Image recognition device, image recognition system, image recognition program, and image recognition method

Abstract

To provide an image recognition device capable of determining whether a hand in a finger area is left or right from a captured image, even without prior hand registration processing or use of a controller. [Solution] The image recognition device has a detection means for detecting an area containing fingers in a captured image, an evaluation value acquisition means for acquiring an evaluation value indicating the likelihood that the area is an area containing the fingers, an angle acquisition means for acquiring information on the orientation of the arm of the hand included in the area, a first discrimination means for determining whether the hand included in the area is left-handed or right-handed based on multiple pieces of information including information on the orientation of the arm, and a second discrimination means for determining whether an area among the multiple areas detected by the detection means is left-handed or right-handed if the discrimination results by the first discrimination means for the multiple areas detected by the detection means are the same.

Description

[Technical Field]

[0001] The present invention relates to an image recognition device that detects an area of ​​an object or the like from an image. [Background technology]

[0002] In a device such as a head-mounted display (hereinafter referred to as HMD) equipped with a camera, a finger area may be detected from a captured image (captured image) and used for operations such as hand gestures. At this time, information on whether the detected finger area is the right hand or the left hand may also be used. Patent Document 1 discloses a method of registering the left and right hands of a person wearing an HMD in advance and distinguishing between the left and right hands through image recognition, and a method of distinguishing between the left and right hands based on pre-registered information on which hand a controller is worn. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2023-143634 Summary of the Invention [Problem to be solved by the invention]

[0004] However, while the technology disclosed in Patent Document 1 is capable of detecting finger areas and distinguishing between left and right hands, it requires prior registration of the left and right hands, as well as controller usage status and controller wearing information, which places restrictions on how the HMD can be used.

[0005] The present invention has been made in consideration of these problems, and aims to provide an image recognition device that can distinguish between the left and right hands in the finger area from a captured image, even without prior hand registration processing or the use of a controller. [Means for solving the problem]

[0006] In order to achieve the above object, the image recognition device of the present invention is characterized by having: a detection means for detecting an area containing fingers in a captured image; an evaluation value acquisition means for acquiring an evaluation value indicating the likelihood that the area is an area containing the fingers; an angle acquisition means for acquiring information on the orientation of the arm of the hand included in the area; a first discrimination means for discriminating between left and right of the hand included in the area based on a plurality of pieces of information including information on the orientation of the arm; and a second discrimination means for discriminating between left and right of an area among the plurality of areas whose evaluation value is higher than a first threshold value when the discrimination results by the first discrimination means for the plurality of areas detected by the detection means are the same. [Effects of the Invention]

[0007] According to the present invention, it is possible to determine whether the hand in the finger area is left or right from a captured image, even without prior hand registration processing or use of a controller. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an internal configuration of an image recognition device according to a first embodiment of the present invention. [Figure 2] 1 is a diagram schematically illustrating an example of image data processed by an image recognition device according to a first embodiment of the present invention. [Figure 3] FIG. 2 is a flowchart illustrating the operation of the image recognition device according to the first embodiment of the present invention. [Figure 4] FIG. 2 is a diagram schematically illustrating an example of data stored in the image recognition device according to the first embodiment of the present invention. [Figure 5] 2 is a diagram schematically illustrating an example of image data acquired by an image recognition device according to a first embodiment of the present invention. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0009] Each embodiment will be described below with reference to the drawings. Identical or equivalent components, members, and processes shown in each drawing will be assigned the same reference numerals, and redundant descriptions will be omitted where appropriate. Also, some components, members, and processes will be omitted in each drawing. The following embodiments do not limit the present invention, and not all combinations of features described in the present embodiments are necessarily essential to the solution of the present invention. The configurations of the embodiments may be modified or changed as appropriate depending on the specifications of the device to which the present invention is applied and various conditions (such as usage conditions and usage environment). In the following embodiments, identical components will be described with the same reference numerals.

[0010] (First embodiment) <Internal configuration of image recognition device> 1 is a block diagram illustrating a head-mounted display (HMD) 100, which is an image recognition device according to a first embodiment. In the first embodiment, an HMD is assumed as an example of a device constituting the image recognition device, but the present invention is not limited to these embodiments. For example, the image recognition device may be a smartphone or tablet terminal equipped with a camera, or may be other devices such as a personal computer (PC) or a digital camera.

[0011] The HMD 100 is a head-mounted display device (electronic device) that can be worn on the head of a user. The HMD 100 includes a camera for capturing an image of the area in front of the user and a display for displaying the image to the user. The display of the HMD 100 displays a composite image that combines an image captured by the HMD 100 of the area in front of the user with content such as CG (computer graphics) in a format that corresponds to the posture of the HMD 100. This allows the user to experience virtual reality or mixed reality with their eyes.

[0012] The internal configuration of the HMD 100 will be described with reference to Fig. 1. In the HMD 100, a control unit 101, a ROM 102, a RAM 103, and an imaging unit 104 are connected to a system bus 105. The control unit 101, the ROM 102, the RAM 103, the imaging unit 104, and the system bus 105 are hardware resources that make up the HMD 100. Note that each component of the HMD 100 may be an image recognition system configured with individual hardware.

[0013] The control unit 101 controls each unit of the HMD 100 in accordance with input signals and programs described below. The control unit 101 has at least one CPU that executes programs stored in a ROM 102 and at least one other circuit. Note that instead of the control unit 101 controlling the entire device, the entire device may be controlled by multiple pieces of hardware sharing the processing.

[0014] The ROM 102 is an electrically erasable and recordable non-volatile memory, and stores programs and the like to be executed by the control unit 101. When the power is turned on to the HMD 100, the control unit 101 reads the programs from the ROM 102 and starts controlling the HMD 100. The ROM 102 is made up of, for example, a flash memory.

[0015] The RAM 103 is a storage unit used as a work area by programs executed by the control unit 101. The RAM 103 is made up of, for example, a volatile memory (DRAM) using a semiconductor element.

[0016] Although the control unit 101, ROM 102, and RAM 103 have been described as separate hardware resources, these functions may be integrated into a single LSI.

[0017] The imaging unit 104 is composed of a stereo camera, and captures color or monochrome images of a scene using the two cameras mounted on the left and right sides, and outputs the video signals to the system bus 105. This video signal undergoes various image processing by the control unit 101, and is stored in the RAM 103 as left and right image data. The imaging unit 104 is composed of, for example, an optical system that controls an optical lens unit and aperture, zoom, focus, etc., and an imaging element that converts the light (image) introduced through the optical lens unit into an electrical video signal. The imaging element generally used is a CMOS imaging element (CMOS image sensor) that uses a CMOS, or a CCD imaging element (CCD image sensor) that uses a CCD.

[0018] <Flow for determining whether an area containing fingers is left or right> A flowchart illustrating the operation of the HMD 100 according to the first embodiment will be described with reference to FIGS. 2(a), 2(b), 2(c), 3, 4(a), and 4(b). The process illustrated in this flowchart is implemented by the control unit 101 of the HMD 100 controlling each unit of the HMD 100 in accordance with input signals and programs. Specifically, this process is implemented by the control unit 101 expanding a program stored in the ROM 102 into the RAM 103 and executing it. The process illustrated in the flowchart of FIG. 3 is executed when the user launches an application on the HMD 100 and is executed each time the imaging unit 104 acquires an image (each time an image is captured). The application is, for example, an application selected by the user on the home screen (home space) after launching the HMD 100, and includes an app that allows the user to interact with a virtual object using hand gestures. The timing of execution of this flowchart is not limited to when the user launches an application on the HMD 100. For example, the timing may be when the user launches the HMD 100 or when a virtual object is displayed in the mixed reality space.

[0019] 2(a), 2(b), and 2(c) are diagrams schematically illustrating an example of image data processed by the image recognition device of the first embodiment. In FIG. 2(a), an image 200 is an example of image data generated by the control unit 101 from a video signal acquired by the imaging unit 104, and the image 200 includes a hand 201 of the HMD wearer. Further, FIGS. 2(b) and 2(c) will be described later.

[0020] In step S301, the control unit 101 acquires the video signal output from the imaging unit 104 to the system bus 105, generates image data, and proceeds to step S302. For example, the control unit 101 stores the image 200 in FIG.

[0021] In step S302, the control unit 101 performs a process of detecting human fingers on the image data generated in step S301 and a process of classifying gestures indicated by the detected fingers, and then proceeds to step S303. The process of detecting human fingers is a region detection process that detects a region of a predetermined shape (hereinafter referred to as an object region) that is estimated to include fingers. For example, a rectangle is an example of the predetermined shape. Furthermore, the region of such a predetermined shape is detected as the smallest predetermined shape that is estimated to include fingers. Note that, depending on the image, the rectangular region estimated to include fingers may not be the smallest region. In the process of detecting human hands, depending on the appearance of the subject or the like included in the image data, only one object region, multiple object regions, or no object region at all may be detected. Note that the process of detecting human hands may be realized, for example, using a trained deep learning model or a rule-based algorithm. Furthermore, in the first embodiment, the predetermined shape of the object region is rectangular, but the predetermined shape is not necessarily limited to rectangular. It may be, for example, a polygon, a circle, an ellipse, or the like, as long as all object regions have the same shape. Furthermore, the area detection process and the process of classifying the gestures indicated by the detected fingers may be performed simultaneously by the same means, or may be performed sequentially, or may be performed by different means.

[0022] Here, referring to FIG. 2(b), a diagram in which the detected regions 211 and 221 are superimposed on the image 200 will be described as an example of the results of executing the process of step S302 on the image 200 of FIG. 2(a) in the first embodiment. Here, the coordinate system in the captured image is defined as having the upper left corner of the image 200 as the origin, the vertical downward direction as the y-axis, and the horizontal rightward direction as the x-axis. As coordinates on the captured image, the coordinates of the center of the rectangle of the region 211 are (cx1, cy1), and the coordinates of the center of the rectangle of the region 221 are (cx2, cy2). In the example of FIG. 2(b), the region 221 is a region where a space that is not a human finger has been mistakenly detected. In FIG. 2(b), the center coordinates of the region 211 are (cx1, cy1) = (75, 55), and the center coordinates of the region 221 are (cx2, cy2) = (85, 10).

[0023] In step S303, the control unit 101 checks whether or not there is an area detected in step S302, and if there is an area detected, the process proceeds to step S304, and if there is no area detected, the process proceeds to step S324.

[0024] Steps S304 to S309 are loop processes for estimating the skeleton, obtaining the arm orientation, and evaluating the region for all regions detected in step S302.

[0025] In step S304, the control unit 101 selects one area that has not yet undergone the above-described loop processing from among all areas detected in step S302, and starts the above-described loop processing.

[0026] In step S305, the control unit 101 performs a position detection process (skeleton estimation process) to detect detection target points in the area selected in step S304, and then proceeds to step S306. In the position detection process, the control unit 101 detects the positions of each joint point including the fingertip of each finger, the wrist point, and the wrist sub-point as detection target points in two-dimensional coordinates. Here, the wrist point refers to the wrist joint point, and the wrist sub-point refers to any point (specific detection target point) that exists between the wrist point and the elbow joint and within the area where the position detection process was performed. In the first embodiment, the coordinates detected by the control unit 101 in the position detection process are two-dimensional coordinates, but the detected coordinates may be three-dimensional coordinates, or some may be two-dimensional coordinates and the rest may be three-dimensional coordinates. Furthermore, the position detection process may be realized, for example, using a trained deep learning model or a rule-based algorithm.

[0027] 2(c), a diagram will be described in which the processing target area and the positions of the detected joint points, wrist point, and wrist-under point are superimposed on the image 200 as an example of the results of executing the processing of step S305 on the image 200 of FIG. 2(a) in the first embodiment. The black circles indicate the positions of the joint points detected within the detected area. The black circles in area 211 include a wrist point 214 and a wrist-under point 215, and the black circles in area 221 include a wrist point 224 and a wrist-under point 225. The coordinates of the wrist point 214 are (wx1, wy1), the coordinates of the wrist-under point 215 are (ux1, uy1), the coordinates of the wrist point 224 are (wx2, wy2), and the coordinates of the wrist-under point 215 are (ux2, uy2). Also, let (wx1, wy1) = (80, 60), (ux1, uy1) = (85, 70), (wx2, wy2) = (85, 15), and (ux2, uy2) = (90, 23). Note that area 221 does not actually include a hand and is an area that was mistakenly detected, but the area and the joint points within the area have been detected.

[0028] In step S306, the control unit 101 executes angle acquisition processing to acquire, as the arm direction, the angle of the arm based on the two-dimensional coordinates of the positions of the wrist point and the wrist subpoint detected in step S305, and proceeds to step S307.

[0029] Here, with reference to Fig. 2(c), the angle of the arm calculated from the captured image will be described. In image 200 in Fig. 2(c), control unit 101 determines the angle between a line connecting wrist point 214 and wrist-bottom point 215 and a line extending vertically from wrist-bottom point 215 as the angle of the arm of the hand detected in region 211. Similarly, the angle between a line connecting wrist point 224 and wrist-bottom point 225 and a line extending vertically from wrist-bottom point 225 as the angle of the arm of the hand detected in region 221. As described above, the position coordinates of the wrist joint point in object region 211 are (wx1, wy1) = (80, 60), and the position coordinates of the wrist-bottom point in object region 211 are (ux1, uy1) = (85, 70). Furthermore, the position coordinates of the wrist joint point in the object area 221 are (wx2, wy2) = (85, 15), and the position coordinates of the wrist bottom point in the object area 221 are (ux2, uy2) = (90, 23). Therefore, the arm angle 231 is arctan((ux1-wx1) / (uy1-wy1)) = 30.0°, and the arm angle 232 is arctan((ux2-wx2) / (uy2-wy2)) = 38.68°.

[0030] In step S307, the control unit 101 stores the center coordinates of the area detected in step S302 in the RAM 103 as information indicating the position of the area, and then the process proceeds to step S308.

[0031] In the example of Figure 2(b), the control unit 101 stores the center coordinates (cx1, cy1) = (75, 55) of the area 211 and the center coordinates (cx2, cy2) = (85, 10) of the area 221 in the RAM 103 together with the time when step S301 was executed.

[0032] Here, with reference to FIG. 4(a), the center coordinates and left / right discrimination information of the detected regions stored in RAM 103 will be described. FIG. 4(a) shows an example of the center coordinates and left / right discrimination information of the detected regions stored in RAM 103. Time t is the time when image 200 is acquired in step S301, and time t-1 is the time when the image data immediately before image 200 is acquired in step S301. Regions 1 and 2 are regions detected in step S302 from the image data acquired at that time, and their center coordinates and left / right discrimination information acquired by a discrimination means (described later) are stored. In FIG. 4(a), image 200 shown in FIG. 2(a) is acquired in step S301 at time t, and region 211 is designated region 1, with center coordinates (cx1, cy1) = (75, 55) stored. Furthermore, region 221 is designated region 2, with center coordinates (cx2, cy2) = (85, 10) stored. In addition, the center coordinates (74, 53) are stored for area 1 detected from the image acquired at time t-1, and the center coordinates (82, 10) are stored for area 2 detected from the image acquired at time t-1.

[0033] In step S308, control unit 101 acquires the center coordinates of previously detected areas from RAM 103, executes evaluation value acquisition processing to acquire an evaluation value indicating the likelihood that the area selected in step S304 is an area containing fingers, and proceeds to step S309. Control unit 101 uses the distance between the area selected in step S304 and the center coordinates of previously detected areas, and the position of the area selected in step S304 with respect to the image data acquired in S301, as criteria for the likelihood that the area is an area containing fingers.

[0034] Specifically, the control unit 101 first calculates the distance between the center coordinate of the region selected in step S304 and the center coordinate of all regions at the immediately preceding time t−1 acquired from the RAM 103. If any of the calculated distances is equal to or less than a predetermined threshold, the control unit 101 determines that the center coordinate of the region selected in step S304 is close to the center coordinate of the corresponding region at time t−1. In the first embodiment, the predetermined threshold for determining that the distance between the center coordinates of regions is close is set to thr_ct=5 pixels. If the region selected in step S304 is close to any region, the control unit 101 then determines whether the region selected in step S304 fits within the captured image acquired in step S301. If the control unit 101 determines that the region selected in step S304 fits within the captured image acquired in step S301, the control unit 101 assigns a high evaluation value to the region. On the other hand, if the control unit 101 does not determine that the region selected in step S304 fits within the captured image acquired in step S301, the control unit 101 assigns a low evaluation value. For example, if the captured image does not include the entire hand but only part of the hand, the process may detect an area that extends beyond the captured image. In such a case, the detected area does not fit within the captured image, and a low evaluation value is assigned. Note that if the distance between the center coordinates of the area selected in step S304 and the center coordinates of all areas at the previous time t-1 acquired from RAM 103 is not equal to or less than a predetermined threshold, a low evaluation value is assigned. In other words, it is determined that the area selected in step S304 is not in the vicinity of all areas at time t-1, and a low evaluation value is assigned. In addition, in the first embodiment, a high evaluation value is 1.0 and a low evaluation value is 0.0.

[0035] In the first embodiment, the previously detected area is the area detected in the image data acquired at time t-1, but it may also be an area detected at an earlier time, or it may be an area detected at any one of multiple past times, or all areas previously detected and stored in RAM 103.

[0036] In the first embodiment, the conditions for assigning a high evaluation value are that the center coordinates of the area selected in step S304 are close to the center coordinates of the previously detected area, and that the area selected in step S304 is within the image data. However, a high evaluation value may be assigned if only one of these conditions is met, and a low evaluation value may be assigned to the area if not. The evaluation value acquired in step S308 may be a binary value or a continuous value. The processing of step S308 may be performed before steps S305 and S306.

[0037] In the example of FIG. 4(a), when region 211 is selected in step S304, the center coordinates 212, 213 of region 211 are (cx1, cy1) = (75, 55), as described above. Of the regions detected at time t-1, the Euclidean distance to region 1, whose center coordinates are (74, 53), is √5 pixels, and the Euclidean distance to region 2, whose center coordinates are (82, 10), is √2074 pixels. Because the distance between region 211 and region 1 at time t-1 is less than or equal to thr_ct, the control unit 101 determines that region 211 is in the vicinity of region 1 detected at time t-1. Furthermore, region 211 is contained within the image data. Therefore, the control unit 101 assigns a high evaluation value of 1.0 to region 211.

[0038] Furthermore, when region 221 is selected in step S304, the center coordinates 222, 223 of region 221 are (cx2, cy2) = (85, 10). Of the regions detected at time t-1, the Euclidean distance to region 1, whose center coordinates are (74, 53), is √1970 pixels, and the Euclidean distance to region 2, whose center coordinates are (82, 10), is 3 pixels. Because the distance between region 221 and region 2 at time t-1 is less than or equal to thr_ct, region 221 is near region 2 detected at time t-1. Furthermore, as shown in FIG. 2, region 221 is contained within image 200. Therefore, control unit 101 assigns a high evaluation value of 1.0 to region 221.

[0039] Here, if there is no area that is close to the area 221 among the areas detected at time t-1, the control unit 101 will assign the low evaluation value 0.0 to the area 221.

[0040] In step S309, if there are any areas detected in step S302 that have not yet been selected in step S304, the control unit 101 continues the loop processing, and if processing of all areas has been completed, ends the loop processing and proceeds to step S310.

[0041] In step S310, the control unit 101 performs a first determination process for all regions detected in step S302 to determine whether the fingers included in the region are left-handed or right-handed, and then proceeds to step S311. In the determination process in step S310, the control unit 101 performs left-handed determination based on the arm orientation information acquired in step S307, the center coordinate information of the region acquired in step S302, and the relative position information between the regions. Then, all results are combined to determine the left-handed determination result as the first determination process. Here, relative position information refers to information regarding the relative positional relationship between the regions. For example, the distance between the center coordinates of two regions or the difference in x-coordinates between the center coordinates of two regions corresponds to relative position information.

[0042] In the determination process based on the arm orientation, the control unit 101 provisionally determines the arm orientation as a right hand if the arm orientation information acquired in step S306 is at an angle greater than or equal to the minimum value thr_deg_min and less than or equal to the maximum value thr_deg_max relative to the vertical direction. If the angle is greater than or equal to the minimum value -thr_deg_max and less than or equal to the maximum value -thr_deg_min relative to the vertical direction, the control unit 101 provisionally determines the arm orientation as a left hand. Otherwise, the control unit 101 determines that the arm orientation cannot be determined as either a right hand or a left hand. In the first embodiment, thr_deg_min=10° and thr_deg_max=150°.

[0043] As a discrimination process based on the center coordinates of the region, the control unit 101 provisionally discriminates the hand as a left hand when the center coordinates of the region are located to the left of the half of the image data acquired in step S301, and provisionally discriminates the hand as a right hand otherwise.

[0044] In the left-right discrimination process based on the positions of the regions, when there are multiple regions detected in step S302, if the center coordinates of the regions are horizontally separated by a threshold value thr_rel or more, the control unit 101 provisionally determines that the right region is a right hand and the left region is a left hand. Otherwise, it is determined that it is impossible to distinguish between a right hand and a left hand. In the first embodiment, thr_rel=30 pixels.

[0045] For each provisional determination result of the three determination processes described above, if the provisional determination result shows that there are more left hands than right hands, the control unit 101 determines that the determination result is left hand, and if not, determines that the determination result is right hand.

[0046] In the examples of FIGS. 2(c) and 4(a), based solely on the arm orientation information, region 211 is provisionally determined to be a right hand because the arm orientation angle is 30.0°, as described above, greater than the minimum value thr_deg_min and less than the maximum value thr_deg_max. Based solely on the information about the center coordinates of the region, region 211 is provisionally determined to be a right hand because it is not located to the left of the half of the image data, as shown in FIG. 2. Based solely on the relative position information between the regions, region 211 is provisionally determined to be a left hand because the horizontal center coordinate of region 211 is 75, which is located more than thr_rel to the left of the horizontal center coordinate of region 221, 85. The provisional determination results of each of the three determination processes are right hand, right hand, and left hand, and since there are more right hands, region 211 is determined to be a right hand.

[0047] Based only on the arm orientation information, region 221 is provisionally determined to be a right hand because the arm orientation angle is 38.68° as described above, which is greater than the minimum value thr_deg_min and less than the maximum value thr_deg_max. Based only on the information about the region's center coordinate, region 221 is provisionally determined to be a right hand because it does not exist to the left of the halfway point of the image data, as shown in FIG. 2. Based only on the relative position information between regions, region 221 is provisionally determined to be a right hand because its horizontal center coordinate is located thr_rel or more to the right of the horizontal center coordinate of region 211. The provisional determination results are right hand, right hand, right hand, and since there are more right hands, region 221 is determined to be a right hand.

[0048] In step S311, the control unit 101 checks whether the results of the first discrimination process in step S310 for all areas detected in step S302 overlap, and if they overlap, proceeds to step S312, and if they do not overlap, proceeds to step S323.

[0049] In the examples of FIG. 2(c) and FIG. 4(a), the results of the first discrimination process in step S310 are both determined to be the right hand, which is a duplicate, so the control unit 101 proceeds to step S312.

[0050] Steps S312 to S315 are loop processes for determining left and right again for all areas detected in step S302 according to the evaluation value of the area.

[0051] In step S312, the control unit 101 selects one area from all areas detected in step S302 for which the loop process from step S312 to step S315 has not yet been performed, and starts the loop process from step S312 to step S315.

[0052] In step S313, the control unit 101 checks whether the evaluation value acquired in step S308 for the region selected in step S312 is equal to or greater than a predetermined value, and if so, proceeds to step S314, and if less than the predetermined value, proceeds to step S315. In the first embodiment, the predetermined value is set to thr_e=0.5.

[0053] 2(c), as described above, an evaluation value of 1.0 is assigned to both area 211 and area 221 in step S308. Therefore, since both are equal to or greater than thr_e, the control unit 101 proceeds to step S314 regardless of whether area 211 or area 221 is selected in step S312.

[0054] In step S314, the control unit 101 performs second determination processing to determine whether the region selected in step S312 is left or right based on the arm orientation acquired in step S306, and then proceeds to step S315. In the first embodiment, the control unit 101 performs processing similar to the determination processing based on the arm orientation in step S310 to determine whether the region is left or right. In the first embodiment, thr_deg_min=5° and the maximum value thr_deg_max=160°. In the first embodiment, the values ​​of thr_deg_min and thr_deg_max are different between S310 and S314, but they may be the same value, or only one of them may be the same.

[0055] 2(c), when region 211 is selected in step S312, the angle of the arm direction in region 211 is 30.0° as described above, which is greater than or equal to the minimum value thr_deg_min and less than or equal to the maximum value thr_deg_max, so region 211 is determined to be a right hand. Also, when region 221 is selected in step S312, the angle of the arm direction in region 221 is 38.68° as described above, which is greater than or equal to the minimum value thr_deg_min and less than or equal to the maximum value thr_deg_max, so it is determined to be a right hand.

[0056] In step S315, if there are any areas detected in step S302 that have not yet been selected in step S312, the control unit 101 continues the loop processing, and if processing of all areas has been completed, ends the loop processing and proceeds to step S316.

[0057] In step S316, if the control unit 101 determines that there is only one area, among the areas detected in step S302, for which the evaluation value acquired in step S308 is less than a predetermined value, the process proceeds to step S317. On the other hand, if the control unit 101 does not determine that there is only one area, among the areas detected in step S302, for which the evaluation value acquired in step S308 is less than a predetermined value, the process proceeds to step S318. In the first embodiment, the predetermined value is thr_e=0.5, which is the same as in step S313, as described above.

[0058] 2(c), as described above, an evaluation value of 1.0 is assigned to both the area 211 and the area 221 in step S308. Therefore, since both are equal to or greater than thr_e, the process proceeds to step S318.

[0059] In step S317, the control unit 101 determines whether the area detected in step S302 is a left-right area for the area not processed in step S314 based on the determination result of the area processed in step S314, and proceeds to step S323. In the first embodiment, the control unit 101 determines either the left or right side, which does not overlap with the determination result of the area processed in step S314, as the determination result for the area not processed in step S314. For example, suppose two areas are detected in step S302, one of which has an evaluation value of 1.0 and the other has an evaluation value of 0.0. In this case, if the left and right sides of the area with an evaluation value of 1.0 are determined to be the right hand in S314, the control unit 101 determines in step S317 the area with an evaluation value of 0.0 to be the left hand. When determining whether the area is a left-right area in this manner, the left-right determination result based on the arm orientation is prioritized for the area with a higher evaluation value, and the left-right determination result for the area with a lower evaluation value is determined according to the left-right determination result of the area with the higher evaluation value. This reduces the number of cases where the left / right discrimination result of an area with a low evaluation value (i.e., a relatively high possibility of erroneous detection) results in an incorrect discrimination of the left / right of an area with a high evaluation value (i.e., a relatively low possibility of erroneous detection).

[0060] In step S318, the control unit 101 checks whether the results of the second determination process in step S314 for all areas detected in step S302 overlap, and if there is an overlap, the process proceeds to step S319. If there is no overlap, the process proceeds to step S323. That is, if two areas are detected in step S318 from the captured image and the two areas correspond to one right hand and one left hand, the process proceeds to step S323. Also, if two areas are detected in step S318 from the captured image and both areas are determined to be the right hand or both left hand, the process proceeds to step S319. Also, if three or more areas are detected in step S318 from the captured image, either the right hand or the left hand overlaps, and the process proceeds to step S319.

[0061] In the examples of FIGS. 2(c) and 4(a), the result of the second determination process in step S314 is that both the area 211 and the area 221 overlap with the right hand, and therefore the control unit 101 proceeds to step S319.

[0062] Steps S319 to S322 are loop processes for further distinguishing left from right for all areas detected in step S302 using the information on the areas detected in S302 at time t-1.

[0063] In step S319, control unit 101 selects one area that has not yet undergone the loop process from step S319 to step S322 from among all areas detected in step S302, and starts the loop process from step S319 to step S322.

[0064] In step S320, if the control unit 101 determines that the area selected in step S319 is near any area by the processing in step S308, the control unit 101 proceeds to step S321, and if not, the control unit 101 proceeds to step S322.

[0065] 2(c), as described above, in the process of step S308, it is determined that region 211 is in the vicinity of region 1 at time t-1, and region 221 is in the vicinity of region 2 at time t-1. Therefore, regardless of the region selected in step S319, the control unit 101 proceeds to step S321.

[0066] In step S321, the control unit 101 performs a third discrimination process in which the left-right information of the area at time t-1 that was determined to be in the vicinity of the area selected in step S319 is used as the left-right discrimination result for the area selected in step S319, and then proceeds to step S322.

[0067] In the examples of FIGS. 2(c) and 4(a), control unit 101 determines in step S308 that the vicinity of region 211 is region 1 at time t-1, and the left-right determination result of region 1 at time t-1 stored in RAM 103 is a right hand. Therefore, when region 211 is selected in step S319, control unit 101 determines the right hand as the left-right determination result of region 211, and proceeds to step S322. Similarly, the vicinity of region 221 is region 2 at time t-1, and the left-right determination result of this region is a left hand. Therefore, when region 221 is selected in step S319, control unit 101 determines the left hand as the left-right determination result of region 221, and proceeds to step S322.

[0068] In step S322, if there are any areas detected in step S302 that have not yet been selected in step S319, the control unit 101 continues the loop processing, and if processing of all areas has been completed, ends the loop processing and proceeds to step S323.

[0069] The second determination process and the third determination process may be interchanged, that is, the process of step S314 and the processes from step S320 to step S321 may be interchanged.

[0070] In step S323, the control unit 101 stores the center coordinates indicating the position of the area detected in step S302 and the final left / right discrimination result of that area in the RAM 103, and then the process proceeds to step S324.

[0071] 4(b) shows an example of the center coordinates and left-right discrimination information of the detected regions stored in RAM 103 as a result of the processing of step S323 for the image acquired at time t. In addition to the information in FIG. 4(a), the final left-right discrimination result of region 211 detected in step S301 is linked to time t and stored in step S323 as left-right discrimination information for region 1 at time t. Similarly, the final left-right discrimination result of region 221 detected in step S301 is linked to time t and stored in step S323 as left-right discrimination information for region 2 at time t.

[0072] In step S324, the control unit 101 determines whether to end the processing of the flowchart in Fig. 3. For example, if the user inputs an end instruction via an operation device (not shown) or if the control unit 101 is unable to acquire image data, the control unit 101 determines to end the processing of Fig. 3. On the other hand, if it is determined not to end the processing in step S313, the control unit 101 returns the processing to step S301.

[0073] As described above, in this embodiment, the control unit 101 detects an area including the fingers of a person, which is the detection target object, from the image data. Next, it detects the coordinates of the skeleton including the wrist point and the sub-wrist point within that area, and obtains the arm orientation of the area. It determines whether the fingers are left or right based on the arm orientation of the area and the position of the area, and if necessary, determines whether the fingers are left or right based on the arm orientation of an area with a high evaluation value. Furthermore, if necessary, it uses the left or right determination result of an area detected in the past that is located nearby as the left or right determination result for the area. This makes it possible to determine whether the hand is left or right in the finger area from the captured image even without prior hand registration processing or use of a controller.

[0074] Furthermore, there are cases where an area that does not actually include a hand but is detected (an area that is actually erroneously detected) and an area that actually includes a hand (an area that is correctly detected) are detected from the captured image. In such cases, if left-right discrimination is performed for each area, there is a risk that the area that is actually erroneously detected will be mistakenly determined as left-right for the area that is correctly detected. Therefore, by following the above-described flowchart, even if an area that is actually erroneously detected and an area that is correctly detected are detected from the captured image, it is possible to prevent an error in the left-right discrimination for the area that is correctly detected. In other words, it is possible to prevent an error in the left-right discrimination even when an area that actually includes a hand is detected due to the left-right discrimination result of an area that is actually erroneously detected.

[0075] <Explanation of the correspondence between images acquired in chronological order and the flowchart> The correspondence between images acquired in time series and the flowchart will be described with reference to FIGS. 5(a), 5(b), and 2(c). As described above, FIG. 2(c) shows image 200 acquired at time t. FIG. 5(b) shows image 600, acquired at time t-1, which is the image acquired immediately before the image acquired at time t. FIG. 5(a) shows image 500, acquired at time t-2, which is the image acquired immediately before the image acquired at time t-1. That is, the time series is time t-2, time t-1, and time t, and the images are acquired in the order of image 500, image 600, and image 200. Here, FIGS. 5(a) and 5(b) show images acquired in step S301 and subjected to the processes up to step S309. That is, the area including the hand has been detected, and joint points including the fingertips, finger joints, and wrist have been detected as detection target points, and the arm orientation has been acquired. The positions of the regions are stored in the RAM 103, and the images in which the evaluation of each region is also performed are shown.

[0076] First, in step S301, when image 500 is acquired at time t-2, region 511 is detected in the processing of step S303, and therefore the process proceeds to step S304. Furthermore, since only region 511 is detected from image 500, the processing from step S304 to step S309 is performed only once. Here, it is assumed that in step S306, an arm orientation that is provisionally determined to be a right hand is acquired in region 511. Furthermore, it is assumed that (cx, cy) = (73, 51) is stored in RAM 103 as the center coordinates of region 511 in step S307. Furthermore, it is assumed that (wx, wy) = (78, 56) is acquired as the coordinates of wrist joint point 514, and (ux, uy) = (83, 66) is acquired as the coordinates of wrist-bottom point 515. That is, arm angle 531 is arctan((ux-wx) / (uy-wy))=30.0°, and in step S310, the hand included in region 511 is determined to be a right hand based on information about the arm orientation, etc. In the processing of step S311, only region 511 is detected from image 500, so the process proceeds to step S323, and the position of region 511 and the fact that the hand detected in region 511 is a right hand are stored in RAM 103.

[0077] Next, in step S301, image 600 is acquired at time t−1. In the processing of step S303, area 611 and area 621 are detected, and therefore the processing proceeds to step S304. Here, area 611 is a correctly detected area that actually includes a hand, and area 621 is an area that does not actually include a hand and is actually a false detection. Furthermore, since area 611 and area 621 are detected from image 600, the processing from step S304 to step S309 is performed twice. Here, it is assumed that in step S306, arm orientations are acquired in area 611 and area 621 such that both are provisionally determined to be right hands. Furthermore, it is assumed that in step S307, (cx, cy)=(74, 53) are stored in RAM 103 as the center coordinates of area 611, and (cx, cy)=(82, 10) are stored in RAM 103 as the center coordinates of area 621. Then, in step S308, by referring to the center coordinates (cx, cy) = (73, 51) of region 511 in image 500, region 611 is determined to be in the vicinity of region 511. Furthermore, since region 611 is contained within image 600, a high evaluation value of 1.0 is assigned to region 611. Furthermore, region 621 is determined not to be in the vicinity of region 511, and a low evaluation value of 0.0 is assigned to region 621.

[0078] Next, in step S310, it is determined that both of the hands included in region 611 and region 621 are right hands based on information about the arm orientation, etc. Specifically, it is assumed that in region 611, the coordinates of wrist joint point 614 are (wx, wy) = (79, 58) and the coordinates of wrist-bottom point 615 are (ux, uy) = (84, 68). That is, arm angle 631 is arctan((ux - wx) / (uy - wy)) = 30.0°, and in step S310, it is determined that the hand included in region 611 is right based on information about the arm orientation, etc. Furthermore, it is assumed that in region 621, the coordinates of wrist joint point 624 are (wx, wy) = (82, 15) and the coordinates of wrist-bottom point 625 are (ux, uy) = (87, 23). That is, arm angle 632 is arctan((ux-wx) / (uy-wy))=38.68°, and in step S310, the hand included in region 621 is determined to be a right hand based on information about the arm orientation, etc. In other words, in the process of step S311, the first determination process of step S310 would be repeated, so the process proceeds to step S312. Here, the processes of steps S312 to S315 are performed twice. When region 611 is selected in step S312, an evaluation value of 1.0 is assigned, so the process proceeds to step S314, where second determination process is performed to determine that the hand is a right hand based on information about the arm orientation. When region 621 is selected in step S312, an evaluation value of 0.0 is assigned, so the second determination process of step S314 is not performed, and the loop process from step S312 to step S315 ends. Then, in the processing of step S316, there is only one area, area 621, whose evaluation value is less than the predetermined value, so the process proceeds to step S317. In the processing of step S317, area 611 has been determined to be a right hand by the second determination processing, so area 621 is determined to be a left hand. Then, in step S323, the position of area 611 and the fact that the hand detected in area 611 is a right hand, and the position of area 621 and the fact that the hand detected in area 621 is a left hand are stored in RAM 103.

[0079] Next, in step S301, image 200 is acquired at time t. As described above, in the second determination process in step S314, it is determined that both area 211 and area 221 are the right hand based on the arm orientation. Then, via the processes in steps S316 to S318, the loop process from step S319 to step S322 is performed.

[0080] (Other embodiments) The present invention can also be realized by executing the following process: software (program) that realizes the functions of the above-described embodiments is supplied to a system or device via a network or various storage media, and the computer (or control unit, MPU, etc.) of the system or device reads and executes the program code. In this case, the program and the storage medium storing the program constitute the present invention.

[0081] While the present invention has been described in detail above based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0082] Note that each functional unit in each of the above embodiments (variations) may or may not be individual hardware. The functions of two or more functional units may be realized by common hardware. Each of multiple functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Furthermore, each functional unit may or may not be realized by hardware such as an ASIC, FPGA, or DSP. For example, an apparatus may have a processor and a memory (storage medium) in which a control program is stored. Then, the functions of at least some of the functional units of the apparatus may be realized by the processor reading and executing the control program from the memory.

[0083] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0084] In addition, in each of the examples described above, the term "processor" refers to a processor in a broad sense, and includes general-purpose processors (e.g., CPUs) and dedicated processors (e.g., GPUs, ASICs, FPGAs, and programmable logic devices, etc.).

[0085] The disclosure of this embodiment includes the following configuration, method, and program.

[0086] [Configuration 1] An image recognition device for determining whether a hand included in a captured image is left or right, a detection means for detecting an area including fingers in the captured image; an evaluation value acquisition means for acquiring an evaluation value indicating a likelihood that the region includes the finger; An angle acquisition means for acquiring information on the orientation of the arm of the hand included in the area; a first discrimination means for discriminating between left and right of the hand included in the region based on a plurality of pieces of information including information on the orientation of the arm; and second discrimination means for discriminating between left and right of an area having an evaluation value higher than a first threshold value among the plurality of areas when the discrimination results by the first discrimination means for the plurality of areas detected by the detection means are the same. An image recognition device characterized by:

[0087] [Configuration 2] The second discrimination means discriminates between left and right hands included in an area where the evaluation value is higher than the first threshold value based on information about the orientation of the arm in the area. 2. The image recognition device according to configuration 1,

[0088] [Configuration 3] a storage means for storing in a storage unit a position on the first captured image of the first region detected by the detection means and a result of left / right discrimination in association with each other in the first captured image; and a third discrimination means for determining the left / right discrimination result of the first region stored in the storage unit as the left / right discrimination result of the second region when, in a second captured image captured after the first captured image, the left / right discrimination result determined by the second discrimination means between a second region detected by the detection means and a third region different from the second region is the same and the distance between the position of the second region on the second captured image and the position of the first region on the first captured image is smaller than a second threshold value. 3. The image recognition device according to configuration 1 or 2.

[0089] [Configuration 4] a storage unit that stores in a storage unit a position of the first region detected by the detection unit on the first captured image and a result of left / right discrimination in association with each other, The second discrimination means determines the left / right discrimination result of the first area stored in the storage unit as the left / right discrimination result of the second area when a distance between a position of the second area detected by the detection means on the second captured image and a position of the first area on the first captured image is smaller than a second threshold value in a second captured image captured after the first captured image. 4. The image recognition device according to any one of configurations 1 to 3.

[0090] [Configuration 5] The detection means further performs classification of the hand gesture. 5. The image recognition device according to any one of configurations 1 to 4.

[0091] [Configuration 6] a storage unit configured to store the position of the area detected by the detection unit on the captured image in a storage unit; The evaluation value acquisition means increases the evaluation value when a distance between a position on the captured image of the area detected by the detection means and a position of the area stored in the storage unit is smaller than a second threshold value, or when the area detected by the detection means is contained within the captured image. 6. The image recognition device according to any one of configurations 1 to 5.

[0092] [Configuration 7] The apparatus further includes a position detection means for detecting the position of a detection target point for each area detected by the detection means. 7. The image recognition device according to any one of configurations 1 to 6,

[0093] [Configuration 8] The position detection means detects detection target points including at least the position of a wrist joint point and the position of a specific detection target point which is any one point between the position of the wrist joint point and the elbow joint and within a region; The angle acquisition means acquires information about the orientation of the arm based on the position of the wrist joint point and the position of the specific detection target point. 8. The image recognition device according to configuration 7,

[0094] [Configuration 9] The first discrimination means further discriminates whether the hand included in the region is left or right based on information about the position of the region on the captured image or information about the relative positional relationship between the plurality of regions detected by the detection means. 9. The image recognition device according to any one of configurations 1 to 8.

[0095] [Configuration 10] When the discrimination results by the first discrimination means for the plurality of regions detected by the detection means are the same, the second discrimination means does not discriminate between left and right of a region among the plurality of regions whose evaluation value is lower than the first threshold value. 10. The image recognition device according to any one of configurations 1 to 9.

[0096] [Configuration 11] The method further includes a determination unit that determines the left and right sides of the region in which the evaluation value is lower than the first threshold value based on the determination result by the second determination unit. 11. The image recognition device according to configuration 10.

[0097] [Configuration 12] The captured image includes a first region where the evaluation value is higher than the first threshold value and a second region where the evaluation value is lower than the first threshold value, and when the second discrimination means discriminates that the hand included in the first region is a right hand, the determination means determines that the hand included in the second region is a left hand based on the discrimination result by the second discrimination means. 12. The image recognition device according to claim 11,

[0098] [method] An image recognition method for determining whether a hand included in a captured image is left or right, comprising: a detection step of detecting an area including fingers in the captured image; an evaluation value acquisition step of acquiring an evaluation value indicating a likelihood that the region includes the finger; an angle acquisition step of acquiring information on the orientation of the arm of the hand included in the region; a first determination step of determining whether the hand included in the region is left or right based on a plurality of pieces of information including information on the orientation of the arm; a second discrimination step of discriminating between left and right of a region having the evaluation value higher than a first threshold value among the plurality of regions when the discrimination results of the first discrimination step for the plurality of regions detected by the detection step are the same. An image recognition method comprising:

[0099] [program] A program for causing a computer to function as each of the means of the image recognition device according to any one of configurations 1 to 12.

[0100] [system] An image recognition system for determining whether a hand included in a captured image is left or right, a detection device for detecting an area including fingers in the captured image; an evaluation value acquisition device that acquires an evaluation value indicating a likelihood that the region includes the finger; an angle acquisition device that acquires information about the orientation of the arm of the hand included in the area; a first discrimination device that discriminates between left and right hands included in the region based on a plurality of pieces of information including information on the orientation of the arm; and a second discrimination device that, when the discrimination results by the first discrimination device for the plurality of regions detected by the detection device are the same, discriminates between left and right of a region among the plurality of regions whose evaluation value is higher than a first threshold value. An image recognition system characterized by:

Claims

1. An image recognition device for determining whether a hand included in a captured image is left or right, a detection means for detecting an area including fingers in the captured image; an evaluation value acquisition means for acquiring an evaluation value indicating a likelihood that the region includes the finger; An angle acquisition means for acquiring information on the orientation of the arm of the hand included in the area; a first discrimination means for discriminating between left and right of the hand included in the region based on a plurality of pieces of information including information on the orientation of the arm; and second discrimination means for discriminating between left and right of an area having an evaluation value higher than a first threshold value among the plurality of areas when the discrimination results by the first discrimination means for the plurality of areas detected by the detection means are the same. An image recognition device characterized by:

2. The second discrimination means discriminates between left and right hands included in an area where the evaluation value is higher than the first threshold value based on information about the orientation of the arm in the area.

2. The image recognition device according to claim 1.

3. a storage means for storing in a storage unit a position on the first captured image of the first region detected by the detection means and a result of left / right discrimination in association with each other in the first captured image; and a third discrimination means for determining the left / right discrimination result of the first region stored in the storage unit as the left / right discrimination result of the second region when, in a second captured image captured after the first captured image, the left / right discrimination result determined by the second discrimination means between a second region detected by the detection means and a third region different from the second region is the same and the distance between the position of the second region on the second captured image and the position of the first region on the first captured image is smaller than a second threshold value.

3. The image recognition device according to claim 1 or 2.

4. a storage unit that stores in a storage unit a position of the first region detected by the detection unit on the first captured image and a result of left / right discrimination in association with each other, The second discrimination means determines the left / right discrimination result of the first area stored in the storage unit as the left / right discrimination result of the second area when a distance between a position of the second area detected by the detection means on the second captured image and a position of the first area on the first captured image is smaller than a second threshold value in a second captured image captured after the first captured image.

2. The image recognition device according to claim 1.

5. The detection means further performs classification of the hand gesture.

2. The image recognition device according to claim 1.

6. a storage unit configured to store the position of the area detected by the detection unit on the captured image in a storage unit; The evaluation value acquisition means increases the evaluation value when a distance between a position on the captured image of the area detected by the detection means and a position of the area stored in the storage unit is smaller than a second threshold value, or when the area detected by the detection means is contained within the captured image.

2. The image recognition device according to claim 1.

7. The apparatus further includes a position detection means for detecting the position of a detection target point for each area detected by the detection means.

2. The image recognition device according to claim 1.

8. the position detection means detects detection target points including at least the position of a wrist joint point and the position of a specific detection target point which is any one point between the position of the wrist joint point and the elbow joint and within a region; The angle acquisition means acquires information about the orientation of the arm based on the position of the wrist joint point and the position of the specific detection target point.

8. The image recognition device according to claim 7.

9. The first discrimination means further discriminates whether the hand included in the region is left or right based on information about the position of the region on the captured image or information about the relative positional relationship between the plurality of regions detected by the detection means.

2. The image recognition device according to claim 1.

10. When the discrimination results by the first discrimination means for the plurality of regions detected by the detection means are the same, the second discrimination means does not discriminate between left and right of a region among the plurality of regions whose evaluation value is lower than the first threshold value.

2. The image recognition device according to claim 1.

11. The method further includes a determination unit that determines the left and right sides of the region in which the evaluation value is lower than the first threshold value based on the determination result by the second determination unit.

11. The image recognition device according to claim 10.

12. The captured image includes a first region where the evaluation value is higher than the first threshold value and a second region where the evaluation value is lower than the first threshold value, and when the second discrimination means discriminates that the hand included in the first region is a right hand, the determination means determines that the hand included in the second region is a left hand based on the discrimination result by the second discrimination means.

12. The image recognition device according to claim 11.

13. An image recognition method for determining whether a hand included in a captured image is left or right, comprising: a detection step of detecting an area including fingers in the captured image; an evaluation value acquisition step of acquiring an evaluation value indicating a likelihood that the region includes the finger; an angle acquisition step of acquiring information on the orientation of the arm of the hand included in the region; a first determination step of determining whether the hand included in the region is left or right based on a plurality of pieces of information including information on the orientation of the arm; a second discrimination step of discriminating between left and right of a region having the evaluation value higher than a first threshold value among the plurality of regions when the discrimination results of the first discrimination step for the plurality of regions detected by the detection step are the same. An image recognition method comprising:

14. A program for causing a computer to function as each of the means of the image recognition device according to claim 1.

15. An image recognition system for determining whether a hand included in a captured image is left or right, a detection device for detecting an area including fingers in the captured image; an evaluation value acquisition device that acquires an evaluation value indicating a likelihood that the region includes the finger; an angle acquisition device that acquires information about the orientation of the arm of the hand included in the area; a first discrimination device that discriminates between left and right hands included in the region based on a plurality of pieces of information including information on the orientation of the arm; and a second discrimination device that, when the discrimination results by the first discrimination device for the plurality of regions detected by the detection device are the same, discriminates between left and right of a region among the plurality of regions whose evaluation value is higher than a first threshold value. An image recognition system characterized by:

Images