Image recognition device, image recognition system, image recognition program, and image recognition method

Abstract

To provide an image recognition device capable of determining whether a hand in a finger area is left or right from a captured image, even without prior hand registration processing or use of a controller. [Solution] The image recognition device has a detection means for detecting an area containing fingers in a captured image, an evaluation value acquisition means for acquiring an evaluation value indicating the likelihood that the area is an area containing the fingers, an angle acquisition means for acquiring information on the orientation of the arm of the hand included in the area, a first discrimination means for determining whether the hand included in the area is left-handed or right-handed based on multiple pieces of information including information on the orientation of the arm, and a second discrimination means for determining whether an area among the multiple areas detected by the detection means is left-handed or right-handed if the discrimination results by the first discrimination means for the multiple areas detected by the detection means are the same.

Description

Technical Field

[0001] The present invention relates to an image recognition device that detects regions such as objects from images.

Background Art

[0002] In a device such as a head-mounted display equipped with a camera (hereinafter referred to as an HMD), in some cases, a finger region is detected from a captured image (captured image) and used for operations by hand gestures. At this time, information indicating whether the detected finger region is the right hand or the left hand may also be used. Patent Document 1 discloses a method of registering the left and right hands of the wearer of the HMD in advance and discriminating them by image recognition, and a method of discriminating the left and right hands based on information indicating which of the left and right hands the previously registered controller is attached to.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the technology disclosed in Patent Document 1, although it is possible to detect the finger region and discriminate the left and right hands, it is necessary to register the left and right hands in advance or register the usage status of the controller and the attachment information of the controller, which restricts the usage form of the HMD.

[0005] The present invention has been made in view of these problems, and an object of the present invention is to provide an image recognition device that can discriminate the left and right hands of a finger region from a captured image even when there is no prior hand registration process or controller usage.

Means for Solving the Problems

[0006] To achieve the above objective, the image recognition device of the present invention includes: detection means for detecting a region in an captured image that is presumed to contain fingers; evaluation value acquisition means for acquiring an evaluation value indicating the likelihood that the region contains fingers; angle acquisition means for acquiring information on the orientation of the arm of the hand presumed to be included in the region; first discrimination means for determining the left or right of the hand presumed to be included in the region based on the information on the orientation of the arm and the information on the position of the region in the captured image; second discrimination means for determining the left or right of the region whose evaluation value is higher than the first threshold when two regions are detected by the detection means and the discrimination results by the first discrimination means for the two regions are the same; and linking the region and the left or right discrimination result of the region. The device comprises a storage unit and a storage means for storing information in a storage unit, wherein, if two regions are detected by the detection means and the discrimination results of the first discrimination means for the two regions are different, the storage means stores in the storage unit the positions of the two regions and the discrimination results of the first discrimination means for each of the two regions linked together; if two regions are detected by the detection means and the discrimination results of the first discrimination means for the two regions are the same, the storage means stores in the storage unit the left and right discrimination results of the second discrimination means linked together for the region whose evaluation value is higher than the first threshold; and for the region whose evaluation value is lower than the first threshold, the storage means stores in the storage unit the left or right discrimination result of the second discrimination means that differs from the left or right discrimination result of the region whose evaluation value is higher than the first threshold. [Effects of the Invention]

[0007] According to the present invention, even without prior hand registration processing or the use of a controller, it is possible to determine the left or right hand from the captured image of the finger region. [Brief explanation of the drawing]

[0008] [Figure 1] This is a diagram illustrating the internal configuration of an image recognition device according to the first embodiment of the present invention. [Figure 2]This figure schematically represents an example of image data processed by an image recognition device according to the first embodiment of the present invention. [Figure 3] This diagram illustrates a flowchart showing the operation of an image recognition device according to the first embodiment of the present invention. [Figure 4] This figure schematically represents an example of data stored by the image recognition device in the first embodiment of the present invention. [Figure 5] This figure schematically represents an example of image data acquired by the image recognition device in the first embodiment of the present invention. [Modes for carrying out the invention]

[0009] The embodiments will be described below with reference to the drawings. The same or equivalent components, members, and processes shown in each drawing will be denoted by the same reference numerals, and redundant explanations will be omitted as appropriate. Furthermore, some components, members, and processes will be omitted in each drawing. The following embodiments are not limiting to the present invention, and not all combinations of features described in these embodiments are essential to the solutions of the present invention. The configuration of the embodiments may be modified or changed as appropriate depending on the specifications of the device to which the present invention is applied and various conditions (operating conditions, operating environment, etc.). In the following embodiments, the same components will be denoted by the same reference numerals.

[0010] (First embodiment) <Internal Configuration of Image Recognition Device> Figure 1 is a block diagram illustrating a head-mounted display (HMD) 100, which is an image recognition device according to the first embodiment. In the first embodiment, an HMD is assumed as an example of a device that constitutes an image recognition device, but the present invention is not limited to these embodiments. For example, it may be a smartphone or tablet terminal equipped with a camera, or other devices such as a personal computer (PC) or digital camera.

[0011] The HMD100 is a head-mounted display device (electronic device) that can be worn on the user's head. The HMD100 is equipped with a camera to capture images of the area directly in front of the user and a display to show the images to the user. The HMD100's display shows a composite image, which is a combination of the image captured by the HMD100 of the area directly in front of the user and content such as computer graphics (CG) that is shaped according to the posture of the HMD100. This allows the user to experience virtual reality or mixed reality with their own eyes.

[0012] Referring to Figure 1, the internal configuration of the HMD100 will be explained. The HMD100 has a control unit 101, ROM 102, RAM 103, and imaging unit 104 connected to a system bus 105. The control unit 101, ROM 102, RAM 103, imaging unit 104, and system bus 105 are the hardware resources that constitute the HMD100. Note that each component of the HMD100 may be an image recognition system composed of individual hardware.

[0013] The control unit 101 controls various parts of the HMD 100 according to the input signals and the program described later. The control unit 101 has at least one CPU that executes the program stored in the ROM 102, and at least one other circuit. Alternatively, instead of the control unit 101 controlling the entire device, multiple hardware components may share the processing to control the entire device.

[0014] ROM102 is an electrically erasable and recordable non-volatile memory that stores programs executed by the control unit 101. When power is turned on to the HMD100, the control unit 101 reads the program from ROM102 and starts controlling the HMD100. ROM102 consists of, for example, flash memory.

[0015] RAM 103 is a memory unit used as a work area by the program executed by the control unit 101. RAM 103 consists of, for example, volatile memory (DRAM) using semiconductor elements.

[0016] Although the control unit 101, ROM 102, and RAM 103 have been described here as separate hardware resources, these functions may also be integrated into a single LSI.

[0017] The imaging unit 104 consists of a stereo camera, and captures color or monochrome images of the scene with its two mounted cameras (left and right), outputting the video signal to the system bus 105. This video signal undergoes various image processing by the control unit 101 and is stored in the RAM 103 as left and right image data. The imaging unit 104 consists of, for example, an optical lens unit, an optical system that controls aperture, zoom, and focus, and an image sensor that converts the light (image) introduced through the optical lens unit into an electrical video signal. Generally, either a CMOS image sensor (CMOS image sensor) using CMOS technology or a CCD image sensor (CCD image sensor) using CCD technology are used as the image sensor.

[0018] <Flowchart for determining left / right distinction in the area containing fingers> Referring to FIGS. 2(a), 2(b), 2(c), 3, 4(a), and 4(b), a flowchart showing the operation of the HMD 100 in the first embodiment will be described. The processes shown in this flowchart are realized by the control unit 101 of the HMD 100 controlling each part of the HMD 100 according to an input signal or a program. That is, this process is realized by expanding the program recorded in the ROM 102 into the RAM 103 and executed by the control unit 101. The process shown in the flowchart of FIG. 3 is executed from the timing when the user starts an application on the HMD 100, and is executed every time the imaging unit 104 acquires an image (every time shooting is performed). The application is, for example, an application selected by the user on the home screen (home space) after starting the HMD 100, and examples of the application include an application that allows the user to interfere with a virtual object using hand gestures. Note that the timing when this flowchart is executed is not limited to the timing when the user starts an application on the HMD 100. For example, it may be the timing when the user starts the HMD 100 or the timing when a virtual object is displayed in the mixed reality space.

[0019] Here, FIGS. 2(a), 2(b), and 2(c) are diagrams schematically showing an example of image data processed by the image recognition device of the first embodiment. In FIG. 2(a), the image 200 is an example of image data generated by the control unit 101 from the video signal acquired by the imaging unit 104, and the image 200 includes the hand 201 of the HMD wearer. The descriptions of FIGS. 2(b) and 2(c) will be described later.

[0020] In step S301, the control unit 101 acquires the video signal output from the imaging unit 104 to the system bus 105, generates image data, and proceeds to step S302. For example, the control unit 101 stores the image 200 in FIG. 2(a) in the RAM 103.

[0021] In step S302, the control unit 101 performs a process to detect human fingers and a process to classify the gestures shown by the detected fingers on the image data generated in step S301, and then proceeds to step S303. The process to detect human fingers is a region detection process that detects regions of a predetermined shape (hereinafter referred to as object regions) that are estimated to contain fingers. For example, a rectangle can be used as the predetermined shape. In addition, the smallest region of the predetermined shape that is estimated to contain fingers is detected. Note that, depending on the image, the rectangular region estimated to contain fingers may not be the smallest region. In the process to detect human hands, depending on how the subject etc. included in the image data is depicted, only one object region may be detected, multiple object regions may be detected, or none may be detected. Note that the process to detect human hands may be implemented using, for example, a trained deep learning model or a rule-based algorithm. In addition, in the first embodiment, the predetermined shape of the object region was set to a rectangle, but the predetermined shape is not necessarily limited to a rectangle, and may be a polygon, circle, ellipse, etc., as long as all object regions have the same shape. Furthermore, this region detection process and the classification process of the gestures shown by the detected fingers may be performed simultaneously using the same means, sequentially, or by different means.

[0022] Here, referring to Figure 2(b), we will describe an example of the result of performing the processing in step S302 on image 200 of Figure 2(a) in the first embodiment, in which the detected regions 211 and 221 are superimposed on image 200. Here, the coordinate system in the captured image is defined as the upper left of image 200 as the origin, the vertical downward direction as the y-coordinate axis, and the horizontal right direction as the x-coordinate axis. As coordinates on the captured image, the coordinates of the center of the rectangle of region 211 are (cx1,cy1), and the coordinates of the center of the rectangle of region 221 are (cx2,cy2). In the example of Figure 2(b), region 221 is a region where space other than human fingers was misdetected. In Figure 2(b), the center coordinates of region 211 are (cx1,cy1)=(75,55), and the center coordinates of region 221 are (cx2,cy2)=(85,10).

[0023] In step S303, the control unit 101 checks whether there is a region detected in step S302. If there is a region detected, the unit proceeds to step S304; otherwise, the unit proceeds to step S324.

[0024] Steps S304 to S309 are loop processes for estimating the skeleton, obtaining the arm orientation, and evaluating the region for all areas detected in step S302.

[0025] In step S304, the control unit 101 selects one region from all the regions detected in step S302 that has not yet undergone the aforementioned loop processing, and starts the aforementioned loop processing.

[0026] In step S305, the control unit 101 performs a position detection process (skeleton estimation process) to detect target points in the region selected in step S304, and proceeds to step S306. As part of the position detection process, the control unit 101 detects the positions of each joint point, including the fingertips of each finger, the wrist point, and the point below the wrist as target points in two-dimensional coordinates. Here, the wrist point is the joint point of the wrist, and the point below the wrist is any one point (a specific target point) that lies between the wrist point and the elbow joint and within the region where the position detection process was performed. In the first embodiment, the coordinates detected by the control unit 101 in the position detection process are two-dimensional coordinates, but the detected coordinates may also be three-dimensional coordinates, or partly two-dimensional coordinates and the rest three-dimensional coordinates. Furthermore, the position detection process may be implemented, for example, using a trained deep learning model, or it may be implemented using a rule-based algorithm.

[0027] Here, referring to Figure 2(c), in the first embodiment, an example of the result of performing the processing in step S305 on image 200 of Figure 2(a) will be described, showing the processing area and the positions of each detected joint point, wrist point, and wrist sub-point superimposed on image 200. The black circles indicate the positions of the detected joint points within the detected areas. The black circles in area 211 include the wrist point 214 and the wrist sub-point 215, and the black circles in area 221 include the wrist point 224 and the wrist sub-point 225. The coordinates of wrist point 214 are (wx1, wy1), the coordinates of wrist sub-point 215 are (ux1, uy1), the coordinates of wrist point 224 are (wx2, wy2), and the coordinates of wrist sub-point 215 are (ux2, uy2). Furthermore, (wx1,wy1)=(80,60), (ux1,uy1)=(85,70), (wx2,wy2)=(85,15), and (ux2,uy2)=(90,23). Note that region 221 does not actually contain a hand and is a false detection region that was mistakenly detected, but region detection and joint point detection within the region were performed.

[0028] In step S306, the control unit 101 performs an angle acquisition process to obtain the arm angle based on the two-dimensional coordinates of the wrist point and the lower wrist point detected in step S305, and then proceeds to step S307.

[0029] Here, with reference to Figure 2(c), the arm angle calculated from the captured image will be explained. In image 200 of Figure 2(c), the control unit 101 defines the angle between the line connecting the wrist point 214 and the wrist lower point 215 and the line extending vertically from the wrist lower point 215 as the arm angle of the hand detected in region 211. Similarly, the angle between the line connecting the wrist point 224 and the wrist lower point 225 and the line extending vertically from the wrist lower point 225 as the arm angle of the hand detected in region 221. As mentioned above, the position coordinates of the wrist joint point in object region 211 are (wx1, wy1) = (80, 60), and the position coordinates of the wrist lower point in object region 211 are (ux1, uy1) = (85, 70). Furthermore, the position coordinates of the wrist joint point in object region 221 are (wx2, wy2) = (85, 15), and the position coordinates of the lower wrist point in object region 221 are (ux2, uy2) = (90, 23). Therefore, the arm angle 231 is arctan((ux1-wx1) / (uy1-wy1)) = 30.0°, and the arm angle 232 is arctan((ux2-wx2) / (uy2-wy2)) = 38.68°.

[0030] In step S307, the control unit 101 stores the center coordinates of the region in the RAM 103 as information representing the position of the region detected in step S302, and proceeds to step S308.

[0031] In the example shown in Figure 2(b), the control unit 101 stores the center coordinates of region 211 (cx1,cy1)=(75,55) and the center coordinates of region 221 (cx2,cy2)=(85,10) in RAM 103 along with the time when step S301 was executed.

[0032] Here, with reference to Figure 4(a), the center coordinates and left / right discrimination information of the detected region stored in RAM 103 will be explained. Figure 4(a) is an example of the center coordinates and left / right discrimination information of the detected region stored in RAM 103. Time t is the time when image 200 was acquired in step S301, and time t-1 is the time when image data was acquired in step S301 immediately before image 200. Region 1 and Region 2 are regions detected in step S302 with respect to the image data acquired at that time, and their center coordinates and left / right discrimination information acquired by the discrimination means described later are stored. In Figure 4(a), the image 200 shown in Figure 2(a) was acquired in step S301 at time t, and region 211 is stored as region 1 with center coordinates (cx1,cy1)=(75,55). Also, region 221 is stored as region 2 with center coordinates (cx2,cy2)=(85,10). Furthermore, the central coordinates (74,53) of region 1, detected from the image acquired at time t-1, are stored, and the central coordinates (82,10) of region 2, detected from the image acquired at time t-1, are stored.

[0033] In step S308, the control unit 101 obtains the center coordinates of previously detected regions from the RAM 103 and performs an evaluation value acquisition process to obtain an evaluation value indicating the likelihood that the region selected in step S304 contains fingers, and then proceeds to step S309. The control unit 101 uses the distance between the center coordinates of the region selected in step S304 and the previously detected regions, and the position of the region selected in step S304 relative to the image data acquired in S301, as criteria for determining the likelihood that the region contains fingers.

[0034] Specifically, the control unit 101 first calculates the distance between the center coordinates of the region selected in step S304 and the center coordinates of all regions at the previous time t-1 obtained from the RAM 103. If any of the calculated distances is less than or equal to a predetermined threshold, the control unit 101 determines that the center coordinates of the region selected in step S304 and the center coordinates of the corresponding region at time t-1 are neighbors. In the first embodiment, the predetermined threshold for determining whether the distance between the center coordinates of regions is considered neighboring is set to thr_ct = 5 pixels. If the control unit 101 determines that the region selected in step S304 is a neighbor of any region, it then determines whether the region selected in step S304 fits within the captured image acquired in step S301. If the control unit 101 determines that the region selected in step S304 fits within the captured image acquired in step S301, it assigns a high value as the evaluation value to that region. On the other hand, if the control unit 101 does not determine that the region selected in step S304 fits within the captured image acquired in step S301, it assigns a low value as the evaluation value. For example, if the captured image does not include the entire hand but only a part of it, the detection process may result in the detected area extending beyond the captured image. In such cases, since the detected area does not fit within the captured image, a low value will be assigned to the evaluation value. Furthermore, if the distance between the center coordinates of the area selected in step S304 and the center coordinates of all areas at the previous time t-1 obtained from RAM 103 is not below a predetermined threshold, a low value will be assigned to the evaluation value. In other words, it is determined that the area selected in step S304 is not in the vicinity of all areas at time t-1, and a low value will be assigned to the evaluation value. In the first embodiment, a high evaluation value is set to 1.0 and a low evaluation value to 0.0.

[0035] In the first embodiment, the region detected in the past is defined as the region detected in the image data acquired at time t-1, but it may also be a region detected at an earlier time. Furthermore, it may be a region detected at any one of several past time points, or it may be all regions that have been detected in the past and stored in RAM 103.

[0036] In the first embodiment, the conditions for assigning a high evaluation value were that the center coordinates of the region selected in step S304 are near the center coordinates of a previously detected region, and that the region selected in step S304 is contained within the image data. However, a high evaluation value may be assigned if only one of these conditions is met, and a low evaluation value may be assigned to the region if neither condition is met. Also, the evaluation value obtained in step S308 may be a binary value or a continuous value. Furthermore, the processing in step S308 may be performed before steps S305 and S306.

[0037] In the example shown in Figure 4(a), if region 211 is selected in step S304, the center coordinates 212,213 of region 211 are (cx1,cy1)=(75,55) as described above. Among the regions detected at time t-1, the Euclidean distance from the center coordinates (74,53) of region 1 is √5 pixels, and the Euclidean distance from the center coordinates (82,10) of region 2 is √2074 pixels. Since the distance between region 211 and region 1 at time t-1 is less than or equal to thr_ct, the control unit 101 determines that region 211 is in the vicinity of region 1 detected at time t-1. Furthermore, region 211 is contained within the image data. Therefore, the control unit 101 assigns a high evaluation value of 1.0 to region 211.

[0038] Furthermore, if region 221 is selected in step S304, the center coordinates 222,223 of region 221 are (cx2,cy2)=(85,10). Among the regions detected at time t-1, the Euclidean distance from the center coordinates (74,53) of region 1 is √1970 pixels, and the Euclidean distance from the center coordinates (82,10) of region 2 is 3 pixels. Since the distance between region 221 and region 2 at time t-1 is less than or equal to thr_ct, region 221 is in the vicinity of region 2 detected at time t-1. Moreover, as shown in Figure 2, region 221 is contained within image 200. Therefore, the control unit 101 assigns a high evaluation value of 1.0 to region 221.

[0039] Here, if no region is found to be adjacent to region 221 among the regions detected at time t-1, the control unit 101 will assign a low evaluation value of 0.0 to region 221.

[0040] In step S309, if there are any regions detected in step S302 that have not yet been selected in step S304, the control unit 101 continues the loop processing. Once processing of all regions is complete, the loop processing ends and the unit proceeds to step S310.

[0041] In step S310, the control unit 101 performs a first discrimination process to determine the left or right of the fingers included in all regions detected in step S302, and then proceeds to step S311. In the discrimination process in step S310, the control unit 101 performs left / right discrimination based on the arm orientation information obtained in step S307, the center coordinate information of the regions obtained in step S302, and the relative position information between the regions. Then, all the results are combined to determine the left / right discrimination result as the first determination process. Here, relative position information refers to information about the relative positional relationship between regions. For example, the distance between the center coordinates of two regions or the difference in the x-coordinates of the center coordinates of two regions constitute relative position information.

[0042] The control unit 101 performs a discrimination process based on the orientation of the arm. If the angle of the arm orientation information obtained in step S306 is between the minimum value thr_deg_min and the maximum value thr_deg_max relative to the vertical, it is tentatively identified as the right hand. If the angle is between the minimum value -thr_deg_max and the maximum value -thr_deg_min relative to the vertical, it is tentatively identified as the left hand. Otherwise, it is considered impossible to distinguish between the right and left hands. In the first embodiment, thr_deg_min = 10° and thr_deg_max = 150°.

[0043] The control unit 101 performs a discrimination process based on the center coordinates of the region. If the center coordinates of the region are to the left of the half of the image data acquired in step S301, it tentatively identifies it as a left hand; otherwise, it tentatively identifies it as a right hand.

[0044] The control unit 101 performs left / right discrimination processing based on the relative positions of the regions. When multiple regions are detected in step S302, if the center coordinates of the regions are separated horizontally by a threshold thr_rel or more, it tentatively identifies the right region as the right hand and the left region as the left hand. Otherwise, it considers it impossible to distinguish between the right and left hands. In the first embodiment, thr_rel = 30 pixels.

[0045] The control unit 101, for each of the three discrimination processes described above, determines the result to be left hand if the number of left-handed people is greater than the number of right-handed people in the provisional discrimination results, and otherwise determines the result to be right hand.

[0046] In the examples in Figures 2(c) and 4(a), region 211 is tentatively identified as a right hand because, based solely on the arm orientation information, the arm orientation angle is 30.0° as described above, and it is greater than or equal to the minimum value thr_deg_min and less than or equal to the maximum value thr_deg_max. Region 211 is tentatively identified as a right hand because, based solely on the information of the region's center coordinate, it does not exist to the left of the halfway point of the image data, as shown in Figure 2. Region 211 is tentatively identified as a left hand because, based solely on the relative position information between regions, the horizontal center coordinate of region 211 is 75, which is more than thr_rel to the left of the horizontal center coordinate of region 221, which is 85. The tentative identification results for each of the three identification processes are right hand, right hand, and left hand, and since there are more right hands, region 211 is identified as a right hand.

[0047] Based solely on the arm orientation information, region 221 is tentatively identified as a right hand because its arm orientation angle is 38.68°, as mentioned above, and is greater than or equal to the minimum value thr_deg_min and less than or equal to the maximum value thr_deg_max. Based solely on the information of the region's center coordinate, region 221 is tentatively identified as a right hand because, as shown in Figure 2, it does not exist to the left of the halfway point of the image data. Based solely on the relative position information between regions, region 221 is tentatively identified as a right hand because its horizontal center coordinate is located more than thr_rel to the right of the horizontal center coordinate of region 211. The tentative identification results are right hand, right hand, right hand, and since there are more right hands, region 221 is identified as a right hand.

[0048] In step S311, the control unit 101 checks whether the results of the first discrimination process in step S310 for all regions detected in step S302 overlap. If there is an overlap, the unit proceeds to step S312; otherwise, the unit proceeds to step S323.

[0049] In the examples of Figures 2(c) and 4(a), the result of the first discrimination process in step S310 is determined to be the right hand in both cases, which is redundant, so the control unit 101 proceeds to step S312.

[0050] Steps S312 to S315 are loop processes that re-determine left and right for all regions detected in step S302, according to the evaluation value of each region.

[0051] In step S312, the control unit 101 selects one region from all the regions detected in step S302 that has not yet undergone the loop processing from step S312 to step S315, and starts the loop processing from step S312 to step S315.

[0052] In step S313, the control unit 101 checks whether the evaluation value obtained in step S308 is greater than or equal to a predetermined value for the region selected in step S312. If it is greater than or equal to the predetermined value, the unit proceeds to step S314; otherwise, the unit proceeds to step S315. In the first embodiment, the predetermined value is set to thr_e = 0.5.

[0053] In the example shown in Figure 2(c), as mentioned above, both region 211 and region 221 are assigned an evaluation value of 1.0 in step S308. Therefore, since both are greater than or equal to thr_e, the control unit 101 proceeds to step S314 regardless of whether region 211 or region 221 is selected in step S312.

[0054] In step S314, the control unit 101 performs a second discrimination process to determine the left or right of the region selected in step S312 based on the arm orientation obtained in step S306, and proceeds to step S315. In the first embodiment, the control unit 101 performs the same process as the discrimination process based on the arm orientation in step S310 to determine the left or right of the region. In the first embodiment, thr_deg_min = 5° and the maximum value thr_deg_max = 160°. In the first embodiment, the values ​​of thr_deg_min and thr_deg_max are different in S310 and S314, but they may be the same value, or only one of them may be the same.

[0055] In the example in Figure 2(c), if region 211 is selected in step S312, the angle of the arm direction in region 211 is 30.0° as described above, and since it is greater than or equal to the minimum value thr_deg_min and less than or equal to the maximum value thr_deg_max, region 211 is determined to be a right hand. Also, if region 221 is selected in step S312, the angle of the arm direction in region 221 is 38.68° as described above, and since it is greater than or equal to the minimum value thr_deg_min and less than or equal to the maximum value thr_deg_max, it is determined to be a right hand.

[0056] In step S315, if there are any regions detected in step S302 that have not yet been selected in step S312, the control unit 101 continues the loop processing. Once processing of all regions is complete, the loop processing ends and the unit proceeds to step S316.

[0057] In step S316, if the control unit 101 determines that there is only one region among the regions detected in step S302 whose evaluation value obtained in step S308 is less than a predetermined value, it proceeds to step S317. If the control unit 101 does not determine that there is only one region among the regions detected in step S302 whose evaluation value obtained in step S308 is less than a predetermined value, it proceeds to step S318. In the first embodiment, the predetermined value is thr_e=0.5, which is the same as in step S313, as described above.

[0058] In the example shown in Figure 2(c), as mentioned above, both region 211 and region 221 are assigned an evaluation value of 1.0 in step S308. Therefore, since both are greater than or equal to thr_e, the process proceeds to step S318.

[0059] In step S317, the control unit 101 determines the left or right of the region detected in step S302 that was not processed in step S314, based on the determination result of the region that was processed in step S314, and proceeds to step S323. In the first embodiment, the control unit 101 determines either the left or right, which does not overlap with the determination result of the region processed in step S314, as the determination result of the region that was not processed in step S314. For example, suppose two regions are detected in step S302, and the evaluation value of one region is 1.0 and the evaluation value of the other region is 0.0. In this case, if the left and right of the region with an evaluation value of 1.0 were determined to be the right hand in S314, the control unit 101 will determine the region with an evaluation value of 0.0 to be the left hand in step S317. When determining left or right in this way, the region with the higher evaluation value will have priority based on the left or right determination result based on the orientation of the arm, and the left or right of the region with the lower evaluation value will be determined according to the left or right determination result of the region with the higher evaluation value. Therefore, it is possible to reduce the cases in which the left / right discrimination result of the region with a low evaluation value (i.e., a relatively high probability of false detection) causes the left / right discrimination of the region with a high evaluation value (i.e., a relatively low probability of false detection) to be incorrect.

[0060] In step S318, the control unit 101 checks whether the results of the second discrimination process in step S314 for all regions detected in step S302 overlap. If there is an overlap, the unit proceeds to step S319; otherwise, it proceeds to step S323. Specifically, in step S318, if two regions are detected from the captured image and the two regions correspond to one right hand and one left hand, the unit proceeds to step S323. Also, in step S318, if two regions are detected from the captured image and both regions are determined to be either the right hand or both left hand, the unit proceeds to step S319. Furthermore, in step S318, if three or more regions are detected from the captured image, either the right hand or the left hand will overlap, so the unit proceeds to step S319.

[0061] In the examples of Figures 2(c) and 4(a), the result of the second discrimination process in step S314 is that both region 211 and region 221 overlap with the right hand, so the control unit 101 proceeds to step S319.

[0062] Steps S319 to S322 are a loop process that uses the information of the region detected in S302 at time t-1 to further determine whether the region is left or right for all the regions detected in step S302.

[0063] In step S319, the control unit 101 selects one region from all the regions detected in step S302 that has not yet undergone the loop processing from step S319 to step S322, and starts the loop processing from step S319 to step S322.

[0064] In step S320, the control unit 101 proceeds to step S321 if the region selected in step S319 is determined to be near any region by the processing in step S308, otherwise proceeds to step S322.

[0065] In the example shown in Figure 2(c), as described above, the process in step S308 determines that region 211 is a neighbor of region 1 at time t-1, and region 221 is a neighbor of region 2 at time t-1. Therefore, regardless of the region selected in step S319, the control unit 101 proceeds to step S321.

[0066] In step S321, the control unit 101 performs a third discrimination process in which the left-right information of the region at time t-1 that was determined to be adjacent to the region selected in step S319 is used as the left-right discrimination result of the region selected in step S319, and proceeds to step S322.

[0067] In the examples of Figures 2(c) and 4(a), the control unit 101 determines in step S308 that the vicinity of region 211 is region 1 at time t-1, and the left / right determination result for region 1 at time t-1 stored in RAM 103 is a right hand. Therefore, if the control unit 101 selected region 211 in step S319, it takes the right hand as the left / right determination result for region 211 and proceeds to step S322. Similarly, the vicinity of region 221 is region 2 at time t-1, and the left / right determination result for this region is a left hand. Therefore, if the control unit 101 selected region 221 in step S319, it takes the left hand as the left / right determination result for region 221 and proceeds to step S322.

[0068] In step S322, the control unit 101 continues the loop processing if there are still regions detected in step S302 that were not selected in step S319. Once processing of all regions is complete, the loop processing ends and the unit proceeds to step S323.

[0069] Note that the second and third discrimination processes may be swapped. That is, the process in step S314 may be swapped with the processes in steps S320 to S321.

[0070] In step S323, the control unit 101 stores the center coordinates representing the position of the region detected in step S302 and the final left / right determination result of that region in the RAM 103, and proceeds to step S324.

[0071] Figure 4(b) shows an example of the center coordinates and left / right discrimination information of the detected region stored in RAM 103 as a result of processing in step S323 for an image acquired at time t. In addition to the information in Figure 4(a), the final left / right discrimination result of region 211 detected in step S301 is stored in step S323 as left / right discrimination information for region 1 at time t, linked to time t. Similarly, the final left / right discrimination result of region 221 detected in step S301 is stored in step S323 as left / right discrimination information for region 2 at time t, linked to time t.

[0072] In step S324, the control unit 101 determines whether to terminate the process shown in the flowchart of Figure 3. For example, if the user inputs a termination command via an operating device (not shown) or if the control unit 101 is unable to acquire image data, the control unit 101 determines to terminate the process shown in Figure 3. On the other hand, if it determines in step S313 that the process should not be terminated, the control unit 101 returns to step S301.

[0073] As described above, in this embodiment, the control unit 101 detects a region containing the human fingers, which is the object to be detected, from the image data. Next, it detects the coordinates of the skeleton, including the wrist point and the lower wrist point, within that region and obtains the orientation of the arm in the region. Based on the orientation of the arm in the region and the position of the region, it determines the left or right of the fingers, and if necessary, it determines the left or right based on the arm orientation of the region with the highest evaluation value. Furthermore, if necessary, it uses the left or right determination result of a nearby region that has been detected in the past as the left or right determination result for the region. This makes it possible to determine the left or right of the hand in the finger region from the captured image, even without prior hand registration processing or the use of a controller.

[0074] Furthermore, in captured images, there are cases where areas that do not actually contain a hand are detected (actually false detection areas) and areas that actually contain a hand (correct detection areas) are detected. In such cases, if left / right discrimination is performed on each area, there is a risk that the left / right discrimination of the correctly detected area will be incorrect due to the area that is actually false detection. Therefore, by following the flowchart described above, even if areas that are actually false detections and areas that are correctly detected are detected in the captured image, it is possible to prevent incorrect left / right discrimination of the correctly detected area. In other words, it is possible to prevent incorrect left / right discrimination of an area that actually contains a hand, even if the detection result of the area that is actually false detection is incorrect.

[0075] <Explanation of the correspondence between images acquired in chronological order and the flowchart> Referring to Figures 5(a), 5(b), and 2(c), the correspondence between the images acquired in chronological order and the flowchart will be explained. As mentioned above, Figure 2(c) shows image 200 acquired at time t. Figure 5(b) shows image 600, acquired at time t-1, which is the image acquired immediately before the image acquired at time t. Figure 5(a) shows image 500, acquired at time t-2, which is the image acquired immediately before the image acquired at time t-1. In other words, the chronological order is time t-2, time t-1, time t, and the images are acquired in the order of image 500, image 600, and image 200. Here, Figures 5(a) and 5(b) show the images after processing up to step S309 has been performed on the image acquired in step S301. That is, the region containing the hand has been detected, and the detection target points are the fingertips, finger joints, and joint points including the wrist, and the orientation of the arm has been acquired. Furthermore, the location of each region is stored in RAM103, and the image shows the evaluation of each region.

[0076] First, in step S301, if image 500 is acquired at time t-2, the process in step S303 proceeds to step S304 because region 511 has been detected. Also, since only region 511 is detected from image 500, the processes from step S304 to step S309 are performed only once. Here, in step S306, it is assumed that the orientation of the arm in region 511 has been acquired in a way that it is tentatively identified as a right hand. Also, in step S307, it is assumed that the center coordinates of region 511, (cx,cy)=(73,51), are stored in RAM103. Furthermore, it is assumed that the coordinates of the wrist joint point 514, (wx,wy)=(78,56), and the coordinates of the wrist lower point 515, (ux,uy)=(83,66), have been acquired. In other words, the arm angle 531 is arctan((ux-wx) / (uy-wy))=30.0°, and in step S310, based on information regarding the orientation of the arm, it is determined that the hand included in region 511 is a right hand. In the processing of step S311, since only region 511 is detected from image 500, the process proceeds to step S323, and the position of region 511 and the fact that the hand detected within region 511 is a right hand are stored in RAM 103.

[0077] Next, in step S301, at time t-1, image 600 is acquired. In the processing of step S303, since regions 611 and 621 are detected, the process proceeds to step S304. Here, region 611 is a correctly detected region that actually contains a hand, and region 621 is a region that does not actually contain a hand and is actually a false detection. Also, since regions 611 and 621 are detected from image 600, the processing from step S304 to step S309 is performed twice. Here, in step S306, it is assumed that the orientation of the arm is acquired in both regions 611 and 621 such that they are both tentatively identified as a right hand. Also, in step S307, it is assumed that the center coordinates of region 611, (cx,cy)=(74,53), and the center coordinates of region 621, (cx,cy)=(82,10), are stored in RAM 103. Then, in step S308, the center coordinates (cx,cy)=(73,51) of region 511 in image 500 are referenced, and it is determined that region 611 is a neighbor of region 511. Furthermore, since region 611 is contained within image 600, a high evaluation value of 1.0 is assigned to region 611. Region 621 is determined not to be a neighbor of region 511, and a low evaluation value of 0.0 is assigned to region 621.

[0078] Next, in step S310, based on information regarding the orientation of the arm, it is determined that both the hand included in region 611 and region 621 is a right hand. Specifically, in region 611, the coordinates of the wrist joint point 614 are (wx,wy)=(79, 58), and the coordinates of the wrist lower point 615 are (ux,uy)=(84, 68). That is, the arm angle 631 is arctan((ux-wx) / (uy-wy))=30.0°, and in step S310, based on information regarding the orientation of the arm, it is determined that the hand included in region 611 is a right hand. Also, in region 621, the coordinates of the wrist joint point 624 are (wx,wy)=(82, 15), and the coordinates of the wrist lower point 625 are (ux,uy)=(87, 23). In other words, the arm angle 632 is arctan((ux-wx) / (uy-wy))=38.68°, and in step S310, based on information regarding the arm's orientation, it is determined that the hand included in region 621 is a right hand. That is, in the processing of step S311, the first determination process in step S310 is duplicated, so the process proceeds to step S312. Here, the processing from step S312 to step S315 is performed twice. If region 611 is selected in step S312, an evaluation value of 1.0 is assigned, so the process proceeds to step S314, where a second determination process is performed to determine that it is a right hand based on information regarding the arm's orientation. Also, if region 621 is selected in step S312, an evaluation value of 0.0 is assigned, so the second determination process in step S314 is not performed, and the loop processing from step S312 to step S315 ends. Then, in step S316, since there is only one region, region 621, where the evaluation value is less than a predetermined value, the process proceeds to step S317. In step S317, since region 611 has been determined to be a right hand by the second determination process, region 621 is determined to be a left hand. Then, in step S323, the position of region 611 and the fact that the hand detected in region 611 is a right hand, as well as the position of region 621 and the fact that the hand detected in region 621 is a left hand, are stored in RAM 103.

[0079] Next, in step S301, at time t, image 200 is acquired. As mentioned above, in the second discrimination process in step S314, based on the orientation of the arm, it is determined that both region 211 and region 221 are the right hand. Then, the loop processing from step S319 to step S322 is performed, via the processing in step S316 and step S318.

[0080] (Other embodiments) Furthermore, the present invention can also be realized by performing the following process: that is, supplying software (program) that realizes the functions of the above-described embodiment to a system or device via a network or various storage media, and having the computer (or control unit or MPU, etc.) of the system or device read and execute the program code. In this case, the program and the storage medium storing the program constitute the present invention.

[0081] Although the present invention has been described in detail above based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Some of the above embodiments may be combined as appropriate.

[0082] Furthermore, each functional unit in each of the above embodiments (each modified example) may or may not be individual hardware. The functions of two or more functional units may be implemented by common hardware. Each of the multiple functions of a single functional unit may be implemented by individual hardware. Two or more functions of a single functional unit may be implemented by common hardware. In addition, each functional unit may or may not be implemented by hardware such as an ASIC, FPGA, or DSP. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. The functions of at least some of the functional units of the device may be implemented by the processor reading and executing the control program from the memory.

[0083] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0084] Furthermore, in each of the examples described above, "processor" refers to a processor in a broad sense, including general-purpose processors (e.g., CPUs) and specialized processors (e.g., GPUs, ASICs, FPGAs, and programmable logic devices, etc.).

[0085] This embodiment includes the following configurations, methods, and programs.

[0086] [Configuration 1] An image recognition device that determines the left and right of a hand included in an captured image, A detection means for detecting the region in the captured image that includes fingers, An evaluation value acquisition means that acquires an evaluation value indicating the likelihood that the region includes the fingers, An angle acquisition means for acquiring information on the orientation of the hand and arm included in the said region, A first determination means for determining the left and right of the hands included in the region based on a plurality of pieces of information including information on the orientation of the arm, If the discrimination results obtained by the first discrimination means for multiple regions detected by the detection means are the same, the second discrimination means determines the left and right sides of the region among the multiple regions in which the evaluation value is higher than the first threshold. An image recognition device characterized by the following features.

[0087] [Configuration 2] The second determination means determines, based on the information of the arm orientation in the region, which hand is included in the region where the evaluation value is higher than the first threshold. The image recognition device according to configuration 1, characterized in that...

[0088] [Configuration 3] A storage means stores in a storage unit the position of the first region detected by the detection means on the first image and the left / right discrimination result in the first image. In a second image captured after the first image, if the left / right discrimination result determined by the second discrimination means is the same for the second region detected by the detection means and the third region which is different from the second region, and the distance between the position of the second region on the second image and the position of the first region on the first image is smaller than the second threshold, the third discrimination means determines the left / right discrimination result of the first region stored in the storage unit as the left / right discrimination result of the second region. An image recognition device according to configuration 1 or 2, characterized by the above.

[0089] [Structure 4] The system further includes a storage means that stores in the storage unit the position of the first region detected by the detection means on the first image, and the left / right discrimination result, in the first image image. The second determination means determines the left / right determination result of the first region stored in the storage unit as the left / right determination result of the second region if, in a second image captured after the first image captured, the distance between the position of the second region detected by the detection means on the second image captured and the position of the first region on the first image captured is smaller than the second threshold. An image recognition device according to any one of configurations 1 to 3, characterized in that it is an image recognition device.

[0090] [Composition 5] The detection means further performs classification of hand gestures. An image recognition device according to any one of configurations 1 to 4, characterized by the above.

[0091] [Composition 6] The system further includes a storage means for recording the position of the region detected by the detection means on the captured image in the storage unit, The evaluation value acquisition means increases the evaluation value when the distance between the position of the region detected by the detection means on the captured image and the position of the region stored in the storage unit is smaller than a second threshold, or when the region detected by the detection means falls within the captured image. An image recognition device according to any one of configurations 1 to 5, characterized in that it is an image recognition device.

[0092] [Composition 7] The system further includes a position detection means for detecting the position of a target point for each region detected by the aforementioned detection means. An image recognition device according to any one of configurations 1 to 6, characterized by the above.

[0093] [Structure 8] The position detection means detects a target point that includes at least the position of the wrist joint point and the position of a specific target point which is any one point located between the wrist joint point and the elbow joint and within the region. The angle acquisition means acquires information about the orientation of the arm based on the position of the wrist joint and the position of the specific detection target point. The image recognition device according to configuration 7, characterized by the features described above.

[0094] [Composition 9] The first determination means further determines whether the hand included in the region is left or right based on information about the position of the region on the captured image or information about the relative positional relationship between the plurality of regions detected by the detection means. An image recognition device according to any one of configurations 1 to 8, characterized in that it is an image recognition device.

[0095] [Configuration 10] If the determination results obtained by the first determination means for the plurality of regions detected by the detection means are the same, the second determination means does not determine the left and right sides of the plurality of regions where the evaluation value is lower than the first threshold. An image recognition device according to any one of configurations 1 to 9, characterized in that it is an image recognition device.

[0096] [Composition 11] The system further includes a determination means that determines the areas to the left and right of the region in which the evaluation value is lower than the first threshold, based on the determination result of the second determination means. The image recognition device according to configuration 10, characterized in that...

[0097] [Composition 12] If the captured image includes a first region in which the evaluation value is higher than the first threshold, and a second region in which the evaluation value is lower than the first threshold, and the second discrimination means determines that the hand included in the first region is a right hand, the determination means determines that the hand included in the second region is a left hand based on the discrimination result by the second discrimination means. The image recognition device according to configuration 11, characterized in that...

[0098] [method] An image recognition method for determining the left and right of a hand included in an captured image, A detection step of detecting a region in the captured image that includes fingers, An evaluation value acquisition step is to acquire an evaluation value that indicates the likelihood that the region includes the fingers, An angle acquisition step to acquire information on the orientation of the hand and arm included in the said region, A first determination step of determining the left and right of the hands included in the region based on a plurality of pieces of information including information on the orientation of the arm, If the discrimination results from the first discrimination step are the same for the multiple regions detected in the detection step, the second discrimination step determines the left and right sides of the region whose evaluation value is higher than the first threshold. An image recognition method characterized by the following features.

[0099] [program] A program for causing a computer to function as one of the means of an image recognition device described in any one of items 1 to 12.

[0100] [system] An image recognition system that determines the left and right of a hand included in an captured image, A detection device for detecting the region in the captured image that includes fingers, An evaluation value acquisition device that acquires an evaluation value indicating the likelihood that the region includes the fingers, An angle acquisition device that acquires information on the orientation of the hand and arm included in the aforementioned region, A first discrimination device that determines the left and right of the hands included in the region based on a plurality of pieces of information including information on the orientation of the arm, If the discrimination results of the first discrimination device are the same for multiple regions detected by the detection device, the device also includes a second discrimination device that distinguishes between the left and right sides of a region whose evaluation value is higher than the first threshold among the multiple regions. An image recognition system characterized by the following features.

Claims

1. An image recognition device that determines the left and right of a hand included in an captured image, A detection means for detecting a region in the captured image that is presumed to contain fingers, An evaluation value acquisition means that acquires an evaluation value indicating the likelihood that the region includes the fingers, An angle acquisition means for acquiring information on the orientation of the hand and arm that is estimated to be included in the aforementioned region, A first determination means for determining the left or right hand of a hand estimated to be included in the region based on information regarding the orientation of the arm and information regarding the position of the region in the captured image, If the detection means detects two regions, and the discrimination results of the first discrimination means for the two regions are the same, a second discrimination means determines the left and right sides of the two regions where the evaluation value is higher than the first threshold, The device includes a storage means that stores the aforementioned region and the left / right determination result of the aforementioned region in a storage unit, The aforementioned storage means is If the detection means detects two regions, and the discrimination results of the first discrimination means for the two regions are different, the positions of the two regions and the discrimination results of the first discrimination means for each of the two regions are linked and stored in the storage unit. If the detection means detects two regions and the discrimination results of the first discrimination means for the two regions are the same, the region whose evaluation value is higher than the first threshold is stored in the storage unit by linking the left and right discrimination results of the second discrimination means, and the region whose evaluation value is lower than the first threshold is stored in the storage unit by linking the left or right discrimination result of the second discrimination means that differs from the left or right discrimination result of the region whose evaluation value is higher than the first threshold. An image recognition device characterized by the following features.

2. The second determination means determines, based on the information of the orientation of the arm in the region, which hand is included in the region where the evaluation value is higher than the first threshold. The image recognition device according to feature 1.

3. The storage means associates the position of the first region detected by the detection means on the first image with the left / right discrimination result in the first image and stores it in the storage unit. The third discrimination means further includes, if, in a second image captured after the first image, the left / right discrimination result determined by the second discrimination means is the same for the second region detected by the detection means and a third region different from the second region, and the distance between the position of the second region on the second image and the position of the first region on the first image is smaller than a second threshold, the left / right discrimination result of the first region stored in the storage unit is determined as the left / right discrimination result of the second region. The image recognition device according to claim 1 or 2.

4. The storage means stores in the storage unit the position of the first region detected by the detection means on the first captured image and the left / right discrimination result, respectively. The second determination means determines the left / right determination result of the first region stored in the storage unit as the left / right determination result of the second region if, in a second image captured after the first image captured, the distance between the position of the second region detected by the detection means on the second image captured and the position of the first region on the first image captured is smaller than the second threshold. The image recognition device according to feature 1.

5. The detection means further performs classification of hand gestures. The image recognition device according to feature 1.

6. The storage means further stores in the storage unit the position on the captured image of the region detected by the detection means. The evaluation value acquisition means increases the evaluation value when the distance between the position of the region detected by the detection means on the captured image and the position of the region stored in the storage unit is smaller than a second threshold, or when the region detected by the detection means falls within the captured image. The image recognition device according to feature 1.

7. The system further includes a position detection means for detecting the position of a target point for each region detected by the aforementioned detection means. The image recognition device according to feature 1.

8. The position detection means detects a target point that includes at least the position of the wrist joint and the position of a specific target point which is any one point located between the wrist joint and the elbow joint and within the said region. The angle acquisition means acquires information about the orientation of the arm based on the position of the wrist joint and the position of the specific detection target point. The image recognition device according to feature 7.

9. The information relating to the position of the region in the captured image is either information about the position of the region on the captured image or information about the relative positional relationship between the two regions detected by the detection means. The image recognition device according to feature 1.

10. If the second discrimination means determines that the discrimination result obtained by the first discrimination means for the two regions detected by the detection means is the same, it does not discriminate between the left and right sides of the two regions where the evaluation value is lower than the first threshold. The image recognition device according to feature 1.

11. An image recognition method for determining the left and right of a hand included in an captured image, A detection step of detecting a region in the captured image that is presumed to contain fingers, An evaluation value acquisition step is to acquire an evaluation value that indicates the likelihood that the region includes the fingers, An angle acquisition step to acquire information on the orientation of the hand and arm that is estimated to be included in the aforementioned region, A first determination step of determining the left or right of the hand estimated to be included in the region based on the information regarding the orientation of the arm and the information regarding the position of the region in the captured image, If two regions are detected by the detection step, and the discrimination results from the first discrimination step for the two regions are the same, a second discrimination step is performed to determine the left and right sides of the two regions where the evaluation value is higher than the first threshold, The system includes a storage step of associating the region with the left / right determination result of the region and storing it in a storage unit, In the aforementioned storage step, If two regions are detected by the detection step, and the discrimination results for the two regions by the first discrimination step are different, the positions of the two regions and the discrimination results for each of the two regions by the first discrimination step are linked and stored in the storage unit. If two regions are detected by the detection step, and the discrimination results of the first discrimination step for the two regions are the same, the region whose evaluation value is higher than the first threshold is stored in the storage unit by linking the left and right discrimination results of the second discrimination step, and the region whose evaluation value is lower than the first threshold is stored in the storage unit by linking the left or right discrimination result that differs from the left or right discrimination result of the region whose evaluation value is higher than the first threshold, as determined by the second discrimination step. An image recognition method characterized by the following features.

12. A program for causing a computer to function as each of the means of the image recognition apparatus described in claim 1.

13. An image recognition system that determines the left and right of a hand included in an captured image, A detection device for detecting a region in the captured image that is presumed to contain fingers, An evaluation value acquisition device that acquires an evaluation value indicating the likelihood that the region includes the fingers, An angle acquisition device that acquires information on the orientation of the hand and arm that is estimated to be included in the aforementioned region, A first discrimination device for determining the left or right hand of a hand estimated to be included in the region based on information regarding the orientation of the arm and information regarding the position of the region in the captured image, If the detection device detects two regions, and the discrimination results of the first discrimination device for the two regions are the same, a second discrimination device is used to determine the left and right sides of the two regions, where the evaluation value is higher than the first threshold. The system includes a memory device that stores the aforementioned region and the left / right determination result of the aforementioned region in association with each other. The aforementioned storage device is If the detection device detects two regions, and the discrimination results of the first discrimination device for the two regions are different, the positions of the two regions and the discrimination results of the first discrimination device for each of the two regions are linked and stored in the system. If the detection device detects two regions and the discrimination results of the first discrimination device for the two regions are the same, the region whose evaluation value is higher than the first threshold is stored by linking the left and right discrimination results of the second discrimination device, and the region whose evaluation value is lower than the first threshold is stored by linking the left or right discrimination result of the second discrimination device that differs from the left or right discrimination result of the region whose evaluation value is higher than the first threshold. An image recognition system characterized by the following features.

Images