Information processing apparatus, information processing method, and program

Abstract

To improve the accuracy of classifying an object to be detected in a region.SOLUTION: An information processing apparatus acquires, from an image, regions of objects and results of classifying the objects with respect to the regions, and selects two or more regions having overlapping relationship from among the acquired regions. The information processing apparatus corrects the results of classifying the objects, on the basis of overlapping relationships between the selected two or more regions.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing technique for detecting a detection target object from an image. [Background technology]

[0002] Patent Document 1 discloses a technique for extracting a hand region from a captured image as a region of an object to be detected, and determining whether the hand is left or right based on the center of gravity of the hand region and the positions of the fingertips. Furthermore, Non-Patent Document 1 describes a learning model (object detection model) called YOLO, which is a representative machine learning technology for object detection, and discloses a technology capable of detecting object regions such as hands. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-204183 A [Non-patent literature]

[0004] [Non-Patent Document 1] Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016). You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 779-788). Summary of the Invention [Problem to be solved by the invention]

[0005] In the technology of Patent Document 1, the hand region detected as the region of the detection target object is classified as either a right hand or a left hand based on the center of gravity of the hand region, but it is difficult to correctly classify the hand region as either left or right depending on the angle of the hand. In addition, the technology disclosed in Non-Patent Document 1 may erroneously detect the hand region depending on the learning data or learning method used in the machine learning of the object detection model, or the shape of the hand, making it impossible to correctly classify the hand region as either left or right.

[0006] Therefore, an object of the present invention is to improve classification accuracy in the region of a detection target object. [Means for solving the problem]

[0007] The information processing device of the present invention is characterized by having an acquisition means for acquiring an object region and an object classification result for the region from an image, a selection means for selecting two or more regions having an overlapping relationship from the multiple regions acquired by the acquisition means, and a correction means for correcting the object classification result based on the overlapping relationship of the two or more regions selected by the selection means. Effect of the Invention

[0008] According to the present invention, it is possible to improve classification accuracy in a region of a detection target object. [Brief description of the drawings]

[0009] [Figure 1] FIG. 2 is a diagram illustrating an example of a functional configuration of an image recognition apparatus according to a first embodiment. [Diagram 2] 4A to 4C are diagrams showing images before and after correction by a first correction unit. [Diagram 3] FIG. 13 is an explanatory diagram of an overlapping relationship of hand regions. [Figure 4] 4 is a flowchart of a process in the image recognition device according to the first embodiment. [Diagram 5] FIG. 11 is a diagram illustrating an example of a functional configuration of an image recognition device according to a second embodiment. [Figure 6] 13A and 13B are diagrams illustrating images before and after correction by a second correction unit. [Figure 7] 13 is a flowchart of a process in an image recognition device according to a second embodiment. [Figure 8] FIG. 13 is a diagram illustrating an example of a functional configuration of an image recognition device according to a third embodiment. [Figure 9] 13A to 13C are diagrams illustrating images before and after correction by a third correction unit. [Figure 10] 13 is a flowchart of a process in an image recognition device according to a third embodiment. [Figure 11] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The following embodiments do not limit the present invention, and not all of the combinations of features described in the present embodiments are necessarily essential to the solution of the present invention. The configurations of the embodiments may be appropriately modified or changed depending on the specifications of the device to which the present invention is applied and various conditions (such as conditions of use and environment of use). In the following embodiments, the same configurations are described with the same reference symbols.

[0011] In this embodiment, an image recognition device mounted on a head mounted display (hereinafter referred to as HMD) equipped with a camera will be described as an application example of an information processing device. A captured image obtained from the camera of the HMD corresponds to an image seen from the viewpoint of a user wearing the HMD, and the image recognition device of this embodiment detects image areas of the right and left hands of the user (hereinafter referred to as hand areas) from the captured image from the user's viewpoint. The image of the hand area detected by the image recognition device of this embodiment can be used, for example, as an image for detecting wrist joint points when recognizing a hand gesture or the like. A technique using machine learning is known as a hand gesture recognition technique, and for example, the positions of hand joint points and the like are detected from an image for detecting wrist joint points, and hand gestures and the like are recognized from changes in the positions of the wrist joint points.

[0012] Here, it is known that the detection accuracy of the wrist joints affects the resolution of the images of the hand regions of the right and left hands input to a machine learning model for detecting hand joint points. In other words, the detection accuracy of the wrist joints can be improved by using a hand region image with high resolution. In addition, the technology of Patent Document 1 described above does not detect joint points including the fingertips of the hand, so it cannot be applied to hand gesture recognition technology that requires information on the wrist joint points. In addition, the object detection model by machine learning disclosed in Non-Patent Document 1 described above has difficulty in correctly classifying the left and right of the hand based on the hand shape alone. The image recognition device of this embodiment can improve the classification accuracy of the left and right of the detected hand region by utilizing the overlapping manner of the hand region and the classification result of the left and right hand.

[0013] <First embodiment> FIG. 1 is a block diagram showing an example of the functional configuration of an image recognition device 100 which is an example of an information processing device according to the first embodiment. The image recognition device 100 of this embodiment has an image acquisition unit 101, a first detection unit 102, a first selection unit 103, a second selection unit 104, a first correction unit 105, a control unit 106, a memory 107, and a display unit 108.

[0014] The image acquisition unit 101 acquires data of an image captured by a camera (imaging device) mounted on the HMD. That is, the image acquired by the image acquisition unit 101 is an image captured from the viewpoint of a user wearing the HMD. The image acquisition unit 101 outputs the acquired captured image to the first detection unit 102 and the control unit 106.

[0015] The first detection unit 102 detects a hand region from an input photographed image using a machine learning model that has been machine-learned to detect a hand region as a region of a detection target object, and acquires information indicating a classification result of whether the detected hand region is a left or right hand region for each detected hand region. The information indicating the classification result is information indicating a result of determining whether the hand region is a right hand or a left hand, and will be referred to as a hand label hereinafter. The machine learning model is an object detection model that has been trained specifically for image detection such as detecting the left and right hand regions of a user wearing an HMD from an image from the viewpoint of the user. In the case of this embodiment, the first detection unit 102 is assumed to be equipped with a machine learning model using, for example, a CNN (Convolutional Neural Network). The first detection unit 102 that detects a hand region using a machine learning model detects the hand region as, for example, a rectangular frame (hereinafter referred to as a hand frame). Then, the first detection unit 102 outputs information on the hand region (hand frame) and hand label detected from the photographed image as hand information to the first selection unit 103, the second selection unit 104, and the control unit 106.

[0016] The first selection unit 103 acquires a value representing the overlapping relationship between the overlapping hand regions based on the multiple hand information (information on multiple hand regions and hand labels) acquired by the first detection unit 102. Furthermore, when the value representing the overlapping relationship between the hand regions is equal to or greater than a predetermined first threshold, the first selection unit 103 selects one hand region among the multiple hand regions having the overlapping relationship as a first hand region and selects the hand label in the first hand region as a first hand label. In this embodiment, the first selection unit 103 selects the hand region having the smallest size among the multiple hand regions having the overlapping relationship as the first hand region and selects the hand label in the first hand region as the first hand label.

[0017] In this embodiment, the first selection unit 103 obtains a first IoU (Intersection over Union) 1 calculation value as a value representing the overlapping relationship between the hand regions. IoU is, for example, a value obtained by dividing the area of ​​the intersection of two regions by the area of ​​the union of the two regions, in other words, a value indicating the overlapping ratio. Since IoU is a value representing the overlapping ratio in this way, the closer the IoU value is to 1, the more the two overlapping hand regions overlap. Furthermore, the first selection unit 103 determines whether the first IoU 1 is equal to or greater than a predetermined first threshold value that is set in advance. The first threshold value is a value that is set in advance to determine that the hand regions are in an overlapping relationship. For example, when the area of ​​the overlapping region that is the intersection of the first region and the second region that are in an overlapping relationship is RAI1, the area of ​​the total region that is the union of the first region and the second region is RAU1, and the first threshold value is Th1, the first selection unit 103 performs calculation and comparison according to the following formula (1).

[0018] IoU1=(RAI1 / RAU1)≧Th1 Formula (1)

[0019] Then, as shown in formula (1), when the first IoU1 is equal to or greater than the first threshold Th1, the first selection unit 103 selects the smallest hand region among the multiple overlapping hand regions as the first hand region and selects the hand label in the first hand region as the first hand label. The first hand region and the first hand label selected by the first selection unit 103 are output to the first correction unit 105.

[0020] The second selection unit 104 selects another hand region that is superimposed on the first hand region selected by the first selection unit 103 as a second hand region among the multiple hand regions acquired by the first detection unit 102, and selects a hand label in the second hand region as a second hand label. The second hand region and the second hand label selected by the second selection unit 104 are output to the first correction unit 105.

[0021] The first correction unit 105 corrects the first hand label based on the first hand region and the first hand label input from the first selection unit 103 and the second hand region and the second hand label input from the second selection unit 104. In this embodiment, the first correction unit 105 corrects the first hand label by rewriting it with the second hand label.

[0022] In this embodiment, when the value representing the overlapping relationship between the overlapping area of ​​the first hand region and the second hand region and the second hand region is equal to or less than a predetermined second threshold, the first correction unit 105 corrects the first hand label of the first hand region to be rewritten with the second hand label. The first correction unit 105 acquires a second IoU2 calculation value as a value representing the overlapping relationship between the overlapping area of ​​the first hand region and the second hand region and the second hand region. As described above, IoU2 is a value obtained by dividing the area of ​​the intersection of the two regions by the area of ​​the union of the two regions (a value indicating the overlap ratio), and the closer the IoU2 value is to 1, the more the two hand regions overlap. Then, the first correction unit 105 compares the second IoU2 with a predetermined second threshold. The second threshold is a threshold value that is preset to determine whether the classification result of the hand region acquired by the first detection unit 102 is correct or not. For example, if the area of ​​the overlapping area which is the intersection of the first hand area and the second hand area is RAI2, the area of ​​the total area which is the union of the overlapping area and the second hand area is RAU2, and the second threshold is Th2, the first correction unit 105 performs the calculation and comparison process of the following equation (2).

[0023] IoU2=(RAI2 / RAU2)≦Th2 Formula (2)

[0024] In addition, when the second IoU2 in formula (2) is equal to or smaller than the second threshold Th2, the first correction unit 105 determines that the first hand label is a misjudged label and corrects the first hand label in the first hand region to be rewritten with the second hand label. On the other hand, when the second IoU2 exceeds the second threshold Th2, the first correction unit 105 determines that the first hand label is a correctly judged label and does not perform correction by rewriting the first hand label.

[0025] The memory 107 holds an information processing program executed by the image recognition device 100 of this embodiment and a machine learning model for detecting a hand region used in the first detection unit 102. Note that the information processing program here refers to a program for executing the processes of various flowcharts described later in this embodiment. Also, the memory 107 stores, under the control of the control unit 106, a captured image captured by a camera and acquired by the image acquisition unit 101, hand information acquired by the first detection unit 102, and the like. The display unit 108 is a monitor such as a liquid crystal monitor provided in the HMD, and displays an image according to display image data generated by the control unit 106 as described below.

[0026] The control unit 106 performs overall control of the image recognition device 100. The control unit 106 is realized by, for example, a CPU (Central Processing Unit) or the like. The control unit 106 also detects the wrist joint from the image of the hand region that has been subjected to correction processing or the like by the first correction unit 105, that is, the image of the hand region where the hand classification result is determined to be correct, and also performs hand gesture recognition or the like from a change in the wrist joint point position. Note that the wrist joint detection technology and gesture recognition technology are well-known technologies, and therefore detailed description thereof will be omitted here.

[0027] Furthermore, the control unit 106 reads out the captured image acquired by the image acquisition unit 101 and stored in the memory 107, and further generates display image data to be displayed on the monitor by the display unit 108 using the hand information and the information indicating the result of the wrist joint detection described above. For example, when the hand information is input from the first detection unit 102, the control unit 106 generates display image data in which an image of a hand frame and a hand label representing a hand area is superimposed on the captured image based on the hand information. Furthermore, when the hand information is corrected by the first correction unit 105, the control unit 106 generates display image data in which an image of a hand area and a hand label based on the corrected hand information is superimposed on the captured image. Note that the control unit 106 can also generate display image data in which information indicating the result of the wrist joint detection from the hand information is superimposed on the captured image, but a description of the display of the information indicating the result of the wrist joint detection will be omitted.

[0028] FIG. 2 is a diagram used to explain the hand region and hand label acquired by the first detection unit 102 of the image recognition device 100, and the correction process of the hand label, and shows an example of a display image in which an image of a hand frame and hand label representing the hand region is superimposed on a captured image. Image 200 in Fig. 2(a) is an example of a display image in which hand frames and hand labels of multiple hand regions acquired from a captured image by first detection unit 102 are superimposed on the captured image. In the example of Fig. 2(a), two hand regions represented by hand frames 202 and 204 are detected by first detection unit 102. In addition, in the case of Fig. 2(a), hand label 203 in hand frame 202 is "left", indicating a classification result that the hand is a left hand, and hand label 205 in hand frame 204 is "right", indicating a classification result that the hand is a right hand.

[0029] Here, the hand 201 in the image 200 in Fig. 2(a) is the right hand of the user wearing the HMD. That is, the hand label 203 in the hand frame 202 is erroneously determined as "left" instead of "right", so that the hand label 203 in the hand frame 202 is erroneously determined as "left". The first correction unit 105 performs the above-mentioned correction process based on the information output from the first selection unit 103 and the second selection unit 104, thereby correcting the hand label 203 in the hand frame 202 to be "right".

[0030] Image 210 in FIG. 2(b) shows an example of a display image in which hand frame 202 and hand label 206 are superimposed on hand 201 in the captured image after correction processing by first correction unit 105 is performed on the hand region detected by first detection unit 102 from the captured image. That is, first correction unit 105 corrects hand label 203, which is erroneously determined among hand frames 202, 204 and hand labels 203, 205 before correction shown in FIG. 2(a), by rewriting it with hand label 205 to obtain hand label 206. In addition, control unit 106 deletes hand frame 204 used for correcting erroneously determined hand label 203 of hand frame 202 and makes it invisible. As a result, the image displayed on the monitor by display unit 108 becomes image 210 as shown in FIG. 2(b).

[0031] FIG. 3 is a diagram used to explain an example of a calculation of the first IoU1 in the first selection unit 103 and an example of a calculation of the second IoU2 in the first correction unit 105. In FIG. FIG. 3(a) is a diagram used to explain an example of calculation of the first IoU1 performed by the first selection unit 103 for the example of the image 200 in FIG. 2(a) described above. The first selection unit 103 calculates the first IoU1 by dividing the area of ​​the overlapping area 301, which is the intersection of the two hand areas of the hand frame 202 and the hand frame 204, by the area of ​​the total area 302, which is the union of the two hand areas of the hand frame 202 and the hand frame 204. Then, as described above, the first selection unit 103 compares the first IoU1 with the first threshold Th1. If the first IoU1 is equal to or greater than the first threshold Th1, the first selection unit 103 selects the hand area of ​​the smallest hand frame 202 out of the two hand areas of the hand frame 202 and the hand frame 204 as the first hand area, and selects the hand label of the first hand area as the first hand label. In addition, the second selection unit 104 selects another hand area that is superimposed on the first hand area (i.e., a hand area of ​​a hand frame 204 other than the hand frame 202) as the second hand area, and selects the hand label of the second hand area as the second hand label.

[0032] FIG. 3(b) is a diagram used to explain a calculation example of the second IoU2 in the first correction unit 105. The first correction unit 105 calculates the second IoU2 by dividing the area of ​​the overlapping area 301, which is the intersection of the two hand areas of the hand frame 202 and the hand frame 204 in FIG. 3(a), by the area of ​​the total area 303, which is the union of the overlapping area 301 and the hand frame 204. Then, as described above, the first correction unit 105 determines whether the second IoU2 is equal to or less than the second threshold Th2. If the second IoU2 is equal to or less than the second threshold Th2, the first correction unit 105 corrects the first hand label of the first hand area to the second hand label. On the other hand, if the second IoU2 exceeds the second threshold Th2, the first correction unit 105 does not perform correction by rewriting the first hand label.

[0033] Fig. 4 is a flowchart showing the flow of information processing in the image recognition device 100 shown in Fig. 1. Note that each process shown in the flowchart in Fig. 4 is performed by each functional unit shown in Fig. 1 under the control of the control unit 106. First, in the processing of step S401, the first detection unit 102 acquires a hand region and a hand label from the image input by the image acquisition unit 101, and sends them as hand information to the first selection unit 103, the second selection unit 104, and the control unit 106.

[0034] Next, in step S402, the first selection unit 103 determines whether the first IoU1 calculated from the overlapping hand regions is equal to or greater than a first threshold Th1. If the first IoU1 is equal to or greater than the first threshold Th1, the first selection unit 103 proceeds to step S403, and if the first IoU1 is less than the first threshold Th1, the first selection unit 103 proceeds to step S408.

[0035] When the process proceeds to step S403, the first selection unit 103 selects the smallest hand region among the overlapping hand regions as the first hand region, and selects the hand label of the first hand region as the first hand label. In the example of image 200 in FIG. 2(a) described above, the first selection unit 103 selects the hand region and hand label of the hand frame 202.

[0036] Next, in step S404, the second selection unit 104 selects another hand region that overlaps with the first hand region selected by the first selection unit 103 as a second hand region, and selects the hand label of the second hand region as a second hand label. In the case of the example of image 200 in FIG. 2(a) described above, the second selection unit 104 selects the hand region and hand label of hand frame 204 that overlaps with hand frame 202.

[0037] Next, in step S405, the first correction unit 105 calculates a second IoU2 from the first hand region and the second hand region, and determines whether the second IoU2 is equal to or smaller than a second threshold Th2. If the second IoU2 is equal to or smaller than the second threshold Th2, the first correction unit 105 proceeds to step S406, and if the second IoU2 exceeds the second threshold Th2, the first correction unit 105 proceeds to step S408.

[0038] When the process proceeds to step S406, the first correction unit 105 rewrites the first hand label in the first hand region by overwriting it with the second hand label in the second hand region. In the example of Fig. 2(a) and Fig. 2(b) described above, the first correction unit 105 rewrites the hand label 203 in the hand frame 202 with the hand label 205 to create the hand label 206. Furthermore, in step S407, the first correction unit 105 deletes the second hand region and the second hand label.

[0039] Next, in the process of step S408, control unit 106 generates a display image in which the hand frame and hand label of the hand area acquired in steps S402 to S407 are superimposed on the captured image, and sends the display image to display unit 108 to display it on the monitor. Thereafter, the control unit 106 determines whether or not an end instruction has been received from the user, for example, via an operation device not shown, and if an end instruction has not been received, returns the process to step S401. On the other hand, if an end instruction has been received, the control unit 106 ends the process of the flowchart in FIG. 4.

[0040] In the examples of Figures 2 and 3 described above, the user's right hand is detected and the hand label is corrected if it is erroneously determined. However, it is possible to obtain the hand label of the user's left hand in a similar manner and correct any erroneous determination.

[0041] As described above, the image recognition device 100 according to the first embodiment obtains a value representing the overlapping relationship between overlapping hand regions, and corrects the hand label based on the result of comparing the value representing the overlapping relationship with a threshold. As a result, according to this embodiment, it is possible to perform accurate hand detection and left / right hand classification.

[0042] Second Embodiment In the first embodiment, a method for correcting an erroneously determined hand label using a value representing an overlapping relationship between hand regions (a value indicating an overlapping ratio) was described. In the second embodiment, in addition to the method described in the first embodiment, an example is described in which a reliability (accuracy) indicating the likelihood that an object (hand in this embodiment) detected from a captured image is a detection target object is obtained, and correction is performed based on the reliability.

[0043] FIG. 5 is a block diagram showing an example of a functional configuration of an image recognition device 500 which is an example of an information processing device according to the second embodiment. The image recognition device 500 according to this embodiment is mounted on an HMD in the same manner as in the example of the first embodiment. The image recognition device 500 according to the second embodiment has an image acquisition unit 101, a second detection unit 501, a first selection unit 103, a second selection unit 104, a second correction unit 502, a control unit 106, a memory 107, and a display unit 108. The image acquisition unit 101, the first selection unit 103, the second selection unit 104, the control unit 106, the memory 107, and the display unit 108 are generally similar to those of the first image recognition device 100 described above, and therefore detailed descriptions thereof will be omitted. Below, the second detection unit 501 and the second correction unit 502, which are functional units different from those of the image recognition device 100 according to the first embodiment, will be mainly described.

[0044] The second detection unit 501 has a function similar to that of the first detection unit 102 described in the first embodiment, and also has an estimation function for estimating the likelihood that the detected object region is the detection target object (the likelihood that the region is a hand in this embodiment). The second detection unit 501 calculates a reliability (accuracy) indicating the likelihood of the hand region detected from the captured image in the same manner as described above, and acquires the reliability as a hand estimation score. Note that a technique for calculating the reliability indicating the likelihood of the object as an estimation score can be a technique used in a known R-CNN (Regional Convolutional Neural Network) or the like. The second detection unit 501 outputs the hand estimation score to the second correction unit 502.

[0045] The second correction unit 502 has a function of correcting hand information (correcting hand labels) by using the hand estimation score input from the second detection unit 501, in addition to the same functions as those described in the first embodiment. The second correction unit 502 of the second embodiment calculates the second IoU2 value by Equation (2) in the same manner as the first correction unit 105 of the first embodiment, and further compares the second IoU2 value with the second threshold Th2. When the second IoU2 value is equal to or less than the second threshold Th2, the second correction unit 502 corrects the first hand label in the first hand region to the second hand label in the same manner as the first correction unit 105 described above. On the other hand, when the second IoU2 value exceeds the second threshold Th2, the second correction unit 502 compares the hand estimation scores between the hand regions in a superimposed relationship calculated by the second detection unit 501. Then, the second correction unit 502 determines that the orientation of the hand label with a large (high) hand estimation score is highly reliable, and adopts the hand region and hand label with the higher reliability as hand information. On the other hand, the second correction unit 502 deletes the hand region and hand label with a low hand estimation score together with the hand estimation score.

[0046] Fig. 6 is a diagram used to explain hand information (hand area, hand label, hand estimation score) acquired by second detection unit 501 of image recognition device 500 according to the second embodiment, and hand label correction processing by second correction unit 502. Fig. 6 shows an example of a display image in which an image showing a hand frame representing a hand area, a hand label, and a hand estimation score are superimposed on a captured image.

[0047] Image 600 in Fig. 6(a) is an example of a display image in which the hand frame, hand label, and hand estimation score of the hand region acquired by second detection unit 501 from the captured image are superimposed on the captured image. In the example of Fig. 6(a), an example is shown in which second detection unit 501 has acquired two hand regions represented by hand frames 602 and 605. In the case of Fig. 6(a), the hand label 603 of hand frame 602 is "left", indicating that it is classified as a left hand, and the value of hand estimation score 604 is 80. On the other hand, the hand label 606 of hand frame 605 is "right", indicating that it is classified as a right hand, and the value of hand estimation score 607 is 90.

[0048] Here, the hand 601 in the image 600 in Fig. 6(a) is the right hand of the user wearing the HMD. That is, the hand label 603 in the hand frame 602 in the image 600 in Fig. 6(a) is "left" instead of "right", so the hand label 603 in the hand frame 602 is misjudged. Also, the hand estimation score 604 of the hand frame 602 is 80, while the hand estimation score 607 of the hand frame 605 is 90, meaning that the hand estimation score 607 of the hand frame 605 is higher.

[0049] An image 608 in FIG. 6(b) shows an example of a display image after the second correction unit 502 performs correction processing on the hand label of the hand region acquired by the second detection unit 501 from the captured image. As described above, the second correction unit 502 determines whether the value of the second IoU2 calculated by the formula (2) is equal to or less than the second threshold Th2, similar to the first correction unit 105 in the first embodiment. However, in the case of the image 600 example in FIG. 6(a), since there is not much difference in size between the hand frame 602 and the hand frame 605, it is assumed that the value of the second IoU2 exceeds the second threshold Th2. If the value of the second IoU2 exceeds the second threshold Th2, the second correction unit 502 compares the hand estimation score 604 and the hand estimation score 607 acquired by the second detection unit 501. Then, the second correction unit 502 leaves the hand information with the larger hand estimation score and deletes the hand information with the smaller hand estimation score. That is, in the case of image 600 in Fig. 6(a), the value of hand estimation score 604 of hand frame 602 is 80, while the value of hand estimation score 607 of hand frame 605 is 90, so second correction unit 502 deletes the hand region, hand label 603, and hand estimation score 604 of hand frame 602. That is, the displayed image after deleting hand frame 602, hand label 603, and hand estimation score 604 in Fig. 6(a) becomes image 608 in which hand frame 605, hand label 606, and hand estimation score 607 are superimposed on hand 601 in the captured image, as shown in Fig. 6(b).

[0050] Fig. 7 is a flowchart showing the flow of information processing in the image recognition device 500 of the second embodiment. Steps S402 to S404, S408, and S409 are generally similar to the processing in Fig. 4 described above, and detailed explanations thereof will be omitted. The following explanation will focus on steps S701 and S702 to S705, which are different processes from the flowchart in Fig. 4.

[0051] First, in the process of step S701, the second detection unit 501 acquires a hand region, a hand label, and a hand estimation score from the image input by the image acquisition unit 101. The second detection unit 501 sends the hand region and the hand label to the control unit 106, the first selection unit 103, and the second selection unit 104 as in the first embodiment, and sends the hand estimation score to the second correction unit 502. After this step S701, the process of the image recognition device 500 proceeds to the above-mentioned step S402. Then, after the processes of the above-mentioned steps S403 and S404, the process of the image recognition device 500 proceeds to step 702.

[0052] In step S702, the second correction unit 502 determines whether the second IoU2 is equal to or less than the second threshold Th2. If the second IoU2 is equal to or less than the second threshold Th2, the second correction unit 502 proceeds to step S704, whereas if the second IoU2 exceeds the second threshold Th2, the second correction unit 502 proceeds to step S703.

[0053] When proceeding to step S703, the second correction unit 502 compares the hand estimation scores of the overlapping hand regions, and deletes the hand frame, hand label, and hand estimation score in the hand region with the smaller hand estimation score, i.e., the lower reliability. In the case of the example of image 600 shown in FIG. 6(a) described above, the second correction unit 502 compares the hand estimation score 604 with the hand estimation score 607, and deletes the hand estimation score 604 with the lower score, and the corresponding hand frame 602 and hand label 603. After this step S703, the second correction unit 502 proceeds to step S408.

[0054] Also, when the process proceeds to step S704, the second correction unit 502 rewrites the first hand label in the first hand region by overwriting it with the second hand label in the second hand region, similar to the process in step S406 described above. In the case of the examples in Figures 6(a) and 6(b) described above, the second correction unit 502 rewrites the hand label 603 in the hand frame 602 to a hand label 606. Furthermore, in step S705, the first correction unit 105 deletes the second hand region and the second hand label together with the hand estimation score. In the case of the example of the image 600 in FIG. 6(a) described above, the second correction unit 105 erases the hand frame 605 in the image 600. After step S705, the second correction unit 502 proceeds to step S408.

[0055] After processing step S408 similar to that of FIG. 4 described above, when proceeding to step S409, the control unit 106 determines whether or not there is an end instruction from the user via an operation device (not shown), and if there is no end instruction, returns the processing to step S701.

[0056] In the example of Figure 6, we have described an example in which the user's right hand is detected and the hand label is corrected if it is incorrectly determined. However, it is also possible to detect the user's left hand and correct the hand label in a similar manner. As described above, in the image recognition device 500 according to the second embodiment, in addition to correcting the hand label based on the overlapping of the hand regions as in the first embodiment, the hand label is corrected using the hand estimation score, thereby enabling more accurate correction. That is, according to the second embodiment, it is possible to realize more accurate hand detection and left / right hand classification.

[0057] <Third embodiment> In the first and second embodiments, examples of correcting erroneously determined hand labels for each of the left and right hands detected from a photographed image are described. In the third embodiment, in addition to the functions described in the first and second embodiments, an example of correcting hand labels for hand regions that are in an overlapping relationship based on hand information for hand regions that are not in an overlapping relationship between the left and right hands detected from a photographed image is described.

[0058] FIG. 8 is a block diagram showing an example of a functional configuration of an image recognition device 800 which is an example of an information processing device according to the third embodiment. The image recognition device 800 according to this embodiment is mounted on an HMD as in the above-described embodiment. The image recognition device 800 according to the third embodiment has an image acquisition unit 101, a first detection unit 102, a first selection unit 103, a second selection unit 104, a third selection unit 801, a third correction unit 802, a control unit 106, a memory 107, and a display unit 108. The image acquisition unit 101, the first detection unit 102, the first selection unit 103, the second selection unit 104, the control unit 106, the memory 107, and the display unit 108 are generally similar to those of the first image recognition device 100 described above, and therefore detailed descriptions thereof will be omitted. Below, the third selection unit 801 and the third correction unit 802, which are functional units different from those of the image recognition device 100 according to the first embodiment, will be mainly described.

[0059] Hand information of multiple hand regions acquired from the captured image by the first detection unit 102 is input to the third selection unit 801. The third selection unit 801 determines a hand region that is not overlapping among the multiple hand regions detected by the first detection unit 102 as a third hand region, selects a hand label of the third hand region as a third hand label, and outputs them to the third correction unit 802 as hand information.

[0060] The third correction unit 802 has a function of performing correction processing based on the hand information input from the third selection unit 801, in addition to the same functions as those described in the first embodiment. The third correction unit 802 calculates the value of the second IoU2 according to formula (2) in the same manner as the first correction unit 105 in the first embodiment, and further compares the second IoU2 with the second threshold Th2. If the second IoU2 is equal to or less than the second threshold Th2, the third correction unit 802 corrects the first hand label to be rewritten with the second hand label in the same manner as the first embodiment described above. On the other hand, in the third embodiment, if the value of the second IoU2 is equal to or less than the second threshold Th2, the third correction unit 802 determines whether or not a third region and a third hand label that do not have a superimposed relationship exist. If a third hand region and a third hand label exist, the third correction unit 802 performs a correction process to rewrite the first hand label with the third hand label.

[0061] In the image recognition device 800 of the third embodiment, the first detection unit 102 may have the function of the second detection unit 501 of the second embodiment described above. In this case, the third correction unit 802 also has the function of the second correction unit 502 of the second embodiment described above.

[0062] Fig. 9 is a diagram used to explain hand information (hand area, hand label) acquired by the first detection unit 102 of the image recognition device 800 according to the third embodiment, and correction processing of the hand label by the third correction unit 802. Fig. 9 shows an example of a display image in which an image of a hand frame representing a hand area and an image representing a hand label are superimposed on a captured image.

[0063] Image 900 in Fig. 9(a) is an example of a display image in which a hand frame and a hand label of a hand region acquired by first detection unit 102 from a captured image are superimposed on the captured image. In the example of Fig. 9(a), two hands, hand 901 and hand 906, are captured in image 900. Assume that first detection unit 102 detects two hand regions, hand frame 902 and hand frame 904, which have an overlapping relationship for one hand 901, and detects one hand region, hand frame 907, which does not have an overlapping relationship for the other hand 906. Furthermore, in the case of Fig. 9(a), hand label 908 of hand frame 907 is "right", which indicates a classification result that the hand is a right hand, and hand label 903 of hand frame 902 and hand label 905 of hand frame 904 are both "right", which indicates a classification result that the hand is a right hand.

[0064] Here, one hand 901 in image 900 in Fig. 9(a) is the left hand of a user wearing an HMD, and the other hand 901 is the right hand of the user. A hand label 903 for hand frame 902 and a hand label 905 for hand frame 904 in image 900 in Fig. 9(a) are "right" instead of "left", which should be correct. In other words, both the hand label 903 and the hand label 905 are misclassified results. On the other hand, a hand label 908 for hand frame 907 in image 900 is "right", so the hand label 908 for hand frame 907 was correctly classified.

[0065] An image 609 in FIG. 9B shows an example of a display image after correction processing by the third correction unit 802 is performed based on the hand frame and hand label selected by the third selection unit 801. The third correction unit 802 calculates a second IoU2 for the hand frame 902 and hand frame 904 that are superimposed on each other, in the same manner as the first correction unit 105 in the first embodiment, and determines whether the second IoU2 is equal to or smaller than the second threshold Th2. If the second IoU2 is equal to or smaller than the second threshold Th2, the third correction unit 802 corrects the hand label in the same manner as described above.

[0066] On the other hand, when the second IoU2 exceeds the second threshold Th2, the third correction unit 802 judges whether or not there is a hand region that does not have an overlapping relationship, and when there is a hand region that does not have an overlapping relationship, judges that the hand region is a region in which the hand label is correctly determined. Then, the third correction unit 802 corrects the first hand label in the first hand region of the hand frames that have an overlapping relationship to be a different hand label corresponding to the hand label of the correctly determined hand region. That is, when the hand label of the correctly determined hand region is "right", the third correction unit 802 rewrites the hand label of the hand frame that has an overlapping relationship to "left", which is a different hand label corresponding to "right". In the example of the image 900 in FIG. 9(a), the third correction unit 802 rewrites the hand label 903 of the hand frame 902 that has an overlapping relationship to "left", which is a different hand label corresponding to the hand label 908 of the hand frame 907 that does not have an overlapping relationship.

[0067] Furthermore, the third correction unit 802 deletes the second hand region and the second hand label from among the hand frames and hand labels that overlap. That is, the third correction unit 802 deletes the hand frame 904, which is the second hand region, and its hand label 905 from among the two hand frames 902 and hand frames 904 that overlap as shown in Fig. 9(a). As a result, the image 909 shown in Fig. 9(b) becomes an image in which the hand frame 902, whose hand label 910 has been corrected to "left", is superimposed on the hand 901, and the hand frame 907, whose hand label 908 is "right", is superimposed on the hand 906.

[0068] Fig. 10 is a flowchart showing the flow of information processing in the image recognition device 800 of the third embodiment. Steps S401 to S403, S408, and S409 are generally similar to the processing in Fig. 4 described above, and therefore detailed explanations thereof will be omitted. The following description will focus on steps S1000 to S1005, which are different processes from the flowchart in Fig. 4.

[0069] In step S401 of the third embodiment, the first detection unit 102 acquires a hand region and a hand label from a captured image. In the example of Fig. 9(a), hand frames 902 and 904 of two hand regions having an overlapping relationship and their hand labels 903 and 905 are acquired for one hand 901, and a hand frame 907 and a hand label 908 of a hand region having no overlapping relationship are acquired for the other hand 906.

[0070] Next, when the process proceeds from step S402 to step S403, in the case of image 900 in the example of Fig. 9(a), a hand frame 902 and a hand label 903 are selected as the first region and the hand label. Then, after step S403, the process of image recognition device 800 proceeds to step S1000.

[0071] When proceeding to step S1000, the second selection unit 104 selects the second hand region and the second hand label as in the above embodiment, and outputs them to the third correction unit 802. In the case of the image 900 in FIG. 9(a), a hand frame 904 and a hand label 905 are selected as the second region and the hand label. Also, in step S1000, if there is a hand region that is not in an overlapping relationship among the multiple hand regions detected by the first detection unit 102, the third selection unit 801 selects the hand region as a third hand region and the hand label of the third hand region as a third hand label. Then, the third selection unit 801 outputs the third hand region and the third hand label as hand information to the third correction unit 802. In the case of the image 900 in FIG. 9(a), a hand frame 907 and a hand label 908 are selected as the third hand region and the hand label that are not in an overlapping relationship.

[0072] Next, in the process of step S1001, the third correction unit 802 determines whether the second IoU2 is equal to or less than the second threshold Th2, as in the above-described embodiment. If the second IoU2 exceeds the second threshold Th2, the third correction unit 802 advances the process to step S408. On the other hand, if the second IoU2 is equal to or less than the second threshold Th2, the third correction unit 802 advances the process to step S1002.

[0073] In step S1002, the third correction unit 802 determines whether or not there is a third hand region and hand label selected by the third selection unit 801 when there is a hand region that is not superimposed. If the third correction unit 802 determines that there is a third hand region and hand label, the process proceeds to step S1003, whereas if it determines that there is no third hand region and hand label, the process proceeds to step S1004.

[0074] When the process proceeds to step S1003, the third correction unit 802 overwrites the first hand label to a different hand label corresponding to the third hand label. In the example of Fig. 9(a) and Fig. 9(b), the third hand label, hand label 908, is "right", so the hand label 903 of the hand frame 902 is rewritten to "left". After this step S1003, the third correction unit 802 proceeds to the process of step S1005.

[0075] Also, when the process proceeds to step S1004, the third correction unit 802 overwrites the first hand label in the first hand region with the second hand label in the second hand region, as in the process of step S406 described above. In the case of the examples of Figures 9(a) and 9(b) described above, in step S1004, the third correction unit 502 rewrites the hand label 903 in the hand frame 902 with the hand label 905. After step S1004, the third correction unit 802 proceeds to the process of step S1005.

[0076] In step S1005, the third correction unit 802 deletes the hand frame and hand label of the second hand region. As a result, as shown in FIG 9B, an image 909 is displayed on the monitor in which a hand frame 902 with a hand label 910 corrected to "left" is superimposed on a hand 901, and a hand frame 907 with a hand label 908 of "right" is superimposed on a hand 906. Thereafter, the process proceeds to step S408, and further to step S409, where the control unit 106 determines whether or not there is an end instruction from the user via an operation device (not shown), and if there is no end instruction, the process returns to step S401.

[0077] As described above, the image recognition device 800 according to the third embodiment can improve classification accuracy by correcting hand labels of overlapping hand regions based on hand labels of non-overlapping hand regions in addition to the correction process similar to that of the first embodiment. Note that the third embodiment has been described using an example in which both left and right hands are captured in a captured image from a user's viewpoint, but the third embodiment can also be applied to the case of one hand.

[0078] In the above example, an HMD equipped with a camera was given as an example of an application of an information processing device, but the present invention is not limited to an HMD and can also be applied to other information processing devices such as smartphones, tablet terminals, digital cameras, and personal computers. In the above example, an example was given in which there were two hand regions that overlapped, but this is not limited to this, and this embodiment can also be applied to cases in which three or more hand regions overlap.

[0079] <Hardware configuration example> FIG. 11 is a diagram illustrating an example of a main hardware configuration of an HMD to which an image recognition device according to an embodiment is applied. In FIG. 11, a CPU 1120 centrally controls each device connected via a bus 1110. The CPU 1120 reads and executes a program stored in a ROM (read-only memory) 1130. An operating system (OS), each information processing program according to this embodiment, a device driver, and the like are stored in the ROM 1130. These programs are read from the ROM 1130, expanded in a RAM (random access memory) 1140, and appropriately executed by the CPU 1120. A camera 1150 is an imaging device that captures a subject such as an object. A monitor 1160 is a monitor such as a liquid crystal display, and an image or the like is displayed by a display unit 108. A sensor 1170 is a motion sensor or the like, and detects the orientation and movement of the HMD in the real space. An I / F 1180 inputs a signal from an external device in a format that can be processed by the HMD of this embodiment, and outputs a signal in a format that can be processed by the external device.

[0080] The CPU 1120 loads the information processing program according to this embodiment from the ROM 1130 into the RAM 1140 and executes it, thereby realizing the image recognition device in each of the above-mentioned embodiments and the processing of each of the above-mentioned flowcharts. When configuring hardware as an alternative to software processing using the CPU 1120, it is sufficient to configure a calculation unit or circuit corresponding to each functional unit in each of the above-mentioned embodiments. That is, each functional unit in the information processing device of this embodiment may be entirely implemented by hardware, or may be partially executed by software. When executed by software, the software is executed using the CPU 1120, ROM 1130, RAM 1140, etc. in FIG. 11.

[0081] The present invention can also be realized by supplying a program for implementing one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions. The above-mentioned embodiments are merely examples of the implementation of the present invention, and the technical scope of the present invention should not be interpreted as being limited by these. In other words, the present invention can be implemented in various forms without departing from its technical concept or main characteristics.

[0082] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) An acquisition means for acquiring an object region from an image and a classification result of the object for the region; A selection means for selecting two or more areas having an overlapping relationship from among the plurality of areas acquired by the acquisition means; a correction means for correcting a classification result of the object based on the overlapping relationship of the two or more regions selected by the selection means; 13. An information processing device comprising: (Configuration 2) The selection means is a first selection means for selecting, as a first region, a smallest region among the two or more regions, in which a first value representing an overlap relationship is equal to or greater than a first threshold; a second selection means for selecting one area different from the first area as a second area from among the two or more areas; having 2. The information processing apparatus according to configuration 1, wherein the correction means corrects the classification result for the first region using the classification result for the second region. (Configuration 3) The information processing device according to configuration 2, wherein the correction means corrects the classification result for the first region with the classification result for the second region when a second value representing an overlapping relationship between the first region and the second region is equal to or smaller than a second threshold value. (Configuration 4) The acquisition means also acquires a confidence level of the classification result of the object for the region, 2. The information processing apparatus according to configuration 1, wherein the correction means corrects a classification result of the object based on the overlapping relationship and the reliability of the two or more regions. (Configuration 5) The selection means is a first selection means for selecting, as a first region, a smallest region among the two or more regions, in which a first value representing an overlap relationship is equal to or greater than a first threshold; a second selection means for selecting one area different from the first area as a second area from among the two or more areas; having The correction means is when a second value representing an overlapping relationship between the first region and the second region is equal to or smaller than a second threshold, correcting the classification result for the first region with the classification result for the second region; 5. The information processing device according to configuration 4, wherein when the second value exceeds a second threshold, one of the first region and the second region, which has a lower reliability, is deleted. (Configuration 6) The selection means selects an area that does not have an overlapping relationship from among the plurality of areas acquired by the acquisition means, The information processing device according to configuration 1, characterized in that the correction means corrects the classification result of the object based on the overlapping relationship of the two or more areas and the classification result of the object for the area that does not have the overlapping relationship. (Configuration 7) The selection means is a first selection means for selecting, as a first region, a smallest region among the two or more regions, in which a first value representing an overlap relationship is equal to or greater than a first threshold; a second selection means for selecting one area different from the first area as a second area from among the two or more areas; a third selection means for selecting the region not having an overlapping relationship as a third region; having The correction means is when a second value representing an overlapping relationship between the first region and the second region is equal to or smaller than a second threshold, correcting the classification result for the first region with the classification result for the second region; 7. The information processing device according to configuration 6, characterized in that, when the second value exceeds a second threshold, the classification result of the first region is corrected based on the classification result of the third region. (Configuration 8) The information processing device according to configuration 7, wherein the correction means corrects the classification result of the first region with a classification result different from the classification result of the third region when the second value exceeds a second threshold value. (Configuration 9) 9. The information processing device according to any one of configurations 2, 3, 5, 7, and 8, wherein the correction means corrects the classification result for the first region so as to overwrite it with the classification result for the second region. (Configuration 10) 10. The information processing device according to any one of configurations 2, 3, 5, 7, 8, and 9, wherein an IoU (Intersection over Union) between regions is used as the value representing the overlapping relationship. (Configuration 11) the object is a hand; 11. The information processing device according to any one of configurations 1 to 10, wherein the acquisition means detects the hand area from the image. (Configuration 12) 12. The information processing device according to claim 11, wherein the acquiring means acquires the classification result indicating the left or right hand. (Configuration 13) 13. The information processing device according to claim 11, wherein the acquisition means detects a region of the user's hand from the image from the user's viewpoint. (Configuration 14) 14. The information processing device according to claim 13, wherein the image is an image captured by a camera mounted on a head mounted display. (Configuration 15) 15. The information processing device according to any one of configurations 11 to 14, further comprising recognition means for detecting positions of hand joint points from the hand area and recognizing a gesture made by the hand based on a change in the positions of the hand joint points. (Method 1) An acquisition step of acquiring an object region from an image and an object classification result for the region; a selection step of selecting two or more regions having an overlapping relationship from among the plurality of regions acquired by the acquisition step; a correction step of correcting a classification result of the object based on the overlapping relationship of the two or more regions selected by the selection step; 13. An information processing method comprising: (Program 1) A program for causing a computer to function as the information processing device according to any one of configurations 1 to 15. [Explanation of symbols]

[0083] 100: image recognition device, 101: image acquisition unit, 102: first detection unit, 103: first selection unit, 104: second selection unit, 105: first correction unit, 106: control unit, 107: memory, 108: display unit

Claims

1. An acquisition means for acquiring an object region from an image and a classification result of the object for the region; A selection means for selecting two or more areas having an overlapping relationship from among the plurality of areas acquired by the acquisition means; a correction means for correcting a classification result of the object based on the overlapping relationship of the two or more regions selected by the selection means; 13. An information processing device comprising:

2. The selection means is a first selection means for selecting, as a first region, a smallest region among the two or more regions, in which a first value representing an overlap relationship is equal to or greater than a first threshold; a second selection means for selecting one area different from the first area as a second area from among the two or more areas; having 2 . The information processing apparatus according to claim 1 , wherein the correction means corrects the classification result for the first region using the classification result for the second region.

3. The information processing device according to claim 2, characterized in that the correction means corrects the classification result for the first region with the classification result for the second region when a second value representing an overlap relationship between the first region and the second region is equal to or less than a second threshold value.

4. The acquisition means also acquires a confidence level of the classification result of the object for the region, The information processing apparatus according to claim 1 , wherein the correction means corrects the classification result of the object based on the overlapping relationship and the reliability of the two or more regions.

5. The selection means is a first selection means for selecting, as a first region, a smallest region among the two or more regions, in which a first value representing an overlap relationship is equal to or greater than a first threshold; a second selection means for selecting one area different from the first area as a second area from among the two or more areas; having The correction means is when a second value representing an overlapping relationship between the first region and the second region is equal to or smaller than a second threshold, correcting the classification result for the first region with the classification result for the second region; 5. The information processing apparatus according to claim 4, wherein, when the second value exceeds a second threshold, one of the first region and the second region, which has a lower reliability, is deleted.

6. The selection means selects an area that does not have an overlapping relationship from among the plurality of areas acquired by the acquisition means, The information processing apparatus according to claim 1 , wherein the correction means corrects the classification result of the object based on the overlapping relationship between the two or more areas and the classification result of the object for the area not having the overlapping relationship.

7. The selection means is a first selection means for selecting, as a first region, a smallest region among the two or more regions, in which a first value representing an overlap relationship is equal to or greater than a first threshold; a second selection means for selecting one area different from the first area as a second area from among the two or more areas; a third selection means for selecting the region not having an overlapping relationship as a third region; having The correction means is when a second value representing an overlapping relationship between the first region and the second region is equal to or smaller than a second threshold, correcting the classification result for the first region with the classification result for the second region; 7. The information processing apparatus according to claim 6, wherein, when the second value exceeds a second threshold value, the classification result of the first region is corrected based on the classification result of the third region.

8. The information processing device according to claim 7, characterized in that, when the second value exceeds a second threshold, the correction means corrects the classification result of the first region with a classification result different from the classification result of the third region.

9. 9. The information processing apparatus according to claim 2, wherein the correction means corrects the classification result for the first region by rewriting the classification result for the second region.

10. 9. The information processing apparatus according to claim 2, wherein an intersection over union (IoU) between regions is used as a value representing the overlapping relationship.

11. the object is a hand; The information processing apparatus according to claim 1 , wherein the acquisition means detects the hand area from the image.

12. The information processing apparatus according to claim 11 , wherein the acquiring means acquires the classification result indicating the left or right hand.

13. The information processing apparatus according to claim 11 , wherein the acquiring means detects a region of the user's hand from the image from the user's viewpoint.

14. 14. The information processing apparatus according to claim 13, wherein the image is an image captured by a camera mounted on a head mounted display.

15. 12. The information processing apparatus according to claim 11, further comprising recognition means for detecting positions of hand joint points from the hand area and recognizing a gesture made by the hand based on a change in the positions of the hand joint points.

16. An acquisition step of acquiring an object region from an image and an object classification result for the region; a selection step of selecting two or more regions having an overlapping relationship from among the plurality of regions acquired by the acquisition step; a correction step of correcting a classification result of the object based on the overlapping relationship of the two or more regions selected by the selection step; 13. An information processing method comprising:

17. Computer, An acquisition means for acquiring an object region from an image and a classification result of the object for the region; A selection means for selecting two or more areas having an overlapping relationship from among the plurality of areas acquired by the acquisition means; a correction means for correcting a classification result of the object based on the overlapping relationship of the two or more regions selected by the selection means; A program that causes the device to function as an information processing device having the above-mentioned configuration.

Images