Information processing device, information processing method, and program

Abstract

An information processing device according to one embodiment of the present invention comprises: an acquisition unit; a first generation unit; a second generation unit; and a recognition unit. The recognition unit acquires a captured image in which an object is captured. The first generation unit generates skeleton information of the object on the basis of the captured image. The second generation unit generates, from the captured image and on the basis of the skeleton information, a recognition image which is an image of a region used for recognizing the object. The recognition unit recognizes the object on the basis of the recognition image and the skeleton information.

Description

Information processing device, information processing method, and program

[0001] The present technology relates to an information processing device, an information processing method, and a program that can be applied to object recognition such as gesture recognition.

[0002] Patent Document 1 discloses a technology for inferring skeleton data based on image data of an operator's hand, and detecting an action (gesture) for operating an electronic device based on the inferred skeleton data.

[0003] Patent No. 7213396

[0004] In the gesture recognition technique described in Patent Document 1, there is a demand for a technique that enables recognition of an object with high accuracy.

[0005] In view of the above circumstances, an object of the present technology is to provide an information processing device, an information processing method, and a program that are capable of recognizing an object with high accuracy.

[0006] To achieve the above object, an information processing device according to one aspect of the present invention includes an acquisition unit, a first generation unit, a second generation unit, and a recognition unit. The recognition unit acquires a captured image of an object. The first generation unit generates skeletal information of the object based on the captured image. The second generation unit generates a recognition image from the captured image based on the skeletal information, which is an image of an area used in recognition processing of the object. The recognition unit recognizes the object based on the recognition image and the skeletal information.

[0007] In this information processing device, skeletal information of the object is generated based on a captured image of the object, and a recognition image used in object recognition processing is generated from the captured image based on the skeletal information.The object is then recognized based on the recognition image and the skeletal information.This makes it possible to recognize the object with high accuracy.

[0008] The second generation unit may detect a portion of interest within the object based on the skeletal information, set an area to be used for recognizing the object based on the detected portion of interest, and generate an image of the set area as the recognition image.

[0009] The second generation unit may generate, as the recognition image, an image smaller in size than the captured image.

[0010] The recognition unit may obtain a recognition result for the object from a machine learning model trained using a photographed image for learning about the object and correct answer information for the recognition result of the object associated with the photographed image for learning, by inputting the recognition image and the skeletal information into the machine learning model.

[0011] The recognition unit may include a skeleton-side machine learning model that, in response to input of skeleton information of the object, outputs a first feature derived from the skeleton based on the skeleton information; an image-side machine learning model that, in response to input of the image for recognition, outputs a first feature derived from the image for recognition based on the image for recognition; and a recognition machine learning model that, in response to input of the first feature derived from the skeleton and the first feature derived from the image, outputs a recognition result of the object.

[0012] The machine learning model for recognition may include a first machine learning model that outputs a second feature derived from a skeleton in response to an input of a first feature derived from the skeleton, a second machine learning model that outputs a second feature derived from an image in response to an input of the first feature derived from an image, and a third machine learning model that outputs a recognition result of the object in response to an input of the second feature derived from the skeleton and the second feature derived from the image.

[0013] The machine learning model for recognition may be configured to train the first machine learning model trained by first pre-training, the second machine learning model trained by second pre-training, and the third machine learning model using first feature amounts derived from a training skeleton, first feature amounts derived from training images, and correct answer information of the recognition result of the object associated with the first feature amounts derived from the training skeleton and the first feature amounts derived from the training images. In this case, the first pre-training may be training the first machine learning model using the first feature amounts derived from the training skeleton and correct answer information of the recognition result of the object associated with the first feature amounts derived from the training skeleton. Furthermore, the second pre-training may be training the second machine learning model using the first feature amounts derived from training images and correct answer information of the recognition result of the object associated with the first feature amounts derived from the training images.

[0014] The first machine learning model and the second machine learning model may be configured so that a difference between a dimension of the second feature derived from the skeleton and a dimension of the second feature derived from the image is within a predetermined range.

[0015] At least one of the skeleton-side machine learning model, the image-side machine learning model, the first machine learning model, the second machine learning model, and the third machine learning model may be configured by a neural network.

[0016] Each of the skeleton-side machine learning model, the image-side machine learning model, the first machine learning model, the second machine learning model, and the third machine learning model may be configured as a neural network.

[0017] The target object may be a predetermined part of a human body, in which case the recognition unit may recognize a gesture made by the predetermined part of the human body.

[0018] The first generation unit may include a machine learning model that outputs skeletal information of the object in response to an input of a photographed image of the object.

[0019] The first generation unit may generate an object image, which is an image of an area in which the object is displayed, from the captured image, and generate the skeletal information based on the generated object image.

[0020] An information processing method according to one embodiment of the present technology is an information processing method executed by a computer system, and includes acquiring a captured image of an object. Skeleton information of the object is generated based on the captured image. A recognition image, which is an image of an area used in a recognition process of the object, is generated from the captured image based on the skeletal information. The object is recognized based on the recognition image and the skeletal information.

[0021] A program according to one form of the present technology causes a computer system to acquire a photographed image of an object, generate skeletal information of the object based on the photographed image, generate an image for recognition from the photographed image based on the skeletal information, which is an image of an area used for recognition processing of the object, and recognize the object based on the image for recognition and the skeletal information.

[0022] FIG. 1 is a schematic diagram showing an example of the configuration of an object recognition system according to a first embodiment. FIG. 2 is a flowchart showing an example of object recognition processing by an information processing device. FIG. 3 is a schematic diagram showing an example of a captured image and skeletal information of a user's hand. FIG. 4 is a schematic diagram showing an example of an image for recognition. FIG. 5 is a schematic diagram showing an example of recognition processing by a recognition unit. FIG. 6 is a schematic diagram showing another example of recognition processing by a recognition unit. FIG. 7 is a schematic diagram showing an example of the configuration of an object recognition system according to a second embodiment. FIG. 8 is a flowchart showing an example of object recognition processing in this embodiment. FIG. 9 is a schematic diagram showing an example of the configuration of a machine learning model for recognition built in a gesture recognition unit. FIG. 10 is a flowchart showing an example of a training method for a machine learning model for recognition configured with three neural networks. FIG. 11 is a schematic diagram showing first pre-training and second pre-training for neural networks. FIG. 12 is a schematic diagram showing another example of hand gestures. FIG. 13 is a schematic diagram showing an example of a hand gesture UI according to the present technology. FIG. 14 is a block diagram showing an example of the hardware configuration of a computer that can be used as an information processing device.

[0023] Hereinafter, embodiments of the present technology will be described with reference to the drawings.

[0024] 1 is a schematic diagram showing a configuration example of an object recognition system according to a first embodiment of the present technology. The object recognition system 1 according to this embodiment includes a television 2, an imaging device (camera) 3, and an information processing device 4.

[0025] In this embodiment, the object recognition system 1 can accurately recognize hand gestures of a user 5 watching a television 2. Then, various controls on the television 2 can be executed in response to the hand gestures of the user 5.

[0026] This allows the user 5 to remotely control the television 2 (non-contact operation) using a hand gesture UI.

[0027] For example, the user 5 makes a hand gesture by extending the index finger of his or her hand 6 (either the right or left hand) and pointing it toward the television 2. This makes it possible to display a cursor (pointer) 7 on the screen of the television 2 at the position where the index finger is pointing. The user 5 can also move the display position of the cursor 7 by moving the position (direction) where the index finger is pointing. Note that while the cursor 7 is illustrated in FIG. 1 as being star-shaped, the shape of the cursor 7 can of course be designed as desired.

[0028] In this embodiment, the user 5 makes a hand gesture in which the thumb and middle finger of the same hand 6 touch with each other while extending the index finger. This allows a selection operation to be input for the display position of the cursor 7. This input operation corresponds to, for example, a touch operation on a touch panel screen of a smartphone or the like. It also corresponds to pressing the left button of a mouse when operating a PC or the like.

[0029] For example, a hand gesture in which the thumb and middle finger are momentarily brought into contact (from a state in which the thumb and middle finger are separated, to a state in which the thumb and middle finger are brought together and then quickly separated again) corresponds to a tap on a touch panel or a left-click on a mouse.

[0030] The hand gesture of touching and maintaining contact between the thumb and middle finger is equivalent to maintaining contact on a touch panel or pressing and holding the left mouse button. By moving the position of the index finger while maintaining contact between the thumb and middle finger, a drag operation can be performed.

[0031] Hereinafter, a hand gesture in which the thumb and middle finger of the same hand 6 are momentarily brought into contact with each other while the index finger is extended will be referred to as a thumb / middle finger click operation.

[0032] For example, the user 5 moves the position of the index finger to move the cursor 7 to one of the genre selection buttons 8 labeled "Movie," "Music," or "News" at the top right of the screen of the television 2 shown in Fig. 1. For example, the user moves the cursor 7 to the genre selection button 8 of the genre they want to watch, and clicks with their thumb or middle finger. This makes it possible to display content of the genre to which the cursor 7 is moved on the screen of the television 2.

[0033] The user 5 also moves the cursor 7 by pointing the index finger to the channel selection button 9 displayed below the genre selection button 8. Then, the user performs a thumb / middle finger click operation. This allows the user to switch channels within each genre.

[0034] In the example shown in Fig. 1, content on channel 103 in the [Movie] genre is displayed on the screen of the television 2. In this state, the cursor 7 is moved to the channel selection button 9 and a thumb / middle finger click operation is performed. This makes it possible to change channels within the [Movie] genre. Of course, similar hand gestures can also be used to change channels in other genres.

[0035] The user 5 also moves the cursor 7 by pointing the index finger to the volume change button 10 displayed at the bottom of the screen, and then performs a thumb / middle finger click operation, thereby adjusting the volume.

[0036] Of course, the types of hand gestures that can be recognized by the object recognition system 1 according to the present technology and the control contents of the television 2 associated with the hand gestures are not limited. Furthermore, the present technology is not limited to only recognizing hand gestures. This point will be described later.

[0037] The television 2, the imaging device 3, and the information processing device 4 are connected to each other via wire or wirelessly so that they can communicate with each other. The connection form between the devices is not limited, and it is possible to use, for example, wireless LAN communication such as Wi-Fi or short-range wireless communication such as Bluetooth (registered trademark).

[0038] 1 , the imaging device 3 is disposed above the television 2, facing the front side (toward the user 5). In this embodiment, the hand 6 of the user 5 is the target object, and is captured by the imaging device 3. The imaging device 3 outputs a captured image of the hand 6 (target object) of the user 5.

[0039] The imaging device 3 may be, for example, a digital camera equipped with an image sensor such as a CMOS (Complementary Metal-Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor. Without being limited to this, any imaging device capable of capturing an image of the hand 6 of the user 5 may be used, such as a monocular camera, a ToF (Time of Flight) camera, a stereo camera, an infrared camera, or a polarization camera. In the present disclosure, the term "image" includes both still images and moving images (video).

[0040] The information processing device 4 has hardware necessary for a computer, such as a processor such as a CPU, a GPU, or a DSP, a memory such as a ROM or a RAM, and a storage device such as an HDD (see FIG. 14 ). The processor loads a program according to the present technology stored in a storage unit or the memory into the RAM and executes the program, thereby executing the information processing method (object recognition method) according to the present technology.

[0041] For example, the information processing device 4 can be realized by any computer such as a PC (Personal Computer). Of course, hardware such as an FPGA or an ASIC may also be used.

[0042] 1 , in the present embodiment, a processor of an information processing device 4 executes a program according to the present technology to configure functional blocks of an acquisition unit 12, a first generation unit 13, a second generation unit 14, and a recognition unit 15. Of course, dedicated hardware such as an IC (integrated circuit) may be used to realize the functional blocks.

[0043] The program is installed in the information processing device 4 via, for example, various recording media. Alternatively, the program may be installed via the Internet or the like. The type of recording medium on which the program is recorded is not limited, and any computer-readable recording medium may be used. For example, any computer-readable non-transitory storage medium may be used.

[0044] 2 is a flowchart showing an example of object recognition processing executed by the information processing device 4. Here, an example is given in which the presence or absence of contact between the thumb and middle finger is output as the recognition result. The object recognition processing shown in FIG. 2 is executed for each frame image of a moving image captured by the imaging device 3 at a predetermined frame rate.

[0045] The acquisition unit 12 of the information processing device 4 acquires a captured image of an object (step 101). In this embodiment, the captured image is a captured image of the hand 6 of the user 5 making a hand gesture.

[0046] The first generating unit 13 generates skeletal information of the object based on the captured image of the object (step 102). In this embodiment, the skeletal information of the hand 6 of the user 5 is generated based on the captured image of the hand 6 of the user 5.

[0047] 3A and 3B are schematic diagrams showing an example of a captured image and skeletal information of a hand 6 of a user 5. Skeletal information 18 shown in Fig. 3B is generated from a captured image 17 of the hand 6 of the user 5 shown in Fig. 3A.

[0048] The skeletal information 18 is generated by a skeletal estimation process that estimates a skeleton based on the captured image 17. Note that the estimated skeleton does not need to match the actual bone structure, but only needs to include line segments (or their endpoints) that indicate the position and bending of each finger. The skeletal estimation process is also called bone estimation or skeleton estimation. There are no specific limitations on the algorithm used in the skeletal estimation process to generate the skeletal information 18, and any algorithm may be used.

[0049] The second generation unit 14 generates a recognition image, which is an image of an area used in the recognition process of the object, from the photographed image 17 based on the skeletal information 18 (step 103). In this embodiment, the recognition image used in the recognition process of the hand 6 of the user 5 is generated from the photographed image 17 of the hand 6 of the user 5 based on the skeletal information 18 of the hand 6 of the user 5.

[0050] 4 is a schematic diagram showing an example of a recognition image 20. The recognition image 20 is an image obtained by cutting out an area necessary for recognizing an object from the photographed image 17. The second generation unit 14 generates an image smaller in size than the photographed image 17 as the recognition image 20.

[0051] For example, the second generation unit 14 detects a noteworthy part in the object based on the skeletal information 18, sets an area to be used for recognizing the object based on the detected noteworthy part, and generates an image of the set area as an image for recognition 20.

[0052] In this embodiment, as a recognition result of object recognition, contact between the thumb 22 and the middle finger 23 is output for the captured image 17 (each frame image) of the hand 6 of the user 5. As shown in Fig. 3B , if the palm is not facing the imaging device 3, the skeleton of the middle finger 23 may be estimated incorrectly.

[0053] On the other hand, regardless of the orientation of the palm, the skeleton estimation of the thumb 22 is performed relatively stably. Therefore, in this embodiment, the image for recognition 20 is generated based on the skeleton estimation of the thumb 22. For example, as shown in FIG. 4 , the tip of the thumb 22 is detected as a portion of interest 24 based on the skeleton information of the thumb 22. Then, the image for recognition 20 is generated based on the portion of interest 24. For example, an image of an area of ​​a predetermined shape and a predetermined size centered on the portion of interest 24 is generated as the image for recognition 20. Of course, the method is not limited to this.

[0054] 4, the recognition image 20 includes information necessary for gesture recognition (determining whether or not the thumb 22 and middle finger 23 are in contact with each other), while information unnecessary for gesture recognition (fingers other than the thumb 22 and middle finger 23, the background, etc.) is deleted. This makes it possible to improve the recognition accuracy of object recognition compared to when the entire photographed image 17 of the hand 6 as shown in FIG. 3A is used as is.

[0055] By using the skeleton information 18, it is possible to generate with high accuracy recognition images 20 that correspond to the shapes, movements, etc. of various hand gestures to be recognized.

[0056] The recognition unit 15 recognizes the object based on the recognition image 20 and the skeletal information 18. In this embodiment, whether or not the thumb 22 and middle finger 23 are in contact with each other in each frame image is output as the recognition result.

[0057] There is no limitation on the method for recognizing an object by the recognition unit 15, and any technique (algorithm) may be used. For example, it is possible to perform recognition processing on an object by an algorithm using a machine learning model constructed by a neural network or the like (machine learning algorithm).

[0058] For example, any machine learning algorithm may be used, such as a deep neural network (DNN), a recurrent neural network (RNN), or a convolutional neural network (CNN). For example, by using an AI (artificial intelligence) that performs deep learning, it becomes possible to recognize an object with high accuracy.

[0059] 5 is a schematic diagram showing an example of recognition processing by the recognition unit 15. In the example shown in Fig. 5, a machine learning model 26 is constructed in the recognition unit 15, which outputs a recognition result of an object in response to an input of a recognition image 20 and skeletal information 18. The recognition unit 15 inputs the recognition image 20 and skeletal information 18 to the machine learning model 26, and the recognition result of the object is acquired and output from the machine learning model 26. In this embodiment, whether or not the thumb 22 and middle finger 23 are in contact with each other is acquired and output from the machine learning model 26 as a recognition result.

[0060] The machine learning model 26 can be trained using, for example, a captured image for learning of an object and correct answer information of the recognition result of the object associated with the captured image for learning. In this embodiment, for example, the machine learning model can be trained using data in which the presence or absence of contact between the thumb 22 and the middle finger 23 is associated as correct answer information with a captured image (frame image) of the hand 6 performing a thumb / middle finger click operation.

[0061] The image (frame image) of the hand 6 performing the thumb / middle finger click operation is an embodiment of a captured image for learning about the object. Also, the presence or absence of contact between the thumb 22 and middle finger 23 is an embodiment of correct answer information for the object recognition result.

[0062] For example, for an image (frame image) of a hand 6 performing a thumb / middle finger click operation, the first generator 13 and the second generator 14 generate skeletal information 18 and a recognition image 20. These two pieces of data and the correct answer information, i.e., whether or not the thumb 22 and middle finger 23 are in contact, are used as training data, and learning is performed based on a machine learning algorithm.

[0063] Through learning, parameters (coefficients) for calculating the correct answer (whether the thumb and middle finger are in contact or not) are updated and generated as learned parameters. A program incorporating the generated learned parameters is generated as the machine learning model 26. For example, the machine learning model 26 can be configured using a neural network.

[0064] The learning method used for the machine learning model 26 is, for example, backpropagation. Backpropagation is a commonly used learning technique for neural network learning. A neural network is a model that originally imitates the neural circuits of the human brain, and has a layer structure consisting of three types of layers: an input layer, an intermediate layer (hidden layer), and an output layer.

[0065] Neural networks with many hidden layers are called deep neural networks, and the deep learning technology used to train them is known as a model that can learn complex patterns hidden in large amounts of data. Backpropagation is one such learning method, and is often used, for example, in training CNNs, which are used to recognize images and videos.

[0066] Furthermore, as a hardware structure for realizing such machine learning, a neurochip / neuromorphic chip incorporating the concept of a neural network can be used.

[0067] The algorithm for training the machine learning model 26 is not limited, and any machine learning algorithm may be used. For example, machine learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, inverse reinforcement learning, active learning, and transfer learning. Furthermore, machine learning models such as a hidden Markov model (HMM) and a support vector machine (SVM) may also be used.

[0068] A functional block (learning unit) that trains the machine learning model 26 may be included in the information processing device 4, and the learning of the machine learning model 26 may be executed by the information processing device 4. On the other hand, the learning unit may be configured outside the information processing device 4. That is, learning may be executed by the learning unit outside the information processing device 4, and the trained machine learning model 26 may be incorporated into the recognition unit 15. Other than that, the method and configuration for training the machine learning model 26 are not limited.

[0069] The application of a machine learning algorithm may be performed for any process within the present disclosure, i.e., any process described within the present disclosure may be subjected to a process using machine learning.

[0070] For example, a machine learning algorithm may be used to generate the skeletal information 18 executed by the first generating unit 13. For example, a machine learning model that outputs skeletal information 18 of an object in response to input of a photographed image 17 of the object may be constructed in the first generating unit 13. The first generating unit 13 inputs the photographed image 17 of the object to the machine learning model, and outputs skeletal information of the object from the machine learning model.

[0071] 6 is a schematic diagram showing another example of the recognition process by the recognition unit 15. As shown in Fig. 6, a recognition result of an object may be output by processing using a rule-based algorithm in response to input of a recognition image 20 and skeletal information 18. In this embodiment, the recognition result output is whether or not the thumb 22 and middle finger 23 are in contact with each other.

[0072] There are no particular limitations on the specific algorithms and the like that are executed as rule-based processing. Any rule-based algorithm, such as matching technology, image recognition technology, or analysis technology, may be used.

[0073] In both the recognition process using the machine learning algorithm shown in Fig. 5 and the recognition process using the rule-based algorithm shown in Fig. 6, a small-sized recognition image 20 is used. This makes it possible to reduce the processing time required for the recognition process compared to using the entire photographed image 17 of the hand 6 as is, as shown in Fig. 3A. Furthermore, the recognition image 20 includes information necessary for the recognition process, while information unnecessary for the recognition process is deleted. This results in high recognition accuracy.

[0074] In this embodiment, it is possible to determine whether or not a thumb / middle finger click operation has been performed based on whether or not the thumb 22 and middle finger 23 are in contact with each frame image. For example, if a change from no contact → contact → no contact is recognized as a recognition result for a predetermined number of frame images acquired within a predetermined time, it is possible to determine that a thumb / middle finger click operation has been performed by the user 5. Note that the presence or absence of the thumb / middle finger click operation is also included in the object recognition result related to the present technology.

[0075] In this embodiment, it is also possible to determine whether the thumb 22 and middle finger 23 are maintaining contact with each frame image based on whether the thumb 22 and middle finger 23 are in contact with each frame image. If it is recognized that the position where the index finger is pointed has been moved while the thumb 22 and middle finger 23 are maintaining contact with each frame image, it is possible to determine that a drag operation has been performed. It should be noted that the determination of whether the position where the index finger is pointed has been moved can also be performed with high accuracy in a short processing time based on, for example, a recognition image of the area around the index finger and skeletal information. It should be noted that the whether or not the state where the thumb 22 and middle finger 23 are maintaining contact and the whether or not a drag operation has been performed are also included in the object recognition results of the present technology.

[0076] Based on the recognition result output by the recognition unit 15, a control unit (not shown) provided in the television 2 controls the television 2. For example, when a thumb / middle finger click operation is recognized, a control corresponding to the thumb / middle finger click operation is executed.

[0077] For example, the content display may be switched in response to the selection of the genre selection button 8 or the channel selection button 9 shown in Fig. 1, or the volume may be controlled in response to the selection of the volume change button 10. Of course, control in response to a drag operation may also be performed.

[0078] The control unit provided in the television 2 has hardware necessary for a computer, such as a processor, memory, and storage device, and comprehensively controls each operation of the television 2. The recognition result from the recognition unit 15 is input to the control unit, and control is performed in accordance with the recognition result. Of course, a functional block for controlling the television 2 may be built in the information processing device 4.

[0079] 1 , a television 2, an imaging device 3, and an information processing device 4 are configured separately. For example, an RGB monocular camera (web camera) or the like is placed above the television 2 and communicably connected to the information processing device 4, thereby enabling the construction of an object recognition system 1 according to the present technology.

[0080] Even if an imaging device 3 with a simple configuration such as an RGB monocular camera is used, rather than an imaging device 3 that can acquire distance information such as a ToF camera or stereo camera, it is possible to perform recognition processing on the object with extremely high accuracy.

[0081] 1 , the television 2, the imaging device 3, and the information processing device 4 may be integrally configured as appropriate. For example, the object recognition system 1 according to the present technology can be constructed using a television with a built-in camera, in which the television 2 and the imaging device 3 are integrally configured.

[0082] Furthermore, the functions of the information processing device 4 (the acquisition unit 12, the first generation unit 13, the second generation unit 14, and the recognition unit 15) may be mounted on the control unit of the television 2. That is, a computer system in which the television 2, the imaging device 3, and the information processing device 4 are integrated can also be configured as an embodiment of the information processing device according to this embodiment.

[0083] As described above, in the object recognition system 1 and information processing device 4 according to this embodiment, skeletal information 18 of the hand 6 (object) is generated based on the captured image 17 of the hand 6 (object), and a recognition image 20 used in recognition processing of the hand 6 (object) is generated from the captured image 17 based on the skeletal information 18. Then, the hand 6 (object) is recognized based on the recognition image 20 and the skeletal information 18. This makes it possible to recognize the hand 6 (object) with high accuracy.

[0084] Conventionally, hand gesture UIs using RGB monocular cameras have had the problem of lacking robustness against changes in hand orientation and individual differences in hand shape, making it difficult to realize remote control using a hand gesture UI such as that shown in Figure 1.

[0085] It is also conceivable to recognize hand gestures and the like using only skeletal information as an object recognition technique. However, as illustrated in FIG. 3B , it may be impossible to correctly estimate the skeleton of the hand 6 when the palm is not facing the imaging device 3. Even if the skeleton itself can be correctly estimated, it is impossible to determine the thickness of the fingers from the skeletal information, and individual differences in the thickness of the fingers may prevent correct recognition of gestures such as finger contact.

[0086] 3B is used to determine whether or not the thumb 22 and middle finger 23 are in contact with each other. In this case, even if a user 5 with thick fingers has the thumb 22 and middle finger 23 in contact with each other, the skeleton of the thumb 22 and the skeleton of the middle finger 23 are far apart, and therefore it may be determined that the thumb 22 and the middle finger 23 are not in contact with each other.

[0087] Furthermore, even though a user 5 with thin fingers performs a thumb / middle finger click operation and separates the thumb 22 and middle finger 23, the skeleton of the thumb 22 and the skeleton of the middle finger 23 are close to each other, and the user may perceive that contact between the thumb 22 and the middle finger 23 is maintained.

[0088] In this object recognition system 1, gesture recognition is performed using not only skeletal information 18 shown in Fig. 3B but also a recognition image 20 generated from the original captured image 17. This makes it possible to realize object recognition that is robust against changes in the orientation of the hand 6 and individual differences in the shape of the hand 6, and to recognize the hand 6 with high accuracy. As a result, it is possible to realize remote control using a hand gesture UI as shown in Fig. 1.

[0089] Furthermore, by using the recognition image 20 instead of the entire original captured image 17, it is possible to improve the recognition accuracy and reduce the processing time. Therefore, it is possible to realize high-speed gesture recognition, and it is possible to fully prevent the user 5 from feeling stressed due to "sluggish movements."

[0090] Furthermore, even in an electronic device with limited computational resources, such as a television 2, by incorporating each function of the information processing device 4 (the acquisition unit 12, the first generation unit 13, the second generation unit 14, and the recognition unit 15), it becomes possible to perform high-speed object recognition.

[0091] Furthermore, by employing a recognition process using a machine learning algorithm such as that shown in FIG. 5, it is possible to perform gesture recognition for a wide variety of gestures at high speed with high accuracy.

[0092] Second Embodiment An object recognition system according to a second embodiment of the present technology will be described. In the following description, descriptions of parts similar to those of the object recognition system 1 described in the above embodiment will be omitted or simplified.

[0093] 7 is a schematic diagram showing an example configuration of an object recognition system 30 according to this embodiment. In this embodiment, an RGB monocular camera 31 is used as an imaging device. Furthermore, an electronic device 32 controlled in response to a user's hand gesture is configured as an embodiment of an information processing device according to the present technology, and executes an object recognition method according to the present technology.

[0094] That is, the object recognition system 30 according to this embodiment has a configuration similar to that of the object recognition system 1 according to the first embodiment shown in FIG. 1 in which the television 2 and the information processing device 4 are integrally configured.

[0095] As shown in FIG. 2 , the electronic device 32 includes a hand detection unit 33, a skeleton estimation unit 34, an image cropping unit 35, a skeleton feature extraction unit 36, an image feature extraction unit 37, a gesture recognition unit 38, and a device control unit 39.

[0096] Each of these blocks is realized, for example, by a processor of a control unit provided in the electronic device 32 executing a program according to the present technology. Of course, hardware such as an IC (integrated circuit) may be used to realize each functional block.

[0097] In this embodiment, the hand detection unit 33 shown in Fig. 2 functions as the acquisition unit 12 shown in Fig. 1. The hand detection unit 33 and the skeleton estimation unit 34 shown in Fig. 2 function as the first generation unit 13 shown in Fig. 1. Therefore, the hand detection unit 33 functions as both the acquisition unit 12 and the first generation unit 13 shown in Fig. 1.

[0098] 1 may be configured separately. A photographed image of a hand may be output from the separately configured acquisition unit 12 to the hand detection unit 33. In this case, the hand detection unit 33 functions only as the first generation unit 13.

[0099] In this embodiment, the image trimming unit 35 functions as the second generation unit 14 shown in Fig. 1. The skeleton feature extraction unit 36, the image feature extraction unit 37, and the gesture recognition unit 38 function as the recognition unit 15 shown in Fig. 1.

[0100] The device control unit 39 receives the recognition result from the gesture recognition unit 38 and controls the electronic device 32 in accordance with the recognition result.

[0101] Fig. 8 is a flowchart showing an example of object recognition processing in this embodiment. The flowchart shown in Fig. 8 shows the flow from when a user makes a hand gesture to when the electronic device 32 is controlled. In this embodiment, too, an example is given of a case where the presence or absence of contact between the thumb and middle finger is output as the recognition result.

[0102] The RGB monocular camera 31 captures an image of the user at a predetermined frame rate and outputs the captured image (frame image). The hand detection unit 33 acquires the image of the user captured by the RGB monocular camera 31 (step 201).

[0103] The hand detection unit 33 detects the user's hand, which is the target object, from the acquired photographed image of the user. Then, a hand image, which is an image of the area in which the user's hand is displayed, is generated from the photographed image of the user. The hand image is an image in which only the area of ​​the user's hand, which is the target object, is cut out, and for example, an image in which the hand 6 shown in FIG. 3 is cut out is generated (step 202).

[0104] The algorithm for generating a hand image from a captured image of a user is not limited. For example, a machine learning algorithm using a machine learning model such as FPN can be used. By using a machine learning algorithm, it is possible to generate a hand image of a user with high accuracy. Of course, a rule-based algorithm may also be used.

[0105] When a user's hand is detected from a captured image of the user, the captured image of the user is an embodiment of a captured image of an object according to the present technology. Also, the hand image generated by the hand detection unit 33 is an embodiment of an image of an area where an object is displayed according to the present technology.

[0106] Skeleton estimation unit 34 generates skeletal information based on the hand image generated by hand detection unit 33. That is, skeleton estimation unit 34 estimates the hand skeleton from the hand image and outputs it as skeletal information (step 203).

[0107] The algorithm for generating hand skeletal information from a hand image is not limited. For example, a machine learning algorithm using a machine learning model such as an Efficient Net can be adopted. By using a machine learning model, it is possible to generate hand skeletal information with high accuracy. Of course, a rule-based algorithm may also be used.

[0108] In this manner, in this embodiment, an object image, which is an image of an area in which an object is displayed, is generated from the captured image, and skeletal information is generated based on the object image, thereby enabling skeletal information to be generated with high accuracy.

[0109] Based on the skeletal information, the image of a portion of the hand is cut out (trimmed) from the hand image by the image trimming unit 35, and a recognition image to be used in the recognition process is generated. For example, as shown in Figure 4, a recognition image 20 smaller in size than the original captured image is generated using a target portion 24 at the tip of the thumb 22 as a reference (step 204).

[0110] The skeleton-side feature extraction unit 36 ​​generates and outputs skeleton-derived features (hereinafter referred to as skeleton-derived first features) based on the skeletal information of the hand. The image-side feature extraction unit 37 generates and outputs image-derived features (hereinafter referred to as image-derived first features) based on the recognition image generated by the image cropping unit 35 (step 205).

[0111] In this embodiment, a machine learning model (hereinafter referred to as a skeleton-side machine learning model) that outputs a first feature derived from the skeleton based on the skeletal information of an object in response to input of the skeletal information of the object is constructed in the skeleton-side feature extraction unit 36. The skeleton information is input to the skeleton-side machine learning model by the skeleton-side feature extraction unit 36, and the first feature derived from the skeleton is acquired and output from the skeleton-side machine learning model.

[0112] As the skeleton-side machine learning model, for example, a machine learning model such as an MLP (multilayer perceptron) can be adopted. Of course, this is not limited to this, and other machine learning models may also be adopted. In this embodiment, the skeleton-side machine learning model is configured by a neural network.

[0113] Furthermore, a machine learning model (hereinafter referred to as the image-side machine learning model) that outputs an image-derived first feature based on an input recognition image in response to an input recognition image is constructed within the image-side feature extraction unit 37. The image-side feature extraction unit 37 inputs the recognition image to the image-side machine learning model, and the image-derived first feature is acquired and output from the image-side machine learning model.

[0114] As the image-side machine learning model, for example, a machine learning model such as an Efficient Net can be adopted. Of course, this is not limiting, and other machine learning models may also be adopted. In this embodiment, the image-side machine learning model is configured by a neural network.

[0115] Generally, the original image contains more information than the skeletal information generated from the image. That is, in this embodiment, the hand image contains more information than the skeletal information generated from the hand image. Therefore, it is conceivable that the output of the first feature values ​​derived from the image side using the image-side machine learning model will require a longer processing time than the output of the first feature values ​​derived from the skeleton side using the skeleton-side machine learning model.

[0116] In this embodiment, a recognition image used in the recognition process of an object is generated with high accuracy by the image trimming unit 35. That is, a recognition image with a small size that is useful for the recognition process is generated with high accuracy.

[0117] Therefore, for example, it is possible to reduce the size of the neural network that constitutes the image-side machine learning model. As a result, it is possible to shorten the processing time for generating the image-derived first feature. Furthermore, the recognition image has data (images of other fingers, etc.) that is not related to the recognition process (whether the thumb and middle finger are in contact or not) reduced. As a result, it is possible to generate the image-derived first feature with high accuracy, and it is possible to improve the recognition accuracy of object recognition.

[0118] Furthermore, by constructing a neural network specialized for small-sized images (images for recognition), it is possible to further reduce processing time and further improve the accuracy of generating the first feature derived from the image, thereby further improving the recognition accuracy of object recognition.

[0119] The gesture recognition unit 38 executes a recognition process based on the first feature amount derived from the skeleton and the first feature amount derived from the image, and outputs a recognition result of the object. In this manner, in this embodiment, the first feature amount derived from the skeleton and the first feature amount derived from the image are used in combination to perform gesture recognition (step 206).

[0120] In this embodiment, a machine learning model for recognition is constructed in the gesture recognition unit 38, which outputs a recognition result of an object in response to input of a first feature amount derived from a skeleton and a first feature amount derived from an image. The gesture recognition unit 38 inputs the first feature amount derived from a skeleton and the first feature amount derived from an image to the machine learning model for recognition, and acquires and outputs a recognition result of the object from the machine learning model for recognition. In this embodiment, the machine learning model for recognition is constructed using a neural network.

[0121] 9 is a schematic diagram showing an example of the configuration of a machine learning model for recognition built in the gesture recognition unit 38. As shown in FIG. 9, in this embodiment, the machine learning model for recognition is made up of three neural networks A to C.

[0122] The neural network A outputs a second feature quantity derived from the skeleton in response to an input of a first feature quantity derived from the skeleton. The neural network A constitutes an embodiment of a first machine learning model according to the present technology.

[0123] Neural network B outputs a second feature quantity derived from an image in response to an input of a first feature quantity derived from an image. Neural network B constitutes an embodiment of a second machine learning model according to the present technology.

[0124] The neural network C outputs a recognition result of an object in response to input of the skeleton-derived second feature amount and the image-derived second feature amount. The neural network C constitutes an embodiment of a third machine learning model according to the present technology.

[0125] 10 is a flowchart showing an example of a training method for a machine learning model for recognition that is configured by three neural networks A to C. As shown in Fig. 10, a first pre-training is performed on neural network A (step 301).

[0126] 11A is a schematic diagram showing first pre-learning for neural network A. The first pre-learning is training for neural network A using first feature amounts derived from a training skeleton and correct answer information of the recognition result of an object associated with the first feature amounts derived from the training skeleton.

[0127] In this embodiment, a captured image (frame image) of a hand performing a thumb / middle finger click operation is prepared as a captured image for learning about an object. Furthermore, the correct answer information of the object recognition result associated with the captured image for learning is associated with whether or not the thumb and middle finger are in contact with each other. That is, a pair of a captured image (frame image) of a hand performing a thumb / middle finger click operation and whether or not the thumb and middle finger are in contact with each other is prepared.

[0128] The captured learning image (frame image) is then input to hand detection unit 33 shown in Fig. 7. Hand detection unit 33 generates a hand image (hereinafter referred to as a learning hand image) from the captured learning image, and inputs the generated image to skeleton estimation unit 34. Skeleton estimation unit 34 generates skeletal information (hereinafter referred to as learning skeletal information) from the learning hand image, and inputs the skeletal information to skeleton-side feature extraction unit 36.

[0129] The skeleton-side feature extraction unit 36 ​​outputs a first feature derived from the skeleton information for training. This first feature derived from the skeleton can be used as the first feature derived from the skeleton for training shown in FIG. 11A .

[0130] 11A , a training neural network 41 is connected to the output side of neural network A. The training neural network 41 is a neural network designed to output a recognition result of an object in response to input of the second feature amount derived from the skeleton output from neural network A.

[0131] For example, a linear classifier can be used as the learning neural network 41. That is, the neural network A can be trained by supervised learning using a linear classifier.

[0132] Neural network A and learning neural network 41 are trained together using training data that includes the first feature amount derived from the training skeleton and pre-prepared correct answer information of the recognition result of the object (whether the thumb and middle finger are in contact with each other). Neural network A is trained by this first pre-training, and the trained parameters are incorporated into neural network A. Note that learning neural network 41 is not used when performing object recognition.

[0133] As shown in FIG. 10, a second pre-training is performed on neural network B (step 302).

[0134] 11B is a schematic diagram showing second pre-learning for neural network B. The second pre-learning is training for neural network B using first feature amounts derived from training images and correct answer information on the recognition results of the object associated with the first feature amounts derived from the training images.

[0135] A photographed image (frame image) for learning, which has been prepared in advance, is input to hand detection unit 33 shown in Fig. 7. A hand image for learning is generated from the photographed image for learning by hand detection unit 33, and input to skeleton estimation unit 34. Skeleton estimation unit 34 generates skeleton information for learning from the hand image for learning, and inputs the information to image cropping unit 35.

[0136] The image trimming unit 35 generates a recognition image (hereinafter referred to as a training recognition image) based on the training skeleton information, and inputs the generated image to the image-side feature extraction unit 37. The image-side feature extraction unit 37 outputs a first feature derived from the training recognition image. This first feature derived from the image can be used as the first feature derived from the training image shown in FIG. 11B.

[0137] 11B, a training neural network 42 is connected to the output side of neural network B. The training neural network 42 is a neural network designed to output a recognition result of an object in response to input of a second feature amount derived from an image output from neural network B. A linear classifier, for example, can be used as the training neural network 42.

[0138] Neural network B and the training neural network 42 are trained together using the first feature amount derived from the training image and prepared correct answer information of the recognition result of the object (whether the thumb and middle finger are in contact or not) as training data. Neural network B is trained by this second pre-training, and the trained parameters are incorporated into neural network B. Note that the training neural network 42 is not used when performing object recognition.

[0139] 10, training is performed on all three neural networks A to C (step 303). This overall training is performed on neural network A trained by the first pre-training, neural network B trained by the second pre-training, and neural network C. Therefore, transfer learning is performed on neural networks A and B.

[0140] In the overall learning in step 303, random values ​​are used as the initial values ​​of the parameters of neural network C. On the other hand, for neural networks A and B, the learned parameters learned in the first pre-learning and the second pre-learning, respectively, are used as the initial values.

[0141] In addition, the overall learning in step 303 is performed using the first feature amounts derived from the training skeleton and the first feature amounts derived from the training image, and correct answer information on the recognition result of the object associated with the first feature amounts derived from the training skeleton and the first feature amounts derived from the training image.

[0142] By inputting a previously prepared captured image (frame image) for learning into hand detection unit 33, a first feature amount derived from the skeleton for learning is obtained from skeleton-side feature amount extraction unit 36. Also, a first feature amount derived from the image for learning is obtained from image-side feature amount extraction unit 37. These two first feature amounts and correct answer information of the previously prepared object recognition result (whether the thumb and middle finger are in contact) are used as training data, and all of neural networks A to C are trained together.

[0143] As mentioned above, the original image generally contains more information than the skeleton information generated from the image. In this embodiment, by generating a small-sized image for recognition, it is possible to reduce the amount of information on the image side. However, there may be cases where the amount of information in the image for recognition is greater than that of the skeleton information.

[0144] Due to a difference in the amount of information between the skeleton information and the image for recognition, a difference may occur between the dimension of the skeleton-derived first feature amount output from the skeleton-side feature amount extraction unit 36 ​​and the dimension of the image-derived first feature amount output from the image-side feature amount extraction unit 37. Specifically, the dimension of the image-derived first feature amount may be larger than the dimension of the skeleton-derived first feature amount.

[0145] In this way, when a machine learning model for recognition that takes two features of different dimensions as input is trained all at once using a first feature derived from the training skeleton and a first feature derived from the training image, stable training may not be achieved, making it difficult to improve recognition accuracy.

[0146] For example, the learning speed may vary, such that the learning speed using first features derived from a training image with a large dimension may be faster than the learning speed using first features derived from a training skeleton with a small dimension. As a result, it may become difficult to improve the recognition accuracy of the machine learning model for recognition.

[0147] In this embodiment, as shown in Figures 10 and 11, first pre-learning using first feature amounts derived from training skeletons for neural network A and second pre-learning using first feature amounts derived from training images for neural network B are individually executed. Then, overall learning is performed on neural networks A to C. This realizes stable learning, making it possible to improve the recognition accuracy of the machine learning model for recognition composed of neural networks A to C. As a result, it is possible to improve the recognition accuracy of object recognition.

[0148] It is also possible to align the dimensions of the skeleton-derived second feature quantity with the dimensions of the image-derived second feature quantity by appropriately designing neural networks A and B. That is, each of the first machine learning model formed by neural network A and the second machine learning model formed by neural network B is configured so that the difference between the dimensions of the skeleton-derived second feature quantity and the dimensions of the image-derived second feature quantity falls within a predetermined range.

[0149] For example, the first machine learning model and the second machine learning model can be configured so that the difference in dimension is zero, that is, the dimension of the skeleton-derived second feature amount and the dimension of the image-derived second feature amount are equal to each other.

[0150] For example, neural network A outputs a second feature derived from the skeleton so that the dimension is equal to that of the input first feature derived from the skeleton. Neural network B outputs a second feature derived from the image so that the dimension is smaller than that of the input first feature derived from the image and so that the dimension is the same as that of the second feature derived from the skeleton. For example, such a design is possible. Of course, this is not limiting.

[0151] The two second features whose dimensions have been aligned by neural networks A and B are input to neural network C. This makes it possible to realize stable learning for neural networks A to C, and thus realize high recognition accuracy.

[0152] Returning to FIG. 8, the device control unit 39 controls the electronic device 32 based on the recognition result output from the gesture recognition unit 38 (step 207).

[0153] In this embodiment as well, it is possible to achieve the effects described in the first embodiment, and to realize high-speed, high-precision object recognition.

[0154] [Variations of Hand Gestures] Figure 12 is a schematic diagram showing another example of a hand gesture. As shown in Figure 12A, when performing skeleton estimation on a hand gesture in which the index finger 44 is crossed under (on the palm side of) the middle finger 23, if the index finger 44 is hidden by the middle finger 23, the skeleton estimation of the index finger 44 may result in an incorrect result. In this case, it is difficult to obtain a highly accurate recognition result using only skeleton information.

[0155] 12B , the object recognition system according to the present technology can generate a highly accurate recognition image 20 by cutting out an image based on the results of skeletal estimation of the middle finger 23. For example, the tip or joint of the middle finger 23 is set as a focus area, and the recognition image 20 is generated based on the focus area. This makes it possible to generate a recognition image 20 that is useful for determining whether the index finger 44 and the middle finger 23 are crossed. Using the recognition image 20, it is possible to realize high-speed, highly accurate hand gesture recognition processing.

[0156] This technology can also be applied to the recognition of various other hand gestures. That is, for various hand gestures, it is possible to generate a recognition image based on skeletal information and realize high-speed recognition processing based on the recognition image and skeletal information.

[0157] By applying this technology, it is possible to realize various methods for controlling electronic devices using a hand gesture UI. For example, for various electronic devices controlled by a remote controller, such as a television, an air conditioner, or a lighting device, by appropriately assigning each button on the remote controller to a hand gesture, it is possible to operate the electronic device without using a remote controller.

[0158] Alternatively, a user may create a hand shape as if they are virtually holding a remote controller (without actually holding the remote controller). Then, the user makes a gesture as if they are pressing each button on the remote controller with their thumb. The gesture is recognized with high accuracy, and the electronic device corresponding to the button the user intended to select is controlled. Such a hand gesture UI can also be realized.

[0159] As shown in Figure 1, for electronic devices with displays such as televisions, smartphones, tablet devices, PCs, smart glasses, and HMDs, the display coordinates of the cursor are determined using the hand's skeletal structure (e.g., the index finger), and pressing / not pressing the cursor is assigned to a hand gesture. This makes it possible to operate the cursor without using a mouse or touch panel.

[0160] For example, smartphones, notebook PCs, smart glasses, HMDs, etc. often have built-in cameras, and the cameras can be used as the imaging device 3 shown in FIG.

[0161] Furthermore, although gaming monitors are often not operated using a remote controller and do not support cursor-type operation, application of the present technology makes it possible to perform operations such as adjusting the screen brightness and volume using a hand gesture UI. In this way, the present technology can be applied even to electronic devices that are not operated using a remote controller or that do not support cursor-type operation.

[0162] 13 is a schematic diagram showing an example of a hand gesture UI according to the present technology. In this example, a paint application that enables users to freely draw characters and pictures is installed on a smartphone, tablet terminal, or the like.

[0163] The user makes a hand gesture of extending the index finger toward the canvas area 46 where the paint application is to draw characters or pictures, thereby making it possible to display a cursor 47 at the position where the index finger is pointing in the canvas area 46, as shown in Fig. 13A .

[0164] In this state, when the thumb and middle finger are brought into contact with each other, the drawing mode is turned on, and drawing becomes possible in a predetermined color. As shown in Fig. 13B, by moving the index finger while keeping the thumb and middle finger in contact, it becomes possible to draw letters or pictures. With this technology, since it is possible to recognize contact between the thumb and middle finger with high speed and accuracy, it is possible to realize such a hand gesture UI.

[0165] Other Embodiments The present technology is not limited to the above-described embodiments, and various other embodiments can be realized.

[0166] This technology can also be applied to recognition of things other than hand gestures. For example, it can recognize the shapes, movements, gestures, etc. of various parts of the human body other than hands with high accuracy.

[0167] For example, applying this technology enables highly accurate recognition when recognizing whether multiple objects (hands) have come into contact with each other, such as when multiple users shake hands or high-five. It is also possible to recognize with high accuracy whether a user's foot or hand has come into contact with a ball, such as when a user kicks a soccer ball or attacks a volleyball.

[0168] This technology can also be applied to determining whether a user performing Tai Chi, dancing, etc. is performing a correct martial art or dance. For example, it is possible to accurately determine whether a martial art or dance that involves raising the foot and touching it to the hand is being performed correctly.

[0169] For example, a recognition image is generated based on skeletal information for a portion where contact is desired to be determined, and contact can be recognized with high accuracy based on the skeletal information and the recognition image.

[0170] Note that contact is made by multiple objects, and the present technology can be applied to multiple objects. That is, skeletal information for each of the multiple objects is generated from a captured image of the multiple objects. Then, a recognition image used for recognition processing (presence or absence of contact) of the multiple objects is generated from the skeletal information. The multiple objects are recognized based on the recognition image and the skeletal information. Such processing is also included in the object recognition according to the present technology. Note that the thumb and middle finger can also be considered as multiple objects.

[0171] In this way, by using a specific part of a human body as the target of this technology, it is possible to recognize various gestures performed by the specific part of the human body. Of course, this technology is not limited to a human body, and it is also possible to apply this technology to a specific part of another living thing as the target. Furthermore, this technology can also be applied to recognize non-living objects, such as umbrellas and bicycles, as long as the object's skeleton can be estimated.

[0172] It is also possible that multiple recognition images are generated from a captured image of an object based on skeletal information. For example, a first recognition image with the tip of the thumb as the first target part and a second recognition image with the tip of the little finger as the second target part are generated from a captured image of a user's hand. Then, a hand gesture is recognized based on these first recognition image, second recognition image, and skeletal information. Such processing is also possible.

[0173] In addition, this technology can also be applied to various recognition processes for objects, such as the position of the object, the state of the object, the movement of the object, the activity level of the object, the distance of the object from a predetermined position, the posture of the object, and whether or not the object has performed an input operation.

[0174] In the holistic learning for neural networks A to C shown in step 303 of Figure 10, the learned parameters learned in the first and second pre-learning for neural networks A and B may be fixed, and the holistic learning may be performed. That is, in the holistic learning, only the parameters of neural network C may be learned. For example, if the quality of the training data prepared for the holistic learning is poor, fixing the parameters of neural networks A and B may result in higher recognition accuracy for the entire neural networks A to C. On the other hand, if the quality of the training data prepared for the holistic learning is high, the recognition accuracy for the entire neural networks A to C may be improved by further training the learned parameters for neural networks A and B as well.

[0175] Depending on the type of gesture to be recognized, the entire original captured image may be generated as the image for recognition. Of course, it is also possible to set the generated image for recognition to be smaller than the original captured image.

[0176] FIG. 14 is a block diagram showing an example of the hardware configuration of a computer 60 that can be used as the information processing device 4.

[0177] The computer 60 includes a CPU 61, a ROM 62, a RAM 63, an input / output interface 65, and a bus 64 interconnecting these components. The input / output interface 65 is connected to a display unit 66, an input unit 67, a storage unit 68, a communication unit 69, a drive unit 70, and other components. The display unit 66 is a display device using, for example, an LCD or EL display. The input unit 67 is a keyboard, a pointing device, a touch panel, or other operating device. If the input unit 67 includes a touch panel, the touch panel may be integrated with the display unit 66. The storage unit 68 is a non-volatile storage device such as a HDD, flash memory, or other solid-state memory. The drive unit 70 is a device capable of driving a removable storage medium 71 such as an optical storage medium or magnetic recording tape. The communication unit 69 is a modem, router, or other communication device connectable to a LAN, WAN, or the like for communicating with other devices. The communication unit 69 may communicate via either a wired or wireless connection. The communication unit 69 is often used separately from the computer 60. Information processing by the computer 60 having the hardware configuration described above is realized by software stored in the storage unit 68 or the ROM 62, etc., working in cooperation with the hardware resources of the computer 60. Specifically, the information processing method according to the present invention is realized by loading a program constituting the software stored in the ROM 62, etc., into the RAM 63 and executing it. The program is installed in the computer 60 via, for example, a recording medium 71. Alternatively, the program may be installed in the computer 60 via a global network, etc. Alternatively, any computer-readable, non-transitory storage medium may be used.

[0178] The information processing method (object recognition method) and program according to the present invention may be executed by cooperation among multiple computers connected to each other via a network or the like, thereby constructing an information processing system or information processing device according to the present invention. In other words, the information processing method and program according to the present invention can be executed not only in a computer system composed of a single computer, but also in a computer system in which multiple computers operate in conjunction with each other. In this disclosure, a "system" refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are contained in the same housing. Therefore, both multiple devices housed in separate housings and connected via a network and a single device housed in a single housing and multiple modules are systems.

[0179] The execution of the information processing method and program according to the present invention by a computer system includes both cases where, for example, the acquisition of photographed images, the generation of skeletal information, the generation of images for recognition, the recognition of objects, and the construction / learning of machine learning models (neural networks) are performed by a single computer, and cases where each process is performed by a different computer. Furthermore, the execution of each process by a specific computer includes having another computer execute part or all of the process and obtaining the results. In other words, the information processing method and program according to the present invention can also be applied to a cloud computing configuration in which a single function is shared and processed collaboratively by multiple devices via a network.

[0180] The configurations of the imaging device, electronic device, information processing device, and machine learning model (neural network), as well as the process flows for generating skeletal information, generating images for recognition, and building / learning the machine learning model (neural network), which have been described with reference to the drawings, are merely exemplary embodiments and may be modified as desired without departing from the spirit of the present invention. In other words, any other configurations, algorithms, etc. may be adopted for implementing the present invention.

[0181] In this disclosure, terms such as "substantially," "almost," and "roughly" may be used as appropriate to facilitate understanding of the description. However, there is no clear difference between using and not using terms such as "substantially," "almost," and "approximately." In other words, in this disclosure, concepts that define shape, size, positional relationship, state, etc., such as "center," "middle," "uniform," and "equal," are concepts that include "substantially center," "substantially central," "substantially uniform," and "substantially equal." For example, states that fall within a predetermined range (e.g., a range of ±10%) based on "completely centered," "completely central," "completely uniform," and "completely equal" are also included. Therefore, even if terms such as "substantially," "almost," and "approximately" are not used, concepts expressed by adding "substantially," "almost," and "approximately" may be included. Conversely, states expressed by adding terms such as "substantially," "almost," and "approximately" do not necessarily exclude perfect states.

[0182] In the present disclosure, expressions using "than", such as "greater than A" and "smaller than A", are expressions that comprehensively include both concepts that include the case where it is equivalent to A and concepts that do not include the case where it is equivalent to A. For example, "greater than A" is not limited to cases that do not include equivalent to A, but also includes "A or greater". Furthermore, "smaller than A" is not limited to "less than A" but also includes "A or less". When implementing the present technology, specific settings and the like can be appropriately adopted from the concepts included in "greater than A" and "smaller than A" so that the effects described above can be achieved.

[0183] It is also possible to combine at least two of the features of the present invention described above. That is, the various features described in each embodiment may be combined in any way without distinguishing between the embodiments. Furthermore, the various effects described above are merely examples and are not intended to be limiting, and other effects may also be achieved.

[0184] Note that the present technology can also be configured as follows. (1) An information processing device comprising: an acquisition unit that acquires a captured image of an object; a first generation unit that generates skeletal information of the object based on the captured image; a second generation unit that generates an image for recognition from the captured image based on the skeletal information, the image being an image of an area used for recognition processing of the object; and a recognition unit that recognizes the object based on the image for recognition and the skeletal information. (2) The information processing device described in (1), wherein the second generation unit detects a portion of interest in the object based on the skeletal information, sets an area used for recognition of the object based on the detected portion of interest, and generates an image of the set area as the image for recognition. (3) The information processing device described in (1) or (2), wherein the second generation unit generates an image smaller in size than the captured image as the image for recognition. (4) The information processing device according to any one of (1) to (3), wherein the recognition unit acquires a recognition result of the object from a machine learning model trained using a captured image for learning of the object and correct answer information of a recognition result of the object associated with the captured image for learning by inputting the image for recognition and the skeletal information. (5) The information processing device according to any one of (1) to (4), wherein the recognition unit includes: a skeleton-side machine learning model that outputs a skeleton-derived first feature based on the skeletal information of the object in response to input of the skeletal information of the object; an image-side machine learning model that outputs an image-derived first feature based on the image for recognition in response to input of the image for recognition; and a recognition machine learning model that outputs a recognition result of the object in response to input of the skeleton-derived first feature and the image-derived first feature.(6) The information processing device according to (5), wherein the machine learning model for recognition includes a first machine learning model that outputs a second feature derived from a skeleton in response to an input of a first feature derived from the skeleton, a second machine learning model that outputs a second feature derived from an image in response to an input of the first feature derived from the image, and a third machine learning model that outputs a recognition result of the object in response to an input of the second feature derived from the skeleton and the second feature derived from the image. (7) An information processing device according to (6), wherein the machine learning model for recognition is configured to train the first machine learning model trained by first pre-learning, the second machine learning model trained by second pre-learning, and the third machine learning model using first feature amounts derived from a training skeleton and first feature amounts derived from a training image, and correct answer information of the recognition result of the object associated with the first feature amounts derived from the training skeleton and the first feature amounts derived from the training image, wherein the first pre-learning is training the first machine learning model using the first feature amounts derived from the training skeleton and correct answer information of the recognition result of the object associated with the first feature amounts derived from the training skeleton, and the second pre-learning is training the second machine learning model using the first feature amounts derived from a training image and correct answer information of the recognition result of the object associated with the first feature amounts derived from the training image. (8) The information processing device according to (6) or (7), wherein the first machine learning model and the second machine learning model are configured so that a difference between a dimension of the skeleton-derived second feature amount and a dimension of the image-derived second feature amount falls within a predetermined range. (9) The information processing device according to any one of (6) to (8), wherein at least one of the skeleton-side machine learning model, the image-side machine learning model, the first machine learning model, the second machine learning model, and the third machine learning model is configured by a neural network.(10) The information processing device according to (9), wherein each of the skeleton-side machine learning model, the image-side machine learning model, the first machine learning model, the second machine learning model, and the third machine learning model is configured by a neural network. (11) The information processing device according to any one of (1) to (10), wherein the object is a predetermined part of a human body, and the recognition unit recognizes a gesture made by the predetermined part of the human body. (12) The information processing device according to any one of (1) to (11), wherein the first generation unit includes a machine learning model that outputs skeletal information of the object in response to input of a captured image of the object. (13) The information processing device according to any one of (1) to (12), wherein the first generation unit generates an object image that is an image of a region in which the object is displayed from the captured image, and generates the skeletal information based on the generated object image. (14) An information processing method executed by a computer system: acquiring a photographed image of an object, generating skeletal information of the object based on the photographed image, generating an image for recognition from the photographed image based on the skeletal information, which is an image of an area used in recognition processing of the object, and recognizing the object based on the image for recognition and the skeletal information. (15) A program that causes a computer system to execute the following: acquiring a photographed image of an object, generating skeletal information of the object based on the photographed image, generating an image for recognition from the photographed image based on the skeletal information, which is an image of an area used in recognition processing of the object, and recognizing the object based on the image for recognition and the skeletal information.

[0185] REFERENCE SIGNS LIST 1, 30... object recognition system 2... television 3... imaging device 4... information processing device 5... user 6... hand 17... photographed image 18... skeletal information 20... image for recognition 22... thumb 23... middle finger 24... target part 26... machine learning model 41, 42... neural network for learning 44... index finger 60... computer

Claims

1. An information processing device comprising: an acquisition unit that acquires a captured image of an object; a first generation unit that generates skeletal information of the object based on the captured image; a second generation unit that generates an image for recognition, which is an image of an area from the captured image to be used for recognition processing of the object, based on the skeletal information; and a recognition unit that recognizes the object based on the image for recognition and the skeletal information.

2. An information processing device as described in claim 1, wherein the second generation unit detects a portion of interest in the object based on the skeletal information, sets an area to be used for recognizing the object based on the detected portion of interest, and generates an image of the set area as the image for recognition.

3. An information processing device according to claim 1, wherein the second generation unit generates an image smaller in size than the captured image as the recognition image.

4. An information processing device as described in claim 1, wherein the recognition unit obtains the recognition result of the object from a machine learning model trained using a captured image for learning of the object and correct answer information for the recognition result of the object associated with the captured image for learning by inputting the image for recognition and the skeletal information into the machine learning model.

5. An information processing device as described in claim 1, wherein the recognition unit includes a skeleton-side machine learning model that outputs a first feature derived from the skeleton based on the skeletal information of the object in response to input of the skeletal information of the object, an image-side machine learning model that outputs a first feature derived from the image for recognition based on the image for recognition in response to input of the image for recognition, and a recognition machine learning model that outputs a recognition result of the object in response to input of the first feature derived from the skeleton and the first feature derived from the image.

6. An information processing device as described in claim 5, wherein the machine learning models for recognition include a first machine learning model that outputs a second feature derived from the skeleton in response to an input of a first feature derived from the skeleton, a second machine learning model that outputs a second feature derived from the image in response to an input of a first feature derived from the image, and a third machine learning model that outputs a recognition result of the object in response to an input of the second feature derived from the skeleton and the second feature derived from the image.

7. An information processing device according to claim 6, wherein the machine learning model for recognition is configured to train the first machine learning model trained by a first pre-learning, the second machine learning model trained by a second pre-learning, and the third machine learning model using a first feature amount derived from a training skeleton and a first feature amount derived from a training image, and correct answer information of the recognition result of the object associated with the first feature amount derived from the training skeleton and the first feature amount derived from the training image, wherein the first pre-learning is training for the first machine learning model using the first feature amount derived from the training skeleton and correct answer information of the recognition result of the object associated with the first feature amount derived from the training skeleton, and wherein the second pre-learning is training for the second machine learning model using the first feature amount derived from a training image and correct answer information of the recognition result of the object associated with the first feature amount derived from the training image.

8. An information processing device according to claim 6, wherein the first machine learning model and the second machine learning model are configured such that a difference between a dimension of the second feature derived from the skeleton and a dimension of the second feature derived from the image is within a predetermined range.

9. An information processing device according to claim 6, wherein at least one of the skeleton-side machine learning model, the image-side machine learning model, the first machine learning model, the second machine learning model, and the third machine learning model is configured by a neural network.

10. An information processing device according to claim 9, wherein each of the skeleton-side machine learning model, the image-side machine learning model, the first machine learning model, the second machine learning model, and the third machine learning model is configured as a neural network.

11. An information processing device according to claim 1, wherein the target object is a predetermined part of a human body, and the recognition unit recognizes a gesture made by the predetermined part of the human body.

12. An information processing device according to claim 1, wherein the first generation unit includes a machine learning model that outputs skeletal information of the object in response to an input of a photographed image of the object.

13. An information processing device according to claim 1, wherein the first generation unit generates an object image, which is an image of an area in which the object is displayed from the captured image, and generates the skeletal information based on the generated object image.

14. An information processing method implemented by a computer system, which comprises: acquiring an image of an object; generating skeletal information of the object based on the captured image; generating an image for recognition, which is an image of an area from the captured image used for recognition processing of the object, based on the skeletal information; and recognizing the object based on the image for recognition and the skeletal information.

15. A program that causes a computer system to execute the following operations: acquiring an image of an object; generating skeletal information of the object based on the captured image; generating an image for recognition, which is an image of an area from the captured image used in recognition processing of the object, based on the skeletal information; and recognizing the object based on the image for recognition and the skeletal information.

Images