Matching program, matching method, and information processing device

Abstract

This matching program causes a computer to execute processing of: acquiring an image; detecting skeleton information of a whole body from the image; detecting skeleton information of limb extremities from the image; calculating a first index on the basis of the positions of limb bases included in the skeleton information of the whole body and the positions of limb bases included in the skeleton information of limb extremities; calculating a second index on the basis of the orientations of limbs obtained from the skeleton information of the whole body and the orientations of the limb bases obtained from the skeleton information of limb extremities; and matching the skeletal information of limb extremities to each limb included in the skeleton information of the whole body on the basis of the first index and the second index.

Description

Matching program, matching method, and information processing device

[0001] The present invention relates to a matching program, a matching method, and an information processing device.

[0002] Activity recognition technology is known for recognizing various human actions from video. For example, in manufacturing and other workplaces, task recognition is sometimes implemented to recognize individual elemental tasks such as "tightening a screw," "soldering," and "attaching a cover" as part of business improvement efforts aimed at improving productivity and quality. In such cases, task recognition requires not only the movement of a person's limbs, but also the posture estimation of the fine movements of the extremities, such as the fingers.

[0003] Here, since the posture estimation of the whole body and each of the fingers is realized by individual AI (Artificial Intelligence) models, the whole body skeleton and finger skeleton detected individually from the camera image are matched, and the finger skeleton is linked to the person's whole body skeleton.

[0004] As an example of such matching technology, Prior Art 1 has been proposed, which aims to determine which region belongs to a hand when occlusion occurs. For example, Prior Art 1 calculates the correlation between the hand region and the worker region extracted from the camera image based on their respective positions and the image features (color information) of the hand region and the worker region, and matches highly correlated regions together.

[0005] US Patent Application Publication No. 2022 / 0405967 International Patent Application Publication No. 2023 / 275940

[0006] However, in the above-mentioned prior art 1, there is an aspect in which the accuracy of matching skeletal information of the whole body and the extremities is reduced.

[0007] For example, in the above-mentioned prior art 1, the correlation used for matching is calculated based on image features such as color information. Therefore, if the appearance features of people appearing in the video are similar, the matching accuracy decreases. In particular, in manufacturing sites, where people wear the same work clothes and gloves, the possibility of matching the finger skeletons and whole body skeletons of different people increases.

[0008] In one aspect, an object of the present invention is to provide a matching program, a matching method, and an information processing device that can improve the matching accuracy of skeletal information of the whole body and the extremities.

[0009] One embodiment of the matching program causes a computer to perform the following processes: acquire an image, detect skeletal information of the entire body from the image, detect skeletal information of the extremities from the image, calculate a first index based on the positions of the bases of the limbs included in the skeletal information of the entire body and the positions of the bases of the limbs included in the skeletal information of the extremities, calculate a second index based on the orientation of the limbs obtained from the skeletal information of the entire body and the orientation of the bases of the limbs obtained from the skeletal information of the extremities, and match the skeletal information of the extremities for each of the limbs included in the skeletal information of the entire body based on the first index and the second index.

[0010] According to one embodiment, it is possible to improve the matching accuracy of skeletal information of the whole body and the extremities.

[0011] FIG. 1 is a diagram illustrating an example of a system to which a matching function is applied. FIG. 2 is a schematic diagram illustrating an example of a method for recognizing higher-level actions. FIG. 3 is a schematic diagram illustrating one aspect of a problem-solving approach. FIG. 4 is a block diagram illustrating an example of the functional configuration of a server device. FIG. 5 is a block diagram illustrating an example of the functional configuration of an object recognition unit. FIG. 6 is a schematic diagram illustrating an example of a method for detecting a whole-body skeleton. FIG. 7 is a schematic diagram illustrating an example of a method for extracting a hand region. FIG. 8 is a schematic diagram illustrating an example of a detection result of the finger skeleton. FIG. 9 is a schematic diagram illustrating an example of a method for calculating a first vector. FIG. 10 is a schematic diagram illustrating an example of a method for calculating a first index. FIG. 11 is a schematic diagram illustrating an example of a method for calculating a second vector. FIG. 12 is a schematic diagram illustrating an example of a calculation result of a second vector. FIG. 13 is a schematic diagram illustrating an example of a method for calculating a second index. FIG. 14 is a flowchart illustrating the procedure of a matching process. FIG. 15 is a diagram illustrating an example of a hardware configuration.

[0012] Hereinafter, examples for implementing the matching program, matching method, and information processing device according to the present disclosure will be described with reference to the accompanying drawings. Note that these examples merely illustrate one example or aspect, and the structure, action, function, properties, characteristics, methods, uses, etc. according to the present disclosure are not limited by these examples.

[0013] <Example 1> <Example of application system> Fig. 1 is a diagram showing an example of a system to which a matching function is applied. Fig. 1 shows a behavior recognition system 1 to which a matching function is applied, which matches skeletal information of the whole body and the extremities output by two posture estimation models that estimate the posture of the whole body and the posture of the extremities, respectively.

[0014] The behavior recognition system 1 provides a service for recognizing the behavior of a subject, for example, a person, from an image captured by a camera 2. For example, the behavior recognition system 1 recognizes higher-level behaviors that become complex as a result of a combination of multiple actions. Examples of such higher-level behaviors include work behaviors, suspicious behaviors, purchasing behaviors, and any other behaviors that may be recognized.

[0015] Except for the above-mentioned behavior recognition system 1, recognizing "higher-level behaviors" requires the preparation of a large amount of training data, such as videos containing the behavior. Preparing such a large amount of training data can be practically difficult, or even if it is possible to prepare a large amount of training data, it can take time and effort, making it difficult to introduce into the field.

[0016] In order to eliminate the need to prepare such a large amount of training data, the above-mentioned behavior recognition system 1 recognizes higher-level behaviors based on combinations of "elementary behaviors" recognized from "basic actions" performed in daily life through time-series analysis, which will be described later.

[0017] As shown in FIG. 1, the behavior recognition system 1 may include an object recognition function 3, a basic action recognition function 4, a higher-level behavior recognition function 6, and the like.

[0018] The object recognition function 3 is a function for recognizing objects from the video captured by the camera 2. The term "object" here refers to the subject of the video that is the target of behavior recognition, and may include, for example, a person who is the target of behavior recognition. Objects may include non-human animals as well as objects related to behavior recognition, such as tools used by people and fixtures, goods, or equipment placed in the environment.

[0019] For example, the object recognition function 3 recognizes an object for each frame of video. The term "recognition" here may include not only the recognition of the class of individual objects but also the detection of the location of an object, i.e., object detection. Such object recognition can be achieved, by way of example only, by a machine learning model that has been trained for object recognition according to any machine learning algorithm, such as deep learning.

[0020] Then, the object recognition function 3 extracts skeletal information including skeletal parts, for example, the positions of each joint, by applying a process called pose estimation or skeleton detection to objects of the "human" class among the objects. The coordinates of each joint may be two-dimensional coordinates on the image or may be coordinates in three-dimensional space.

[0021] The basic action recognition function 4 is a function that recognizes basic actions from the skeletal information of each frame. The term "basic actions" as used herein may include "whole-body actions," which are movements of a person's entire body, and "partial actions," which are movements of only a part of a person's body. Examples of "whole-body actions" include "walking," "running," and "standing still." Examples of "partial actions" include "raising the right hand," "looking down," and "looking forward." These "whole-body actions" and "partial actions" are "basic actions" performed in daily life and are therefore simpler than "higher-level actions." Therefore, it is easy to collect video footage of "basic actions," and high-accuracy recognition can be achieved using a trained machine learning model. As just one example, recognition of basic actions can be achieved using a machine learning model, such as a neural network or a classifier, that has been trained to recognize basic actions according to an arbitrary machine learning algorithm.

[0022] The higher-level behavior recognition function 6 is a function for recognizing higher-level behaviors. As shown in Fig. 1, the higher-level behavior recognition function 6 may include a component behavior recognition function 6A, a scenario recognition function 6B, and the like.

[0023] The elemental action recognition function 6A is a function that recognizes elemental actions by performing a time series analysis of the recognition results of basic actions. For example, the elemental action recognition function 6A compares time series data of the recognition results for one or more basic actions with the elemental action recognition rules 7A. The elemental action recognition rules 7A define time series patterns for one or more basic actions for each elemental action to be recognized. Examples of time series patterns include the continuity of the basic action and the state transition of the basic action.

[0024] The scenario recognition function 6B is a function that recognizes higher-level actions by checking the recognition results of elemental actions against scenario recognition rules 7B. The scenario recognition rules 7B define a combination of elemental actions as a scenario for each higher-level action to be recognized. Such a scenario can be defined by sequence, logical operation, etc.

[0025] These element behavior recognition rules 7A and scenario recognition rules 7B can be set not only by system definition but also by user definition. For example, the element behavior recognition rules 7A and scenario recognition rules 7B can be customized by a user U, such as a system engineer of a business provider that provides the above-mentioned behavior recognition system 1 or a customer of the business provider, by editing the element behavior recognition rules 7A and scenario recognition rules 7B.

[0026] FIG. 2 is a schematic diagram showing an example of a method for recognizing higher-level actions. FIG. 2 shows, as an example only, time-series data of basic action recognition results for the whole body and each part of the body, including the right arm, face, and waist. Furthermore, as an example only, FIG. 2 shows an example in which basic action IDs (IDentifications) corresponding to labels of the basic action recognition results are plotted in time series. Furthermore, in FIG. 2, hatched portions of the time-series data of the basic action recognition results plotted for the whole body and each part satisfy the time-series patterns of the basic actions defined in the component action recognition rule 7A.

[0027] In the example shown in FIG. 2 , the time series data of the recognition results of the basic movements of the whole body, right arm, and face from time t1 to time t3 satisfy the time series pattern for the basic movements of the whole body, right arm, and face defined as the recognition rule for element action "B" in the element action recognition rule 7A. As a result, the element action recognition function 6A recognizes element action "B" at time t3. Furthermore, the time series data of the recognition results of the basic movement of the right arm from time t3 to time t7 satisfy the time series pattern for the basic movement of the right arm defined as the recognition rule for element action "A" in the element action recognition rule 7A. As a result, the element action recognition function 6A recognizes element action "A" at time t7. Furthermore, the time series data of the recognition results of the basic movement of the waist from time t5 to time t9 satisfy the time series pattern for the basic movement of the waist defined as the recognition rule for element action "P" in the element action recognition rule 7A. As a result, the element action recognition function 6A recognizes element action "P" at time t9. Furthermore, the time-series data of the recognition results of the basic movements of the whole body and right arm from time t6 to time t10 satisfy the time-series pattern for the basic movements of the whole body and right arm defined as the recognition rule for elemental action "J" in elemental action recognition rule 7A. As a result, elemental action recognition function 6A recognizes elemental action "J" at time t10. The order in which the combination of these four elemental actions "A," "B," "J," and "P" is recognized matches the scenario order "B," "A," "P," and "J" defined as the recognition rule for higher-level action "X" in scenario recognition rule 7B. Therefore, scenario recognition function 6B recognizes higher-level action "X" at time t10.

[0028] <Example of use case> Below, as an example of higher-level behavior, we will give an example of task recognition being implemented to recognize individual elemental tasks such as "tightening a screw," "soldering," and "attaching a cover" from the perspective of business improvement aimed at improving productivity and quality at manufacturing sites.

[0029] Furthermore, from the perspective of realizing the above-mentioned task recognition, an example of human posture estimation will be given, in which the posture of the whole body and the posture of the fingers are estimated. While the technology related to posture estimation models that estimate the posture of the whole body and the posture of the fingers individually has been well established, the technology related to posture estimation models that estimate the posture of the whole body and the posture of the fingers simultaneously has yet to be established.

[0030] From this perspective, we will give an example of separately estimating the whole-body posture and the finger posture using two posture estimation models, for example, a whole-body skeletal model that detects skeletal information of the whole body and a finger skeletal model that detects skeletal information of the fingers.

[0031] Hereinafter, skeletal information of the entire body may be referred to as "whole body skeletal information," and skeletal information of the fingers may be referred to as "finger skeletal information."

[0032] When the whole body skeleton and finger skeletons are detected individually, the finger skeletons are linked to the person's whole body skeleton by matching the whole body skeleton output by the whole body skeleton model with the finger skeletons output by the finger skeleton model, from the perspective of realizing the recognition of higher-level actions as shown in Figure 2.

[0033] By combining whole-body skeletal information and hand and finger skeletal information in this way, the entire body of the person to be recognized, including the extremities, can be incorporated into the rule settings of elemental behavior recognition rules 7A and scenario recognition rules 7B.

[0034] Therefore, the elemental action recognition rules 7A and the scenario recognition rules 7B enable the expression of a large amount of information, thereby improving the reproducibility of the recognition of higher-level actions. For example, even if task recognition is performed using only whole-body skeletal information, it is difficult to distinguish between "the action of tightening a screw by hand" and "the action of soldering," but by also using finger skeletal information for task recognition, it becomes possible to distinguish between the two.

[0035] <One Aspect of the Problem> As explained in the Background Art section above, in the above-mentioned Prior Art 1, there is an aspect in which the accuracy of matching skeletal information of the whole body and the extremities decreases.

[0036] For example, in the above-mentioned prior art 1, the correlation used for matching is calculated based on image features such as color information. Therefore, if the appearance features of people appearing in the video are similar, the matching accuracy decreases. In particular, in manufacturing sites, where people wear the same work clothes and gloves, the possibility of matching the finger skeletons and whole body skeletons of different people increases.

[0037] <One aspect of the problem-solving approach> Therefore, the matching function according to this embodiment matches the two based on a first index determined from the wrist position in the whole-body skeletal information and the wrist position in the finger skeletal information, and a second index determined from the orientation of the forearm in the whole-body skeletal information and the orientation of the wrist in the finger skeletal information.

[0038] 3 is a schematic diagram illustrating one aspect of the problem-solving approach. Fig. 3 shows an example of an image frame captured by camera 2, in which an occlusion occurs when the left wrist of person U1 standing in front obscures the right wrist of person U2 standing behind him.

[0039] 3 also shows a map 20A in which part of the whole-body skeletal information and part of the hand and finger skeletal information detected from the image 20 are mapped onto the image 20. For example, as a result of object recognition for the image 20, bounding boxes corresponding to two people, person U1 and person U2, are obtained. Therefore, two pieces of whole-body skeletal information are detected from each of the two images corresponding to the two bounding boxes. On the other hand, four pieces of hand and finger skeletal information are detected from each of the four images corresponding to both wrists of person U1 and both wrists of person U2.

[0040] Of these, map 20A plots the left elbow joint position E13 and left wrist joint position W13 of person U1 from two pieces of whole-body skeletal information, and the wrist joint positions W21 to W24 of both person U1 and person U2 from four pieces of finger skeletal information.

[0041] Furthermore, FIG. 3 shows a map 20B in which a vector corresponding to the orientation of the forearm obtained from the whole-body skeletal information and a vector corresponding to the orientation of the fingers obtained from the finger skeletal information are mapped onto the image 20.

[0042] The orientation of the forearm may refer to the orientation from the elbow joint position to the wrist joint position in the whole-body skeletal information. Hereinafter, the vector corresponding to the orientation from the elbow joint position to the wrist joint position may be referred to as the "elbow-wrist vector."

[0043] The wrist orientation may refer to the orientation from the wrist joint position to the finger joint positions in the finger skeletal information. For example, a composite vector obtained by combining five vectors corresponding to the orientation from the wrist joint position to each of the metacarpo phalangeal (MP) joints of a specific finger, for example, the middle finger, may be referred to as a "wrist-to-finger vector."

[0044] Here, as one aspect of matching the whole body skeleton and finger skeleton, we will explain a scene in which the left forearm, one of the two forearms included in the whole body skeleton information of person U1, is matched with four pieces of finger skeleton information.

[0045] As one aspect, a first index, such as the distance or similarity of the wrist positions, calculated from the left wrist joint position W13 in the whole-body skeletal information of person U1 and the wrist joint positions W21 to W24 included in each of the four finger skeletal information can be used as one of the evaluation criteria for matching.

[0046] The closer the wrist joint positions are between the whole body skeletal information and the finger skeletal information, the more likely that the whole body skeletal information and the finger skeletal information belong to the same person. For example, in the example of map 20A shown in Figure 3, among the wrist joint positions W21 to W24 included in each of the four finger skeletal information, the right wrist joint position W22 of person U2 and the left wrist joint position W23 of person U1 have a high similarity to the left wrist joint position W13 in the whole body skeletal information of person U1. In other words, among the wrist joint positions W21 to W24 included in each of the four finger skeletal information, the left wrist joint position W24 of person U2 and the right wrist joint position W21 of person U1 have a low similarity to the left wrist joint position W13 in the whole body skeletal information of person U1.

[0047] As a further aspect, a second index, such as the similarity between vectors, calculated from the elbow-to-wrist vector V13 corresponding to the left forearm in the whole-body skeletal information of person U1 and the four wrist-to-finger vectors V21 to V24 corresponding to each of the four finger skeletal information can be used as one of the evaluation criteria for matching.

[0048] As the angle between the elbow-wrist vector and the wrist-fingers vector becomes smaller, the wrist joint approaches a neutral position, increasing the possibility that the whole body skeletal information and the finger skeletal information belong to the same person.

[0049] Furthermore, as the angle between the elbow-wrist vector and the wrist-fingers vector increases, the degree of deviation from the range of motion of the wrist joint increases, making it less likely that the whole-body skeletal information and the hand and finger skeletal information belong to the same person.

[0050] 3, the angle between the elbow-wrist vector V13 and the wrist-to-finger vector V23 is smaller than the angle between the elbow-wrist vector V13 and the wrist-to-finger vector V24. In other words, the angle between the elbow-wrist vector V13 and the wrist-to-finger vector V21 is larger than the angle between the elbow-wrist vector V13 and the wrist-to-finger vector V22.

[0051] According to these two indicators, even when occlusion occurs, it can be evaluated that the finger skeletal information including the wrist joint position V23 is the best among the four finger skeletal information to match the left wrist of person U1's whole body skeletal information.

[0052] For example, among the wrist joint positions W21 to W24 included in each of the four sets of finger skeletal information, the right wrist joint position W22 of person U2 and the left wrist joint position W23 of person U1 are both close to the left wrist joint position W13 in the whole-body skeletal information of person U1. Therefore, if only the distance or similarity of wrist positions were used as an evaluation index, it would be difficult to distinguish between the finger skeletal information including the left wrist joint position W23 of person U1 and the finger skeletal information including the right wrist joint position W22 of person U2, which is more suitable as a matching partner to be combined with the left wrist in the whole-body skeletal information of person U1. However, by also using the similarity between vectors as an evaluation index, this distinction becomes possible. That is, because the angle between the elbow-wrist vector V13 and the wrist-fingers vector V22 is large, the finger skeletal information including the right wrist joint position W22 of person U2 can be excluded from the matching partners to be combined with the left wrist in the whole-body skeletal information of person U1.

[0053] Furthermore, the angle between the elbow-wrist vector V13 and the wrist-to-finger vector V23 and the angle between the elbow-wrist vector V13 and the wrist-to-finger vector V24 are both small. Therefore, if the similarity between vectors alone were used as an evaluation index, it would be difficult to distinguish between the hand and finger skeletal information including the left wrist joint position W23 of person U1 and the hand and finger skeletal information including the left wrist joint position W24 of person U2, which is more suitable as a matching partner to be combined with the left wrist in the whole-body skeletal information of person U1. However, by also using the distance and similarity of the wrist positions as evaluation indexes, this distinction becomes possible. That is, because the left wrist joint position W24 of person U2 is far from the left wrist joint position W13 in the whole-body skeletal information of person U1, the hand and finger skeletal information including the left wrist joint position W24 of person U2 can be excluded from the matching partners to be combined with the left wrist in the whole-body skeletal information of person U1.

[0054] These two indices are based on the whole body skeleton and finger skeleton and do not depend on the external characteristics of the person appearing in the video, so they do not affect matching accuracy even if the external characteristics of the people appearing in the video are similar.

[0055] Therefore, the matching function according to this embodiment can improve the matching accuracy of whole-body skeletal information and finger skeletal information. Furthermore, the matching function according to this embodiment can eliminate the need to build a skeletal detection model that simultaneously detects whole-body and finger skeletal information. In addition, the matching function according to this embodiment can be applied to whole-body skeletal information and finger skeletal information detected from video captured by a monocular camera such as an existing security camera. This eliminates the need to install a special camera such as an RGB-D camera, thereby facilitating the implementation of the above-mentioned behavior recognition system 1.

[0056] 3 illustrates an example of occlusion that occurs when parts of multiple people overlap in the front-to-back direction of the optical center of camera 2, but the condition for occlusion to occur is not limited to the presence of multiple people. For example, it is possible to provide robust matching even for occlusion that occurs when parts of one person overlap in the front-to-back direction of the optical center of camera 2, such as when one person crosses their arms or when one person twists their wrist.

[0057] <Configuration of Server Device> Next, a description will be given of an example functional configuration of the server device 10 that provides the behavior recognition service realized by the above-described behavior recognition system 1. Fig. 4 is a block diagram showing an example functional configuration of the server device 10. Fig. 4 schematically shows blocks related to functions corresponding to the behavior recognition service that the server device 10 has.

[0058] The server device 10 is an example of an information processing device that provides the above-mentioned behavior recognition service. For example, the server device 10 can be realized as a Software as a Service (SaaS) type application. This allows the above-mentioned behavior recognition service to be provided as a cloud service. Alternatively, the server device 10 can be realized as a web server that provides the above-mentioned behavior recognition service on-premises.

[0059] 4 illustrates an example in which the matching function is packaged with the behavior recognition service and provided as part of the behavior recognition service, but is not limited to this. For example, the matching function may be provided independently of the behavior recognition service. In this case, the behavior recognition service and the matching function may be provided using different hardware resources, such as physical devices or physical elements.

[0060] As shown in Fig. 4, the server device 10 includes a communication control unit 11, a storage unit 12, and a control unit 13. Note that Fig. 4 only illustrates a selection of functional units related to the functions corresponding to the above-described behavior recognition service, and the server device 10 may include functional units other than those illustrated.

[0061] The communication control unit 11 is a functional unit that controls communication with other devices such as the camera 2. In one embodiment, the communication control unit 11 can be realized by a network interface card.

[0062] In one aspect, the communication control unit 11 can receive images frame by frame from the camera 2. In another aspect, the communication control unit 11 can output various settings related to image capture by the camera 2, such as the frame rate and resolution, to the camera 2.

[0063] The storage unit 12 is a functional unit that stores various types of data. As just one example, the storage unit 12 is realized by an internal, external, or auxiliary storage of the server device 10. For example, the storage unit 12 stores information such as an element behavior recognition rule 12A corresponding to the element behavior recognition rule 7A shown in Fig. 1 and a scenario recognition rule 12B corresponding to the scenario recognition rule 7B shown in Fig. 1.

[0064] The control unit 13 is a functional unit that performs overall control of the server device 10. For example, the control unit 13 may be realized by a hardware processor. As shown in Fig. 4, the control unit 13 has an image acquisition unit 14, an object recognition unit 15, a basic behavior recognition unit 16, a component behavior recognition unit 17, and a scenario recognition unit 18. Note that the control unit 13 may also be realized by hardwired logic or the like.

[0065] The video acquisition unit 14 is a processing unit that acquires video. In one aspect, the video acquisition unit 14 can acquire images transmitted from the camera 2 frame by frame. In another aspect, the video acquisition unit 14 can buffer images transmitted from the camera 2 frame by frame in a predetermined storage area and acquire a series of images as video. Here, the information source from which the video acquisition unit 14 acquires video may be any information source and is not limited to the camera 2. For example, the video acquisition unit 14 can acquire video from a storage device that accumulates video, such as an auxiliary storage device such as a hard disk or optical disk, or a removable medium such as a memory card or USB (Universal Serial Bus) memory. In addition, the video acquisition unit 14 can acquire video from an external device other than the camera 2 via a network NW.

[0066] The object recognition unit 15 is a processing unit that recognizes objects from video. The object recognition unit 15 may correspond to the object recognition function 3 shown in FIG. 1. In one embodiment, the object recognition unit 15 recognizes objects for each frame of video. Then, the object recognition unit 15 applies a process called pose estimation or skeleton detection to objects that belong to the "human" class, thereby extracting skeleton information including skeletal parts, for example, the positions of each joint.

[0067] The basic action recognition unit 16 is a processing unit that recognizes basic actions. The basic action recognition unit 16 may correspond to the basic action recognition function 4 shown in FIG. 1. In one embodiment, the basic action recognition unit 16 inputs the position of each joint detected by the object recognition unit 15 to the machine learning model on a frame-by-frame basis. As a result, basic action recognition results for the entire body and each part are obtained on a frame-by-frame basis as the output of the machine learning model.

[0068] The component action recognition unit 17 is a processing unit that recognizes component actions by comparing time series data of the recognition result for one or more basic actions with the component action recognition rules 12A stored in the storage unit 12. The component action recognition rules 12A define a time series pattern for one or more basic actions for each component action to be recognized. For example, examples of the time series pattern include the continuity of the basic action and the state transition of the basic action.

[0069] The scenario recognition unit 18 is a processing unit that recognizes higher-level actions by comparing the recognition results of the elemental actions with the scenario recognition rules 12B stored in the storage unit 12. The scenario recognition rules 12B define a combination of elemental actions as a scenario for each higher-level action to be recognized. Such a scenario can be defined by sequence, logical operation, etc.

[0070] These element behavior recognition rules 12A and scenario recognition rules 12B can be set not only by system definition but also by user definition. For example, a user U, such as a system engineer on the side of a business providing the behavior recognition service or a customer of the behavior recognition service, can edit the element behavior recognition rules 12A and scenario recognition rules 12B, thereby enabling customization by the user U.

[0071] <Configuration of Object Recognition Unit> Next, an example of the functional configuration of the object recognition unit 15 will be described. Fig. 5 is a block diagram showing an example of the functional configuration of the object recognition unit 15. As shown in Fig. 5, the object recognition unit 15 has a whole body skeleton detection unit 15A, a hand region extraction unit 15B, a finger skeleton detection unit 15C, a first vector calculation unit 15D, a first index calculation unit 15E, a second vector calculation unit 15F, a second index calculation unit 15G, and a matching unit 15H.

[0072] The whole-body skeleton detection unit 15A is a processing unit that detects the whole-body skeleton. FIG. 6 is a schematic diagram illustrating an example of a whole-body skeleton detection method. Similar to FIG. 3 , FIG. 6 illustrates an example of a frame of video captured by the camera 2, in which occlusion occurs due to the left wrist of a person U1 standing in front obscuring the right wrist of a person U2 standing behind the person U1. As shown in FIG. 6 , the whole-body skeleton detection unit 15A inputs the image 20 to a machine learning pipeline implemented by a DL (Deep Learning) framework that implements machine learning tasks for object recognition and whole-body skeleton detection. As a result, the whole-body skeleton detection unit 15A acquires whole-body skeleton information 31 and whole-body skeleton information 32 output by the machine learning pipeline for each object, for example, person U1 and person U2, whose class is "human." The whole-body skeleton information 31 and whole-body skeleton information 32 may include, for example, coordinates of joints such as the head, shoulders, spine, elbows, wrists, hips, knees, and ankles, located on the image 20.

[0073] The hand region cutout unit 15B is a processing unit that cuts out hand regions. Here, the "hand region" may refer to the area from the wrist to the fingertips. FIG. 7 is a schematic diagram illustrating an example of a hand region cutout method. FIG. 7 shows whole-body skeletal information 31 and whole-body skeletal information 32 obtained as the whole-body skeletal detection results shown in FIG. 6. As shown in FIG. 7, the hand region cutout unit 15B detects hand regions P1 to P4 including the wrist for each wrist joint included in the whole-body skeletal information 31 and whole-body skeletal information 32. Such hand region detection can be achieved by skin-color region detection or template matching, or by a machine learning model that has been trained to detect hand shapes. Then, the hand region cutout unit 15B cuts out partial images 21 to 24 corresponding to each of the four hand regions P1 to P4 from image 20.

[0074] The finger skeleton detection unit 15C is a processing unit that detects the finger skeleton. Here, "finger skeleton" may include the wrist joint and finger joints, for example, any part from the first joint to the fourth joint of the five fingers. In one embodiment, the finger skeleton detection unit 15C detects the finger skeleton from each hand region extracted by the hand region extraction unit 15B. Such finger skeleton detection may be achieved using a machine learning model that inputs an image and outputs the finger skeleton. FIG. 8 is a schematic diagram illustrating an example of the finger skeleton detection results. In FIG. 8, finger skeleton information 41 to 44 detected from each of the hand regions P1 to P4 shown in FIG. 7 is mapped. As shown in FIG. 8, each of the finger skeleton information 41 to 44 may include not only the wrist joint positions W21 to W24 but also the coordinates of the finger joint positions, for example, any joint from the first joint to the fourth joint of the five fingers, on the image 20.

[0075] The first vector calculation unit 15D is a processing unit that calculates a first vector corresponding to the orientation of the forearm in the whole-body skeletal information. An example of such a first vector is an elbow-wrist vector corresponding to the orientation from the elbow joint position to the wrist joint position.

[0076] Fig. 9 is a schematic diagram showing an example of a method for calculating a first vector. Fig. 9 shows an example in which elbow-wrist vectors V11 to V14 are calculated from the whole-body skeletal information 31 and the whole-body skeletal information 32 shown in Fig. 6. Furthermore, in Fig. 9, pixels corresponding to the elbow joint position among the joints of the whole body are plotted with white circles, and pixels corresponding to the wrist joint position are plotted with hatched circles.

[0077] 9 , the first vector calculation unit 15D calculates elbow-wrist vectors V11 to V14 for each of the four arms included in the whole-body skeleton information 31, i.e., the left and right arms of person U1, and the arms included in the whole-body skeleton information 32, i.e., the left and right arms of person U2. For example, an elbow-wrist vector V11 corresponding to two points, i.e., the elbow joint position E11 and the wrist joint position W11, included in the whole-body skeleton information 31, is calculated. Furthermore, an elbow-wrist vector V12 corresponding to two points, i.e., the elbow joint position E12 and the wrist joint position W12, included in the whole-body skeleton information 32, is calculated. Furthermore, an elbow-wrist vector V13 corresponding to two points, i.e., the elbow joint position E13 and the wrist joint position W13, included in the whole-body skeleton information 31, is calculated. Furthermore, an elbow-wrist vector V14 corresponding to two points, i.e., the elbow joint position E14 and the wrist joint position W14, included in the whole-body skeleton information 32, is calculated.

[0078] The first index calculation unit 15E is a processing unit that calculates a first index based on the wrist joint position included in the whole-body skeletal information and the wrist joint position included in the finger skeletal information. Examples of such a first index include the distance and similarity between the wrist positions. FIG. 10 is a schematic diagram showing an example of a method for calculating the first index. As shown in FIG. 10, the first index calculation unit 15E calculates, as the first index, distances d1 to d4 between the wrist joint position W13 included in the whole-body skeletal information 31 and the wrist joint positions W21 to W24 included in each of the four pieces of finger skeletal information 41 to 44.

[0079] The second vector calculation unit 15F is a processing unit that calculates a second vector corresponding to the orientation of the fingers in the finger skeletal information. An example of such a second vector is a wrist-to-finger vector corresponding to the orientation from the wrist joint position to the finger joint position. In one aspect, the second vector calculation unit 15F calculates the wrist-to-finger vector for each piece of finger skeletal information detected by the finger skeletal detection unit 15C based on the finger skeletal information.

[0080] FIG. 11 is a schematic diagram showing an example of a method for calculating a second vector. As shown in FIG. 11, the second vector calculation unit 15F calculates a wrist-to-finger vector V21 corresponding to two points: the wrist joint position W21 included in the finger skeletal information 41; and the position W210 of the middle finger MP joint included in the finger skeletal information 41. Here, FIG. 11 illustrates, as merely one example, an example in which the wrist-to-finger vector V21 is calculated from the finger skeletal information 41. However, it goes without saying that the wrist-to-finger vectors V22 to V24 can be calculated from each of the finger skeletal information 42 to 44 using similar logic. As a result, the calculation result of the second vector shown in FIG. 12 is obtained. FIG. 12 is a schematic diagram showing an example of the calculation result of the second vector. As shown in FIG. 12, the second vector calculation unit 15F can calculate the wrist-to-finger vectors V21 to V24 for each of the four finger skeletal information 41 to 44 shown in FIG. 8.

[0081] The second index calculation unit 15G is a processing unit that calculates the second index based on the orientation of the forearm in the whole-body skeletal information and the orientation of the wrist in the finger skeletal information. In one aspect, the second index calculation unit 15G calculates, as the second index, the angle between the elbow-wrist vector calculated by the first vector calculation unit 15D and the wrist-to-finger vector calculated by the second vector calculation unit 15F.

[0082] Fig. 13 is a schematic diagram showing an example of a method for calculating the second index. Fig. 13 shows an example in which the angles formed between the elbow-wrist vector V13, one of the elbow-wrist vectors V11 to V14 shown in Fig. 9, and the wrist-to-finger vectors V21 to V24 shown in Fig. 12 are calculated. As shown in Fig. 13, the second index calculation unit 15G calculates the angle θ31 formed by the elbow-wrist vector V13 and the wrist-to-finger vector V21, the angle θ32 formed by the elbow-wrist vector V13 and the wrist-to-finger vector V22, the angle θ33 formed by the elbow-wrist vector V13 and the wrist-to-finger vector V23, and the angle θ34 formed by the elbow-wrist vector V13 and the wrist-to-finger vector V24.

[0083] The matching unit 15H is a processing unit that matches hand and finger skeletal information for each arm included in all of the whole-body skeletal information. In one embodiment, the matching unit 15H executes the following process a number of times corresponding to the total number M of arms included in all of the whole-body skeletal information. That is, the matching unit 15H calculates N matching scores between the mth arm and N pieces of unmatched hand and finger skeletal information. For example, such matching scores can be calculated based on a first index calculated by the first index calculation unit 15E, such as the distance between the wrist positions, and a second index calculated by the second index calculation unit 15G, such as the angle between two vectors. For example, the matching score can be calculated by performing statistical processing, such as arithmetic averaging or weighted averaging, on the first index and the second index. Alternatively, the matching score value can be scaled to a specific numerical range, such as 0 to 1 or 0 to 100, or normalized to a score in which the likelihood of belonging to the same person increases as the value increases.

[0084] The matching unit 15H then determines whether the highest matching score among the N matching scores is equal to or greater than a threshold. If the highest matching score is equal to or greater than the threshold, the matching unit 15H matches the hand and finger skeleton information for which the highest matching score was calculated with the mth arm as a matching partner for the mth arm. The reason for imposing the matching condition that the highest matching score be equal to or greater than the threshold is to prevent a situation in which a correct matching partner cannot be found in subsequent matching when a mismatch occurs when matching the arm and hand skeletons of full-body skeletons of different people. The matching unit 15H then repeats the above matching until matching partners for all M arms included in the full-body skeleton information are determined.

[0085] In this way, by repeating the above matching until matching partners for the M arms are determined, all finger skeletons can be matched to the M arms included in all whole body skeleton information.

[0086] With this type of matching, as explained using Figure 3, even when an image 20 is input in which wrist occlusion occurs between multiple people, it is possible to match finger skeletal information 43 including wrist joint position V23 corresponding to the left wrist of person U1 as a matching partner for the left wrist of whole-body skeletal information 31 of person U1.

[0087] <Processing Flow> Next, a processing flow of the server device 10 according to this embodiment will be described. Fig. 14 is a flowchart showing the procedure of the matching processing. This processing can be started when a video is acquired by the video acquisition unit 14, and is merely an example.

[0088] As shown in FIG. 14, when an image is acquired by the image acquisition unit 14 (step S101), the whole body skeleton detection unit 15A detects the whole body skeleton from the image acquired in step S101 (step S102).

[0089] Next, the hand region extraction unit 15B extracts a partial image corresponding to each hand region detected from the video acquired in step S101 (step S103).Then, the hand skeleton detection unit 15C detects the hand skeleton from the partial image corresponding to each hand region extracted in step S103 (step S104).

[0090] Thereafter, loop processing 1 is executed, which repeats the processing from step S105 to step S111 described below a number of times corresponding to the total number M of arms included in all the whole-body skeleton information.

[0091] That is, the first vector calculation unit 15D calculates the elbow-wrist vector related to the m-th arm as the first vector (step S105).

[0092] Then, loop process 2 is executed, which repeats the processes from step S106 to step S108 described below a number of times corresponding to the total number N of finger skeletal information that has not yet been matched.

[0093] That is, the first index calculation unit 15E calculates the distance between the wrist joint position of the mth arm and the wrist joint position included in the nth finger skeletal information as the first index (step S106).

[0094] Next, the second vector calculation unit 15F calculates, as a second vector, a wrist-to-finger vector corresponding to two points: the wrist joint position included in the n-th finger skeletal information and the middle finger MP joint position included in the n-th finger skeletal information (step S107).

[0095] Then, the second index calculation unit 15G calculates, as a second index, the angle between the elbow-wrist vector calculated from the mth arm in step S105 and the wrist-fingers vector calculated from the nth hand skeletal information in step S107 (step S108).

[0096] By repeating this loop process 2, the first index and the second index for the m-th arm are calculated for each of N pieces of finger skeletal information.

[0097] Then, the matching unit 15H calculates N matching scores between the mth arm and the N pieces of finger skeletal information based on the first index and the second index calculated for each of the N pieces of finger skeletal information that have not yet been matched (step S109).

[0098] Then, the matching unit 15H determines whether the highest matching score among the N matching scores calculated in step S109 is equal to or greater than a threshold value (step S110).

[0099] Here, if the highest matching score is equal to or greater than the threshold value (Yes in step S110), the matching unit 15H matches the finger skeleton information for which the highest matching score was calculated with the mth arm as the matching partner for the mth arm (step S111).

[0100] If the highest matching score is not equal to or greater than the threshold (No in step S110), the possibility of a mismatch increases. In this case, the process of step S111 is skipped.

[0101] By repeating this loop process 1, all finger skeletons can be matched to the M arms included in all the whole body skeleton information.

[0102] <One Aspect of Effect> As described above, the server device 10 according to the present embodiment matches the two based on a first index calculated from the wrist position in the whole-body skeletal information and the wrist position in the finger skeletal information, and a second index calculated from the forearm orientation in the whole-body skeletal information and the wrist orientation in the finger skeletal information. These two indexes are based on the whole-body skeleton and the finger skeletal information and do not depend on the appearance characteristics of the person appearing in the video. Therefore, even if the appearance characteristics of the people appearing in the video are similar, the matching accuracy is not affected.

[0103] Therefore, the server device 10 according to this embodiment can improve the matching accuracy of whole-body skeletal information and finger skeletal information. Furthermore, the server device 10 according to this embodiment can eliminate the need to build a skeletal detection model that simultaneously detects whole-body and finger skeletal information. In addition, the server device 10 according to this embodiment can be applied to whole-body skeletal information and finger skeletal information detected from video captured by a monocular camera such as an existing security camera. This eliminates the need to install a special camera such as an RGB-D camera, thereby facilitating the implementation of the above-described behavior recognition system 1.

[0104] Second Embodiment Although the embodiments of the present disclosure have been described above, various applications are possible, and further, the present disclosure may be implemented in various different forms other than the first embodiment described above.

[0105] <Exercise of Creativity> The matters described in the above embodiment, such as specific examples of the elbow-wrist vector and the wrist-finger vector, are merely examples and can be changed. Furthermore, the order of processing in the flowcharts described in the embodiment can also be changed within a consistent range.

[0106] (1) Application Example of Wrist-to-Finger Vector In the above-described first embodiment, an example was given in which the wrist-to-finger vector was defined only by a specific finger, for example, the middle finger, but this is not limiting. For example, the wrist-to-finger vector may be defined by a composite vector obtained by combining five vectors corresponding to the directions from the wrist joint to each of the MP joints of the five fingers. Furthermore, in the above-described first embodiment, an example was given in which the MP joint, i.e., the third joint, was used to define the wrist-to-finger vector, but the first, second, or fourth joint of the finger may also be used.

[0107] (2) Application Example of Extremity Distals In the above-described first embodiment, examples of estimating the posture of the whole body and the posture of the fingers have been given as examples of human posture, but the present invention is not limited to this. For example, the posture of the toes can also be estimated as the posture of the extremities in addition to the posture of the fingers. In this case, the matching process shown in FIG. 14 can be similarly applied even when toe skeletal information is detected instead of finger skeletal information by replacing "wrist" with "ankle," "elbow" with "knee," "hand region" with "foot region," "elbow-wrist vector" with "knee-ankle vector," and "wrist-finger vector" with "ankle-finger vector."

[0108] <System> The information including the processing procedures, control procedures, specific names, various data, and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, any one or more of the functional units of the information processing device 10, including the whole body skeleton detection unit 15A, the hand area extraction unit 15B, the finger skeleton detection unit 15C, the first vector calculation unit 15D, the first index calculation unit 15E, the second vector calculation unit 15F, the second index calculation unit 15G, and the matching unit 15H, may be configured as separate devices.

[0109] Furthermore, the components of each device shown in the figure are functional concepts and do not necessarily have to be physically configured as shown. In other words, the specific form of distribution and integration of each device is not limited to that shown. In other words, all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc. Note that each configuration may also be a physical configuration.

[0110] Furthermore, all or any part of the processing performed by the illustrated device may be realized by a program executed by a hardware processor such as an MPU (Micro-Processing Unit) or a CPU (Central Processing Unit), or may be realized as hardware using wired logic.

[0111] <Hardware> Next, an example of the hardware configuration of the information processing device described in the first embodiment will be described. Fig. 15 is a diagram showing an example of the hardware configuration. As shown in Fig. 15, the information processing device 10 includes a communication device 10a, a storage device 10b, a memory 10c, and a processor 10d. Note that the components shown in Fig. 15 may be connected to each other via a bus or the like.

[0112] The communication device 10a is a network interface card, etc. The storage device 10b is a storage device such as a hard disk drive (HDD) or a solid state drive (SSD). For example, the storage device 10b stores programs and databases that operate the functions shown in FIG. 5.

[0113] The processor 10d reads out a program that executes the same processing as the processing unit shown in FIG. 5 from the storage device 10b or the like and loads it into the memory 10c, thereby operating a process that executes the functions described in FIG.

[0114] Such a process realizes the same functions as the processing units of the information processing device 10. For example, the processor 10d reads from the storage device 10b or the like a program having the same functions as the whole body skeleton detection unit 15A, the hand area clipping unit 15B, the finger skeleton detection unit 15C, the first vector calculation unit 15D, the first index calculation unit 15E, the second vector calculation unit 15F, the second index calculation unit 15G, the matching unit 15H, etc. Then, the processor 10d executes a process that executes the same processing as the whole body skeleton detection unit 15A, the hand area clipping unit 15B, the finger skeleton detection unit 15C, the first vector calculation unit 15D, the first index calculation unit 15E, the second vector calculation unit 15F, the second index calculation unit 15G, the matching unit 15H, etc.

[0115] In this way, the information processing device 10 operates as an information processing device that executes a matching output method by reading and executing a program. The information processing device 10 can also realize functions similar to those of the above-described embodiment by reading the program from a recording medium using a medium reading device and executing the read program. Note that the program in these other embodiments is not limited to being executed by the information processing device 10. For example, the present invention can also be applied to cases where another computer or server executes the program, or where these execute the program in cooperation with each other.

[0116] The above program can be distributed via a network such as the Internet. The above program can also be recorded on any recording medium and executed by a computer by reading it from the recording medium. For example, the recording medium can be a hard disk, a flexible disk (FD), a CD-ROM, a magneto-optical disk (MO), a digital versatile disk (DVD), or the like.

[0117] REFERENCE SIGNS LIST 1 Behavior recognition system 2 Camera 10 Server device 11 Communication control unit 12 Memory unit 12A Elemental behavior recognition rules 12B Scenario recognition rules 13 Control unit 14 Video acquisition unit 15 Object recognition unit 15A Whole body skeleton detection unit 15B Hand area extraction unit 15C Hand and finger skeleton detection unit 15D First vector calculation unit 15E First index calculation unit 15F Second vector calculation unit 15G Second index calculation unit 15H Matching unit 16 Basic action recognition unit 17 Elemental behavior recognition unit 18 Scenario recognition unit

Claims

1. A matching program that causes a computer to perform the following processes: acquire an image; detect skeletal information of the entire body from the image; detect skeletal information of the extremities from the image; calculate a first index based on the positions of the bases of the limbs included in the skeletal information of the entire body and the positions of the bases of the limbs included in the skeletal information of the extremities; calculate a second index based on the orientation of the limbs obtained from the skeletal information of the entire body and the orientation of the bases of the limbs obtained from the skeletal information of the extremities; and match the skeletal information of the extremities for each of the limbs included in the skeletal information of the entire body based on the first index and the second index.

2. The matching program of claim 1, characterized in that: the skeletal information of the extremities corresponds to skeletal information of the fingers; the first index corresponds to the distance between the wrist position included in the skeletal information of the whole body and the wrist position included in the skeletal information of the fingers; and the second index is the angle between a first vector corresponding to the orientation of the forearm obtained from the skeletal information of the whole body and a second vector corresponding to the orientation of the wrist obtained from the skeletal information of the fingers.

3. The matching program of claim 2, wherein the first vector corresponds to a direction from the elbow to the wrist, and the second vector corresponds to a direction from the wrist to a specific finger.

4. The matching program according to claim 3, wherein the second vector is a vector obtained by combining five vectors corresponding to the directions from the wrist to each of the five fingers.

5. A matching method characterized by a computer performing the following processes: acquiring an image; detecting skeletal information of the entire body from the image; detecting skeletal information of the extremities from the image; calculating a first index based on the positions of the bases of the limbs included in the skeletal information of the entire body and the positions of the bases of the limbs included in the skeletal information of the extremities; calculating a second index based on the orientation of the limbs obtained from the skeletal information of the entire body and the orientation of the bases of the limbs obtained from the skeletal information of the extremities; and matching the skeletal information of the extremities for each of the limbs included in the skeletal information of the entire body based on the first index and the second index.

6. The matching method described in claim 5, characterized in that the skeletal information of the extremities corresponds to skeletal information of the fingers, the first index corresponds to the distance between the wrist position included in the skeletal information of the whole body and the wrist position included in the skeletal information of the fingers, and the second index is the angle between a first vector corresponding to the orientation of the forearm obtained from the skeletal information of the whole body and a second vector corresponding to the orientation of the wrist obtained from the skeletal information of the fingers.

7. The matching method of claim 6, wherein the first vector corresponds to a direction from the elbow to the wrist, and the second vector corresponds to a direction from the wrist to a particular finger.

8. A matching method according to claim 7, wherein the second vector is a vector obtained by combining five vectors corresponding to the directions from the wrist to each of the five fingers.

9. An information processing device characterized by having a control unit that executes the following processes: acquire an image; detect whole-body skeletal information from the image; detect skeletal information of extremities from the image; calculate a first index based on the positions of the bases of the limbs included in the whole-body skeletal information and the positions of the bases of the limbs included in the skeletal information of the extremities; calculate a second index based on the orientation of the limbs obtained from the whole-body skeletal information and the orientation of the bases of the limbs obtained from the skeletal information of the extremities; and match the skeletal information of the extremities for each of the limbs included in the whole-body skeletal information based on the first index and the second index.

10. The information processing device of claim 9, wherein the skeletal information of the extremities corresponds to skeletal information of the fingers, the first index corresponds to the distance between the wrist position included in the skeletal information of the whole body and the wrist position included in the skeletal information of the fingers, and the second index is the angle between a first vector corresponding to the orientation of the forearm obtained from the skeletal information of the whole body and a second vector corresponding to the orientation of the wrist obtained from the skeletal information of the fingers.

11. The information processing device according to claim 10, wherein the first vector corresponds to a direction from the elbow to the wrist, and the second vector corresponds to a direction from the wrist to a specific finger.

12. The information processing device according to claim 11, wherein the second vector is a vector obtained by combining five vectors corresponding to the directions from the wrist to each of the five fingers.

Images