Selection program, selection method, and information processing device

Abstract

To provide a selection program, a selection method and an information processing device which select a feature amount according to a way of concealing of a portion of a person.SOLUTION: An information processing device acquires learning data which is made of a label about a person and an image in which a portion of the person is concealed and has a first image and a second image obtained by imaging the same person, generates skeleton information in which a plurality of joint positions of the person and the confidence level of each joint position are associated with each other from the first image and second image included in the learning data, generates a plurality of feature amounts for the first image and second image on the basis of the skeleton information, specifies a matching degree between the label of the first image and a label of the second image having the similar partial feature amount using the partial feature amount included in the plurality of feature amounts, and selects the first feature amount from the plurality of feature amounts on the basis of the matching degree.SELECTED DRAWING: Figure 14

Description

[Technical Field]

[0001] The present invention relates to a selection program and the like. [Background technology]

[0002] In various investigations, police officers often use a photograph of a criminal as a clue to check surveillance camera footage to see if the criminal is visible in the footage. Conventional technology exists that extracts features of a person from footage captured by cameras installed in multiple locations and compares the extracted features with the features of the person under investigation. Using such conventional technology can reduce the burden of the verification process. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-60866 [Patent Document 2] Patent Publication No. 2021-51376 [Patent Document 3] Patent Publication No. 2021-33395 [Patent Document 4] US Patent Application Publication No. 2020 / 082157 Summary of the Invention [Problem to be solved by the invention]

[0004] However, if there is an obstacle between the camera and the person, part of the person's body will be hidden, making it impossible to perform matching accurately. The part of a person's body that is hidden is called "occlusion."

[0005] Fig. 17 is a diagram illustrating an example of occlusion. An image 3 shown in Fig. 17 includes a person 3a and a person 3b. An ankle 4 of person 3a is hidden by person 3b, resulting in occlusion.

[0006] In a situation where occlusion occurs as described in FIG. 17, if the feature amounts of person 3a are extracted and matched, the feature amounts will differ from those when all parts of person 3a are visible, and therefore matching of person 3a cannot be performed accurately.

[0007] For example, if occlusion occurs in the ankle part 4, it is thought that the matching accuracy when occlusion occurs can be improved by selecting some feature amounts suitable for matching from multiple feature amounts of person 3a. However, since the parts where occlusion occurs change, it is necessary to select feature amounts according to how the parts of the person are hidden.

[0008] In one aspect, the present invention aims to provide a selection program, a selection method, and an information processing device that can select feature amounts depending on how parts of a person are hidden. [Means for solving the problem]

[0009] In the first proposal, a computer is caused to perform the following process. The computer acquires training data consisting of labels related to a person and images in which the person's body parts are hidden, the training data having a first image and a second image of the same person. The computer generates skeletal information from the first image and the second image included in the training data, in which multiple joint positions of the person are associated with the reliability of each joint position. The computer generates multiple feature amounts for the first image and the second image based on the skeletal information. The computer uses some of the multiple feature amounts to identify the degree of match between the label of the first image and the label of the second image, some of which have similar feature amounts. The computer selects a first feature amount from the multiple feature amounts based on the degree of match. [Effects of the Invention]

[0010] The feature amount can be selected depending on how parts of the person are hidden. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating an example of a system according to the present embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of learning data. [Figure 3] FIG. 3 is a diagram (1) for explaining the pre-processing. [Figure 4] FIG. 4 is a diagram (2) for explaining the pre-processing. [Figure 5] FIG. 5 is a diagram (3) for explaining the pre-processing. [Figure 6] FIG. 6 is a diagram illustrating an example of the data structure of the feature definition table. [Figure 7] FIG. 7 is a diagram showing an example of video data. [Figure 8] FIG. 8 is a functional block diagram illustrating the configuration of an information processing device according to this embodiment. [Figure 9] FIG. 9 is a diagram illustrating an example of the data structure of the determination table. [Figure 10] FIG. 10 is a diagram for explaining feature amounts obtained from one image frame. [Figure 11] FIG. 11 is a diagram for explaining feature amounts obtained from a plurality of image frames. [Figure 12] FIG. 12 is a diagram illustrating an example of the data structure of a feature vector. [Figure 13] FIG. 13 is a diagram for explaining the processing of the matching unit. [Figure 14] FIG. 14 is a flowchart showing the procedure of the pre-processing of the information processing device. [Figure 15] FIG. 15 is a flowchart showing the processing procedure of the collation process of the information processing device. [Figure 16] FIG. 16 is a diagram illustrating an example of a hardware configuration of a computer that realizes the same functions as the information processing apparatus of the embodiment. [Figure 17] FIG. 17 is a diagram illustrating an example of occlusion. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the selection program, selection method, and information processing device disclosed in the present application will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to these embodiments. [Example]

[0013] Fig. 1 is a diagram showing an example of a system according to this embodiment. As shown in Fig. 1, this system 1 has a camera 10 and an information processing device 100. The camera 10 and the information processing device 100 are connected to each other via a network 30. Although the example shown in Fig. 1 shows only the camera 10, the system according to this embodiment may also include other cameras.

[0014] The camera 10 captures video within its capture range and transmits the captured video data to the information processing device 100. In the following description, the video data transmitted by the camera 10 to the information processing device 100 will be referred to as video data. In this embodiment, the description will be given using video data of a person.

[0015] The video data includes a plurality of image frames in a time series. Each image frame is assigned a frame number in ascending chronological order. One image frame is a still image captured by the camera 10 at a certain timing.

[0016] The information processing device 100 is a device that acquires video data from the camera 10 and performs person matching. For example, the information processing device 100 performs pre-processing and matching processing. The pre-processing and matching processing will be described below in this order.

[0017] An example of "pre-processing" performed by the information processing device 100 will be described. The information processing device 100 performs the pre-processing using training data prepared in advance. FIG. 2 is a diagram showing an example of the training data. As shown in FIG. 2, the training data 50 includes a database 50a and a query 50b.

[0018] Database 50a associates correct labels with video data. The correct label is information that uniquely identifies a person included in the video data. The video data is video data that has been captured in advance by camera 10 (or another camera). In the example shown in FIG. 2, correct label "X" is associated with video data 51a. Correct label "Y" is associated with video data 52a. Correct label "Z" is associated with video data 53a. For example, it is indicated that video data 51a shows a person identified by correct label X.

[0019] Query 50b associates a correct label with video data. The correct label is information that uniquely identifies a person included in the video data. The video data is video data that has been captured in advance by camera 10 (or another camera). In the example shown in FIG. 2, a correct label "X" is associated with video data 51b. A correct label "Y" is associated with video data 52b. A correct label "Z" is associated with video data 53b. For example, it is indicated that a person identified by the correct label X appears in video data 51b.

[0020] Here, since the correct answer label of video data 51a and video data 51b is the same, "X," the person shown in video data 51a and the person shown in video data 51b are the same person. Note that video data 51a and video data 51b are video data of the corresponding person captured at different times or in different places. In other words, video data 51a is not a copy of video data 51b.

[0021] The information processing device 100 executes the following processing for each piece of video data registered in the learning data 50. The information processing device 100 inputs each image frame included in the video data into a machine-learned learning model such as OpenPose, thereby detecting skeletal information of a person included in each image frame.

[0022] FIG. 3 is a diagram (1) for explaining pre-processing. The information processing device 100 detects skeletal information 20b by inputting an image frame 20a into a learning model. The skeletal information 20b includes two-dimensional coordinates (x, y) of multiple joints of a person. The multiple joints include the nose, neck, left and right eyes, ears, shoulders, elbows, wrists, buttocks, knees, ankles, etc. In the following description, the two-dimensional coordinates of the joints will be referred to as "joint coordinates."

[0023] When the learning model receives an input of an image frame and outputs the joint coordinates of each joint, it also outputs a reliability indicating the likelihood of the joint coordinates.

[0024] The information processing device 100 determines an occlusion part based on the reliability of the joint coordinates output from the learning model. In this embodiment, a person's body parts are associated in advance with the types of joints included in the parts, and when the reliability of the joint coordinates corresponding to a part is less than a threshold, the corresponding part is identified as an "occlusion part."

[0025] In the example described with reference to FIG. 3, in the skeleton information 20b, the reliability of the joint coordinates for the joints "left ankle, right ankle" is less than the threshold, and the occlusion part is the "ankle."

[0026] The information processing device 100 detects skeletal information for each image frame included in the video data, and identifies multiple feature amounts based on the joint coordinates and the transition of the joint coordinates included in the skeletal information. The multiple feature amounts include the height of the right arm, the height of the left arm, the height of the right knee, the height of the left knee, the maximum stride length, the time the feet are in the air, the angles of each joint, and the line of sight while walking. In the following description, the multiple feature amounts will be referred to as a "feature vector."

[0027] The information processing device 100 repeatedly executes the above process for each piece of video data registered in the query 50b, and identifies an occlusion region and a feature vector for each piece of video data. Although not shown in the figure, the information processing device 100 registers an occlusion region and a feature vector for each piece of video data in the query 50b of FIG.

[0028] The information processing device 100 repeatedly executes the above process for each piece of video data registered in the database 50a, and identifies a feature vector for each piece of video data. Although not shown in the figure, the information processing device 100 registers the feature vector for each piece of video data in the database 50a of FIG.

[0029] The information processing device 100 selects an optimal combination of feature amounts for each occlusion region by performing the following process. FIG. 4 is a diagram (2) for explaining the pre-processing. The information processing device 100 defines in advance a plurality of partial feature amounts with modified combinations from among all feature amounts included in the feature amount vector. The number of feature amounts (types of feature amounts) included in the partial feature amount is assumed to be less than the number of feature amounts in the feature amount vector.

[0030] In the example shown in FIG. 4, the feature vector includes the right arm height, left arm height, right knee height, left knee height, maximum stride length, and time the feet are in the air. Partial feature A includes the left arm height, right arm height, and so on. Partial feature B includes the maximum stride length and time the feet are in the air, and partial feature C includes the left knee height, right knee height, and so on. In the example shown in FIG. 4, partial features A, B, and C are shown, but other partial features can also be defined.

[0031] FIG. 5 is a diagram (3) for explaining the pre-processing. The information processing device 100 identifies the degree of match of the labels of each partial feature for each occlusion region, and identifies the partial feature with the highest degree of match of the labels as the partial feature of the corresponding occlusion region. In the following explanation, in order to distinguish between the partial feature identified from the video data of the database 50a and the partial feature identified from the video data of the query 50b, the partial feature identified from the video data of the query 50b will be referred to as the "query partial feature." In the explanation of FIG. 5, the occlusion region in the video data 51b to 53b of the query 50b will be described as the "ankle."

[0032] The information processing device 100 extracts the value of the partial feature amount A from the feature amount vectors of the video data 51a to 53a in the database 50a. The information processing device 100 extracts the value of the query partial feature amount A from the feature amount vectors of the video data 51b to 53b in the query 50b.

[0033] The information processing device 100 calculates the similarity (cosine similarity) between the partial feature A of each of the video data 51a to 53a and the query partial feature A of the video data 51b, and identifies the correct label of the video data corresponding to the partial feature A that has the maximum similarity to the query partial feature. For example, if the similarity between the partial feature A of the video data 51a and the query partial feature A of the video data 51b is the maximum similarity, the correct label is "X" and they match.

[0034] The information processing device 100 calculates the similarity between the partial feature A of each of the video data 51a to 53a and the query partial feature A of the video data 52b, and identifies the correct label of the video data corresponding to the partial feature A that has the maximum similarity to the query partial feature. For example, if the similarity between the partial feature A of the video data 52a and the query partial feature A of the video data 52b is set to be the maximum similarity, the correct labels will not match (X and Y do not match).

[0035] The information processing device 100 calculates the similarity between the partial feature A of each of the video data 51a to 53a and the query partial feature A of the video data 53b, and identifies the correct label of the video data corresponding to the partial feature A that has the maximum similarity to the query partial feature. For example, if the similarity between the partial feature A of the video data 51a and the query partial feature A of the video data 51b is the maximum similarity, the correct label is "X", which is the same.

[0036] In the above example, when the information processing device 100 performs a search using the partial feature amount A for the video data of the occlusion part "ankle," the degree of match of the label is "2 / 3."

[0037] As with partial feature A, the information processing apparatus 100 executes a search using partial feature B, C and other partial feature amounts for the video data of the occlusion part "ankle" to identify the degree of match of the labels.

[0038] The information processing device 100 compares the degree of match of the labels for each partial feature and identifies the partial feature with the highest degree of match of the labels as the partial feature of the occlusion region. For example, if the degree of match of the labels when partial feature A is used for the occlusion region "ankle" is the highest, the information processing device 100 identifies the partial feature of the occlusion region "ankle" as partial feature A.

[0039] The information processing device 100 identifies the partial feature amounts of other occlusion regions by performing the same process as described above for the other occlusion regions. Based on the identification results, the information processing device 100 registers the relationship between the occlusion region and the partial feature amounts in the feature amount definition table.

[0040] Fig. 6 is a diagram showing an example of the data structure of a feature definition table. As shown in Fig. 6, this feature definition table 52 associates occlusion areas, the number of feature amounts, and combinations of feature amounts. An occlusion area indicates an area in video data where occlusion has occurred. The number of feature amounts is the number of feature amounts included in a partial feature amount corresponding to the occlusion area. A feature amount combination indicates a combination of each feature amount constituting a partial feature amount.

[0041] Next, a description will be given of an example of the "matching process" executed by the information processing device 100. In the description of the matching process, the video data input from the camera 10 and that is the target of the matching process will be referred to as "matching target video data" as appropriate.

[0042] The information processing device 100 inputs each image frame of the video data to be matched into a learning model to identify each piece of skeletal information and the reliability of each joint coordinate. The information processing device 100 identifies an occlusion part in the video data to be matched based on the reliability of each joint coordinate.

[0043] The information processing device 100 compares the identified occlusion region with the feature definition table 52 to identify a combination of features corresponding to the identified occlusion region. In the following description, the identified combination of features will be referred to as a "specific feature."

[0044] The information processing device 100 identifies a feature vector based on the coordinates of each joint included in the skeletal information obtained from the video data to be matched and the transition of the joint information. The information processing device 100 extracts a specific feature from the feature vector of the video data to be matched. In the following description, the specific feature extracted from the feature vector of the video data to be matched is referred to as a "first specific feature."

[0045] The information processing device 100 extracts specific features from the feature vectors of each piece of video data in the database 50a, which were identified in the pre-processing. In the following description, the specific features extracted from the feature vectors of each piece of video data in the database 50a will be referred to as "second specific features."

[0046] The information processing device 100 calculates the cosine similarity between the first specific feature and each of the second specific features, and identifies, as a matching result, the correct label of the video data from which the second specific feature is extracted, which has the largest cosine similarity with the first specific feature.

[0047] As described above, in the pre-processing, the information processing device 100 identifies the degree of match of the labels of each partial feature for each occlusion region, and identifies the partial feature with the highest degree of match of the labels as the partial feature corresponding to the corresponding occlusion region. This makes it possible to select features according to occlusion. Furthermore, because it is possible to select features according to occlusion, it is possible to perform matching processing with high accuracy even for video data in which occlusion has occurred.

[0048] FIG. 7 is a diagram showing an example of video data. The example shown in FIG. 7 shows video data (image frames) 5A, 5B, and 5C. In video data 5A, the entire person is captured. In video data 5B, occlusion occurs in the person's waist area. In video data 5C, occlusion occurs in the person's lower body area. In conventional technology, matching is performed using all feature amounts regardless of the occluded area, so matching accuracy decreases depending on the amount and position of the hidden area. While matching accuracy does not decrease for video data 5A, it decreases for video data 5B and 5C due to the influence of occlusion. For example, in video data 5C, despite the feature amounts related to the lower body being unreliable due to occlusion, the feature amounts related to the lower body are used as is, resulting in reduced matching accuracy.

[0049] On the other hand, the information processing device 100 according to this embodiment selects a combination of feature amounts according to the occlusion region and performs matching, so that even if occlusion occurs as in the video data 5B and 5C, it is possible to prevent a decrease in matching accuracy. For example, in the video data 5C, the information processing device 100 performs matching using a highly reliable combination of feature amounts other than those of the lower body, so that it is possible to maintain matching accuracy.

[0050] Next, an example of the configuration of the information processing device 100 according to this embodiment will be described. Fig. 8 is a functional block diagram showing the configuration of the information processing device according to this embodiment. As shown in Fig. 8, this information processing device 100 has a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.

[0051] The communication unit 110 is connected to the camera 10 via the network 30 and receives video data. For example, the communication unit 110 is realized by a network interface card (NIC) or the like. The communication unit 110 may also be connected to other external devices or the like via the network 30.

[0052] The input unit 120 is an input device that inputs various types of information to the information processing device 100. The input unit 120 corresponds to a keyboard, a mouse, a touch panel, or the like.

[0053] The display unit 130 is a display device that displays information output from the control unit 150. The display unit 130 corresponds to a liquid crystal display, an organic EL (Electro Luminescence) display, a touch panel, or the like.

[0054] The storage unit 140 includes learning data 50, a learning model 51, a feature definition table 52, a video buffer 60, and a judgment table 61. The storage unit 140 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk.

[0055] The training data 50 is data used in pre-processing, and corresponds to the training data 50 described in Fig. 2. As described in Fig. 2, the training data 50 includes a database 50a and a query 50b.

[0056] The learning model 51 is a machine-learned learning model such as OpenPose. By inputting an image frame to the learning model 51, the joint coordinates of each joint and the reliability of each joint coordinate are output. For example, the learning model is a neural network (NN).

[0057] The feature definition table 52 is a table that defines an optimal combination of features for each occlusion region, and is generated in pre-processing. The data structure of the feature definition table 52 is the same as that of the feature definition table 52 described in FIG. 6.

[0058] The video buffer 60 is a buffer that holds video data acquired from the camera 10 (video data to be matched).

[0059] The determination table 61 is a table used when determining an occlusion part. FIG. 9 is a diagram showing an example of the data structure of the determination table. As shown in FIG. 9, this determination table 61 associates an occlusion part with a joint. For example, the joint corresponding to the occlusion part "head" is "nose, neck." Therefore, when the reliability corresponding to the joint coordinates of the nose and neck is less than a threshold, the determination table 61 defines that the occlusion part is the "head." The same applies to the relationship between the occlusion part and the joint in the determination table 61. The relationship between the occlusion part and the joint in the determination table 61 may be changed as appropriate.

[0060] The control unit 150 includes an acquisition unit 151, a generation unit 152, a selection unit 153, and a matching unit 154. The control unit 150 is realized by, for example, a central processing unit (CPU) or a micro processing unit (MPU). The control unit 150 may also be implemented by, for example, an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0061] The acquisition unit 151 acquires the video data to be matched from the camera 10 and registers the acquired video data to be matched in the video buffer 60. The acquisition unit 151 may acquire the learning data 50, the machine-learned learning model 51, etc. from an external device and register them in the storage unit 140.

[0062] The generation unit 152 generates feature vectors of the video data in the database 50a and the query 50b included in the training data 50. The processing by the generation unit 152 corresponds to the processing for generating feature vectors described in the pre-processing above. The generation unit 152 executes the following processing for each video data registered in the training data 50.

[0063] The generation unit 152 detects skeletal information of a person included in each image frame by inputting each image frame included in the video data into the learning model 51. The skeletal information includes the joint coordinates of each joint and the reliability of each joint coordinate. The generation unit 152 compares the joint coordinates (joint type) of which reliability is less than a threshold value with the determination table 61 to identify an occlusion part.

[0064] The generation unit 152 generates a feature vector of the video data based on the joint coordinates included in the skeletal information and the transition of the joint coordinates. The feature vector includes feature vectors obtained from one image frame and feature vectors obtained from multiple image frames.

[0065] FIG. 10 is a diagram illustrating feature amounts obtained from one image frame. As shown in FIG. 10, feature amounts such as stride length f1, arm height f2, and knee height f3 are extracted from one image frame. The stride length f1 is the distance between the joint coordinates of the left and right ankles included in the skeletal information 10-1 of a certain image frame. The arm height f2 of the skeletal information 10-1 is the distance from the left ankle (right ankle) to the left wrist (right wrist). The knee height f3 of the skeletal information 10-1 is the distance from the left ankle (right ankle) to the left knee (right knee). For example, the generation unit 152 calculates the stride length f1 from the skeletal information of each image frame included in the video data and calculates the average value of each stride length f1 to calculate the feature amount of the stride length. The same applies to the feature amounts related to the arm height and knee height.

[0066] FIG. 11 is a diagram illustrating feature quantities obtained from multiple image frames. Here, the description will be given using skeletal information 11-1, 11-2, and 11-3. Skeletal information 11-1 is skeletal information extracted from the image frame with frame number n-2. Skeletal information 11-2 is skeletal information extracted from the image frame with frame number n-1. Skeletal information 11-3 is skeletal information extracted from the image frame with frame number n. For example, if the right ankle starts floating in frame number n-2 and lands on the ground in frame number n, the generation unit 152 calculates the time from frame number n-2 to frame number n as the floating time of the foot. The generation unit 152 calculates the floating time for the video data multiple times and calculates the maximum (or average) value of each floating time, thereby calculating the feature quantity of the floating time.

[0067] The generation unit 152 calculates other feature amounts (height of each joint, joint angle, transition of each joint, line of sight while walking, etc.) using conventional technology.

[0068] Fig. 12 is a diagram showing an example of the data structure of a feature vector. As shown in Fig. 12, a value is set for each feature (type of feature) in the feature vector.

[0069] The generation unit 152 repeatedly executes the above process for each piece of video data registered in the database 50a, and identifies a feature vector for each piece of video data. Although not shown in the figure, the generation unit 152 registers the feature vector for each piece of video data in the database 50a in FIG.

[0070] The generation unit 152 repeatedly executes the above process for each piece of video data registered in the query 50b, and identifies an occlusion region and a feature vector for each piece of video data. Although not shown in the figure, the generation unit 152 registers the occlusion region and the feature vector for each piece of video data in the query 50b in FIG.

[0071] The selection unit 153 identifies the degree of match of the labels of each partial feature for each occlusion region based on the learning data 50, and selects the partial feature with the highest degree of match of the label as the partial feature of the corresponding occlusion region. The selection unit 153 registers the selection result in the feature definition table 52. The processing of the selection unit 153 corresponds to the pre-processing described in FIG. 5. Similar to the description of FIG. 5, the selection unit 153 will be described assuming that the occlusion region of the video data 51b to 53b of the query 50b is the "ankle."

[0072] The selection unit 153 extracts the value of the partial feature amount A from the feature amount vectors of the video data 51a to 53a in the database 50a. The selection unit 153 extracts the value of the query partial feature amount A from the feature amount vectors of the video data 51b to 53b in the query 50b.

[0073] The selection unit 153 calculates the similarity (cosine similarity) between the partial feature A of each of the video data 51a to 53a and the query partial feature A of the video data 51b, and identifies the correct label of the video data corresponding to the partial feature A that has the maximum similarity to the query partial feature. For example, if the similarity between the partial feature A of the video data 51a and the query partial feature A of the video data 51b is the maximum similarity, the correct label is "X" and they match.

[0074] The selection unit 153 calculates the similarity between the partial feature A of each of the video data 51a to 53a and the query partial feature A of the video data 52b, and identifies the correct label of the video data corresponding to the partial feature A that has the maximum similarity to the query partial feature. For example, if the similarity between the partial feature A of the video data 52a and the query partial feature A of the video data 52b is set to be the maximum similarity, the correct labels will not match (X and Y do not match).

[0075] The selection unit 153 calculates the similarity between the partial feature A of each of the video data 51a to 53a and the query partial feature A of the video data 53b, and identifies the correct label of the video data corresponding to the partial feature A that has the maximum similarity to the query partial feature. For example, if the similarity between the partial feature A of the video data 51a and the query partial feature A of the video data 53b is the maximum similarity, the correct label is "X", which is the same.

[0076] In the above example, when the selection unit 153 searches the video data of the occlusion part "ankle" using the partial feature amount A, the degree of match of the label is "2 / 3".

[0077] As with partial feature A, the selection unit 153 executes a search using partial feature B, C, and other partial feature amounts for the video data of the occlusion part "ankle" to identify the degree of match of the labels.

[0078] The selection unit 153 compares the degree of coincidence of the labels for each partial feature and identifies the partial feature with the highest degree of coincidence of the labels as the partial feature of the occlusion region. For example, if the degree of coincidence of the labels when partial feature A is used for the occlusion region "ankle" is the highest, the selection unit 153 selects the partial feature of the occlusion region "ankle" as partial feature A.

[0079] The selection unit 153 executes the same process as above for other occlusion regions to identify the partial feature amounts of the other occlusion regions. Based on the identification results, the selection unit 153 registers the relationship between the occlusion region and the partial feature amounts in the feature amount definition table 52.

[0080] Here, the occlusion part is a part corresponding to the joint coordinates with low reliability among the joint coordinates, and therefore, the feature definition table 52 can be said to associate information on reliability with partial feature amounts.

[0081] The matching unit 154 identifies the identification information of a person included in the video data to be matched by matching the video data to be matched with the video data in the database 50a of the learning data 50. The processing of the matching unit 154 corresponds to the above-mentioned matching processing.

[0082] Fig. 13 is a diagram for explaining the processing of the matching unit. The matching unit 154 inputs each image frame of the matching target video data 40 registered in the video buffer 60 into the learning model 51, and identifies each piece of skeletal information and the reliability of each joint coordinate. The matching unit 154 identifies an occlusion part in the matching target video data based on the reliability of each joint coordinate and the decision table 61. In the example shown in Fig. 13, the occlusion part is the "ankle."

[0083] The collation unit 154 compares the identified occlusion region with the feature definition table 52 to identify a combination of features (specific feature) corresponding to the identified occlusion region. For example, if the occlusion region is the "ankle," the specific feature will be "the height of the left arm, the angle of the right arm, etc."

[0084] The matching unit 154 identifies a feature vector based on the coordinates of each joint included in the skeletal information obtained from the video data to be matched and the transition of the joint information. The process by which the matching unit 154 identifies a feature vector is similar to the process by which the generation unit 152 identifies a feature vector. The matching unit 154 extracts a first specific feature from the feature vector of the video data to be matched. In the example shown in FIG. 13, the matching unit 154 extracts a first specific feature 40a.

[0085] The matching unit 154 extracts second specific features from the feature vectors of each video data in the database 50a identified in the pre-processing. In the example shown in Fig. 13, the matching unit 154 extracts second specific features 45b from the feature vector of video data 45.

[0086] The matching unit 154 calculates the cosine similarity between the first specific feature 40a and the second specific feature 45a. The matching unit 154 calculates the cosine similarity between the first specific feature 40a and a second specific feature extracted from other video data. The matching unit 154 identifies, as the matching result, the correct label of the video data from which the second specific feature was extracted, which has the largest cosine similarity with the first specific feature. The matching unit 154 outputs the matching result to the display unit 130 for display.

[0087] Next, an example of a processing procedure of the information processing device 100 according to this embodiment will be described. Fig. 14 is a flowchart showing the processing procedure of pre-processing of the information processing device. As shown in Fig. 14, the generation unit 152 of the information processing device 100 acquires video data from the learning data 50 (step S101).

[0088] The generation unit 152 extracts skeletal information from each image frame of the video data using the learning model 51 (step S102). The generation unit 152 determines an occlusion part based on the reliability of each joint coordinate and the determination table 61 (step S103). The generation unit 152 calculates a feature vector of the video data (step S104).

[0089] The selection unit 153 of the information processing device 100 identifies the degree of matching of the labels for each occlusion region and each partial feature (step S105). The selection unit 153 identifies the partial feature corresponding to the occlusion region based on the degree of matching of the labels (step S106).

[0090] The selection unit 153 registers the relationship between the occlusion part and the identified partial feature amount (combination of feature amounts) in the feature amount definition table 52 (step S107).

[0091] 15 is a flowchart showing the processing procedure of the matching process of the information processing device. As shown in Fig. 15, the acquisition unit 151 of the information processing device 100 acquires the matching target video data from the camera 10 and registers it in the video buffer 60 (step S201).

[0092] The matching unit 154 of the information processing device 100 extracts skeletal information from each image frame of the video data to be matched using the learning model 51 (step S202). The matching unit 154 identifies an occlusion part based on the reliability of each joint coordinate and the determination table 61 (step S203).

[0093] The matching unit 154 identifies a specific feature based on the identified occlusion region and the feature definition table 52 (step S204). The matching unit 154 calculates the similarity between the first specific feature and each of the second specific features (step S205).

[0094] The matching unit 154 identifies a correct label corresponding to the video data of the second specific feature that has the greatest similarity to the first specific feature (step S206). The matching unit 154 outputs the identification result to the display unit 130 for display (step S207).

[0095] Next, the effects of the information processing device 100 according to this embodiment will be described. The information processing device 100 identifies the degree of coincidence of the labels of each partial feature for each occlusion region based on the video data registered in the learning data 50, and selects a partial feature corresponding to the occlusion region based on the degree of coincidence of the labels. This makes it possible to select a feature according to the occlusion.

[0096] The information processing device 100 associates the partial features selected based on the degree of matching of the labels with the occlusion region, and registers them in the feature definition table 52. During the matching process, by using the feature definition table 52, it is possible to select and use a combination of features suitable for the occlusion region.

[0097] The information processing device 100 calculates the similarity between the partial feature of the video data of the query 50b and the partial feature of the video data of the database 50a, and calculates the degree of match using the match or mismatch of the labels of the video data pair with the highest similarity. This makes it possible to identify a combination of feature that can prevent a decrease in matching accuracy when occlusion occurs.

[0098] The information processing device 100 identifies a combination of features suitable for the occlusion area based on the occlusion area of the video data to be matched and the feature definition table 52, and performs matching using the identified combination of features. This makes it possible to accurately identify a person appearing in the video data to be matched, even when occlusion occurs.

[0099] Incidentally, the information processing device 100 according to the present embodiment identifies an occlusion area based on the reliability of joint coordinates, but the present invention is not limited to this. For example, when the information processing device 100 detects an obstruction between a person and a camera using an object detection sensor or the like, the information processing device 100 may identify an occlusion area according to the position of the obstruction.

[0100] Next, an example of the hardware configuration of a computer that realizes the same functions as the information processing device 100 described in the above embodiment will be described. Fig. 16 is a diagram showing an example of the hardware configuration of a computer that realizes the same functions as the information processing device of the embodiment.

[0101] 16, the computer 200 has a CPU 201 that executes various types of arithmetic processing, an input device 202 that accepts data input from a user, and a display 203. The computer 200 also has a communication device 204 that exchanges data with the camera 10, external devices, etc. via a wired or wireless network, and an interface device 205. The computer 200 also has a RAM 206 that temporarily stores various types of information, and a hard disk drive 207. The devices 201 to 207 are connected to a bus 208.

[0102] The hard disk device 207 stores an acquisition program 207a, a generation program 207b, a selection program 207c, and a matching program 207d. The CPU 201 reads out each of the programs 207a to 207d and loads them into the RAM 206.

[0103] The acquisition program 207a functions as the acquisition process 206a. The generation program 207b functions as the generation process 206b. The selection program 207c functions as the selection process 206c. The matching program 207d functions as the matching process 206d.

[0104] The processing of the acquisition process 206a corresponds to the processing of the acquisition unit 151. The processing of the generation process 206b corresponds to the processing of the generation unit 152. The processing of the selection process 206c corresponds to the processing of the selection unit 153. The processing of the matching process 206d corresponds to the processing of the matching unit 154.

[0105] It should be noted that each of the programs 207a to 207d does not necessarily have to be stored in the hard disk drive 207 from the beginning. For example, each of the programs may be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, or IC card that is inserted into the computer 200. Then, the computer 200 may read and execute each of the programs 207a to 207d.

[0106] The following supplementary notes are further disclosed regarding the embodiments including the above examples.

[0107] (Supplementary Note 1) Acquire learning data including a label related to a person and an image in which a part of the person is hidden, the learning data including a first image and a second image of the same person; generating skeletal information in which a plurality of joint positions of the person are associated with reliability of each joint position from the first image and the second image included in the learning data; generating a plurality of feature amounts for the first image and the second image based on the skeletal information; using a portion of the plurality of feature amounts, a degree of agreement between a label of a first image and a label of a second image, the portion of the feature amounts being similar; selecting a first feature amount from the plurality of feature amounts based on the degree of match; A selection program that causes a computer to execute a process.

[0108] (Supplementary Note 2) The selection program according to Supplementary Note 1, further causing a computer to execute a process of registering the selected first feature amount and the reliability in a storage device in association with each other.

[0109] (Appendix 3) The selection program described in Appendix 1, characterized in that the process of determining the degree of match calculates the similarity between a portion of the feature of the first image and a portion of the feature of the second image, and determines the label of the second image that has the greatest similarity to the portion of the feature of the first image by performing this process for each first image with a different label.

[0110] (Appendix 4) The selection program described in Appendix 2, characterized in that when a third image is acquired, the selection program further causes the computer to execute a process of identifying a first feature corresponding to the reliability of each joint position of the third image based on information registered in the storage device, and extracting a label of a first image similar to the third image based on the identified first feature.

[0111] (Supplementary Note 5) Acquire learning data including a label related to a person and an image in which a part of the person is hidden, the learning data including a first image and a second image of the same person; generating skeletal information in which a plurality of joint positions of the person are associated with reliability of each joint position from the first image and the second image included in the learning data; generating a plurality of feature amounts for the first image and the second image based on the skeletal information; using a portion of the plurality of feature amounts, a degree of agreement between a label of a first image and a label of a second image, the portion of the feature amounts being similar; selecting a first feature amount from the plurality of feature amounts based on the degree of match; A selection method characterized by causing a computer to execute processing.

[0112] (Supplementary Note 6) The selection method according to Supplementary Note 5, further comprising causing the computer to execute a process of registering the selected first feature amount and the reliability in a storage device in association with each other.

[0113] (Appendix 7) The selection method described in Appendix 5, characterized in that the process of determining the degree of match is performed by calculating the similarity between a portion of the feature of the first image and a portion of the feature of the second image, and performing a process of determining the label of the second image that has the greatest similarity to the portion of the feature of the first image for each first image with a different label.

[0114] (Appendix 8) The selection method described in Appendix 6, characterized in that when a third image is acquired, the selection method further causes the computer to execute a process of identifying a first feature corresponding to the reliability of each joint position of the third image based on the information registered in the storage device, and extracting a label of a first image similar to the third image based on the identified first feature.

[0115] (Supplementary Note 9) A generation unit that acquires learning data consisting of labels related to a person and images in which parts of the person are hidden, the learning data having a first image and a second image of the same person, generates skeletal information from the first image and the second image included in the learning data, in which multiple joint positions of the person are associated with reliability of each joint position, and generates multiple feature amounts for the first image and the second image based on the skeletal information; a selection unit that uses some of the feature amounts included in the plurality of feature amounts to identify a degree of agreement between a label of a first image and a label of a second image, the some of which are similar to the feature amounts, and selects a first feature amount from the plurality of feature amounts based on the degree of agreement; An information processing device comprising:

[0116] (Supplementary Note 10) The information processing device according to Supplementary Note 9, wherein the selection unit further executes a process of registering the selected first feature amount and the reliability in association with each other in a storage device.

[0117] (Appendix 11) The information processing device described in Appendix 9 is characterized in that the selection unit calculates the similarity between a portion of the feature of the first image and a portion of the feature of the second image, and identifies the label of the second image that has the greatest similarity to the portion of the feature of the first image, by performing this process for each first image with a different label, thereby identifying the degree of match.

[0118] (Appendix 12) The information processing device according to appendix 10, further comprising a matching unit that, when a third image is acquired, identifies a first feature corresponding to the reliability of each joint position of the third image based on information registered in the storage device, and extracts a label of a first image similar to the third image based on the identified first feature. [Explanation of symbols]

[0119] 50 training data 51 Learning Model 52 Feature definition table 60 video buffers 61 Decision Table 100 Information processing device 110 Communications Department 120 Input section 130 Display section 140 Storage section 150 control section 151 Acquisition Department 152 Generation part 153 Selection Department 154 Matching Unit

Claims

1. acquiring training data including a label relating to a person and an image in which a body part of the person is hidden, the training data including a first image and a second image of the same person; generating skeletal information in which a plurality of joint positions of the person are associated with reliability of each joint position from the first image and the second image included in the learning data; generating a plurality of feature amounts for the first image and the second image based on the skeleton information; Identifying, as an occlusion part, a joint position whose reliability is less than a threshold value among a plurality of joint positions of the person included in the first image; using a portion of the plurality of feature amounts, a degree of agreement between a label of a first image and a label of a second image, the portion of the feature amounts being similar; The part of the feature amounts with the highest degree of match is selected as the feature amounts of the occlusion part. A selection program that causes a computer to execute a process.

2. 2. The selection program according to claim 1, further causing the computer to execute a process of registering the selected feature quantity and the reliability in a storage device in association with each other.

3. The selection program according to claim 1, characterized in that the process of determining the degree of match calculates a similarity between a portion of a feature of the first image and a portion of a feature of the second image, and determines the label of the second image that has the greatest similarity to the portion of the feature of the first image by performing this process for each first image with a different label.

4. The selection program according to claim 2, further comprising causing the computer to execute a process of, when a third image is acquired, identifying a first feature corresponding to the reliability of each joint position of the third image based on the information registered in the storage device, and extracting a label of a first image similar to the third image based on the identified first feature.

5. acquiring training data including a label relating to a person and an image in which a body part of the person is hidden, the training data including a first image and a second image of the same person; generating skeletal information in which a plurality of joint positions of the person are associated with reliability of each joint position from the first image and the second image included in the learning data; generating a plurality of feature amounts for the first image and the second image based on the skeleton information; Identifying, as an occlusion part, a joint position whose reliability is less than a threshold value among a plurality of joint positions of the person included in the first image; using a portion of the plurality of feature amounts, a degree of agreement between a label of a first image and a label of a second image, the portion of the feature amounts being similar; The part of the feature amounts with the highest degree of match is selected as the feature amounts of the occlusion part. A selection method characterized by causing a computer to execute processing.

6. a generation unit that acquires learning data including a first image and a second image of the same person, the learning data being made up of labels relating to a person and an image in which parts of the person are hidden, and generates skeletal information from the first image and the second image included in the learning data, in which multiple joint positions of the person are associated with reliability of each joint position, and generates multiple feature amounts for the first image and the second image based on the skeletal information; a selection unit that identifies, from among a plurality of joint positions of the person included in the first image, a joint position for which the reliability is less than a threshold as an occlusion region, and that uses a portion of the plurality of feature amounts to identify a degree of agreement between a label of the first image and a label of the second image for which the portion of feature amounts is similar, and selects the portion of feature amount with the greatest degree of agreement as the feature amount of the occlusion region; An information processing device comprising:

Images