Specific program, specific method, and information processing device

Abstract

To improve the accuracy of detecting an image of a person to be identified.SOLUTION: An information processing device 100 acquires a plurality of first images captured by a plurality of cameras that have different depression angles of images to be captured and different installation locations and a second image including a person to be identified. The information processing device 100 generates a first feature quantity related to joints of the person from the second image. The information processing device 100 generates a second feature quantity used for matching the person to be identified based on information on the depression angles of the cameras and the first feature quantity. The information processing device 100 identifies an image having the person to be identified from the plurality of first images captured by the plurality of cameras based on the second feature quantity.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a specific program and the like. [Background technology]

[0002] In various investigations, images (video) from cameras installed in multiple locations are used. For example, when a crime occurs, police officers may use a photo of the perpetrator as a clue to check the images from the surveillance cameras to see if the perpetrator is captured in the images. Police officers also use information on the locations of the cameras that captured the perpetrator to identify the perpetrator's walking path, which helps with the investigation.

[0003] There is a conventional technology that extracts features such as a person's skeletal structure from images taken by cameras installed in multiple locations, and compares the extracted features with the features of the person under investigation to detect an image of a specific person. By using such a conventional technology, the burden of the verification work can be reduced. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2018-5555 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, when matching is performed using predetermined feature amounts as in the above-described conventional technology, there is a problem in that the accuracy of detecting an image of a person to be identified decreases.

[0006] Each camera is installed by a different installer, such as a local government, an individual, or the police, and the camera's depression angle varies. For example, if a camera is installed in a convenience store, the depression angle of the camera is set so that the camera's coverage area includes the interior and front of the store. On the other hand, if a local government or police installs a camera in a busy shopping district, the depression angle of the camera is adjusted so that the entire shopping district can be viewed from a crime prevention perspective.

[0007] Here, when the depression angle of the camera changes, the way a person is viewed changes. Figure 17 is a diagram for explaining how a person is viewed depending on the depression angle of the camera. In Figure 17, image 3 is an image taken by the first camera, which is installed at a height of 1 m and has a shallow depression angle. Image 4 is an image taken by the second camera, which is installed at a height of 3 m and has a deep depression angle.

[0008] Image 3 in Figure 17 shows people 3a, 3b, and 3c photographed by the first camera at different times n, n+1, and n+2. People 3a to 3c are the same person. For example, person 3a in image 3 is a person photographed at time n. Person 3b in image 3 is a person photographed at time n+1. Person 3c in image 3 is a person photographed at time n+2. When the depression angle is shallow, people 3a, 3b, and 3c are always photographed from the side, so the angle of their necks remains constant.

[0009] Image 4 in FIG. 17 shows people 4a, 4b, and 4c photographed by the second camera at different times n, n+1, and n+2. People 4a to 4c are the same person. For example, person 4a in image 4 is a person photographed at time n. Person 4b in image 4 is a person photographed at time n+1. Person 4c in image 4 is a person photographed at time n+2. Compared to person 4b, more of the front of person 4a's body is captured, so the neck angle of person 4a is larger than that of person 4b. On the other hand, more of the back of person 4c's body is captured, so the neck angle of person 4c is smaller than that of person 4b.

[0010] As explained in Figure 17, the feature amount of a person changes depending on the depression angle of the camera, so if you try to match images taken with cameras with different depression angles, the matching accuracy will decrease. For example, if you try to match the person in image 3 with the person in image 4 using the neck angle as a feature amount, the feature amount may not match even if they are the same person.

[0011] In one aspect, the present invention aims to provide an identification program, an identification method, and an information processing device that can improve the accuracy of detecting an image of a person to be identified. [Means for solving the problem]

[0012] In the first proposal, the computer is caused to execute the following process. The computer acquires a plurality of first images captured by a plurality of cameras installed at different locations and with different depression angles for capturing the images, and a second image including a person to be identified. The computer generates a first feature amount related to the person's joints from the second image. The computer generates a second feature amount used to match the person to be identified based on the information about the depression angles of the cameras and the first feature amount. The computer identifies an image including the person to be identified from the plurality of first images captured by the plurality of cameras based on the second feature amount. [Effects of the Invention]

[0013] This can improve the accuracy of detecting an image of a person to be identified. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram illustrating an example of a system according to the present embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of learning data. [Figure 3] FIG. 3 is a diagram (1) for explaining the pre-processing. [Figure 4] FIG. 4 is a diagram (2) for explaining the pre-processing. [Figure 5]FIG. 5 is a diagram (3) for explaining the pre-processing. [Figure 6] FIG. 6 is a diagram illustrating an example of the data structure of the feature definition table. [Figure 7] FIG. 7 is a diagram illustrating an example of the data structure of the camera management table. [Figure 8] FIG. 8 is a diagram showing a comparative example of matching accuracy. [Figure 9] FIG. 9 is a functional block diagram illustrating the configuration of an information processing device according to this embodiment. [Figure 10] FIG. 10 is a diagram for explaining feature amounts obtained from one image frame. [Figure 11] FIG. 11 is a diagram for explaining feature amounts obtained from a plurality of image frames. [Figure 12] FIG. 12 is a diagram illustrating an example of the data structure of a feature vector. [Figure 13] FIG. 13 is a diagram for explaining the processing of the matching unit. [Figure 14] FIG. 14 is a flowchart showing the procedure of the pre-processing of the information processing device. [Figure 15] FIG. 15 is a flowchart showing the processing procedure of the collation process of the information processing device. [Figure 16] FIG. 16 is a diagram illustrating an example of a hardware configuration of a computer that realizes the same functions as the information processing apparatus of the embodiment. [Figure 17] FIG. 17 is a diagram for explaining how a person appears depending on the depression angle of the camera. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, embodiments of the identification program, identification method, and information processing device disclosed in the present application will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to these embodiments. [Example]

[0016] Fig. 1 is a diagram showing an example of a system according to this embodiment. As shown in Fig. 1, this system 1 includes cameras 10a, 10b, 10c, and 10d, and an information processing device 100. The cameras 10a to 10d and the information processing device 100 are connected to each other via a network 30. Although the example shown in Fig. 1 shows only the cameras 10a to 10d, the system according to this embodiment may include other cameras.

[0017] The cameras 10a to 10d capture images within their capture range and transmit the captured image data to the information processing device 100. In the following description, the cameras 10a to 10d will be referred to as "camera 10" unless otherwise distinguished. The image data transmitted by the camera 10 to the information processing device 100 will be referred to as image data. In this embodiment, the description will be given using image data of a person. The image data is assigned with identification information of the camera 10 that captured the image data. The identification information of the camera 10 will be referred to as "camera identification information."

[0018] The video data includes a plurality of image frames in a time series. Each image frame is assigned a frame number in ascending chronological order. One image frame is a still image captured by the camera 10 at a certain timing.

[0019] The information processing device 100 is a device that acquires video data from the camera 10 and performs person matching. For example, the information processing device 100 performs pre-processing and matching processing. The pre-processing and matching processing will be described below in this order.

[0020] An example of "pre-processing" performed by the information processing device 100 will be described. The information processing device 100 performs the pre-processing using training data prepared in advance. FIG. 2 is a diagram showing an example of the training data. As shown in FIG. 2, the training data 50 includes a database 50a and a query 50b.

[0021] The database 50a associates camera identification information, correct labels, and video data. The camera identification information is information that identifies the camera 10. For example, the camera identification information for cameras 10a, 10b, 10c, and 10d are C10a, C10b, C10c, and C10d, respectively. The correct label is information that uniquely identifies a person included in the video data. The video data is video data that was previously captured by the camera 10 (or another camera).

[0022] In the example shown in FIG. 2, camera identification information "C10a", a correct label "X", and video data 51a are associated with each other. Camera identification information "C10b", a correct label "Y", and video data 52a are associated with each other. Camera identification information "C10c", a correct label "Z", and video data 53a are associated with each other. Camera identification information "C10d", a correct label "W", and video data 54a are associated with each other. For example, video data 51a was captured by camera 10a, and it is shown that a person identified by the correct label X is captured in the video data 51a.

[0023] Query 50b associates a correct label, video data, and a depression angle. The correct label is information that uniquely identifies a person included in the video data. The video data is video data that was previously captured by camera 10 (or another camera). The depression angle indicates the depression angle of the camera that captured the corresponding video data.

[0024] In the example shown in FIG. 2, the correct label "X" is associated with video data 51b and a depression angle of "40°." The correct label "Y" is associated with video data 52b and a depression angle of "40°." The correct label "Z" is associated with video data 53b and a depression angle of "40°." The correct label "W" is associated with video data 54b and a depression angle of "40°." For example, video data 51b is shown to have been captured by a camera with a depression angle of "50°" and to contain a person identified by the correct label X.

[0025] Here, since the correct answer label of video data 51a and video data 51b is the same, "X," the person shown in video data 51a and the person shown in video data 51b are the same person. Note that video data 51a and video data 51b are video data of the corresponding person captured at different times or in different places. In other words, video data 51a is not a copy of video data 51b.

[0026] The information processing device 100 executes the following processing for each piece of video data registered in the learning data 50. The information processing device 100 inputs each image frame included in the video data into a machine-learned learning model such as OpenPose, thereby detecting skeletal information of a person included in each image frame.

[0027] FIG. 3 is a diagram (1) for explaining pre-processing. The information processing device 100 detects skeletal information 20b by inputting an image frame 20a into a learning model. The skeletal information 20b includes two-dimensional coordinates (x, y) of multiple joints of a person. The multiple joints include the nose, neck, left and right eyes, ears, shoulders, elbows, wrists, buttocks, knees, ankles, etc. In the following description, the two-dimensional coordinates of the joints will be referred to as "joint coordinates."

[0028] The information processing device 100 detects skeletal information for each image frame included in the video data, and identifies multiple feature amounts based on the joint coordinates and the transition of the joint coordinates included in the skeletal information. The multiple feature amounts include the height of the right arm, the height of the left arm, the height of the right knee, the height of the left knee, the maximum stride length, the time the feet are in the air, the angles of each joint, and the line of sight while walking. In the following description, the multiple feature amounts will be referred to as a "feature vector."

[0029] The information processing device 100 repeatedly executes the above process for each piece of video data registered in the database 50a and the query 50b, and identifies a feature vector for each piece of video data.

[0030] The information processing device 100 selects an optimal combination of feature amounts for each depression angle by performing the following process. FIG. 4 is a diagram (2) for explaining the pre-processing. The information processing device 100 defines in advance a plurality of partial feature amounts with modified combinations from among all feature amounts included in the feature amount vector. The number of feature amounts (types of feature amounts) included in the partial feature amount is assumed to be less than the number of feature amounts in the feature amount vector.

[0031] In the example shown in FIG. 4, the feature vector includes the right arm height, left arm height, right knee height, left knee height, maximum stride length, and time the feet are in the air. Partial feature A includes the left arm height, right arm height, and so on. Partial feature B includes the maximum stride length and time the feet are in the air, and partial feature C includes the left knee height, right knee height, and so on. In the example shown in FIG. 4, partial features A, B, and C are shown, but other partial features can also be defined.

[0032] FIG. 5 is a diagram (3) for explaining the pre-processing. The information processing device 100 determines the degree of match between the labels of each partial feature for each depression angle, and determines the partial feature with the highest degree of match between the labels as the partial feature for the corresponding depression angle. In the following description, in order to distinguish between the partial feature determined from the video data of the database 50a and the partial feature determined from the video data of the query 50b, the partial feature determined from the video data of the query 50b will be referred to as the "query partial feature." In the description of FIG. 5, the depression angle of the video data 51b to 53b of the query 50b will be described as "40°."

[0033] The information processing device 100 extracts the value of the partial feature amount A from the feature amount vectors of the video data 51a to 54a in the database 50a. The information processing device 100 extracts the value of the query partial feature amount A from the feature amount vectors of the video data 51b to 53b in the query 50b.

[0034] The information processing device 100 calculates the similarity (cosine similarity) between the partial feature A of each of the video data 51a to 54a and the query partial feature A of the video data 51b, and identifies the correct label of the video data corresponding to the partial feature A that has the maximum similarity to the query partial feature. For example, if the similarity between the partial feature A of the video data 51a and the query partial feature A of the video data 51b is the maximum similarity, the correct label is "X", which is the same.

[0035] The information processing device 100 calculates the similarity between the partial feature A of each of the video data 51a to 54a and the query partial feature A of the video data 52b, and identifies the correct label of the video data corresponding to the partial feature A that has the maximum similarity to the query partial feature. For example, if the similarity between the partial feature A of the video data 52a and the query partial feature A of the video data 52b is the maximum similarity, the correct labels will not match (X and Y do not match).

[0036] The information processing device 100 calculates the similarity between the partial feature A of each of the video data 51a to 54a and the query partial feature A of the video data 53b, and identifies the correct label of the video data corresponding to the partial feature A that has the maximum similarity to the query partial feature. For example, if the similarity between the partial feature A of the video data 51a and the query partial feature A of the video data 51b is the maximum similarity, the correct label is "X", which is the same.

[0037] In the above example, when the information processing device 100 performs a search using the partial feature amount A for video data with a depression angle of "40°," the degree of match of the labels is "2 / 3."

[0038] As with partial feature A, the information processing apparatus 100 executes a search using partial feature B, C, and other partial feature for the video data with a depression angle of 40°, and identifies the degree of match of the labels.

[0039] The information processing device 100 compares the degree of match of the labels for each partial feature and identifies the partial feature with the highest degree of match of the labels as the partial feature of the corresponding depression angle. For example, if the degree of match of the labels when partial feature A is used for a depression angle of 40° is highest, the information processing device 100 identifies the partial feature of the depression angle of 40° as partial feature A.

[0040] The information processing device 100 identifies the partial feature amounts of other depression angles by performing the same process as described above for the other depression angles. Based on the identification results, the information processing device 100 registers the relationship between the depression angle and the partial feature amount in the feature amount definition table.

[0041] Fig. 6 is a diagram showing an example of the data structure of a feature quantity definition table. As shown in Fig. 6, this feature quantity definition table 52 associates depression angles, the number of feature quantities, and combinations of feature quantities. The depression angle indicates the depression angle of the camera that captured the video data. The number of feature quantities is the number of feature quantities included in the partial feature quantity corresponding to the depression angle. The feature quantity combination indicates the combination of each feature quantity that makes up the partial feature quantity.

[0042] Next, an example of the "matching process" executed by the information processing device 100 will be described. In the description of the matching process, video data showing a person to be identified will be used. The video data showing a person to be identified will be referred to as "matching target video data" as appropriate. The matching target video data is assigned camera identification information of the camera 10 that captured the matching target video data.

[0043] The information processing device 100 inputs each image frame of the video data to be matched into a learning model, thereby identifying each piece of skeletal information.

[0044] The information processing device 100 compares the camera identification information with the camera management table to identify the depression angle of the camera 10 that captured the video data to be matched. FIG. 7 is a diagram showing an example of the data structure of the camera management table. As shown in FIG. 7, this camera management table 53 associates camera identification information, depression angle, and installation position (latitude, longitude). The camera identification information is information that identifies a camera. The depression angle indicates the depression angle of the corresponding camera. The installation position indicates the installation position (latitude, longitude) of the corresponding camera.

[0045] The information processing device 100 compares the identified depression angle of the camera 10 with the feature definition table 52 to identify a combination of feature amounts corresponding to the identified depression angle. In the following description, the identified combination of feature amounts will be referred to as a "specific feature amount."

[0046] The information processing device 100 identifies a feature vector based on the coordinates of each joint included in the skeletal information obtained from the video data to be matched and the transition of the joint information. The information processing device 100 extracts a specific feature from the feature vector of the video data to be matched. In the following description, the specific feature extracted from the feature vector of the video data to be matched is referred to as a "first specific feature."

[0047] The information processing device 100 extracts specific features from the feature vectors of each piece of video data in the database 50a, which were identified in the pre-processing. In the following description, the specific features extracted from the feature vectors of each piece of video data in the database 50a are referred to as "second specific features." Camera identification information is attached to the second specific features.

[0048] The information processing device 100 calculates the cosine similarity between the first specific feature and each of the second specific features. The information processing device 100 identifies, as a matching result, each of the video data from which the second specific feature is extracted, whose cosine similarity with the first specific feature is equal to or greater than a threshold. Furthermore, the information processing device 100 identifies the installation positions of each camera that captured the identified person based on the camera identification information of the second specific feature whose cosine similarity with the first specific feature is equal to or greater than a threshold and the camera management table 53.

[0049] As described above, the information processing device 100 identifies a specific feature based on the depression angle of the camera 10 that captured the video data to be matched, which includes the person to be identified, and extracts a first specific feature from the video data to be matched. Based on the first specific feature and the second specific feature of each piece of video data in the database 50a, the information processing device 100 identifies video data with a second specific feature similar to the first specific feature. This improves the accuracy of detecting an image of the person to be identified. The person to be identified is a person related to a case handled by the police. For example, the person to be identified is a criminal.

[0050] FIG. 8 is a diagram showing a comparative example of matching accuracy. The horizontal axis of graph G1 in FIG. 8 corresponds to the depression angle of camera 10. The vertical axis corresponds to matching accuracy. Line segment l1 shows the relationship between the depression angle of the camera and matching accuracy according to the conventional technology. Line segment l2 shows the relationship between the depression angle of the camera and matching accuracy according to the present invention. As shown in FIG. 8, the present invention has improved matching accuracy when compared to the conventional technology at depression angles of 30° to 60°.

[0051] Next, an example of the configuration of the information processing device 100 according to this embodiment will be described. Fig. 9 is a functional block diagram showing the configuration of the information processing device according to this embodiment. As shown in Fig. 9, this information processing device 100 has a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.

[0052] The communication unit 110 is connected to the camera 10 via the network 30 and receives video data. For example, the communication unit 110 is realized by a network interface card (NIC) or the like. The communication unit 110 may also be connected to other external devices or the like via the network 30.

[0053] The input unit 120 is an input device that inputs various types of information to the information processing device 100. The input unit 120 corresponds to a keyboard, a mouse, a touch panel, etc. The user may operate the input unit 120 to specify the camera 10 that captured the video data to be matched, or may directly input the video data to be matched via an input interface.

[0054] The display unit 130 is a display device that displays information output from the control unit 150. The display unit 130 corresponds to a liquid crystal display, an organic EL (Electro Luminescence) display, a touch panel, or the like.

[0055] The storage unit 140 includes learning data 50, a learning model 51, a feature definition table 52, a camera management table 53, and a video buffer 60. The storage unit 140 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk.

[0056] The training data 50 is data used in pre-processing, and corresponds to the training data 50 described in Fig. 2. As described in Fig. 2, the training data 50 includes a database 50a and a query 50b.

[0057] The learning model 51 is a machine-learned learning model such as OpenPose. By inputting an image frame to the learning model 51, the joint coordinates of each joint and the reliability of each joint coordinate are output. For example, the learning model is a neural network (NN).

[0058] The feature quantity definition table 52 is a table that defines the optimal combination of feature quantities for each depression angle, and is generated in pre-processing. The data structure of the feature quantity definition table 52 is the same as that of the feature quantity definition table 52 described in FIG. 6.

[0059] The camera management table 53 is a table that holds information about the camera 10. The data structure of the camera management table 53 is the same as that of the camera management table 53 described in FIG.

[0060] The video buffer 60 is a buffer that holds video data (video data to be matched) acquired from the camera 10. Note that the video buffer 60 may hold video data to be matched that is directly input to the information processing device 100 from an input interface (not shown).

[0061] The control unit 150 includes an acquisition unit 151, a generation unit 152, a selection unit 153, and a matching unit 154. The control unit 150 is realized by, for example, a central processing unit (CPU) or a micro processing unit (MPU). The control unit 150 may also be implemented by, for example, an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0062] The acquisition unit 151 acquires the matching target video data from the camera 10 and registers the acquired matching target video data in the video buffer 60. The acquisition unit 151 may acquire the matching target video data that is directly input to the information processing device 100 from an input interface and register the acquired matching target video data in the video buffer 60. The acquisition unit 151 may acquire the learning data 50, the machine-learned learning model 51, the camera management table 53, etc. from an external device and register them in the storage unit 140.

[0063] The generation unit 152 generates feature vectors of the video data in the database 50a and the query 50b included in the training data 50. The processing by the generation unit 152 corresponds to the processing for generating feature vectors described in the pre-processing above. The generation unit 152 executes the following processing for each video data registered in the training data 50.

[0064] The generation unit 152 generates feature vectors of the video data in the database 50a and the query 50b included in the training data 50. The processing by the generation unit 152 corresponds to the processing for generating feature vectors described in the pre-processing above. The generation unit 152 executes the following processing for each video data registered in the training data 50.

[0065] The generation unit 152 detects skeletal information of a person included in each image frame by inputting each image frame included in the video data into the learning model 51. The skeletal information includes the joint coordinates of each joint.

[0066] The generation unit 152 generates a feature vector of the video data based on the joint coordinates included in the skeletal information and the transition of the joint coordinates. The feature vector includes feature vectors obtained from one image frame and feature vectors obtained from multiple image frames.

[0067] FIG. 10 is a diagram illustrating feature amounts obtained from one image frame. As shown in FIG. 10, feature amounts such as stride length f1, arm height f2, and knee height f3 are extracted from one image frame. The stride length f1 is the distance between the joint coordinates of the left and right ankles included in the skeletal information 10-1 of a certain image frame. The arm height f2 of the skeletal information 10-1 is the distance from the left ankle (right ankle) to the left wrist (right wrist). The knee height f3 of the skeletal information 10-1 is the distance from the left ankle (right ankle) to the left knee (right knee). For example, the generation unit 152 calculates the stride length f1 from the skeletal information of each image frame included in the video data and calculates the average value of each stride length f1 to calculate the feature amount of the stride length. The same applies to the feature amounts related to the arm height and knee height.

[0068] FIG. 11 is a diagram illustrating feature quantities obtained from multiple image frames. Here, the description will be given using skeletal information 11-1, 11-2, and 11-3. Skeletal information 11-1 is skeletal information extracted from the image frame with frame number n-2. Skeletal information 11-2 is skeletal information extracted from the image frame with frame number n-1. Skeletal information 11-3 is skeletal information extracted from the image frame with frame number n. For example, if the right ankle starts floating in frame number n-2 and lands on the ground in frame number n, the generation unit 152 calculates the time from frame number n-2 to frame number n as the floating time of the foot. The generation unit 152 calculates the floating time for the video data multiple times and calculates the maximum (or average) value of each floating time, thereby calculating the feature quantity of the floating time.

[0069] The generation unit 152 calculates other feature amounts (height of each joint, joint angle, transition of each joint, line of sight while walking, etc.) using conventional technology.

[0070] Fig. 12 is a diagram showing an example of the data structure of a feature vector. As shown in Fig. 12, a value is set for each feature (type of feature) in the feature vector.

[0071] The generation unit 152 repeatedly executes the above process for each piece of video data registered in the database 50a and the query 50b, and identifies a feature vector for each piece of video data. Although not shown, the generation unit 152 registers the feature vector for each piece of video data in the database 50a and the query 50b in FIG. 2.

[0072] The selection unit 153 identifies the degree of match of the labels of each partial feature for each depression angle based on the learning data 50, and selects the partial feature with the highest degree of match of the labels as the partial feature for the corresponding depression angle. The selection unit 153 registers the selection result in the feature definition table 52. The processing of the selection unit 153 corresponds to the pre-processing described in FIG. 5. As with the description of FIG. 5, the selection unit 153 will be described assuming that the depression angle of the video data 51b to 53b of the query 50b is "40°."

[0073] The selection unit 153 extracts the value of the partial feature amount A from the feature amount vectors of the video data 51a to 54a in the database 50a. The selection unit 153 extracts the value of the query partial feature amount A from the feature amount vectors of the video data 51b to 53b in the query 50b.

[0074] The selection unit 153 calculates the similarity (cosine similarity) between the partial feature A of each of the video data 51a to 54a and the query partial feature A of the video data 51b, and identifies the correct label of the video data corresponding to the partial feature A that has the maximum similarity to the query partial feature. For example, if the similarity between the partial feature A of the video data 51a and the query partial feature A of the video data 51b is the maximum similarity, the correct label is "X", which is the same.

[0075] The selection unit 153 calculates the similarity between the partial feature A of each of the video data 51a to 54a and the query partial feature A of the video data 52b, and identifies the correct label of the video data corresponding to the partial feature A that has the maximum similarity to the query partial feature. For example, if the similarity between the partial feature A of the video data 52a and the query partial feature A of the video data 52b is set to be the maximum similarity, the correct labels will not match (X and Y do not match).

[0076] The selection unit 153 calculates the similarity between the partial feature A of each of the video data 51a to 54a and the query partial feature A of the video data 53b, and identifies the correct label of the video data corresponding to the partial feature A that has the maximum similarity to the query partial feature. For example, if the similarity between the partial feature A of the video data 51a and the query partial feature A of the video data 53b is the maximum similarity, the correct label is "X", which is the same.

[0077] In the above example, when the selection unit 153 searches for video data with a depression angle of "40°" using the partial feature amount A, the degree of match of the label is "2 / 3".

[0078] As with partial feature A, the selection unit 153 executes a search using partial feature B, C, and other partial feature for the video data with a depression angle of 40°, and identifies the degree of match of the labels.

[0079] The selection unit 153 compares the degree of match of the labels for each partial feature and identifies the partial feature with the highest degree of match of the labels as the partial feature of the depression angle. For example, if the degree of match of the labels when partial feature A is used for a depression angle of 40° is the highest, the selection unit 153 selects the partial feature of the depression angle of 40° as partial feature A.

[0080] The selection unit 153 executes the same process as above for other depression angles to identify partial feature amounts for the other depression angles. Based on the identification results, the selection unit 153 registers the relationship between the depression angles and partial feature amounts in the feature amount definition table 52.

[0081] The matching unit 154 matches the video data to be matched with the video data in the database 50a of the learning data 50, thereby identifying video data in which a person similar to the person appearing in the video data to be matched is shown. The processing of the matching unit 154 corresponds to the above-mentioned matching processing.

[0082] 13 is a diagram for explaining the processing of the matching unit. Matching unit 154 inputs each image frame of matching target video data 40 registered in video buffer 60 to learning model 51 and identifies each skeletal information. For example, the depression angle of the camera that captured matching target video data 40 shown in FIG. 13 is assumed to be "40°."

[0083] Based on the camera identification information assigned to the video data to be compared and the camera management table 53, the comparison unit 154 identifies the depression angle of the camera that captured the video data to be compared.

[0084] The collation unit 154 compares the identified depression angle with the feature definition table 52 to identify a combination of features (specific feature) corresponding to the identified depression angle. For example, if the depression angle is "40°," the specific feature will be "the height of the left arm, the angle of the right arm, etc."

[0085] The matching unit 154 identifies a feature vector based on the coordinates of each joint included in the skeletal information obtained from the video data to be matched and the transition of the joint information. The process by which the matching unit 154 identifies a feature vector is similar to the process by which the generation unit 152 identifies a feature vector. The matching unit 154 extracts a first specific feature from the feature vector of the video data to be matched. In the example shown in FIG. 13, the matching unit 154 extracts a first specific feature 40a from the video data 40 to be matched.

[0086] The matching unit 154 extracts second specific features from the feature vectors of each video data in the database 50a identified in the pre-processing. In the example shown in Fig. 13, the matching unit 154 extracts second specific features 45b from the feature vector of video data 45.

[0087] The matching unit 154 calculates the cosine similarity between the first specific feature 40a and the second specific feature 45a. The matching unit 154 calculates the cosine similarity between the first specific feature 40a and a second specific feature extracted from other video data. The matching unit 154 extracts, as a matching result, video data from which the second specific feature was extracted, whose cosine similarity with the first specific feature is equal to or greater than a threshold. The matching unit 154 outputs the matching result to the display unit 130 for display.

[0088] Furthermore, the matching unit 154 identifies the installation positions of the cameras that captured each piece of video data, based on the camera identification information of the video data from which the second specific feature is extracted, the second specific feature having a cosine similarity with the first specific feature equal to or greater than a threshold, and the camera management table 53. The matching unit 154 outputs the identified installation positions of the cameras to the display unit 130 as a movement trajectory of the person to be identified, and causes the display unit 130 to display the same.

[0089] Next, an example of a processing procedure of the information processing device 100 according to this embodiment will be described. Fig. 14 is a flowchart showing the processing procedure of pre-processing of the information processing device. As shown in Fig. 14, the generation unit 152 of the information processing device 100 acquires video data from the learning data 50 (step S101).

[0090] The generation unit 152 extracts skeletal information from each image frame of the video data using the learning model 51 (step S102). The generation unit 152 identifies the depression angle based on the camera identification information of the camera that captured the video data and the camera management table 53 (step S103). The generation unit 152 calculates a feature vector of the video data (step S104).

[0091] The selection unit 153 of the information processing device 100 identifies the degree of match of the labels for each depression angle and each partial feature (step S105). The selection unit 153 identifies the partial feature corresponding to the depression angle based on the degree of match of the labels (step S106).

[0092] The selection unit 153 registers the relationship between the depression angle and the identified partial feature amount (combination of feature amounts) in the feature amount definition table 52 (step S107).

[0093] 15 is a flowchart showing the processing procedure of the matching process of the information processing device. As shown in Fig. 15, the acquisition unit 151 of the information processing device 100 acquires the matching target video data from the camera 10 or the input interface, and registers it in the video buffer 60 (step S201).

[0094] Matching unit 154 of information processing device 100 extracts skeletal information from each image frame of the video data to be matched using learning model 51 (step S202). Matching unit 154 identifies the depression angle based on the camera identification information of the camera that captured the video data to be matched and camera management table 53 (step S203).

[0095] The matching unit 154 identifies a specific feature based on the identified depression angle and feature definition table 52 (step S204). The matching unit 154 calculates the similarity between the first specific feature and each of the second specific feature (step S205).

[0096] The matching unit 154 extracts from the database 50a video data of a second specific feature whose similarity to the first specific feature is equal to or greater than a threshold (step S206). The matching unit 154 identifies the movement trajectory of the person to be identified based on the installation positions of the cameras that captured the video data of the second specific feature whose similarity to the first specific feature is equal to or greater than a threshold (step S207). The matching unit 154 outputs the matching result to the display unit 130 (step S208).

[0097] Next, the effects of the information processing device 100 according to this embodiment will be described. The information processing device 100 identifies a specific feature based on the depression angle of the camera 10 that captured the video data to be matched, which includes the person to be identified, and extracts a first specific feature from the video data to be matched. The information processing device 100 identifies video data with a second specific feature similar to the first specific feature based on the first specific feature and the second specific feature of each video data in the database 50a. This can improve the accuracy of detecting an image of the person to be identified.

[0098] The information processing device 100 generates a movement trajectory of the specified person based on the installation position of the camera that captured the video data of the second specific feature whose similarity to the first specific feature is equal to or greater than a threshold value. This makes it possible to easily identify the locations where the specified person has moved, which can be useful in investigations.

[0099] The information processing device 100 identifies the degree of match of the labels of each partial feature for each depression angle based on the video data registered in the learning data 50, and selects the partial feature corresponding to the depression angle based on the degree of match of the labels. This makes it possible to select a combination of features according to the depression angle.

[0100] Next, an example of the hardware configuration of a computer that realizes the same functions as the information processing device 100 described in the above embodiment will be described. Fig. 16 is a diagram showing an example of the hardware configuration of a computer that realizes the same functions as the information processing device of the embodiment.

[0101] 16, the computer 200 has a CPU 201 that executes various types of arithmetic processing, an input device 202 that accepts data input from a user, and a display 203. The computer 200 also has a communication device 204 that exchanges data with the camera 10, external devices, etc. via a wired or wireless network, and an interface device 205. The computer 200 also has a RAM 206 that temporarily stores various types of information, and a hard disk drive 207. The devices 201 to 207 are connected to a bus 208.

[0102] The hard disk device 207 stores an acquisition program 207a, a generation program 207b, a selection program 207c, and a matching program 207d. The CPU 201 reads out each of the programs 207a to 207d and loads them into the RAM 206.

[0103] The acquisition program 207a functions as the acquisition process 206a. The generation program 207b functions as the generation process 206b. The selection program 207c functions as the selection process 206c. The matching program 207d functions as the matching process 206d.

[0104] The processing of the acquisition process 206a corresponds to the processing of the acquisition unit 151. The processing of the generation process 206b corresponds to the processing of the generation unit 152. The processing of the selection process 206c corresponds to the processing of the selection unit 153. The processing of the matching process 206d corresponds to the processing of the matching unit 154.

[0105] It should be noted that each of the programs 207a to 207d does not necessarily have to be stored in the hard disk drive 207 from the beginning. For example, each of the programs may be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, or IC card that is inserted into the computer 200. Then, the computer 200 may read and execute each of the programs 207a to 207d.

[0106] The following supplementary notes are further disclosed regarding the embodiments including the above examples.

[0107] (Appendix 1) A plurality of first images taken by a plurality of cameras installed at different locations and with different depression angles for the images taken, and a second image including a person to be identified are acquired, generating a first feature amount relating to a joint of a person from the second image; generating a second feature amount to be used for matching the person to be identified based on information about the depression angle of the camera and the first feature amount; Identifying an image including the person to be identified from the plurality of first images captured by the plurality of cameras based on the second feature amount. A specific program that causes a computer to execute a process.

[0108] (Appendix 2) The specific program described in Appendix 1 is characterized in that it further causes a computer to execute the following process: extracting the first feature from the plurality of first images to which labels relating to people included in the captured images have been assigned, and from a plurality of third images to which the labels and a certain depression angle have been assigned; generating a plurality of types of second feature from the first feature; and selecting the second feature to be used at the certain depression angle based on the degree of agreement between the label of the first image and the label of the third image to which the second feature is similar.

[0109] (Appendix 3) The identification program described in Appendix 1 is characterized in that the identification process calculates the similarity between second features corresponding to the plurality of first images and second features corresponding to the second images, and identifies a first image having second features whose similarity with the second features corresponding to the second images is equal to or greater than a threshold as an image containing the person to be identified.

[0110] (Appendix 4) The identification program described in Appendix 3, characterized in that the identification process further identifies the movement trajectory of the person to be identified based on the location of a camera that captured a first image of a second feature whose similarity with the second feature corresponding to the second image is equal to or greater than a threshold.

[0111] (Appendix 5) The identification program described in Appendix 1, characterized in that the person to be identified is a person related to a case handled by the police.

[0112] (Appendix 6) A plurality of first images taken by a plurality of cameras installed at different locations and with different depression angles for the images taken, and a second image including a person to be identified are acquired, generating a first feature amount relating to a joint of a person from the second image; generating a second feature amount to be used for matching the person to be identified based on information about the depression angle of the camera and the first feature amount; Identifying an image including the person to be identified from the plurality of first images captured by the plurality of cameras based on the second feature amount. A method for identifying a target object, characterized in that the processing is carried out by a computer.

[0113] (Appendix 7) The identification method described in Appendix 6, characterized in that the computer further executes the process of extracting the first feature from the plurality of first images to which labels relating to people included in the captured images have been assigned and from a plurality of third images to which the labels and a certain depression angle have been assigned, generating a plurality of types of second feature from the first feature, and selecting the second feature to be used at the certain depression angle based on the degree of agreement between the label of the first image and the label of the third image to which the second feature is similar.

[0114] (Appendix 8) The identification method described in Appendix 6, characterized in that the identification process calculates the similarity between second feature amounts corresponding to the plurality of first images and second feature amounts corresponding to the second images, and identifies a first image having second feature amounts whose similarity with the second feature amount corresponding to the second image is equal to or greater than a threshold value as an image containing the person to be identified.

[0115] (Appendix 9) The identification method described in Appendix 8, characterized in that the identification process further identifies the movement trajectory of the person to be identified based on the location of a camera that captured a first image of a second feature whose similarity with the second feature corresponding to the second image is equal to or greater than a threshold.

[0116] (Appendix 10) The identification method described in Appendix 6, characterized in that the person to be identified is a person related to a case handled by the police.

[0117] (Appendix 11) An acquisition unit that acquires a plurality of first images taken by a plurality of cameras that have different depression angles of the images to be taken and are installed in different locations, and a second image including a person to be identified; a matching unit that generates a first feature amount related to the joints of the person from the second image, generates a second feature amount to be used for matching the person to be identified based on information about the depression angle of the camera and the first feature amount, and identifies an image including the person to be identified from the plurality of first images taken by the plurality of cameras based on the second feature amount; An information processing device comprising:

[0118] (Appendix 12) The information processing device according to appendix 11, further comprising a selection unit that extracts the first feature from the plurality of first images to which labels relating to people included in the captured images have been assigned and a plurality of third images to which the labels and a certain depression angle have been assigned, generates a plurality of types of second feature from the first feature, and selects the second feature to be used at the certain depression angle based on the degree of agreement between the label of the first image and the label of the third image to which the second feature is similar.

[0119] (Appendix 13) The information processing device described in Appendix 11 is characterized in that the matching unit calculates the similarity between second features corresponding to the plurality of first images and second features corresponding to the second images, and identifies a first image having second features whose similarity with the second features corresponding to the second images is equal to or greater than a threshold value as an image containing the person to be identified.

[0120] (Appendix 14) The information processing device described in Appendix 13, characterized in that the matching unit further identifies the movement trajectory of the person to be identified based on the location of a camera that captured a first image of a second feature whose similarity with the second feature corresponding to the second image is greater than or equal to a threshold.

[0121] (Appendix 15) The information processing device according to appendix 11, wherein the person to be identified is a person related to a case handled by the police. [Explanation of symbols]

[0122] 50 training data 51 Learning Model 52 Feature definition table 53 Camera Management Table 60 video buffers 100 Information processing device 110 Communications Department 120 Input section 130 Display section 140 Storage section 150 control section 151 Acquisition Department 152 Generation part 153 Selection Department 154 Matching Unit

Claims

1. A plurality of first images are acquired which are taken by a plurality of cameras which are installed at different locations and have different depression angles for the images taken, and a second image which includes the person to be identified; generating a plurality of partial feature amounts by combining feature amounts for each joint of the person from the second image, and generating a first feature amount from the plurality of partial feature amounts; generating a second feature amount to be used for matching the person to be identified based on information about the depression angle of the camera and the first feature amount; Identifying an image including the person to be identified from the plurality of first images captured by the plurality of cameras based on the second feature amount. A specific program that causes a computer to execute a process.

2. 2. The program according to claim 1, further causing a computer to execute the following process: extracting first features from the plurality of first images to which labels relating to people included in the captured images have been assigned and from a plurality of third images to which the labels and a certain depression angle have been assigned; generating a plurality of types of second features from the first features; and selecting a second feature to be used at the certain depression angle based on a degree of agreement between the label of a first image and the label of the third image to which the second features are similar.

3. The identification program according to claim 1, characterized in that the identification process calculates a similarity between second features corresponding to the plurality of first images and second features corresponding to the second images, and identifies a first image having second features whose similarity with the second features corresponding to the second images is equal to or greater than a threshold value as an image containing the person to be identified.

4. The identification program of claim 3, characterized in that the identification process further identifies the movement trajectory of the person to be identified based on the location of a camera that captured a first image of a second feature whose similarity to the second feature corresponding to the second image is greater than or equal to a threshold value.

5. 2. The identification program according to claim 1, wherein the person to be identified is a person related to a case handled by the police.

6. A plurality of first images are acquired which are taken by a plurality of cameras which are installed at different locations and have different depression angles for the images taken, and a second image which includes the person to be identified; generating a plurality of partial feature amounts by combining feature amounts for each joint of the person from the second image, and generating a first feature amount from the plurality of partial feature amounts; generating a second feature amount to be used for matching the person to be identified based on information about the depression angle of the camera and the first feature amount; Identifying an image including the person to be identified from the plurality of first images captured by the plurality of cameras based on the second feature amount. A method for identifying a target object, characterized in that the processing is carried out by a computer.

7. an acquisition unit that acquires a plurality of first images captured by a plurality of cameras that have different depression angles for the captured images and are installed in different locations, and a second image that includes the person to be identified; a matching unit that generates a plurality of partial feature amounts by combining feature amounts for each joint of the person from the second image, generates a first feature amount from the plurality of partial feature amounts, generates a second feature amount to be used for matching the person to be identified based on information about the depression angle of the camera and the first feature amount, and identifies an image including the person to be identified from the plurality of first images taken by the plurality of cameras based on the second feature amount; An information processing device comprising:

Images