Calculation device, calculation method, and calculation program

Abstract

To improve accuracy of authentication processing.SOLUTION: A calculation apparatus 10 estimates skeleton data with a skeleton estimation model for estimating skeleton data relating to a human skeleton by using image data as input. The calculation apparatus 10 calculates weight values for each joint on the basis of reliability of estimation results for each joint, and calculates similarity between skeleton data by use of the calculated weight values for each joint. The calculation apparatus 10 determines whether or not, the similarity is equal to or higher than a predetermined threshold, and determines authentication success when the similarity is equal to or higher than the predetermined threshold, and determines authentication failure when the similarity is lower than the predetermined threshold.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a calculation device, a calculation method, and a calculation program. [Background technology]

[0002] In recent years, various biometric authentication techniques have become known for authenticating individuals. For example, one such authentication technique estimates the posture of a person to be authenticated by estimating the joint positions of the person from image data including the entire body of the person, and performs identity authentication based on the similarity between the estimated posture and a posture registered in advance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-013999 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional personal authentication methods have the problem that the accuracy of the authentication process may be low. For example, in conventional personal authentication methods, it is difficult to estimate the location of some joints depending on the image, and the reliability of the estimation result varies depending on the location of each joint, resulting in a problem of reduced authentication accuracy. [Means for solving the problem]

[0005] In order to solve the above-mentioned problems and achieve the object, the calculation device of the present invention is characterized in having a first estimation unit that estimates skeletal data related to the skeleton of the person using a skeletal estimation model based on input image data including a person, a calculation unit that calculates a weight value of each joint based on the reliability of the estimation result of each joint, and a calculation unit that calculates a similarity between the skeletal data estimated by the first estimation unit and skeletal data estimated from specified image data using the weight value of each joint calculated by the calculation unit.

[0006] Furthermore, the calculation method of the present invention is a calculation method executed by a calculation device, and is characterized by including a first estimation step of estimating skeletal data using a skeletal estimation model that estimates skeletal data related to the skeleton of the person using image data including a person as input, a calculation step of calculating a weight value for each joint based on the reliability of the estimation result of each joint, and a calculation step of calculating a similarity between the skeletal data estimated by the first estimation step and skeletal data estimated from specified image data using the weight value for each joint calculated by the calculation step.

[0007] In addition, the calculation program of the present invention is characterized in that it causes a computer to execute the following steps: a first estimation step of estimating skeletal data using a skeletal estimation model that estimates skeletal data related to the skeleton of the person using image data including a person as input; a calculation step of calculating a weight value for each joint based on the reliability of the estimation result of each joint; and a calculation step of calculating a similarity between the skeletal data estimated by the first estimation step and skeletal data estimated from specified image data using the weight value for each joint calculated by the calculation step. Effect of the Invention

[0008] According to the present invention, it is possible to improve the accuracy of authentication processing. [Brief description of the drawings]

[0009] [Figure 1]FIG. 1 is a block diagram illustrating an example of the configuration of a calculation device according to the first embodiment. [Diagram 2] FIG. 2 is a diagram for explaining the process of calculating the similarity of skeletons. [Diagram 3] FIG. 3 is a diagram illustrating an overview of the authentication process performed by the calculation device according to the first embodiment. [Figure 4] FIG. 4 is a flowchart illustrating an example of a processing flow in the calculation device according to the first embodiment. [Diagram 5] FIG. 5 is a diagram illustrating an overview of authentication processing performed by the calculation device according to the second embodiment. [Figure 6] FIG. 6 is a diagram illustrating the variation in the estimated positions of the joints. [Figure 7] FIG. 7 is a diagram illustrating a computer that executes a calculation program. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, embodiments of a calculation device, a calculation method, and a calculation program according to the present application will be described in detail with reference to the drawings. Note that the calculation device, the calculation method, and the calculation program according to the present application are not limited to these embodiments.

[0011] [First embodiment] In the following embodiment, the configuration of the calculation device 10 according to the first embodiment and the processing flow of the calculation device 10 will be described in order, and finally, the effects of the first embodiment will be described.

[0012] [Configuration of the calculation device] First, the configuration of a calculation device 10 will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the configuration of a calculation device according to a first embodiment. The calculation device 10 is a device that performs personal authentication by acquiring image data of a person to be authenticated, estimating skeletal data of the person, and calculating a similarity with correct skeletal data.

[0013] Specifically, the calculation device 10 estimates skeletal data using a skeletal estimation model that estimates skeletal data related to a person's skeleton, with image data as input. The calculation device 10 also estimates skeletal data using the skeletal estimation model with image data previously stored in the storage unit 13 as input.

[0014] The calculation device 10 then calculates a weight value for each joint based on the reliability of the estimation result for each joint in the estimated skeletal data, and calculates the similarity between the skeletal data using the calculated weight value for each joint. The calculation device 10 then determines whether the similarity is equal to or greater than a predetermined threshold, and if the similarity is equal to or greater than the predetermined threshold, determines that the authentication has been successful, and if the similarity is less than the predetermined threshold, determines that the authentication has failed.

[0015] 1, the calculation device 10 includes a communication processing unit 11, a control unit 12, and a storage unit 13. The processing of each unit included in the calculation device 10 will be described below.

[0016] The communication processing unit 11 controls communication regarding various information exchanged with the connected device. For example, the communication processing unit 11 receives image data to be processed for skeletal estimation from an external device. The storage unit 13 stores data and programs required for various processes by the control unit 12, and has a registration information storage unit 13a. For example, the storage unit 13 is a storage device such as a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory.

[0017] The registration information storage unit 13a stores an image of the whole body registered in advance by the user. For example, the registration information storage unit 13a stores an image of the whole body of the user in a state where the user takes a predetermined pose (for example, raising both hands) in front of a camera as an image of the whole body. This pose may be freely determined by the user at the time of registration, or may be a pose that is determined in advance and is notified only to legitimate users. The data stored in advance in the registration information storage unit 13a does not necessarily have to be an image, and may store the result of skeletal estimation estimated from predetermined image data and the reliability of each joint. That is, the registration information storage unit 13a may store in advance, for example, the result of skeletal estimation estimated using a skeletal estimation model from an image of the whole body of the user in a predetermined pose, and the reliability of each joint output from the skeletal estimation model.

[0018] The control unit 12 has an internal memory for storing programs that define various processing procedures and required data, and executes various processes using these. For example, the control unit 12 has an acquisition unit 12a, a first estimation unit 12b, a second estimation unit 12c, a calculation unit 12d, a calculation unit 12e, and an authentication unit 12f. Here, the control unit 12 is, for example, an electronic circuit such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or a GPU (Graphical Processing Unit), or an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0019] The acquiring unit 12a acquires image data including a person. For example, the acquiring unit 12a acquires image data from a camera that captures the whole body of the person to be authenticated, and outputs the image data to the first estimating unit 12b.

[0020] The first estimation unit 12b estimates skeletal data using a skeletal estimation model that estimates skeletal data related to a person's skeleton, using the image data acquired by the acquisition unit 12a as input. For example, the first estimation unit 12b inputs the image data into the skeletal estimation model to identify the position of each part of the person's skeleton in the image data, and estimates the positions of the "right shoulder", "right upper arm", "right forearm", "left shoulder", "left upper arm", "left forearm", "right upper thigh", "right lower leg", "left upper thigh", and "left lower leg" as parts corresponding to each joint.

[0021] The second estimation unit 12c estimates skeletal data using a skeletal estimation model with image data stored in advance in the registration information storage unit 13a as input. For example, the second estimation unit 12c reads image data stored in the registration information storage unit 13a and inputs the read image data to the skeletal estimation model to identify the position of each part of the person's skeleton in the image data, and estimates the positions of the "right shoulder", "right upper arm", "right forearm", "left shoulder", "left upper arm", "left forearm", "right upper thigh", "right lower leg", "left upper thigh", and "left lower leg" as parts corresponding to each joint.

[0022] The calculation unit 12d calculates the weight value of each joint based on the reliability of the estimation result of each joint. Specifically, the calculation unit 12d calculates the weight value of each joint based on the reliability of the estimation result of each joint in the skeleton data estimated by the first estimation unit 12b and the skeleton data estimated by the second estimation unit 12c. For example, the calculation unit 12d calculates the weight value of each joint based on the reliability of the estimation result of each joint output from the skeleton estimation model. Note that, when the reliability of the estimation result of each joint is stored in advance in the registration information storage unit 13a, the calculation unit 12d may read the reliability from the registration information storage unit 13a and calculate the weight value of each joint based on the read reliability.

[0023] Here, a specific example of the calculation process of the weight value of each joint will be described. For example, the calculation unit 12d calculates the average of the reliability of the estimation result of each joint output from the skeleton estimation model when the first estimation unit 12b performs skeleton estimation and the reliability of the estimation result of each joint output from the skeleton estimation model when the second estimation unit 12c performs skeleton estimation, for each joint, using the following formula (1), to calculate the weight value of each joint. In the following formula, J means a set of joints, j means a certain joint, θ means the angle of the joint j, and conf means the reliability. In the following, the skeleton data estimated by the first estimation unit 12b will be referred to as "skeletal data A", and the skeleton data estimated by the second estimation unit 12c will be referred to as "skeletal data B".

[0024]

number

[0025] The calculation unit 12e calculates the similarity between the skeletal data estimated by the first estimation unit 12b and the skeletal data estimated from the predetermined image data, using the weight value of each joint calculated by the calculation unit 12d. Specifically, the calculation unit 12e calculates the similarity between the skeletal data estimated by the first estimation unit 12b and the skeletal data estimated by the second estimation unit 12c, using the weight value of each joint calculated by the calculation unit 12d. Note that, when the result of skeletal estimation is stored in advance in the registration information storage unit 13a, the calculation unit 12e may read the result of skeletal estimation from the registration information storage unit 13a and calculate the similarity between the read result of skeletal estimation and the skeletal data estimated by the first estimation unit 12b.

[0026] Here, a specific example of the similarity calculation process will be described. For example, the calculation unit 12e calculates the similarity for each of all joints as shown in the following formula (2), and calculates the sum of the values ​​obtained by weighting the similarity for each joint as the similarity score for the user's skeleton.

[0027]

number

[0028] Here, the process of calculating the similarity of the skeleton will be described with reference to FIG. 2. FIG. 2 is a diagram for explaining the process of calculating the similarity of the skeleton. As illustrated in FIG. 2, the calculation unit 12e specifies the position of each part of the skeleton from the whole-body image for each of the skeleton data A estimated by the first estimation unit 12b and the skeleton data B estimated by the second estimation unit 12c, and specifies the angles of the "right shoulder", "right upper arm", "right forearm", "left shoulder", "left upper arm", "left forearm", "right upper thigh", "right lower leg", "left upper thigh" and "left lower leg" as each joint. The calculation unit 12e calculates the closeness of the angles of the vectors as the similarity, for example, by cosine similarity. In this case, the closer the angles of the two vectors are, the closer the value is to 1.

[0029] The authentication unit 12f determines whether the similarity calculated by the calculation unit 12e is greater than or equal to a predetermined threshold, and if the similarity is greater than or equal to the predetermined threshold, determines that the authentication has been successful, and if the similarity is less than the predetermined threshold, determines that the authentication has failed.

[0030] Here, an overview of the authentication process by the calculation device 10 will be described with reference to Fig. 3. Fig. 3 is a diagram for explaining an overview of the authentication process by the calculation device. As illustrated in Fig. 3, the calculation device 10 inputs acquired image data to a skeleton estimation model to estimate skeleton data, and inputs preregistered image data to the skeleton estimation model to estimate skeleton data. Here, the calculation device 10 also obtains the reliability of each joint output from the skeleton estimation model.

[0031] Then, the calculation device 10 calculates a similarity score for the two estimated skeletal data, taking into account the reliability of each joint. After that, the calculation device 10 determines whether the calculated similarity is equal to or greater than a predetermined threshold, and determines that the authentication is successful if the similarity is equal to or greater than the predetermined threshold, and determines that the authentication is unsuccessful if the similarity is less than the predetermined threshold. Note that the calculation device 10 may perform the process up to the calculation of the similarity score, and the authentication process may be performed by another device.

[0032] [Calculation device processing procedure] Next, an example of a processing procedure by the calculation device 10 according to the first embodiment will be described with reference to Fig. 4. Fig. 4 is a flowchart showing an example of a processing flow in the calculation device according to the first embodiment.

[0033] As illustrated in FIG. 4, in the calculation device 10, when the acquisition unit 12a acquires image data including the entire body of a person (Yes at step S101), the first estimation unit 12b estimates skeletal data using a skeletal estimation model that estimates skeletal data related to the person's skeleton, using the image data acquired by the acquisition unit 12a as an input (step S102).

[0034] The second estimation unit 12c estimates skeletal data using a skeletal estimation model with image data registered in advance in the registration information storage unit 13a as input (step S103). If the results of skeletal estimation are stored in advance in the registration information storage unit 13a, this process can be omitted. Next, the calculation unit 12d calculates a weight value for each joint based on the reliability of the estimation results for each joint in the skeletal data estimated by the first estimation unit 12b and the skeletal data estimated by the second estimation unit 12c (step S104).

[0035] Then, the calculation unit 12e uses the calculated weight value of each joint to calculate the similarity between the skeletal data estimated by the first estimation unit 12b and the skeletal data estimated by the second estimation unit 12c (step S105).

[0036] Thereafter, the authentication unit 12f determines whether the similarity calculated by the calculation unit 12e is equal to or greater than a predetermined threshold, and if the similarity is equal to or greater than the predetermined threshold, determines that the authentication has been successful, and if the similarity is less than the predetermined threshold, determines that the authentication has failed (step S106).

[0037] [Advantages of the first embodiment] The calculation device 10 according to the first embodiment estimates skeletal data using a skeletal estimation model that estimates skeletal data related to a person's skeleton using image data as input. The calculation device 10 then calculates a weight value for each joint based on the reliability of the estimation result for each joint, and calculates the similarity between the skeletal data using the calculated weight value for each joint. The calculation device 10 then determines whether the similarity is equal to or greater than a predetermined threshold, and if the similarity is equal to or greater than the predetermined threshold, determines that the authentication has been successful, and if the similarity is less than the predetermined threshold, determines that the authentication has failed. This allows the calculation device 10 to improve the accuracy of the authentication process.

[0038] That is, the calculation device 10 calculates the similarity score to be used for authentication by taking into consideration the reliability of the estimation result. For example, the calculation device 10 can increase the reliability of authentication by reducing the contribution of a joint with low reliability to authentication.

[0039] [Second embodiment] In the above-mentioned first embodiment, a case where the weight value of each joint is calculated based on the reliability of the estimation result of each joint output from the skeleton estimation model has been described, but the present invention is not limited to this. For example, during authentication, instead of using a still image obtained from a camera, a moving image may be used to calculate the weight value of each joint based on the degree of variation in the estimated position of each joint. Therefore, in the following second embodiment, a case where a moving image is used to calculate the weight value of each joint based on the degree of variation in the estimated position of each joint will be described. Note that the description of the same configuration and processing as in the first embodiment will be omitted.

[0040] Here, an overview of the authentication process by the calculation device 10A according to the second embodiment will be described with reference to FIG. 5. FIG. 5 is a diagram illustrating an overview of the authentication process by the calculation device according to the second embodiment. As illustrated in FIG. 5, the acquisition unit 12a of the calculation device 10A according to the second embodiment acquires a plurality of image data from video data including a person. Then, the first estimation unit 12b receives the plurality of image data acquired by the acquisition unit 12a as input, and estimates skeletal data corresponding to each image data by using a skeletal estimation model.

[0041] The second estimation unit 12c receives a plurality of image data stored in advance in the registration information storage unit 13a and estimates skeletal data corresponding to each image data using a skeletal estimation model. The calculation unit 12d calculates a weight value for each joint based on the magnitude of variation in the estimated position of each joint in the skeletal data estimated by the first estimation unit 12b and the skeletal data estimated by the second estimation unit 12c.

[0042] Here, the variation in the estimated position of the joint will be described with reference to Fig. 6. Fig. 6 is a diagram for explaining the variation in the estimated position of the joint. As illustrated in Fig. 6, for example, when the estimated position of the right shoulder is different for each of a plurality of images included in the video, the calculation unit 12d determines that there is a variation in the estimated position and reduces the reliability of the right shoulder joint.

[0043] Then, the calculation device 10A calculates a similarity score for the two estimated skeletal data, taking into account the reliability of each joint. After that, the calculation device 10A determines whether the calculated similarity is equal to or greater than a predetermined threshold, and determines that the authentication is successful if the similarity is equal to or greater than the predetermined threshold, and determines that the authentication is unsuccessful if the similarity is less than the predetermined threshold.

[0044] [Effects of the second embodiment] The calculation device 10A according to the second embodiment acquires a plurality of image data from video data including a person, and estimates skeletal data corresponding to each image data using a skeletal estimation model with the acquired plurality of image data as input. The calculation device 10A also estimates skeletal data corresponding to each image data using a skeletal estimation model with the plurality of image data stored in advance in the storage unit 13 as input. Then, the calculation device 10A calculates a weight value for each joint based on the degree of variation in the estimated position of each joint in each estimated skeletal data. After that, the calculation device 10A calculates a similarity score for the two estimated skeletal data, taking into account the reliability of each joint.

[0045] In this way, in the calculation device 10A according to the second embodiment, the weighting value of a joint whose estimated position is not determined in the video is lowered, and the weighting value of a joint whose estimated position has little variation is increased, thereby enabling calculation of weighting values ​​that are more suited to the actual situation, and thus enabling improvement in the accuracy of the authentication process.

[0046] [System configuration, etc.] Moreover, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or a part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions. For example, the first estimation unit and the second estimation unit may be integrated. Furthermore, all or any part of each processing function performed by each device can be realized by a CPU or GPU and a program analyzed and executed by the CPU or GPU, or can be realized as hardware by wired logic.

[0047] Furthermore, among the processes described in this embodiment, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified.

[0048] [program] Also, a program in which the processing performed by the information processing device described in the above embodiment is written in a language executable by a computer can be created. For example, a program in which the processing performed by the calculation device 10, 10A according to the embodiment is written in a language executable by a computer can be created. In this case, the computer executes the program, thereby obtaining the same effect as in the above embodiment. Furthermore, such a program may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read and executed by a computer to realize the same processing as in the above embodiment.

[0049] Fig. 7 is a diagram showing a computer that executes a calculation program. As shown in Fig. 7, the computer 1000 includes, for example, a memory 1010, a CPU 1020, a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070, and these components are connected by a bus 1080.

[0050] The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012, as exemplified in FIG. 7. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090, as exemplified in FIG. 7. The disk drive interface 1040 is connected to a disk drive 1100, as exemplified in FIG. 7. A removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, as exemplified in FIG. 7. The video adapter 1060 is connected to a display 1130, as exemplified in FIG. 7.

[0051] 7, the hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, the above programs are stored in, for example, the hard disk drive 1090 as program modules in which instructions to be executed by the computer 1000 are written.

[0052] Moreover, the various data described in the above embodiment are stored as program data, for example, in the memory 1010 or the hard disk drive 1090. Then, the CPU 1020 reads out the program module 1093 and the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as necessary, and executes various processing procedures.

[0053] Note that the program module 1093 and program data 1094 relating to the program are not limited to being stored in the hard disk drive 1090, and may be stored in, for example, a removable storage medium and read by the CPU 1020 via a disk drive or the like. Alternatively, the program module 1093 and program data 1094 relating to the program may be stored in another computer connected via a network (such as a local area network (LAN) or wide area network (WAN)) and read by the CPU 1020 via the network interface 1070.

[0054] The above-described embodiments and their modifications are included in the technology disclosed in this application, as well as in the scope of the invention described in the claims and their equivalents. [Explanation of symbols]

[0055] 10 Calculation device 11 Communication processing unit 12 Control section 12a Acquisition part 12b First Presumption 12c Second Presumption 12d calculation section 12e Calculation part 12f Authentication section 13 Storage section 13a Registration information storage unit

Claims

1. a first estimation unit that receives a plurality of image data acquired from video data including a person, and estimates skeletal data corresponding to each image using a skeletal estimation model that estimates skeletal data related to a skeleton of the person; a second estimation unit that receives a plurality of image data stored in advance in a storage unit, and estimates the skeletal data corresponding to each image by using the skeletal estimation model; a calculation unit that calculates a weight value of each joint based on a degree of variation in an estimated position of each joint in the skeletal data estimated by the first estimation unit and the skeletal data estimated by the second estimation unit; a calculation unit that calculates a similarity between skeletal data estimated by the first estimation unit and skeletal data estimated by the second estimation unit using the weight value of each joint calculated by the calculation unit; A calculation device comprising:

2. 2. The calculation device according to claim 1, further comprising an authentication unit that determines whether the similarity calculated by the calculation unit is equal to or greater than a predetermined threshold, and determines that authentication has been successful if the similarity is equal to or greater than the predetermined threshold, and determines that authentication has failed if the similarity is less than the predetermined threshold.

3. A calculation method performed by a calculation device, comprising: a first estimation step of estimating, using a plurality of image data acquired from video data including a person as input, skeletal data corresponding to each image using a skeletal estimation model that estimates skeletal data related to the skeletal structure of the person; a second estimation step of estimating the skeletal data corresponding to each image by using the skeletal estimation model, using a plurality of image data stored in advance in a storage unit as an input; a calculation step of calculating a weight value for each joint based on a degree of variation in an estimated position of each joint in the skeletal data estimated in the first estimation step and the skeletal data estimated in the second estimation step; a calculation step of calculating a similarity between the skeletal data estimated in the first estimation step and the skeletal data estimated in the second estimation step, using the weight values ​​of each joint calculated in the calculation step; A calculation method comprising:

4. a first estimation step of estimating, using a plurality of image data acquired from video data including a person as input, skeletal data corresponding to each image using a skeletal estimation model that estimates skeletal data related to a skeleton of the person; a second estimation step of estimating the skeletal data corresponding to each image by using the skeletal estimation model, using a plurality of image data stored in advance in a storage unit as an input; a calculation step of calculating a weight value of each joint based on a degree of variation in an estimated position of each joint in the skeletal data estimated in the first estimation step and the skeletal data estimated in the second estimation step; a calculation step of calculating a similarity between the skeletal data estimated in the first estimation step and the skeletal data estimated in the second estimation step, using the weight value of each joint calculated in the calculation step; A calculation program that causes a computer to execute the above.

Images