Authentication verification using soft biometrics

Abstract

The technology described herein may be embodied in a method for verifying the authentication state of a device. In one aspect, the method includes capturing first auxiliary user identification information for short-term authentication verification of an object during a main authentication process. The first auxiliary user identification information includes first eyebrow biometric information of the object and first eyeglass frame information of the object. A short-term authentication verification process is performed to verify the identity of the object. In response to determining that a match score representing the degree of match between the first auxiliary user identification information and the captured second auxiliary user identification information is equal to or greater than a predetermined threshold, the authentication state of the object is verified. In response to determining that the match score is less than a predetermined threshold, the authentication state of the object is ended.

Description

Technical Field

[0001] This specification relates to image capture devices. Background Art

[0002] Biometrics allow for the identification and authentication of people based on a set of identifiable and verifiable data that is unique and specific to them. Authentication using biometrics involves the process of comparing a person's characteristic data to that person's biometric "template" to determine similarity. Biometric authentication systems can be grouped based on the physical characteristics used to authenticate the user. Examples of such systems include authentication based on fingerprints, hand geometry, palm veins, retinal scans, iris scans, facial recognition, signatures, and voice analysis. Summary of the invention

[0003] In one aspect, a method for allowing continued access to a soft biometric-based security system is described herein. The method includes: during a primary authentication process using a device to obtain a primary user identification information of a subject, capturing first auxiliary user identification information for short-term authentication verification of the subject, wherein the first auxiliary user identification information includes the subject's first eyebrow biometric information and the subject's first eyeglass frame information; after the primary authentication process, determining to perform the short-term authentication verification process to verify the identity of the subject; performing the short-term authentication verification process, including: capturing second auxiliary user identification information of the subject, including the subject's second eyebrow biometric information and the subject's second eyeglass frame information; comparing the first auxiliary user identification information with the second auxiliary user identification information, including determining a matching score indicating the degree of matching between the first auxiliary user identification information and the second auxiliary user identification information; determining whether the matching score is equal to or greater than a predetermined threshold score; in response to determining that the matching score is equal to or greater than the predetermined threshold score, verifying that the authentication state of the subject is passed and allowing continued use of the device; and in response to determining that the matching score is less than the predetermined threshold score, ending the authentication state of the subject, including prohibiting the subject from using the device.

[0004] In another aspect, a system is described herein, the system comprising an image acquisition device and an image analysis engine comprising one or more processors. The image acquisition device is configured to: during a main authentication process of using a device to obtain a main user identification information of an object, capture first auxiliary user identification information for short-term authentication verification of the object, wherein the first auxiliary user identification information includes the first eyebrow biometric information of the object and the first eyeglass frame information of the object; after the main authentication process, determine to perform the short-term authentication verification process to verify the identity of the object; perform the short-term authentication verification process, including: capturing second auxiliary user identification information of the object, including the second eyebrow biometric information of the object and the second eyeglass frame information of the object; comparing the first auxiliary user identification information with the second auxiliary user identification information, including determining a matching score indicating the degree of matching between the first auxiliary user identification information and the second auxiliary user identification information; determining whether the matching score is equal to or greater than a predetermined threshold score; in response to determining that the matching score is equal to or greater than the predetermined threshold score, verifying that the authentication state of the object is passed and allowing continued use of the device; and in response to determining that the matching score is less than the predetermined threshold score, ending the authentication state of the object, including prohibiting the object from using the device.

[0005] In another aspect, one or more machine-readable storage devices including machine-readable instructions are described herein, the machine-readable instructions being configured to cause one or more processing devices to perform various operations. The operations include: during a primary authentication process using a device to obtain a primary user identification information of a subject, capturing first auxiliary user identification information for short-term authentication verification of the subject, wherein the first auxiliary user identification information includes first eyebrow biometric information of the subject and first eyeglass frame information of the subject; after the primary authentication process, determining to perform the short-term authentication verification process to verify the identity of the subject; performing the short-term authentication verification process, including: capturing second auxiliary user identification information of the subject, including second eyebrow biometric information of the subject and second eyeglass frame information of the subject; comparing the first auxiliary user identification information with the second auxiliary user identification information, including determining a matching score indicating a degree of matching between the first auxiliary user identification information and the second auxiliary user identification information; determining whether the matching score is equal to or greater than a predetermined threshold score; in response to determining that the matching score is equal to or greater than the predetermined threshold score, verifying that the authentication state of the subject is passed and allowing continued use of the device; and in response to determining that the matching score is less than the predetermined threshold score, ending the authentication state of the subject, including prohibiting the subject from using the device.

[0006] Embodiments of the above aspects may include one or more of the following features. In response to determining that the match score is less than the predetermined threshold score, the main authentication process may be re-executed. Determining to execute the short-term authentication verification process may include determining that a predetermined amount of time has occurred since the main authentication process was completed. The short-term authentication verification process may be repeated at multiple subsequent time points. The device may be an all-in-one device or a mobile device. The first auxiliary user identification information and the second auxiliary user identification information may be acquired by a front camera of the mobile device. The first auxiliary user identification information may include a first clothing feature of the object, the second auxiliary user identification information may include a second clothing feature of the object, and comparing the first auxiliary user identification information with the second auxiliary user identification information may include comparing the first clothing feature with the second clothing feature. The matching score may include a fusion score of two or more of an eyebrow matching score, a glasses frame matching score, and a clothing feature matching score. The short-term authentication verification process may be performed without prompting the object for user input.

[0007] The various embodiments described herein can provide one or more of the following advantages. The reliability and / or security of the biometric authentication system can be improved by verifying the user's previous authentication. By allowing the biometric authentication system to use soft biometrics to verify previous authentications, additional processing can be preemptively terminated, thereby creating an additional layer of security. If the user returns to a previously authorized (biometric or otherwise) device after a short period of time, the user can be verified and the user's authentication state maintained without the need for additional, explicit biometric scans, to improve the user experience. In some cases, the techniques described herein allow for improved system security while reducing the requirement for additional hardware. In some cases, this in turn can reduce the costs associated with the underlying biometric authentication system.

[0008] One of the main issues associated with implementing biometrics with behavioral characteristics is the complexity of user compliance, which can lead to significantly reduced usability and overall user experience. The ideal best biometric for mobile devices would be one that distinguishes the largest possible population in the least invasive, cheapest and most accurate way. Therefore, passive identification methods can have a huge advantage over interactive methods due to transparency, ease of integration and independence from user compliance. Therefore, passive authentication verification based on soft biometrics can improve the security of the system. In typical use of a mobile device, the user's face and part of the user's upper body can be seen in the camera's field of view, adding an option to recognize the user or verify previous authentication through facial and / or other soft biometric patterns. Images can provide a lot of data related to the user's identity and behavior, and the use of images is transparent to the user experience.

[0009] The techniques described herein improve the security / reliability of biometric authentication systems that rely on identifying users based on facial recognition, eyeprint recognition, iris recognition, etc. Specifically, a pre-processing step is described herein that allows for a determination to be made as to whether the current user in a session is the same user that was previously authenticated at the start of the session. Upon detecting that the current user is different from the previously authenticated user, the current user may be prevented from further access to the system without explicit authentication of the current user, thereby adding a layer of security to the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 An exemplary environment is shown in which the techniques described herein may be used.

[0011] Figure 2 An example of a system using eyebrows as a soft biometric for short-term authentication is shown.

[0012] Figure 3 An exemplary convolutional neural network for determining eyebrow-based deep features of a user is shown.

[0013] Figures 4A-4C An example of a system using eyeglass frames as a soft biometric for short-term authentication is shown.

[0014] Figure 5A-5B Another example of a system using eyeglass frames as a soft biometric for short-term authentication is shown.

[0015] Figure 6 An exemplary convolutional neural network using eyeglass frames as a soft biometric for short-term authentication is shown.

[0016] Figure 7 An example of a system using clothing features as soft biometrics for short-term authentication is shown.

[0017] Figure 8 The use of clothing features as soft biometrics for short-term authentication using a non-learning based approach is shown.

[0018] Fig. 9 The use of clothing features as soft biometrics for short-term authentication using a learning-based approach is shown.

[0019] Fig.10 An example of a system in which the techniques described herein may be used is shown.

[0020] Fig.11 is a flow chart of an exemplary process for verifying the authentication status of a device.

[0021] Fig.12 is a block diagram representing an example of a computing device.

[0022] Repeat use of reference characters herein and in the drawings is intended to represent the same or analogous features or elements. DETAILED DESCRIPTION

[0023] This paper proposes an image analysis method in which images captured by an image acquisition device using a biometric authentication system are analyzed to verify the user's previous authentication.

[0024] Some desktop and mobile applications that require secure access may be configured to authenticate the user during the initial login phase where the user enters an authenticated state. Many biometric authentication systems involve authenticating the user to a security system based on a primary authentication, which may include identifying the user's face, eye pattern, iris, etc. Such biometric authentication systems may involve capturing one or more videos or images of the user and performing corresponding recognition processing on the captured images.

[0025] Malicious attempts to compromise the security of such biometric authentication systems can include using the system by a malicious user after the genuine user is logged into the system. For example, if a user is distracted and walks away from the device, another user could take control of the device, potentially accessing sensitive information without proper authentication. A malicious user could use the device and masquerade as the genuine user while the genuine user's session is still active.

[0026] Therefore, persistence or continuity of authentication state during user interaction may be required, especially for sensitive applications. For such applications, it may be necessary to set up a continuous (e.g., repeated) authentication system after the initial login phase for authentication verification.

[0027] The authentication system can use soft biometrics to verify the user's authentication. Soft biometrics (e.g., gender, skin color, face shape, typing and swiping behavior, etc.) are physical or behavioral characteristics that can provide personal information. In isolation, some soft biometrics may not be sufficient to uniquely define the identity of a user. However, using soft biometrics to verify previous authentication can provide various advantages. For example, soft biometrics may be easier to collect, less intrusive to users, and compliant than other biometrics. Multiple soft biometrics can be fused together and combined to overcome the uniqueness of a single one. As another example, one or more soft biometrics can be used with one or more other biometrics to improve the security and robustness of other biometrics. In an exemplary system, features such as clothing, eyebrows, and glasses seen within the field of view of a device's camera (e.g., a front-facing camera) can be used as soft biometrics to authenticate the user after the main authentication. Although clothing and glasses are not physical parts of the human body, they can contain visual information that can be used for short-term authentication.

[0028] Figure 1Exemplary environments 100a and 100b in which the techniques described herein may be used are shown. For example, the techniques described herein may be used with various types of devices, such as a mobile device 102. The mobile device 102 may include, for example, a front-facing camera. When a user 103 is using the mobile device 102, the face and upper body of the user 103 may be in the field of view 104 of the front-facing camera.

[0029] Kiosk 106 can be another exemplary environment for using the technology described herein. Kiosk 106 can be used for various purposes that require authenticating a user through one or more biometric authentication processes. For example, kiosk 106 can include an automatic teller machine (ATM) that allows a user to withdraw money from a bank account. Generally, kiosk 106 can be deployed in various types of locations to interactively authenticate a user, even without any active participation by the user.

[0030] The kiosk 106 may include a camera 108 configured to capture images of a user interacting with the kiosk 106, such as a user 110 within a field of view 112 of the camera 108. The captured images may be processed to identify / authenticate a legitimate user, and / or to allow or deny access to services / products provided through the kiosk 106. As described below, the captured images may be used to verify a user's prior authentication based on soft biometrics.

[0031] User 110 may participate in an initial, primary authentication process 114 before using kiosk 106. For example, primary authentication process 114 may include recognizing the user's face, eye prints, irises, etc. As another example, user 110 may provide a user identification and credentials. Similarly, user 103 may participate in primary authentication process 116 before using a particular application on mobile device 102.

[0032] During the primary authentication process 114 or 116, initial auxiliary user identification information or soft biometrics may be captured for user 110 or user 103, respectively. The auxiliary biometric may be information about a user that may be used for short-term authentication of the user. For example, the auxiliary biometric may include the shape of an eyebrow 118, the shape of an eyeglass frame 120, or the appearance of clothing 122. The auxiliary or soft biometric may be a characteristic of the user that is unlikely to change within a predetermined amount of time (e.g., five minutes) after the primary authentication process 114 or 116 has been completed. The initial auxiliary biometric captured when the user is first authenticated may be compared to subsequently captured auxiliary biometrics to determine whether the user attempting to access the corresponding device is the same user as the authenticated user.

[0033] For example, the short-term authentication verification process 124 or 126 may be performed respectively after the user 103 or the user 110 has been authenticated. For example, the short-term authentication verification processes 124 and 126 may be performed periodically (e.g., every minute). In the short-term authentication verification processes 124 and 126, the current auxiliary biometric features may be photographed for the user 103 or the user 110, respectively. For example, the current eyebrows, glasses frames, and clothing features may be photographed. The short-term authentication verification processes 124 and 126 may include comparing the corresponding initial auxiliary biometric features with the corresponding current auxiliary biometric features. For example, a matching score may be calculated for the corresponding device, which indicates the degree of match between the initial auxiliary biometric features and the current auxiliary biometric features photographed on the device.

[0034] In response to determining that the match score is equal to or greater than a predetermined threshold score, the user's authentication status can be verified. For example, the match score calculated on the mobile device 102 may be greater than the predetermined threshold score, and performing the short-term authentication verification process 124 on the mobile device 102 may result in allowing continued use of the mobile device 102 by the user 103.

[0035] As another example, in response to determining that the match score is less than a predetermined threshold score, the user's authentication state can be ended. For example, the match score calculated for the integrated machine 106 may be less than the predetermined threshold score, and the execution of the short-term authentication verification process 126 may result in locking the integrated machine 106. For example, the user 110 may have left the integrated machine 106, and the camera 108 may have captured an image of another, currently unauthorized user, and the short-term authentication verification process 126 determined that the other user is different from the user 110. For example, the integrated machine 106 can remain locked until the next successful completion of the main authentication process 114.

[0036] Figure 2 An example of a system using eyebrows as a soft biometric for short-term authentication is shown. For example, an image 202 of a user 203 is obtained using an image capture device. For example, the image 202 may be obtained using a front-facing camera of a mobile device, a camera included in or otherwise associated with an all-in-one device, or the like. The image 202 may be an image of the user 203 obtained during or as part of an initial authentication process, or as part of a short-term re-authentication process.

[0037] Image 202 may be processed using a facial landmark detection library 204 to obtain facial coordinates of landmarks in a right eyebrow region of interest (ROI) 206 (e.g., corresponding to the right eyebrow of user 203) and a left eyebrow ROI 208. For example, image 202 may be processed using an image processing library that may include features for machine learning and data analysis, particularly features for facial landmark detection.

[0038] The labeled right eyebrow ROI 210 corresponding to the right eyebrow ROI 206 shows the identified coordinates of the facial feature points identified in the right eyebrow ROI 206. Each coordinate can have an identifier. For example, the coordinate 212 with the identifier "19" specifies a specific location on the right eyebrow 214 in the labeled right eyebrow ROI 210. As another example, the coordinate 216 with the identifier "40" specifies a specific location on the top of the right eye 218 in the labeled right eyebrow ROI 210. Similar coordinates can be identified and drawn in the labeled left eyebrow ROI 219. If the user 203 wears glasses, the ROI can include the eyebrows and glasses, as shown in the right eyebrow ROI 220 and the left eyebrow ROI 221. Although not shown, the right eyebrow ROI 220 and the left eyebrow ROI 221 can be annotated with the identified coordinates, such as the labeled right eyebrow ROI 210 and the labeled left eyebrow ROI 219.

[0039] In the feature extraction stage 222, the right eyebrow feature 224 and the left eyebrow feature 226 are extracted based on the previously determined facial coordinates. For example, the right eyebrow feature 224 and the left eyebrow feature 226 may both include a depth feature 228, a local feature 230, and a global feature 232.

[0040] A VGG (Visual Geometry Group) convolutional neural network (CNN) may be used to extract deep features 228, as shown below. Figure 3 Described in more detail. A CNN can learn features in later (i.e., deeper) layers, such as image edge and contour detectors from earlier layers. "Deeper" layers can respond by creating their own feature filters for more complex patterns in the input, such as texture, shape, or changes in earlier processed features.

[0041] Local features may be extracted using histogram of oriented gradient (HOG) 230. The HOG technique may include counting the orientations of gradients that occur in local portions of an image. Global features may be extracted using "gist" descriptors 232.

[0042] A key descriptor can be obtained by dividing the feature map into a 4 by 4 grid (e.g., including 16 grid cells, such as grid cell 234), for which a directional histogram is extracted. As another example, the feature values ​​within each grid cell can be averaged. Multiple feature maps can be obtained by convolving the input image at different scales and directions. Key descriptors can be formed by concatenating the averages of multiple feature maps. Key descriptors can summarize gradient information (e.g., scale and direction) of different parts of an image, providing a rough description (e.g., gist) of the scene contained in the image.

[0043] Global features 232 may describe the image as a whole to summarize the entire image, while local features 230 may describe image patches (e.g., key points in the image). For example, global features 232 may include contour representations, shape descriptors, and texture features. Local features 230 may represent texture in an image patch.

[0044] A linear SVM (Support Vector Machine) classifier 236 can create output scores 238 for the right eyebrow and the left eyebrow based on the right eyebrow feature 224 and the left eyebrow feature 226. In addition, a fusion score 240 for the left eyebrow and the right eyebrow can be determined based on the output scores 238 for the left eyebrow and the right eyebrow, respectively. The fusion score 240 can be determined by merging multiple output scores 238.

[0045] In the test evaluation phase 242, the output score 238 and the fused score 240 (or, in some embodiments, only the fused score 240) of the first image 202 are compared to the corresponding score of the second image 202. For example, the first image can be an image of the user 203 obtained in an initial (or primary) authentication process, and the second image can be an image of the user 203 obtained at a later point in time, such as during a short-term re-authentication process. In the test evaluation phase 242, a match score indicating the likelihood that the first image and the second image are images of the same user (e.g., user 203) can be determined. If the match score is equal to or greater than a predetermined threshold score, the user 203 can be allowed to continue using the device or system. If the match score is less than the predetermined threshold score, the user can be denied access to the device or system.

[0046] Figure 3 An exemplary convolutional neural network 300 for determining a user's eyebrow-based depth features is shown. CNN 300 may be a VGG-16 network with sixteen layers, or may be another type of network. CNN 300 is a feedforward neural network in which the connection pattern between neurons is based on the organization of the animal visual cortex. CNN 300 may be arranged in convolutional layers alternating with subsampling layers.

[0047] CNN 300 includes an input layer 302 of a ROI image of a user's eyebrow region. CNN includes two convolutional layers 304, each with 64 filters. Convolutional layer 304 and other convolutional layers mentioned below may each include a rectified linear unit (ReLU) for normalization.

[0048] The maximum pooling layer 306 can be used for downsampling. Maximum pooling can eliminate non-maximum values, which can: 1) reduce upper layer calculations; 2) provide position invariance over a larger local area, which can lead to faster convergence, better selection of fixed features, and enhanced generalization.

[0049] The downsampled data may be processed by three convolutional layers 308, each with 128 filters. Another max pooling layer 310 may be used for further downsampling. A back-end convolutional layer 312 with 256 filters per layer is used, and further downsampling is performed using a max pooling layer 314. Further convolutions are performed using multiple sets of convolutional layers 316 and 318, followed by a max pooling layer 320 and a final max pooling layer 322. The features of size 3x10x512 obtained from the final max pooling layer 322 are flattened into a feature vector of 1x1x15360, which may be used as a deep feature representation as a fully connected layer 324.

[0050] Figures 4A-4C An example of a system 400 for short-term authentication using eyeglass frames as a soft biometric is shown. Short-term authentication using eyeglass frames may include a ROI detection stage 402 and the use of a cascaded CNN. A first CNN 404 may be used for eyeglass frame detection. The weights generated from the weight generation stage 406 may be provided to a second CNN 408, which may be used for image segmentation. The ROI detection stage 402, the first CNN 404, and the weight generation stage 406 are described below with respect to Figure 4B The second CNN 408 and the post-processing stage 410 are described in more detail below. Figure 4C Describe in more detail.

[0051] like Figure 4B As shown, during the ROI detection stage 402, facial feature points may be identified in input images 412 and 414. Input image 412 is of a user wearing glasses, and input image 414 is of a user not wearing glasses. The coordinates of the identified feature points are shown in annotated images 416 and 418. ROI 420 and ROI 422 may be generated by cropping input image 412 and / or input image 414, respectively, based on the identified feature point coordinates.

[0052] The first CNN 404 can accept ROI 420 and ROI 422 as input (e.g., as shown in input layer 424 and input layer 426, respectively). The first CNN 404 includes a convolution layer 428, a maximum pooling layer 430, a back-end convolution layer 432, a back-end maximum pooling layer 434, flattening layers 436 and 438, a fully connected layer 440, and a softmax layer 442. The output of the first CNN 404 includes case partitions 444 and weights 446. The weights 446 represent microstructural features of the large-scale image.

[0053] like Figure 4C As shown, the second CNN 408 includes an input layer 448 of the eye image, a convolution layer 450, a maximum pooling layer 452, a back-end convolution layer 454, an upsampling layer 456 and an upsampling layer 458, a heat map image 459 (e.g., sixteen heat map images), and a glasses mask 460. For example, the glasses mask 460 can be the sum / average of the sixteen heat map images. The post-processing operation 410 can include summing or averaging (or other mathematical operations) and binarization operations. A glasses mask 462 can be generated based on the post-processing operation 410, wherein the glasses mask corresponds to the eye input image 448.

[0054] Figure 5A-5B Another example of a system 500 using glasses as a soft biometric for short-term authentication is shown. Figure 5A As shown, in the ROI detection stage 509, glasses ROI 502 and no-glasses ROI 504 may be generated based on the glasses image 506 and no-glasses image 508, respectively, as described above for system 400. The repair stage 510 may include a repair process 512 that receives the no-glasses ROI 504 and the glasses image 514 to generate a repaired eye ROI 516.

[0055] like Figure 5B As shown, as part of the mask prediction stage 522, in the training stage 518, the CNN 520 can be trained using the inpainted eye ROI 516. In the testing stage 524, the CNN 520 can be tested using images of real glasses. The CNN 520 can be used to generate a mask 526. In the cleaning stage 528, a morphological cleaning process 530 can be used to create a cleaned mask 532 from the CNN mask output 534.

[0056] The cleaned mask 532 or mask 462 can be used for glasses-based user authentication processing. For example, the cleaned mask 532 or mask 462 can be multiplied by the RGB (red, green, blue) channels of the original ROI (e.g., the extended eye region) to obtain the glasses frame. Local, global, and deep feature descriptors, such as HOG, GIST, and VGG-16 descriptors, can be extracted from the segmented glasses. The extracted features can be used for SVM training and classification to authenticate objects based on the glasses frame shape. In some embodiments, users can be authenticated based on eyebrow and glasses frame features.

[0057] Figure 6An exemplary convolutional neural network 600 is shown that uses eyeglass frames as a soft biometric for short-term authentication. For example, CNN 600 can be CNN 520. CNN 600 includes an input eye image 602 (e.g., an image of a user wearing glasses). CNN 600 also includes a convolution layer 604, a maximum pooling layer 606, a back-end convolution layer 608, a deconvolution layer 610, an upsampling layer 612, a back-end deconvolution layer 614, and a custom (e.g., Lambda) layer 616. Eyeglass mask 618 (e.g., mask 526) can be the output of CNN 600.

[0058] Figure 7 An example of a system using clothing features as soft biometrics for short-term authentication is shown. In addition to eyebrow and / or eyeglass frame soft biometrics, clothing features can also be used as an auxiliary form of identification and can be used in conjunction with one or more other types of identification information. Using clothing features can include upper body pose clothing ROI segmentation, and robust matching of features extracted from the clothing ROI. Both learning and non-learning methods can be used for clothing feature matching.

[0059] In some embodiments, a CNN 702 (e.g., U-Net) is used for clothing ROI segmentation. The CNN 702 includes a contraction portion (e.g., encoder 704) and an expansion path (e.g., decoder 706). The encoder 704 may include repeated applications of two 3x3 convolutions followed by a rectified linear unit (ReLU) and a 2x2 maximum pooling operation. For example, each decoder layer may include upsampling using a 2x2 upper convolution, a series of corresponding feature maps from the contraction path, and two 3x convolutions followed by ReLU. Given an input upper body image 708, the output of the CNN 702 is a mask 710.

[0060] Figure 8A non-learning based approach is shown for short-term authentication using clothing features as soft biometrics. The non-learning based approach for clothing matching may include a discriminant for a predefined distance metric (e.g., Euclidean distance or Manhattan distance). In some embodiments, a Speeded Up Robust Features (SURF) approach is used. The SURF approach may be used for local feature detection and descriptor generation. To detect interest points, the SURF approach may use a Hessian matrix with a Gaussian smoothing approximation. Similar to the Scale Invariant Feature Transform (SIFT) approach, interest points may be calculated at different scales of an image pyramid. Descriptors around each interest point may be calculated using a first-order Haar wavelet response representing the distribution of pixel intensities within a block. Using the sum of absolute differences (e.g., Manhattan distances), a matching score may be calculated as the number of matched SURF points between an initial clothing ROI (e.g., collected during the main authentication process) and a verification clothing ROI (e.g., collected at a later time).

[0061] For example, clothing ROI 802 can be compared with second clothing ROI 804, for example, when second clothing 804 is a region of an image of the same clothing as shown in clothing ROI 802 (e.g., acquired at different time points). The matching score for comparing clothing ROI 802 with second clothing ROI 804 can be the sum of matching SURF points, for example, where each matching SURF point pair can be shown as a connecting line (e.g., connection 806). The matching score for comparing ROI 802 with second ROI 804 can be greater than a threshold (e.g., twenty matches), indicating that the second clothing ROI may correspond to a user who has recently recorded wearing the same clothing. Therefore, the authentication verification process can succeed. As another example, the matching score of the comparison of clothing ROI 808 with clothing ROI 810 of the impostor clothing can be less than a threshold (e.g., as shown, the matching score can be 2). Therefore, the authentication verification process may fail.

[0062] Fig. 9The use of clothing features as soft biometrics using a learning-based approach for short-term authentication is shown. In a learning-based approach, training data can be used to learn a discriminant (e.g., a similarity measure). For example, tile texture features can be used to train an SVM as a learned similarity measure. The trained SVM can then be used for authentication verification. In some embodiments, local binary patterns (LBP), histograms of oriented gradients, and color histograms can be used. LBP is a visual descriptor that can encode the difference between a given center pixel and adjacent pixels. The HOG method can include calculating the local gradient direction of a dense grid with local contrast normalization. Both LBP and HOG can operate on grayscale images. The color histogram method can generate color information from histograms of red, green, and blue channels.

[0063] Features can be extracted by dividing the clothing ROI into 2x3 non-overlapping tiles of 4 different image scales (e.g., 1x, 0.5x, and 0.25x). For example, the clothing ROI has been divided into tiles 902, 904, and 906 of different scales. The LBP, HOG, and CH feature vectors can be concatenated into a single vector and used to train and test SVMs (e.g., linear SVMs). For example, features 908, 910, and 912 extracted from tiles 902, 904, and 906, respectively, can be concatenated into a single vector 914.

[0064] For clothing-based authentication verification with deep features, a VGG neural network can be used. For example, deep features can be obtained from the 4th and 5th pooling layers of the VGG-16 network. The image features extracted from the 4th and 5th pooling layers can be aggregated with Fisher vectors. The output of clothing-based authentication verification can be a real (e.g., the same) user 916 or an imposter user 918. Clothing-based authentication verification can be combined with methods based on eyebrows and / or glasses frames.

[0065] Fig.10 An exemplary system 1000 is shown in which the techniques described herein may be used. The system 1000 includes an image acquisition device 1005 configured to take a picture of a subject. In some embodiments, the image acquisition device 1005 may be disposed in an infrared camera. In some embodiments, the image acquisition device 1005 includes one or more processing devices configured to generate digital data from the captured image. The image acquisition device 1005 may capture a first image of a user during a primary authentication process and capture a second image of the user at a later time during an authentication verification process.

[0066] The system 1000 also includes an image analysis engine 1010, which can be configured to analyze images captured by the image acquisition device 1005. In some embodiments, the image analysis engine 1010 resides on a computing device (e.g., a server) that is remotely located relative to the image acquisition device 1005 and / or the authentication engine 1015. For example, the image acquisition device 1005 can be set at an all-in-one machine or a mobile device, and the all-in-one machine or mobile device communicates with a remote server running the image analysis engine 1010 via a network. The image analysis engine 1010 can then provide the output of the analysis to the authentication engine 1015 located on the all-in-one machine or mobile device. In some embodiments, the authentication engine 1015 can also reside on a remote server (e.g., the same server where the image analysis engine 1010 resides, or on a different server), and communicate the results of the authentication process to the all-in-one machine or mobile device.

[0067] The image analysis engine 1010 may be configured to analyze the captured images in various ways to determine features of the captured images that may be used for user identification. For example, for the first image and the second image, the image analysis engine 1010 may identify eyebrow features, eyeglass frame features, and clothing features.

[0068] The authentication engine 1015 can be configured to perform a primary authentication process and an authentication verification process. The primary authentication process can include, for example, using one or more primary biometric features (e.g., face, iris) and / or using some other means of user identification (e.g., entered credentials). During the primary authentication process, first auxiliary user identification information can be stored for the user, such as eyebrow features, eyeglass frame features, clothing features, etc.

[0069] The authentication verification process may include capturing second auxiliary user identification information for the subject, including a second eyebrow feature, a second eyeglass frame feature, and a second clothing feature. The authentication engine 1015 may compare the first auxiliary user identification information with the second auxiliary user identification information, and determine a matching score representing the degree of matching between the first auxiliary user identification information and the second auxiliary user identification information based on the comparison result. In response to determining that the matching score is equal to or greater than a predetermined threshold score, the authentication engine 1015 may verify the authentication status of the object and allow continued use of the device. As another example, in response to determining that the matching score is less than a predetermined threshold score, the authentication engine 1015 may end the authentication status of the object, which may include prohibiting the object from using the device.

[0070] For example, the network described herein can be trained using photos obtained from a front-facing mobile device camera or other device. Other images can be used for network testing. Both training and test images can include images taken under various types of lighting (e.g., outdoor daylight, office lighting).

[0071] Fig.11 is a flow chart of an exemplary process 1100 for verifying the authentication status of a device. In some embodiments, at least a portion of the process 1100 may be provided in reference to Figure 1 The process 1100 is described as being performed by one or more processing devices within a machine such as machine 106 or within a mobile device such as mobile device 103. In some implementations, at least a portion of the process 1100 may be performed by an authentication engine 1015, which may be located within a machine, within a mobile device, or at one or more servers.

[0072] The operation of processing 1100 includes capturing first auxiliary user identification information (1110) for short-term authentication verification of the object during a main authentication process of obtaining the main user identification information of the object using a device. The first auxiliary user identification information includes the first eyebrow biometric information of the object and the first eyeglass frame information of the object. The first auxiliary user identification information may include clothing features. The main user identification information may be associated with the main biometric feature, such as facial or iris features. As another example, the main user identification information may be a user identification. The device may be a mobile device, an all-in-one device, or some other type of device. When the device is a mobile device, the first auxiliary user identification information may be acquired using a front camera of the mobile device. For example, the front camera may capture the face and upper body of the user.

[0073] The operations of process 1100 also include determining to perform a short-term authentication verification process to verify the identity of the subject after the main authentication process (1120). For example, it can be determined that a predetermined amount of time (e.g., two minutes) has occurred since the main authentication process was completed and the short-term authentication verification process should be performed. The short-term authentication verification process can be repeated at multiple subsequent time points (e.g., every two minutes).

[0074] The operations of process 1100 also include performing a short-term authentication verification process 1125. The short-term authentication verification process may be performed automatically without prompting the subject for user input.

[0075] Performing the short-term authentication verification process includes capturing the second auxiliary user identification information of the subject (1130). The second auxiliary user identification information includes the subject's second eyebrow biometric information and the subject's second eyeglass frame information. The second auxiliary user identification information may include the subject's second clothing feature.

[0076] Performing the short-term authentication verification process also includes comparing the first auxiliary user identification information with the second auxiliary user identification information (1140), including determining a match score representing the degree of match between the first auxiliary user identification information and the second auxiliary user identification information. For example, the first eyebrow biometric information can be compared with the second eyebrow biometric information to create an eyebrow matching score. The eyebrow matching score can be a fusion of the left eyebrow matching score and the right eyebrow matching score. The first eyeglass frame information can be compared with the second eyeglass frame information to create an eyeglass frame matching score. The first clothing feature can be compared with the second clothing feature to create a clothing feature matching score.

[0077] The matching score may be a fusion score of two or more of the eyebrow matching score, the eyeglass frame matching score, and the clothing feature matching score. Before using the fusion score, the individual scores may be normalized (e.g., in a min-max manner). Different types of fusion scores may be used. For example, a weighted fusion score may be used. For example, a weighted score level (WSL) fusion score may be calculated using formula (1):

[0078] WSL=CL*W CL +EB*W EB +GL*W GL (1)

[0079] Among them, CL is the score based on clothing, W CL is based on clothing weight, EB is based on eyebrow weight, WEB is based on eyebrow weight, GL is based on glasses weight, W GL The weights are based on the glasses. The individual weights can be determined based on the maximum accuracy obtained.

[0080] Performing the short-term authentication verification process also includes determining whether the match score is equal to or greater than a predetermined threshold score (1150). The predetermined threshold score may correspond to a predetermined likelihood (eg, 98%).

[0081] Performing the short-term authentication verification process also includes verifying the authentication status of the object in response to determining that the match score is equal to or greater than a predetermined threshold score, allowing continued use of the device (1160).

[0082] Performing the short-term authentication verification process also includes, in response to determining that the match score is less than a predetermined threshold score, ending the subject's authentication state, including prohibiting the subject from using the device (1170). In some embodiments, in response to determining that the match score is less than the predetermined threshold score, re-performing the main authentication process (e.g., requesting the user to re-login or re-provide the main biometric data).

[0083] Fig.12An example of a computing device 1200 and a mobile device 1250 using the techniques described herein is shown. Figure 1 , the all-in-one device 106 may include, in part or in whole, one or more of the computing device 1200 or the mobile device 1250. For example, the all-in-one device 106 may be the mobile device 1250. The computing device 1200 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The mobile device 1250 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the technology described and / or claimed herein.

[0084] The computing device 1200 includes a processor 1202, a memory 1204, a storage device 1206, a high-speed controller 1208 connected to the memory 1204 and a high-speed expansion port 1210, and a low-speed controller 1212 connected to a low-speed bus 1214 and the storage device 1206. Each of the components 1202, 1204, 1206, 1208, 1210, and 1212 is interconnected using various buses and can be mounted on a general motherboard or in other suitable ways. The processor 1202 can process instructions for execution within the computing device 1200, including instructions stored in the memory 1204 or stored on the storage device 1206, to display graphical information of a graphical user interface (GUI) on an external input / output device such as a display 1216 coupled to the high-speed controller 1208. In other embodiments, multiple processors and / or multiple buses, as well as multiple memories and multiple types of memories, can be used as appropriate. Additionally, multiple computing devices 1200 may be connected, with each computing device providing portions of the necessary operations (eg, as a server bank, a group of blade servers, or a multi-processor system).

[0085] The memory 1204 stores information within the computing device 1200. In one embodiment, the memory 1204 is one or more volatile memory units. In another embodiment, the memory 1204 is one or more non-volatile memory units. The memory 1204 may also be another form of computer-readable medium, such as a magnetic disk or optical disk.

[0086] The storage device 1206 can provide mass storage for the computing device 1200. In one embodiment, the storage device 1206 can be or include a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a magnetic tape device, a flash memory or other similar solid-state memory device or array of devices, including devices in a storage area network or other configuration. A computer program product can be tangibly embodied in an information carrier. The computer program product can also include instructions that, when executed, perform one or more methods such as those described above. The information carrier is a computer-readable medium or a machine-readable medium, such as a memory 1204, a storage device 1206, a memory on a processor 1202, or a propagated signal.

[0087] The high-speed controller 1208 manages bandwidth-intensive operations of the computing device 1200, while the low-speed controller 1212 manages less bandwidth-intensive operations. This allocation of functions is merely an example. In one embodiment, the high-speed controller 1208 is coupled to the memory 1204, the display 1216 (e.g., through a graphics processor or accelerator), and the high-speed expansion port 1210 that can accept various expansion cards (not shown). In this embodiment, the low-speed controller 1212 is coupled to the storage device 1206 and the low-speed expansion port 1214. The low-speed expansion port, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or, for example, through a network adapter to a network device such as a switch or router.

[0088] As shown in the figures, computing device 1200 can be implemented in a variety of different forms. For example, it can be implemented as a standard server 1220, or multiple times in a group of such servers. It can also be implemented as part of a rack server system 1224. In addition, it can be implemented in a personal computer such as a laptop computer 1222. Optionally, components from computing device 1200 can be combined with other components in a mobile device (not shown) such as device 1250. Each such device can contain one or more of computing device 1200, mobile device 1250, and the entire system can be composed of multiple computing devices 1200, mobile devices 1250 that communicate with each other.

[0089] Mobile device 1250 includes, among other components, a processor 1252, a memory 1264, an input / output device such as a display 1254, a communication interface 1266, and a transceiver 1268. Device 1250 may also have a storage device such as a micro drive or other device to provide additional storage. Each of components 1252, 1264, 1254, 1266, and 1268 are interconnected using various buses, and several components may be mounted on a common motherboard or in other suitable ways.

[0090] The processor 1252 can execute instructions within the mobile device 1250, including instructions stored in the memory 1264. The processor can be implemented as a chipset including a single or multiple analog and digital processors. The processor can provide, for example, coordination of other components of the device 1250, such as control of a user interface, applications run by the device 1250, and wireless communications performed by the device 1250.

[0091] The processor 1252 can communicate with the user through the control interface 1258 and the display interface 1256 coupled to the display 1254. The display 1254 can be, for example, a thin film transistor liquid crystal display (TFTLCD) or an organic light emitting diode (OLED) display or other appropriate display technology. The display interface 1256 may include appropriate circuits for driving the display 1254 to present graphics and other information to the user. The control interface 1258 can receive commands from the user and convert them to submit to the processor 1252. In addition, the external interface 1262 can provide communication with the processor 1252 to enable near field communication of the device 1250 with other devices. The external interface 1262 can be provided for, for example, wired communication in some embodiments or wireless communication in other embodiments, and multiple interfaces can also be used.

[0092] The memory 1264 stores information within the mobile device 1250. The memory 1264 may be implemented as one or more of the following: one or more computer-readable media, one or more volatile memory units, or one or more non-volatile memory units. An expansion memory 1274 may also be provided and connected to the device 1250 via an expansion interface 1272, which may include, for example, a single in-line memory module (SIMM) card interface. Such an expansion memory 1274 may provide additional storage space for the device 1250, or may also store applications or other information for the device 1250. Specifically, the expansion memory 1274 may include instructions for performing or supplementing the above-mentioned processing, and may also include security information. Therefore, for example, the expansion memory 1274 may be provided as a security module for the device 1250, and may be programmed with instructions that allow the secure use of the device 1250. In addition, a security application may be provided by a SIMM card, together with additional information, such as placing identification information on the SIMM card in an unbreakable manner.

[0093] The memory may include, for example, flash memory and / or non-volatile random access memory (NVRAM), as described below. In one embodiment, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform, for example, one or more of the methods described above. The information carrier is a computer-readable medium or a machine-readable medium, such as the memory 1264, the expansion memory 1274, a memory on the processor 1252, or a propagated signal that can be received by, for example, the transceiver 1268 or the external interface 1262.

[0094] Device 1250 can communicate wirelessly via a communication interface 1266, which may include digital signal processing circuitry as needed. Communication interface 1266 can provide communications in various modes or protocols, such as GSM voice calls, SMS, EMS or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000 or GPRS, etc. For example, such communications can be performed via a radio frequency transceiver 1268. In addition, short-range communications can be performed such as using Bluetooth, WiFi or other such transceivers (not shown). In addition, a global positioning system (GPS) receiver module 1270 can provide additional wireless data related to navigation and location to device 1250, which can be used appropriately by applications running on device 1250.

[0095] Device 1250 may also communicate audio using audio codec 1260, which may receive voice information from a user and convert it into usable digital information. Audio codec 1260 may similarly generate sounds audible to a user in a handheld device such as device 1250, for example, through a speaker. Such sounds may include sounds from voice phone calls, may include recorded sounds (e.g., voice messages, music files, etc.), and may also include sounds generated by applications running on device 1250.

[0096] As shown in the accompanying drawings, mobile device 1250 can be implemented in many different forms. For example, it can be implemented as a cellular phone. It can also be implemented as part of a smart phone 1282, a personal digital assistant, a tablet computer or other similar mobile devices.

[0097] Various implementations of the systems and techniques described herein can be realized in digital electronic circuitry, integrated circuits, specially designed application specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs executable and / or compilable on a programmable system that includes at least one programmable processor, which can be special purpose or general purpose, coupled thereto to receive data and instructions from a storage system, at least one input device, and at least one output device, and to send data and instructions to the storage system, at least one input device, and at least one output device.

[0098] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions.

[0099] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor) for displaying information to the user and a keyboard and a pointer device (e.g., a mouse or trackball) that the user can use to provide input to the computer. Other types of devices can also be used to provide interaction with a user. For example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback). Any form of input from the user can be received, including sound, voice, or tactile input.

[0100] The systems and techniques described herein can be implemented in a computing system that includes a back-end component (e.g., a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or includes a combination of such back-end, middleware, or front-end components. The components of the system can be connected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0101] A computing system may include clients and servers. Clients and servers are generally remote from each other and generally interact through a communication network. The relationship of client and server arises by virtue of computer programs running on respective computers and having a client-server relationship to each other.

[0102] Although several implementations have been described in detail above, other modifications may be made without departing from the scope of the inventive concepts described herein and, accordingly, other implementations are within the scope of the appended claims.

Claims

1. A method for verifying the authentication status of a device, characterized in that: The method comprises: During a primary authentication process of acquiring primary user identification information of a subject using a device, capturing first auxiliary user identification information for short-term authentication verification of the subject, wherein the primary user identification information is associated with a primary biometric feature or is a user identification, the first auxiliary user identification information is associated with a soft biometric feature, and the first auxiliary user identification information includes first eyebrow biometric information of the subject and first eyeglass frame information of the subject; After the main authentication process, determining to perform the short-term authentication verification process to verify the identity of the subject; and, Executing the short-term authentication verification process includes: photographing second auxiliary user identification information of the subject, including second eyebrow biometric information of the subject and second eyeglass frame information of the subject; comparing the first auxiliary user identification information with the second auxiliary user identification information, including determining a match score representing a degree of match between the first auxiliary user identification information and the second auxiliary user identification information; determining whether the match score is equal to or greater than a predetermined threshold score; In response to determining that the match score is equal to or greater than the predetermined threshold score, verifying the authentication status of the subject to allow continued use of the device; and In response to determining that the match score is less than the predetermined threshold score, ending the authentication state of the subject includes prohibiting the subject from using the device.

2. The method according to claim 1, characterized in that Also includes: In response to determining that the match score is less than the predetermined threshold score, the primary authentication process is re-executed.

3. The method according to claim 1, characterized in that Determining that the short-term authentication verification process is performed includes determining that a predetermined amount of time has occurred since the main authentication process was completed.

4. The method according to claim 3, characterized in that Also includes: The short-term authentication verification process is repeated at a plurality of subsequent time points.

5. The method according to claim 1, characterized in that The device is an all-in-one device.

6. The method according to claim 1, characterized in that The device is a mobile device.

7. The method according to claim 6, characterized in that The first auxiliary user identification information and the second auxiliary user identification information are obtained using a front camera of the mobile device.

8. The method according to claim 7, characterized in that The first auxiliary user identification information includes a first clothing feature of the object, the second auxiliary user identification information includes a second clothing feature of the object, and comparing the first auxiliary user identification information with the second auxiliary user identification information includes comparing the first clothing feature with the second clothing feature.

9. The method according to claim 8, characterized in that The matching score includes a fusion score of two or more of an eyebrow matching score, a glasses frame matching score, and a clothing feature matching score.

10. The method according to claim 1, characterized in that The short-term authentication verification process is performed without prompting the subject for user input.

11. A system, characterized in that: include: an image acquisition device configured to capture an image of a subject; and An image analysis engine, comprising one or more processing devices, wherein the image analysis engine is configured to: During a primary authentication process of acquiring primary user identification information of a subject using a device, capturing first auxiliary user identification information for short-term authentication verification of the subject, wherein the primary user identification information is associated with a primary biometric feature or is a user identification, the first auxiliary user identification information is associated with a soft biometric feature, and the first auxiliary user identification information includes first eyebrow biometric information of the subject and first eyeglass frame information of the subject; After the main authentication process, determining to perform the short-term authentication verification process to verify the identity of the subject; and Executing the short-term authentication verification process includes: photographing second auxiliary user identification information of the subject, including second eyebrow biometric information of the subject and second eyeglass frame information of the subject; comparing the first auxiliary user identification information with the second auxiliary user identification information, including determining a match score representing a degree of match between the first auxiliary user identification information and the second auxiliary user identification information; determining whether the match score is equal to or greater than a predetermined threshold score; In response to determining that the match score is equal to or greater than the predetermined threshold score, verifying the authentication status of the subject to allow continued use of the device; and In response to determining that the match score is less than the predetermined threshold score, ending the authentication state of the subject includes prohibiting the subject from using the device.

12. The system according to claim 11, characterized in that Also includes: In response to determining that the match score is less than the predetermined threshold score, the primary authentication process is re-executed.

13. The system according to claim 11, characterized in that Determining that the short-term authentication verification process is performed includes determining that a predetermined amount of time has occurred since the main authentication process was completed.

14. The system of claim 13, wherein: The image analysis engine is configured to repeat the short-term authentication verification process at a plurality of subsequent time points.

15. The system of claim 11, wherein: The first auxiliary user identification information includes a first clothing feature of the object, the second auxiliary user identification information includes a second clothing feature of the object, and comparing the first auxiliary user identification information with the second auxiliary user identification information includes comparing the first clothing feature with the second clothing feature.

16. One or more machine-readable storage devices comprising machine-readable instructions, characterized in that: The machine-readable instructions are configured to cause one or more processing devices to perform operations including: During a primary authentication process of acquiring primary user identification information of a subject using a device, capturing first auxiliary user identification information for short-term authentication verification of the subject, wherein the primary user identification information is associated with a primary biometric feature or is a user identification, the first auxiliary user identification information is associated with a soft biometric feature, and the first auxiliary user identification information includes first eyebrow biometric information of the subject and first eyeglass frame information of the subject; After the main authentication process, determining to perform the short-term authentication verification process to verify the identity of the subject; and Executing the short-term authentication verification process includes: photographing second auxiliary user identification information of the subject, including second eyebrow biometric information of the subject and second eyeglass frame information of the subject; comparing the first auxiliary user identification information with the second auxiliary user identification information, including determining a match score representing a degree of match between the first auxiliary user identification information and the second auxiliary user identification information; determining whether the match score is equal to or greater than a predetermined threshold score; In response to determining that the match score is equal to or greater than the predetermined threshold score, verifying the authentication status of the subject to allow continued use of the device; and In response to determining that the match score is less than the predetermined threshold score, ending the authentication state of the subject includes prohibiting the subject from using the device.

17. The machine-readable storage device of claim 16, wherein: The operations further include re-performing the primary authentication process in response to determining that the match score is less than the predetermined threshold score.

18. The machine-readable storage device of claim 16, wherein: Determining to perform the short-term authentication verification process includes determining that a predetermined amount of time has occurred since completing the main authentication process.

19. The machine-readable storage device of claim 18, wherein: The operations also include repeating the short-term authentication verification process at a plurality of subsequent time points.

20. The machine-readable storage device of claim 16, wherein: The first auxiliary user identification information includes a first clothing feature of the object, the second auxiliary user identification information includes a second clothing feature of the object, and comparing the first auxiliary user identification information with the second auxiliary user identification information includes comparing the first clothing feature with the second clothing feature.

Images