A system that enables compatibility with multiple configurations of biometric authentication hardware

Abstract

The biometric identification system processes input data acquired by an input device to determine an embedding used to identify the user. Different types of input devices, or hardware configurations of input devices, may produce different outputs. Each hardware configuration may be associated with a respective representation data. A set of converter networks is used to convert the embedding from one representation data associated with a first type of device or hardware configuration to another representation data. This allows users to interact across different hardware configurations without having to re-enroll with different input devices or hardware configurations. Opportunistic updates are made to the embedding as embeddings native to a particular hardware configuration are acquired from the user.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. patent application Ser. No. 18 / 051,214, entitled "System to Provide Multiconfiguration Biometric Hardware Compatibility," filed October 31, 2022, the contents of which are incorporated by reference into this disclosure. [Background technology]

[0002] Biometric input data can be used to assert the identity of a user.

[0003] The detailed description will be set forth with reference to the accompanying drawings. In the drawings, the left-most digit(s) of a reference number identifies the figure in which the reference number first appears. Use of the same reference number in different figures indicates similar or identical items or features. The drawings are not necessarily drawn to scale, and in some figures, proportions or other aspects may be exaggerated to facilitate an understanding of particular aspects. [Brief explanation of the drawings]

[0004] [Figure 1] 1A and 1B illustrate a system for achieving compatibility of multiple configurations of biometric authentication hardware, according to some embodiments. [Figure 2] 1 illustrates processing training input data to determine transformer training data, according to some embodiments. [Figure 3] 1 illustrates a transducer module during training, according to some embodiments. [Figure 4] 1 illustrates transforming representation data to achieve compatibility with alternative hardware configurations, according to some embodiments. [Figure 5] 1 illustrates a query using native and transformed representation data, according to some embodiments. [Figure 6]1 illustrates compatibility matrix data showing hardware configurations and the respective representation data that is converted, according to some embodiments. [Figure 7] FIG. 1 is a flow diagram for registering with a first hardware configuration and backfilling expression data to a second hardware configuration, according to some embodiments. [Figure 8] FIG. 1 is a block diagram of a computing device for implementing the system, according to some embodiments.

[0005] Although embodiments are described herein by way of example, those skilled in the art will recognize that the embodiments are not limited to the illustrated examples or drawings. The drawings and their detailed description are not intended to limit the embodiments to the particular forms disclosed; on the contrary, it should be understood that the invention is to cover all modifications, equivalents, and alternatives falling within the spirit and scope as defined by the appended claims. The headings used herein are for organizational purposes only and are not meant to be used to limit the scope of the description and claims. As used throughout this application, the term "may" is used in the permissive sense (i.e., meaning having the potential to), rather than the essential sense (i.e., meaning required). Similarly, the terms "include," "including," and "includes" mean including, but not limited to, DETAILED DESCRIPTION OF THE INVENTION

[0006] Input data used for biometric identification, etc., may be acquired using different hardware configurations of input devices. An input device may acquire input data using one or more modalities. For example, one modality may include an image of the skin on the surface of a user's palm, and a second modality may include an image of subcutaneous features, such as veins, in the user's palm.

[0007] Different hardware configurations may have different physical configurations, operating characteristics, etc. Physical configurations may differ by the physical arrangement of components such as the camera(s), lighting fixture(s), the type of camera(s) used, the wavelength of light used, the resolution of the input data acquired, etc. This may result in different hardware configurations acquiring different types of input data. For example, a first hardware configuration may acquire input data using two modalities, a second hardware configuration may acquire input data using a single modality and 1 megapixel image resolution, a third hardware configuration may acquire input data using a single modality with 5 megapixel image resolution, a fourth hardware configuration may acquire input data using three modalities, etc.

[0008] The overall performance of a biometric authentication system may be maximized by tailoring a particular combination of hardware configurations with subsequent data processing. During operation, input data may be processed using an embedding network that provides as output representation data. For example, the embedding network may provide as output a fixed-length embedding vector that represents features in the input data. The input data may be processed in other ways as well. For example, the input data may be preprocessed using one or more filters, image transforms, etc. to generate a canonical version of the input data for subsequent processing.

[0009] The embedded network may be trained using input data associated with a particular hardware configuration. Continuing with the previous example, a first hardware configuration using two modalities may utilize a first embedded network trained to process first-modality images and a second embedded network trained to process second-modality images. In comparison, a third hardware configuration using a single modality at relatively high resolution may utilize a third embedded network trained to process high-resolution input data for that single modality.

[0010] The expression data of a particular user obtained during the registration process may later be used to determine the user's asserted identity. During the registration process, registered user data is determined that associates identification data with the registered expression data. Once registered, the system may be used to identify unknown users. For example, after registration, the unknown user uses an input device to provide input data, such as image data of a hand. The unidentified user's input data is processed to determine query expression data. The query expression data may then be compared with registered expression data of previously registered users. If the query expression data is deemed to match the registered expression, identity may be asserted.

[0011] As noted above, the input data generated by different hardware configurations may differ. As a result, the subsequent data processing of the output from those respective hardware configurations may also differ. Continuing with the previous example, each hardware configuration may utilize an embedded network trained using input data native to the respective hardware configuration.

[0012] From an operational and user perspective, it is desirable for users to register once to use the system. The registration process may take several minutes and may involve interaction with more computationally intensive systems, human operators, etc. This poses challenges in situations where each hardware configuration may obtain different input data or utilize different data processing systems, such as different embedded networks.

[0013] Traditionally, users have had to go through a registration process for every hardware configuration. With each different hardware configuration deployed, users and system operators must go through a time-consuming and costly registration process. This is inconvenient for users and becomes increasingly expensive to perform, especially as the number of users using the system increases.

[0014] This disclosure describes techniques for enabling compatibility of multi-configuration biometric authentication hardware by propagating representation data associated with one hardware configuration to another hardware configuration, thereby allowing a user to enroll only once. First input data is acquired and processed in a first hardware configuration to determine first native representation data associated with the first hardware configuration. The first native representation data is associated with a first representation space, such as a first embedding space of the output of a first trained embedding network used by the first hardware configuration.

[0015] The first native representation data is then transformed into another representation space associated with another hardware configuration with which compatibility is maintained. For example, a transformation matrix may indicate which hardware configurations are mutually compatible. A trained one-way transformer network may be used to transform the native representation data into transformed representation data in a given representation space. The given representation space is associated with a specific hardware configuration.

[0016] The converted expression data may then be used for subsequent queries involving input data acquired using a different hardware configuration. For example, if a user is registered using a first hardware configuration and later identified using a second hardware configuration, second native expression data based on input data acquired using the second hardware configuration may be added to the user's registered user data for later use. This facilitates an opportunistic update process in which native expression data is added as the user interacts with a compatible hardware configuration. In some implementations, while using registered user data, the system may use the converted expression data when native expression data is unavailable and preferentially utilize native expression data when available.

[0017] Use of the techniques described in this disclosure significantly improves the user experience. Users only need to perform the enrollment process once and still benefit from the use of different hardware configurations of input devices supported by the system. System operators benefit from eliminating the need to perform costly re-enrollments. Both users and system operators benefit from significant performance improvements, such as increased identification accuracy and reduced latency, that result from using a data processing system tuned for a specific hardware configuration. Additionally, both system operators and users benefit from significantly reduced constraints on further development of input devices and associated data processing. For example, new hardware configurations may be cheaper and allow for larger deployments, increasing usability for users and allowing users to use biometric identification in more locations. Exemplary System

[0018] 1A and 1B illustrate a system 100 for enabling compatibility of multiple configurations of biometric hardware, according to some embodiments. System 100 is described in the context of a biometric identification system for determining a user's identity. However, the systems and techniques described herein may be used in other contexts.

[0019] For clarity of illustration, and not by way of limitation, FIGS. 1A and 1B show different portions of system 100.

[0020] A user's hand 102 is shown positioned on an input device 104. The input device 104 may include a computing device and a camera 108 (see FIG. 1B ). The camera 108 has a field of view (FOV) 110. During operation of the input device 104, the camera 108 captures images of objects within the FOV 110, such as the hand 102, and provides input image data 112. The input device 104 may include other components not shown. For example, the input device 104 may include a light that illuminates objects within the FOV 110.

[0021] As shown in FIG. 1A , system 100 may include a variety of different input devices 104(1), 104(2), ..., 104(D). Input devices 104 may be associated with a particular hardware configuration 192. For example, input device 104(1) may be associated with a first hardware configuration 192(1), a second input device 104(2) may be associated with a second hardware configuration 192(2), etc. Each hardware configuration 192 may include multiple input devices 104. In some implementations, input device 104 may be incorporated into or include a smartphone device, a tablet device, a laptop computer, a desktop computer, a home security device, etc. For example, input device 104 may include a doorbell camera. In other implementations, input device 104 may include a dedicated biometric input device for point-of-sale scanners, entry to a secure area, etc.

[0022] The hardware configuration 192 may vary based on one or more of the physical hardware or the operation of that hardware. Different physical configurations may vary depending on the physical arrangement of components such as the camera(s) 108, lenses or other optical components, lighting fixture(s), the type of camera(s) used, the wavelength of light used, and the resolution of the input image data 112 acquired. In some implementations, monochromatic light associated with a particular wave may be used. In other implementations, the light may include a range or band of wavelengths. For example, the set of wavelengths may include visible light, near-infrared, mid-infrared, far-infrared, etc.

[0023] The hardware configurations 192 may also differ based on how the physical configurations are utilized, such as through different operating modes. For example, first and second hardware configurations 192 may have the same physical configuration, but one or more features or functions associated with the operation of the input device 104 may be limited or operated differently. For example, export restrictions implemented via program instructions executed on the input device 104 may prevent the operation of a particular modality. In another example, the same input device 104(1) may be associated with a first hardware configuration 192(1) used a first time and a second hardware configuration 192(2) used a second time. Continuing with this example, during the day, the input device 104(1) may be used with the first hardware configuration 192(1), with the lighting fixtures operated in a first lighting mode. At a later time, such as at night, the same input device 104(1) is operated with the second hardware configuration 192(1), with the lighting fixtures operated in a second lighting mode. The lighting modes may vary based on the number of lighting fixtures used, the intensity of light emitted by the lighting fixtures during operation, the wavelength of light emitted by the lighting fixtures, etc.

[0024] The different hardware configurations 192 may also be distinguished based on the data processing that occurs within the input device 104. For example, a first hardware configuration 192 may include an image processing pipeline that utilizes Joint Photographic Experts Group (JPEG) compression at a quality setting of 50, a second hardware configuration 192 may include an image processing pipeline that utilizes JPEG compression at a quality setting of 25, and a third hardware configuration 192 may include an image processing pipeline that uses Portable Network Graphics (PNG) compression. Each of these configurations may use the same physical configuration or may share one or more common hardware elements, such as a particular type of image signal processor (ISP).

[0025] In the illustrated implementation, the hand 102 is held above the camera 108, with the FOV 110 extending upward. In other implementations, other configurations may be used. For example, the camera 108 may have an FOV 110 extending downward, and the user may place the hand 102 in the FOV 110 below the input device 104. In another example, the camera 108 may have an FOV 110 extending horizontally, and the user may face the input device 104 and place the hand 102 within the FOV 110.

[0026] The input device 104 may acquire input data 112 using one or more modalities. In one implementation, the input device 104 is configured to acquire images of the hand 102 illuminated using infrared light having two or more specific polarizations, such as with different illumination patterns. For example, during a movement, a user may present the hand 102 with the palm or palm region facing the input device 104. As a result, the input image data 112 provides an image of the front of the hand 102. In other implementations, the input image data 112 may include the back of the hand 102. Separate images may be acquired using different combinations of polarizations provided by the infrared light.

[0027] Depending on the polarization used, the image generated by the input device 104 may be of a first modality feature or a second modality feature. The first modality may utilize images acquired by a camera 108 in which the hand 102 is illuminated with light having a first polarization and which has a polarizer that also passes light having the first polarization through the camera 108. The first modality feature may include features near or on the surface of the user's hand 102. For example, the first modality feature may include surface features such as creases, wrinkles, scars, dermal papillae, etc. in at least the epidermis of the hand 102. Images acquired using the first modality may be associated with one or more surface features.

[0028] The second modality features include features beneath the epidermis. The second modality may utilize images acquired by a camera 108 in which the hand 102 is illuminated with light having a second polarization and a polarizer that passes light having a first polarization through the camera 108. For example, the second modality features may include subcutaneous anatomical structures such as veins, bones, and soft tissue. Some features may be visible in both the first modality image and the second modality image. For example, a crease in the palm may include a superficial first modality feature as well as a second modality feature deeper within the palm. Images acquired using the second modality may be associated with one or more subcutaneous features.

[0029] Separate images of the first and second modalities may be acquired using different combinations of polarization provided by the infrared light. The input image data 112 may include information associated with one or more modalities. As shown in FIG. 1B , the input image data 112 may include first modality image data 114 and second modality image data 116. The first modality image data 114 and second modality image data 116 of the same object may be acquired in rapid succession to one another. For example, the camera 108 may operate at 60 frames per second and acquire the first modality image data 114 in a first frame and the second modality image data 116 in a second frame.

[0030] In other implementations, other modalities may be used. A third modality may include color images acquired using a red-green-blue (RGB) camera using visible light. A fourth modality may include depth or distance data that provides information about the shape of an object, such as the hand 102.

[0031] In the implementations shown herein, the input device 104 does not include guides, scan plates, or other structures that constrain the posture or position of the hand 102. The omission of guides may improve hygienic operation of the system. For example, by removing the guides, the user's hand 102 does not physically contact the structures, eliminating the possibility of contact transmission of contaminants, diseases, etc. The elimination of physical contact may eliminate the need for surface cleaning of the guides between users.

[0032] In another embodiment, the input device 104 may include a structure, such as a guide or scan plate, to constrain the movement of at least a portion of the hand 102. For example, the scan plate may include a flat glass plate on which the hand 102 may be placed, and the camera 108 may then capture an image.

[0033] In some implementations, the input device 104 may also provide hardware configuration data 194 associated with or included within the input image data 112. For example, the hardware configuration data 194 may indicate a particular hardware configuration 192 with which the input device 104 is associated. In some implementations, a model number, serial number, or other information may be provided instead of or in addition to the hardware configuration data 194.

[0034] A user may utilize the system 100 by performing an enrollment process. An enrollment module 118 (see FIG. 1B) may coordinate the enrollment process. Enrollment may associate biometric information, such as expression data or transformed expression data, with specific information, such as name, account number, etc.

[0035] During the registration process, a user opts in and presents their hand 102 to an input device 104. The input device 104 provides input image data 112 to a computing device 106 executing an embedded network module 120. The embedded network module 120 includes a neural network implementing at least one embedded network module 120 that receives the input image data 112 as input and provides representation data 134 as output. The representation data 134 represents at least some of the features depicted in the input image data 112. In some implementations, the representation data 134 may include vector values ​​in a representation space 132.

[0036] In one embodiment, where the input image data 112 includes data associated with multiple modalities, a single embedded network module 120 may be used to process the multi-modality input image data 112. In another embodiment, where the input image data 112 includes data associated with multiple modalities, multiple embedded network modules 120 may be used to process each modality of the input image data 112.

[0037] In some implementations, the input device 104 may include a computing device and may execute the embedded network module(s) 120. In other implementations, the input device 104 may encrypt and transmit the input image data 112 or data based thereon to another computing device 106, such as a server.

[0038] During the registration process, the submitted expression data 134 may be checked to determine whether the user has been previously registered. Successful registration may include storage of identification data 142, such as name, phone number, account number, etc., and storage of one or more of expression data 134 or converted expression data 156, as registered user data 140. In some implementations, registered user data 140 may include additional information associated with processing of input image data 112 with embedded network module 120. For example, registered user data 140 may include intermediate layer data, such as values ​​from the penultimate layer of embedded network module 120.

[0039] Returning to FIG. 1A , a user may use system 100 by presenting hand 102 to one of participating input devices 104 to achieve interoperability. In some implementations, a user may register using any of the input devices 104 associated with participating hardware configurations 192. In this illustration, it is assumed that the user is registered using input device 104(1) associated with first hardware configuration 192(1). Input device 104(1) obtains first input image data 112(1) and provides it to first embedded network module 120(1) associated with first hardware configuration 192(1). In some implementations, other pre-processing operations may be performed on input image data 112(1) before it is processed by embedded network module 120(1). Input device 104(1) also provides hardware configuration data 194(1) indicating the first hardware configuration 192(1) to which first input device 104(1) belongs.

[0040] The first embedding network module 120(1) processes the first input image data 112(1) and determines the first representation data 134(1). For example, the first representation data 134(1) may include fixed-length embedding vectors.

[0041] First representation data 134(1) is associated with a first representation space 132(1). The first representation space 132(1) is associated with a first hardware configuration 192(1). The representation data 134 generated by the embedded network module(s) 120 associated with a particular representation space 132 is also referred to as a native representation 196. In comparison, the transformed representation data 156 is also referred to as a transformed representation 198.

[0042] As shown in this figure, a second input device 104(2) associated with a second hardware configuration 192(2) is also used. The second input device 104(2) provides second input image data 112(2) to a second embedded network module 120(2). The second embedded network module 120(2) is associated with the second hardware configuration 192(2). The second input device 104(2) may also provide hardware configuration data 194(2).

[0043] The second embedded network module 120(2) processes the second input image data 112(2) and determines the second representation data 134(2).

[0044] Second representation data 134(2) is associated with second representation space 132(2), which is associated with second hardware configuration 192(2).

[0045] The comparison module 160 may compare representation data that is within the same representation space 132. As noted above, conventional techniques involve users re-registering to provide native representations 196 for each of the participating representation spaces 132.

[0046] As shown in FIG. 1A, system 100 may include one or more transformer modules 150(1), 150(2), ..., 150(T). Transformer modules 150 are trained to transform or convert representation data 134 from one representation space 132 to another representation space 132 using transformer training data 152. Upon completion of training, transformer modules 150 include weight data 154 indicating weights, bias values, or other values ​​associated with nodes in their neural networks that represent the resulting training. The training and operation of transformer modules 150 are described in more detail in the following figures.

[0047] Once training is complete, the transformer module(s) 150 are used to transform the expression data 134 into transformed expression data 156. For example, the transformer module 150 may transform the first expression data 134(1) into the first transformed expression data 156(1). As a result, during subsequent use, the second embedded network module 120(2) may be used, and the comparison module 160 may operate by comparing the query expression data 162 in the second representation space 132(2) with the enrolled user data 140, which includes the transformed expression data 156. By utilizing the transformer module(s) 150, during operation of the system 100, the comparison module 160 or other modules may use the transformed expression data 156 until the native expression 196 is available.

[0048] In one embodiment where the input image data 112 and corresponding representation data 134 include data associated with multiple modalities, a single transformer module 150 may be used to process the representation data 134. In another embodiment, multiple transformer modules 150 may be used to transform the representation data 134, or portions thereof associated with particular modalities, into transformed representation data 156.

[0049] The transformer module 150 may be unidirectional in that it is trained to receive as input representation data 134 associated with a first representation space 132(1) and provide as output transformed representation data 156 associated with a second representation space 132(2).

[0050] As shown in FIG. 1A , first expression data 134(1) native to a first representation space 132(1) may be processed by a first transducer module 150(1) to determine first transformed expression data 156(1). The first transformed expression data 156(1) associated with a second representation space 132(2) may then be used to perform queries, stored as registered user data 140, and so on. Similarly, second expression data 134(2) native to a second representation space 132(2) may be processed by a second transducer module 150(2) to determine second transformed expression data 156(2). The second transformed expression data 156(2) associated with the first representation space 132(1) may then be used to perform queries, stored as registered user data 140, and so on.

[0051] Although two representation spaces 132 and associated converter modules 150 are shown, the system 100 may be expanded to any number of representation spaces 132. In this illustration, two-way compatibility between the representation spaces 132 and the associated hardware configurations 192 is achieved, i.e., a user may continue to use the input device 104 of either hardware configuration 192.

[0052] In other implementations, compatibility may be asymmetric. For example, if first hardware configuration 192(1) is degraded and discontinued, first representation data 134(1) is converted to second representation space 132(2) for use with input device 104 of second hardware configuration 192(2), but not vice versa.

[0053] It is understood that enrolled user data 140 may include multiple instances of an expression associated with a particular user. For example, a single user may have multiple native expressions 196, multiple translated expressions 198, etc. stored in enrolled user data 140.

[0054] During a subsequent use, such as a second time, the (as yet unidentified) user presents hand 102 to input device 104. The resulting query input image data 112 may be processed by first embedded network module 120(1) to determine query expression data 162 (not shown) within first expression space 132(1). Comparison module 160 compares query expression data 162 with expression data 134 and transformed expression data 156 stored in enrolled user data 140 to determine asserted identification data 164. In one implementation, asserted identification data 164 may include a user identifier associated with previously stored expression data 134 or transformed expression data 156 within enrolled user data 140 that is closest to the query expression data 162 associated with the user who presented hand 102. The comparison module 160 may utilize other considerations, such as requiring that the query expression data 162 be less than or equal to a maximum distance in expression data from a particular user's expression data 134 or transformed expression data 156 before determining the asserted identification data 164.

[0055] In some implementations, the comparison module 160 may utilize different thresholds or techniques for comparison based on whether the enrolled user data 140 is a native representation 196 or a transformed representation 198. For example, the threshold maximum distance for a comparison between a native representation 196 and a transformed representation 198 of the query expression data 162 may be smaller than that involving a comparison with a previously stored native representation 196.

[0056] The asserted identification data 164 may then be used by subsequent systems or modules. For example, the asserted identification data 164 or information based thereon may be provided to a facility management module 166.

[0057] Facility management module 166 may use asserted identity data 164 to associate an identity with a user as they move around the facility. For example, facility management module 166 may use data from cameras or other sensors in the environment to determine the user's location. Given the user's known path from an entrance using input device 104, the user identity indicated in identification data 142 may be associated with the user as they use the facility. For example, the identified user may go to a shelf, retrieve an item, and exit the facility. Facility management module 166 may determine that conversation data indicating the retrieval of the item is associated with the user identifier specified in asserted identity data 164 and may charge the account associated with the user identifier. In another embodiment, facility management module 166 may include a point-of-sale system. A user may present hand 102 at checkout to assert their identity and make a payment using a payment account associated with their identity.

[0058] System 100 may continue to undergo changes over time. Different embedding network modules 120 may include different neural network architectures, may use different training data, etc. The representation spaces 132 generated by different embedding network modules 120 may differ. For example, second representation space 132(2) may have a different number of dimensions than first representation space 132(1). In another example, first representation space 132(1) and second representation space 132(2) may have the same overall dimensionality, but one or more specified dimensions in first representation space 132(1) are not collinear with one or more specified dimensions in second representation space 132(2). In some implementations, representation spaces 132 may share one or more common dimensions or may not be completely contiguous.

[0059] The systems and techniques described herein are described with respect to images of human hands. These systems and techniques may be used with other forms of data, other types of objects, etc. For example, these techniques may be used in facial recognition systems, object recognition systems, etc.

[0060] 2 illustrates at 200 a method for processing training input data to determine transformer training data 152, according to some embodiments. The preparation of the transformer training data 152 may be performed by one or more computing devices 106. The transformer training data 152 is obtained for use in training the transformer module 150, which training is represented as weight data 154.

[0061] Illustrated is training input data 202. The training input data 202 may include one or more of actual input data 204 with associated label data 240 or synthetic input data 206 with associated label data 240. The actual input data 204 may include actual input image data 112 obtained from individuals who have opted in to provide training data. In one embodiment, the training input data 202 may exclude individuals who have registered to use the system for identification purposes. In another embodiment, some registered users may opt in to explicitly allow input image data 112 obtained during registration to be stored as actual input data 204 for later training.

[0062] The synthetic input data 206 may include synthetic data that matches the expected input image data 112. For example, the synthetic input data 206 may include output from a generative adversarial network (GAN) trained to generate synthetic images of a user's hands. In some implementations, the synthetic data may be based on the actual input data 204. In other implementations, other techniques may be used to determine the synthetic data 206.

[0063] The label data 240 may include information such as a sample identifier (ID) 242, a modality label 244, and a model label 246. The sample ID 242 indicates a particular training identity. The modality label 244 indicates whether the associated input data represents a first modality, a second modality, etc. The model label 246 may indicate the embedded network module 120 used to determine the training representation data 220, as described below.

[0064] The training input data 202 is processed by at least two embedded models implemented in two embedded network modules. In the following examples, the first embedded network module 120(1) may be considered the “old” or “existing” embedded network module 120, and the second embedded network module 120(2) may be considered the “new” or “updated” embedded network module 120. For these examples, it may be assumed that the first embedded network module 120(1) will be degraded and discontinued for use as an embedded network module 120 at some future point, after which the second embedded network module 120(2) will be used.

[0065] The first embedding network module 120(1) is used to process input data from the training input data 202 to generate first training representation data 220(1) within the first representation space 132(1). In some implementations, the first training representation data 220(1) includes or is based on intermediate layer data 210 and embedding layer data 212. The intermediate layer data 210 may include values ​​associated with one or more layers of the first embedding network module 120(1) while processing the input. The embedding layer data 212 includes representation data provided by the output of the first embedding network module 120(1). In one implementation, the intermediate layer data 210 may include values ​​of the penultimate layer of the neural network of the first embedding network module 120(1). The penultimate layer may include the layer preceding the final output of the embedding layer data 212. In one implementation, the intermediate layer data 210 may include values ​​of the fully connected linear layer preceding the output of the embedding layer data 212. For example, the embedding layer data 212 may have vectors of size 128, while the intermediate layer data 210 has vectors of size 1280.

[0066] Continuing with the above embodiment, the first training representation data 220(1) may include a concatenation of the intermediate layer data 210 and the embedded layer data 212. In other embodiments, the intermediate layer data 210 and the embedded layer data 212 may be combined in other ways.

[0067] In some implementations, the use of intermediate layer data 210 significantly improves the overall performance of the system.

[0068] The same training input data 202 is also processed by a second embedding network module 120(2) to generate second training representation data 220(2). This pair of training representation data 220(1) and 220(2) may be associated with each other by a common value of sample ID 242. Thus, the pair represents the same input data from training input data 202, as represented in two different representation spaces 132. Each instance of training representation data 220 may have associated label data 240. This associated label data 240 may include a model label 246 that indicates the embedding network module 120 used to generate the particular training representation data 220.

[0069] The transformer training data 152, including the first training representation data 220(1), the second training representation data 220(2), and the associated or implied label data 240, may not be used to train the transformer network module 310 within the transformer module 150, as described below.

[0070] 3 illustrates a transformer module 150 during training, according to some embodiments. The transformer module 150 may be implemented by one or more computing devices 106. The transformer module 150 includes a transformer network module 310, a classification module 312, a similarity loss module 314, and a divergence loss module 316.

[0071] The transformer network module 310 may include a neural network. During training, the transformer network module 310 receives as input first training representation data 220(1) associated with the first representation space 132(1) and generates as output first transformed representation data 156. As training progresses, the quality of the resulting first transformed representation data 156 may be expected to improve, as described below, by the returned loss value 360.

[0072] The first transformed representation data 156 is processed by a first classification module 312(1) to determine a first classification loss 342. In one implementation, the classification module 312 may utilize a HyperSpheical loss function, as illustrated with respect to Equations 1 and 2. In other implementations, other classification loss functions may be used. For example, other classification functions such as Softmax, Cosine, AM-Softmax, Arcface, large margin cosine loss, etc. may be used.

[0073] The HyperSpheical loss (HSL) function may also be used during training of the embedding network of the embedding network module 120. The HSL loss minimizes L, which is the sum of a cross-entropy term and a regularization term for regularizing the confidence scores (weighted by λ). j denotes the classifier weight for the jth class. C is the total number of training classes. M is the mini-batch size. m in these equations is a fixed angular margin.

number

number

[0074] The second training representation data 220(2) is processed by a second classification module 312(2) to determine a second classification loss 348. The second classification module 312(2) may utilize the same loss function as the first classification module 312(1). For example, the second classification module 312(2) may utilize a HyperSpherical loss function.

[0075] The similarity loss module 314 receives as input the first transformed expression data 156 and the second training expression data 220(2) and determines a similarity loss 344.

[0076] In one embodiment, the similarity loss module 314 may implement a mean squared error (MSE) and cosine distance loss function. In other embodiments, other loss functions may be used. For example, an MSE loss may be used.

[0077] The divergence loss module 316 receives as input the first classification loss 342 and the second classification loss 348 and determines the divergence loss 346. In one implementation, the divergence loss module 316 may implement a Kullback-Leibler divergence (KLD) function.

[0078] Loss value(s) 360, including one or more of the first classification loss 342, second classification loss 348, similarity loss 344, or divergence loss 346, are then returned to the transformer network module 310 for subsequent iterations during training.

[0079] 4 illustrates, at 400, transforming representation data to achieve compatibility with another hardware configuration 192, according to some embodiments. In some embodiments, the described operations may be performed using one or more of an input device 104, a computing device 106, or other devices. The diagram illustrates a first, second, and third time, each separated by a dotted line.

[0080] The first time, first input image data 112(1) of user “Alex” is obtained using a first input device 104(1) associated with a first hardware configuration 192(1). For example, registration module 118 may be used to register user “Alex.” First hardware configuration data 194(1) may also be associated with the first input image data 112(1). The first input image data 112(1) is processed by first embedded network module 120(1) to determine first representation data 134(1) representing Alex in a first representation space 132(1). The first representation data 134(1) is a native representation 196 for the first representation space 132(1).

[0081] In some implementations, first representation data 134(1) includes or is based on intermediate layer data 210 and embedding layer data 212. Intermediate layer data 210 may include values ​​associated with one or more layers of first embedding network module 120(1) while processing input image data 112(1). Embedding layer data 212 includes representation data provided by the output of first embedding network module 120(1). In one implementation, intermediate layer data 210 may include values ​​of the penultimate layer of the neural network of first embedding network module 120(1). The penultimate layer may include a layer preceding the final output of embedding layer data 212. In one implementation, intermediate layer data 210 may include values ​​of a fully connected linear layer preceding the output of embedding layer data 212. First representation data 134(1) may include a concatenation of intermediate layer data 210 and embedding layer data 212. In other implementations, the intermediate layer data 210 and the embedded layer data 212 may be combined in other ways.

[0082] 3, is used to convert or translate the first representation data 134(1) into first transformed representation data 156(1), which is in the second representation space 132(2). This first transformed representation data 156(1) may be stored in the enrolled user data 140 along with the associated identification data 142 of “Alex.”

[0083] At the second time, registered user data 140 for user “Alex” may include Alex’s first expression data 134(1), first transformed expression data 156(1), and associated identification data 142 (not shown).

[0084] A third time, a second input device 104(2) associated with a second hardware configuration 192(2) is used to obtain second input image data 112(2) of a user “Alex” who wishes to be identified by system 100. Second hardware configuration data 194(2) may also be associated with second input image data 112(2). The second input image data 112(2) is processed by second embedded network module 120(2) to determine second representation data 134(2) representing Alex in second representation space 132(2). The second representation data 134(2) is a native representation 196(2) for second representation space 132(2). Due to the previous actions of converter module 150(1) in the second time, second representation space 132(2) now includes first transformed representation data 156(1). For the second representation space 132(2), the first transformed representation data 156(1) is transformed representation 198.

[0085] During operation of system 100, comparison module 160 may compare second expression data 134(2) representing a query with previously stored first transformed expression data 156(1). If comparison module 160 determines that second expression data 134(2) and first transformed expression data 156(1) correspond to the same user, second expression data 134(2) may be stored as registered user data 140 in second representation space 132(2).

[0086] Here, registered user data 140 includes first expression data 134(1) for first expression space 132(1) and second expression data 134(2) for second expression space 132(2).

[0087] In other alternative embodiments (not shown), another converter module 150 may be used to process second representation data 134(2) and provide second transformed representation data 156(2) for inclusion in first representation space 132(1). This bidirectionality between representation spaces 132 may be used to provide continuous updates to the representation data 134 stored therein. As a result, system 100 may experience improved performance while adapting to changes in the user. For example, as the appearance of a user's hand 102 changes over time, the exchange of transformed representation data 156 may facilitate continued user operability with the involved hardware configuration 192.

[0088] In some implementations, the operations of the converter module 150 may be performed on-demand. In other implementations, the converter module 150 may operate to process the expression data 134 in batches. Thus, such conversion may be completed before a query utilizing the converted expression data 156 is received. This implementation may reduce or eliminate latency in asserting identity that may result from an online or on-demand conversion process.

[0089] The implementation described with respect to FIG. 4 determines transformed expression data 156 that may be stored as registered user data 140 for later use, such as by comparison module 160 .

[0090] In the implementation described with respect to FIG. 5, the native representation 196 (if available) of the registered user data 140 may be compared with the native representation of the query expression data 134(1) and the query transformed expression data 156.

[0091] 5 illustrates a query using native and transformed representation data at 500, according to some implementations. In some implementations, the operations described may be performed using one or more of the input device 104, the computing device 106, or other devices.

[0092] In this illustration, query input image data 112 is obtained using an input device 104(1) associated with a first hardware configuration 192(1). First hardware configuration data 194(1) may also be associated with the query input image data 112. The input image data 112 is processed by a first embedded network module 120(1) to determine first query expression data 134(1) in a first representation space 132(1). The query first expression data 134(1) is a native representation 196 for the first representation space 132(1).

[0093] The management module 502 may coordinate the operation of one or more converter modules 150(1)-(T). The management module 502 may operate based on compatibility matrix data 504. The compatibility matrix data 504 may include information indicating which hardware configurations 192 and converter modules 150 should be used to maintain compatibility of multiple configurations. The compatibility matrix data 504 is described in further detail in FIG. 6.

[0094] Based on the compatibility matrix data 504, the management module 502 determines that the first hardware configuration 192(1) should maintain compatibility with the second hardware configuration 192(2) associated with the second representation space 132(2). The query expression data 134(1) is processed by the converter module 150(2) to determine the query first transformed expression data 156(1). The query first transformed expression data 156(1) is associated with the second representation space 132(2).

[0095] The first query expression data 134(1) may be provided to a comparison module 160. The comparison module 160 may compare the first query expression data 134(1) with registered user data 140 associated with the first expression space 132(1), such as the first expression data 134(1).

[0096] Query first transformed expression data 156(1) may also be provided to comparison module 160. Comparison module 160 may compare query first transformed expression data 156(1) with registered user data 140 associated with second expression space 132(2), such as second expression data 134(2).

[0097] Comparison module 160 may then provide as output asserted identity data 164 indicating the asserted identity of the user and their hand 102 presented to input device 104(1). In some implementations, comparison module 160 may also use transformed expression data 156.

[0098] In some implementations, comparison module 160 may perform two or more "cross" comparisons to determine asserted identity data 164. If at least a threshold portion of the cross-comparisons indicate a correspondence above the threshold, identity may be asserted. For example, each instance of query transformed expression data 156 may be compared to each instance of previously stored expression data 134 associated with the same expression space 132, and query expression data 134 may be compared to each instance of previously determined transformed expression data 156. Continuing the example, if both sets of comparisons all indicate respective matches, identity corresponding to all matches may be asserted.

[0099] 6 illustrates compatibility matrix data 504 at 600, which indicates hardware configurations and the respective representation data to be converted, according to some embodiments. The compatibility matrix data 504 may be stored in memory 820 of one or more of the devices described herein. The compatibility matrix data 504 is illustrated as a table for ease of explanation and not necessarily as a limitation. In other embodiments, other data structures may be used to store the compatibility matrix data 504.

[0100] In this figure, various physical configurations 602(1), 602(2), 602(3), 602(4)... 602(P) are depicted along with associated hardware configurations 192(1), 192(2), 192(3), 192(4), 192(5)... 192(N). As previously mentioned, a single physical configuration 602 of input device 104 may be associated with multiple hardware configurations 192. For example, the same physical configuration 602(4) is associated with both hardware configurations 192(4) and 192(5). Continuing with this example, hardware configuration 192(4) may be associated with a first lighting mode, such as low light / nighttime operation, while hardware configuration 192(5) is associated with a second lighting mode, such as bright light / daytime operation.

[0101] Also shown are a first representation space 132(1), a second representation space 132(2), a fourth representation space 132(4), a fifth representation space 132(5), ..., 132(S). Compatibility attributes may be specified for various combinations of hardware configurations 192 and representation spaces 132. These may include a null entry to indicate incompatibility; "native," which indicates that this is the native representation space 132 and therefore no conversion is required; "convert," which indicates that the corresponding converter module 150 is used to generate converted representation data 156 in this space; and "no conversion," which indicates that no conversion is performed. The "no conversion" option may be utilized in situations where compatibility is undesirable. For example, incompatibility may be acceptable in situations where the resulting operation using the converted representation data 156 would cause the performance of the system 100 to fall below a specified threshold.

[0102] The compatibility supported by system 100 may be asymmetric. For example, in this illustration, hardware configuration 192(3) is compatible with first representation space 132(1) (associated with first hardware configuration 192(1)), but is incompatible with second representation space 132(2) associated with second hardware configuration 192(2).

[0103] 7 is a flow diagram 700 for registering with a first hardware configuration 192(1) and backfilling expression data to a second hardware configuration 192(2), according to some embodiments. In some embodiments, the described operations may be performed using one or more of an input device 104, a computing device 106, or other devices.

[0104] At 702, first input image data 112(1) is determined. For example, the first input image data 112(1) associated with one or more modalities may be acquired using a first input device 104(1) associated with a first hardware configuration 192(1). Continuing with this example, the first input image data 112(1) may include first modality image data 114 and second modality image data 116.

[0105] At 704, a first hardware configuration 192(1) associated with the first input image data 112(1) is determined. For example, the first input device 104(1) may transmit hardware configuration data 194(1) indicating the first hardware configuration 192(1).

[0106] At 706, first representation data 134(1) in first representation space 132(1) is determined based on first input image data 112(1). For example, first input image data 112(1) is processed by first embedded network module 120(1) to determine first representation data 134(1). In some implementations, the selection of the embedded network module 120 to be used may be based on hardware configuration data 194.

[0107] At 708, first identification data 142(1) associated with first expression data 134(1) is determined. For example, during the enrollment process, enrollment module 118 may obtain first identification data 142(1) from a user who presented the hand 102 associated with first input image data 112(1).

[0108] At 710, first enrolled user data 140 is determined based on first expression data 134(1) and first identification data 142(1). For example, first expression data 134(1) may be associated with first identification data 142(1).

[0109] At 712, the second representation space 132 or the second hardware configuration 192(2) is determined to be associated with registered user data 140. For example, the registered user data 140 may be associated with a particular set of compatibility matrix data 504 that specifies hardware among a specified set of hardware configurations 192.

[0110] At 714, first transformed expression data 156(1) is determined based on the first expression data 134(1) associated with the first representation space 132(1). For example, the first expression data 134(1) may be processed by the first converter module 150(1) to determine the first transformed expression data 156(1). The first transformed expression data 156(1) is associated with the second representation space 132(2). The first transformed expression data 156(1) may be associated with the enrolled user data 140. For example, the first transformed expression data 156(1) may be associated with the identification data 142 determined above.

[0111] In some implementations, the selection of the converter module(s) 150 to be used may be based on one or more of the hardware configuration data 194 or the compatibility matrix data 504 .

[0112] At 716, second input image data 112(2) is determined. For example, a second input device 104(2) associated with a second hardware configuration 192(2) may be used to acquire the second input image data 112(2) associated with one or more modalities. Continuing with this example, the second input image data 112(2) may include second modality image data 116 and third modality image data acquired using an RGB camera.

[0113] At 718, a second hardware configuration 192(2) associated with the second input image data 112(2) is determined. For example, the second input device 104(2) may transmit hardware configuration data 194(2) indicating the second hardware configuration 192(2).

[0114] At 720, second representation data 134(2) in second representation space 132(2) is determined based on the second input image data 112(2). For example, the second input image data 112(2) is processed by second embedded network module 120(2) to determine second representation data 134(2). In some implementations, the selection of the embedded network module 120 to be used may be based on hardware configuration data 194.

[0115] At 722, based on a comparison of second expression data 134(2) and first transformed expression data 156(1), it is determined that second expression data 134(2) is associated with first identification data 142. For example, comparison module 160 may determine asserted identification data 164 indicating that hand 102 represented in second input image data 112(1) is associated with hand 102 represented in first input image data 112(2).

[0116] At 724, first enrolled user data 140 may be determined based on second expression data 134(2). For example, second expression data 134(2) may be stored in enrolled user data 140 as the user's native expression 196.

[0117] 8 is a block diagram of a computing device 106 for implementing the system 100, according to some embodiments. The computing device 106 may be within the input device 104, may include a server, etc. The computing device 106 may be physically present at a facility, accessible over a network, or a combination of both. The computing device 106 does not require end-user knowledge of the physical location and configuration of the system that delivers the service. Common terms associated with the computing device 106 may include “embedded system,” “on-demand computing,” “software as a service (SaaS),” “platform computing,” “network-accessible platform,” “cloud services,” “data center,” etc. The services provided by the computing device 106 may be distributed across one or more physical or virtual devices.

[0118] One or more power sources 802 may be configured to provide adequate power to operate components within the computing device 106. The one or more power sources 802 may include batteries, capacitors, fuel cells, solar cells, wireless power receivers, conductive couplings suitable for attachment to a power source such as that provided by a power utility, and the like. The computing device 106 may include one or more hardware processors 804 configured to execute one or more stored instructions. The processors 804 may include one or more cores. One or more clocks 806 may provide information indicating dates, times, instants, and the like. For example, the processor 804 may use data from the clock 806 to associate a particular interaction with a particular point in time.

[0119] Computing device 106 may include one or more communication interfaces 808, such as an input / output (I / O) interface 810 and a network interface 812. The communication interface 808 allows computing device 106 or components thereof to communicate with other devices or components. The communication interface 808 may include one or more I / O interfaces 810. The I / O interfaces 810 may include an Integrated Circuit (I2C), a Serial Peripheral Interface Bus (SPI), a Universal Serial Bus (USB) developed by the USB Implementers Forum, RS-232, etc.

[0120] The I / O interface(s) 810 may couple to one or more I / O devices 814. The I / O devices 814 may include input devices such as one or more of sensors 816, a keyboard, a mouse, an input device, etc. The I / O devices 814 may also include output devices 818, such as one or more of a display device, a printer, an audio speaker, etc. In some embodiments, the I / O devices 814 may be physically integrated with the computing device 106 or may be external. The sensors 816 may include a camera 108, a smart card reader, a touch sensor, a microphone, etc.

[0121] The network interface 812 may be configured to facilitate communication between the computing device 106 and other devices, such as routers and access points. The network interface 812 may include devices configured to couple to a personal area network (PAN), a local area network (LAN), a wireless local area network (WLAN), a wide area network (WAN), etc. For example, the network interface 812 may include devices compatible with Ethernet, Wi-Fi, Bluetooth, etc.

[0122] Computing device 106 may also include one or more buses or other internal communication hardware or software that allow data to be transferred between various modules and components of computing device 106 .

[0123] 8, computing device 106 includes one or more memories 820. Memory 820 may include one or more non-transitory computer-readable storage media (CRSMs). CRSMs may be any one or more of electronic, magnetic, optical, quantum, and mechanical computer storage media. Memory 820 provides storage of computer-readable instructions, data structures, program modules, and other data for operation of computing device 106. While some example functional modules are illustrated as stored in memory 820, the same functionality may alternatively be implemented in hardware, firmware, or as a system-on-chip (SoC).

[0124] Memory 820 may include at least one operating system (OS) module 822. OS module 822 is configured to manage hardware resource devices, such as I / O interface 810, I / O devices 814, and communication interface 808, and to provide various services to applications or modules executing on processor 804. OS module 822 may implement a variant of the FreeBSD operating system as promulgated by the FreeBSD Project, other UNIX or UNIX-like variants, a variant of the Linux operating system as promulgated by Linus Torvalds, and the Windows operating system from Microsoft Corporation of Redmond, Washington, USA, etc.

[0125] The communications module 826 may be configured to establish communications with the computing device 106, a server, other computing devices 106, or other devices. The communications may be authenticated, encrypted, or the like.

[0126] Also stored in memory 820 may be a data store 824 and one or more of the following modules, which may run as foreground applications, background tasks, daemons, and the like. Data store 824 may use flat files, databases, linked lists, trees, executable code, scripts, or other data structures for storing information. In some implementations, data store 824 or portions of data store 824 may be distributed across one or more other devices, including computing devices 106 and network-attached storage devices, and the like.

[0127] The data store 824 may store one or more of the transformer training data 152, the trained weight data 154, the registered user data 140, the query expression data 162, the compatibility matrix data 504, etc. The memory 820 may store the embedded network module(s) 120, the transformer module(s) 150, the management module 502, the comparison module 160, the facility management module 166, etc.

[0128] In some implementations, the input image data 112 may be temporarily stored during processing by the embedded network module 120. For example, the input device 104 may obtain the input image data 112, determine the expression data 134 based on the input image data 112, and then clear the input image data 112. The resulting expression data 134 may then be transmitted to a server or other computing device 106 for performing enrollment, comparisons to assert identity, etc.

[0129] For example, facility management module 166 may perform various functions such as tracking items between different inventory locations or carts, generating restock orders, directing the operation of robots within the facility, and associating particular user identities with users within the facility using asserted identification data 164. During operation, facility management module 166 may access input image data 112 or sensor data 832, such as data from other sensors.

[0130] Information used by facility management module 166 may be stored in data store 824. For example, data store 824 may be used to store physical layout data 830, sensor data 832, query aggregate expression data 704, asserted identity data 164, user location data 836, interaction data 838, etc. For example, sensor data 832 may include input image data 112 obtained from input devices 104 associated with the facility.

[0131] Physical layout data 830 may provide information indicating where input devices 104, cameras, weight sensors, antennas for wireless receivers, inventory locations, etc. are located relative to one another within a facility. For example, physical layout data 830 may include information representing a map or floor plan of a facility with the relative locations of gates and inventory locations with input devices 104.

[0132] Facility management module 166 may generate user location data 836 that indicates the user's location within the facility. For example, facility management module 166 may use image data captured by a camera to determine the user's location. In other implementations, other techniques may be used to determine user location data 836. For example, data from a smart floor may be used to determine the user's location.

[0133] Identification data 142 may be associated with user location data 836. For example, a user enters a facility and scans their hand 102 with an input device 104, resulting in asserted identification data 164 associated with the time of entry and the location of the input device 104. Tracking data indicating the user's path beginning from the location of the input device 104 at the time of entry may associate user location data 836 with the user identifier in asserted identification data 164.

[0134] A particular interaction may be associated with a particular user's account based on user location data 836 and interaction data 838. For example, if user location data 836 indicates a user's presence in front of inventory location 892 at a time of 09:02:02, and interaction data 838 indicates a selection of a quantity of one item from an area on inventory location 892 at 09:04:13, the user may be charged a fee for the selection.

[0135] Facility management module 166 may use sensor data 832 to generate interaction data 838. Interaction data 838 may include information regarding the type of items involved, the number involved, whether the interaction was a pick or place, and so forth. Interactions may include a user picking an item from an inventory location, placing an item in an inventory location, touching an item at an inventory location, searching for an item at an inventory location, and so forth. For example, facility management module 166 may generate interaction data 838 indicating which items a user selected from a particular lane on a shelf and may then use this interaction data 838 to adjust the total number of inventory contained in that lane. Interaction data 838 may then be used to charge a fee to an account associated with a user identifier associated with the user who selected the item.

[0136] The facility management module 166 may process the sensor data 832 and generate output data. For example, based on the interaction data 838, the amount of a certain type of item at a particular inventory location may fall below a threshold restocking level. The system may generate output data including a restocking order indicating the inventory location, area, and quantity needed to replenish the stock to a predetermined level. The restocking order may then be used to instruct a robot to restock that inventory location.

[0137] Other modules 840 may also reside in memory 820, and other data 842 may reside in data store 824. For example, a billing module may use interaction data 838 and asserted identity data 164 to bill an account associated with a particular user.

[0138] The devices and techniques described in this disclosure may be used in a variety of other settings. For example, system 100 may be used in conjunction with a point-of-sale (POS) device. A user may present hand 102 to input device 104 to provide an indication of intent and authorization to pay via an account associated with asserted identification data 164. In another example, a robot may incorporate input device 104. The robot may use asserted identification data 164 to determine whether to deliver a package to a user and, based on asserted identification data 164, to determine which package to deliver.

[0139] Although the input to system 100 is discussed with respect to image data, the system may be used with other types of input. For example, the input may include data obtained from one or more sensors, data generated by another system, etc. For example, instead of image data generated by camera 108, the input to system 100 may include an array of data. Other modalities may be used. For example, the first modality may be visible light, the second modality may be sonar, etc.

[0140] Although system 100 is discussed with respect to processing biometric data, the system may be used with other types of data. For example, input may include satellite weather imagery, weather data, product images, data indicative of chemical composition, etc. For example, instead of image data generated by camera 108, input to system 100 may include an array of data.

[0141] The processes discussed herein may be implemented in hardware, software, or a combination thereof. In the software context, the described operations represent computer-executable instructions stored on one or more non-transitory computer-readable storage media that, when executed by one or more processors, perform the described operations. Generally, computer-executable instructions include routines, programs, objects, components, and data structures that perform particular functions or implement particular abstract data types. Those skilled in the art will readily recognize that particular steps or operations depicted in the above figures may be removed, combined, or performed in an alternative order. Any steps or operations may be performed serially or in parallel. Furthermore, the order in which operations are described is not intended to be limiting.

[0142] Embodiments may be provided as a software program or computer program product that includes a non-transitory computer-readable storage medium having stored thereon instructions (in compressed or uncompressed format), which can be used to program a computer (or other electronic device) to perform the processes or methods described herein. The computer-readable storage medium may be one or more of an electronic storage medium, a magnetic storage medium, an optical storage medium, a quantum storage medium, and the like. For example, the computer-readable storage medium may include, but is not limited to, a hard drive, an optical disk, a read-only memory (ROM), a random access memory (RAM), an erasable programmable ROM (EPROM), an electronically erasable programmable ROM (EEPROM), a flash memory, a magnetic or optical card, a solid-state memory device, or any other type of physical medium suitable for storing electronic instructions. Furthermore, embodiments may also be provided as a computer program product that includes a transitory machine-readable signal (in compressed or uncompressed format). Examples of transitory machine-readable signals include signals that can be configured to be accessed by a computer system or machine that hosts or executes a computer program, including, but not limited to, signals transmitted over one or more networks, whether modulated or unmodulated with a carrier. For example, a transitory machine-readable signal may include the transmission of software over the Internet.

[0143] The separate instances of those programs may be running on or distributed across any number of separate computer systems. Thus, while particular steps have been described as being performed by particular devices, software programs, processes, or entities, this need not be the case, and various alternative implementations will be appreciated by those skilled in the art.

[0144] Additionally, those skilled in the art will readily recognize that the above-described techniques may be utilized in a variety of devices, environments, and situations. Although the subject matter has been described in language specific to structural features or methodological acts, it will be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claims.

[0145] Embodiments of the present disclosure can be described in light of the following clauses.

[0146] (Article 1) a memory storing first computer-executable instructions; Determining first input image data acquired using a first input device associated with the first hardware configuration; determining first representation data based on processing the first input image data with the first embedding model, the first representation data being associated with a first representation space; determining first transformed representation data based on processing the first representation data with a first transformer network, the first transformed representation data being associated with a second representation space; and a hardware processor that executes first computer-executable instructions to: determine first data indicating an association between the first representation data and the first transformed representation data.

[0147] (Clause 2) The hardware processor: determining first identification data, The system described in clause 1, wherein the first data executes first computer-executable instructions to determine whether the first identification data is associated with the first expression data and the first converted expression data.

[0148] (Article 3) The hardware processor: determining that second data indicating a correspondence between the first representation data associated with the first representation space and the first enrolled user data is less than a threshold; determining that third data indicating a correspondence between the first transformed expression data associated with the second expression space and the first registered user data is greater than or equal to a threshold value; The system described in clause 1 or 2 executes first computer-executable instructions to: determine, based on the third data, that first registered user data is associated with the first input image data.

[0149] (Article 4) The hardware processor: The system of any one of clauses 1 to 3, wherein the system executes first computer-executable instructions to: store first expression data in association with first registered user data.

[0150] (Article 5) The hardware processor: Determining second input image data acquired using a second input device associated with a second hardware configuration; determining second representation data based on processing second input image data with a second embedding model, the second representation data being associated with a second representation space; and determining that second data indicating a correspondence between the second expression data and the first transformed expression data is greater than or equal to a threshold; The system described in any of clauses 1 to 4 executes first computer-executable instructions to perform the following: and determine third data indicating an association between the first expression data and the second expression data.

[0151] (Article 6) a first input device associated with the first hardware configuration acquires image data using a first lighting mode; and 6. The system of any of clauses 1 to 5, wherein the second representation space is associated with a first input device using a second lighting mode.

[0152] (Article 7) the first input device associated with the first hardware configuration includes a dedicated biometric input device; and The system of any of clauses 1-6, wherein the second representation space is associated with a second hardware configuration including one of a smartphone device, a tablet device, a laptop computer, a home security device, or a desktop computer.

[0153] (Article 8) Determining first input image data acquired using a first input device associated with the first hardware configuration; determining first representation data based on processing the first input image data with the first embedding model, the first representation data being associated with a first representation space; determining first transformed representation data based on processing the first representation data with a first transformer network, the first transformed representation data being associated with a second representation space; determining first data indicative of an association between the first representation data and the first transformed representation data.

[0154] (Article 9) determining the first identification data; and the first data associates the first identification data with the first representation data and the first transformed representation data; The method described in clause 8.

[0155] (Article 10) determining that second data indicating a correspondence between the first representation data associated with the first representation space and the first enrolled user data is less than a threshold; determining that third data indicating a correspondence between the first transformed expression data associated with the second expression space and the first registered user data is greater than or equal to a threshold value; 10. The method of claim 8 or 9, further comprising: determining, based on the third data, that the first registered user data is associated with the first input image data.

[0156] (Article 11) 11. The method of any of clauses 8 to 10, further comprising storing the first expression data in association with the first registered user data.

[0157] (Article 12) Determining second input image data acquired using a second input device associated with a second hardware configuration; determining second representation data based on processing second input image data with the first embedding model, the second representation data being associated with a second representation space; and determining that second data indicating a correspondence between the second expression data and the first transformed expression data is greater than or equal to a threshold; 12. The method of any of clauses 8-11, further comprising: determining third data indicative of an association between the first expression data and the second expression data.

[0158] (Article 13) the first representation space is associated with input image data representing images acquired using one or more of a first set of wavelengths of light or a first illumination mode; and 13. The method of any of clauses 8 to 12, wherein the second representation space is associated with input image data representing images acquired using one or more of a second set of wavelengths of light or a second illumination mode.

[0159] (Article 14) a memory storing first computer-executable instructions; determining first representation data based on processing first input image data with a first embedding model, the first representation data being associated with a first representation space; determining first transformed representation data based on processing the first representation data with a first transformer network, the first transformed representation data being associated with a second representation space; and a hardware processor that executes first computer-executable instructions to: determine first data indicating an association between the first representation data and the first transformed representation data.

[0160] (Article 15) The hardware processor shall: determining first enrolled user data based on one or more of the first expression data or the first transformed expression data; determining a first set of representation spaces based on first registered user data; The system described in clause 14 executes first computer-executable instructions to: determine a set of first transformed representation data based on processing the first representation data using respective transformer networks, each transformer network being associated with a respective one of a set of first representation spaces.

[0161] (Article 16) The hardware processor shall: determining a first hardware configuration associated with the first expression data; determining a first set of representation spaces based on the first hardware configuration; The system of clause 14 or 15 executes first computer-executable instructions to: determine a set of first transformed representation data based on processing the first representation data using respective transformer networks, each transformer network being associated with a respective one of the set of first representation spaces.

[0162] (Article 17) The hardware processor shall: determining first identification data, The system of any of clauses 14 to 16, wherein the first data executes first computer-executable instructions to associate and determine the first identification data with the first representation data and the first converted representation data.

[0163] (Article 18) The hardware processor shall: determining second representation data based on processing second input image data with the second embedding model, the second representation data being associated with a second representation space; determining that second data indicating a correspondence between the second expression data and the first transformed expression data is greater than or equal to a threshold; 19. The system of any of clauses 14 to 18, wherein the system executes first computer-executable instructions to: determine third data indicating an association between the first expression data and the second expression data.

[0164] (Article 19) a first input device associated with the first hardware configuration, the first input device acquiring first input image data using one or more image sensors detecting a first set of wavelengths; a second input device associated with the second hardware configuration, the second input device acquiring second input image data using one or more image sensors detecting a second set of wavelengths; and 19. The system of any of clauses 14-18, further comprising: a hardware processor that executes first computer-executable instructions to determine second representation data based on processing second input image data using the second embedding model, wherein the second representation data is associated with a second representation space.

[0165] (Article 20) a first input device, including a smartphone device, a tablet device, a laptop computer, a desktop computer, or a home security device, the first input device acquiring first input image data; a second input device, including a smartphone device, a tablet device, a laptop computer, a desktop computer, a network-connected camera, or a home security device, the second input device acquiring second input image data; and 20. The system of any of clauses 14-19, further comprising: a hardware processor that executes first computer-executable instructions to determine second representation data based on processing second input image data using the second embedding model, wherein the second representation data is associated with a second representation space.

Claims

1. Determining first input image data acquired using a first input device associated with the first hardware configuration; determining first representation data based on processing the first input image data with a first embedding model, the first representation data being associated with a first representation space; determining first transformed representation data based on processing the first representation data with a first transformer network, the first transformed representation data being associated with a second representation space; determining first data indicative of an association between the first representation data and the first transformed representation data; Including, method.

2. determining first identification data; the first data associates the first identification data with the first expression data and the first transformed expression data; The method of claim 1.

3. determining that second data indicating a correspondence between the first expression data associated with the first expression space and first enrolled user data is less than a threshold; determining that third data indicating a correspondence between the first transformed expression data associated with the second expression space and the first registered user data is equal to or greater than the threshold value; determining, based on the third data, that the first registered user data is associated with the first input image data; 3. The method of claim 1 or 2, further comprising:

4. storing the first expression data in association with the first registered user data; The method of any one of claims 1 to 3, further comprising:

5. Determining second input image data acquired using a second input device associated with a second hardware configuration; determining second representation data based on processing the second input image data with the first embedding model, the second representation data being associated with the second representation space; determining that second data indicating a correspondence between the second expression data and the first transformed expression data is equal to or greater than a threshold; determining third data indicative of an association between the first expression data and the second expression data; The method of any one of claims 1 to 4, further comprising:

6. the first representation space is associated with input image data representing images acquired using one or more of a first set of wavelengths of light or a first illumination mode; and the second representation space is associated with input image data representing an image acquired using one or more of a second set of wavelengths of light or a second illumination mode; The method according to any one of claims 1 to 5.

7. a memory storing first computer-executable instructions; determining first representation data based on processing first input image data with a first embedding model, the first representation data being associated with a first representation space; determining first transformed representation data based on processing the first representation data with a first transformer network, the first transformed representation data being associated with a second representation space; determining first data indicative of an association between the first representation data and the first transformed representation data; a hardware processor that executes the first computer-executable instructions to: Including, system.

8. The hardware processor includes: determining first enrolled user data based on one or more of the first expression data or the first transformed expression data; determining a first set of representation spaces based on the first enrolled user data; 8. The system of claim 7, wherein the system executes the first computer-executable instructions to: determine a set of first transformed representation data based on processing the first representation data with a respective transformer network, wherein a respective transformer network is associated with a respective one of the first set of representation spaces.

9. The hardware processor includes: determining a first hardware configuration associated with the first expression data; determining a set of first representation spaces based on the first hardware configuration; 9. The system of claim 7 or 8, wherein the system executes the first computer-executable instructions to: determine a set of first transformed representation data based on processing the first representation data with a respective transformer network, wherein a respective transformer network is associated with a respective one of the first set of representation spaces.

10. The hardware processor includes: determining first identification data, 10. The system of claim 7, wherein the first data executes the first computer-executable instructions to perform the determining step of associating the first identification data with the first expression data and the first converted expression data.

11. the first representation space is associated with a first hardware configuration using a first lighting mode; and the second representation space is associated with a second hardware configuration using a second lighting mode; The system according to any one of claims 7 to 10.

12. The hardware processor includes: determining second representation data based on processing second input image data with a second embedding model, wherein the second representation data is associated with the second representation space; determining that second data indicating a correspondence between the second expression data and the first transformed expression data is equal to or greater than a threshold; and determining third data indicative of an association between the first expression data and the second expression data. A system according to any one of claims 7 to 11.

13. a first input device associated with a first hardware configuration, the first input device acquiring the first input image data using one or more image sensors detecting a first set of wavelengths; a second input device associated with a second hardware configuration, the second input device acquiring second input image data using one or more image sensors detecting a second set of wavelengths; and determining second representation data based on processing the second input image data with a second embedding model, wherein the second representation data is associated with the second representation space; the hardware processor executing the first computer-executable instructions to: The system according to any one of claims 7 to 12, further comprising:

14. a first input device, including a smartphone device, a tablet device, a laptop computer, a desktop computer, or a home security device, which acquires the first input image data; and a second input device, including a smartphone device, a tablet device, a laptop computer, a desktop computer, a network-connected camera, or a home security device, wherein the second input device acquires second input image data; and determining second representation data based on processing the second input image data with a second embedding model, the second representation data being associated with the second representation space; the hardware processor executing the first computer-executable instructions to: The system according to any one of claims 7 to 13, further comprising:

15. The system of any one of claims 7 to 14, wherein the hardware processor executes the first computer-executable instructions to store the first expression data in association with the first registered user data.

Images