Method and apparatus for identifying users based on on-device training

Abstract

A method and apparatus for identifying a user based on on-device training are provided. A method for identifying a user based on on-device training includes: performing on-device training on a feature extractor based on reference data corresponding to a generalized user and user data; determining a registration feature vector based on an output from the feature extractor in response to an input of user data; determining a test feature vector based on an output from the feature extractor in response to an input of test data; and performing user identification on a test user based on a result of comparing the registration feature vector with the test feature vector.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2019-0108199, filed on September 2, 2019, in the Korean Intellectual Property Office, and Korean Patent Application No. 10-2019-0127239, filed on October 14, 2019, in the Korean Intellectual Property Office, the disclosures of which are incorporated herein in their entirety by reference for all purposes. Technical Field

[0002] The following description relates to an on-device training-based user identification method and apparatus. Background Art

[0003] For example, technical automation of recognition has been achieved through neural network models implemented as processors of specialized computing architectures that, after extensive training, can provide computationally intuitive mappings between input patterns and output patterns. The training capability to produce such mappings can be referred to as the learning capability of the neural network. Furthermore, due to the specialized training, such a specially trained neural network can therefore have the generalization capability to generate relatively accurate outputs for, for example, input patterns for which the neural network may not have been trained. Summary of the invention

[0004] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0005] In a general aspect, an identification method includes: receiving user data for user registration input by a valid user; performing on-device training on a feature extractor based on the user data and reference data corresponding to a generalized user; determining a registration feature vector based on an output from the feature extractor in response to an input of the user data; receiving test data for user identification input by a test user; determining a test feature vector based on an output from the feature extractor in response to an input of the test data; and performing user identification on the test user based on a result of comparing the registration feature vector with the test feature vector.

[0006] The feature extractor may include a first neural network having set parameters and a second neural network having adjustable parameters. The adjustable parameters of the second neural network may be adjusted by on-device training. The first neural network may be pre-trained to extract features from input data based on a large user database (DB). The step of performing on-device training may include: assigning labels of different values ​​to user data and reference data, respectively; and performing on-device training based on a result of comparing the labels with an output from the feature extractor in response to input of the user data and the reference data.

[0007] The feature extractor may include a first neural network having set parameters and a second neural network having adjustable parameters. The step of performing on-device training may include: inputting user data to the first neural network; inputting reference data and an output from the first neural network in response to the input of the user data to the second neural network; and performing on-device training based on the output from the second neural network. The reference data may include a generalized feature vector corresponding to a generalized user. The generalized feature vector may be generated by grouping feature vectors corresponding to a plurality of generalized users into clusters.

[0008] The step of performing user identification may include: performing user identification based on a result of comparing a distance between a registration feature vector and a test feature vector with a threshold value. Determining the distance between the registration feature vector and the test feature vector based on one of a cosine distance between the registration feature vector and the test feature vector and a Euclidean distance between the registration feature vector and the test feature vector. When the registration feature vector is determined, the identification method may further include: storing the determined registration feature vector in a registered user database.

[0009] In another general aspect, a recognition method includes: obtaining a feature extractor including a first neural network having set parameters and a second neural network having adjustable parameters; performing on-device training on the feature extractor based on user data corresponding to valid users and reference data corresponding to generalized users; and when the on-device training is completed, performing user recognition on test data using the feature extractor.

[0010] In another general aspect, an on-device training method for a feature extractor disposed in a user device, the feature extractor comprising a first neural network that is pretrained and has set parameters and a second neural network having adjustable parameters, the on-device training method comprising: obtaining user data input by a valid user; inputting the user data into the first neural network; and adjusting the adjustable parameters of the second neural network by inputting preset reference data and an output from the first neural network in response to the input of the user data into the second neural network.

[0011] The reference data may include 1000 or fewer feature vectors, 500 or fewer feature vectors, or 100 or fewer feature vectors.

[0012] In another general aspect, an identification device includes: a processor; and a memory including instructions that can be executed in the processor. When the instructions are executed by the processor, the processor can be configured to: receive user data for user registration input by a valid user; perform on-device training on a feature extractor based on the user data and reference data corresponding to a generalized user; determine a registration feature vector based on an output from the feature extractor in response to the input of the user data; receive test data for user identification input by a test user; determine a test feature vector based on an output from the feature extractor in response to the input of the test data; and perform user identification on the test user based on a result of comparing the registration feature vector with the test feature vector.

[0013] In another general aspect, an identification device includes: a processor; and a memory including instructions that can be executed in the processor. When the instructions are executed by the processor, the processor can be configured to: obtain a feature extractor including a first neural network with set parameters and a second neural network with adjustable parameters; perform on-device training on the feature extractor based on user data corresponding to a valid user and reference data corresponding to a generalized user; and when the on-device training is completed, perform user identification on test data using the feature extractor.

[0014] In another general aspect, a method includes: pre-training a first neural network of a feature extractor on a server side; after the first neural network is pre-trained, setting the feature extractor to a device; training a second neural network of the feature extractor on the device using data input to the device; and performing user identification on test data input to the device using the feature extractor.

[0015] The data input to the device may include user data for user registration input by a valid user and reference data corresponding to a generalized user.

[0016] The method may include performing user identification by comparing a registration feature vector corresponding to the user data with a test feature vector corresponding to the test data.

[0017] Other features and aspects will be apparent from the following detailed description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a diagram illustrating an example of operations for user registration and user identification to be performed by an identification device.

[0019] Figure 2 is a diagram illustrating an example of processing to be performed for pre-training, user registration, and user identification.

[0020] Figure 3 is a diagram illustrating an example of pre-training.

[0021] Figure 4 is a diagram illustrating an example of operations to be performed by a recognition apparatus for on-device training and user registration.

[0022] Figure 5 is a diagram illustrating an example of on-device training.

[0023] Figure 6 is a diagram showing an example of generating a generalized user model.

[0024] Figure 7 is a diagram illustrating an example of an operation for user identification to be performed by an identification device.

[0025] Figure 8 and Fig. 9 is a diagram illustrating an example of a change in distribution of a feature vector based on on-device training.

[0026] Fig.10 is a flow chart illustrating an example of a recognition method based on on-device training.

[0027] Fig.11 is a flow chart illustrating another example of a recognition method based on on-device training.

[0028] Fig.12 is a diagram illustrating an example of a recognition apparatus based on on-device training.

[0029] Fig.13 is a diagram illustrating an example of a user device.

[0030] Throughout the drawings and detailed description, unless otherwise described or provided, the same figure reference numerals will be understood to refer to the same elements, features, and structures. The drawings may not be to scale, and the relative sizes, proportions, and depictions of the elements in the drawings may be exaggerated for clarity, illustration, and convenience. DETAILED DESCRIPTION

[0031] The following specific embodiments are provided to help the reader obtain a comprehensive understanding of the methods, devices and / or systems described herein. However, after understanding the disclosure of the present application, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be clear. For example, the order of operations described herein is only an example and is not limited to the order of operations set forth herein, but the order of operations can be changed as clearly after understanding the disclosure of the present application, except for the operations that must occur in a specific order. In addition, for greater clarity and simplicity, the description of features known after understanding the disclosure of the present application can be omitted.

[0032] The features described herein may be implemented in different forms and are not to be construed as being limited to the examples described herein. Rather, the examples described herein are provided to illustrate only some of the many possible ways to implement the methods, devices, and / or systems described herein, which will be clear after understanding the disclosure of the present application.

[0033] Throughout the specification, when an element such as a layer, region, or substrate is described as being "on," "connected to," or "bonded to" another element, the element may be directly "on," "connected to," or "bonded to" the other element, or one or more other elements may be present between them. Conversely, when an element is described as being "directly on," "directly connected to," or "directly bonded to" another element, no other elements may be present between them. As used herein, the term "and / or" includes any one of the associated listed items and any combination of any two or more.

[0034] Although terms such as "first", "second", and "third" may be used herein to describe various members, components, regions, layers, or portions, these members, components, regions, layers, or portions are not limited by these terms. Instead, these terms are only used to distinguish one member, component, region, layer, or portion from another member, component, region, layer, or portion. Therefore, without departing from the teachings of the examples described herein, the first member, first component, first region, first layer, or first portion referred to may also be referred to as the second member, second component, second region, second layer, or second portion.

[0035] The terms used herein are only used to describe various examples and are not intended to limit the present disclosure. Unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. The terms "comprise", "include" and "have" indicate the presence of the features, quantities, operations, components, elements and / or combinations thereof set forth, but do not exclude the presence or addition of one or more other features, quantities, operations, components, elements and / or combinations thereof.

[0036] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those commonly understood by those skilled in the art to which the present disclosure belongs and the meanings commonly understood after understanding the disclosure of the present application. Unless explicitly defined as such herein, terms (such as those defined in general dictionaries) will be interpreted as having the same meaning as their meanings in the context of the relevant art and the disclosure of the present application, and will not be interpreted in an idealized or overly formal manner.

[0037] Furthermore, in the description of examples, when it is considered that a detailed description of a structure or function known therefrom after understanding the disclosure of the present application will lead to an obscure interpretation of the examples, such description will be omitted.

[0038] Hereinafter, examples will be described in detail with reference to the accompanying drawings, and in the drawings, like reference numerals denote like elements throughout.

[0039] Figure 1 is a diagram showing an example of operations for user registration and user identification to be performed by an identification device. Figure 1 , the identification device 110 registers the valid user 101 in the identification device 110 based on the user data of the valid user 101. The valid user 101 may be one or more users, and one or more users may be registered in the identification device 110. The valid user 101 may be a person having the right to use the identification device 110 (for example, the owner or administrator of the device in which the identification device 110 is set or embedded). The valid user 101 may also be referred to as a genuine user. Registering the valid user 101 in the identification device 110 may be referred to as a user registration process herein. Through the user registration process, the identification information (for example, a registration feature vector) of the valid user 101 is stored in the identification device 110 or another device or apparatus associated with the identification device 110. When the valid user 101 is registered through the user registration process, the valid user 101 may then be referred to as a registered user.

[0040] The test user 102 may be an unidentified person who has not been identified, and the test user 102 attempts user identification through the identification device 110 to use the identification device 110. The test user 102 may be a valid user 101, or an imposter indicating a person who does not have the right to use the identification device 110. The identification device 110 may perform user identification on the test user 102 by comparing test data of the test user 102 with user data. Performing user identification on the test user 102 may be referred to herein as a user identification process. The user identification process may be performed after performing a user registration process.

[0041] User identification may include user verification and user identification. User verification may be performed to determine whether test user 102 is a registered user, and user identification may be performed to determine which of multiple users is test user 102. For example, when there are multiple registered users and test user 102 is one of the multiple registered users, user identification may be performed to determine one registered user corresponding to test user 102.

[0042] The result of user identification (hereinafter referred to as "identification result") may include at least one of the result of user verification (verification result) and the result of user identification (identification result). For example, when the test user 102 is a registered user, the identification device 110 may output a verification result corresponding to a successful identification. In this example, when there are multiple registered users, the identification result may include an identification result indicating which of the multiple registered users corresponds to the test user 102. However, when the test user 102 is an imposter, the identification device 110 may output a verification result corresponding to an unsuccessful identification.

[0043] The user data may be associated with the valid user 101, and the test data may be associated with the test user 102. The user data may be input into the identification device 110 by the valid user 101, the user data may be input into another device or apparatus including the identification device 110 to be transmitted to the identification device 110, or the user data may be input into another device or apparatus separate from the identification device 110 to be transmitted to the identification device 110. Similarly, the test data may be input into the identification device 110 by the test user 102, the test data may be input into another device or apparatus including the identification device 110 to be transmitted to the identification device 110, or the test data may be input into another device or apparatus separate from the identification device 110 to be transmitted to the identification device 110.

[0044] Data input to the recognition device 110 (such as user data and test data) may be referred to as input data. The input data may include voice or image. For example, in the case of speaker recognition, the input data may include voice, speech or audio. In the case of facial recognition, the input data may include a facial image. In the case of fingerprint recognition, the input data may include a fingerprint image. In the case of iris recognition, the input data may include an iris image. The recognition device 110 may perform user authentication based on at least one of such various authentication methods. The modality of each of the user data, test data, reference data and training data may correspond to at least one authentication method used by the recognition device 110. In the following, for ease of description, examples will be described with respect to speaker recognition. However, the examples may also be applied to other authentication methods besides speaker recognition.

[0045] The recognition device 110 may perform user recognition using a feature extractor 120. The feature extractor 120 includes a neural network (e.g., a first neural network 121 and a second neural network 122). At least a portion of the neural network may be implemented by software, hardware including a neural processor, or a combination thereof. For example, the neural network may be a deep neural network (DNN) including, for example, a fully connected network, a deep convolutional network, and a recurrent neural network (RNN). The DNN may include multiple layers including an input layer, at least one hidden layer, and an output layer.

[0046] A neural network may be trained to perform a given operation by mapping input data and output data that are in a nonlinear relationship based on deep learning. Deep learning may be a type of machine learning that is performed based on a large data set to solve a given problem. Deep learning may be interpreted as an optimization process that finds a point at which energy is minimized. Through supervised learning or unsupervised learning of deep learning, weights corresponding to an architecture or model of a neural network may be obtained, and input data and output data may be mapped to each other through the weights obtained as described above. Although the feature extractor 120 is Figure 1 Although shown as being located outside the recognition device 110 , the feature extractor 120 may be located inside the recognition device 110 .

[0047] The recognition device 110 may input the input data to the feature extractor 120, and in response to the input of the input data, register the user in the recognition device 110 or generate a recognition result based on the output from the feature extractor 120. In one example, the recognition device 110 may apply preprocessing to the input data, and input the input data obtained by applying the preprocessing to the feature extractor 120. Through the preprocessing, the input data may be changed into a form suitable for the feature extractor 120 to extract features therefrom. For example, when the input data corresponds to an audio wave, the audio wave may be converted into a frequency spectrum through the preprocessing.

[0048] The feature extractor 120 may output output data in response to the input of input data. The output data of the feature extractor 120 may be referred to as a feature vector herein. Optionally, the output data of the feature extractor 120 may also be referred to as an embedding vector, which indicates that the output data includes identification information of the user. In the user registration process for the valid user 101, the feature extractor 120 may output a feature vector in response to the input of user data. The output feature vector may be referred to as a registration feature vector herein, and is stored in the recognition device 110 or another device or apparatus associated with the recognition device 110 as identification information of the valid user 101. In the user identification process for the test user 102, the feature extractor 120 may output a feature vector in response to the input of test data. The output feature vector may be referred to as a test feature vector herein.

[0049] The recognition device 110 may generate a recognition result by comparing the registration feature vector with the test feature vector. For example, the recognition device 110 may determine the distance between the registration feature vector and the test vector, and generate a recognition result based on the result of comparing the determined distance with a threshold. In this example, when the determined distance is less than the threshold, the registration feature vector and the test feature vector may be represented as matching each other, and when the determined distance is not less than the threshold, the registration feature vector and the test feature vector may be represented as not matching each other.

[0050] For example, when there are multiple registered users, there may be multiple registered feature vectors for each registered user. In this example, the recognition device 110 may generate a recognition result by comparing the test feature vector with each registered feature vector. When the test feature vector matches one of the registered feature vectors, the recognition device 110 may output a recognition result corresponding to a successful recognition. The recognition result may include a recognition result associated with a registered user corresponding to the registered feature vector that matches the test feature vector. For example, the recognition result may include a recognition result associated with one of the registered users corresponding to the test user 102.

[0051] The feature extractor 120 includes a first neural network 121 and a second neural network 122. The first neural network 121 may be pre-trained or pre-trained based on a large user database (DB) (also referred to as a non-specific general user database), and the second neural network 122 may be additionally trained based on user data in a user registration process. Here, the term "pre" or "pre-" may indicate a time point before the user registration process is performed (e.g., a time point of development and production of the feature extractor 120). The large user database may correspond to a non-specific general user, and the user data may correspond to a specific user (e.g., a valid user 101). In one example, the training of the first neural network 121 may be performed by a server in the steps of development and production of the feature extractor 120, and is referred to as pre-training or first training. In addition, the training of the second neural network 122 may be performed by a device or apparatus including the recognition device 110 in a user registration process, and is referred to as on-device training or second training. Here, the "device" in the term "on-device training" may indicate a user device in which the recognition device 110 is set or embedded.

[0052] The first neural network 121 may have set parameters, and the second neural network 122 may have adjustable parameters. Parameters used herein may include weights. When the first neural network 121 is trained by pre-training, the parameters of the first neural network 121 may be set and not changed by on-device training. The parameters being set may also be described as the parameters being frozen, and the set parameters may also be referred to as frozen parameters. The parameters of the second neural network 122 may be adjusted by on-device training. The first neural network 121 may extract features from input data in a general manner, and the second neural network 122 may remap the features extracted by the first neural network 121 so that the features are specific to the user of the individual device.

[0053] In user recognition, a mismatch between training data and actual user data may result in poor recognition performance. For example, actual user data will not be used for pre-training of the first neural network 121, and thus the level of recognition performance of the feature extractor 120 including only the first neural network 121 may not be satisfactory. However, in this example, on-device training of the second neural network 122 may be performed based on actual user data, and thus may help reduce such mismatches. For example, when using a general feature extractor to which only pre-training is applied, it may not be easy to identify users with similar features (e.g., family members). However, when using the feature extractor 120 described herein, actual user data of each user may be used for on-device training, and thus, multiple users with similar features may be relatively accurately identified.

[0054] In addition, for on-device training, in addition to user data, reference data corresponding to generalized users can also be used. For example, a generalized user can be understood as a typical user or a representative user among non-specific general users. By using on-device training of user data and reference data, the feature extractor 120 can extract features from the user data that are distinguishable from the features in the reference data. As a result, it is possible to more accurately identify the feature vectors of the impostor and the feature vectors of the registered user, thereby improving the recognition performance. The on-device training using user data and reference data will be described in more detail below.

[0055] Figure 2 is a diagram showing an example of a process to be performed for pre-training, user registration, and user identification. Figure 2 , in operation 210, pre-training is performed. Pre-training may be performed based on a large user database corresponding to non-specific general users. Through pre-training, the first neural network 201 of the feature extractor 200 may be trained. Pre-training may be performed on the server side. After operation 210 is performed, the feature extractor 200 may be set or embedded in a device and distributed to users.

[0056] In operation 220, when user data is input by a valid user for user registration, on-device training is performed. In operation 230, user registration is performed. Operations 220 and 230 may be collectively referred to as a user registration process. On-device training may be performed in a user registration process. On-device training may be performed based on user data corresponding to a specific user (e.g., a valid user) and reference data corresponding to a generalized user. Through the on-device training, the second neural network 202 of the feature extractor 200 is trained. Before performing the on-device training, the second neural network 202 may be initialized by an identity matrix.

[0057] After operation 220 is performed, the feature extractor 200 may become dedicated to the registered user. In operation 230, after training on the device is completed, user data of the valid user is input to the feature extractor 200. A registered feature vector is determined based on an output from the feature extractor 200 in response to the input of the user data. When the registered feature vector is determined, the determined registered feature vector is stored in the registered user database.

[0058] In operation 240, user identification is performed. Here, operation 240 may be referred to as a user identification process. In this operation, test data for user identification input by a test user is input to the feature extractor 200, and a test feature vector is determined based on an output of the feature extractor 200 in response to the input of the test data. Based on the result of comparing the registration feature vector with the test feature vector, user identification is performed on the test user. Operations 220 to 240 may be performed by the device.

[0059] Figure 3 is a diagram showing an example of pre-training. Figure 3 , the training device 310 uses the large user database 320 to train the neural network 330 to extract features from the input data. For example, the large user database 320 may include training data associated with a plurality of non-specific general users, and a label may be assigned to each or each group of training data. The training data may include speech or images. For example, in the case of speaker recognition, the input data may include voice, speech, or audio.

[0060] The neural network 330 includes an input layer 331, at least one hidden layer 332, and an output layer 333. For example, the input layer 331 may correspond to the training data, and the output layer 333 may correspond to an activation function (such as Softmax). Through pre-training of the neural network 330, the parameters (e.g., weights) of the hidden layer 332 may be adjusted. When assigning labels, different labels may be assigned to each training data, and through pre-training based on the labels and the training data, the neural network 330 may output different output data in response to different input data. For example, different labels may be assigned to each group of training data, and through pre-training based on the labels and the training data, the neural network 330 may output different groups of output data in response to different groups of input data. This capability of the neural network 330 may be interpreted as a feature extraction function.

[0061] For example, a first label may be assigned to the first training data, and a second label may be assigned to the second training data. In this example, the neural network 330 may respond to the input of the first training data and output the first output data, and respond to the input of the second training data and output the second output data. The training device 310 may then compare the first output data with the first label, and adjust the parameters of the hidden layer 332 so that the first output data and the first label may become identical to each other. Similarly, the training device 310 may compare the second output data with the second label, and adjust the parameters of the hidden layer 332 so that the second output data and the second label may become identical to each other. The training device 310 may pre-train the neural network 330 by repeating such a process based on the large user database 320.

[0062] In one example, the training process may be performed by a batch unit. For example, a process of inputting training data to the neural network 330 and obtaining output data corresponding to the output from the neural network 330 in response to the input of the training data may be performed by the batch unit (e.g., a process of inputting a set of training data to the neural network 330 and obtaining a set of output data corresponding to the output from the neural network 330 in response to the input of the training data may be performed by the batch unit), and pre-training using the large user database 320 may be performed by repeating such a process by the batch unit.

[0063] The output layer 333 may convert the feature vector output from the hidden layer 332 into a form corresponding to the label. Through pre-training, the parameters of the hidden layer 332 may be set to values ​​suitable for the training target, and when the pre-training is completed, the parameters of the hidden layer 332 may be set or fixed. Subsequently, the output layer 333 may be removed from the neural network 330, and the first neural network of the feature extractor may be configured using the portion 340 including the input layer 331 and the hidden layer 332.

[0064] When pre-training is completed, the neural network 330 may perform a feature extraction function to output different output data in response to different input data, for example, to output different groups of output data in response to different groups of input data. This feature extraction function may exhibit maximum performance when the training data is the same as the actual data used in the user registration process and the user identification process. However, the training data and the actual data may generally be different from each other. It is only theoretically possible to reduce the mismatch between the training data and the actual data by including the actual data in the training data and performing retraining to improve recognition performance.

[0065] However, it may be necessary to train the neural network 330 using the large user database 320 until the neural network 330 has a feature extraction function, and such training may require a large amount of computing resources. Typically, a user device may have limited computing resources, and therefore, such training may be performed at a large-scale server end. Therefore, according to one example, a dual training method including pre-training and on-device training is provided. The dual training method may generate a first neural network of a feature extractor by training the neural network 330 using the large user database 320, and generate a second neural network of the feature extractor based on actual data. Therefore, the mismatch between the training data and the actual data may be reduced or minimized, and a feature extractor dedicated to the user device may be provided.

[0066] Figure 4 is a diagram showing an example of operations to be performed by a recognition device for on-device training and user registration. Figure 4 , a valid user inputs user data for user registration. The recognition device 410 performs on-device training on the feature extractor 420 based on the user data. The feature extractor 420 includes a first neural network 421 and a second neural network 422. Parameters of the first neural network 421 can be set or fixed by pre-training, and parameters of the second neural network 422 can be adjusted by on-device training. For on-device training, reference data can be used. The recognition device 410 obtains reference data from the generalized user model 430 and inputs the obtained reference data to the second neural network 422. The user data may correspond to a valid user, and the reference data may correspond to a generalized user. The generalized user model 430 will be described in more detail below.

[0067] The recognition device 410 adjusts the parameters of the second neural network 422 by assigning different labels to each or each group of user data and reference data, and comparing the labels with the output from the feature extractor 420 in response to the input of the user data and the reference data. For example, the parameters of the second neural network 422 may be adjusted so that the output of the feature extractor 420 in response to the input of the user data (i.e., the output of the second neural network 422) becomes the same as the label assigned to the user data, and the output of the feature extractor 420 in response to the input of the reference data (i.e., the output of the second neural network 422) becomes the same as the label assigned to the reference data. As described above, the recognition device 410 may train the feature extractor 420 so that the feature extractor 420 may output different feature vectors corresponding to the user data and the reference data, respectively. By training the feature extractor 420 using the user data and the reference data, the registered feature vector of the registered user may be more accurately identified, and the registered feature vector of the registered user and the feature vector of the imposter may be more accurately identified. Therefore, through on-device training, the feature extractor 420 may have a recognition capability to identify each registered user, and a verification capability to distinguish a registered user from an imposter and verify a registered user.

[0068] When the on-device training is completed, the recognition device 410 inputs the user data into the feature extractor 420, and obtains a feature vector in response to the input of the user data output from the feature extractor 420. The recognition device 410 stores the feature vector output by the feature extractor 420 as a registered feature vector in the registered user database 440. The registered feature vector can then be used for user recognition processing.

[0069] Figure 5 is a diagram showing an example of on-device training. Figure 5 , the recognition device 510 performs on-device training on the feature extractor 520 using the user data and the reference data. The user data is input to the first neural network 521, and the reference data is input to the second neural network 522. The reference data is obtained from the generalized user model 540. The recognition device 510 inputs the user data to the first neural network 521. When the first neural network 521 responds to the input of the user data and outputs a feature vector, the recognition device 510 inputs the output feature vector to the second neural network 522. The reference data can be generated using a neural network configured to perform feature extraction similar to the first neural network 521. It can be interpreted that the output from the first neural network 521 is input from the first neural network 521 to the second neural network 522 without being controlled by the recognition device 510.

[0070] The second neural network 522 can be Figure 3 The neural network 330 of is trained by performing a similar process. It can be interpreted as: Figure 3The training data is replaced by Figure 5 The feature vector corresponding to the user data and the reference vector in the example of . The second neural network 522 includes an input layer 523, at least one hidden layer 524, and an output layer 525. For example, the input layer 523 may correspond to the input data including the feature vector corresponding to the user data and the reference data, and the output layer 525 may correspond to an activation function (such as, flexible maximum). The parameters (e.g., weights) of the hidden layer 524 may be adjusted by on-device training. For example, the parameters of the hidden layer 524 may be adjusted so that the output of the feature extractor 420 in response to the input of the user data (i.e., the output of the second neural network 522) becomes the same as the label assigned to the user data, and the output of the feature extractor 420 in response to the input of the reference data (i.e., the output of the second neural network 522) becomes the same as the label assigned to the reference data. The second neural network 522 may be constructed using a portion 530 including the input layer 523 and the hidden layer 524.

[0071] A label may be assigned to each of the user data and the reference data. By training on a device based on the user data, the reference data, and different labels assigned to the respective data, the feature extractor 520 may become capable of outputting different output data in response to different user data and reference data. For example, a label may be assigned to each group of user data and reference data. By training on a device based on the user data, the reference data, and different labels assigned to the respective groups of data, the feature extractor 520 may become capable of outputting different groups of output data in response to different groups of user data and reference data. For example, the first neural network 521 may extract features from the input data in a general manner, and the second neural network 522 may remap the features extracted by the first neural network 521 so that the features become specific to the user of the individual device.

[0072] In one example, the training process may be performed by a batch processing unit. For example, a process of inputting user data and reference data to the feature extractor 520 and obtaining output data corresponding to the output from the feature extractor 520 may be performed by a batch processing unit (e.g., a process of inputting one or a group of user data and reference data to the feature extractor 520 and obtaining one or a group of output data corresponding to the output from the feature extractor 520 may be performed by a batch processing unit), and on-device training using the user data and reference data may be performed by repeating such a process via a batch processing unit. When the on-device training is completed, the parameters of the hidden layer 524 may be set or fixed. Subsequently, the output layer 525 may be removed from the second neural network 522, and the second neural network 522 may be determined or decided with the output layer 525 removed therefrom.

[0073] Through on-device training as described above, the mismatch between training data and actual data can be reduced or minimized. For example, the recognition capability of identifying registered feature vectors through user data and the verification capability of identifying or distinguishing registered feature vectors from imposter feature vectors through reference data can be improved.

[0074] Figure 6 is a diagram showing an example of generating a generalized user model. Figure 6 , input data is extracted from the large user database 610 and input to the neural network 620. For example, the neural network 620 may correspond to Figure 4 The first neural network 421 of the present invention can be used to generate a feature vector based on the input data. The large user database 610 can be used to generate a feature vector based on the input data. Figure 3 The large user database 320 may be the same or different.

[0075] exist Figure 6 In the example of FIG. 6 , the feature vectors output by the neural network 620 are indicated by small circles on the vector plane 630. These feature vectors may correspond to a plurality of generalized users included in the large user database 610, and are also referred to as basic feature vectors. As vectors representing basic feature vectors, typical feature vectors (e.g., θ 1 ,θ 2 ,…,θ c ). For example, typical feature vectors θ can be selected by grouping basic feature vectors into clusters 1 ,θ 2 ,…,θ c (For example, one or more feature vectors are selected from each of some or all of the clusters obtained by grouping as typical feature vectors). Typical feature vector θ 1 ,θ 2 ,…,θ c can correspond to a generalized user and is also called a generalized eigenvector. In addition, the typical eigenvector θ 1 ,θ 2 ,…,θ c It can be included in the generalized user model 640 as reference data and used for on-device training. For example, there may be dozens or hundreds of such typical feature vectors. For example, the number of typical feature vectors may be 1000 or less, 500 or less, or 100 or less. The typical feature vector may correspond to the data actually or truly processed by the user device through deep learning or training. For example, 10 utterances or voices may be collected from each of approximately 100,000 users, and a database including approximately one million utterances or voices may be configured. Based on the database, approximately 100 typical feature vectors may be generated.

[0076] Figure 7is a diagram showing an example of an operation to be performed by an identification device for user identification. Figure 7 , the recognition device 710 inputs the test data into the feature extractor 720. Figure 7 In the example of , the feature extractor 720 is in a state where training is completed on the device. The feature extractor 720 outputs a test feature vector in response to input of test data. The test data may be input by a test user in a user verification process. The test user may be an unidentified person who attempts to use the identification device 710 for user identification through the identification device 710. The test user may be a valid user or an imposter.

[0077] The recognition device 710 obtains a registration feature vector from the registered user database 730, performs user recognition on the test user by comparing the registration feature vector with the test feature vector, and generates a recognition result. For example, the recognition device 710 determines the distance between the registration feature vector and the test feature vector, and generates a recognition result based on the result of comparing the determined distance with a threshold. For example, the distance between the registration feature vector and the test feature vector may be determined based on the cosine distance or the Euclidean distance between the registration feature vector and the test feature vector.

[0078] Figure 8 and Fig. 9 is a diagram showing an example of a change in the distribution of feature vectors based on training on a device. Figure 8 In the example of FIG. 8 , the registration feature vectors are indicated on vector planes 810 and 820. The registration feature vectors are indicated by small circles, and small circles with the same pattern indicate registration feature vectors of the same registered user. In one example, the registration feature vectors on vector plane 810 are obtained by a feature extractor that has not been trained on the application device, and the registration feature vectors on vector plane 820 are obtained by a feature extractor that has been trained on the application device. Figure 8 As shown in , the registration feature vectors can be remapped to be specific to the registered users through on-device training. Therefore, registered users (e.g., registered users with similar features (such as family members)) can be more accurately identified from each other.

[0079] Reference Fig. 9 ,and Figure 8 Compared to vector planes 810 and 820 shown in FIG. 1 , vector planes 910 and 920 also include feature vectors of impostors. Fig. 9In the example of , the feature vector of the impostor is simply referred to as the impostor feature vector and is indicated by a six-pointed star. In this example, the registration feature vector and the impostor feature vector on the vector plane 910 are obtained by a feature extractor that has not been trained on the device, and the registration feature vector and the impostor feature vector on the vector plane 920 are obtained by a feature extractor that has been trained on the device. Fig. 9 As shown in , in addition to the registered feature vectors, the impostor feature vectors can also be remapped to be specific to the registered user through on-device training. Therefore, the registered user and the impostor can be more accurately identified or distinguished from each other, thereby more accurately verifying the registered user.

[0080] Fig.10 is a flow chart showing an example of a recognition method based on on-device training. Fig.10 In operation 1010, the recognition device receives user data for user registration input by a valid user. In operation 1020, the recognition device performs on-device training on a feature extractor based on the user data and reference data corresponding to a generalized user. In operation 1030, the recognition device determines a registration feature vector based on an output from the feature extractor in response to an input of the user data. In operation 1040, the recognition device receives test data for user identification input by a test user. In operation 1050, the recognition device determines a test feature vector based on an output from the feature extractor in response to an input of the test data. In operation 1060, the recognition device performs user identification on the test user based on a result of comparing the registration feature vector with the test feature vector. For a more detailed description of the recognition method based on on-device training, reference may be made to the above reference. Figures 1 to 9 Provide a description.

[0081] Fig.11 is a flow chart showing another example of a recognition method based on on-device training. Fig.11 In operation 1110, the recognition device obtains a feature extractor including a first neural network with set parameters and a second neural network with adjustable parameters. In operation 1120, the recognition device performs on-device training on the feature extractor based on user data corresponding to the valid user and reference data corresponding to the generalized user. In operation 1130, when the on-device training is completed, the recognition device performs user recognition using the feature extractor. For a more detailed description of the recognition method based on on-device training, reference may be made to the above reference. Figures 1 to 10 Provide a description.

[0082] Fig.121 is a diagram showing an example of a recognition device based on on-device training. The recognition device 1200 may receive input data including user data and test data, and process the operation of a neural network associated with the received input data. For example, the operation of the neural network may include user recognition. The recognition device 1200 may perform one or more or all of the operations or methods described herein with respect to processing a neural network, and provide a result of processing the neural network to a user.

[0083] See also Fig.12 , the recognition device 1200 includes at least one processor 1210 and a memory 1220. The memory 1220 may be connected to the processor 1210, and store instructions executable by the processor 1210, and data to be processed by the processor 1210 or data processed by the processor 1210. The memory 1220 may include a non-transitory computer-readable medium (e.g., a high-speed random access memory (RAM)) and / or a non-volatile computer-readable storage medium (e.g., at least one disk storage device, a flash memory device, and other non-volatile solid-state memory devices).

[0084] The processor 1210 can execute instructions to perform the above reference Figures 1 to 11 One or more or all of the operations or methods described. In one example, when the instructions stored in the memory 1220 are executed by the processor 1210, the processor 1210 may receive user data for user registration input by a valid user, perform on-device training on the feature extractor 1225 based on the user data and reference data corresponding to the generalized user, determine a registration feature vector based on an output from the feature extractor 1225 in response to the input of the user data, receive test data for user identification input by a test user, determine a test feature vector based on an output from the feature extractor 1225 in response to the input of the test data, and perform user identification on the test user based on a result of comparing the registration feature vector with the test feature vector.

[0085] In another example, when the instructions stored in the memory 1220 are executed by the processor 1210, the processor 1210 may obtain a feature extractor 1225 including a first neural network with set parameters and a second neural network with adjustable parameters, perform on-device training on the feature extractor 1225 based on user data corresponding to a valid user and reference data corresponding to a generalized user, and perform user identification using the feature extractor 1225 when the on-device training is completed.

[0086] Fig.131 is a diagram showing an example of a user device. The user device 1300 may receive input data and process the operation of a neural network associated with the received input data. For example, the operation of the neural network may include user identification. The user device 1300 may include the above reference Figures 1 to 12 The identification device described above and performs the Figures 1 to 12 Describes the operation or functionality of an identified device.

[0087] Reference Fig.13 , the user device 1300 includes a processor 1310, a memory 1320, a camera 1330, a storage device 1340, an input device 1350, an output device 1360, and a network interface 1370. The processor 1310, the memory 1320, the camera 1330, the storage device 1340, the input device 1350, the output device 1360, and the network interface 1370 may communicate with each other via a communication bus 1380. For example, the user device 1300 may include a smart phone, a tablet personal computer (PC), a laptop computer, a desktop computer, a wearable device, a smart home appliance, a smart speaker, a smart car, etc.

[0088] The processor 1310 may execute functions and instructions in the user device 1300. For example, the processor 1310 may process instructions stored in the memory 1320 or the storage device 1340. The processor 1310 may execute the above-mentioned Figures 1 to 12 One or more or all of the operations or methods described.

[0089] The memory 1320 may store information to be used for processing the operation of the neural network. The memory 1320 may include a computer-readable storage medium or a computer-readable storage device. The memory 1320 may store instructions to be executed by the processor 1310 and store relevant information when the software or application is being executed by the user device 1300.

[0090] The camera 1330 may capture still images, motion or video images, or both. The camera 1330 may capture an image of a facial area to be input by a user for facial verification. The camera 1330 may also provide a three-dimensional (3D) image including depth information of an object.

[0091] Storage device 1340 may include a computer-readable storage medium or a computer-readable storage device. Storage device 1340 may store a larger amount of information for a longer period of time than memory 1320. For example, storage device 1340 may include a magnetic hard disk, an optical disk, a flash memory, a floppy disk, and other types of non-volatile memory known in the relevant technical field.

[0092] The input device 1350 may receive input from the user through conventional input methods (e.g., keyboard and mouse) and new input methods (e.g., touch input, voice input, and image input). For example, the input device 1350 may include a keyboard, a mouse, a touch screen, a microphone, and other devices that may detect input from the user and transmit the detected input to the user device 1300. Through the input device 1350, data of the user's fingerprint, iris, voice, speech, and audio, etc. may be input.

[0093] The output device 1360 can provide output from the user device 1300 to the user through a visual channel, an auditory channel, or a tactile channel. For example, the output device 1360 may include a display, a touch screen, a speaker, a vibration generator, and other devices that can provide output to the user. The network interface 1370 can communicate with an external device through a wired network or a wireless network.

[0094] In this regard Figure 1 , Fig.12 and Fig.13The identification equipment, training equipment, user device and other equipment, devices, units, modules and other components described are implemented by hardware components. Examples of hardware components that can be used to perform the operations described in this application include, where appropriate, controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators and any other electronic components configured to perform the operations described in this application. In other examples, one or more hardware components in the hardware components that perform the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). Processors or computers can be implemented by one or more processing elements (such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field programmable gate arrays, programmable logic arrays, microprocessors or any other devices or combinations of devices configured to respond and execute instructions in a limited manner to achieve the desired result). In one example, a processor or computer includes or is connected to one or more memories storing instructions or software executed by a processor or computer. The hardware components implemented by a processor or a computer can execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) for performing the operations described in this application. The hardware components can also access, manipulate, process, create and store data in response to the execution of instructions or software. For simplicity, the singular term "processor" or "computer" can be used for the description of the examples described in this application, but in other examples, multiple processors or computers can be used, or the processor or computer can include multiple processing elements or multiple types of processing elements or both. For example, a single hardware component or two or more hardware components can be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components can be implemented by one or more processors, or a processor and a controller, and one or more other hardware components can be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller can implement a single hardware component or two or more hardware components. The hardware components may have any one or more of different processing configurations, examples of which include: a single processor, independent processors, parallel processors, single instruction single data (SISD) multiprocessing, single instruction multiple data (SIMD) multiprocessing, multiple instruction single data (MISD) multiprocessing, and multiple instruction multiple data (MIMD) multiprocessing.

[0095] exist Figures 2 to 11The method for performing the operations described in the present application shown in the embodiment is performed by computing hardware (e.g., by one or more processors or computers), which is implemented as described above as execution instructions or software to perform the operations performed by the method described in the present application. For example, a single operation or two or more operations may be performed by a single processor or two or more processors, or a processor and a controller. One or more operations may be performed by one or more processors, or a processor and a controller, and one or more other operations may be performed by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller may perform a single operation or two or more operations.

[0096] The instructions or software for controlling the processor or computer to implement the hardware components and perform the methods described above are written as computer programs, code segments, instructions or any combination thereof to individually or collectively instruct or configure the processor or computer to operate as a machine or special-purpose computer to perform the operations performed by the hardware components and methods described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) directly executed by the processor or computer. In another example, the instructions or software include high-level code executed by the processor or computer using an interpreter. Ordinary programmers in the art can easily write instructions or software based on the block diagrams and flow charts shown in the accompanying drawings and the corresponding descriptions in the specification, and the block diagrams and flow charts shown in the accompanying drawings and the corresponding descriptions in the specification disclose algorithms for performing the operations performed by the hardware components and methods described above.

[0097] The instructions or software for controlling a processor or computer to implement the hardware components and perform the methods described above, and any associated data, data files, and data structures are recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as a multimedia card or a micro card (for example, Secure Digital (SD) or Extreme Digital (XD)), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk, and any other device configured to store instructions or software and any associated data, data files and data structures in a non-transitory manner and provide the instructions or software and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the instructions.

[0098] Although the present disclosure includes specific examples, it will be clear to those skilled in the art that various changes in form and detail may be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are considered to be illustrative only and not for limiting purposes. The description of features or aspects in each example will be considered to be applicable to similar features or aspects in other examples. If the described techniques are performed in a different order, and / or if the components in the described systems, architectures, devices, or circuits are combined in different ways, and / or replaced or supplemented by other components or their equivalents, suitable results may be achieved. Therefore, the scope of the present disclosure is not limited by specific embodiments, but by the claims and their equivalents, and all changes within the scope of the claims and their equivalents should be interpreted as included in the present disclosure.

Claims

1. An identification method, comprising: receiving user data for user registration input by a valid user, wherein the valid user comprises a person having the right to use the identification device; performing on-device training on a feature extractor based on the user data and reference data corresponding to a generalized user, wherein the generalized user includes a typical user or a representative user among non-specific general users; determining a registration feature vector based on an output from a feature extractor responsive to an input of user data; receiving test data for user identification input by a test user, wherein the test user includes an unidentified person who has not yet been identified; determining a test feature vector based on output from the feature extractor in response to input of test data; and Based on the result of comparing the enrollment feature vector with the test feature vector, user identification is performed on the test user, Wherein user data is associated with a valid user and test data is associated with a test user, The feature extractor includes a first neural network with set parameters and a second neural network with adjustable parameters. The first neural network is pre-trained based on a large user database. The step of performing on-device training includes: inputting user data into a first neural network, inputting reference data and an output from the first neural network in response to the input of user data into a second neural network, and performing on-device training based on the output from the second neural network.

2. The identification method according to claim 1, in, The adjustable parameters of the second neural network are tuned through on-device training.

3. The identification method according to claim 2, wherein: The first neural network is pre-trained to extract features from input data based on a large user database.

4. The identification method according to claim 1, wherein: The steps to perform on-device training include: Assigning labels with different values ​​to user data and reference data respectively; and On-device training is performed based on a result of comparing the label with an output from a feature extractor in response to input of user data and reference data.

5. The identification method according to claim 1, wherein: The reference data includes generalized feature vectors corresponding to generalized users, The generalized feature vector is generated by grouping feature vectors corresponding to a plurality of generalized users into clusters.

6. The identification method according to claim 1, wherein: The steps to perform user identification include: User identification is performed based on the result of comparing the distance between the registration feature vector and the test feature vector with a threshold value.

7. The identification method according to claim 6, wherein: A distance between the registration feature vector and the test feature vector is determined based on one of a cosine distance between the registration feature vector and the test feature vector and a Euclidean distance between the registration feature vector and the test feature vector.

8. The identification method according to claim 1, further comprising: The determined registration feature vector is stored in the registered user database. 9 . A non-transitory computer-readable storage medium storing instructions, which, when executed by a processor, causes the processor to perform the identification method according to claim 1 .

10. An identification method, comprising: Obtaining a feature extractor including a first neural network having set parameters and a second neural network having adjustable parameters; performing on-device training on a feature extractor based on user data corresponding to a valid user and reference data corresponding to a generalized user, wherein the valid user includes a person having the right to use the identification device and the generalized user includes a typical user or a representative user among non-specific general users; and When on-device training is complete, perform user identification on the test data using the feature extractor. The user data is associated with a valid user, the test data is associated with a test user, and the test user includes an unidentified person who has not yet been identified. The first neural network is pre-trained based on a large user database. The step of performing on-device training includes: inputting user data into a first neural network, inputting reference data and an output from the first neural network in response to the input of user data into a second neural network, and performing on-device training based on the output from the second neural network.

11. The identification method according to claim 10, wherein: The adjustable parameters of the second neural network are tuned through on-device training.

12. An on-device training method for a feature extractor disposed in a user device, the feature extractor comprising a pre-trained first neural network having set parameters and a second neural network having adjustable parameters, the on-device training method comprising: obtaining user data input by a valid user, wherein a valid user includes a person having the right to use the user device; inputting user data into the first neural network; and By inputting preset reference data and the output from the first neural network in response to the input of user data into the second neural network, adjusting the adjustable parameters of the second neural network, Among them, user data is associated with a valid user. Therein, the first neural network is pre-trained based on a large user database.

13. The on-device training method according to claim 12, wherein: The reference data consisted of 1000 or fewer feature vectors.

14. The on-device training method according to claim 12, wherein: The reference data consisted of 500 or fewer feature vectors.

15. The on-device training method according to claim 12, wherein: The reference data consisted of 100 or fewer feature vectors.

16. The on-device training method according to claim 12, wherein: The reference data includes generalized feature vectors corresponding to generalized users.

17. The on-device training method according to claim 16, wherein: A generalized feature vector is generated by grouping feature vectors corresponding to a plurality of generalized users into clusters.

18. An identification device comprising: processor; as well as memory, including instructions that can be executed in the processor, Wherein, when the instruction is executed by a processor, the processor is configured to: receiving user data for user registration input by a valid user, wherein the valid user comprises a person having the right to use the identification device; performing on-device training on a feature extractor based on the user data and reference data corresponding to a generalized user, wherein the generalized user includes a typical user or a representative user among non-specific general users; determining a registration feature vector based on an output from a feature extractor responsive to an input of user data; receiving test data for user identification input by a test user, wherein the test user includes an unidentified person who has not yet been identified; determining a test feature vector based on output from the feature extractor in response to input of test data; and Based on the result of comparing the enrollment feature vector with the test feature vector, user identification is performed on the test user, Wherein user data is associated with a valid user and test data is associated with a test user, The feature extractor includes a first neural network with set parameters and a second neural network with adjustable parameters. The first neural network is pre-trained based on a large user database. The processor is configured to: input user data into a first neural network, input reference data and an output from the first neural network in response to the input of user data into a second neural network; and perform on-device training based on the output from the second neural network.

19. The identification device according to claim 18, in, The adjustable parameters of the second neural network are tuned through on-device training.

20. The identification device according to claim 19, wherein: The first neural network is pre-trained to extract features from input data based on a large user database.

21. The identification device according to claim 18, wherein: The processor is configured as: Assigning different value labels to user data and reference data respectively; and On-device training is performed based on a result of comparing the label with an output from a feature extractor in response to input of user data and reference data.

22. The identification device according to claim 18, wherein: The reference data includes generalized feature vectors corresponding to generalized users, The generalized feature vector is generated by grouping feature vectors corresponding to a plurality of generalized users into clusters.

23. The identification device according to claim 18, wherein: The processor is configured as: User identification is performed based on the result of comparing the distance between the registration feature vector and the test feature vector with a threshold value.

24. The identification device according to claim 23, wherein: A distance between the registration feature vector and the test feature vector is determined based on one of a cosine distance between the registration feature vector and the test feature vector and a Euclidean distance between the registration feature vector and the test feature vector.

25. The identification device according to claim 18, wherein: The processor is configured to store the determined registration feature vector in a registered user database.

26. An identification device comprising: processor; as well as memory, including instructions that can be executed in the processor, Wherein, when the instruction is executed by a processor, the processor is configured to: Obtaining a feature extractor including a first neural network having set parameters and a second neural network having adjustable parameters; performing on-device training on a feature extractor based on user data corresponding to valid users and reference data corresponding to generalized users, wherein valid users include persons having the right to use the identification device, and generalized users include typical or representative users among non-specific general users; and When on-device training is complete, perform user identification on the test data using the feature extractor. wherein the user data is associated with a valid user, and the test data is associated with a test user, the test user including an unidentified person who has not yet been identified, The first neural network is pre-trained based on a large user database. The processor is configured to: input user data into a first neural network, input reference data and an output from the first neural network in response to the input of user data into a second neural network; and perform on-device training based on the output from the second neural network.

27. The identification device according to claim 26, wherein: The adjustable parameters of the second neural network are tuned through on-device training.

28. A recognition method comprising: Pre-training a first neural network of a feature extractor on a server side; After the first neural network is pre-trained, setting the feature extractor to the device; performing on-device training on a second neural network of a feature extractor on the device using data input to the device, wherein the data input to the device includes user data for user registration input by a valid user and reference data corresponding to a generalized user, wherein the valid user includes a person having a right to use the device, and the generalized user includes a typical user or a representative user among non-specific general users; and Use a feature extractor to perform user identification on the test data input to the device, wherein the user data is associated with a valid user, and the test data is associated with a test user, the test user including an unidentified person who has not yet been identified, The first neural network has set parameters and is pre-trained based on a large user database. The step of performing on-device training includes: inputting user data into a first neural network, inputting reference data and an output from the first neural network in response to the input of user data into a second neural network, and performing on-device training based on the output from the second neural network.

29. The identification method according to claim 28, further comprising: User identification is performed by comparing the registration feature vector corresponding to the user data with the test feature vector corresponding to the test data.

Images