Data identification device and method and training device and method

Abstract

Disclosed are a data recognition device and method and a training device and method. The data recognition device can pool orthogonal component vectors using feature maps extracted from input data, generate an embedding vector by recombining the orthogonal component vectors using attention weights, and use the embedding vector to identify a user corresponding to the input data.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2018-0126436, filed on October 23, 2018, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety for all purposes by reference. Technical Field

[0002] The following description relates to data identification. Background Art

[0003] Technical automation of data recognition has been achieved through neural network models implemented as processors of dedicated computing architectures, wherein the processor-implemented neural network models after extensive training can provide a computationally intuitive mapping between input patterns and output patterns. The training ability to produce such a mapping can be referred to as the learning ability of the neural network. In addition, due to the specialized training, such a specially trained neural network can therefore have a generalization ability to generate relatively accurate outputs with respect to, for example, input patterns for which the neural network may not have been trained. However, because such operations or applications are performed through such specialized computing architectures and in an automated manner that is different from the way they are performed in non-computer-implemented or non-automated methods, they also cause problems or defects that occur solely due to the automated and dedicated computing architecture manner in which they are implemented.

[0004] Currently, when neural networks (e.g., deep neural networks (DNNs)) are used for data recognition (e.g., speech recognition, image recognition, etc.), it is often difficult to distinguish changes in input data from the same source (e.g., the same speaker, the same object in an image, etc.), which results in low accuracy in data recognition and fails to meet user needs. Summary of the invention

[0005] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0006] In a general aspect, a data recognition method includes: extracting a feature map from input data by a feature extraction layer based on a data recognition model; pooling component vectors from the feature map by a pooling layer based on the data recognition model; and generating an embedding vector by recombining the component vectors by a combination layer based on the data recognition model.

[0007] The data identification method may further include identifying the input data based on the generated embedding vector.

[0008] The step of identifying the input data may include indicating, among a plurality of registered users, a user mapped to a registered vector verified to match the embedded vector.

[0009] The step of pooling out component vectors may include: extracting a first component vector from the feature map based on a pooling layer, and extracting a second component vector whose cosine similarity with the first component vector is less than a threshold similarity from the feature map based on the pooling layer.

[0010] The step of pooling component vectors may include: extracting component vectors that are orthogonal to each other from the feature map.

[0011] The step of generating the embedding vector may include calculating attention weights of the component vectors from the component vectors through an attention layer, and calculating the embedding vector by applying the attention weights to the component vectors.

[0012] The step of calculating the embedding vector may include generating a weighted component vector by applying an attention weight to each component vector, and calculating the embedding vector by linearly combining the weighted component vectors.

[0013] The step of generating the embedding vector may include generating a normalized component vector by normalizing the component vector.

[0014] The step of extracting the feature map may include: extracting a certain number of feature maps using a feature extraction layer including a convolution layer. The number of feature maps may be proportional to the number of component vectors to be pooled.

[0015] In another general aspect, a training method for training a data recognition model, the training method comprising: calculating the orthogonal loss of temporary component vectors extracted from training inputs by a feature extraction layer and a pooling layer based on the data recognition model; calculating the recognition loss of a temporary embedding vector calculated by recombining the temporary component vectors by a combination layer based on the data recognition model; and training the data recognition model based on the orthogonal loss and the recognition loss.

[0016] The step of calculating the recognition loss may include calculating the recognition loss between a temporary output recognized from the temporary embedding vector and a training output provided for the training input.

[0017] The step of calculating the recognition loss may include calculating a temporary output by propagating a temporary embedding vector from a feature extraction layer to a combination layer via a pooling layer.

[0018] The step of calculating the recognition loss may include calculating a temporary attention weight from the temporary component vector based on an attention layer of the data recognition model, and generating a temporary embedding vector by applying the temporary attention weight to the temporary component vector.

[0019] The step of generating the temporary embedding vector may include calculating a weighted component vector by applying a temporary attention weight to each temporary component vector, and generating the temporary embedding vector by linearly combining the weighted component vectors.

[0020] The step of calculating the recognition loss may include generating a normalized temporary vector by normalizing the temporary component vector.

[0021] The step of calculating the orthogonal loss may include: extracting a temporary feature map from the training input based on a feature extraction layer, extracting a temporary component vector from the temporary feature map based on a pooling layer, and calculating the orthogonal loss based on the orthogonality between each of the temporary component vectors and another of the temporary component vectors.

[0022] The step of calculating the orthogonal loss may include calculating the sum of cosine similarities of the temporary component vectors as the orthogonal loss.

[0023] The training may include: calculating a total loss based on the orthogonal loss and the recognition loss, and updating parameters of the data recognition model to minimize the total loss.

[0024] In another general aspect, a data recognition device includes: a memory configured to store a data recognition model including a feature extraction layer, a pooling layer, and a combination layer; a processor configured to extract a feature map from input data based on the feature extraction layer, pool a feature vector from the feature map based on the pooling layer, and generate an embedding vector by recombining component vectors based on the combination layer.

[0025] In another general aspect, a speaker recognition method includes: extracting a feature map from speech data by a feature extraction layer based on a speech recognition model, pooling component vectors from the feature map by a pooling layer based on the speech recognition model, generating an embedding vector by recombining the component vectors through a combination layer based on the speech recognition model, and indicating a recognition result of recognizing a speaker of the speech data based on the generated embedding vector.

[0026] The step of indicating the recognition result may include verifying whether the speaker is a registered user based on a result of comparing the embedding vector with a registration vector corresponding to a registered user.

[0027] Indicating the recognition result may include determining that authentication of the voice data is successful in response to a cosine similarity between the embedded vector and the enrollment vector exceeding a threshold recognition value, and unlocking the device in response to the authentication being determined to be successful.

[0028] In another general aspect, a speaker recognition device includes: a memory configured to store a speech recognition model including a feature extraction layer, a pooling layer, and a combination layer; a processor configured to extract a feature map from speech data based on the feature extraction layer of the speech recognition model, pool component vectors from the feature map based on the pooling layer of the speech recognition model, generate an embedding vector by recombining the component vectors based on the combination layer of the speech recognition model, and indicate a recognition result of recognizing a speaker of the speech data based on the generated embedding vector.

[0029] Other features and aspects will be apparent from the following detailed description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a diagram illustrating an example of a data recognition system.

[0031] Figure 2 is a diagram illustrating an example of a speech recognition model.

[0032] Figure 3 is a diagram showing an example of changes in vector distribution.

[0033] Figure 4 is a flowchart illustrating an example of a data identification method.

[0034] Figure 5 is a diagram illustrating an example of a data recognition model.

[0035] Figure 6 is a diagram illustrating an example of extraction of a feature map in a data recognition model.

[0036] Figure 7 is a diagram illustrating an example of pooling and recombination of component vectors in a data recognition model.

[0037] Figure 8 is a diagram illustrating an example of a data recognition device.

[0038] Fig. 9 is a flow chart illustrating an example of a training method.

[0039] Fig.10 is a diagram illustrating an example of training of a data recognition model.

[0040] Fig.11 is a diagram illustrating an example of a training device.

[0041] Fig.12 is a diagram illustrating an example of a computing device.

[0042] Throughout the drawings and detailed description, unless otherwise described or provided, the same figure reference numerals will be understood to refer to the same elements, features and structures. The drawings may not be drawn to scale, and the relative sizes, proportions and depictions of the elements in the drawings may be exaggerated for clarity, illustration and convenience. DETAILED DESCRIPTION

[0043] The following specific embodiments are provided to help the reader gain a comprehensive understanding of the methods, devices and / or systems described herein. However, after understanding the disclosure of the present application, various permutations, modifications and equivalents of the methods, devices and / or systems described herein will be apparent. For example, the order of operations described herein is merely an example and is not limited to the order set forth herein, but in addition to the operations that must occur in a particular order, changes that will be apparent may be made after understanding the disclosure of the present application. In addition, for clarity and brevity, descriptions of features known in the art may be omitted.

[0044] The features described herein may be implemented in different forms and shall not be construed as being limited by the examples described herein. Rather, the examples described herein have been provided only to illustrate some of the many possible ways of implementing the methods, devices and / or systems described herein that will be apparent after understanding the disclosure of the present application.

[0045] Although terms such as "first," "second," and "third" may be used herein to describe various members, components, regions, layers, or portions, these members, components, regions, layers, or portions are not limited by these terms. Rather, these terms are only used to distinguish one member, component, region, layer, or portion from another member, component, region, layer, or portion. Therefore, without departing from the teachings of the examples described herein, a first member, component, region, layer, or portion referred to may also be referred to as a second member, component, region, layer, or portion.

[0046] Throughout the specification, when a component is described as being "connected to" or "coupled to" another component, the component may be directly "connected to" or "coupled to" another component, or there may be one or more other components in between. In contrast, when an element is described as being "directly connected to" or "directly coupled to" another element, there may be no other elements in between. Similarly, similar expressions, for example, "between" and "immediately between", "adjacent to" and "immediately adjacent to" should also be interpreted in the same manner. As used herein, the term "and / or" includes any one and any combination of any two or more of the relevant listed items.

[0047] The terms used herein are only used to describe various examples and are not used to limit the present disclosure. Unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. The terms "comprise", "include" and "have" list the existence of the stated features, numbers, operations, components, elements and / or their combinations, but do not exclude the existence or addition of one or more other features, numbers, operations, components, elements and / or their combinations.

[0048] Unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which the present disclosure belongs based on understanding the disclosure of the present application. Unless explicitly defined as such herein, terms (such as those defined in general dictionaries) will be interpreted as having a meaning consistent with their meaning in the context of the relevant field and the disclosure of the present application, and will not be understood in an idealized or overly formal way.

[0049] Furthermore, in the description of the exemplary embodiments, when it is considered that a detailed description of a structure or function thus known after understanding the disclosure of the present application will lead to an obscure interpretation of the exemplary embodiments, such description will be omitted.

[0050] Figure 1 is a diagram illustrating an example of a data recognition system.

[0051] The data recognition system described herein refers to a system configured to recognize data input from a user. In the case where the data recognition system recognizes speech data, the data recognition system may be referred to as a speech recognition system. In the case where the data recognition system recognizes image data, the data recognition system may be referred to as an image recognition system. Although the speech recognition system will be mainly described below, the data recognition system is not limited thereto.

[0052] Reference Figure 1 The speech recognition system includes a speech recognition device and a speech recognition server 130 .

[0053] The speech recognition device described herein may be configured to collect sound or voice and obtain an audio signal as a digital electrical signal from the collected sound or voice. For example, the speech recognition device may collect the voice sound of a person as a nearby sound and distinguish the voice or voice of the user of the speech recognition device from the nearby sound. In addition, the speech recognition device may identify a registered user from the speech signal.

[0054] In this specification, the term "identification" is used as a concept including verification and identification. Verification can be an operation of determining whether input data is true or false. For example, verification can be an operation of determining whether the user indicated by the input data is a registered user. Identification can be an operation of determining a tag indicated by the input data among multiple tags. For example, each tag can indicate the identity (ID) of each registered user.

[0055] For example, the speech recognition device may be an electronic device owned by an individual user. In this example, the speech recognition device may include, for example, a smart phone, a smart tablet, a wearable device (such as a smart bracelet), a personal digital assistant (PDA), a laptop computer, etc. For another example, the speech recognition device may also be an electronic device shared by multiple users. In this example, the speech recognition device may include, for example, Figure 1 Shown is a voice recognition speaker 103 shared by users 113 and 114 or a voice recognition television (TV) 104 shared by users 115 , 116 , and 117 .

[0056] The speech recognition system can recognize the user's voice. The speech recognition may include operations of distinguishing the user's voice included in the audio signal, recognizing the user based on the voice, converting the user's voice into speech expressed in a language, and converting the speech into text.

[0057] The speech recognition server 130 may recognize a speech signal and identify a user. In addition, the speech recognition server 130 may generate text data corresponding to the speech signal. The speech recognition server 130 may receive a speech signal of a user from a speech recognition device, and generate text data corresponding to the speech signal based on the received speech signal. The speech recognition server 130 may send the text data generated by converting the speech signal to the speech recognition device. The text data generated by the speech recognition server 130 may be used for various services to be provided to the user of the speech recognition device.

[0058] exist Figure 1In the example shown, the first terminal 101 is a personal electronic device of a first user 111, and the second terminal 102 is a personal electronic device of a second user 112. In this example, the first terminal 101 may collect a voice signal indicating an instruction to make a call (e.g., an audio call or an audio / video call) to the second terminal 102 of the second user 112 from the first user 111. Then the voice recognition server 130 may receive the voice signal indicating the instruction of the first user 111 from the first terminal 101. The voice recognition server 130 may obtain text data corresponding to the instruction from the voice signal. The voice recognition server 130 may send the obtained text data to the first terminal 101. When the first terminal 101 receives the text data recognized from the instruction spoken by the first user 111, the first terminal 101 may execute a phone application and control the phone application to make a call to the second user 112.

[0059] In another example, the voice recognition device may use the text data recognized by the voice recognition server 130 to generate a schedule for a user of the voice recognition device, retrieve data from a web service, send a text message, or provide a service to retrieve data stored in the voice recognition device.

[0060] However, the operations of the voice recognition device and the voice recognition server 130 are not limited to the foregoing description. For example, the voice recognition device may collect a voice signal and autonomously recognize a user from the voice signal.

[0061] In this specification, voice-based user recognition will be mainly described as an example of data recognition. Figure 2 Describe in detail the speech recognition model used in the speech recognition system.

[0062] Figure 2 is a diagram illustrating an example of a speech recognition model.

[0063] Reference Figure 2 , refer to the above Figure 1 The described speech recognition system may use a speech recognition model 220. The speech recognition model 220 may be configured to identify a user from a speech signal. A speech recognition server may store the speech recognition model 220, and the speech recognition server may identify a user from a speech signal based on the speech recognition model 220. For example, a speech recognition device may send a speech signal or speech feature data obtained by converting from a speech signal to a speech recognition server, and the speech recognition server may perform speech-based user recognition using the received information. However, the examples are not limited to the examples described above, so the speech recognition device may autonomously store the speech recognition model 220 and use the stored speech recognition model 220 to identify a user.

[0064] In one example, the speech recognition model 220 may be implemented by various architectures, such as, for example, a Gaussian mixture model (GMM), a deep neural network (DNN), and a bidirectional long short-term memory (BLSTM). However, the speech recognition model 220 is not limited to the example implemented by the architecture of machine learning, and may therefore be implemented by a combined architecture of at least one of the examples. For example, a neural network may be a recognition model that uses multiple artificial nodes to perform complex calculations implemented by hardware or a combination of hardware and software. The neural network may perform recognition or learning (or training) through these artificial nodes.

[0065] In one example, a neural network may include multiple layers. Figure 2 As shown in FIG. 1 , the neural network includes an input layer 210, at least one hidden layer 221, and an output layer 230. The input layer 210 receives input data 201 (e.g., speech data) and sends the received input data to the hidden layer 221, and the output layer 230 generates output data based on signals received from nodes of the hidden layer 221. Figure 2 The input layer 210 and the output layer 230 are shown to be outside the speech recognition model 220, but the present invention is not limited thereto. For example, the input layer 210 and the output layer 230 may be in the speech recognition model 220.

[0066] The hidden layer 221 is connected to the input layer and the output layer by being arranged between the input layer and the output layer, and the speech recognition system transforms the input data into a predictable value through the hidden layer 221. The nodes included in the input layer and the hidden layer 221 are connected to each other through a connection line having a connection weight, and the nodes included in the hidden layer 221 and the output layer are also connected to each other through a connection line having a connection weight. Among the nodes of the neural network, the nodes included in the hidden layer 221 are called hidden nodes 229.

[0067] A type of neural network including multiple hidden layers is called a DNN, and learning or training a DNN is called deep learning. A DNN may correspond to a convolutional neural network (CNN), a recurrent neural network (RNN), a deep belief network, a fully connected network, a bidirectional neural network, a restricted Boltzmann machine, or may include different or overlapping neural network network parts having full connections, convolutional connections, recurrent connections, and / or bidirectional connections, respectively.

[0068] The input layer, hidden layer 221 and output layer include a plurality of nodes. The hidden layer 221 may be a convolution filter or a fully connected layer in a convolutional neural network (CNN), or various types of filters or layers grouped by specific functions or characteristics.

[0069] For example, the speech recognition device or speech recognition server can input the output of the previous hidden node included in the previous hidden layer to each hidden layer 221 through a connection line with a connection weight, and generate the output of the hidden node 229 included in the hidden layer 221 based on the activation function and the value obtained by applying the connection weight to the output of the previous hidden node. In one example, in order to fire the output to the next hidden node, the result of the activation function may need to exceed the threshold of the current hidden node. In this example, the node may remain inactive without signaling the next node until the threshold strength of activation is reached by the input vector.

[0070] The output layer can output an embedding vector. The embedding vector can be an abstract vector of speech characteristics or characteristics of the speaker. The embedding vector can be used for user identification.

[0071] The application of speech-based user recognition is described as referring to Figure 1 and Figure 2 In the following, reference will be made to Figures 3 to 12 Describe in detail the operations of identifying data and training data identification models.

[0072] Figure 3 is a diagram showing an example of changes in vector distribution.

[0073] Reference Figure 3 , the data recognition model can be configured to output similar embedding vectors for input data obtained from users with the same identity. In addition, the data recognition model can be configured to output different embedding vectors for input data obtained from users with different identities. For example, the data recognition model can be configured to eliminate unnecessary information from the input data obtained from the user and generate a distinctive embedding vector. A data recognition model that uses voice data as input is called a voice recognition model.

[0074] For example, when the first input data x is obtained by a convolutional neural network (CNN), 1 and the second input data x 2 Extract the first vector v 1 and the second vector v 2 When Figure 3 As shown in , a first vector distribution 301 and a second vector distribution 302 may be obtained.

[0075] In one example, the data recognition device may extract multiple component vectors from each set of input data through a pooling layer of the data recognition model. For example, the data recognition device may pool multiple component vectors so that each component vector in the multiple component vectors is orthogonal or roughly orthogonal to another component vector in the multiple component vectors. Therefore, the data recognition device may decompose the input vector into component vectors by pooling the component vectors from the input data through the pooling layer. Figure 3 In the example shown in FIG. , the data recognition device converts the first vector v 1 Decomposed into two first component vectors v 1，1 and v 1，2 The data recognition device converts the second vector v 2 Decomposed into two second component vectors v 2，1 and v 2，2 In this example, Figure 3 As shown in , the vector corresponding to the first component v can be obtained 1，1 The distribution 311 corresponds to the first component vector v 1，2 The distribution 312 corresponds to the second component vector v 2，1 The distribution 321 and the corresponding second component vector v 2，2 Distribution 322.

[0076] The first input data x 1 The subspace of 1，1 and v 1，2 Expand or expand (span). The second input data x 2 The subspace of is represented by the second component vector v 2，1 and v 2，2 As shown in vector coordinates 380, when the first component vector v 1，1 and v 1，2 When they are orthogonal to each other, the first input data x 1 The subspace of can be further expanded. Similarly, when the second component vector v 2，1 and v 2，2 When they are orthogonal to each other, the second input data x 2 The subspace of can also be further expanded. For example, Equation 1 represents the described method of passing the i-th input data x i The component vector v i，1 and v i，2 The size of the area that will be expanded or expanded.

[0077] [Equation 1]

[0078] |v i，1 ||v i，2 |sinθ

[0079] In Equation 1, θ represents the i，1 and v i，2 The angle between them. When the component vector v i，1 and v i，2 When they are orthogonal to each other, θ is 90° and the input data x i The subspace of is maximized.

[0080] Therefore, when the component vectors corresponding to each set of input data are orthogonal to each other, the area where the subspace corresponding to one set of input data overlaps with the subspace corresponding to another set of input data can be maximized. When such an overlapping area is large, the data recognition device is more likely to generate similar embedding vectors for the same speaker based on the weighted sum of the component vectors. In addition, if Figure 3 As shown in , the data identification device can reassemble the component vectors so that the embedding vectors corresponding to the input data obtained from different speakers are distinguishable from each other.

[0081] The data recognition device can generate an embedding vector having a distribution (eg, distributions 391 and 392) that is distinguishable for different speakers and similar for the same speaker by recombining orthogonal component vectors. The embedding vector generated as described above can be robust to global changes.

[0082] For ease of description, Figure 3 The component vectors shown in can be interpreted as two-dimensional (2D) geometric vectors. However, the dimension of the component vector is not limited to two dimensions, so the component vector can be an m-dimensional vector, where m represents an integer greater than or equal to 1.

[0083] Figure 4 is a flowchart illustrating an example of a data identification method.

[0084] Reference Figure 4 In operation 410, the data recognition device extracts a feature map from the input data based on a feature extraction layer of the data recognition model. The feature extraction layer may include, for example, a convolution layer. The feature map may be a map in which the input data is abstracted by the convolution layer. Figure 6 Describe the extraction of feature maps in detail.

[0085] In operation 420, the data recognition device pools component vectors from the feature map based on a pooling layer of the data recognition model. The pooling layer may be configured to pool component vectors from the feature map, and the data recognition device may, for example, use the pooling layer to extract component vectors that are orthogonal to each other or roughly orthogonal to each other. The component vector refers to a vector decomposed from the feature map. Figure 7 Describes pooling of component vectors in detail.

[0086] In operation 430, the data recognition device generates an embedding vector by recombining the component vectors based on the combination layer of the data recognition model. The combination layer may be configured to linearly combine the component vectors. The embedding vector refers to a vector that recombines the component vectors, and may be, for example, a sum of the component vectors to which weights are applied. The embedding vector may be used to identify the user who inputs the input data. Figure 7 Describe the recombination of the component vectors in detail.

[0087] Figure 5 is a diagram illustrating an example of a data recognition model.

[0088] Reference Figure 5 , the data recognition device calculates a recognition result 509 from the input data 501 based on the data recognition model 590. For example, the recognition result 509 may be a result of identifying a user corresponding to the input data 501. For example, in the case where the input data 501 is image data, the data recognition device may identify a user whose image is captured in the image data. For another example, in the case where the input data 501 is voice data, the data recognition device may identify a user who utters a voice included in the voice data. However, the output of the data recognition model 590 is not limited to the examples described above, and may vary based on the design.

[0089] exist Figure 5 In the example shown, the data recognition model 590 includes a feature extraction layer 510, a pooling layer 520, and a combination layer 530. The combination layer 530 includes a linear transformation layer 531 and an attention layer 532. The feature extraction layer 510 refers to a layer configured to extract a feature map 511 from the input data 501. The pooling layer 520 refers to a layer configured to pool component vectors 521 from the feature map 511. The combination layer 530 refers to a layer configured to linearly combine component vectors. The linear transformation layer 531 refers to a layer configured to linearly transform component vectors. The attention layer 532 refers to a layer configured to calculate the attention weights of the linear combination of component vectors. The parameters of the data recognition model 590 can be, for example, connection weights between nodes. The parameters of the data recognition model 590 can be pre-trained.

[0090] The data recognition device propagates input data 501 from the input layer to the output layer of the data recognition model 590. The data recognition device generates a recognition result 509 from the embedding vector 550 generated based on the data recognition model 590.

[0091] In one example, the data recognition device may recognize, through a classifier, a label corresponding to the embedding vector 550. For example, the data recognition device may calculate the probability that the input data 501 corresponds to each of the plurality of users using the embedding vector 550. The data recognition device may recognize the user with the greatest probability among the users as the user corresponding to the input data 501. In this example, the recognition result 509 may indicate the probability that the input data 501 corresponds to each of the plurality of users, the label of the user corresponding to the greatest probability, and the like.

[0092] In another example, the data recognition device may generate a target embedding vector 550 from input data 501 (hereinafter also referred to as target data) as a target to be verified based on the data recognition model 590. The data recognition device may retrieve a vector that matches the target embedding vector 550 from a plurality of reference embedding vectors 550 pre-stored in a device (e.g., a mobile terminal such as a smart phone). The data recognition device may verify whether the target embedding vector 550 and each reference embedding vector 550 are equal to or similar to each other. For example, the data recognition device may calculate the similarity between the target embedding vector 550 and each reference embedding vector 550, such as cosine similarity. In this example, in response to the calculated similarity exceeding a threshold recognition value, the data recognition device may determine that the two embedding vectors 550 are equal to or similar to each other.

[0093] In response to the calculated similarity exceeding the threshold identification value, the data identification device may determine that the verification of the target data is successful. Conversely, in response to the calculated similarity being less than or equal to the threshold identification value, the data identification device may determine that the verification of the target data is unsuccessful. Before the target data is input into the data identification device, the reference embedding vector 550 may be stored in an apparatus or device equipped with the data identification device.

[0094] The data identification device may indicate the result of identifying the speaker of the target data based on the target embedding vector 550. For example, in response to confirming that the verification is successful, the data identification device may unlock the device. Unlocking may indicate, for example, an operation of allowing access to at least one application in the device. The data identification device may allow access to each application in the device based on the access rights set for the registered user before performing the verification. In response to input from the user, the device may execute the application that the registered user is allowed to access.

[0095] Hereinafter, operations performed using each layer of the data recognition model 590 will be described in detail.

[0096] Figure 6 is a diagram illustrating an example of extraction of a feature map in a data recognition model.

[0097] Reference Figure 6 , the feature extraction layer 510 includes a convolutional layer. In addition, the feature extraction layer 510 may include additional layers. Figure 6 In the example shown in , the additional layer pools the convolution result.

[0098] In the example, the data recognition device extracts convolution data 615 from the input data 501 based on the convolution layer. The data recognition device calculates the convolution data 615 by performing a convolution operation on the kernel 611 with a stride with respect to the input data 501. In addition, the data recognition device extracts a feature map 511 by pooling the convolution data 615.

[0099] In one example, the data identification device may use a feature extraction layer 510 including a convolution layer to extract a certain number of feature maps 511. In this example, the number of feature maps 511 to be extracted is proportional to the number of component vectors 521 to be pooled. For example, in the case where j component vectors 521 are to be pooled, the data identification device may extract d feature maps 511 for each component vector. Therefore, the data identification device may extract a total of j×d feature maps, where j represents an integer greater than or equal to 2, and d represents an integer greater than or equal to 1.

[0100] Figure 7 is a diagram illustrating an example of pooling and recombination of component vectors in a data recognition model.

[0101] Reference Figure 7 , the data recognition device extracts multiple component vectors 521 from the feature map based on the pooling layer 520. For example, as shown, the data recognition device extracts component vectors 521 that are orthogonal to each other from the feature map. The pooling layer 520 may include parameters that are trained so that the component vectors 521 with the smallest cosine similarity between them are output from the feature map.

[0102] The data identification device pools component vectors from the corresponding feature maps 511 in the plurality of feature maps 511. For example, as shown, the data identification device extracts a first component vector v from the first feature map 711 in the feature map 511 based on the pooling layer 520. 1 The data recognition device extracts the first component vector v from the second feature map 719 in the feature map 511 based on the pooling layer 520. 1 The second component vector v whose cosine similarity is less than the threshold similarity j The data recognition device pools j component vectors 521 from the j×d feature maps 511. Figure 5 In the example shown in , the corresponding feature map is f i to f i×d Can be used to pool each component vector v i , where i represents an integer equal to or greater than 1 and equal to or less than j.

[0103] The data recognition device generates an embedding vector 550 by combining component vectors 521 based on a combination layer 530. For example, as shown, the combination layer 530 includes a linear transformation layer 531. The data recognition device transforms the component vectors 521 based on the linear transformation layer 531 to transform the vectors into a format suitable for linear combination. In addition, the data recognition device normalizes the component vectors 521 to generate normalized component vectors. Through such normalization, the data recognition device can limit the influence of size while maintaining the directionality or direction of the vector transformed from the component vector. The linear transformation layer 531 and the normalization can be represented by Equation 2.

[0104] [Equation 2]

[0105]

[0106] In Equation 2, for ease of description, only the first component vector v will be described. 1 and the second component vector v 2 In Equation 2, f() represents the linear transformation function of each component vector. 1 and 2 They represent the normalized first component vector and the normalized second component vector respectively. The magnitude of the normalized component vector can be between 0 and 1.

[0107] The data recognition device determines the size of each normalized component vector through the attention layer 532. For example, as shown in Equation 3, the data recognition device calculates the attention weight of the component vector 521 from the component vector 521 through the attention layer 532.

[0108] [Equation 3]

[0109] α 1 =f a ( 1 ), α 2 =f a ( 2 )

[0110] In Equation 3, for ease of description, only the first component vector v 1 The attention weight and the second component vector v 2 The attention weight of . In Equation 3, f a () represents the mathematical modeling function of the attention layer 532. 1 and α 2 Respectively represent the first component vector v 1 The first attention weight and the second component vector v 2 The second attention weight of .

[0111] The data recognition device calculates the embedding vector 550 by applying the attention weight to each component vector 521. For example, the data recognition device generates a weighted component vector 521 by applying the attention weight to each component vector 521. The data recognition device calculates the embedding vector 550 by linearly combining the weighted component vectors 521. The embedding vector 550 can be represented by Equation 4.

[0112] [Equation 4]

[0113]

[0114] In Equation 4, for ease of description, only the normalized first component vector and the normalized second component vector A linear combination of . In equation 4, 550 represents an embedding vector 550 that linearly combines the weighted component vectors 521.

[0115] The combination layer 530 including the linear transformation layer 531 and the attention layer 532 may include parameters that are trained so that the orthogonal component vectors 521 are recombined to be optimally combined. Therefore, the data recognition device can form similar subspaces for the same user and different subspaces for different users by combining the component vectors 521 through the combination layer 530.

[0116] Figure 8 is a diagram illustrating an example of a data recognition device.

[0117] Reference Figure 8 , the data identification device 800 includes a processor 810 , a memory 820 and a data acquirer 830 .

[0118] The processor 810 may extract a feature map from the input data based on the feature extraction layer. The processor 810 may pool a component vector from the feature map based on the pooling layer. The processor 810 may generate an embedding vector by recombining the component vectors based on the combination layer. The processor may recognize the input data based on the embedding vector. The processor 810 may verify whether the speaker is a registered user based on the result of comparing the embedding vector and the registration vector. For example, the processor 810 may indicate a user among multiple registered users that is mapped to a registration vector that is verified to match the embedding vector. The registration vector may indicate a vector stored in the memory 820 and corresponding to the registered user. The operation of the processor 810 is not limited to the content described above, and may perform the above-referenced Figures 1 to 7 Describes one or more operations.

[0119] The memory 820 may store a data recognition model including a feature extraction layer, a pooling layer, and a combination layer. The memory 820 may temporarily or permanently store data required to perform the data recognition method described herein.

[0120] The data acquirer 830 may obtain input data. For example, the data acquirer 830 may include a camera sensor, and the camera sensor may capture an image including at least a portion of the user's body through a single frame or multiple frames. For example, the body portion may include the user's face, eyes, irises, and fingerprints, etc. However, examples of body portions are not limited to the aforementioned examples and may vary based on design. For another example, the data acquirer 830 may include a voice sensor, and the voice sensor may capture a user's voice signal and generate voice data from the captured voice signal.

[0121] The processor 810 of the data recognition device 800 can perform a speech-based speaker recognition operation, a speech-based user verification operation, a speech-based payment operation, a speech-based wake-up operation, etc. based on the embedding vector. The data recognition device 800 can also perform other verification operations by generating the embedding vector as described above for other types of data other than speech data.

[0122] The data identification device 800 may be implemented as a mobile device (e.g., a smartphone). However, examples of the data identification device 800 are not limited to the examples described above. The data identification device 800 may also be implemented as an automated teller machine (ATM), a vehicle, a security system (e.g., an access control door), etc.

[0123] Fig. 9 is a flow chart illustrating an example of a training method.

[0124] A training method for a training data recognition model (hereinafter referred to as a training method) may be performed by a training device. The training device may use training data to train a data recognition model. A data recognition model that has not yet completed training is referred to herein as a temporary model. The training data may include a pair of training inputs and training outputs corresponding to the training inputs. The training outputs may be, for example, user labels provided for the training inputs.

[0125] Reference Fig. 9 In operation 910, the training device calculates an orthogonal loss of a temporary component vector extracted from a training input based on a feature extraction layer and a pooling layer of a data recognition model. For example, the training device may extract a temporary component vector by forward propagating the training input to the feature extraction layer and the pooling layer. The temporary component vector may indicate a vector extracted based on the feature extraction layer and the pooling layer before the training is completed. Fig.10 Describe the orthogonality loss in detail.

[0126] In operation 920, the training device calculates the recognition loss of the temporary embedding vector calculated by recombining the temporary component vectors by the combination layer based on the data recognition model. The temporary embedding vector may indicate a vector of the temporary component vectors recombined by the combination layer before the training is completed. Fig.10 Describe the identification loss in detail.

[0127] In operation 930, the training device trains the data recognition model based on the orthogonal loss and the recognition loss. For example, the training device may calculate the total loss from the sum of the orthogonal loss and the recognition loss. The training device may update the parameters of the data recognition model so that the total loss is minimized.

[0128] Fig.10 is a diagram illustrating an example of training of a data recognition model.

[0129] Reference Fig.10 , the training device extracts a temporary feature map 1011 from the training input 1091 based on the feature extraction layer 510. The training device extracts a plurality of temporary component vectors 1021 from the temporary feature map 1011 based on the pooling layer 520. The training device calculates an orthogonal loss 1081 based on the orthogonality (e.g., orthogonality level) between each temporary component vector in the plurality of temporary component vectors 1021 and another temporary component vector in the plurality of temporary component vectors 1021. For example, as shown in Equation 5, the training device may calculate the sum of the cosine similarities of the plurality of temporary component vectors 1021 as the orthogonal loss 1081.

[0130] [Equation 5]

[0131]

[0132] In Equation 5, v k and v l denote the kth component vector and the lth component vector, respectively, where k and l denote integers equal to or greater than 1 and equal to or less than j. As described above, the training device calculates the sum of the cosine similarities of the plurality of temporary component vectors 1021 as the orthogonal loss 1081. When the parameters of the data recognition model 590 are updated so that the orthogonal loss 1081 is minimized, the component vectors may be orthogonal to each other or roughly orthogonal to each other. In Equation 5, t denotes the transpose of the vector, and v k t represents the transpose of the kth component vector. orthogonal represents the orthogonality loss 1081.

[0133] The training device calculates a temporary output 1009 by propagating the temporary embedding vector 550 from the feature extraction layer 510 via the pooling layer 520 to the combination layer 530. For example, as shown, the training device calculates a temporary attention weight from the temporary component vector 1021 based on the attention layer 532 of the data recognition model 590. In this example, the training device generates a normalized temporary vector by normalizing the temporary component vector 1021. The training device generates a temporary embedding vector 550 by applying the temporary attention weight to the temporary component vector 1021 (e.g., the normalized temporary vector). The training device calculates a weighted component vector by applying the temporary attention weight to each temporary component vector 1021. The training device generates a temporary embedding vector 550 by linearly combining the weighted component vectors.

[0134] As shown in Equation 6, the training device calculates the recognition loss 1082 between the temporary output 1009 recognized from the temporary embedding vector 550 and the training output 1092 provided for the training input 1091.

[0135] [Equation 6]

[0136]

[0137] In equation 6, represents a temporary embedding vector 550, and denote the weight based on the temporary output 1009 and the weight based on the training output 1092, respectively. In Equation 6, b denotes a constant, L softmax denotes the recognition loss 1082, and spk denotes the index indicating the individual speakers to be included for training.

[0138] The training device calculates a total loss 1083 based on the orthogonal loss 1081 and the recognition loss 1082. For example, the total loss 1083 can be expressed by Equation 7.

[0139] [Equation 7]

[0140] L total =L softmax +L orthogonal

[0141] The training device updates the parameters of the data recognition model 590 so that the total loss 1083 is minimized. For example, the training device may repeat the update of the parameters of the data recognition model 590 until the total loss 1083 converges. The training device may train the data recognition device by back propagation learning. For example, during training, the connection weights between nodes of different hidden layers may be recursively adjusted by back propagation until the data recognition model 590 is trained to have a desired accuracy rate or be lower than a maximum error rate.

[0142] Fig.11 is a diagram illustrating an example of a training device.

[0143] Reference Fig.11 , the training device 1100 includes a processor 1110 and a memory 1120 .

[0144] The processor 1110 may calculate an orthogonal loss of a temporary component vector extracted from a training input based on a feature extraction layer and a pooling layer of the data recognition model 1122. The processor 1110 may calculate a recognition loss of a temporary embedding vector calculated by recombining the temporary component vector based on a combination layer of the data recognition model 1122. The processor 1110 may train the data recognition model 1122 based on the orthogonal loss and the recognition loss. However, the operation of the processor 1110 is not limited to the above, and thus the processor 1110 may perform the above-mentioned reference Fig.10 One or more of the operations described.

[0145] The memory 1120 may store the data recognition model 1122 and the training data 1121. The memory 1120 may store the temporary model and the temporary parameters corresponding to the temporary model while the data recognition model 1122 is being trained. When the training is completed, the memory 1120 may store the trained data recognition model 1122 and the parameters corresponding to the trained data recognition model 1122. The architecture of the data recognition model 1122 may be similar to that described above with reference to Figure 5 The described architecture.

[0146] Fig.12 is a diagram illustrating an example of a computing device.

[0147] Reference Fig.12 , the computing device 1200 may be configured to use the data recognition method described herein to recognize input data. In one example, the computing device 1200 may correspond to the above reference Figure 8 The data recognition device 800 described above. For example, the computing device 1200 may be an image processing device, a smart phone, a wearable device, a tablet computer, a netbook, a laptop computer, a desktop computer, a personal digital assistant (PDA), a set-top box, a home appliance, a biometric door lock, a security device, a vehicle starter, etc. The computing device 1200 may include the above reference Figure 8 The functions of the data identification device 800 described above. In addition, the computing device 1200 may also include the above reference Fig.11 The functionality of training device 1100 is described.

[0148] Reference Fig.12, the computing device 1200 includes a processor 1210, a storage device 1220, a camera 1230, an input device 1240, an output device 1250, and a network interface 1260. The processor 1210, the storage device 1220, the camera 1230, the input device 1240, the output device 1250, and the network interface 1260 can communicate with each other through a communication bus 1270.

[0149] The processor 1210 may execute functions and instructions. For example, the processor 1210 may process instructions stored in the storage device 1220. The processor 1210 may execute the functions and instructions described above with reference to Figures 1 to 11 One or more of the operations described.

[0150] The storage device 1220 may store information and data required for the processor 1210 to perform operations. The storage device 1220 may include a computer-readable storage medium or a computer-readable storage device. The storage device 1220 may store instructions to be executed by the processor 1210 and store relevant information while the computing device 1200 is executing software or applications.

[0151] The camera 1230 may capture an image including a plurality of image frames.

[0152] The input device 1240 may receive input from the user through tactile, video, audio, or touch input. For example, the input device 1240 may include a keyboard, a mouse, a touch screen, a microphone, and other devices that can detect input from the user and send the detected input to the computing device 1200. For example, the input device 1240 may obtain voice data from the user as input data.

[0153] The output device 1250 can provide the output of the computing device 1200 to the user through visual, auditory or tactile channels. For example, the output device 1250 may include a display, a touch screen, a speaker, a vibration generator, and other devices that can provide output to the user. The network interface 1260 can communicate with external devices through a wired or wireless network.

[0154] This is achieved through hardware components Figure 1 , Figure 2 , Figure 5 , Figure 6 , Figure 7 , Figure 8 , Fig.10 , Fig.11 and Fig.12The described data identification equipment, training equipment and other equipment, units, modules, devices and other components are implemented by hardware components. Examples of hardware components that can be used to perform the operations described in this application include: controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators and any other electronic components configured to perform the operations described in this application. In other examples, one or more hardware components in the hardware components that perform the operations described in this application are implemented by computing hardware (for example, by one or more processors or computers). Processors or computers can be implemented by one or more processing elements (such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field programmable gate arrays, programmable logic arrays, microprocessors or any other devices or combinations of devices configured to respond and execute instructions in a limited manner to achieve the desired result). In one example, a processor or computer includes or is connected to one or more memories storing instructions or software executed by a processor or computer. The hardware components implemented by a processor or a computer can execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) for performing the operations described in this application. The hardware components can also access, manipulate, process, create and store data in response to the execution of instructions or software. For simplicity, the singular term "processor" or "computer" can be used for the description of the examples described in this application, but in other examples, multiple processors or computers can be used, or a processor or computer can include multiple processing elements or multiple types of processing elements or both. For example, a single hardware component or two or more hardware components can be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components can be implemented by one or more processors, or a processor and a controller, and one or more other hardware components can be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller can implement a single hardware component or two or more hardware components. The hardware components may have any one or more of different processing configurations, where examples of different processing configurations include: a single processor, independent processors, parallel processors, single instruction single data (SISD) multiprocessing, single instruction multiple data (SIMD) multiprocessing, multiple instruction single data (MISD) multiprocessing, and multiple instruction multiple data (MIMD) multiprocessing.

[0155] Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 , Fig. 9 and Fig.10 The method for performing the operations described in the present application shown in the embodiment is performed by computing hardware (e.g., by one or more processors or computers), wherein the computing hardware is implemented as the instructions or software described above for performing the operations performed by the method described in the present application. For example, a single operation or two or more operations may be performed by a single processor or two or more processors, or a processor and a controller. One or more operations may be performed by one or more processors, or a processor and a controller, and one or more other operations may be performed by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller may perform a single operation or two or more operations.

[0156] The instructions or software for controlling the processor or computer to implement the hardware components and perform the methods described above are written as computer programs, code segments, instructions or any combination thereof to individually or collectively instruct or configure the processor or computer to operate as a machine or special-purpose computer to perform the operations performed by the hardware components and methods described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) directly executed by the processor or computer. In another example, the instructions or software include high-level code executed by the processor or computer using an interpreter. Ordinary programmers in the art can easily write instructions or software based on the block diagrams and flow charts shown in the accompanying drawings and the corresponding descriptions in the specification, wherein the block diagrams and flow charts shown in the accompanying drawings and the corresponding descriptions in the specification disclose algorithms for performing the operations performed by the hardware components and methods described above.

[0157] Instructions or software for controlling a processor or computer-implemented hardware component and performing the method described above, as well as any associated data, data files, and data structures, may be recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R, BD-R, BD-RW, BD- RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia mini card or card (for example, Secure Digital (SD) or Extreme Digital (XD)), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, wherein any other device is configured to: store instructions or software and any associated data, data files and data structures in a non-temporary manner, and provide instructions or software and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the instructions.

[0158] Although the present disclosure includes specific examples, it will be clear to those skilled in the art that various changes in form and detail may be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are to be considered to be descriptive only and not for limiting purposes. The description of the features or aspects in each example will be considered to be applicable to similar features or aspects in other examples. If the described techniques are performed in a different order, and / or if the components in the described systems, architectures, devices, or circuits are combined in different ways, and / or replaced or supplemented by other components or their equivalents, suitable results may be achieved. Therefore, the scope of the present disclosure is not limited by specific embodiments, but by the claims and their equivalents, and all changes within the scope of the claims and their equivalents will be interpreted as included in the present disclosure.

Claims

1. A data identification method, comprising: Extracting a feature map from input data based on a feature extraction layer of a data recognition model, wherein the input data includes one of speech data and image data; The pooling layer based on the data recognition model pools component vectors from the feature map; generating an embedding vector by recombining the component vectors through a combination layer based on a data recognition model; and Recognize input data based on the embedding vector, The step of pooling component vectors includes: Extracting a first component vector from the feature map based on a pooling layer, and extracting a second component vector having a cosine similarity with the first component vector less than a threshold similarity from the feature map based on the pooling layer; or Component vectors that are orthogonal to each other are extracted from the feature map.

2. The data identification method according to claim 1, wherein: The steps to identify input data include: A user mapped to the enrollment vector verified to match the embedded vector is indicated among the plurality of enrollment users.

3. The data identification method according to claim 1, wherein: The steps to generate the embedding vector include: Calculating the attention weights of the component vectors from the component vectors by an attention layer; and The embedding vector is calculated by applying attention weights to the component vectors.

4. The data identification method according to claim 3, wherein: The steps to calculate the embedding vector include: generating a weighted component vector by applying an attention weight to each component vector; and The embedding vector is calculated by linearly combining the weighted component vectors.

5. The data identification method according to claim 1, wherein: The steps to generate the embedding vector include: Generates a normalized component vector by normalizing the component vector.

6. The data identification method according to claim 1, wherein: The steps of extracting feature maps include: A number of feature maps are extracted using a feature extraction layer, wherein the feature extraction layer includes a convolutional layer, wherein the number of feature maps is proportional to the number of component vectors to be pooled.

7. A training method for a training data recognition model, comprising: Calculating an orthogonal loss of a temporary component vector extracted from a training input by a feature extraction layer and a pooling layer based on a data recognition model, wherein the training input includes one of speech data and image data; calculating a recognition loss of a temporary embedding vector calculated by recombining the temporary component vectors by a combination layer based on a data recognition model; and The data recognition model is trained based on orthogonal loss and recognition loss.

8. The training method according to claim 7, wherein: The steps to calculate the recognition loss include: Compute the recognition loss between the temporary output recognized from the temporary embedding vector and the training output provided for the training input.

9. The training method according to claim 8, wherein: The steps to calculate the recognition loss also include: The temporary output is computed by propagating the temporary embedding vector from the feature extraction layer through the pooling layer to the combination layer.

10. The training method according to claim 7, wherein: The steps to calculate the recognition loss include: Computing temporary attention weights from the temporary component vectors based on the attention layer of the data recognition model; and A temporary embedding vector is generated by applying temporary attention weights to the temporary component vectors.

11. The training method according to claim 10, wherein: The steps to generate a temporary embedding vector include: calculating a weighted component vector by applying a temporary attention weight to each temporary component vector; and A temporary embedding vector is generated by linearly combining the weighted component vectors.

12. The training method according to claim 7, wherein: The steps to calculate the recognition loss include: A normalized temporary vector is generated by normalizing the temporary component vectors.

13. The training method according to claim 7, wherein: The steps to calculate the orthogonal loss include: Extracting a temporary feature map from the training input based on a feature extraction layer; Extracting a temporary component vector from the temporary feature map based on a pooling layer; and An orthogonality loss is calculated based on the orthogonality between each of the temporary component vectors and another of the temporary component vectors.

14. The training method according to claim 7, wherein: The steps to calculate the orthogonal loss include: The sum of the cosine similarities of the temporary component vectors is calculated as the orthogonal loss.

15. The training method according to claim 7, wherein: The training includes: Calculate the total loss based on the orthogonal loss and the identification loss; and Update the parameters of the data recognition model to minimize the total loss. 16 . A non-transitory computer-readable storage medium storing instructions which, when executed by a processor, cause the processor to perform the data identification method of claim 1 .

17. A data recognition device, comprising: A memory configured to: store a data recognition model including a feature extraction layer, a pooling layer, and a combination layer; as well as The processor is configured to: extract a feature map from input data based on the feature extraction layer, pool component vectors from the feature map based on the pooling layer, generate an embedding vector by recombining the component vectors based on the combination layer, and recognize input data based on the embedding vector, wherein the input data includes one of speech data and image data, The step of pooling component vectors includes: Extracting a first component vector from the feature map based on a pooling layer, and extracting a second component vector having a cosine similarity with the first component vector less than a threshold similarity from the feature map based on the pooling layer; or Component vectors that are orthogonal to each other are extracted from the feature map.

18. A method for speaker identification, comprising: A feature extraction layer based on a speech recognition model extracts feature maps from speech data; The pooling layer based on the speech recognition model pools component vectors from the feature map; Generate an embedding vector by recombining the component vectors through a combination layer based on a speech recognition model; Identify the speaker of speech data based on the embedding vector; as well as a recognition result indicating a speaker of the recognized speech data, The step of pooling component vectors includes: Extracting a first component vector from the feature map based on a pooling layer, and extracting a second component vector having a cosine similarity with the first component vector less than a threshold similarity from the feature map based on the pooling layer; or Component vectors that are orthogonal to each other are extracted from the feature map.

19. The speaker recognition method according to claim 18, wherein: The steps for indicating the recognition result include: Whether the speaker is a registered user is verified based on the result of comparing the embedding vector with the registration vector corresponding to the registered user.

20. The speaker recognition method according to claim 19, wherein: The step of indicating the recognition result also includes: In response to a cosine similarity between the embedded vector and the enrollment vector exceeding a threshold recognition value, determining that the verification of the speech data succeeded; and In response to the authentication being determined to be successful, the device is unlocked.

21. A speaker recognition device, comprising: A memory configured to: store a speech recognition model including a feature extraction layer, a pooling layer, and a combination layer; The processor is configured to: extract a feature map from the speech data based on a feature extraction layer of the speech recognition model, pool component vectors from the feature map based on a pooling layer of the speech recognition model, recombine the component vectors through a combination layer based on the speech recognition model to generate an embedding vector, identify a speaker of the speech data based on the embedding vector, and indicate a recognition result of identifying the speaker of the speech data, The step of pooling component vectors includes: Extracting a first component vector from the feature map based on a pooling layer, and extracting a second component vector having a cosine similarity with the first component vector less than a threshold similarity from the feature map based on the pooling layer; or Component vectors that are orthogonal to each other are extracted from the feature map.

22. The speaker recognition device according to claim 21, wherein The step of identifying the speaker based on the embedding vector includes identifying the speaker based on a similarity between the embedding vector and a registration vector corresponding to a registered user.

23. The speaker recognition device according to claim 21, wherein The step of recombining the component vectors based on the combination layer includes generating weighted component vectors by applying attention weights to the component vectors, and linearly combining the weighted component vectors.

24. The speaker recognition device according to claim 21, wherein The step of extracting the feature map includes: extracting convolution data from the speech data based on a convolution layer of the feature extraction layer, and pooling the convolution data.

25. An image recognition device, comprising: A memory configured to: store a data recognition model including a feature extraction layer, a pooling layer, and a combination layer; as well as The processor is configured as: Extracting a feature map from image data obtained from a captured image of a user based on a feature extraction layer, Based on the pooling layer, the component vector is pooled from the feature map. Generate an embedding vector by recombining the component vectors based on the combination layer, Identify users based on their embedding vectors, and indicating the recognition result of the recognized user, The step of pooling component vectors includes: Extracting a first component vector from the feature map based on a pooling layer, and extracting a second component vector having a cosine similarity with the first component vector less than a threshold similarity from the feature map based on the pooling layer; or Component vectors that are orthogonal to each other are extracted from the feature map.

Images