Image recognition method, image recognition device and system

Abstract

An image recognition method, an image recognition apparatus, and a system are provided. The image recognition method includes extracting feature data from an input image received by an image sensor using a feature extraction layer; and outputting a recognition result of an object appearing in the input image by applying a fixed mask and a variable mask to the extracted feature data, wherein the variable mask is adjusted in response to the extracted feature data.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2019-0161553, filed on December 6, 2019, with the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference for all purposes. Technical Field

[0002] The following description relates to techniques used for image recognition. Background Technology

[0003] Recent research has focused on solving the problem of classifying input patterns into specific groups, and studies have been conducted on methods for applying efficient and accurate pattern recognition performed by humans to computers. One area of ​​this research is artificial neural networks. To solve the problem of classifying input patterns into specific groups, neural networks employ algorithms that generate mappings between input and output patterns. The ability to generate such mappings is known as the learning ability of artificial neural networks. Furthermore, even for input patterns that have not yet been used for learning, artificial neural networks can generalize to generate relatively accurate outputs based on the learning results. Summary of the Invention

[0004] The present invention is provided in a simplified form to describe the choice of concepts further described in the following detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to help determine the scope of the claimed subject matter.

[0005] In one general aspect, a method for image recognition includes: extracting feature data from an input image received by an image sensor using a feature extraction layer; and outputting a recognition result of an object appearing in the input image by applying a fixed mask and a variable mask to the extracted feature data, wherein the variable mask is adjusted in response to the extracted feature data.

[0006] The steps for outputting the recognition result may include: calculating first recognition data from the extracted feature data by applying a fixed mask to the extracted feature data; calculating second recognition data from the extracted feature data by applying a variable mask to the extracted feature data; and determining the recognition result based on the first recognition data and the second recognition data.

[0007] The steps of calculating the first identification data may include: generating a general feature map relating to the region of interest by applying a fixed mask to the extracted feature data; and calculating the first identification data from the general feature map.

[0008] The steps of calculating the second identification data may include: generating a sensor-specific feature map related to the region of interest of the image sensor by applying a variable mask to a target feature map corresponding to the extracted feature data; and calculating the second identification data from the sensor-specific feature map.

[0009] The steps for generating a sensor-specific feature map may include applying the corresponding value from a variable mask to a single value in the target feature map.

[0010] The method may further include: calculating third identification data from the extracted feature data using a softmax function and a fully connected layer. The step of determining the identification result may include: determining the identification result based on the third identification data, in addition to the first and second identification data.

[0011] The steps for outputting the recognition results may include: using at least a portion of a sensor-specific layer including a variable mask, adjusting one or more values ​​of the variable mask based on the extracted feature data.

[0012] The step of adjusting one or more values ​​of the variable mask may include: determining the value of the variable mask using a softmax function from the result of multiplying the transposed query feature map and the key feature map, the result of multiplying the transposed query feature map and the key feature map corresponding to the result of applying a convolutional filter to the extracted feature data.

[0013] The steps for outputting the recognition result may include: determining the recognition result by weighting the first recognition data based on a fixed mask and the second recognition data based on a variable mask.

[0014] The step of determining the weighted sum as the identification result may include: applying a weight that is greater than the weight applied to the first identification data to the second identification data.

[0015] The method may further include: receiving parameters of a sensor-specific layer including a variable mask from an external server in response to an update command; and updating the sensor-specific layer using the received parameters.

[0016] The method may further include: requesting sensor-specific parameters from an external server, the sensor-specific parameters corresponding to optical characteristics similar to or the same as those of the image sensor.

[0017] The method may further include maintaining the value of a fixed mask while updating the parameters of the sensor-specific layer.

[0018] The steps for outputting the recognition results may include: calculating the recognition results based on a fixed mask and multiple variable masks.

[0019] The parameters of a sensor-specific layer that includes one of the plurality of variable masks may differ from the parameters of another sensor-specific layer that includes another of the plurality of variable masks.

[0020] The steps for outputting the recognition result may include: generating authenticity information indicating whether the object is a real object or a pseudo object as the recognition result.

[0021] The method may further include: granting permissions based on the identification result; and allowing access to either or both of the operations of the electronic terminal and the data of the electronic terminal based on the permissions.

[0022] The steps for outputting the recognition results may include: after generating the recognition results, visualizing the recognition results on a display.

[0023] In another general aspect, a non-transitory computer-readable storage medium may store instructions that, when executed by a processor, cause the processor to perform the methods described above.

[0024] In another general aspect, an apparatus for image recognition includes an image sensor and a processor. The image sensor is configured to receive an input image. The processor is configured to: extract feature data from the input image using a feature extraction layer; and output a recognition result of an object appearing in the input image by applying a fixed mask and a variable mask to the extracted feature data, wherein the variable mask adjusts in response to the extracted feature data.

[0025] The output of the recognition result may include: calculating first recognition data from the extracted feature data by applying a fixed mask to the extracted feature data; calculating second recognition data from the extracted feature data by applying a variable mask to the extracted feature data; and determining the recognition result based on the sum of the first recognition data and the second recognition data.

[0026] The sum can be determined by applying a weight that is greater to the second identification data than to the first identification data.

[0027] The steps of calculating the first identification data may include: generating a general feature map relating to a region of interest by applying a fixed mask to the extracted feature data; and calculating the first identification data from the general feature map. The steps of calculating the second identification data include: generating a sensor-specific feature map relating to a region of interest of an image sensor by applying a variable mask to a target feature map corresponding to the extracted feature data; and calculating the second identification data from the sensor-specific feature map.

[0028] In another general aspect, a system with image recognition includes an image recognition device and a server. The image recognition device is configured to: extract feature data from a received input image using a feature extraction layer; and output a recognition result of an object appearing in the input image by applying a fixed mask and a variable mask to the extracted feature data. The variable mask is included in a sensor-specific layer of the image recognition device and is adjusted in response to the extracted feature data. The server is configured to: assign parameters of an additionally trained sensor-specific layer to the image recognition device in response to either or both an update request from the image recognition device and the completion of additional training of the sensor-specific layer of the server's recognition model. The image recognition device is configured to: update the sensor-specific layer of the image recognition device based on the assigned parameters.

[0029] The server is also configured to assign parameters of an additional trained sensor-specific layer to another image recognition device, which includes an image sensor determined to be similar to the image sensor of the image recognition device.

[0030] Other features and aspects will become clear from the following detailed description, drawings, and claims. Attached Figure Description

[0031] Figure 1 An example of a recognition model is shown.

[0032] Figure 2 This is a flowchart illustrating an example of an image recognition method.

[0033] Figure 3 and Figure 4 An example of the structure of the recognition model is shown.

[0034] Figure 5 and Figure 6 An example of the structure of the recognition model is shown.

[0035] Figure 7 An example of a concern layer is shown.

[0036] Figure 8 An example of the structure of the recognition model is shown.

[0037] Figure 9 An example of training a recognition model is shown.

[0038] Figure 10 This illustrates an example of parameter updates for the sensor-specific layer in the recognition model.

[0039] Figure 11 and Figure 12 This is a block diagram illustrating an example of an image recognition device.

[0040] Throughout the accompanying drawings and detailed embodiments, the same reference numerals denote the same elements, features, and structures. The drawings may not be to scale, and for clarity, illustration, and convenience, the relative dimensions, scale, and depiction of elements in the drawings may be exaggerated. Detailed Implementation

[0041] The following detailed embodiments are provided to aid the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to the order set forth herein, but may be changed as will become clear upon understanding this disclosure, except for operations that must occur in a specific order. Furthermore, for clarity and brevity, descriptions of features known upon understanding this disclosure may be omitted.

[0042] The features described herein may be implemented in different forms and should not be construed as being limited to the examples described herein. Rather, only the examples described herein have been provided to illustrate some of the many feasible ways of implementing the methods, apparatus, and / or systems described herein, which will become clear upon understanding the disclosure of this application.

[0043] It should be noted here that the use of the term "may" (e.g., what an example or embodiment may include or implement) with respect to an example or embodiment indicates that there exists at least one example or embodiment that includes or implements such a feature, but all examples and embodiments are not limited thereto.

[0044] Throughout this specification, when an element (such as a layer, region, or substrate) is described as being "on" another element, "connected to," or "bonded to" another element, the element may be directly "on" the other element, directly "connected to," or "bonded to" the other element, or there may be one or more other elements in between. Conversely, when an element is described as being "directly on" another element, directly "connected to," or "bonded to" another element, there may be no other elements in between. As used herein, the term "and / or" includes any one of the associated listed items and any combination of any two or more.

[0045] While terms such as “first,” “second,” and “third” may be used herein to describe various components, assemblies, regions, layers, or parts, these components, assemblies, regions, layers, or parts should not be limited by these terms. Rather, these terms are used only to distinguish one component, assembly, region, layer, or part from another. Thus, without departing from the teaching of the examples described herein, the first component, first assembly, first region, first layer, or first part mentioned in the examples may also be referred to as a second component, second assembly, second region, second layer, or second part.

[0046] The terminology used herein is for the purpose of describing various examples only and is not intended to limit disclosure. Unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. The terms “comprising,” “including,” and “having” specify the presence of the features, quantities, operations, components, elements, and / or combinations thereof described, but do not preclude the presence or addition of one or more other features, quantities, operations, components, elements, and / or combinations thereof.

[0047] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains, based on an understanding of this disclosure. Unless expressly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and shall not be interpreted in an idealized or overly formalized sense.

[0048] Figure 1 An example of a recognition model is shown.

[0049] Image recognition devices can use feature data extracted from an input image to identify a user. For example, an image recognition device can extract feature data from an input image based on at least a portion of multiple layers of a recognition model (e.g., a feature extraction layer). Feature data is data in which characteristics of an image are abstracted, for example, and represented, for example, in the form of a vector. Feature data in the form of a two-dimensional vector or higher is also referred to as a "feature map". In this disclosure, a feature map represents feature data in the form of a 2D vector or a 2D matrix.

[0050] A recognition model is, for example, a model designed to extract feature data from an image and output a result that identifies the object appearing in the image from the extracted feature data. For example, a recognition model is a machine learning structure and includes a neural network 100.

[0051] Neural network 100 can be, for example, a deep neural network (DNN). DNNs can include fully connected networks, deep convolutional networks, recurrent neural networks, etc. Neural network 100 can perform object classification, object recognition, speech recognition, and image recognition by mapping input and output data with a non-linear relationship based on deep learning. Deep learning is a machine learning technique used to solve problems related to image or speech recognition from large datasets, and it is performed by mapping input and output data via supervised or unsupervised learning.

[0052] Here, identification includes data verification and data identification. Verification can be an operation to determine whether input data is true or false. As an example, verification is an operation used to determine whether an object indicated by an input image (e.g., a human face) is the same as an object indicated by a reference image. As another example, liveness verification can be an operation to determine whether an object indicated by an input image is a real object or a fake object.

[0053] An image recognition device can verify whether the data extracted and acquired from an input image is the same as the registered data in the image recognition device. When the extracted data matches the registered data, the image recognition device can determine that the user corresponding to the input image has been successfully verified. When multiple pieces of registered data are stored in the image recognition device, the image recognition device can sequentially perform verification of the data extracted and acquired from the input image for each piece of registered data.

[0054] Identification is a classification operation that determines which label is indicated by input data among multiple labels. For example, each label indicates a category (e.g., the identity (ID) of a registered user). For example, identification can determine whether a user included in the input data is male or female.

[0055] Reference Figure 1 The neural network 100 includes, for example, an input layer 110, a hidden layer 120, and an output layer 130. The input layer 110, the hidden layer 120, and the output layer 130 each include multiple artificial nodes.

[0056] For ease of description, Figure 1 The hidden layer 120 is shown to include three layers. However, the hidden layer 120 may include a variety of numbers of layers. Furthermore, Figure 1 The neural network 100 is shown to include a separate input layer for receiving input data. However, the input data can be directly fed into the hidden layer 120. In the neural network 100, nodes in layers other than the output layer 130 are connected to nodes in subsequent layers via links to transmit output signals. The number of links corresponds to the number of nodes included in the subsequent layers.

[0057] The output of the activation function associated with the weighted inputs of the artificial nodes included in the previous layers can be input to each artificial node in the hidden layer 120. The weighted inputs are obtained by multiplying the inputs of the artificial nodes included in the previous layers by weights. These weights can be referred to as parameters of the neural network 100. Activation functions include sigmoid, hyperbolic tangent, and Rectified Linear Unit (ReLU), and nonlinearity is formed in the neural network 100 through these activation functions. The weighted inputs of the artificial nodes included in the previous layers can be input to the artificial nodes in the output layer 130.

[0058] Once input data is provided, neural network 100 can compute a function value of the number of categories identified in output layer 130 through hidden layer 120, and can identify the input data as the category with the maximum value among multiple categories. Although neural network 100 identifies input data, the embodiment is not limited to such an example. Neural network 100 can verify input data against reference data (e.g., registration data). The following description of the recognition process is primarily described as a verification process, but the following description of the recognition process can also be applied to the identification process as long as the recognition process does not contradict the identification process.

[0059] When the width and depth of the neural network 100 are sufficiently large, the neural network 100 can have the capability to achieve the intended function. When the neural network 100 learns a sufficient amount of training data through appropriate training processing, the neural network 100 can achieve optimal estimation performance.

[0060] Although the neural network 100 has been described above as an example of a recognition model, recognition models are not limited to neural network 100. The following will mainly describe an example verification operation based on feature data extracted using the feature extraction layer of the recognition model.

[0061] Figure 2 This is a flowchart illustrating an example of an image recognition method.

[0062] Image recognition devices can receive input images via image sensors. The input image can be an image related to an object (e.g., an image acquired by capturing at least a portion of the object). A portion of the object can be a body part related to the object's unique biometric characteristics. In an example where the object is a person, the portion of the object is, for example, a face, fingerprint, veins, etc. This disclosure includes descriptions of examples where the input image includes a human face, but is not limited to such examples. The input image can be, for example, a color image, and can include multiple channel images for each channel constituting a color space. For example, with respect to the RGB color space, the input image includes a red channel image, a green channel image, and a blue channel image. The color space is not limited to the examples and can include YCbCr, etc. The input image is not limited to the foregoing examples and can include depth images, infrared images, ultrasound images, radar scan images, etc.

[0063] In operation 210, the image recognition device uses a feature extraction layer to extract feature data from the input image received by the image sensor. For example, the feature extraction layer is... Figure 1 The hidden layer 120 may include one or more convolutional layers. The output of each convolutional layer is the result of applying the convolution operation to the data input to the corresponding convolutional layer through a sweep of the kernel filter. In an example where the input image includes multiple channel images, the image recognition device can use the feature extraction layer of the recognition model to extract feature data for each channel image and can propagate the feature data to subsequent layers of the recognition model.

[0064] In operation 220, the image recognition device outputs a recognition result of an object appearing in the input image based on a variable mask and a fixed mask adjusted in response to the extracted feature data. A fixed mask is a mask with the same value for different input images. A variable mask is a mask with different values ​​for different input images.

[0065] A mask includes mask weights for excluding, retaining, and modifying values ​​included in data. Masks can be applied to data containing multiple values ​​via element-wise (e.g., element-by-element) operations. For example, a value in the data is multiplied by the corresponding mask weight in the mask. As described below, a mask includes mask weights for enhancing and / or retaining values ​​corresponding to regions of interest in the data, and weakening and / or excluding values ​​corresponding to the remaining regions in the data. For example, mask weights can have real values ​​ranging from 0 to 1, but the range of mask weight values ​​is not limited to such examples. The data to which the mask is applied can also be referred to as "masked data."

[0066] For reference, the following description assumes that the size and dimensions of the mask are the same as the size and dimensions of the data to which the mask will be applied. For example, when the data to which the mask will be applied is a two-dimensional (2D) vector of size 32×32, the mask is a two-dimensional vector of size 32×32. However, the foregoing description is merely illustrative and is not intended to be limited to such examples. The size and dimensions of the mask may differ from the size and dimensions of the data.

[0067] Image recognition devices can compute multiple mask data by applying masks to extracted feature data and target data extracted from the feature data. The image recognition device can then use this multiple mask data to compute the recognition result.

[0068] Figure 3 and Figure 4 An example of the structure of the recognition model is shown.

[0069] Figure 3 An example of the structure of recognition model 310 is shown. The image recognition device can use recognition model 310 to output recognition result 309 from input image 301. For example, when a pair of images does not exist, the image recognition device uses recognition model 310 to output recognition result 309 from a single image.

[0070] The recognition model 310 includes, for example, a feature extraction layer 311, a fixed layer 312, and a sensor-specific layer 313. The feature extraction layer 311 is designed to extract feature data from the input image 301. The fixed layer 312 is designed to apply a fixed mask 321 to the data propagated from the feature extraction layer 311 (e.g., feature data) and output first recognition data from the data with the fixed mask 321 applied. The sensor-specific layer 313 is configured to apply a variable mask 322 to the data propagated from the feature extraction layer 311 (e.g., target feature maps extracted from the feature data through one or more convolutional layers) and output second recognition data from the data with the variable mask 322 applied.

[0071] The recognition model 310 can be customized based on the type of image sensor of the electronic terminal in which the recognition model 310 is set. For example, the parameters of the fixed layer 312 of the recognition model 310 can remain unchanged regardless of the type of image sensor. Furthermore, the parameters of the sensor-specific layer 313 (e.g., connection weights between artificial nodes, etc.) can vary based on the type of image sensor. The type of image sensor is classified based on, for example, the optical characteristics of the image sensor. If image sensors have the same or similar optical characteristics and are different models, they can be classified as the same type.

[0072] The image recognition device can extract feature data from the input image 301 through the feature extraction layer 311. As described above, feature data is data in which the characteristics of the image are abstracted, and can be data in vector form (e.g., feature vector), but is not limited thereto.

[0073] An image recognition device can calculate multiple recognition data from the same feature data by using masks individually. As an example, the image recognition device calculates first recognition data from the extracted feature data based on a fixed mask 321. This first recognition data is the result calculated from the data to which the fixed mask 321 is applied, and may also be referred to as "general recognition data." As another example, the image recognition device calculates second recognition data from the extracted feature data based on a variable mask 322. This second recognition data is the result calculated from the data to which the variable mask 322 is applied, and may also be referred to as "sensor-specific data."

[0074] The image recognition device can determine a recognition result 309 based on first recognition data and second recognition data. Each of the first and second recognition data can indicate at least one of the probability that an object appearing in the input image 301 is a real object and the probability that an object appearing in the input image 301 is a fake object. The probability that an object appearing in the input image 301 is a real object can have a real value ranging from 0 to 1. A probability close to 0 indicates that an object appearing in the input image may be a fake object. A probability close to 1 indicates that an object appearing in the input image may be a real object. The image recognition device can determine the recognition result 309 by combining the first and second recognition data. For example, the image recognition device calculates the recognition result 309 by weighting the first and second recognition data.

[0075] Figure 4 Show the corresponding details Figure 3 The structure of recognition model 310 and recognition model 400.

[0076] as Figure 3 For example, an image recognition device can use the feature extraction layer 405 of the recognition model 400 to extract feature data 492 from an input image 401. Examples of calculating first recognition data 494 for feature data 492 using a fixed layer 410 and examples of calculating second recognition data 498 for feature data 492 using a sensor-dedicated layer 420 will be described below.

[0077] An image recognition device can generate a general feature map 493 related to a region of interest by applying a fixed mask 411 to feature data 492. For example, the image recognition device applies a mask weight corresponding to the corresponding value in the fixed mask 411 to each value of the feature data 492 based on element-wise operations. The region of interest is a region of interest relating to a portion of an object in the data, and can be, for example, a region including components related to a human face. In the fixed mask 411, the mask weight in the region of interest can be greater than the mask weight in the remaining regions. Therefore, the general feature map 493 is a feature map that enhances the components related to a human face in the feature data 492 while weakening or excluding other components.

[0078] An image recognition device can compute first recognition data 494 from a general feature map 493. For example, the image recognition device uses a recognizer 412 of a fixed layer 410 to compute the first recognition data 494. The recognizer 412 is configured to output recognition data from the general feature map 493. For example, the recognizer is a classifier and outputs a first verification score vector (e.g., first verification score vector = [probability of being a real object, probability of being a fake object]), where the first verification score vector indicates the probability that an object appearing in the input image 401 is a real object and the probability that an object appearing in the input image 401 is a fake object. The classifier may include fully connected layers (also called “FC layers”) and a softmax operation.

[0079] For reference, in this disclosure, the verification score is primarily described as an example of recognition data, but the disclosure is not limited to such an example. Recognition data may include information indicating the probability that an object appearing in an input image belongs to each of k categories, where k is an integer greater than or equal to 2. Furthermore, although the softmax operation is representatively described as an operation used to compute recognition data, this is merely an example, and other nonlinear mapping functions may also be applicable.

[0080] Furthermore, before applying the variable mask 495 to the target feature map 496, the image recognition device may adjust the variable mask 495 in response to the propagation of feature data 492. For example, the image recognition device may use at least a portion of a plurality of layers of a sensor-specific layer 420 including the variable mask 495 (e.g., mask adjustment layer 421) to adjust one or more values ​​of the variable mask 495 based on the feature data 492. The mask weights of the variable mask 495 may be updated whenever the input image 401 is input. For example, the mask adjustment layer 421 may be implemented as part of a focus layer. (See reference...) Figure 7 A description is provided regarding the mask adjustment layer 421.

[0081] An image recognition device can generate a sensor-specific feature map 497 related to a region of interest of an image sensor by applying an adjusted variable mask 495 to a target feature map 496 corresponding to feature data 492. For example, the image recognition device uses a target extraction layer 422 to extract the target feature map 496 from the feature data 492. The target extraction layer 422 may include one or more convolutional layers. The target feature map 496 is a feature map obtained, for example, by applying one or more convolution operations to the feature data 492. The image recognition device generates the sensor-specific feature map 497 by applying the corresponding value in the variable mask 495 to a single value of the target feature map 496. For example, the image recognition device applies mask weights corresponding to the corresponding values ​​in the variable mask 495 to each value of the target feature map 496 based on element-wise operations.

[0082] In this disclosure, the region of interest (ROI) of an image sensor represents a region in the data that relates to a portion of the object and the optical characteristics of the image sensor. For example, the ROI of an image sensor can be a region in the data that includes the main components for object recognition, taking into account the optical characteristics of the image sensor (e.g., lens shading, image sensor sensitivity, etc.). As described above, since the mask weights of the variable mask 495 are adjusted for each input, the ROI of the image sensor is changed for each input. The sensor-specific feature map is a feature map in the target feature map that enhances the ROI related to the optical characteristics of the image sensor and the object. The optical characteristics of the image sensor can be applied by referring to, for example, in more detail... Figure 9 and Figure 10 The parameters of the sensor-specific layer 420, which are determined through training, are described.

[0083] The image recognition device can calculate second recognition data 498 from the sensor-specific feature map 497. For example, the image recognition device uses the recognizer 423 of the sensor-specific layer 420 to calculate the second recognition data 498. The recognizer 423 is designed to output recognition data from the sensor-specific feature map 497. For example, the recognizer 423 is a classifier and outputs a second verification score vector (e.g., second verification score vector = [probability of being a real object, probability of being a fake object]), where the second verification score vector indicates the probability that an object appearing in the input image 401 is a real object and the probability that an object appearing in the input image 401 is a fake object. For reference, even if the recognizer 412 of the fixed layer 410 and the recognizer 423 of the sensor-specific layer 420 have the same structure (e.g., a structure including fully connected layers and softmax operations), the parameters of the recognizer 412 may be different from the parameters of the recognizer 423.

[0084] The image recognition device can generate a recognition result 409 by applying a merging operation 430 to the first recognition data 494 and the second recognition data 498. For example, the image recognition device can determine the recognition result 409 by weighting the first recognition data 494 based on a fixed mask 411 and the second recognition data 498 based on a variable mask 495. The image recognition device can determine the recognition result shown in Equation 1 below.

[0085] [Equation 1]

[0086] Liveness Score=α·score1+β·score2

[0087] In Equation 1, the Liveness Score is the liveness verification score corresponding to the recognition result 409. score1 is the verification score of the first recognition data 494, and score2 is the verification score of the second recognition data 498. α is the weight for the first recognition data 494, and β is the weight for the second recognition data 498. The image recognition device may apply a weight greater than the weight for the first recognition data 494 to the second recognition data 498. Therefore, for example, in Equation 1, β > α. Equation 1 is merely an example. The image recognition device may compute n recognition data items based on the structure of the recognition model, and may compute a weighted sum by applying n weights to each of the n recognition data items. Among the n weights, the weight applied to the recognition data based on the variable mask may be greater than the weight applied to the remaining recognition data items, where n is an integer greater than or equal to 2.

[0088] Figure 5 and Figure 6 An example of the structure of the recognition model is shown.

[0089] Reference Figure 5 In addition to, as referenced Figure 3 and Figure 4 In addition to the recognition data based on the fixed mask 511 and the variable mask 521, the image recognition device can also calculate recognition data based on the verification layer 530. The verification layer 530 includes a recognizer. The first recognition data 581 based on the fixed layer 510 including the fixed mask 511 (or defined mask) can also be referred to as a "hard mask score". The second recognition data 582 based on the sensor-specific layer 520 including the variable mask (or attention mask) 521 can also be referred to as a "soft mask score". The third recognition data 583 based on the basic activity verification model can also be referred to as a "2D activity score". The image recognition device can calculate the first recognition data 581, the second recognition data 582, and the third recognition data 583 separately from the feature data x jointly extracted from a single input image 501 by the feature extraction layer (or feature extraction model) 505.

[0090] In addition to the first identification data 581 and the second identification data 582, the image recognition device can also determine the recognition result 590 based on the third identification data 583. For example, the image recognition device can generate authenticity information indicating whether the object is a real object or a fake object as the recognition result 590. The recognition result 590 is an activity score and may include a value indicating the probability that it is a real object.

[0091] Figure 6 Show more details with Figure 5 The structure corresponding to the structure.

[0092] Reference Figure 6 The recognition model may include a fixed layer 610, a sensor-specific layer 620, and an activity verification model 630. When the recognition model is implemented using an input image 601, the image recognition device may propagate the feature data x extracted by the feature extraction layer 605 of the activity verification model 630 to the fixed layer 610 and the sensor-specific layer 620.

[0093] The fixed layer 610 may include a fixed mask 611, a fully connected layer 613, and a softmax operation 614. For example, an image recognition device may apply the fixed mask 611 to feature data x to compute a general feature map 612, as shown in Equation 2 below. generic .

[0094] [Equation 2]

[0095] Feat generic =M hard ⊙x

[0096] In Equation 2, Feat generic It is a general feature map 612, M hard Here, 611 is a fixed mask, x is feature data, and ⊙ is an element-wise operation (e.g., element-wise multiplication). An image recognition device can apply a softmax operation 614 to a general feature map 612. generic The output value propagated to the fully connected layer 613 is used to calculate the first recognition data 681. For example, feature data x, general feature map 612. generic And the data output by the fully connected layer 613 has the same size (e.g., 32×32).

[0097] The sensor-specific layer 620 may include an interest layer 621, a fully connected layer 623, and a softmax operation 624. (See reference...) Figure 7 The focus layer 621 is described in more detail. For example, an image recognition device can use the focus layer 621 to compute a focus feature map from feature data x as a sensor-specific feature map 622, as shown in Equation 3 below. specific .

[0098] [Equation 3]

[0099] Feat specfic =M soft ⊙h(x)

[0100] In Equation 3, Feat specific It is a sensor-specific feature map 622, M soft It is a variable mask, and h(x) is the target feature map corresponding to the feature data x. (Refer to...) Figure 7 The calculation describes the target feature map h(x). Image recognition devices can achieve this by applying a softmax operation 624 to the sensor-specific feature map 622. specific The output value, propagated to the fully connected layer 623, is used to calculate the second identification data 682. For example, feature data x, sensor-specific feature map 622. specific And the fully connected layer 623 outputs data of the same size (e.g., 32×32).

[0101] The activity verification model 630 may include a feature extraction layer 605 and a recognizer. The image recognition device uses a fully connected layer 631 and a softmax operation 632 to calculate third recognition data (i.e., a 2D activity score) 683 from the extracted feature data x. For example, fully connected layers 613, 623, and 631 output data of the same size (e.g., 32×32).

[0102] The image recognition device calculates the activity score 690 by applying a weighted sum operation 689 to the first recognition data 681, the second recognition data 682, and the third recognition data 683.

[0103] The image recognition device implements the activity verification model 630, the fixed layer 610, and the sensor-dedicated layer 620 in parallel. For example, the image recognition device can simultaneously or within a short period of time propagate feature data x extracted by the feature extraction layer 605 to the fixed layer 610, the sensor-dedicated layer 620, and the activity verification model 630. However, the embodiments are not limited to the foregoing examples, and the image recognition device can propagate the feature data x sequentially to the activity verification model 630, the fixed layer 610, and the sensor-dedicated layer 620. The first identification data 681, the second identification data 682, and the third identification data 683 can be calculated simultaneously, but are not limited to being calculated simultaneously. The first identification data 681, the second identification data 682, and the third identification data 683 can also be calculated at different times based on the computation time required in each of the fixed layer 610, the sensor-dedicated layer 620, and the activity verification model 630.

[0104] Figure 7 An example of concern layer 700 is shown.

[0105] Reference Figure 7 The image recognition device can use the attention layer 700 to adjust one or more values ​​of the variable mask 706. The attention layer 700 may include a mask adjustment layer 710, a target extraction layer 720, and mask operations. The mask adjustment layer 710 may include a query extraction layer 711 and a key extraction layer 712. The query extraction layer 711, the key extraction layer 712, and the target extraction layer 720 may each include one or more convolutional layers, but are not limited to such a configuration.

[0106] The image recognition device can use the query extraction layer 711 to extract a query feature map f(x) from the feature data 705. The image recognition device can use the key extraction layer 712 to extract a key feature map g(x) from the feature data 705. The image recognition device can use the target extraction layer 720 to extract a target feature map h(x). (See reference...) Figure 2 In the example where the input image is a color image and includes multiple channels (e.g., a three-channel image), feature data 705 is extracted for each channel. The query extraction layer 711, key extraction layer 712, and target extraction layer 720 can be configured to extract features for each channel. As a non-limiting example, "key" and "query" relate to attention mechanisms, attention functions, and attention networks in the field of deep learning. The query feature map is the feature map used as the query, and the key feature map is used as the key. The attention network computes the similarity between a given query and each of all keys. The attention network applies the computed similarity to the values ​​mapped using the key. The attention network returns the sum of the applied similarity values. The returned result will be the attention weights (e.g., the attention mask of the present invention). In other words, a query can be an input used to query which of a plurality of keys should be focused on relative to the query. A key can be the key of a key-value pair. For example, each of the plurality of keys is mapped to a corresponding value, and the similarity between the corresponding key and the query can determine the ratio of applying the corresponding value to the attention weights.

[0107] For example, an image recognition device can use the softmax function to determine the value of a variable mask 706 from the result of multiplying the transposed query feature map f(x) and the key feature map g(x), the result of applying a convolutional filter to the feature data 705. The result of multiplying the key feature map g(x) and the transposed query feature map f(x) indicates the level of similarity between a given query and all keys. The variable mask 706 can be determined as shown in Equation 4 below.

[0108] [Equation 4]

[0109] M soft = softmax(f(x)) T g(x))

[0110] In equation 4, M soft Here, 706 is a variable mask, f(x) is the query feature map, and g(x) is the key feature map. The image recognition device can use the variable mask 706M determined according to Equation 4. soft Applied to the target feature map h(x) according to Equation 3. Sensor-specific feature map 709 indicates the use of variable mask 706M. soft The result of masking the target feature map h(x). A sensor-specific feature map 709 is generated for each channel based on the number of channels.

[0111] The attention layer 700 prevents the vanishing gradient problem by referencing the entire image of the encoder in the decoder for each time point. The attention layer 700 references the entire image by focusing on and identifying portions of the image that have high correlation rather than identical values. Although Figure 7 The attention layer can be shown as a self-attention structure that receives the same feature data as the query, key, and value, but the disclosure is not limited to such an example.

[0112] Figure 8 An example of the structure of the recognition model is shown.

[0113] Reference Figure 8 The recognition model 800 may include a feature extraction layer 810, a fixed layer 820, and first sensor-specific layers 831 to nth sensor-specific layers 832, where n is an integer greater than or equal to 2. Each of the first sensor-specific layers 831 to nth sensor-specific layers 832 may include a variable mask. The value of the variable mask may be adjusted in response to feature data extracted from the input image 801 by the feature extraction layer 810. The image recognition device may calculate a recognition result 809 based on the fixed mask of the fixed layer 820 and multiple variable masks of the multiple sensor-specific layers. The image recognition device may determine the recognition result 809 by summing the recognition data calculated from the fixed layer 820 and the first sensor-specific layers 831 to nth sensor-specific layers 832. For example, the image recognition device may determine the recognition result 809 by a weighted sum of multiple recognition data.

[0114] The parameters of a sensor-dedicated layer comprising one of a plurality of variable masks may differ from the parameters of another sensor-dedicated layer comprising another of a plurality of variable masks. Furthermore, the first sensor-dedicated layers 831 to the nth sensor-dedicated layer 832 may be layers with different structures. For example, one of the first sensor-dedicated layers 831 to the nth sensor-dedicated layer 832 may be implemented as a layer of interest, and the remaining sensor-dedicated layers 831 to the nth sensor-dedicated layer 832 may be implemented with structures other than the layer of interest.

[0115] Figure 9An example of training a recognition model is shown.

[0116] Reference Figure 9 The training device can use training data to train a recognition model. Training data may include a pair of data containing training input and training output. The training input is, for example, an image. The training output is, for example, the ground truth value for the recognition of an object appearing in the corresponding image. The training output may have a value (e.g., 1) indicating that the object appearing in the training input image is a real object or a value (e.g., 0) indicating that the object is a false object. After training, the trained recognition model outputs a real value between 0 and 1 as recognition data. This value may indicate the probability that the object appearing in the input image is a real object. However, the disclosure is not limited to the examples described above.

[0117] The training device can compute temporary outputs by propagating training inputs to a temporary recognition model. The recognition model that has not yet been fully trained can be referred to as a "temporary recognition model." The training device can compute feature data using the feature extraction layer 910 of the temporary recognition model and can propagate the feature data to a fixed layer 920, a sensor-specific layer 930, and a validation layer (or recognizer) 940. During propagation, a temporary general feature map 922 and a temporary interest feature map 932 can be computed. The training device can compute a first temporary output from the fixed layer 920, a second temporary output from the sensor-specific layer 930, and a third temporary output from the validation layer 940. The training device can compute a loss based on a loss function from each of the temporary outputs and the training output. For example, the training device can compute a first loss based on the first temporary output and the training output, a second loss based on the second temporary output and the training output, and a third loss based on the third temporary output and the training output.

[0118] [Equation 5]

[0119] Livenessloss=α*Loss1+β*Loss2+γ*Loss3

[0120] The training device calculates a weighted loss as shown in Equation 5 above. In Equation 5, the liveness loss is the total loss 909, Loss1 is the first loss, Loss2 is the second loss, and Loss3 is the third loss. α is the weight for the first loss, β is the weight for the second loss, and γ is the weight for the third loss. The training device can update the parameters of the provisional recognition model until the total loss 909 reaches the threshold loss. Depending on the design of the loss function, the training device can increase or decrease the total loss 909. For example, the training device updates the parameters of the provisional recognition model through backpropagation.

[0121] The training device can update all parameters of the feature extraction layer 910, the fixed layer 920, the sensor-specific layer 930, and the validation layer 940 for an untrained initial recognition model during training. In this example, the training device can use general training data 901 to train the initial recognition model. General training data 901 includes, for example, images acquired by an image sensor as training input. The training images of general training data 901 can be acquired by one type of image sensor, but can also be acquired by various types of image sensors. The recognition model trained using general training data 901 can also be referred to as a "general recognition model". The general recognition model can be, for example, a model installed on a flagship electronic terminal with high performance. The image sensor of a flagship electronic terminal can have relatively high optical performance. In some cases, the optical characteristics of a particular type of image sensor are not reflected in the general recognition model. In this case, the general recognition model can output false rejection (FR) results and false acceptance (FA) results for the corresponding type of image sensor. FR results indicate results that a true false is identified as a false false. FA results indicate results that a false false is identified as a true false.

[0122] The training device can generate a recognition model for a specific type of image sensor from a general recognition model. For example, during training, the training device can fix the values ​​of the fixed mask 921 included in the fixed layer 920 and the parameters of the verification layer 940 in the general recognition model. The training device can update the parameters of the sensor-specific layer 930 in the temporary recognition model during training. As described above, the training device can calculate the total loss 909 and repeatedly adjust the parameters of the sensor-specific layer 930 until the total loss 909 reaches a threshold loss. For example, the training device updates the parameters (e.g., connection weights) of the attention layer 931 and the parameters of the fully connected layers in the sensor-specific layer 930.

[0123] In this example, the training device uses general training data 901 and sensor-specific training data 902 together to train a sensor-specific layer 930 of the recognition model. Sensor-specific training data 902 consists only of training images acquired by a specific type of image sensor. As described above, various types of image sensors are classified based on their optical characteristics. The training device can update the parameters of the sensor-specific layer 930 using the sensor-specific training data 902, based on a loss calculated in a manner similar to that described above.

[0124] In the early stages of launching a new product, the amount of sensor-specific training data 902 may be insufficient. To prevent overfitting due to insufficient training data, the training device can use general training data 901 for training. The amount of general training data 901 can be greater than the amount of sensor-specific training data 902. For example, in addition to a small amount (e.g., tens of thousands) of sensor-specific training data 902, the training device can use general training data 901 (e.g., a database containing millions of image data) to generate a recognition model with a sensor-specific layer 930 dedicated to various optical characteristics. Thus, the training device can generate a recognition model dedicated to a specific type of image sensor from a general recognition model in a relatively short period of time. In the event of a previously unseen spoofing attack, the training device can learn the parameters of the sensor-specific layer and urgently allocate the trained sensor-specific layer parameters to the image recognition device (e.g., Figure 10 (electronic terminals) to quickly defend against spoofing attacks. Sensor-specific training data 902 may include, for example, images corresponding to newly reported FR and FA results.

[0125] Figure 10 This illustrates an example of parameter updates for the sensor-specific layer in the recognition model.

[0126] Reference Figure 10 The image recognition system may include a training device 1010, a server 1050, and electronic terminals 1060, 1070, and 1080.

[0127] The processor 1011 of the training device 1010 can be referred to as follows Figure 9 The recognition model is trained as described above. Even after the initial training of the initial recognition model 1040 is completed, the training device 1010 can perform additional training on the sensor-specific layer 1043 of the recognition model 1040. For example, in response to the occurrence of a new spoofing attack, the training device 1010 can retrain the sensor-specific layer 1043 of the recognition model 1040 based on the training data associated with the new spoofing attack.

[0128] The memory 1012 of the training device 1010 can store the recognition model 1040 before and after the training of the recognition model 1040 is completed. Furthermore, the memory 1012 can store general training data 1020, sensor-specific training data 1030, and parameters of the feature extraction layer 1041, sensor-specific layer 1043, and fixed layer 1042 of the recognition model 1040. Figure 9 When training is complete, the training device 1010 can assign the trained recognition model 1040 through communication with the server 1050 (e.g., wired or wireless communication).

[0129] Server 1050 may assign a subset, rather than all, of the parameters of recognition model 1040 to each electronic terminal. For example, in response to the completion of additional training of the sensor-specific layer 1043 of recognition model 1040, training device 1010 uploads the parameters of the retrained sensor-specific layer 1043 to server 1050. Server 1050 may provide only the parameters of sensor-specific layer 1043 to electronic terminals 1060, 1070, and 1080 of electronic terminal group 1091 that have a specific type of image sensor. For example, electronic terminals 1060, 1070, and 1080 included in electronic terminal group 1091 are equipped with image sensors having the same or similar optical characteristics. In response to at least one of an update request received from one of the electronic terminals 1060, 1070, and 1080, and the completion of additional training on the sensor-specific layer 1043 of the recognition model 1040, the server 1050 may assign the parameters of the additionally trained sensor-specific layer 1043 to the corresponding electronic terminals 1060, 1070, and 1080. The update request may be a signal from a terminal requesting the server to update the recognition model.

[0130] Although Figure 10 The training device 1010 is shown storing one type of recognition model 1040, but the disclosure is not limited to such an example. The training device may store other types of recognition models and provide updated parameters to another group of terminals 1092.

[0131] Each of the electronic terminals 1060, 1070, and 1080 included in the electronic terminal group 1091 can receive parameters of a sensor-dedicated layer 1043, including a variable mask, from an external server 1050 in response to an update command. The update command can be based on user input or can be a command received by the electronic terminal from the server. The electronic terminals 1060, 1070, and 1080 use the received parameters to update the sensor-dedicated layers 1062, 1072, and 1082, respectively. In this example, the electronic terminals 1060, 1070, and 1080 can fix the parameters of fixed layers 1063, 1073, and 1083, as well as the remaining feature extraction layers 1061, 1071, and 1081. For example, the electronic terminals 1060, 1070, and 1080 can maintain the value of the fixed mask during the update of the parameters of the sensor-dedicated layers 1062, 1072, and 1082. For example, electronic terminals 1060, 1070, and 1080 may maintain the values ​​of a fixed mask before and / or after updating the parameters of sensor-dedicated layers 1062, 1072, and 1082. For instance, in cases where FR and FA results based on the unique optical characteristics of a single image sensor are reported, the training device assigns parameters obtained as a result of training sensor-dedicated layer 1043 based on the FR and FA results.

[0132] The electronic terminal can request sensor-specific parameters from the external server 1050 that correspond to the same or similar optical characteristics as the currently installed image sensor. The server 1050 can retrieve the sensor-specific parameters corresponding to the optical characteristics requested by the electronic terminal and provide the retrieved sensor-specific parameters to the corresponding electronic terminal.

[0133] Although, as an example, Figure 10 The example shown illustrates how server 1050 assigns parameters to sensor-specific layer 1043, but the disclosure is not limited to this example. When the value of the fixed mask in fixed layer 1042 is changed, server 1050 can assign the value to electronic terminals 1060, 1070, and 1080. If necessary, electronic terminals 1060, 1070, and 1080 can update fixed layers 1063, 1073, and 1083. For example, when general FR and FA results, independent of the unique optical characteristics of each image sensor, are reported, the training device can adjust the value of the fixed mask in fixed layer 1042. For reference, updating the fixed mask can improve recognition performance in various electronic terminals, including those with general-purpose image sensors of various types. Updating a variable mask corresponding to individual optical characteristics can improve recognition performance in electronic terminals, including those with image sensors having corresponding optical characteristics.

[0134] If a neural network is trained on data acquired using a specific device, that device can achieve a high recognition rate. However, when the same neural network is installed in another device, the recognition rate may decrease. (See reference...) Figure 9 and Figure 10 The recognition model can have sensor-specific layers for each type of image sensor through some additional training instead of retraining the entire network. This allows for more robust protection of the privacy and security of electronic terminals 1060, 1070, and 1080, as emergency patches for the recognition model are possible.

[0135] Figure 11 and Figure 12 This is a block diagram illustrating an example of an image recognition device.

[0136] Reference Figure 11 The image recognition device 1100 includes, for example, an image sensor 1110, a processor 1120, and a memory 1130.

[0137] Image sensor 1110 can receive an input image. Image sensor 1110 can be, for example, a camera sensor that captures color images. Alternatively, image sensor 1110 can be a dual-phase detection (2PD) sensor that uses the difference between the left and right phases to acquire a disparity image of one pixel. Since the disparity image is directly generated by the 2PD sensor, a depth image can be calculated from the disparity image without using a stereo sensor and typical depth extraction techniques.

[0138] Unlike Time-of-Flight (ToF) and structured light depth sensors, the 2PD sensor is mounted on the image recognition device 1100 without incurring additional form factor and sensor costs. For example, unlike a contact image sensor (CIS), the 2PD sensor includes multiple sensing elements, each comprising two photodiodes (e.g., a first photodiode and a second photodiode). Therefore, two images are generated by capturing images with the 2PD sensor. The two images include one sensed by the first photodiode (e.g., the left photodiode) and the other sensed by the second photodiode (e.g., the right photodiode). The two images differ slightly due to the physical distance between the photodiodes. The image recognition device 1100 can use the two images to calculate the parallax caused by the distance difference based on triangulation, etc., and estimate the depth of each pixel from the calculated parallax. Unlike a CIS that outputs three channels, the 2PD sensor outputs one channel image for each of the two photodiodes, thereby reducing memory usage and computational load. This is because a pair of channel images (e.g., two channels in total) are needed to estimate disparity from images acquired by a 2PD sensor, while three pairs of channel images (e.g., six channels in total) are needed to estimate disparity from images acquired by a CIS.

[0139] However, the foregoing description is merely an example, and the image sensor 1110 may include infrared sensors, radar sensors, ultrasonic sensors, depth sensors, etc.

[0140] Processor 1120 can use a feature extraction layer to extract feature data from the input image. Processor 1120 can output recognition results of objects appearing in the input image based on variable and fixed masks adjusted in response to the extracted feature data. When receiving parameters of the sensor-specific layer from a server via communication, processor 1120 can update the parameters of the sensor-specific layer stored in memory 1130.

[0141] The memory 1130 can temporarily or permanently store the recognition model and the data generated during the implementation of the recognition model. When new parameters of the sensor-specific layer are received from the server, the memory 1130 can replace the existing parameters with the received new parameters.

[0142] Reference Figure 12The computing device 1200 can be an apparatus for recognizing images using the image recognition methods described herein. In one example, the computing device 1200 may correspond to... Figure 10 Electronic terminals and / or Figure 11 Image recognition device 1100. For example, computing device 1200 may be an image processing device, smartphone, wearable device, tablet computer, netbook, laptop computer, desktop computer, personal digital assistant (PDA), head-mounted display (HMD), etc.

[0143] Reference Figure 12 The computing device 1200 includes, for example, a processor 1210, a storage device 1220, a camera 1230, an input device 1240, an output device 1250, and a network interface 1260. The processor 1210, storage device 1220, camera 1230, input device 1240, output device 1250, and network interface 1260 can communicate with each other via a communication bus 1270.

[0144] Processor 1210 executes functions and instructions in computing device 1200. For example, processor 1210 processes instructions stored in storage device 1220. Processor 1210 can execute the above-mentioned functions and instructions. Figures 1 to 11 One or more operations described.

[0145] Storage device 1220 stores information and data required for execution by processor 1210. Storage device 1220 may include a computer-readable storage medium or a computer-readable storage device. Storage device 1220 stores instructions to be executed by processor 1210, and stores relevant information while software or applications are being executed by computing device 1200.

[0146] Camera 1230 captures an input image for each image recognition. Camera 1230 may capture multiple images (e.g., multiple frame images). Processor 1210 may output the recognition result of a single image using the recognition model described above.

[0147] Input device 1240 can receive input from a user via tactile, video, audio, or touch input. For example, input device 1240 may include a keyboard, mouse, touchscreen, microphone, and / or other devices that can detect input from the user and transmit the detected input.

[0148] Output device 1250 provides the output of computing device 1200 to a user via a visual channel, an auditory channel, or a tactile channel. For example, output device 1250 may include a display, touchscreen, speaker, vibration generator, and / or other devices that can provide output to a user. Network interface 1260 communicates with external devices via a wired or wireless network. Output device 1250 uses any one or any combination of two or more visual, auditory, and tactile information to provide the user with the recognition result of the input data (e.g., granting and / or denying access).

[0149] The computing device 1200 may grant permissions based on the identification result. The computing device 1200 may, based on the permissions, allow access to at least one of its operations and data. As an example, in response to verifying from the identification result that a user is a registered user in the computing device 1200 and is a genuine object, the computing device 1200 may grant permissions. When the computing device 1200 is locked, it may be unlocked based on the permissions. As another example, in response to verifying from the identification result that a user is a registered user in the computing device 1200 and is a genuine object, the computing device 1200 may allow access to financial payment functions. As yet another example, after generating the identification result, the computing device 1200 visualizes the identification result through an output device 1250 (e.g., a display).

[0150] Figures 1 to 12The neural network 100, recognition models 310, 400, 800, and 1040, processors 1011, 1120, and 1210, memory 1012 and 1130, server 1050, storage device 1220, input device 1240, output device 1250, network interface 1260, neural network, recognition model, processor, memory, and other devices, apparatuses, units, modules, and components that perform the operations described in this application are implemented by hardware components, wherein the hardware components are configured to perform the operations performed by the hardware components described in this application. Examples of hardware components that can be used to perform the operations described in this application include, where appropriate, controllers, sensors, generators, drivers, memory, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components performing the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer may be implemented by one or more processing elements, such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field-programmable gate arrays, programmable logic arrays, microprocessors, or any other means or combination of means configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, the processor or computer includes or is connected to one or more memories storing instructions or software executed by the processor or computer. Hardware components implemented by the processor or computer may execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) to perform the operations described in this application. The hardware components may also access, manipulate, process, create, and store data in response to the execution of instructions or software. For simplicity, the singular terms “processor” or “computer” may be used in the description of the examples described in this application, but in other examples, multiple processors or computers may be used, or a processor or computer may include multiple processing elements or multiple types of processing elements or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or processors and controllers. One or more other hardware components may be implemented by one or more other processors, or additional processors and additional controllers. One or more processors, or processors and controllers, may implement a single hardware component, or two or more hardware components.The hardware components may have any one or more different processing configurations, examples of which include: a single processor, a discrete processor, a parallel processor, a single instruction single data (SISD) multiprocessing, a single instruction multiple data (SIMD) multiprocessing, multiple instruction single data (MISD) multiprocessing, and multiple instruction multiple data (MIMD) multiprocessing.

[0151] Figures 1 to 12 The methods for performing the operations described in this application, as shown, are executed by computing hardware (e.g., one or more processors or a computer), wherein the computing hardware is implemented to execute instructions or software as described above to perform the operations performed by the methods described in this application. For example, a single operation or two or more operations may be executed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be executed by one or more processors, or a processor and a controller, and one or more other operations may be executed by one or more other processors, or additional processors and additional controllers. One or more processors, or a processor and a controller, may execute a single operation or two or more operations.

[0152] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above are written as computer programs, code segments, instructions, or any combination thereof to individually or collectively instruct or configure one or more processors or computers to operate as a machine or special-purpose computer to perform the operations performed by the hardware components and methods described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) that is directly executed by one or more processors or computers. In another example, the instructions or software contain high-level code that is executed by one or more processors or computers using an interpreter. The instructions or software can be written using any programming language based on the block diagrams and flowcharts shown in the accompanying drawings and the corresponding descriptions in the specification, wherein the block diagrams and flowcharts shown in the accompanying drawings and the corresponding descriptions in the specification disclose algorithms for performing the operations performed by the hardware components and methods described above.

[0153] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above, along with any associated data, data files, and data structures, may be recorded, stored, or fixed on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access memory (RAM), flash memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other means, wherein said other means is configured to store instructions or software and any associated data, data files, and data structures in a non-transitory manner and to provide the instructions or software and any associated data, data files, and data structures to one or more processors or computers, such that one or more processors or computers can execute the instructions. In one example, instructions or software, along with any associated data, data files, and data structures, are distributed across a networked computer system, such that the instructions or software, along with any associated data, data files, and data structures, are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0154] While this disclosure includes specific examples, it will be clear upon understanding this disclosure that various changes in form and detail may be made to these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are to be considered descriptive only and not for limiting purposes. The description of features or aspects in each example will be considered applicable to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order, and / or if components in the described system, architecture, apparatus, or circuit are combined in a different manner, and / or replaced or supplemented by other components or their equivalents. Therefore, the scope of this disclosure is not limited by the specific embodiments but by the claims and their equivalents, and all variations within the scope of the claims and their equivalents should be construed as included in this disclosure.

Claims

1. An image recognition method, the image recognition method comprising: Feature extraction layers are used to extract feature data from the input image received by the image sensor; as well as By applying fixed and variable masks to the extracted feature data, the output shows the recognition results of objects appearing in the input image. The variable mask is adjusted in response to the extracted feature data. The step of outputting the recognition result includes: using at least a portion of a sensor-specific layer including a variable mask, adjusting one or more values ​​of the variable mask based on the extracted feature data.

2. The image recognition method according to claim 1, wherein, The steps for outputting the recognition results include: First identification data is calculated from the extracted feature data by applying a fixed mask to the extracted feature data. Second identification data is calculated from the extracted feature data by applying a variable mask to the extracted feature data; and The identification result is determined based on the first identification data and the second identification data.

3. The image recognition method according to claim 2, wherein, The steps for calculating the first identification data include: Generate a general feature map related to the region of interest by applying a fixed mask to the extracted feature data; and The first identification data is calculated from the general feature map.

4. The image recognition method according to claim 2, wherein, The steps for calculating the second identification data include: Sensor-specific feature maps related to the region of interest of the image sensor are generated by applying a variable mask to the target feature map corresponding to the extracted feature data; and The second identification data is calculated from the sensor-specific feature map.

5. The image recognition method according to claim 4, wherein, The steps for generating sensor-specific feature maps include: Sensor-specific feature maps are generated by applying corresponding values ​​from a variable mask to individual values ​​in the target feature map.

6. The image recognition method according to claim 2, further comprising: The third identification data is calculated from the extracted feature data using the softmax function and fully connected layers. The steps for determining the recognition result include: In addition to the first and second identification data, the identification result is also determined based on the third identification data.

7. The image recognition method according to claim 1, wherein, The steps of adjusting one or more values ​​of the variable mask include: The value of the variable mask is determined using the softmax function from the result of multiplying the transposed query feature map and the key feature map, which corresponds to the result of applying convolutional filtering to the extracted feature data.

8. The image recognition method according to claim 1, wherein, The steps for outputting the recognition results include: The weighted sum of the first identification data based on a fixed mask and the second identification data based on a variable mask is determined as the identification result.

9. The image recognition method according to claim 8, wherein, The steps to determine the weighted sum as the identification result include: Apply a weight that is greater than the weight applied to the first identification data to the second identification data.

10. The image recognition method according to claim 1, further comprising: In response to the update command, parameters, including a sensor-specific layer with a variable mask, are received from an external server. as well as Update the sensor-specific layer using the received parameters.

11. The image recognition method according to claim 10, further comprising: Request sensor-specific parameters from an external server. These parameters correspond to optical characteristics that are similar to or the same as those of the image sensor.

12. The image recognition method according to claim 10, further comprising: While updating the parameters of the sensor-specific layer, the values ​​of the fixed mask are maintained.

13. The image recognition method according to claim 1, wherein, A variable mask includes multiple variable masks.

14. The image recognition method according to claim 13, wherein, The parameters of a sensor-specific layer that includes one of the plurality of variable masks are different from the parameters of another sensor-specific layer that includes another of the plurality of variable masks.

15. The image recognition method according to claim 1, wherein, The steps for outputting the recognition results include: Generate information indicating whether the object is a real object or a pseudo object as the recognition result.

16. The image recognition method according to claim 1, further comprising: Grant permissions based on the identification results; as well as Based on permissions, access is permitted to any one or both of the operations and data of the electronic terminal.

17. The image recognition method according to claim 1, wherein, The steps for outputting the recognition results include: After generating the recognition results, visualize the results on the monitor.

18. An image recognition method, the image recognition method comprising: Feature extraction layers are used to extract feature data from the input image received by the image sensor; as well as By applying fixed and variable masks to the extracted feature data, the output shows the recognition results of objects appearing in the input image. The variable mask is adjusted in response to the extracted feature data. The image recognition method further includes: granting permissions based on the recognition results; and allowing access to either or both of the operations and data of the electronic terminal based on the permissions.

19. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the image recognition method as claimed in claim 1.

20. An image recognition device, comprising: An image sensor is configured to acquire an input image; as well as The processor is configured as follows: Use a feature extraction layer to extract feature data from the input image; as well as By applying fixed and variable masks to the extracted feature data, the output shows the recognition results of objects appearing in the input image. The variable mask adjusts in response to the extracted feature data. The step of outputting the recognition result includes: using at least a portion of a sensor-specific layer including a variable mask, adjusting one or more values ​​of the variable mask based on the extracted feature data.

21. The image recognition device according to claim 20, wherein, The steps for outputting the recognition results include: First identification data is calculated from the extracted feature data by applying a fixed mask to the extracted feature data. Second identification data is calculated from the extracted feature data by applying a variable mask to the extracted feature data; and The identification result is determined based on the sum of the first identification data and the second identification data.

22. The image recognition device according to claim 21, wherein, The sum is determined by applying a larger weight to the second identification data than to the first identification data.

23. The image recognition device according to claim 21, wherein, The steps for calculating the first identification data include: Generate a general feature map related to the region of interest by applying a fixed mask to the extracted feature data; and The first recognition data is calculated from the general feature map, and The steps for calculating the second identification data include: Sensor-specific feature maps related to the region of interest of the image sensor are generated by applying a variable mask to the target feature map corresponding to the extracted feature data; and The second identification data is calculated from the sensor-specific feature map.

24. An image recognition system, comprising: The image recognition device is configured as follows: The feature extraction layer is used to extract feature data from the received input image; as well as By applying fixed and variable masks to the extracted feature data, the output shows the recognition results of objects appearing in the input image; The variable mask is included in the sensor-specific layer of the image recognition device and is adjusted in response to the extracted feature data; and The server is configured to: in response to either or both of an update request from the image recognition device and the completion of additional training of the sensor-specific layer of the server's recognition model, assign parameters of the additionally trained sensor-specific layer to the image recognition device. The image recognition device is configured to update the sensor-specific layer of the image recognition device based on the assigned parameters.

25. The image recognition system according to claim 24, wherein, The server is also configured to assign parameters of an additional trained sensor-specific layer to another image recognition device, which includes an image sensor identified as being the same as or similar to the image sensor of the image recognition device.

Images