Active testing apparatus and method

Abstract

Disclosed are activity test apparatuses and methods. The activity test method includes detecting a face region in an input image, generating weight map data related to a face position in the input image based on the detected face region, generating spliced data by splicing the weight map data with feature data generated from an intermediate layer of an activity test model or image data of the input image, and generating an activity test result based on an activity score generated by the activity test model to which the spliced data is provided.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2022-0001677, filed on January 5, 2022, with the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference for all purposes. Technical Field

[0002] The following description relates to devices and methods with activity considerations. Background Technology

[0003] In a user authentication system, a computing device can determine whether to allow a user to access the computing device based on authentication information provided by the user. For example, authentication information may include a password entered by the user or the user's biometric information. Biometric information includes information related to fingerprints, irises, or faces.

[0004] Facial deception techniques can be used to improve the security of user authentication systems. Facial deception determines whether a user's face input to a computing device is fake or real. To this end, features (such as Local Binary Pattern (LBP), Histogram of Oriented Gradients (HOG), and Difference of Gaussians (DoG)) can be extracted from the input image, and the authenticity of the input face can be determined based on these extracted features. Facial deception is a form of attack that uses photographs, moving images, or masks, and identifying such attacks is important in facial verification processing. Summary of the Invention

[0005] The present invention is provided in a simplified form to introduce the choice of concepts further described in the following detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0006] In one general aspect, a method includes: detecting facial regions in an input image; generating weighted map data related to facial locations in the input image based on the detected facial regions; generating stitched data by concatenating the weighted map data with feature data generated from intermediate layers of an activity testing model or image data of the input image; and generating an activity test result based on an activity score generated by the activity testing model provided with the stitched data.

[0007] The weighted map data may include a first region corresponding to facial regions in the input image and a second region corresponding to non-facial regions in the input image. The weights of the first region and the second region may be different from each other.

[0008] The weights of the weighted map data can vary based on the distance from the center of the corresponding region in the weighted map data that corresponds to the facial region.

[0009] The weighted map data may include: a reduced region of the corresponding region, the corresponding region, and an expanded region of the corresponding region. The reduced region, the corresponding region, and the expanded region may overlap and be arranged based on the center of the corresponding region.

[0010] The first weight of the reduced region can be greater than the second weight of the region between the corresponding region and the reduced region, and the second weight can be greater than the third weight of the region between the expanded region and the corresponding region.

[0011] The activity testing model may include providing concatenated data to another intermediate layer of the activity testing model in response to generating concatenated data by concatenating weighted graph data with feature data generated from the intermediate layer. This other intermediate layer may be located after the first intermediate layer.

[0012] The liveness test model may include: providing stitched data to the input layer of the liveness test model in response to generating stitched data by stitching weight map data with image data of the input image.

[0013] The steps to generate weighted graph data may include: using a neural network-based weighted graph generation model to generate weighted graph data.

[0014] The weight of the first region corresponding to the facial region in the weight map data may be different from the weight of the second region corresponding to the occluded region in the facial region in the weight map data.

[0015] The steps for generating stitched data may include: adjusting the size of the weighted graph data to correspond to the size of the feature data, and generating stitched data by stitching the feature data with the resized weighted graph data.

[0016] In one general aspect, embodiments include a non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform any one or any combination of two or more of the operations described herein, or all of them.

[0017] In another general aspect, an apparatus includes: a processor configured to: detect facial regions in an input image; generate weighted map data related to facial locations in the input image based on the detected facial regions; generate stitched data by stitching the weighted map data with feature data generated from intermediate layers of an activity testing model or image data of the input image; provide the stitched data to an activity testing model; and determine an activity test result based on an activity score determined by the activity testing model.

[0018] The weighted map data may include a first region corresponding to facial regions in the input image and a second region corresponding to non-facial regions in the input image. The weights of the first region and the second region may be different from each other.

[0019] The weights of the weighted map data can vary based on the distance from the center of the corresponding region in the weighted map data that corresponds to the facial region.

[0020] The processor can also be configured to provide spliced ​​data to another intermediate layer of the liveness test model in response to generating spliced ​​data by concatenating weight map data with feature data generated from the intermediate layer, and the other intermediate layer may be located after the intermediate layer.

[0021] The processor can also be configured to provide stitched data to the input layer of the liveness test model in response to generating stitched data by stitching weight map data with image data of the input image.

[0022] The processor can also be configured to generate weight map data using a neural network-based weight map generation model. The weights of the first region in the weight map data corresponding to the facial region may differ from the weights of the second region in the weight map data corresponding to the occluded region within the facial region.

[0023] The device may also include a memory for storing instructions. The processor may also be configured to execute instructions that configure the processor to perform the following operations: detect facial regions, generate weighted map data based on the detected facial regions, generate stitched data, and determine activity test results.

[0024] In another general aspect, an electronic device includes: a camera configured to: acquire an input image; and a processor. The processor is configured to: detect facial regions in the input image; generate weighted map data related to facial positions in the input image based on the detected facial regions; generate stitched data by stitching the weighted map data with feature data generated from intermediate layers of an activity testing model fed from the input image or image data of the input image; provide the stitched data to the activity testing model; and determine an activity test result based on an activity score determined by the activity testing model.

[0025] The weighted map data may include a first region corresponding to a facial region in the input image and a second region corresponding to a non-facial region in the input image, and the weights of the first region and the second region may be different from each other.

[0026] The processor can also be configured to provide spliced ​​data to another intermediate layer of the liveness test model in response to generating spliced ​​data by concatenating weight map data with feature data generated from the intermediate layer, and the other intermediate layer may be located after the intermediate layer.

[0027] In another general aspect, an apparatus includes: a processor configured to: detect facial regions in an input image; generate weighted map data relating to facial locations in the input image based on the detected facial regions, the weighted map data including multiple regions, each with different weights; generate stitched data by concatenating one of the regions of the weighted map data with feature data generated from layers of an activity testing model; provide the stitched data to an activity testing model; and determine an activity test result based on an activity score determined by the activity testing model.

[0028] The first region among the multiple regions can correspond to a facial region, and the second region among the multiple regions can correspond to a non-facial region in the input image.

[0029] The first region among the multiple regions can correspond to the reduced region of the facial region, the second region among the multiple regions can correspond to the facial region, and the third region among the multiple regions can correspond to the expanded region of the facial region.

[0030] The activity testing model can be a machine learning model or a neural network model.

[0031] Other features and aspects will become clear from the following detailed description, drawings, and claims. Attached Figure Description

[0032] Figure 1 and Figure 2 Examples of biometric authentication and activity testing according to one or more embodiments are shown.

[0033] Figure 3 This is a flowchart illustrating an example of an activity testing method according to one or more embodiments.

[0034] Figure 4 Examples of generating weighted graph data according to one or more embodiments are shown.

[0035] Figure 5 Another example of generating weighted graph data according to one or more embodiments is shown.

[0036] Figure 6A An example is shown of inputting spliced ​​data into an activity test model according to one or more embodiments.

[0037] Figure 6B An example is shown of inputting spliced ​​data into an activity test model according to one or more embodiments.

[0038] Figure 7 This is a block diagram illustrating an example configuration of an activity testing device according to one or more embodiments.

[0039] Figure 8This is a block diagram illustrating an example configuration of an electronic device according to one or more embodiments.

[0040] Figure 9 Examples of training an activity test model according to one or more embodiments are shown.

[0041] Throughout the accompanying drawings and detailed embodiments, unless otherwise described or provided, the same reference numerals will be understood to denote the same elements, features, and structures. The drawings may not be to scale, and for clarity, illustration, and convenience, the relative sizes, proportions, and depictions of elements in the drawings may be exaggerated. Detailed Implementation

[0042] The following detailed embodiments are provided to aid the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but may be changed as will become clear upon understanding this disclosure, except for operations that must occur in a specific order. Furthermore, for clarity and brevity, descriptions of features known upon understanding this disclosure may be omitted.

[0043] The features described herein may be implemented in different forms and should not be construed as limited to the examples described herein. Rather, the examples described herein are provided merely to illustrate some of the many feasible ways of implementing the methods, apparatus, and / or systems described herein that will be clear upon understanding the disclosure of this application.

[0044] Throughout the specification, when an element (such as a layer, region, or substrate) is described as being "on," "connected to," or "bonded to" another element, the element may be directly "on," directly "connected to," or directly "bonded to" the other element, or one or more other elements may be present in between. Conversely, when an element is described as being "directly" "on," "directly connected to," or "directly bonded to" another element, no other elements may be present in between.

[0045] As used herein, the term “and / or” includes any one of the associated listed items and any combination of any two or more.

[0046] Although terms such as “first,” “second,” and “third” may be used herein to describe various components, assemblies, regions, layers, or parts, these components, assemblies, regions, layers, or parts should not be limited by these terms. Rather, these terms are used only to distinguish one component, assembly, region, layer, or part from another. Thus, without departing from the teaching of the examples described herein, the first component, first assembly, first region, first layer, or first part referred to as the first component, first assembly, first region, first layer, or first part may also be referred to as the second component, second assembly, second region, second layer, or second part.

[0047] For ease of description, spatial relative terms (such as "above," "upper," "lower," and "below") are used herein to describe the relationship between one element and another shown in the figure. In addition to the orientations depicted in the figure, such spatial relative terms are intended to encompass different orientations of the device during use or operation. For example, if the device in the figure is flipped, an element described as "above" or "upper" relative to another element will be "below" or "below" relative to that element. Thus, depending on the spatial orientation of the device, the term "above" includes both upper and lower orientations. The device may also be oriented in other ways (e.g., rotated 90 degrees or in other orientations), and the spatial relative terms used herein will be interpreted accordingly.

[0048] The terminology used herein is for the purpose of describing various examples only and is not intended to limit this disclosure. Unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. The terms “comprising,” “including,” and “having” indicate the presence of the features, quantities, operations, components, elements, and / or combinations thereof stated, but do not preclude the presence or addition of one or more other features, quantities, operations, components, elements, and / or combinations thereof.

[0049] Due to manufacturing techniques and / or errors, variations in the shapes shown in the accompanying drawings may occur. Therefore, the examples described herein are not limited to the specific shapes shown in the drawings, but include shape variations that occur during manufacturing.

[0050] As will be clear upon understanding the disclosure of this application, the features of the examples described herein can be combined in various ways. Furthermore, although the examples described herein have various configurations, other configurations are also possible, as will be clear upon understanding the disclosure of this application.

[0051] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains, upon understanding the disclosure of this application. Unless so expressly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having a meaning consistent with their meaning in the relevant field and in the context of the disclosure of this application, and shall not be interpreted in an idealized or overly formal sense.

[0052] In the following description, examples will be described in detail with reference to the accompanying drawings. When describing examples with reference to the accompanying drawings, the same reference numerals denote the same elements, and repeated descriptions related to the same reference numerals will be omitted.

[0053] Figure 1 and Figure 2 Examples of biometric authentication and liveness (or in vivo) testing according to one or more embodiments are shown.

[0054] Biometric authentication is an authentication technology that uses personal biometric information (such as fingerprints, irises, faces, veins, and skin) to verify users. In biometric authentication, facial verification is an authentication technology that uses a user's facial information to determine whether a user attempting authentication is a valid user. For example, facial verification can be used to authenticate users for login, payment services, and access control.

[0055] Reference Figure 1 Electronic device 120 (e.g., Figure 8 Electronic device 120 (800) can authenticate an object 110 (e.g., a user) attempting to access electronic device 120 via biometric authentication. Electronic device 120 can use a camera 130 included in electronic device 120 to acquire image data related to object 110 and determine the authentication result by analyzing the acquired image data. The biometric authentication process may include extracting features from the image data, comparing the extracted features with registered features of a valid object, and determining whether authentication was successful based on the comparison result. For example, assuming electronic device 120 is locked, it can be unlocked when authentication of object 110 is determined to be successful; conversely, it can remain locked or prevent access to object 110 when authentication of object 110 is determined to be unsuccessful. It should be noted that the use of the term "may" in relation to examples or embodiments (e.g., what an example or embodiment may include or implement) indicates the existence of at least one example or embodiment that includes or implements such features, but all examples and embodiments are not limited thereto.

[0056] The valid user of electronic device 120 can pre-register their biometric features with the electronic device 120 through a registration process. The electronic device 120 can store information used to identify the valid user in a storage device or cloud storage. For example, a facial image or facial features of the valid user extracted from a facial image can be stored as the valid user's registered biometric features.

[0057] An activity test can be performed in the biometric authentication process described above. The activity test checks whether the object 110 is alive and optionally determines whether the biometric authentication method is genuine. For example, the activity test checks whether a face appearing in an image captured by camera 130 is a real human face or a fake face. The activity test can distinguish between inanimate objects (e.g., photographs, paper, moving images, models, masks, etc.) and living objects (e.g., a real human face). Here, the term "activity test" can be replaced by "activity detection." According to the example, electronic device 120 can perform either or both of the activity test and biometric authentication.

[0058] Figure 2 Examples of a fake face 210 and a real face 220 according to one or more embodiments are shown. Electronic device 120 can identify whether a face shown in an image is a real face 220 via an activity test. In one example, electronic device 120 can distinguish between a fake face 210 and a real face 220 shown in any of a screen, photograph, paper, model, etc., via an activity test.

[0059] Reference Figure 1 Invalid users can attempt to use deception techniques to cause false acceptance by the biometric authentication system. For example, in facial verification processing, an invalid user could present a substitute (such as a photograph, motion image, or facial model showing the face of an valid user) to camera 130 to cause false acceptance. The liveness test filters out authentication attempts (or deception attacks) made using these substitutes to prevent false acceptance. As a result of the liveness test, if the object to be authenticated is determined to be inanimate, a final determination of authentication failure can be made without performing authentication to determine the object's validity. However, even in examples where authentication processing is performed, a final determination of authentication failure can be made regardless of the authentication outcome.

[0060] Electronic device 120 can perform an activity test using facial location information during activity test processing. Electronic device 120 can detect facial regions in image data and use the location information of the detected facial regions to generate weighted map data (or facial location map data) related to the location of the facial regions. Electronic device 120 can use the weighted map data by combining the weighted map data with image data or from an activity test model (e.g., Figure 6AThe activity test model 620 determines activity by concatenating the feature data output from its layers. The electronic device 120 can determine the activity test results by performing activity tests using image data and weighted map data, taking into account facial features and features surrounding the face, including contextual information. Activity tests are more accurate when using weighted map data than when not using it.

[0061] Figure 3 This is a flowchart illustrating an example of an activity testing method according to one or more embodiments. The activity testing method can be performed using an activity testing device (e.g., Figure 7 The activity testing equipment (700) is used for the test.

[0062] Reference Figure 3 In operation 310, the activity testing device can detect facial regions in an input image (or face image). Facial regions may include the main parts of the face shown in the input image (e.g., eyes, nose, mouth, and eyebrows), or only a portion of the face, not necessarily the entire face. The activity testing device may receive the input image to determine activity and use at least one of various facial region detection techniques to detect facial regions in the input image. For example, the activity testing device may use a Haar-based cascaded adaptive boosting (Adaboost) classifier, a neural network trained to detect facial regions, or a Viola-Jones detector to detect facial regions in the input data. However, the scope of the example is not limited to this, and the activity testing device may use various other facial region detection techniques to detect facial regions. For example, the activity testing device may detect facial landmarks in the input image and detect the boundary regions of the face including the detected landmarks as facial regions.

[0063] In one example, the activity testing device can detect reference coordinates, height, and width from a reference location to define a facial region in an input image, thereby defining the facial region. The facial region can be detected as, for example, a square region, and in this example, the reference coordinates can be the two-dimensional coordinates of the top-left vertex of the detected square region. However, the example is not limited to this, and the facial region can be detected as, for example, a circle, an ellipse, or a polygon, and the reference coordinates can be defined as, for example, the center position of the facial region or the position of another vertex.

[0064] In operation 320, the activity testing device can generate weighted map data related to facial positions in the input image based on detected facial regions. The weighted map data can represent a weight distribution based on the positions of facial regions in the input image. According to examples, the weighted map data generated by the activity testing device can be defined in various forms. For example, the weighted map data can include a first region corresponding to facial regions in the input image and a second region corresponding to non-facial regions in the input image, and the weights assigned to the first region and the weights assigned to the second region can be different from each other. For example, the weight assigned to the first region can be greater than the weight assigned to the second region.

[0065] In one example, the weight map data may have weights that vary based on the distance from the center of the corresponding region of the weight map data corresponding to the facial region of the input image. The weight map data may include a reduced region of the corresponding region, a total corresponding region, and an extended region of the corresponding region. The reduced region, corresponding region, and extended region may overlap to be arranged based on the center of the corresponding region. In one example, the reduced region, corresponding region, and extended region may include, but are not limited to, the center of the corresponding region. Here, a first weight assigned to the reduced region may be greater than a second weight assigned to the region between the corresponding region and the reduced region, and the second weight may be greater than a third weight assigned to the region between the extended region and the corresponding region.

[0066] In one example, the activity testing device can use a neural network-based weighted graph generation model to generate weighted graph data. An input image can be fed into the weighted graph generation model, and the model can output weighted graph data related to the location of facial regions in the input image. The weighted graph generation model can be a model trained based on training data (e.g., training images) and expected weighted graph data corresponding to the training data. During the training process, the model can update its parameters (e.g., connection weights) to output weighted graph data most similar to the expected weighted graph data corresponding to the input training data. When using a weighted graph generation model, the weights of occluded regions generated by obstacles or accessories (e.g., hats, face masks, sunglasses, and glasses) that may be present in the facial region can be determined more accurately. In the weighted graph data generated by the weighted graph generation model, the weights assigned to a first region of the weighted graph data corresponding to the facial region may differ from the weights assigned to a second region of the weighted graph data corresponding to an occluded region in the facial region. The weight assigned to the first region may be greater than the weight assigned to the second region.

[0067] In operation 330, the activity testing device can generate stitched data by concatenating weighted map data with feature data output from the first intermediate layer of the activity testing model or image data of the input image. The activity testing model (e.g., Figure 6A Activity test model 620 and Figure 6B The activity testing model 650 can be a model that outputs an activity score based on an input image, and can be based on a neural network (e.g., a convolutional neural network (CNN)). When concatenating weight map data with feature data or image data output from the first intermediate layer, the activity testing device can adjust the size (or resolution) of the weight map data to correspond to the size (or resolution) of the feature data or image data, and generate concatenated data by concatenating the feature data or image data with the resized weight map data.

[0068] In operation 340, the activity testing device can input stitched data into the activity testing model. When stitched data is generated by stitching weight map data with feature data output from the first intermediate layer of the activity testing model, the activity testing device can input the stitched data into the second intermediate layer of the activity testing model. The second intermediate layer can be the layer above the first intermediate layer (i.e., the layer following the first intermediate layer). When stitched data is generated by stitching weight map data with image data of the input image, the activity testing device can input the stitched data into the input layer of the activity testing model.

[0069] In operation 350, the activity testing device may determine the activity test result based on an activity score determined by an activity testing model. The activity testing model may be a neural network trained to output an activity score (or determine whether an object is active) based on input data (e.g., image data of an input image, or concatenated data of an input image and weighted graph data). The activity testing model may output a value calculated through intrinsic parameters as the activity score in response to input data. The activity testing model may be, for example, a deep neural network (DNN), a CNN, a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), and a bidirectional recurrent DNN (BRDNN), a deep Q-network, or a combination of two or more of these, but is not limited to the foregoing examples. The activity testing model may be implemented by hardware including a neural processor or a combination of hardware and instructions executed by the hardware (e.g., as a processor configured to execute instructions that configure the processor to implement the activity testing model).

[0070] The activity score output from the activity testing model can be a reference value used to determine whether a test object is alive, and represents a value (such as a numerical value, probability value, or feature value) indicating whether the object (i.e., the test object) corresponds to a real or fake object based on the input data. In one example, the activity score could represent a value indicating whether a facial object corresponds to a real or fake face. The activity testing device can determine whether an object is alive based on whether the activity score meets preset conditions. For example, when the activity score is greater than a threshold, the activity testing device can identify the object as a real, alive object; when the activity score is less than or equal to the threshold, the activity testing device can identify the object as a lifeless, fake object. In one example, when the activity score is a value indicating whether an object corresponds to a real or fake object, the activity testing device can determine whether the object is alive based on the activity score.

[0071] Figure 4 Examples of generating weighted graph data according to one or more embodiments are shown.

[0072] Reference Figure 4 When an activity test is performed on an input image 410 in which object 420 is shown, the activity test device can detect the facial region 430 of object 420 in the input image 410. The position and size of the facial region 430 can be defined, for example, by the coordinates of the upper left vertex of a square defining the facial region 430, as well as the width and height of the square.

[0073] The activity testing device can generate weighted map data 440 by converting the position information of the facial region 430 into a weighted map based on its position. The weighted map data 440 can be the position information of the facial region 430 converted into a two-dimensional (2D) map, and can include weight information based on the position of the facial region 430.

[0074] In the weighted map data 440, different weights can be assigned to the corresponding regions of the weighted map data 440 that correspond to the facial region 430 and the regions that correspond to the non-facial regions. Furthermore, the weighted map data 440 can be implemented with weights that vary according to the distance from the center of the corresponding region of the weighted map data 440 that corresponds to the facial region 430. Figure 4 In the example, the weighted map data 440 may include a corresponding region (1T region) 442 corresponding to the facial region 430, a reduced region (0.5T region) 441 of the corresponding region 442, extended regions (1.5T region and 2T region) 443 and 444 of the corresponding region 442, and an outermost region 445. Regions 441, 442, 443, 444 and 445 may each have a different aspect ratio and are overlapped and arranged based on the center of the corresponding region 442.

[0075] In the weighted map data 440, the weights assigned to each of regions 441, 442, 443, 444, and 445 can be different from each other. For example, a weight of 1 can be assigned to the decreasing region 441 corresponding to the inner center region of the face region 430, and a weight of 0.75 can be assigned to the region between the corresponding region 442 and the decreasing region 441. For example, a weight of 0.5 can be assigned to the region between the extended region 443 and the corresponding region 442, and a weight of 0.25 can be assigned to the region between the extended region 444 and the extended region 443. For example, a weight of 0 can be assigned to the region between the outermost region 445 and the extended region 444. In this way, weights can be assigned, and the weighted map data 440 can show weights that gradually decrease according to the distance from the center of the corresponding region 442 corresponding to the face region 430. However, this is just an example, and the weight distribution of the weighted map data 440 can vary. For example, the distribution of weights can change continuously in proportion to the distance from the corresponding region 442, rather than changing in a stepwise manner as in the weighted map data 440. When a region has a larger weight in the weighted map data 440, the region information of the input image 410 corresponding to that region can have a relatively higher influence on the activity test results compared to the region information of the input image 410 corresponding to another region.

[0076] Figure 5 Examples of generating weighted graph data according to one or more embodiments are shown.

[0077] Reference Figure 5 The weight map data 540 can be generated by a neural network-based weight generation model 530. An input image 510, in which object 520 is shown, can be input to the weight map generation model 530, and the weight map data 540 corresponding to the input image 510 can be output from the weight map generation model 530. The weight map generation model 530 can dynamically assign weights to the weight map data based on contextual information in the input image 510 (such as facial occlusion in object 520). Figure 5 As illustrated in the example, when the input image 510 is assumed to show an occluded region of a face created by an obstacle, the weighted map data 540 generated by the weighted map data generation model 530 may include a first region 542 corresponding to the face region of the object and a second region 544 corresponding to the occluded region in the face region, and the weights assigned to the first region 542 and the weights assigned to the second region 544 may be different from each other. The same weight or a weight that gradually decreases with increasing distance from the center of the first region 542 may be assigned to the first region 542. Simultaneously, a weight of 0 may be assigned to the second region 544 corresponding to the occluded region; therefore, the influence of the occluded region shown in the input image on the activity test results may be reduced.

[0078] Figure 6A An example is shown of inputting spliced ​​data into an activity test model according to one or more embodiments.

[0079] Reference Figure 6A To perform an activity test, the input image 610 can be fed into the activity test model 620. The activity test model 620 can be, for example, a CNN. When generating weight map data (e.g., Figure 4 Weighted graph data 440 or Figure 5 When the weight map data 540 is input, the activity testing device can input the weight map data into the activity testing model 620. The activity testing device can generate spliced ​​data by splicing feature data 630 (e.g., activation map or vector data) output from the first intermediate layer 622 (e.g., a convolutional layer) of the activity testing model 620, and input the spliced ​​data into the second intermediate layer 624 of the activity testing model 620. To generate the spliced ​​data, feature data from another intermediate layer besides the first intermediate layer 622 can be used. For example, the second intermediate layer 624 can be the upper layer directly above the first intermediate layer 622 (e.g., the layer immediately following the first intermediate layer 622).

[0080] When generating stitched data, if the feature data 630 and the weight map data 640 have different sizes (e.g., when the feature data 630 and the weight map data 640 have different horizontal and vertical lengths), the activity testing device can adjust the size of the weight map data 640 to correspond to the size of the feature data 630, so as to stitch the weight map data 640 with the feature data 630. The activity testing device can generate stitched data by stitching the feature 630 with the resized weight map data. In one embodiment, this can be achieved by dividing the weight map data into regions (e.g., such as...) Figure 4 One of the multiple regions shown in the image is concatenated with feature data to generate the concatenated data.

[0081] The activity testing model 620 can output an activity score based on the input image 610 and the stitched data input to the second intermediate layer, and the activity testing device can determine the activity test result based on the activity score. Using the weighted map data 640 makes the facial region of the input image 610 have a greater influence on the activity test result than another region, thus improving the accuracy of the activity test result.

[0082] Figure 6B An example is shown of inputting spliced ​​data into an activity test model according to one or more embodiments.

[0083] Reference Figure 6B In the weighted graph data 650 (e.g., Figure 4 Weighted graph data 440 or Figure 5After the weighted map data 640 is generated, the activity testing device can generate stitched data by stitching the weighted map data 650 with the image data of the input image 610, and input the stitched data into the input layer of the activity testing model 660. The activity testing model 660 can output an activity score based on the input stitched data, and the activity testing device can determine the activity test result based on the activity score.

[0084] Figure 7 This is a block diagram illustrating an example configuration of an activity testing device according to one or more embodiments.

[0085] Reference Figure 7 The activity testing device 700 can perform activity tests on objects shown in an input image. The activity testing device 700 may include a processor 710 and a memory 720.

[0086] Memory 720 may store various types of data used by components (e.g., processor 710). These various types of data may include, for example, instructions and input or output data. Memory 720 may include any one or both of volatile and non-volatile memory. Executing such instructions via processor 710 may configure the processor to implement any, any combination, or all of the operations and / or methods described herein.

[0087] The processor 710 can execute instructions to perform operations on the activity testing device 700. The processor 710 can execute, for example, instructions to configure the processor to control at least one other component (e.g., hardware or hardware-implemented software component) of the activity testing device 700 connected to the processor 710, and to perform various types of data processing or operations.

[0088] As at least part of data processing or operation, processor 710 may store instructions or data in memory 720, process instructions or data stored in memory 720, and store result data in memory 720 or storage device 230. Processor 710 may include a main processor (e.g., a central processing unit (CPU) or an application processor (AP)) or an auxiliary processor (e.g., a graphics processing unit (GPU) and a neural processing unit (NPU)) that may operate independently of or in cooperation with the main processor.

[0089] Processor 710 can perform one or more operations in conjunction with an activity test described or illustrated herein. For example, processor 710 can detect facial regions (i.e., activity test objects) in an input image and generate weight map data related to facial positions in the input image based on the detected facial regions. Processor 710 can generate stitched data by concatenating the weight map data with feature data output from a first intermediate layer of the activity test model or image data of the input image, and input the stitched data into the activity test model. When the stitched data is generated by concatenating the weight map data with feature data output from the first intermediate layer, processor 710 can input the stitched data into a second intermediate layer, which is an upper layer (e.g., a layer after the first intermediate layer) of the activity test model. When the stitched data is generated by concatenating the weight map data with image data of the input image, processor 710 can input the stitched data into the input layer of the activity test model. Processor 710 can then determine an activity test result based on an activity score determined by the activity test model.

[0090] Figure 8 This is a block diagram illustrating an example configuration of an electronic device according to one or more embodiments.

[0091] Reference Figure 8 Electronic device 800 (e.g., Figure 1 The electronic device 120 can be any electronic device in various forms. For example, the electronic device 800 can be a smartphone, tablet computer, personal digital assistant (PDA), netbook, laptop computer, product inspection device, personal computer, wearable device (e.g., augmented reality (AR) glasses, head-mounted display (HMD)), smart car, smart home appliance, security device, ATM, or server device, but is not limited thereto. The electronic device can perform activity testing on the object.

[0092] Electronic device 800 may include processor 810, memory 820, camera 830, sensor 840, input device 850, output device 860, and communication device 870. At least some components of electronic device 800 may be interconnected and transmit signals (e.g., instructions or data) between them via internal-peripheral communication interface 880 (e.g., bus, general purpose input and output (GPIO), serial peripheral interface (SPI), mobile industrial processor interface (MIPI)).

[0093] The processor 810 can control all operations of the electronic device 800 and execute functions and instructions to be performed within the electronic device 800. The processor 810 can execute the activity testing equipment described herein (e.g., Figure 7 The operation of the activity testing equipment (700).

[0094] Memory 820 may store instructions and input / output data that can be executed by processor 810. Memory 820 may include volatile memory (such as random access memory (RAM), dynamic random access memory (DRAM), and static random access memory (SRAM)) and / or non-volatile memory known in the art (such as read-only memory (ROM) and flash memory).

[0095] Camera 830 can capture images. Camera 830 can acquire, for example, color images, black and white images, grayscale images, infrared images, or depth images. Camera 830 can acquire an input image in which an object is shown, and processor 810 can perform an activity test based on the acquired input image.

[0096] Sensor 840 can detect the operating state of electronic device 800 (e.g., power or temperature) or the state of external environment (e.g., user state), and generate an electrical signal or data value corresponding to the detected state. Sensor 840 may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0097] Input device 850 can receive user input from the user via video, audio, or touch input. Input device 850 may include, for example, a keyboard, mouse, touch screen, microphone, or any other device that transmits user input to electronic device 800.

[0098] Output device 860 can provide the output of electronic device 800 to a user through visual, auditory, or tactile channels. Output device 860 may include, for example, a liquid crystal display or light-emitting diode (LED) / organic light-emitting diode (OLED) display, a micro LED, a touch screen, a speaker, a vibration generating device, or any other device capable of providing output to a user.

[0099] Communication device 870 can support establishing a direct (or wired) or wireless communication channel between electronic device 800 and external electronic device, and supports communication through the established communication channel. According to examples, the communication module may include a wireless communication module (e.g., a cellular communication module, a short-range wireless communication module, or a Global Navigation Satellite System (GNSS) communication module) or a wired communication module (e.g., a local area network (LAN) communication module or a power line communication module). The wireless communication module can communicate with external devices via short-range communication networks (e.g., Bluetooth, Wi-Fi Direct, or Infrared Data Association (IrDA)) or long-range communication networks (e.g., traditional cellular networks, 5G networks, next-generation communication networks, the Internet, or computer networks (e.g., LANs or wide area networks (WANs)).

[0100] Figure 9 An example of a training activity test model is shown according to one or more embodiments.

[0101] The activity testing model described herein may have parameters (e.g., connection weights) determined through training processes. (See reference...) Figure 9 In the training process, a large amount of training data 910 (e.g., training images) and the expected value corresponding to each piece of training data 910 are provided. During the training process, the liveness test model 930 can receive the training data 910 and output the result value corresponding to the training data 910 through parameter-based computation. Here, the weight graph generator 920 can generate weight graph data corresponding to the training data 910 based on the training data 910. The weight graph generator 920 can generate, for example... Figure 4 or Figure 5 The weighted graph data is described in the text, therefore, a detailed description of the weighted graph data is not provided here.

[0102] According to the example, the weight map data generated by the weight map generator 920 can be concatenated with the feature data output from the first intermediate layer of the liveness test model 930 or the current training data 910. When concatenated data is generated by concatenating the feature data of the first intermediate layer with the weight map data, the concatenated data can be input to a second intermediate layer directly above the first intermediate layer (e.g., the second intermediate layer can be a layer immediately following the first intermediate layer). When concatenated data is generated by concatenating the training data 910 with the weight map data, the concatenated data can be input to the input layer of the liveness test model 930.

[0103] Training device 940 can update the parameters of active test model 930 based on the output values ​​from active test model 930. Training device 940 can calculate the loss based on the difference between the output values ​​from active test model 930 and the expected values ​​corresponding to training data 910, and adjust the parameters of active test model 930 to reduce the loss. Various loss functions can be used to calculate the loss, and parameter adjustment can be performed, for example, using a backpropagation algorithm. Training device 940 can perform this process iteratively over a large amount of training data 910, thus the parameters of active test model 930 can be desiredly adjusted gradually. In addition to the training methods described herein, training device 940 can use various machine learning algorithms to train active test model 930.

[0104] When the weight graph generator 920 is a weight graph generation model based on a neural network, the training device 940 can also train the weight graph generator 920. Here, training data 910 and expected weight graph data corresponding to the training data 910 are provided, and the training device 940 can update the parameters of the weight graph generator 920 so that the weight graph generator 920 outputs weight graph data that is most similar to the expected weight graph data corresponding to the training data 910 input to the weight graph generator 920.

[0105] Figures 1 to 9The activity testing device, processor, sensor, electronic device, activity testing device 700, processors 710 and 810, electronic device 120 and 800, sensor 840, communication device 870, weight graph generator 920, and training device 940 that perform the operations described in this application are implemented by hardware components configured to perform the operations described in this application. Examples of hardware components that can be used to perform the operations described in this application include, where appropriate, controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components that perform the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer may be implemented by one or more processing elements, such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field-programmable gate arrays, programmable logic arrays, microprocessors, or any other means or combination of means configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, the processor or computer includes or is connected to one or more memories storing instructions or software executed by the processor or computer. Hardware components implemented by the processor or computer may execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) to perform the operations described in this application. The hardware components may also access, manipulate, process, create, and store data in response to the execution of instructions or software. For brevity, the singular terms “processor” or “computer” may be used in the description of the examples described in this application; however, in other examples, multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component, or two or more hardware components, may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or processors and controllers, and one or more other hardware components may be implemented by one or more other processors, or additional processors and additional controllers. One or more processors, or processors and controllers, may implement a single hardware component, or two or more hardware components. Hardware components may have any one or more different processing configurations, examples of which include: a single processor, a discrete processor, a parallel processor, Single Instruction Single Data (SISD) multiple processing, Single Instruction Multiple Data (SIMD) multiple processing, Multiple Instruction Single Data (MISD) multiple processing, and Multiple Instruction Multiple Data (MIMD) multiple processing.

[0106] Figures 1 to 9 The methods for performing the operations described in this application, as shown, are executed by computing hardware (e.g., one or more processors or a computer), which is implemented to execute instructions or software as described above to perform the operations performed by the methods described in this application. For example, a single operation, or two or more operations, may be executed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be executed by one or more processors, or a processor and a controller, and one or more other operations may be executed by one or more other processors, or additional processors and additional controllers. One or more processors, or a processor and a controller, may execute a single operation, or two or more operations.

[0107] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above can be written as computer programs, code segments, instructions, or any combination thereof to individually or collectively instruct or configure one or more processors or computers to operate as machines or special-purpose computers to perform operations performed by the hardware components and methods described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) that is directly executed by one or more processors or computers. In another example, the instructions or software include high-level code that is executed by one or more processors or computers using an interpreter. The instructions or software can be written using any programming language based on the block diagrams and flowcharts shown in the accompanying drawings and the corresponding descriptions in the specification, which disclose algorithms for performing operations performed by the hardware components and methods described above.

[0108] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above, along with any associated data, data files, and data structures, may be recorded, stored, or fixed in, or on, one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access memory (RAM), flash memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other means configured to store instructions or software and any associated data, data files, and data structures in a non-transitory manner and to provide said instructions or software and any associated data, data files, and data structures to one or more processors or computers such that one or more processors or computers can execute the instructions. In one example, instructions or software, along with any associated data, data files, and data structures, are distributed across a networked computer system, such that the instructions or software, along with any associated data, data files, and data structures, are stored, accessed, and executed in a distributed manner by one or more processors or computers.

[0109] While this disclosure includes specific examples, it will be clear upon understanding this disclosure that various changes in form and detail may be made to these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein should be considered descriptive only and not for limiting purposes. The description of features or aspects in each example will be considered applicable to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order, and / or if components in the described system, architecture, apparatus, or circuit are combined in a different manner, and / or replaced or supplemented by other components or their equivalents. Therefore, the scope of this disclosure is not limited by the specific embodiments but by the claims and their equivalents, and all variations within the scope of the claims and their equivalents should be construed as included in this disclosure.

Claims

1. A processor-implemented activity testing method, comprising: Detect facial regions in the input image; Based on the detected facial regions, weight map data related to the facial positions in the input image is generated; The stitched data is generated by stitching together the weighted graph data with the feature data generated from the intermediate layer of the liveness test model or the image data of the input image. as well as The activity test results are generated based on the activity score produced by the activity test model using the provided spliced ​​data. The weighted map data includes a first region corresponding to the facial regions in the input image and a second region corresponding to the non-facial regions in the input image. The weights of the first region and the weights of the second region are different from each other.

2. The activity testing method according to claim 1, wherein, The weights of the weighted map data vary based on the distance from the center of the corresponding region in the weighted map data that corresponds to the facial region.

3. The activity testing method according to claim 2, wherein, The weighted map data includes: the decreasing region of the corresponding region, the corresponding region, and the expanding region of the corresponding region, and The reduced area, corresponding area, and extended area overlap and are arranged based on the center of the corresponding area.

4. The activity testing method according to claim 3, wherein, The first weight of the reduced region is greater than the second weight of the region between the corresponding region and the reduced region, and The second weight is greater than the third weight of the region between the extended region and the corresponding region.

5. The activity testing method according to claim 1, wherein, The step of providing stitched data to the liveness testing model includes: in response to generating stitched data by stitching weight graph data with feature data generated from the intermediate layer, providing the stitched data to another intermediate layer of the liveness testing model, and The other intermediate layer is located after the intermediate layer.

6. The activity testing method according to claim 1, wherein, The steps of providing stitched data to the liveness test model include: in response to generating stitched data by stitching weight map data with image data of the input image, providing the stitched data to the input layer of the liveness test model.

7. The activity testing method according to claim 1, wherein, The steps to generate weighted graph data include: using a neural network-based weighted graph generation model to generate weighted graph data.

8. The activity testing method according to claim 7, wherein, The weight of the first region corresponding to the facial region in the weight map data is different from the weight of the third region corresponding to the occluded area in the facial region in the weight map data.

9. The activity testing method according to claim 1, wherein, The steps to generate the spliced ​​data include: Adjust the size of the weighted graph data to correspond to the size of the feature data; and The spliced ​​data is generated by concatenating feature data with weighted graph data whose size has been adjusted.

10. A non-transitory computer-readable storage medium storing instructions, which, when executed by a processor, cause the processor to perform the activity testing method according to any one of claims 1 to 9.

11. An activity testing device, comprising: The processor is configured as follows: Detect facial regions in the input image; Based on the detected facial regions, weight map data related to the facial positions in the input image is generated; The stitched data is generated by stitching together the weighted graph data with the feature data generated from the intermediate layer of the liveness test model or the image data of the input image. as well as The activity test results are determined based on the activity score determined by the activity test model using the provided spliced ​​data. The weighted map data includes a first region corresponding to the facial regions in the input image and a second region corresponding to the non-facial regions in the input image. The weights of the first region and the weights of the second region are different from each other.

12. The activity testing device according to claim 11, wherein, The weights of the weighted map data vary based on the distance from the center of the corresponding region in the weighted map data that corresponds to the facial region.

13. The activity testing device according to claim 11, wherein, The processor is also configured to: in response to generating concatenated data by concatenating weight map data with feature data generated from the intermediate layer, provide the concatenated data to another intermediate layer of the liveness test model, and The other intermediate layer is located after the intermediate layer.

14. The activity testing device according to claim 11, wherein, The processor is also configured to provide stitched data to the input layer of the liveness test model in response to generating stitched data by stitching weight map data with image data of the input image.

15. The activity testing device according to claim 11, wherein, The processor is also configured to generate weight map data using a neural network-based weight map generation model, and The weight of the first region corresponding to the facial region in the weight map data is different from the weight of the third region corresponding to the occluded region in the facial region in the weight map data.

16. An electronic device comprising: The camera is configured to: acquire the input image, and The processor is configured as follows: Detect facial and non-facial regions in the input image; Based on facial and non-facial regions, weighted map data is generated, which includes multiple regions, each with different weights. The spliced ​​data is generated by concatenating one of the multiple regions of the weighted graph data with feature data generated from the intermediate layer of the liveness test model. as well as The activity test results are generated based on the activity score produced by the activity test model using the provided spliced ​​data. Among them, the activity testing model is a machine learning model or a neural network model. The weighted map data includes a first region corresponding to the facial regions in the input image and a second region corresponding to the non-facial regions in the input image. The weights of the first region and the weights of the second region are different from each other.

17. The electronic device according to claim 16, wherein: The processor is also configured to: in response to generating concatenated data by concatenating weight map data with feature data generated from the intermediate layer, provide the concatenated data to another intermediate layer of the liveness test model, and The other intermediate layer is located after the intermediate layer.

Images