Face recognition method and device and in-memory computing system
By integrating lightweight face detection and recognition models on in-memory computing chips, the high energy consumption problem caused by traditional chip computing architecture is solved, real-time face detection and recognition on low-power devices is realized, and the device's Energy consumption.
Patent Information
- Application Number
- CN202310639397.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-31
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2043-05-31
AI Technical Summary
Traditional chip computing architecture leads to high cost and high power consumption in facial recognition technology, especially during the operation of the algorithm, the frequent transmission of computing core and memory leads to large energy consumption.
The low-power in-memory computing chip integrates a lightweight face detection model and face recognition model. Face detection and recognition are performed through in-memory computing, reducing the data copying process, and real-time face detection and recognition are achieved.
Real-time face detection and recognition is realized on low-power devices with lower energy consumption, and is suitable for drones, ink level boards, wearable devices, etc., reducing the energy consumption of device wake-up and unlocking functions.
Smart Images

Figure CN119068519A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, and in particular to a face recognition method and device, an in-memory computing system, an electronic device, and a computer-readable storage medium. Background Art
[0002] Face recognition is a biometric technology that identifies people based on their facial features. It uses a camera or a webcam to collect images or video streams containing faces, and automatically detects and tracks faces in the images, and then performs facial recognition on the detected faces. It is also commonly called portrait recognition or facial recognition. With the rise of target monitoring, target tracking, human-computer interaction, and crowd counting, face recognition technology has attracted more and more attention.
[0003] With the development of artificial intelligence (AI) applications, more and more cloud algorithms (such as face recognition algorithms) are gradually shifting to the edge. How to migrate high-computing power and high-energy consumption algorithms on the cloud to the edge with limited computing power is a technical challenge.
[0004] In the related technologies, the traditional chip computing architecture has the problem of memory wall. During the operation of the algorithm, a large amount of energy will be consumed in the frequent transmission between the computing core and the memory. Face recognition technology is usually implemented based on the traditional chip computing architecture. Therefore, face recognition technology in the related technologies usually has the problems of high cost and high power consumption. Summary of the invention
[0005] The present disclosure provides a face recognition method and device, an in-memory computing system, an electronic device, and a computer-readable storage medium.
[0006] According to a first aspect of the present disclosure, the present disclosure provides a face recognition method, the face recognition method comprising:
[0007] Get the image to be processed;
[0008] Inputting the image to be processed into an in-memory computing chip configured with a face detection model, performing face detection processing on the image to be processed using a preset face detection model, and obtaining an initial face detection frame in the image to be processed;
[0009] Acquire a target face image according to the initial face detection frame;
[0010] Performing face recognition processing on the target face image using the face recognition model pre-configured in the in-memory computing chip in the in-memory computing chip to obtain a target face feature vector corresponding to the target face image;
[0011] The target face feature vector is compared with the feature vector corresponding to the preset face image to obtain a face recognition result.
[0012] In some embodiments, acquiring a target face image according to the initial face detection frame includes:
[0013] Inputting the initial face detection frame into a programmable logic unit for non-maximum suppression processing to obtain a target face detection frame;
[0014] and inputting the target face detection frame into an in-memory computing chip;
[0015] And the target face detection frame is mapped to the image to be processed in an in-memory computing chip to obtain the target face image corresponding to the target face detection frame on the image to be processed.
[0016] In some embodiments, before performing non-maximum suppression processing, the face recognition method further includes:
[0017] In the programmable logic unit, according to the confidences of the multiple initial face detection frames output by the face detection model, a preset number of initial face detection frames are screened out from the multiple initial face detection frames in descending order of confidence;
[0018] The step of inputting the initial face detection frame into a programmable logic unit for non-maximum suppression processing to obtain a target face detection frame includes:
[0019] A preset number of initial face detection frames are input into a programmable logic unit for non-maximum suppression processing to obtain a target face detection frame.
[0020] In some embodiments, the acquiring of the image to be processed and the face recognition method further include:
[0021] Get the original image;
[0022] Performing image decoding and moving object detection on the original image;
[0023] In response to detecting the presence of a moving object in the original image, the original image is input into a programmable logic unit to perform image preprocessing on the original image to obtain the image to be processed.
[0024] In some embodiments, the image preprocessing includes at least one of the following: image scaling processing, image cropping processing, image format conversion processing, and image enhancement processing.
[0025] In some embodiments, the moving object detection includes performing moving object detection on the original image using a frame difference method.
[0026] In some embodiments, comparing the target face feature vector with a preset face feature vector to obtain a face recognition result includes:
[0027] The target face feature vector is input into a programmable logic unit so that the programmable logic unit performs the following processing:
[0028] Normalizing the target face feature vector to obtain a target face normalized feature vector;
[0029] Compare the normalized feature vector of the target face with the feature vector corresponding to the preset face image in the preset database to obtain the face feature matching degree;
[0030] When the facial feature matching degree is greater than a preset threshold, a face recognition result indicating successful face matching is obtained.
[0031] In some embodiments, the face detection model includes: a Backbone layer, a Neck layer, and a Head layer;
[0032] The Backbone layer is used to extract image features of the input image to be processed;
[0033] The Neck layer is used to reduce or adjust the image features from the Backbone layer;
[0034] The Head layer is used to generate a final network output by inputting the features processed by the Neck layer to obtain an initial face detection frame.
[0035] In some embodiments, the face recognition model is trained using knowledge distillation.
[0036] According to a second aspect of the present disclosure, the present disclosure provides a face recognition device, the face recognition device comprising:
[0037] An acquisition module, configured to acquire an image to be processed;
[0038] A detection module is configured to input the image to be processed into an in-memory computing chip configured with a face detection model, perform face detection processing on the image to be processed using the face detection model, and obtain an initial face detection frame in the image to be processed;
[0039] A processing module, configured to obtain a target face image according to the initial face detection frame;
[0040] A recognition module configured to perform face recognition processing on the target face image using a face recognition model pre-configured in the in-memory computing chip in the in-memory computing chip to obtain a target face feature vector corresponding to the target face image;
[0041] The comparison module is configured to compare the target face feature vector with the feature vector corresponding to the preset face image to obtain a face recognition result.
[0042] According to a third aspect of the present disclosure, the present disclosure provides an in-memory computing system, the in-memory computing system comprising:
[0043] An in-memory computing chip, on which a face detection model is integrated, is configured to perform face detection processing on the image to be processed using the face detection model, obtain an initial face detection frame in the image to be processed, and send the initial face detection frame to the PL end;
[0044] The PL end is configured to obtain a target face image according to the initial face detection frame, and send the target face image to the in-memory computing chip;
[0045] The in-memory computing chip also has a face recognition model integrated thereon, and is further configured to perform face recognition processing on the target face image using the face recognition model, obtain a target face feature vector corresponding to the target face image, and send the target face feature vector to the PL end;
[0046] The PL end is further configured to compare the target face feature vector with a feature vector corresponding to a preset face image to obtain a face recognition result.
[0047] In some embodiments, the PL terminal is configured as:
[0048] Performing non-maximum suppression processing on the initial face detection frame to obtain a target face detection frame;
[0049] The target face detection frame is mapped onto the image to be processed to obtain the target face image corresponding to the target face detection frame on the image to be processed.
[0050] In some embodiments, the in-memory computing system further includes: a PS end, the PS end being deployed on the FPGA device;
[0051] The PS end is configured to: obtain an original image from a camera device; perform image decoding and moving object detection on the original image; and send the original image to the PL end when a moving object is detected in the original image;
[0052] The PL end is also configured to perform image preprocessing on the original image in response to the PS end detecting the presence of a moving object in the original image to obtain the image to be processed, and send the image to be processed to the in-memory computing chip.
[0053] In some embodiments, the PL terminal is configured as:
[0054] Normalizing the target face feature vector to obtain a target face normalized feature vector;
[0055] Compare the normalized feature vector of the target face with the feature vector corresponding to the preset face image in the preset database to obtain the face feature matching degree;
[0056] When the facial feature matching degree is greater than a preset threshold, a facial recognition result of successful face matching is obtained.
[0057] According to a fourth aspect of the present disclosure, the present disclosure provides an electronic device, which includes the above-mentioned in-memory computing system.
[0058] According to a fifth aspect of the present disclosure, the present disclosure provides an electronic device, the electronic device comprising:
[0059] at least one processor; and
[0060] a memory communicatively connected to the at least one processor; wherein,
[0061] The memory stores one or more computer programs that can be executed by the at least one processor, and the one or more computer programs are executed by the at least one processor so that the at least one processor can perform the above-mentioned face recognition method.
[0062] According to a sixth aspect of the present disclosure, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above-mentioned face recognition method when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS
[0063] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. The above and other features and advantages will become more apparent to those skilled in the art by describing detailed example embodiments with reference to the accompanying drawings, in which:
[0064] Figure 1 A flowchart of a face recognition method provided by an embodiment of the present disclosure;
[0065] Figure 2 A schematic diagram of a model architecture of a face detection model in an embodiment of the present disclosure;
[0066] Figure 3 A schematic diagram of a process for obtaining a target face image in an embodiment of the present disclosure;
[0067] Figure 4 A schematic diagram of a flow chart of a method for obtaining an image to be processed in an embodiment of the present disclosure;
[0068] Figure 5 A schematic diagram of a model architecture of a face recognition model in an embodiment of the present disclosure;
[0069] Figure 6 A flowchart of a method for obtaining face recognition results in an embodiment of the present disclosure;
[0070] Figure 7 A schematic diagram of the structure of a face recognition device provided by an embodiment of the present disclosure;
[0071] Figure 8 A schematic diagram of the structure of an in-memory computing system provided in an embodiment of the present disclosure;
[0072] Fig. 9 A schematic diagram of an application scenario of an in-memory computing system provided by an embodiment of the present disclosure;
[0073] Fig.10 A block diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0074] In order to enable those skilled in the art to better understand the technical solutions of the present disclosure, the exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.
[0075] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.
[0076] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.
[0077] The terms used herein are only used to describe specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms "a" and "the" are also intended to include plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "including" and / or "made of" are used in this specification, the presence of the features, wholes, steps, operations, elements and / or components is specified, but the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof is not excluded. "Connected" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect.
[0078] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined as such herein.
[0079] Figure 1 A flow chart of a face recognition method provided by an embodiment of the present disclosure is shown in FIG. Figure 1 As shown, the face recognition method includes: steps S11 to S14.
[0080] Step S10: Obtain an image to be processed.
[0081] Step S11: input the image to be processed into an in-memory computing chip configured with a face detection model, perform face detection processing on the image to be processed using a preset face detection model, and obtain an initial face detection frame in the image to be processed.
[0082] Step S12: Acquire a target face image according to the initial face detection frame.
[0083] Step S13: Performing face recognition processing on the target face image in the in-memory computing chip using the face recognition model pre-configured in the in-memory computing chip to obtain a target face feature vector corresponding to the target face image.
[0084] Step S14: compare the target face feature vector with the feature vector corresponding to the preset face image to obtain a face recognition result.
[0085] According to the face recognition method provided by the embodiment of the present disclosure, a lightweight face detection model and a face recognition model are integrated into a low-power in-memory computing chip to perform face detection and recognition. Real-time face detection and recognition can be achieved with low energy consumption. The method can be applied to low-power devices such as drones, ink tablets, wearable devices, etc. to achieve functions such as device wake-up and unlocking.
[0086] In the embodiment of the present disclosure, in step S11, the image to be processed is input into a face detection model pre-configured in an in-memory computing chip for face detection processing to obtain an initial face detection frame in the image to be processed. The initial face detection frame can be used to represent the position coordinate information of the face in the image predicted by the model.
[0087] In the disclosed embodiment, the input of the face detection model is an image to be processed, the pixel size of the image to be processed is a preset pixel size, and the image format is a preset image format. Exemplarily, the pixel size of the image to be processed is 320*240, and the image format is RGB format. The image to be processed is input into the face detection model, and after the neural network inference calculation of the face detection model, the original result of the face detection is output, and the original result of the face detection includes an initial face detection frame.
[0088] Figure 2 FIG. 1 is a schematic diagram of a model architecture of a face detection model in an embodiment of the present disclosure. To achieve lightweight model design, the number of network channels and the number of network layers of the face detection model are set to smaller values. In some embodiments, for example, Figure 2 As shown in the figure, the face detection model consists of three parts, namely, feature extraction layer, feature processing layer and prediction output layer. The feature extraction layer is the Backbone layer, the feature processing layer is the Neck layer, and the prediction output layer is the Head layer.
[0089] The feature extraction layer, namely the Backbone layer, is the backbone network of the model, which is used to extract the image features of the input image to be processed for subsequent processing and analysis. In the Backbone layer, it is composed of the input layer, the convolution layer (Conv2d), the batch normalization layer (BatchNorm), the nonlinear activation function layer (ReLU6), 4 network modules (Block) and 3 attention mechanism network modules (Block_SE). Among them, the number of channels of the first convolution layer (Conv2d) of the model is set to 16, N = 1, and the N value is the number of network module blocks of the model. The N value can be adjusted according to the computing power of the in-memory computing chip.
[0090] Among them, Figure 2On the right side, the network module (Block) includes three units consisting of a convolutional layer (Conv2d), a batch normalization layer (BatchNorm) and a non-linear activation function layer (ReLU6); the attention mechanism network module (Block_SE) includes a network module (Block) and an attention mechanism module (Squeeze-and-Excitation, SE). The feature map passes through the network module (Block) and is short-circuited with the attention mechanism module. The attention mechanism module (SE) includes a pooling layer (AveragePooling) and two units consisting of a convolutional layer (Conv2d), a batch normalization layer (BatchNorm) and a non-linear activation function layer (ReLU6).
[0091] The feature processing layer, namely the Neck layer, is the middle layer connecting the Backbone layer and the Head. The main function of the Neck layer is to reduce or adjust the image features from the Backbone layer, improve the diversity and robustness of the features, so as to better meet the task requirements. In the Neck layer, the feature map processed by the seventh module of the seven modules of the Backbone layer is processed by convolution (Conv2d) to obtain the first feature map Feature1; the first feature map Featuer1 is processed by upsampling (Upsample) and added (Add) to the feature map processed by the fifth module of the seven modules of the Backbone layer, and then processed by convolution (Conv2d) to obtain the second feature map Feature2; the second feature map Featuer2 is processed by upsampling (Upsample) and added (Add) to the feature map processed by the third module of the seven modules of the Backbone layer, and then processed by convolution (Conv2d) to obtain Feature3; the first feature map Feature1, the second feature map Feature2, and the third feature map Feature3 are processed by corresponding convolution (Conv2d) and input into the Head layer.
[0092] The prediction output layer, namely the Head layer, is the last layer of the model and is the detection head for face detection. It is usually a classifier or regressor, which is used to generate the final network output by inputting the features processed by the Neck layer to obtain the initial face detection frame. In the Head layer, the first feature map Feature1, the second feature map Feature2, and the third feature map Feature3 are respectively processed by convolution (Conv2d) in the Neck layer, and then superimposed and fused (Concat) in the Head layer. After convolution (Conv2d) processing and dimension conversion (permute, view) processing, the three-part feature map vector is output through the softmax layer, which are the first feature map vector, the second feature map vector, and the third feature map vector.
[0093] Among them, the first feature map vector represents the initial face detection frame and its corresponding position coordinates, the second feature map vector represents the initial face detection frame and its corresponding facial key point coordinates, and the third feature map vector represents the initial face detection frame and its corresponding confidence. Exemplarily, the first feature map vector, the second feature map vector and the third feature map vector are data of length 3160*4, 3160*10, and 3160*2, respectively. 3160 represents 3160 initial face detection frames, 4 represents the coordinate dimension (x, y, w, h) of the initial face detection frame, 10 represents the position coordinates (5*(x, y)) of the 5 facial key points corresponding to the initial face detection frame, the facial key points include the two eyes, nose, and two corners of the mouth, 2 represents the confidence, and a total of 4+10+2=16 dimensions of information.
[0094] In the disclosed embodiment, the main computing resources of the neural network calculation of the model are consumed in multiplication and addition operations. The Processing-In-Memory (PIM) chip is suitable for neural network calculation, which can greatly reduce the process of repeated copying of data during the calculation process.
[0095] Figure 3 FIG. 1 is a flow chart of a method for obtaining a target face image in an embodiment of the present disclosure. In some embodiments, for example, Figure 3 As shown, in step S12, obtaining a target face image according to the initial face detection frame may further include:
[0096] Step S31: input the initial face detection frame into the programmable logic unit for non-maximum suppression processing to obtain the target face detection frame.
[0097] The initial face detection frame obtained and selected by the above face detection is subjected to non-maximum suppression processing using a non-maximum suppression algorithm (NMS) to obtain a target face detection frame.
[0098] Among them, the non-maximum suppression algorithm NMS is an algorithm that searches for local maxima and suppresses non-maximum elements.
[0099] In the face detection reasoning process, many initial face detection frames (such as A, B, C, D, E, F, etc.) will be generated, many of which are detecting the same target face, but ultimately each target face only needs one target face detection frame. NMS can be used to select the initial face detection frame with the highest score, assuming it is the initial face detection frame C, and then calculate the corresponding IOU (Intersection over Union) value of the initial face detection frame C and other initial face detection frames. The IOU value is an evaluation indicator for target detection. When the IOU value exceeds the set threshold, the initial face detection frame that exceeds the threshold is suppressed. The suppression method is to set the score of the initial face detection frame to 0. After one round, continue to search for the initial face detection frame with the highest score in the remaining initial face detection frames, and then suppress the initial face detection frame whose IOU value exceeds the threshold with the initial face detection frame, until the initial face detection frame with almost no overlap is retained, so as to obtain the target face detection frame corresponding to each target face.
[0100] In some embodiments, after the initial face detection frame is input into the programmable logic unit and before the non-maximum suppression process is performed, the face recognition method further includes:
[0101] In the programmable logic unit, according to the confidences of the multiple initial face detection frames output by the face detection model, a preset number of initial face detection frames are screened out from the multiple initial face detection frames in descending order of confidence.
[0102] For example, the number of initial face detection frames output by the face detection model is 3160, and the preset number is 100. After sorting the initial face detection frames according to their confidence levels, the initial face detection frames corresponding to the top 100 confidence levels in the confidence level sorting are selected for the next step of processing.
[0103] Furthermore, the initial face detection frame is input into a programmable logic unit for non-maximum suppression processing to obtain a target face detection frame, including: inputting a preset number of initial face detection frames into a programmable logic unit for non-maximum suppression processing to obtain a target face detection frame.
[0104] Step S32: input the target face detection frame into the in-memory computing chip.
[0105] Step S33: Mapping the target face detection frame to the image to be processed in the in-memory computing chip to obtain a target face image corresponding to the target face detection frame on the image to be processed.
[0106] The detected target face detection frame is mapped to the corresponding position coordinates on the image to be processed according to the position coordinates, and the target face image corresponding to the target face detection frame is obtained by cropping. The cropped image is scaled to a predetermined pixel size, such as a pixel size of 112×112, and an affine transformation is performed according to the coordinates of the facial key points (such as eyes, nose, and mouth corners) obtained by face detection to correct the target face image.
[0107] Figure 4 FIG. 1 is a flow chart of a method for obtaining an image to be processed in an embodiment of the present disclosure. In some embodiments, for example, Figure 4 As shown, in step S10, obtaining an image to be processed includes:
[0108] Step S41, obtaining the original image.
[0109] In some embodiments, the main control system controls the camera device to shoot, and receives the original images shot by the camera device. In some embodiments, the main control system receives the original images shot and transmitted by the camera device at regular intervals.
[0110] Step S42: perform image decoding and moving object detection on the original image.
[0111] After the original image is acquired, the main control system decodes the original image and performs moving object detection on the decoded original image.
[0112] In some embodiments, a frame difference method is used to detect moving objects in the original image to determine whether the pixel values of any two corresponding pixel points in the previous and next frame images are consistent. If the number of inconsistent pixel points is greater than a predetermined threshold, it is considered that a moving object appears in the original image.
[0113] Step S43: In response to detecting that a moving object appears in the original image, the original image is input into a programmable logic unit to perform image preprocessing on the original image to obtain an image to be processed.
[0114] The face detection model has certain input requirements. Therefore, after obtaining the original image, before inputting the image into the face detection model for face detection, the image needs to be preprocessed to obtain a processed image that meets the model input requirements.
[0115] In some embodiments, image preprocessing includes at least one of the following: image scaling processing, image format conversion processing, and image enhancement processing.
[0116] Among them, image scaling processing refers to downsampling the image to a preset pixel size, such as downsampling to a 320*240 pixel size. For example, directly taking the values of some pixels in the original image as the pixel values of the scaled image, thereby improving processing efficiency; image format conversion processing refers to converting the image format of the image to a preset image format. For example, the model requires that the input image format is RGB format. If the original image is in RGB format, no format conversion is required. If the original image is in YUV format, format conversion is required; image enhancement processing can be to perform histogram equalization on the image to prevent the image from being too bright or too dark.
[0117] In the disclosed embodiment, in step S13, the acquired target face image is input into the face recognition model pre-configured in the in-memory computing chip, so that the face recognition model is used in the in-memory computing chip to perform face recognition processing on the target face image, thereby obtaining a target face feature vector corresponding to the target face image. The dimension of the target face feature vector can be 256 dimensions.
[0118] Figure 5 FIG. 1 is a schematic diagram of a model architecture of a face recognition model in an embodiment of the present disclosure. In some embodiments, for example, Figure 5 As shown, because the computing power of the in-memory computing PIM chip is limited, the model is designed according to the small model structure, where the network input can be a 64*64 3-channel color image. To achieve lightweight model design, the number of network channels and the number of network layers of the face recognition model are set to smaller values. In some embodiments, such as Figure 5 As shown in the figure, the face recognition model includes a first network unit consisting of a convolutional layer (Conv2d), a batch normalization layer (BatchNorm), and a nonlinear activation function layer (ReLU6), 4 network modules (Block), 3 attention mechanism network modules (Block_SE), a second network unit consisting of a convolutional layer (Conv2d), a batch normalization layer (BatchNorm), and a nonlinear activation function layer (ReLU6), a pooling layer (AveragePooling), and a fully connected layer consisting of a first linear layer (Linear), a batch normalization layer (BatchNorm), a nonlinear activation function layer (ReLU6), and a second linear layer (Linear). Among them, the number of channels of the first convolutional layer (Conv2d) of the model is set to 16, N = 1, and the N value is the number of network module blocks of the model, and the N value can be adjusted according to the computing power of the in-memory computing chip.
[0119] The network module (Block) includes three units consisting of a convolutional layer (Conv2d), a batch normalization layer (BatchNorm) and a non-linear activation function layer (ReLU6); the attention mechanism network module (Block_SE) includes a network module (Block) and an attention mechanism module (Squeeze-and-Excitation, SE). The feature map passes through the network module (Block) and is short-circuited with the attention mechanism module. The attention mechanism module (SE) includes a pooling layer (AveragePooling) and two units consisting of a convolutional layer (Conv2d), a batch normalization layer (BatchNorm) and a non-linear activation function layer (ReLU6).
[0120] In order to improve the model's ability to express facial features, knowledge distillation is used to improve its model reasoning and computing capabilities during the training process. The face recognition model to be learned and trained is used as the student model (Student), and the number of network channels and network layers of the face recognition model are increased to obtain a model with a larger number of network channels and network layers than the face recognition model, which is used as the teacher model (Teacher). For example, the number of network channels of the first convolutional layer in the student model is set to 16, N=1, and the number of network channels of the first convolutional layer in the teacher model is set to 64, N=4.
[0121] The teacher model is a face recognition model that has been pre-trained using a training data set, and the student model is a face recognition model to be trained. Knowledge distillation refers to the transfer of dark knowledge in the teacher model to the student model. Through knowledge distillation, the expression ability of the student model can be as close to or exceed that of the teacher model as possible, thereby achieving the effect of using a low-complexity student model to obtain the network prediction (output) of a high-complexity teacher model, streamlining network parameters, and ensuring network accuracy.
[0122] Figure 6 FIG. 1 is a flow chart of a method for obtaining face recognition results in an embodiment of the present disclosure. In some embodiments, for example, Figure 6 As shown, in step S14, the target face feature vector is compared with the feature vector corresponding to the preset face image to obtain the face recognition result, including: inputting the target face feature vector into the programmable logic unit so that the programmable logic unit performs the following processing:
[0123] Step S61, normalize the target face feature vector to obtain a normalized target face feature vector.
[0124] Among them, the normalization process refers to subtracting the mean of the target face feature vector and then dividing it by its variance.
[0125] Step S62: compare the normalized feature vector of the target face with the feature vector corresponding to the preset face image in the preset database to obtain the face feature matching degree.
[0126] The feature vector corresponding to the preset face image is a feature vector obtained by performing face recognition on the preset face image using a face recognition model and then performing normalization processing on the face feature vector.
[0127] In some embodiments, the facial feature matching degree can be obtained by calculating the similarity between two feature vectors, wherein the similarity between the two feature vectors can be calculated using a preset vector similarity algorithm, for example, the vector similarity algorithm can use a Euclidean distance similarity algorithm, a cosine similarity algorithm, etc.
[0128] Step S63: When the facial feature matching degree is greater than a preset threshold, a face recognition result indicating successful face matching is obtained.
[0129] When the facial feature matching degree is greater than a preset threshold, it means that there is a facial image that is relatively similar to the target facial image in the facial image represented by the preset facial feature vector of the preset database, that is, the target facial image is matched in the preset database, and thus a facial recognition result indicating a successful face match is obtained.
[0130] When the facial feature matching degree is less than or equal to the preset threshold, it means that the facial images represented by the preset facial feature vectors of the preset database are not similar to the target facial image, that is, the target facial image cannot be matched in the preset database, and thus a facial recognition result indicating an unsuccessful face match is obtained.
[0131] In some embodiments, the normalized feature vector of the target face can be compared one by one with multiple preset face feature vectors in a preset database to obtain multiple face feature matching degrees, and the maximum face feature matching degree is obtained from the multiple face feature matching degrees. The maximum face feature matching degree is compared with a preset threshold, thereby determining whether the target face image matches the face image in the preset database.
[0132] In some embodiments, the establishment of a preset database is to first obtain multiple preset facial images and guide the annotation of their identity information, use a facial recognition model to extract their facial feature vectors, and perform normalization processing to obtain the normalized facial feature vector corresponding to each preset facial image, and save it in the preset database.
[0133] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not repeat them. It can be understood by those skilled in the art that in the above-mentioned method of the specific implementation method, the specific execution order of each step should be determined according to its function and possible internal logic.
[0134] Figure 7 A structural diagram of a face recognition device provided by an embodiment of the present disclosure is shown in FIG. Figure 7 As shown, the face recognition device 700 includes: an acquisition module 701, a detection module 702, a processing module 703, a recognition module 704 and a comparison module 705.
[0135] Among them, the acquisition module 701 is configured to acquire the image to be processed; the detection module 702 is configured to input the image to be processed into an in-memory computing chip configured with a face detection model, and perform face detection processing on the image to be processed using the face detection model to obtain an initial face detection frame in the image to be processed; the processing module 703 is configured to acquire the target face image according to the initial face detection frame; the recognition module 704 is configured to perform face recognition processing on the target face image in the in-memory computing chip using the face recognition model pre-configured in the in-memory computing chip, and obtain a target face feature vector corresponding to the target face image; the comparison module 705 is configured to compare the target face feature vector with the feature vector corresponding to the preset face image to obtain a face recognition result.
[0136] The face recognition device 700 provided in the embodiment of the present disclosure is used to implement the face recognition method provided in any of the above embodiments. For specific related descriptions, please refer to the description of the face recognition method in any of the above embodiments, which will not be repeated here.
[0137] Figure 8 A schematic diagram of the structure of an in-memory computing system provided by an embodiment of the present disclosure is shown in FIG. Figure 8 As shown, the in-memory computing system 800 includes: an in-memory computing chip 801 and a PL (Progarmmable Logic) terminal 802.
[0138] Among them, the in-memory computing chip 801 has a face detection model integrated thereon, which is configured to use the face detection model to perform face detection processing on the image to be processed, obtain an initial face detection frame in the image to be processed, and send the initial face detection frame to the PL end 802; the PL end 802 is deployed on an FPGA (Field Programmable Gate Array) device, which is configured to obtain a target face image according to the initial face detection frame, and send the target face image to the in-memory computing chip 801; the in-memory computing chip 801 also has a face recognition model integrated thereon, which is also configured to use the face recognition model to perform face recognition processing on the target face image, obtain a target face feature vector corresponding to the target face image, and send the target face feature vector to the PL end 802; the PL end 802 is also configured to compare the target face feature vector with the feature vector corresponding to the preset face image to obtain a face recognition result.
[0139] In some embodiments, the PL terminal 802 is configured to: perform non-maximum suppression processing on the initial face detection frame to obtain a target face detection frame; map the target face detection frame to the image to be processed to obtain a target face image corresponding to the target face detection frame on the image to be processed.
[0140] In some embodiments, Figure 8 As shown, the in-memory computing system 800 also includes: a PS (Processing System, main control system) terminal 803, and the PS terminal 803 is deployed in the FPGA device. Among them, the PS terminal 801 is configured to obtain the image to be processed, and obtaining the image to be processed includes: obtaining the original image from the camera device; decoding the original image and detecting the moving object; when the moving object is detected in the original image, sending the original image to the PL terminal 802; the PL terminal 802 is also configured to perform image preprocessing on the original image in response to the PS terminal 803 detecting the moving object in the original image, obtain the image to be processed, and send the image to be processed to the in-memory computing chip 801.
[0141] In some embodiments, a Linux operating system can be deployed on the FPGA device as the main control system (PS side) to realize the control and scheduling of hardware resources. Based on the operating system, some algorithms with strong versatility and low performance requirements (such as motion detection algorithms, etc.) can be implemented.
[0142] In some embodiments, the PL end 802 is configured to: normalize the target face feature vector to obtain a normalized target face feature vector; compare the normalized target face feature vector with the feature vector corresponding to a preset face image in a preset database to obtain a face feature matching degree; when the face feature matching degree is greater than a preset threshold, obtain a face recognition result of a successful face match.
[0143] Fig. 9 A schematic diagram of an application scenario of an in-memory computing system provided by an embodiment of the present disclosure, such as Fig. 9 As shown, in some application scenarios, the in-memory computing system 900 can be applied to a terminal device 905, and the main control system (PS end) 903 in the in-memory computing system 900 can be used as a device control end to interact with the camera device 904 and the terminal device 905, such as controlling the camera device 904 to capture images and transmitting face detection results and face recognition results to the terminal device 905.
[0144] In some application scenarios, such as Fig. 9 As shown, the camera device 904 transmits the captured original image to the main control system 903, and the main control system 903 performs image signal decoding and moving object detection to activate the programmable logic unit (PL end) 902; when the main control system 903 detects a moving object, the image is transmitted to the programmable logic unit (PL end) 902 for image preprocessing; the programmable logic unit (PL end) 902 can maintain a normally open, low-computing operation mode in hardware. After activation, the programmable logic unit (PL end) 902 transmits the processed image to be processed to the in-memory computing chip 901 that integrates the face detection model and the face recognition model. After the in-memory computing chip 901 performs face detection processing using the face detection model, The feature map is transmitted back to the programmable logic unit (PL end) 902 for feature map post-processing to obtain a target face detection frame, and the corresponding target face image is cut out from the image to be processed and sent to the in-memory computing chip 901; the in-memory computing chip 901 uses the face recognition model to calculate the face feature vector of the target face image, and then transmits it back to the programmable logic unit (PL end) 902 for face feature comparison to obtain the face recognition result; the programmable logic unit (PL end) 902 feeds back the face detection result and face recognition result to the terminal device 905 through the main control system 903, so that the terminal device 905 can display the face detection result and face recognition result in real time, or use them to implement other functions, such as device wake-up, unlocking, etc.
[0145] In some embodiments, the in-memory computing system 900 may be integrated into the terminal device 905 . Furthermore, the above-mentioned camera device 904 may also be integrated into the terminal device 905 .
[0146] In some application scenarios, the terminal device 905 can be a low-power electronic device such as an ink tablet, a drone, a wearable device, etc., which uses the in-memory computing system 900 to perform face detection and face recognition, thereby realizing functions such as device wake-up display and unlocking.
[0147] In the drone application scenario, the terminal device 905 can also use the in-memory computing system 900 to perform face detection and face recognition to achieve functions such as assisting obstacle avoidance, avoiding pedestrians, and pursuing suspects.
[0148] In the application scenario of wearable devices, the wearable devices are, for example, smart watches, smart glasses, etc. In the application scenario of smart watches, the above-mentioned face recognition method provided in the embodiment of the present disclosure can be used to realize functions such as watch unlocking and waking up the screen. When a single face is recognized and it is the face of the owner of the device, the screen will be lit to display the notification message. When multiple faces are recognized, even if the face of the owner of the device is recognized, the screen can be kept dark and the notification message can be hidden.
[0149] In the application scenario of smart glasses, facial feature matching is achieved locally, but the storage of glasses is limited and the amount of data in the facial feature database is also small. Therefore, the device can upload the facial features to the cloud through the Internet and search and compare the facial features in the large database in the cloud.
[0150] In some embodiments, the terminal device 905 may also be an electronic device such as a smart phone, a computer, a tablet, a television, etc.
[0151] An embodiment of the present disclosure also provides an electronic device, which includes the in-memory computing system provided in the above embodiment. The in-memory computing system can be integrated into the electronic device to realize face detection and face recognition functions.
[0152] Fig.10 A block diagram of another electronic device provided in accordance with an embodiment of the present disclosure.
[0153] Reference Fig.10 An embodiment of the present disclosure provides an electronic device 1000, which includes: at least one processor 1001; at least one memory 1002, and one or more I / O interfaces 1003, connected between the processor 1001 and the memory 1002; wherein the memory 1002 stores one or more computer programs that can be executed by the at least one processor 1001, and the one or more computer programs are executed by the at least one processor 1001, so that the at least one processor 1001 can perform the above-mentioned face recognition method.
[0154] The present disclosure also provides a computer-readable storage medium on which a computer program is stored, wherein the computer program implements the above-mentioned face recognition method when executed by a processor. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.
[0155] The embodiment of the present disclosure also provides a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above-mentioned face recognition method.
[0156] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable storage medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium).
[0157] As is known to those of ordinary skill in the art, the term computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable program instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technology, portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. In addition, it is known to those of ordinary skill in the art that communication media typically contain computer-readable program instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and may include any information delivery medium.
[0158] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.
[0159] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.
[0160] The computer program product described herein may be implemented in hardware, software, or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK), etc.
[0161] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.
[0162] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0163] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0164] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.
[0165] Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for limiting purposes. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly noted, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, those skilled in the art will appreciate that various changes in form and detail may be made without departing from the scope of the present disclosure as set forth in the appended claims.
Claims
1. A face recognition method, characterized in that: The face recognition method comprises: Get the image to be processed; Inputting the image to be processed into an in-memory computing chip configured with a face detection model, performing face detection processing on the image to be processed using a preset face detection model, and obtaining an initial face detection frame in the image to be processed; Acquire a target face image according to the initial face detection frame; Performing face recognition processing on the target face image using the face recognition model pre-configured in the in-memory computing chip in the in-memory computing chip to obtain a target face feature vector corresponding to the target face image; The target face feature vector is compared with the feature vector corresponding to the preset face image to obtain a face recognition result.
2. The face recognition method according to claim 1, characterized in that: The step of acquiring a target face image according to the initial face detection frame includes: Inputting the initial face detection frame into a programmable logic unit for non-maximum suppression processing to obtain a target face detection frame; and inputting the target face detection frame into an in-memory computing chip; And the target face detection frame is mapped to the image to be processed in an in-memory computing chip to obtain the target face image corresponding to the target face detection frame on the image to be processed.
3. The face recognition method according to claim 2, characterized in that: Before performing the non-maximum suppression process, the face recognition method further includes: In the programmable logic unit, according to the confidences of the multiple initial face detection frames output by the face detection model, a preset number of initial face detection frames are screened out from the multiple initial face detection frames in descending order of confidence; The step of inputting the initial face detection frame into a programmable logic unit for non-maximum suppression processing to obtain a target face detection frame includes: A preset number of initial face detection frames are input into a programmable logic unit for non-maximum suppression processing to obtain a target face detection frame.
4. The face recognition method according to claim 1, characterized in that: The step of obtaining the image to be processed comprises: Get the original image; Performing image decoding and moving object detection on the original image; In response to detecting the presence of a moving object in the original image, the original image is input into a programmable logic unit to perform image preprocessing on the original image to obtain the image to be processed.
5. The face recognition method according to claim 4, characterized in that: The image preprocessing includes at least one of the following: image scaling processing, image cropping processing, image format conversion processing, and image enhancement processing.
6. The face recognition method according to claim 4, characterized in that: The moving object detection includes performing moving object detection on the original image using a frame difference method.
7. The face recognition method according to claim 1, characterized in that: The step of comparing the target face feature vector with a preset face feature vector to obtain a face recognition result includes: The target face feature vector is input into a programmable logic unit so that the programmable logic unit performs the following processing: Normalizing the target face feature vector to obtain a target face normalized feature vector; Compare the normalized feature vector of the target face with the feature vector corresponding to the preset face image in the preset database to obtain the face feature matching degree; When the facial feature matching degree is greater than a preset threshold, a face recognition result indicating successful face matching is obtained.
8. The face recognition method according to any one of claims 1 to 7, characterized in that: The face detection model includes: Backbone layer, Neck layer and Head layer; The Backbone layer is used to extract image features of the input image to be processed; The Neck layer is used to reduce the dimension or adjust the image features from the Backbone layer; The Head layer is used to generate a final network output by inputting the features processed by the Neck layer to obtain an initial face detection frame.
9. The face recognition method according to any one of claims 1 to 7, characterized in that: The face recognition model is trained by using knowledge distillation.
10. A face recognition device, characterized in that: include: An acquisition module, configured to acquire an image to be processed; A detection module is configured to input the image to be processed into an in-memory computing chip configured with a face detection model, perform face detection processing on the image to be processed using the face detection model, and obtain an initial face detection frame in the image to be processed; A processing module, configured to obtain a target face image according to the initial face detection frame; A recognition module configured to perform face recognition processing on the target face image using a face recognition model pre-configured in the in-memory computing chip in the in-memory computing chip to obtain a target face feature vector corresponding to the target face image; The comparison module is configured to compare the target face feature vector with the feature vector corresponding to the preset face image to obtain a face recognition result.
11. An in-memory computing system, characterized in that: include: An in-memory computing chip, on which a face detection model is integrated, is configured to perform face detection processing on the image to be processed using the face detection model, obtain an initial face detection frame in the image to be processed, and send the initial face detection frame to the PL end; The PL end is configured to obtain a target face image according to the initial face detection frame, and send the target face image to the in-memory computing chip; The in-memory computing chip also has a face recognition model integrated thereon, and is further configured to perform face recognition processing on the target face image using the face recognition model, obtain a target face feature vector corresponding to the target face image, and send the target face feature vector to the PL end; The PL end is further configured to compare the target face feature vector with a feature vector corresponding to a preset face image to obtain a face recognition result.
12. The in-memory computing system according to claim 11, characterized in that: The PL terminal is configured as: Performing non-maximum suppression processing on the initial face detection frame to obtain a target face detection frame; The target face detection frame is mapped onto the image to be processed to obtain the target face image corresponding to the target face detection frame on the image to be processed.
13. The in-memory computing system according to claim 11, characterized in that: The in-memory computing system further includes: a PS end, wherein the PS end is deployed in an FPGA device; The PS end is configured to: obtain an original image from a camera device; perform image decoding and moving object detection on the original image; and send the original image to the PL end when a moving object is detected in the original image; The PL end is also configured to perform image preprocessing on the original image in response to the PS end detecting the presence of a moving object in the original image to obtain the image to be processed, and send the image to be processed to the in-memory computing chip.
14. The in-memory computing system according to claim 11, characterized in that: The PL terminal is configured as: Normalizing the target face feature vector to obtain a target face normalized feature vector; Compare the normalized feature vector of the target face with the feature vector corresponding to the preset face image in the preset database to obtain the face feature matching degree; When the facial feature matching degree is greater than a preset threshold, a facial recognition result of successful face matching is obtained.
15. An electronic device, characterized in that: Comprising an in-memory computing system as described in any one of claims 11-14.
16. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, and the one or more computer programs are executed by the at least one processor so that the at least one processor can execute the face recognition method as described in any one of claims 1-9.
17. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements the face recognition method according to any one of claims 1 to 9.
Citation Information
Patent Citations
Facial modelling and matching systems and methods
CA3050456A1
In-vehicle face recognition method, equipment and device and storage medium
CN112215179A
End-to-end face detection and recognition method
CN112966660A
Target detection method, device and equipment
CN113159277A
In-memory calculation method and device for graph small sample learning and electronic equipment
CN115034374A
