Face recognition method and apparatus, and in-memory computing system
By integrating lightweight face detection and recognition models on in-memory computing chips, the high energy consumption problem caused by traditional chip computing architecture is solved, and real-time face recognition with low power consumption is achieved, suitable for low-power devices.
Patent Information
- Application Number
- PCT/CN2024/088304
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-31
- Filing Date
- 2024-04-17
- Publication Date
- 2024-12-05
AI Technical Summary
Traditional chip computing architecture leads to high cost and high power consumption in facial recognition technology, especially during the operation of the algorithm, the frequent transmission of computing core and memory leads to large energy consumption.
Using in-memory computing chips to integrate lightweight face detection model and face recognition model, the images are processed through preset face detection model and recognition model, reducing data copying, and real-time face detection and recognition with low power consumption .
Real-time face detection and recognition with low power consumption is realized, suitable for low-power devices, such as drones and wearable devices, reducing the energy consumption of device wake-up and unlocking functions.
Smart Images

Figure CN2024088304_05122024_PF_FP_ABST
Abstract
Description
Face recognition method and device, and in-memory computing system Technical Field
[0001] The present disclosure relates to the field of computer technology, and in particular to a face recognition method and device, an in-memory computing system, an electronic device, and a computer-readable storage medium. Background Art
[0002] Facial recognition is a biometric technology that identifies people based on their facial features. It involves using a camera or video camera to capture images or video streams containing faces, automatically detecting and tracking faces within the images, and then performing facial recognition on the detected faces. This technology is also commonly referred to as portrait recognition or facial recognition. With the rise of applications in target monitoring, target tracking, human-computer interaction, and crowd counting, facial recognition technology is gaining increasing attention.
[0003] With the development of artificial intelligence (AI) applications, more and more cloud algorithms (such as face recognition algorithms) are gradually shifting to the edge side. How to migrate high-computing power and high-energy consumption algorithms on the cloud to the edge with limited computing power is a technical challenge.
[0004] In related technologies, traditional chip computing architecture has the problem of memory wall. During the operation of the algorithm, a large amount of energy will be consumed in the frequent transmission between the computing core and the memory. Face recognition technology is usually implemented based on the traditional chip computing architecture. Therefore, face recognition technology in related technologies usually has the problems of high cost and high power consumption.
[0005] Summary of the Invention
[0006] The present disclosure provides a face recognition method and device, an in-memory computing system, an electronic device, and a computer-readable storage medium.
[0007] According to a first aspect of the present disclosure, the present disclosure provides a face recognition method, the face recognition method comprising:
[0008] Get the image to be processed;
[0009] Inputting the image to be processed into an in-memory computing chip configured with a face detection model, performing face detection processing on the image to be processed using a preset face detection model, and obtaining an initial face detection frame in the image to be processed;
[0010] Acquire a target face image according to the initial face detection frame;
[0011] Performing face recognition processing on the target face image in the in-memory computing chip using a face recognition model pre-configured in the in-memory computing chip to obtain a target face feature vector corresponding to the target face image;
[0012] The target face feature vector is compared with the feature vector corresponding to the preset face image to obtain a face recognition result.
[0013] In some embodiments, acquiring a target face image according to the initial face detection frame includes:
[0014] Inputting the initial face detection frame into a programmable logic unit for non-maximum suppression processing to obtain a target face detection frame;
[0015] and inputting the target face detection frame into the in-memory computing chip;
[0016] The target face detection frame is mapped onto the image to be processed in an in-memory computing chip to obtain the target face image corresponding to the target face detection frame on the image to be processed.
[0017] In some embodiments, before performing the non-maximum suppression process, the face recognition method further includes:
[0018] In the programmable logic unit, based on the confidence levels of the multiple initial face detection frames output by the face detection model, a preset number of initial face detection frames are screened out from the multiple initial face detection frames in descending order of confidence levels;
[0019] The step of inputting the initial face detection frame into a programmable logic unit for non-maximum suppression processing to obtain a target face detection frame includes:
[0020] A preset number of initial face detection frames are input into a programmable logic unit for non-maximum suppression processing to obtain a target face detection frame.
[0021] In some embodiments, the acquiring of the image to be processed and the face recognition method further include:
[0022] Get the original image;
[0023] performing image decoding and moving object detection on the original image;
[0024] In response to detecting the presence of a moving object in the original image, the original image is input into a programmable logic unit to perform image preprocessing on the original image to obtain the image to be processed.
[0025] In some embodiments, the image preprocessing includes at least one of the following: image scaling processing, image cropping processing, image format conversion processing, and image enhancement processing.
[0026] In some embodiments, the moving object detection includes performing moving object detection on the original image using a frame difference method.
[0027] In some embodiments, comparing the target facial feature vector with a preset facial feature vector to obtain a facial recognition result includes:
[0028] The target face feature vector is input into a programmable logic unit, so that the programmable logic unit performs the following processing:
[0029] Normalizing the target face feature vector to obtain a target face normalized feature vector;
[0030] Comparing the normalized feature vector of the target face with the feature vector corresponding to the preset face image in the preset database to obtain the face feature matching degree;
[0031] When the facial feature matching degree is greater than a preset threshold, a face recognition result indicating successful face matching is obtained.
[0032] In some embodiments, the face detection model includes: a Backbone layer, a Neck layer, and a Head layer;
[0033] The Backbone layer is used to extract image features of the input image to be processed;
[0034] The Neck layer is used to reduce or adjust the image features from the Backbone layer;
[0035] The Head layer is used to generate the final network output by inputting the features processed by the Neck layer to obtain an initial face detection frame.
[0036] In some embodiments, the face recognition model is trained using knowledge distillation.
[0037] According to a second aspect of the present disclosure, the present disclosure provides a face recognition device, the face recognition device comprising:
[0038] an acquisition module, configured to acquire an image to be processed;
[0039] a detection module configured to input the image to be processed into an in-memory computing chip configured with a face detection model, perform face detection processing on the image to be processed using the face detection model, and obtain an initial face detection frame in the image to be processed;
[0040] a processing module, configured to acquire a target face image according to the initial face detection frame;
[0041] a recognition module configured to perform face recognition processing on the target face image using a face recognition model pre-configured in the in-memory computing chip in the in-memory computing chip to obtain a target face feature vector corresponding to the target face image;
[0042] The comparison module is configured to compare the target face feature vector with the feature vector corresponding to the preset face image to obtain a face recognition result.
[0043] According to a third aspect of the present disclosure, the present disclosure provides an in-memory computing system, the in-memory computing system comprising:
[0044] An in-memory computing chip, on which a face detection model is integrated, is configured to perform face detection processing on the image to be processed using the face detection model, obtain an initial face detection frame in the image to be processed, and send the initial face detection frame to the PL end;
[0045] The PL end is configured to obtain a target face image according to the initial face detection frame and send the target face image to the in-memory computing chip;
[0046] The in-memory computing chip is further integrated with a face recognition model, and is further configured to perform face recognition processing on the target face image using the face recognition model, obtain a target face feature vector corresponding to the target face image, and send the target face feature vector to the PL end;
[0047] The PL end is further configured to compare the target face feature vector with the feature vector corresponding to the preset face image to obtain a face recognition result.
[0048] In some embodiments, the PL terminal is configured as follows:
[0049] Performing non-maximum suppression processing on the initial face detection frame to obtain a target face detection frame;
[0050] The target face detection frame is mapped onto the image to be processed to obtain the target face image corresponding to the target face detection frame on the image to be processed.
[0051] In some embodiments, the in-memory computing system further includes: a PS end, the PS end being deployed on the FPGA device;
[0052] The PS end is configured to: obtain an original image from a camera device; perform image decoding and motion object detection on the original image; and send the original image to the PL end when a motion object is detected in the original image;
[0053] The PL end is further configured to perform image preprocessing on the original image in response to the PS end detecting the presence of a moving object in the original image, obtain the image to be processed, and send the image to be processed to the in-memory computing chip.
[0054] In some embodiments, the PL terminal is configured as follows:
[0055] Normalizing the target face feature vector to obtain a target face normalized feature vector;
[0056] Comparing the normalized feature vector of the target face with the feature vector corresponding to the preset face image in the preset database to obtain the face feature matching degree;
[0057] When the facial feature matching degree is greater than a preset threshold, a face recognition result indicating successful face matching is obtained.
[0058] According to a fourth aspect of the present disclosure, the present disclosure provides an electronic device comprising the above-mentioned in-memory computing system.
[0059] According to a fifth aspect of the present disclosure, the present disclosure provides an electronic device, the electronic device comprising:
[0060] at least one processor; and
[0061] a memory communicatively connected to the at least one processor; wherein,
[0062] The memory stores one or more computer programs that can be executed by the at least one processor, and the one or more computer programs are executed by the at least one processor so that the at least one processor can perform the above-mentioned face recognition method.
[0063] According to a sixth aspect of the present disclosure, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, which implements the above-mentioned face recognition method when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS
[0064] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. The above and other features and advantages will become more apparent to those skilled in the art by describing detailed example embodiments with reference to the accompanying drawings. In the accompanying drawings:
[0065] FIG1 is a flow chart of a face recognition method provided in an embodiment of the present disclosure.
[0066] FIG2 is a schematic diagram of a model architecture of a face detection model in an embodiment of the present disclosure.
[0067] FIG3 is a flow chart of a method for acquiring a target facial image in an embodiment of the present disclosure.
[0068] FIG4 is a flow chart of a method for obtaining an image to be processed in an embodiment of the present disclosure.
[0069] FIG5 is a schematic diagram of a model architecture of a face recognition model in an embodiment of the present disclosure.
[0070] FIG6 is a flow chart of a method for obtaining face recognition results in an embodiment of the present disclosure.
[0071] FIG7 is a schematic structural diagram of a face recognition device provided in an embodiment of the present disclosure.
[0072] FIG8 is a schematic structural diagram of an in-memory computing system provided in an embodiment of the present disclosure.
[0073] FIG9 is a schematic diagram of an application scenario of an in-memory computing system provided by an embodiment of the present disclosure.
[0074] FIG10 is a block diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION
[0075] To enable those skilled in the art to better understand the technical solutions of the present disclosure, exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.
[0076] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.
[0077] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.
[0078] The terms used herein are only used to describe specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms "a" and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "comprising" and / or "made of" are used in this specification, the presence of the features, wholes, steps, operations, elements and / or components is specified, but the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof is not excluded. Similar words such as "connected" or "connected" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect.
[0079] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as having an idealized or overly formal meaning unless expressly defined as such herein.
[0080] FIG1 is a flow chart of a face recognition method provided by an embodiment of the present disclosure. As shown in FIG1 , the face recognition method includes steps S11 to S14.
[0081] Step S10: Acquire the image to be processed.
[0082] Step S11: input the image to be processed into an in-memory computing chip configured with a face detection model, perform face detection processing on the image to be processed using a preset face detection model, and obtain an initial face detection frame in the image to be processed.
[0083] Step S12: Acquire a target face image according to the initial face detection frame.
[0084] Step S13: performing face recognition processing on the target face image in the in-memory computing chip using the face recognition model pre-configured in the in-memory computing chip to obtain a target face feature vector corresponding to the target face image.
[0085] Step S14: Compare the target face feature vector with the feature vector corresponding to the preset face image to obtain a face recognition result.
[0086] According to the face recognition method provided in the embodiment of the present disclosure, a low-power in-memory computing chip is used to integrate a lightweight face detection model and a face recognition model for face detection and recognition. Real-time face detection and recognition can be achieved with low energy consumption. It can be applied to low-power devices such as drones, ink tablets, and wearable devices to realize functions such as device wake-up and unlocking.
[0087] In an embodiment of the present disclosure, in step S11, the image to be processed is input into a face detection model pre-configured in an in-memory computing chip for face detection processing to obtain an initial face detection frame in the image to be processed. The initial face detection frame can be used to represent the position coordinate information of the face in the image predicted by the model.
[0088] In the embodiment of the present disclosure, the input of the face detection model is an image to be processed, the pixel size of the image to be processed is a preset pixel size, and the image format is a preset image format. For example, the pixel size of the image to be processed is 320*240, and the image format is RGB format. The image to be processed is input into the face detection model, and after the neural network inference calculation of the face detection model, the original result of the face detection is output, and the original result of the face detection includes an initial face detection frame.
[0089] Figure 2 is a schematic diagram of the model architecture of a face detection model in an embodiment of the present disclosure. In order to achieve lightweight model design, the number of network channels and the number of network layers of the face detection model are set to smaller values. In some embodiments, as shown in Figure 2, the face detection model includes three parts, namely, a feature extraction layer, a feature processing layer and a prediction output layer. The feature extraction layer is the Backbone layer, the feature processing layer is the Neck layer, and the prediction output layer is the Head layer.
[0090] The feature extraction layer, also known as the Backbone layer, is the backbone of the model, used to extract image features from the input image for subsequent processing and analysis. The Backbone layer consists of an input layer, a convolutional layer (Conv2d), a batch normalization layer (BatchNorm), a nonlinear activation function layer (ReLU6), four network modules (Block), and three attention mechanism network modules (Block_SE). The first convolutional layer (Conv2d) of the model has 16 channels, with N = 1. N represents the number of network modules in the model and can be adjusted based on the computing power of the in-memory computing chip.
[0091] As shown in the right part of Figure 2, the network module (Block) includes three units consisting of a convolutional layer (Conv2d), a batch normalization layer (BatchNorm) and a nonlinear activation function layer (ReLU6); the attention mechanism network module (Block_SE) includes a network module (Block) and an attention mechanism module (Squeeze-and-Excitation, SE). The feature map passes through the network module (Block) and is short-circuited with the attention mechanism module. The attention mechanism module (SE) includes a pooling layer (AveragePooling) and two units consisting of a convolutional layer (Conv2d), a batch normalization layer (BatchNorm) and a nonlinear activation function layer (ReLU6).
[0092] The feature processing layer, or Neck layer, is the intermediate layer connecting the Backbone layer and the Head layer. The main function of the Neck layer is to reduce or adjust the image features from the Backbone layer, improving the diversity and robustness of the features to better meet the task requirements. In the Neck layer, the feature map processed by the seventh module of the seven modules of the Backbone layer is processed through convolution (Conv2d) to obtain the first feature map Feature1; the first feature map Featur1 is upsampled (Upsample) and added (Add) to the feature map processed by the fifth module of the seven modules of the Backbone layer, and then processed through convolution (Conv2d) to obtain the second feature map Feature2; the second feature map Featur2 is upsampled (Upsample) and added (Add) to the feature map processed by the third module of the seven modules of the Backbone layer, and then processed through convolution (Conv2d) to obtain Feature3; the first feature map Feature1, the second feature map Feature2, and the third feature map Feature3 are respectively processed through corresponding convolution (Conv2d) and input into the Head layer.
[0093] The prediction output layer, or Head layer, is the last layer of the model and is the detection head for face detection. It is usually a classifier or regressor that generates the final network output by inputting the features processed by the Neck layer to obtain the initial face detection frame. In the Head layer, the first feature map Feature1, the second feature map Feature2, and the third feature map Feature3 are respectively convolved (Conv2d) in the Neck layer, and then superimposed and fused (Concat) in the Head layer. After convolution (Conv2d) processing and dimensionality conversion (permute, view) processing, the softmax layer outputs three feature map vectors, namely the first feature map vector, the second feature map vector, and the third feature map vector.
[0094] Among them, the first feature map vector represents the initial face detection frame and its corresponding position coordinates, the second feature map vector represents the initial face detection frame and its corresponding facial key point coordinates, and the third feature map vector represents the initial face detection frame and its corresponding confidence. Exemplarily, the first feature map vector, the second feature map vector and the third feature map vector are data of length 3160*4, 3160*10, and 3160*2, respectively. 3160 represents 3160 initial face detection frames, 4 represents the coordinate dimensions (x, y, w, h) of the initial face detection frame, 10 represents the position coordinates (5*(x, y)) of the 5 facial key points corresponding to the initial face detection frame, and the facial key points include the two eyes, nose, and two corners of the mouth of the face. 2 represents the confidence, for a total of 4+10+2=16 dimensions of information.
[0095] In the embodiment of the present disclosure, the main computing resources of the neural network calculation of the model are consumed in multiplication and addition operations. The Processing-In-Memory (PIM) chip is suitable for neural network calculation and can greatly reduce the process of repeated copying of data during the calculation process.
[0096] FIG3 is a flow chart of a method for acquiring a target face image in an embodiment of the present disclosure. In some embodiments, as shown in FIG3 , in step S12, acquiring the target face image according to the initial face detection frame may further include:
[0097] Step S31: Input the initial face detection frame into the programmable logic unit for non-maximum suppression processing to obtain the target face detection frame.
[0098] The initial face detection frame obtained and selected by the above face detection is subjected to non-maximum suppression processing using the non-maximum suppression algorithm (NMS) to obtain the target face detection frame.
[0099] Among them, the non-maximum suppression algorithm NMS is an algorithm that searches for local maximum values and suppresses non-maximum elements.
[0100] During the face detection inference process, many initial face detection frames (such as A, B, C, D, E, F, etc.) are generated. Many of these initial face detection frames are used to detect the same target face, but ultimately each target face only requires one target face detection frame. NMS can be used to select the initial face detection frame with the highest score, assuming it is the initial face detection frame C. The initial face detection frame C is then compared with other initial face detection frames to calculate the corresponding IOU (Intersection over Union) value. The IOU value is an evaluation metric for target detection. When the IOU value exceeds the set threshold, the initial face detection frame that exceeds the threshold is suppressed. The suppression method is to set the score of the initial face detection frame to 0. After one round, the initial face detection frame with the highest score is continued to be searched among the remaining initial face detection frames, and the initial face detection frames whose IOU values exceed the threshold are then suppressed until the initial face detection frames with almost no overlap are retained. In this way, the target face detection frame corresponding to each target face is obtained.
[0101] In some embodiments, after inputting the initial face detection frame into the programmable logic unit and before performing the non-maximum suppression process, the face recognition method further includes:
[0102] In the programmable logic unit, according to the confidences of the multiple initial face detection frames output by the face detection model, a preset number of initial face detection frames are screened out from the multiple initial face detection frames in descending order of confidence.
[0103] For example, the number of initial face detection frames output by the face detection model is 3160, and the preset number is 100. After sorting the initial face detection frames according to their confidence levels, the initial face detection frames corresponding to the top 100 confidence levels in the confidence level ranking are selected for the next step of processing.
[0104] Furthermore, the initial face detection frame is input into a programmable logic unit for non-maximum suppression processing to obtain a target face detection frame, including: inputting a preset number of initial face detection frames into a programmable logic unit for non-maximum suppression processing to obtain a target face detection frame.
[0105] Step S32: input the target face detection frame into the in-memory computing chip.
[0106] Step S33: Map the target face detection frame to the image to be processed in the in-memory computing chip to obtain a target face image corresponding to the target face detection frame on the image to be processed.
[0107] The target face detection frame detected is mapped to the corresponding position coordinates on the image to be processed according to the position coordinates, and the target face image corresponding to the target face detection frame is obtained by cropping. The cropped image is scaled to a predetermined pixel size, for example, a pixel size of 112×112. An affine transformation is performed based on the coordinates of the facial key points (such as eyes, nose, and mouth corners) obtained by face detection to correct the target face image.
[0108] FIG4 is a flow chart of a method for obtaining an image to be processed in an embodiment of the present disclosure. In some embodiments, as shown in FIG4 , in step S10, obtaining the image to be processed includes:
[0109] Step S41: Acquire the original image.
[0110] In some embodiments, the main control system controls the camera to shoot and receives the original images shot by the camera. In some embodiments, the main control system receives the original images shot and transmitted by the camera at regular intervals.
[0111] Step S42: perform image decoding and moving object detection on the original image.
[0112] After the original image is acquired, the main control system decodes the original image and performs moving object detection on the decoded original image.
[0113] In some embodiments, the frame difference method is used to detect moving objects in the original image to determine whether the pixel values of any two corresponding pixel points in the previous and next frame images are consistent. If the number of inconsistent pixel points is greater than a predetermined threshold, it is considered that a moving object appears in the original image.
[0114] Step S43: In response to detecting that a moving object appears in the original image, the original image is input into the programmable logic unit to perform image preprocessing on the original image to obtain an image to be processed.
[0115] The face detection model has certain input requirements. Therefore, after obtaining the original image, before inputting the image into the face detection model for face detection, the image needs to be preprocessed to obtain a processed image that meets the model input requirements.
[0116] In some embodiments, image preprocessing includes at least one of the following: image scaling processing, image format conversion processing, and image enhancement processing.
[0117] Among them, image scaling processing refers to downsampling the image to a preset pixel size, such as downsampling to a 320*240 pixel size. For example, directly taking the values of some pixels in the original image as the pixel values of the scaled image, thereby improving processing efficiency; image format conversion processing refers to converting the image format of the image to a preset image format. For example, the model requires that the input image format is RGB format. If the original image is in RGB format, no format conversion is required. If the original image is in YUV format, format conversion is required; image enhancement processing can be to perform histogram equalization on the image to prevent the image from being too bright or too dark.
[0118] In the disclosed embodiment, in step S13, the acquired target facial image is input into a pre-configured facial recognition model in the in-memory computing chip, so that the in-memory computing chip uses the facial recognition model to perform facial recognition processing on the target facial image, thereby obtaining a target facial feature vector corresponding to the target facial image. The target facial feature vector may have a dimension of 256.
[0119] Figure 5 is a schematic diagram of the model architecture of a face recognition model in an embodiment of the present disclosure. In some embodiments, as shown in Figure 5, due to the limited computing power of the in-memory computing PIM chip, the model is designed according to a small model structure, wherein the network input can be a 64*64 3-channel color image. To achieve lightweight model design, the number of network channels and the number of network layers of the face recognition model are set to smaller values. In some embodiments, as shown in Figure 5, the face recognition model includes a first network unit consisting of a convolutional layer (Conv2d), a batch normalization layer (BatchNorm), and a nonlinear activation function layer (ReLU6), four network modules (Block), three attention mechanism network modules (Block_SE), a second network unit consisting of a convolutional layer (Conv2d), a batch normalization layer (BatchNorm), and a nonlinear activation function layer (ReLU6), a pooling layer (AveragePooling), and a fully connected layer consisting of a first linear layer (Linear), a batch normalization layer (BatchNorm), a nonlinear activation function layer (ReLU6), and a second linear layer (Linear). The number of channels of the first convolutional layer (Conv2d) of the model is set to 16, N = 1, where N is the number of network module blocks of the model and can be adjusted according to the computing power of the in-memory computing chip.
[0120] The network module (Block) includes three units consisting of a convolutional layer (Conv2d), a batch normalization layer (BatchNorm) and a non-linear activation function layer (ReLU6); the attention mechanism network module (Block_SE) includes a network module (Block) and an attention mechanism module (Squeeze-and-Excitation, SE). The feature map passes through the network module (Block) and is short-circuited with the attention mechanism module. The attention mechanism module (SE) includes a pooling layer (AveragePooling) and two units consisting of a convolutional layer (Conv2d), a batch normalization layer (BatchNorm) and a non-linear activation function layer (ReLU6).
[0121] In order to improve the model's ability to express facial features, knowledge distillation is used during the training process to improve its model reasoning and computing capabilities. The face recognition model to be learned and trained is used as the student model (Student), and the number of network channels and network layers of the face recognition model are increased to obtain a model with a larger number of network channels and network layers than the face recognition model, which is used as the teacher model (Teacher). For example, the number of network channels of the first convolutional layer in the student model is set to 16, N=1, and the number of network channels of the first convolutional layer in the teacher model is set to 64, N=4.
[0122] The teacher model is a face recognition model that has been pre-trained using a training dataset, and the student model is the face recognition model to be trained. Knowledge distillation refers to the process of migrating the dark knowledge in the teacher model to the student model. Through knowledge distillation, the expressive power of the student model can be as close to or even better than that of the teacher model as possible, thereby achieving the effect of using a low-complexity student model to obtain the network prediction (output) of a high-complexity teacher model, streamlining network parameters, and ensuring network accuracy.
[0123] FIG6 is a flow chart of a method for obtaining a face recognition result in an embodiment of the present disclosure. In some embodiments, as shown in FIG6 , in step S14, a target face feature vector is compared with a feature vector corresponding to a preset face image to obtain a face recognition result, including: inputting the target face feature vector into a programmable logic unit, so that the programmable logic unit performs the following processing:
[0124] Step S61: normalize the target face feature vector to obtain a normalized target face feature vector.
[0125] Among them, normalization processing refers to subtracting the mean of the target face feature vector and then dividing it by its variance.
[0126] Step S62: Compare the normalized feature vector of the target face with the feature vector corresponding to the preset face image in the preset database to obtain the face feature matching degree.
[0127] The feature vector corresponding to the preset face image is a feature vector obtained by performing face recognition on the preset face image using a face recognition model and then performing normalization processing on the obtained face feature vector.
[0128] In some embodiments, the facial feature matching degree can be obtained by calculating the similarity between two feature vectors, wherein the similarity between the two feature vectors can be calculated using a preset vector similarity algorithm, for example, the vector similarity algorithm can use a Euclidean distance similarity algorithm, a cosine similarity algorithm, etc.
[0129] Step S63: When the facial feature matching degree is greater than a preset threshold, a face recognition result indicating successful face matching is obtained.
[0130] When the facial feature matching degree is greater than the preset threshold, it means that there is a facial image that is relatively similar to the target facial image in the facial image represented by the preset facial feature vector in the preset database, that is, the target facial image is matched in the preset database, and thus a facial recognition result indicating a successful face match is obtained.
[0131] When the facial feature matching degree is less than or equal to the preset threshold, it means that the facial images represented by the preset facial feature vectors in the preset database are not similar to the target facial image, that is, the target facial image cannot be matched in the preset database, and thus a facial recognition result indicating that the face match is unsuccessful is obtained.
[0132] In some embodiments, the normalized feature vector of the target face can be compared one by one with multiple preset face feature vectors in a preset database to obtain multiple face feature matching degrees, and the maximum face feature matching degree is obtained from the multiple face feature matching degrees. The maximum face feature matching degree is compared with a preset threshold to determine whether the target face image matches the face image in the preset database.
[0133] In some embodiments, the establishment of a preset database is to first obtain multiple preset facial images and guide the labeling of their identity information, use a facial recognition model to extract their facial feature vectors, and perform normalization processing to obtain the normalized facial feature vector corresponding to each preset facial image, and save it in the preset database.
[0134] It is understood that the above-mentioned various method embodiments mentioned in this disclosure can be combined with each other to form combined embodiments without violating the principle logic. Due to space limitations, this disclosure will not go into details. It is understood by those skilled in the art that in the above-mentioned methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.
[0135] Figure 7 is a structural diagram of a face recognition device provided by an embodiment of the present disclosure. As shown in Figure 7, the face recognition device 700 includes: an acquisition module 701, a detection module 702, a processing module 703, a recognition module 704 and a comparison module 705.
[0136] Among them, the acquisition module 701 is configured to acquire the image to be processed; the detection module 702 is configured to input the image to be processed into an in-memory computing chip configured with a face detection model, and perform face detection processing on the image to be processed using the face detection model to obtain an initial face detection frame in the image to be processed; the processing module 703 is configured to obtain a target face image based on the initial face detection frame; the recognition module 704 is configured to perform face recognition processing on the target face image in the in-memory computing chip using the face recognition model pre-configured in the in-memory computing chip to obtain a target face feature vector corresponding to the target face image; the comparison module 705 is configured to compare the target face feature vector with the feature vector corresponding to the preset face image to obtain a face recognition result.
[0137] The face recognition device 700 provided in the embodiment of the present disclosure is used to implement the face recognition method provided in any of the above embodiments. For specific related descriptions, please refer to the description of the face recognition method in any of the above embodiments, and will not be repeated here.
[0138] FIG8 is a schematic structural diagram of an in-memory computing system provided by an embodiment of the present disclosure. As shown in FIG8 , the in-memory computing system 800 includes an in-memory computing chip 801 and a PL (Programmable Logic) terminal 802 .
[0139] Among them, the in-memory computing chip 801 has a face detection model integrated thereon, which is configured to use the face detection model to perform face detection processing on the image to be processed, obtain an initial face detection frame in the image to be processed, and send the initial face detection frame to the PL end 802; the PL end 802 is deployed on an FPGA (Field Programmable Gate Array) device, which is configured to obtain a target face image according to the initial face detection frame, and send the target face image to the in-memory computing chip 801; the in-memory computing chip 801 also has a face recognition model integrated thereon, which is further configured to use the face recognition model to perform face recognition processing on the target face image, obtain a target face feature vector corresponding to the target face image, and send the target face feature vector to the PL end 802; the PL end 802 is further configured to compare the target face feature vector with the feature vector corresponding to the preset face image to obtain a face recognition result.
[0140] In some embodiments, the PL terminal 802 is configured to: perform non-maximum suppression processing on the initial face detection frame to obtain a target face detection frame; map the target face detection frame to the image to be processed to obtain a target face image corresponding to the target face detection frame on the image to be processed.
[0141] In some embodiments, as shown in FIG8 , the in-memory computing system 800 further includes a processing system (PS) terminal 803, which is deployed in an FPGA device. The processing system 803 is configured to acquire an image to be processed. Acquiring the image to be processed includes: acquiring a raw image from a camera; performing image decoding and motion object detection on the raw image; and, upon detecting a motion object in the raw image, sending the raw image to the processor (PL) terminal 802. The processor (PL) terminal 802 is further configured to, in response to the PS terminal 803 detecting a motion object in the raw image, perform image preprocessing on the raw image to obtain an image to be processed, and then send the image to the in-memory computing chip 801.
[0142] In some embodiments, a Linux operating system can be deployed on the FPGA device as the main control system (PS side) to realize the control and scheduling of hardware resources. Based on the operating system, some algorithms with strong versatility and low performance requirements (such as motion detection algorithms, etc.) can be implemented.
[0143] In some embodiments, the PL end 802 is configured to: normalize the target face feature vector to obtain a normalized target face feature vector; compare the normalized target face feature vector with the feature vector corresponding to a preset face image in a preset database to obtain a facial feature matching degree; when the facial feature matching degree is greater than a preset threshold, obtain a facial recognition result indicating a successful face match.
[0144] Figure 9 is a schematic diagram of an application scenario of an in-memory computing system provided by an embodiment of the present disclosure. As shown in Figure 9, in some application scenarios, the in-memory computing system 900 can be applied to a terminal device 905, and the main control system (PS end) 903 in the in-memory computing system 900 can serve as a device control end to interact with the camera device 904 and the terminal device 905, such as controlling the camera device 904 to capture images and transmitting face detection results and face recognition results to the terminal device 905.
[0145] In some application scenarios, as shown in FIG9 , the camera device 904 transmits the captured original image to the main control system 903, and the main control system 903 performs image signal decoding and motion object detection to activate the programmable logic unit (PL end) 902; when the main control system 903 detects a motion object, the image is transmitted to the programmable logic unit (PL end) 902 for image preprocessing; the programmable logic unit (PL end) 902 can maintain a normally open, low-computing-power operation mode in the hardware. After activation, the programmable logic unit (PL end) 902 transmits the processed image to the in-memory computing chip 901 which integrates the face detection model and the face recognition model. The in-memory computing chip 901 uses the face detection model to perform face detection. After detection and processing, the feature map is transmitted back to the programmable logic unit (PL end) 902 for feature map post-processing to obtain the target face detection frame, and the corresponding target face image is cropped out from the image to be processed and sent to the in-memory computing chip 901; the in-memory computing chip 901 uses the face recognition model to calculate the face feature vector of the target face image, and then transmits it back to the programmable logic unit (PL end) 902 for face feature comparison to obtain the face recognition result; the programmable logic unit (PL end) 902 feeds back the face detection result and face recognition result to the terminal device 905 through the main control system 903, so that the terminal device 905 can display the face detection result and face recognition result in real time, or use them to implement other functions, such as device wake-up, unlocking, etc.
[0146] In some embodiments, the in-memory computing system 900 may be integrated into the terminal device 905 . Furthermore, the camera device 904 may also be integrated into the terminal device 905 .
[0147] In some application scenarios, the terminal device 905 can be a low-power electronic device such as an ink tablet, a drone, a wearable device, etc., which uses the in-memory computing system 900 to perform face detection and face recognition, thereby realizing functions such as device wake-up display and unlocking.
[0148] In the drone application scenario, the terminal device 905 can also use the in-memory computing system 900 to perform face detection and face recognition to achieve functions such as assisting obstacle avoidance, avoiding pedestrians, and pursuing suspects.
[0149] In the application scenario of wearable devices, such as smart watches and smart glasses, in the application scenario of smart watches, the above-mentioned face recognition method provided by the embodiment of the present disclosure can be used to realize functions such as watch unlocking and waking up the screen. When a single face is recognized and it is the face of the owner of the device, the screen will be lit to display a notification message. When multiple faces are recognized, even if the face of the owner of the device is recognized, the screen can be kept dark and the notification message can be hidden.
[0150] In the application scenario of smart glasses, facial feature matching is achieved locally, but the storage of glasses is limited and the amount of data in the facial feature database is also small. Therefore, the device can upload facial features to the cloud through the Internet and search and compare facial features in the large database in the cloud.
[0151] In some embodiments, the terminal device 905 may also be an electronic device such as a smart phone, a computer, a tablet, or a television.
[0152] An embodiment of the present disclosure also provides an electronic device, which includes the in-memory computing system provided by the above embodiment. The in-memory computing system can be integrated into the electronic device to realize face detection and face recognition functions.
[0153] FIG10 is a block diagram of another electronic device provided by an embodiment of the present disclosure.
[0154] 10 , an embodiment of the present disclosure provides an electronic device 1000, which includes: at least one processor 1001; at least one memory 1002; and one or more I / O interfaces 1003 connected between the processor 1001 and the memory 1002; wherein the memory 1002 stores one or more computer programs that can be executed by the at least one processor 1001, and the one or more computer programs are executed by the at least one processor 1001 so that the at least one processor 1001 can execute the above-mentioned face recognition method.
[0155] The present disclosure also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-mentioned face recognition method. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.
[0156] An embodiment of the present disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above-mentioned face recognition method.
[0157] It will be understood by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable storage medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium).
[0158] As is well known to those skilled in the art, the term computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information (such as computer-readable program instructions, data structures, program modules or other data). Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technology, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, as is well known to those skilled in the art, communication media typically contains computer-readable program instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.
[0159] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.
[0160] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, and conventional procedural programming languages such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., utilizing an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions. The electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.
[0161] The computer program product described herein may be implemented in hardware, software, or a combination thereof. In one embodiment, the computer program product is implemented as a computer storage medium. In another embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).
[0162] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.
[0163] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.
[0164] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.
[0165] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the prescribed logical function. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the prescribed function or action, or can be implemented by a combination of dedicated hardware and computer instructions.
[0166] Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for purposes of limitation. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly indicated, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, it will be understood by those skilled in the art that various changes in form and detail may be made without departing from the scope of the present disclosure as set forth in the appended claims.
Claims
1. A face recognition method, characterized in that: The face recognition method comprises: Get the image to be processed; Inputting the image to be processed into an in-memory computing chip configured with a face detection model, performing face detection processing on the image to be processed using a preset face detection model, and obtaining an initial face detection frame in the image to be processed; Acquire a target face image according to the initial face detection frame; Performing face recognition processing on the target face image using the face recognition model pre-configured in the in-memory computing chip in the in-memory computing chip to obtain a target face feature vector corresponding to the target face image; The target face feature vector is compared with the feature vector corresponding to the preset face image to obtain a face recognition result.
2. The face recognition method according to claim 1, characterized in that: The step of acquiring a target face image according to the initial face detection frame includes: Inputting the initial face detection frame into a programmable logic unit for non-maximum suppression processing to obtain a target face detection frame; and inputting the target face detection frame into an in-memory computing chip; And the target face detection frame is mapped to the image to be processed in an in-memory computing chip to obtain the target face image corresponding to the target face detection frame on the image to be processed.
3. The face recognition method according to claim 2, characterized in that: Before performing the non-maximum suppression process, the face recognition method further includes: In the programmable logic unit, according to the confidences of the multiple initial face detection frames output by the face detection model, a preset number of initial face detection frames are screened out from the multiple initial face detection frames in descending order of confidence; The step of inputting the initial face detection frame into a programmable logic unit for non-maximum suppression processing to obtain a target face detection frame includes: A preset number of initial face detection frames are input into a programmable logic unit for non-maximum suppression processing to obtain a target face detection frame.
4. The face recognition method according to claim 1, characterized in that: The step of obtaining the image to be processed comprises: Get the original image; Performing image decoding and moving object detection on the original image; In response to detecting the presence of a moving object in the original image, the original image is input into a programmable logic unit to perform image preprocessing on the original image to obtain the image to be processed.
5. The face recognition method according to claim 4, characterized in that: The image preprocessing includes at least one of the following: image scaling processing, image cropping processing, image format conversion processing, and image enhancement processing.
6. The face recognition method according to claim 4, characterized in that: The moving object detection includes performing moving object detection on the original image using a frame difference method.
7. The face recognition method according to claim 1, characterized in that: The step of comparing the target face feature vector with a preset face feature vector to obtain a face recognition result includes: The target face feature vector is input into a programmable logic unit so that the programmable logic unit performs the following processing: Normalizing the target face feature vector to obtain a target face normalized feature vector; Compare the normalized feature vector of the target face with the feature vector corresponding to the preset face image in the preset database to obtain the face feature matching degree; When the facial feature matching degree is greater than a preset threshold, a face recognition result indicating successful face matching is obtained.
8. The face recognition method according to any one of claims 1 to 7, characterized in that: The face detection model includes: Backbone layer, Neck layer and Head layer; The Backbone layer is used to extract image features of the input image to be processed; The Neck layer is used to reduce the dimension or adjust the image features from the Backbone layer; The Head layer is used to generate a final network output by inputting the features processed by the Neck layer to obtain an initial face detection frame.
9. The face recognition method according to any one of claims 1 to 7, characterized in that: The face recognition model is trained by using knowledge distillation.
10. A face recognition device, characterized in that: include: An acquisition module, configured to acquire an image to be processed; A detection module is configured to input the image to be processed into an in-memory computing chip configured with a face detection model, perform face detection processing on the image to be processed using the face detection model, and obtain an initial face detection frame in the image to be processed; A processing module, configured to obtain a target face image according to the initial face detection frame; A recognition module configured to perform face recognition processing on the target face image using a face recognition model pre-configured in the in-memory computing chip in the in-memory computing chip to obtain a target face feature vector corresponding to the target face image; The comparison module is configured to compare the target face feature vector with the feature vector corresponding to the preset face image to obtain a face recognition result.
11. An in-memory computing system, characterized in that: include: An in-memory computing chip, on which a face detection model is integrated, is configured to perform face detection processing on the image to be processed using the face detection model, obtain an initial face detection frame in the image to be processed, and send the initial face detection frame to the PL end; The PL end is configured to obtain a target face image according to the initial face detection frame, and send the target face image to the in-memory computing chip; The in-memory computing chip also has a face recognition model integrated thereon, and is further configured to perform face recognition processing on the target face image using the face recognition model, obtain a target face feature vector corresponding to the target face image, and send the target face feature vector to the PL end; The PL end is further configured to compare the target face feature vector with a feature vector corresponding to a preset face image to obtain a face recognition result.
12. The in-memory computing system according to claim 11, characterized in that: The PL terminal is configured as: Performing non-maximum suppression processing on the initial face detection frame to obtain a target face detection frame; The target face detection frame is mapped onto the image to be processed to obtain the target face image corresponding to the target face detection frame on the image to be processed.
13. The in-memory computing system according to claim 11, characterized in that: The in-memory computing system further includes: a PS end, wherein the PS end is deployed in an FPGA device; The PS end is configured to: obtain an original image from a camera device; perform image decoding and moving object detection on the original image; and send the original image to the PL end when a moving object is detected in the original image; The PL end is also configured to perform image preprocessing on the original image in response to the PS end detecting the presence of a moving object in the original image to obtain the image to be processed, and send the image to be processed to the in-memory computing chip.
14. The in-memory computing system according to claim 11, characterized in that: The PL terminal is configured as: Normalizing the target face feature vector to obtain a target face normalized feature vector; Compare the normalized feature vector of the target face with the feature vector corresponding to the preset face image in the preset database to obtain the face feature matching degree; When the facial feature matching degree is greater than a preset threshold, a facial recognition result of successful face matching is obtained.
15. An electronic device, characterized in that: Comprising an in-memory computing system as claimed in any one of claims 11-14.
16. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, and the one or more computer programs are executed by the at least one processor so that the at least one processor can execute the face recognition method as described in any one of claims 1-9.
17. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements the face recognition method according to any one of claims 1 to 9.
Citation Information
Patent Citations
Mobile terminal CPU real-time multifunctional face detection method
CN112232205A
Target detection processing method, apparatus and device
CN113222108A
Improved lightweight YOLOv4 face mask detection method
CN114419705A
Face recognition system for face recognition in unlabeled videos with domain adversarial learning and knowledge distillation
US20180268203A1
Method for training facial recognition model, and method and apparatus for facial recognition
WO2021218899A1
Cited By
Target detection method and device based on FPGA chip, electronic equipment and medium
CN120375158A
Effective face detection method and device, equipment, storage medium and program product
CN121545206A
