Face recognition method and system based on light field camera

Abstract

The present application discloses a method and system for face recognition based on a light field camera, the method comprising: using a light field camera to collect scene images to obtain light field image data; extracting sub-aperture image data of different viewing angles from the light field image data; detecting and processing the sub-aperture image data of different viewing angles to obtain candidate face regions of interest and two-dimensional face feature data; obtaining three-dimensional data of face regions of interest from the light field image data based on the candidate face regions of interest; processing the three-dimensional data of face regions of interest based on the two-dimensional face feature data to obtain three-dimensional space face feature vectors. The face recognition method based on a light field camera of the present application is based on multi-view images for recognition, effectively reducing the impact of occlusion, illumination changes and posture changes on face detection, having strong environmental adaptability and being able to effectively improve the accuracy of face recognition.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and more specifically, to a face recognition method and system based on a light field camera. Background Art

[0002] Traditional face recognition technology is mainly based on the use of a monocular ordinary camera to collect two-dimensional face image data for recognition. In uncontrolled scenarios, due to the influence of factors such as lighting changes, posture changes, and occlusion, the collected two-dimensional face image data may be missing data, which greatly reduces the accuracy of face recognition. In recent years, researchers have tried to use RGBD cameras or binocular cameras to collect three-dimensional face data for recognition, and to improve the accuracy of face recognition by increasing depth information. However, the RGBD camera method has the problems of low image resolution, small field of view, and low face detection accuracy; the binocular camera method has the problem that the two cameras need to be synchronized in time, and the external parameters of the two cameras are easily disturbed by the outside world, which seriously affects the accuracy of face recognition.

[0003] In summary, at this stage, there is an urgent need to develop a face recognition method and system based on light field cameras to solve the above-mentioned technical problems. Summary of the invention

[0004] One purpose of the present application is to provide a new technical solution for a face recognition method and system based on a light field camera.

[0005] According to a first aspect of the present application, a face recognition method based on a light field camera is provided, the method comprising:

[0006] Using a light field camera to collect scene images to obtain light field image data;

[0007] Extracting sub-aperture image data of different viewing angles from the light field image data;

[0008] Detect and process the sub-aperture image data of different viewing angles to obtain the candidate human face region of interest and two-dimensional human face feature data;

[0009] Acquire three-dimensional data of a human face region of interest from the light field image data based on the candidate human face region of interest;

[0010] The three-dimensional data of the face region of interest is processed according to the two-dimensional face feature data to obtain a three-dimensional face feature vector.

[0011] According to the method of the first aspect of the present application, the sub-aperture image data of different viewing angles include -135° sub-aperture images, -45° sub-aperture images, central sub-aperture images, 45° sub-aperture images and 135° sub-aperture images.

[0012] According to the method of the first aspect of the present application, detection and processing are performed on sub-aperture image data of different viewing angles to obtain candidate human face regions of interest and two-dimensional human face feature data, specifically including:

[0013] Using a preset two-dimensional face detection model to detect and process sub-aperture image data of different viewing angles to obtain multiple rectangular face regions;

[0014] Merging multiple face rectangular regions to obtain the candidate face region of interest;

[0015] The preset two-dimensional human face feature algorithm library is used to perform feature extraction on the candidate human face region of interest to obtain corresponding two-dimensional human face feature data.

[0016] According to the method of the first aspect of the present application, the acquiring three-dimensional data of the face region of interest from the light field image data based on the candidate face region of interest specifically includes:

[0017] Transforming the candidate face region of interest in the central sub-aperture coordinate system into the light field coordinate system to obtain the light field region of interest;

[0018] The light field region of interest is processed to obtain three-dimensional data of the face region of interest.

[0019] According to the method of the first aspect of the present application, the three-dimensional data of the face region of interest is processed according to the two-dimensional face feature data to obtain a three-dimensional face feature vector, specifically comprising:

[0020] Calculating and processing the three-dimensional data of the face region of interest according to the two-dimensional face feature data to obtain three-dimensional feature point coordinate data, three-dimensional feature point normals, three-dimensional feature point first-order gradient data and / or three-dimensional feature point second-order gradient data;

[0021] A preset three-dimensional facial feature model is used to process three-dimensional feature point coordinate data, three-dimensional feature point normals, three-dimensional feature point first-order gradient data and / or three-dimensional feature point second-order gradient data to obtain the three-dimensional space facial feature vector.

[0022] According to the method of the first aspect of the present application, before extracting sub-aperture image data of different viewing angles from the light field image data, the method further includes:

[0023] The obtained light field image data is subjected to distortion correction using the center point distortion correction parameter.

[0024] According to the method of the first aspect of the present application, the process of obtaining the center point distortion correction parameter is:

[0025] Shooting a white scene with the light field camera to obtain white image data;

[0026] Locating and obtaining the center point coordinate data of each microlens from the white image data;

[0027] The center point distortion correction parameters are calculated based on the center point coordinate data of each microlens, the distortion model and the energy equation.

[0028] According to a second aspect of the present application, a face recognition system based on a light field camera is provided, the system comprising:

[0029] The light field camera is configured to capture a scene image to obtain light field image data;

[0030] An extraction module is configured to extract sub-aperture image data of different viewing angles from the light field image data;

[0031] The first processing module is configured to detect and process the sub-aperture image data of different viewing angles to obtain the candidate human face region of interest and two-dimensional human face feature data;

[0032] An acquisition module is configured to acquire three-dimensional data of a human face region of interest from the light field image data based on the candidate human face region of interest;

[0033] The second processing module is configured to process the three-dimensional data of the facial region of interest according to the two-dimensional facial feature data to obtain a three-dimensional spatial facial feature vector.

[0034] According to a third aspect of the present application, an electronic device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps in the face recognition method based on a light field camera as described in the first aspect of the present application are implemented.

[0035] According to a fourth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the face recognition method based on a light field camera as described in the first aspect of the present application are implemented.

[0036] According to an embodiment disclosed in the present application, a method and system for face recognition based on a light field camera of the present application have the following beneficial effects:

[0037] The face recognition method based on a light field camera of the present application is based on multi-view images collected by the light field camera. Since the occluded area may be visible in some viewing angles, the problem of the face being unable to be detected due to partial occlusion can be solved. In addition, the shadow distribution under different viewing angles is different and the face posture is different. The impact of illumination changes and posture changes on face detection can be reduced through multi-view integration. Therefore, the face recognition method based on a light field camera of the present application has strong environmental adaptability and can effectively improve the accuracy of face recognition.

[0038] Other features and advantages of the present application will become apparent from the following detailed description of exemplary embodiments of the present application with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the application and, together with the description, serve to explain the principles of the application.

[0040] Figure 1 A schematic diagram of a flow chart of a face recognition method based on a light field camera provided according to an embodiment;

[0041] Figure 2 Schematic diagram of the process of obtaining the region of interest for the candidate face;

[0042] Figure 3 Schematic diagram of the process of obtaining the facial feature vector in three-dimensional space;

[0043] Figure 4 Schematic diagram of light field image distortion;

[0044] Figure 5 Schematic diagram of light field image distortion correction, where 5(a) is the image before distortion correction, and 5(b) is the image after distortion correction;

[0045] Figure 6 Schematic diagram of vignetting correction for light field images, where 6(a) is the image before vignetting correction, and 5(b) is the image after vignetting correction;

[0046] Figure 7 The figure is a schematic diagram of the calculated 3D data of the face region of interest;

[0047] Figure 8 A structural schematic diagram of a face recognition system based on a light field camera provided according to an embodiment;

[0048] Fig. 9 A schematic diagram of an electronic device. DETAILED DESCRIPTION

[0049] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present application.

[0050] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present application, its application, or uses.

[0051] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0052] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0053] Embodiment 1:

[0054] In uncontrolled scenes, factors such as lighting changes, posture changes, and occlusions can cause data loss in the collected two-dimensional face image data, greatly reducing the accuracy of face recognition. The method of using RGBD cameras to collect three-dimensional face data for recognition has problems such as low image resolution, small field of view, and low face detection accuracy; the method of using binocular cameras to collect three-dimensional face data for recognition has the problem that the two cameras need to be synchronized, and the external parameters of the two cameras are easily disturbed by the outside world, which seriously affects the accuracy of face recognition.

[0055] In order to solve the above technical problems, the present application embodiment provides a face recognition method based on a light field camera, see Figure 1 As shown, the method includes:

[0056] Step S1: using a light field camera to capture a scene image to obtain light field image data;

[0057] In this embodiment, a light field camera is used to simultaneously capture scene images from multiple perspectives through a single exposure.

[0058] It should be noted that the light field camera in this embodiment may be a microlens array light field camera (for example, Lytro Illum, Raytrix R5), a camera array light field camera, or a coded mask light field camera, and this application does not make any specific limitations on this.

[0059] Preferably, a microlens array light field camera is used in this embodiment.

[0060] It should be noted that the scene images collected in this embodiment refer to images of scenes where face recognition is required in specific circumstances, such as security inspection scenes, criminal investigation scenes, or autonomous driving scenes, etc. This application does not make any specific limitations on this.

[0061] Step S2: extracting sub-aperture image data of different viewing angles from the light field image data;

[0062] It should be noted that, in order to improve the image quality and ensure the accuracy of face recognition in this embodiment, a series of processing is first performed on each frame of light field image data collected, specifically including: Bayer conversion, which is used to obtain color light field image data; filtering processing, which is used to filter and denoise the image data and remove high-frequency noise in low-texture areas; vignetting correction, which is used to remove dark corners of the image data and compensate for lens edges.

[0063] It should be noted that it is difficult to formally represent four-dimensional light field data in a three-dimensional world. Usually, the light field is visualized by fixing any two dimensions of the dual-plane light field model and displaying two-dimensional slices. In this embodiment, light field data in the form of a sub-aperture image array can be obtained by fixing the angle dimension, that is: the pixel values ​​at the center of each microlens are extracted in sequence and reassembled into a new image in sequence, and the central sub-aperture image is obtained; similarly, all microlenses are traversed in sequence, and the coordinates of a certain angle are inferred based on the coordinate points of the center of the microlens, and the corresponding pixel values ​​are extracted, and then a new image can be reassembled, which is the sub-aperture image of the angle.

[0064] It should be noted that in order to ensure the accuracy of face recognition while improving the data processing speed, in this embodiment, sub-aperture images of several angles including the central viewing angle can be selected for extraction, such as: central sub-aperture image, 30° sub-aperture image, 45° sub-aperture image, 60° sub-aperture image, 135° sub-aperture image, -30° sub-aperture image, -45° sub-aperture image, -60° sub-aperture image and -135° sub-aperture image, etc. This application does not make any specific limitations on this.

[0065] Preferably, the sub-aperture image data of different viewing angles in the face recognition method based on a light field camera of this embodiment includes a -135° sub-aperture image, a -45° sub-aperture image, a center sub-aperture image, a 45° sub-aperture image and a 135° sub-aperture image.

[0066] In this embodiment, by extracting sub-aperture image data of different viewing angles, the missing of two-dimensional face image data caused by factors such as illumination changes, posture changes, and occlusion in uncontrolled scenes is avoided.

[0067] Step S3: Detect and process the sub-aperture image data of different viewing angles to obtain candidate human face regions of interest and two-dimensional human face feature data;

[0068] It should be noted that the sub-aperture images of different viewing angles extracted in step S2 of this embodiment are all two-dimensional images. Traditional image processing algorithms or pre-trained neural network models can be used to detect the facial region of interest and extract corresponding two-dimensional facial features. This application does not make any specific limitations on this.

[0069] The candidate human face region of interest obtained in this embodiment may be a two-dimensional rectangular frame, and the coordinate data of the pixel points in the frame and the coordinate data of the two-dimensional human face feature points are recorded.

[0070] Step S4: acquiring three-dimensional data of the face region of interest from the light field image data based on the candidate face region of interest;

[0071] Step S4 of this embodiment is to calculate the corresponding three-dimensional data of the human face region of interest (ie, three-dimensional point cloud data) from the light field image data based on the candidate human face region of interest obtained in step S3.

[0072] Optionally, step S4 in the face recognition method based on a light field camera in this embodiment specifically includes:

[0073] Step S41: transforming the candidate face region of interest in the central sub-aperture coordinate system into the light field coordinate system to obtain the light field region of interest;

[0074] Step S42: performing a calculation on the light field region of interest to obtain three-dimensional data of the face region of interest.

[0075] Step S5: Processing the three-dimensional data of the facial region of interest according to the two-dimensional facial feature data to obtain a three-dimensional spatial facial feature vector.

[0076] In step S5 of this embodiment, the two-dimensional facial feature data and the three-dimensional data of the facial region of interest can be processed by a pre-trained 3D face model to directly obtain a three-dimensional spatial facial feature vector; it is also possible to first quickly determine the corresponding three-dimensional facial feature point coordinate data in the three-dimensional data of the facial region of interest based on the two-dimensional facial feature data, and then use a traditional facial feature processing algorithm or a pre-trained neural network model to process the three-dimensional facial feature point coordinate data to obtain the three-dimensional spatial facial feature vector ultimately used for face recognition.

[0077] Optionally, step S3 in the face recognition method based on a light field camera in this embodiment specifically includes:

[0078] Step S31: using a preset two-dimensional face detection model to detect and process sub-aperture image data of different viewing angles to obtain a plurality of rectangular face regions;

[0079] It should be noted that, in this embodiment, the preset two-dimensional face detection model may be a pre-trained ‌YOLO model (You Only Look Once), such as YOLOv1, YOLOv2, YOLOv3 or YOLOv5, and this application does not make any specific limitation on this.

[0080] Step S32: merging multiple face rectangular regions to obtain a candidate face region of interest;

[0081] In this embodiment, multiple rectangular face regions may be merged, and their union may be taken to obtain a candidate face region of interest.

[0082] Step S33: extracting features from the candidate face region of interest using a preset two-dimensional face feature algorithm library to obtain corresponding two-dimensional face feature data.

[0083] It should be noted that the preset two-dimensional face feature algorithm library in this embodiment may be a dlib algorithm library, and of course may be other algorithm libraries, which are not listed one by one in this application.

[0084] For an example, see Figure 2 As shown in the figure, firstly, the YOLO model is used to perform face detection on the -135° sub-aperture image, -45° sub-aperture image, center sub-aperture image, 45° sub-aperture image and 135° sub-aperture image, respectively, to obtain the corresponding five rectangular face regions; then, the five rectangular face regions are merged to obtain the candidate face region of interest (i.e., face ROI region), and finally, the dlib algorithm library is used to extract features from the candidate face region of interest to obtain two-dimensional face feature data.

[0085] In the case of occlusion of the face area, multiple face rectangular areas can be obtained through sub-aperture images of multiple perspectives. Each face rectangular area contains different face areas due to different perspectives. By merging multiple face rectangular areas, they can complement each other and reduce or eliminate the impact of occlusion. In the case of changes in illumination of the face area, multiple face rectangular areas can be obtained through sub-aperture images of multiple perspectives. Each face rectangular area contains different shadow areas of the face area due to different perspectives. By merging multiple face rectangular areas, they can complement each other and reduce or eliminate the impact of shadows.

[0086] Optionally, step S5 in the face recognition method based on a light field camera in this embodiment specifically includes:

[0087] Step S51: Calculate and process the three-dimensional data of the face region of interest based on the two-dimensional face feature data to obtain three-dimensional feature point coordinate data, three-dimensional feature point normals, three-dimensional feature point first-order gradient data and / or three-dimensional feature point second-order gradient data;

[0088] Step S52: using a preset 3D facial feature model to process the 3D feature point coordinate data, the 3D feature point normal, the 3D feature point first-order gradient data and / or the 3D feature point second-order gradient data to obtain a 3D spatial facial feature vector.

[0089] It should be noted that the characteristic of light field is that it not only collects information on light intensity, but also collects information on light direction. Three-dimensional information can be obtained through ray tracing. Three-dimensional facial feature extraction can provide richer information than two-dimensional facial features, thus making face recognition results more reliable.

[0090] It should be noted that the preset three-dimensional facial feature model in this embodiment can be an ANN model, and of course it can also be other models, which are not listed one by one in this application.

[0091] For an example, see Figure 3 As shown, firstly, based on the two-dimensional facial feature data (i.e., 2D facial feature points), the three-dimensional feature point coordinate data (i.e., feature point 3D coordinates) is extracted from the three-dimensional data (i.e., 3D data) of the facial region of interest; then, a simple gradient calculation can be performed to obtain the first-order gradient data of the three-dimensional feature points (i.e., the first-order gradient of the feature points) and the second-order gradient data of the three-dimensional feature points (i.e., the first-order gradient of the feature points), and the three-dimensional feature point normal can also be calculated; finally, the above data is input into the ANN model for fitting processing to obtain the three-dimensional spatial facial feature vector (i.e., 3D facial feature vector).

[0092] Optionally, before step S1, the face recognition method based on a light field camera in this embodiment further includes:

[0093] Step S0: performing distortion correction on the acquired light field image data using the center point distortion correction parameters.

[0094] Optionally, the process of acquiring the center point distortion correction parameter in the face recognition method based on the light field camera of this embodiment is as follows:

[0095] White image data is obtained by photographing a white scene with a light field camera;

[0096] Locate and obtain the center point coordinate data of each microlens from the white image data;

[0097] The center point distortion correction parameters are calculated based on the center point coordinate data of each microlens, the distortion model and the energy equation.

[0098] Optionally, the distortion model in the face recognition method based on the light field camera of this embodiment is expressed as:

[0099] (1)

[0100] (2)

[0101] The energy equation E is expressed as:

[0102] (3)

[0103] Among them, x, y are the horizontal and vertical coordinates of the original image; x d , y d is the horizontal coordinate and vertical coordinate of the center point of the microlens in the original image; u , y u k is the horizontal coordinate and vertical coordinate of the center point of the microlens after correction; 1 , k 2 is the radial distortion coefficient; p 1 , p 2 is the tangential distortion coefficient; r is the distance from the original pixel to the distortion center; x mla , y mla are the standard abscissa and ordinate of the center point of the microlens.

[0104] It should be noted that traditional image correction performs distortion correction by photographing a calibration plate, and the general reprojection error can reach 0.2 pixels. If the magnification of the distance is taken into account, the correction accuracy is generally tens or hundreds of microns. In this embodiment, the center point of the white image is used to correct the light field image, and the ultra-high precision of microlens processing is utilized. Its accuracy is generally at the nanometer level, so it can significantly improve the distortion correction accuracy.

[0105] The face recognition method based on the light field camera of this embodiment is specifically described below with a specific example:

[0106] 1) Calculation of center point distortion correction parameters;

[0107] Capture white images: A light field camera captures a white scene, such as a piece of white paper or a white wall, to obtain a white image with relatively uniform brightness.

[0108] Center point extraction: Locate the center point of each microlens by finding the circular area in the white image.

[0109] Calculate distortion correction parameters: Due to camera distortion, the arrangement of the center points of the white image is not consistent with the arrangement of the center points of the microlenses. Figure 4 As shown, green is the center point of the microlens, and red is the center point of the white image detection. Due to lens distortion, the center point deviates significantly at the four corners far from the center. According to the above distortion models (1), (2) and the constructed energy equation (3), the center point distortion correction parameter k is obtained through a nonlinear optimization algorithm. 1 , k 2 , p 1 , p2 , the center point distortion correction parameters are applied to the original light field image to obtain undistorted light field data.

[0110] 2) Obtain light field data:

[0111] Bayer conversion: Each frame of light field data collected is first converted by Bayer to obtain a color image;

[0112] Filtering: Use bilateral filtering algorithm to remove high-frequency noise in low-texture areas of light field images;

[0113] Distortion correction: The light field image (such as Figure 5 a) is used to perform distortion correction, and the corrected image is a distortion-free image (as shown in Figure 5 b);

[0114] Vignetting Correction: Distortion-free images (such as Figure 6 a) is vignetted to remove dark corners and compensate for lens edges (as shown in Figure 6 b).

[0115] 3) Image data preprocessing: By detecting sub-aperture images of different viewing angles, the candidate face regions of interest and two-dimensional face feature data are obtained. Preprocessing can eliminate most of the data that does not contain the face area, thereby saving solution time and achieving a higher calculation frame rate.

[0116] 4) 3D solution and face recognition:

[0117] Region transformation: transform the candidate face region of interest from the central sub-aperture coordinate system to the original light field coordinate system;

[0118] Light field solution: solve the 3D data of the candidate face interest area corresponding to the original light field, that is, obtain the 3D data of the face interest area, such as Figure 7 As shown;

[0119] 3D face detection: The three-dimensional data of the facial area of ​​interest and the two-dimensional facial feature data are taken as input, and processed by the 3D face model to obtain a three-dimensional face feature vector. The three-dimensional face feature vector is a multidimensional vector representing facial features in 3D space and is used to identify different faces.

[0120] The face recognition method based on the light field camera of the embodiment of the present application first uses the light field camera to collect scene images from multiple perspectives at the same time through a single exposure, then extracts sub-aperture image data of different perspectives from the light field image data, and then detects and processes the sub-aperture image data of different perspectives to obtain candidate face interest regions and two-dimensional face feature data, and then obtains three-dimensional data of face interest regions from the light field image data based on the candidate face interest regions, and finally processes the three-dimensional data of face interest regions according to the two-dimensional face feature data to obtain three-dimensional space face feature vectors, which are finally used for three-dimensional face recognition. The face recognition method of the embodiment of the present application is based on multi-perspective images collected by the light field camera. Since the occluded area may be visible in some perspectives, the problem of the face being unable to be detected due to partial occlusion can be solved. In addition, the shadow distribution under different perspectives is different and the face posture is different. The impact of illumination changes and posture changes on face detection can be reduced through multi-perspective integration. In summary, the face recognition method based on the light field camera of the embodiment of the present application has strong environmental adaptability and can effectively improve the accuracy of face recognition.

[0121] Embodiment 2:

[0122] The present application embodiment provides a face recognition system 1 based on a light field camera, see Figure 8 As shown, the system includes:

[0123] The light field camera 10 is configured to capture scene images to obtain light field image data;

[0124] The extraction module 20 is configured to extract sub-aperture image data of different viewing angles from the light field image data;

[0125] The first processing module 30 is configured to detect and process the sub-aperture image data of different viewing angles to obtain the candidate human face region of interest and two-dimensional human face feature data;

[0126] The acquisition module 40 is configured to acquire three-dimensional data of the human face region of interest from the light field image data based on the candidate human face region of interest;

[0127] The second processing module 50 is configured to process the three-dimensional data of the facial region of interest according to the two-dimensional facial feature data to obtain a three-dimensional spatial facial feature vector.

[0128] Optionally, the sub-aperture image data of different viewing angles extracted by the extraction module 20 in the face recognition system based on the light field camera of this embodiment includes a -135° sub-aperture image, a -45° sub-aperture image, a center sub-aperture image, a 45° sub-aperture image and a 135° sub-aperture image.

[0129] Optionally, the first processing module 30 in the face recognition system based on the light field camera of this embodiment specifically includes:

[0130] A first processing unit is configured to detect and process sub-aperture image data of different viewing angles using a preset two-dimensional face detection model to obtain a plurality of rectangular face regions;

[0131] The second processing unit is configured to merge the multiple face rectangular regions to obtain a candidate face region of interest;

[0132] The third processing unit is configured to extract features from the candidate human face region of interest using a preset two-dimensional human face feature algorithm library to obtain corresponding two-dimensional human face feature data.

[0133] Optionally, the acquisition module 40 in the face recognition system based on the light field camera of this embodiment specifically includes:

[0134] A transformation unit is configured to transform the candidate face region of interest in the central sub-aperture coordinate system into the light field coordinate system to obtain the light field region of interest;

[0135] The solving unit is configured to solve the light field region of interest to obtain three-dimensional data of the face region of interest.

[0136] Optionally, the second processing module 50 in the face recognition system based on the light field camera of this embodiment specifically includes:

[0137] a fourth processing unit, configured to calculate and process the three-dimensional data of the face region of interest according to the two-dimensional face feature data to obtain three-dimensional feature point coordinate data, three-dimensional feature point normals, three-dimensional feature point first-order gradient data and / or three-dimensional feature point second-order gradient data;

[0138] The fifth processing unit is configured to use a preset three-dimensional facial feature model to process the three-dimensional feature point coordinate data, the three-dimensional feature point normal, the three-dimensional feature point first-order gradient data and / or the three-dimensional feature point second-order gradient data to obtain a three-dimensional spatial facial feature vector.

[0139] Optionally, the light field camera-based face recognition system 1 of this embodiment further includes: a correction module configured to perform distortion correction on the acquired light field image data using a center point distortion correction parameter.

[0140] Optionally, the face recognition system based on the light field camera of this embodiment also includes: a correction parameter acquisition module, which is specifically configured to: obtain white image data by shooting a white scene with a light field camera; obtain the center point coordinate data of each microlens from the white image data; and obtain the center point distortion correction parameter based on the center point coordinate data of each microlens, a distortion model and an energy equation.

[0141] It is not difficult to find that the face recognition system based on the light field camera of this embodiment is a system embodiment corresponding to the first embodiment, and this embodiment can be implemented in conjunction with the first embodiment. The relevant technical details mentioned in the first embodiment are still valid in this embodiment, and in order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the first embodiment.

[0142] It is worth mentioning that all modules involved in this embodiment are logic modules. In practical applications, a logic unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, in order to highlight the innovative part of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed by this application, but this does not mean that there are no other units in this embodiment.

[0143] Embodiment three:

[0144] The present application discloses an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of a face recognition method based on a light field camera in any one of the first embodiments disclosed in the present application are implemented.

[0145] Fig. 9 is a structural diagram of an electronic device according to an embodiment of the present application, such as Fig. 9 As shown, the electronic device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, near field communication (NFC) or other technologies. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the housing of the electronic device, or an external keyboard, touchpad or mouse, etc.

[0146] Those skilled in the art will understand that Fig. 9The structure shown in the figure is only a structural diagram of the part related to the technical solution of the present disclosure, and does not constitute a limitation on the electronic device to which the technical solution of the present application is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0147] Embodiment 4:

[0148] The embodiment of the present application discloses a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any one of the face recognition methods based on a light field camera in the first embodiment disclosed in the present application are implemented.

[0149] Please note that the technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. The above embodiments only express several implementation methods of the present application, and their descriptions are relatively specific and detailed, but they cannot be understood as limiting the scope of the patent application. It should be pointed out that for ordinary technicians in this field, without departing from the concept of the present application, several variations and improvements can be made, which all belong to the scope of protection of the present application. Therefore, the scope of protection of the patent application of this application shall be based on the attached claims.

[0150] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

[0151] Although some specific embodiments of the present application have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are only for illustration, not for limiting the scope of the present application. It should be understood by those skilled in the art that the above embodiments may be modified without departing from the scope and spirit of the present application. The scope of the present application is defined by the appended claims.

Claims

1. A face recognition method based on a light field camera, characterized in that: The method comprises: Step S1: using a light field camera to capture a scene image to obtain light field image data; Step S2: directly extracting sub-aperture image data of different viewing angles from the light field image data in a fixed angle dimension manner; Step S3: Detect and process the sub-aperture image data of different viewing angles to obtain candidate human face regions of interest and two-dimensional human face feature data; Step S4: acquiring three-dimensional data of a human face region of interest from the light field image data based on the candidate human face region of interest; Step S5: processing the three-dimensional data of the face region of interest according to the two-dimensional face feature data to obtain a three-dimensional face feature vector for face recognition; The processing of the three-dimensional data of the face region of interest according to the two-dimensional face feature data to obtain a three-dimensional face feature vector specifically includes: Calculating and processing the three-dimensional data of the face region of interest according to the two-dimensional face feature data to obtain three-dimensional feature point coordinate data, three-dimensional feature point normals, three-dimensional feature point first-order gradient data and / or three-dimensional feature point second-order gradient data; Using a preset three-dimensional facial feature model to process three-dimensional feature point coordinate data, three-dimensional feature point normals, three-dimensional feature point first-order gradient data and / or three-dimensional feature point second-order gradient data to obtain the three-dimensional spatial facial feature vector; The sub-aperture image data of different viewing angles are detected and processed to obtain the candidate face region of interest and two-dimensional face feature data, including: Using a preset two-dimensional face detection model to detect and process sub-aperture image data of different viewing angles to obtain multiple rectangular face regions; Merging multiple face rectangular regions to obtain the candidate face region of interest; The preset two-dimensional human face feature algorithm library is used to perform feature extraction on the candidate human face region of interest to obtain corresponding two-dimensional human face feature data.

2. The face recognition method based on light field camera according to claim 1, characterized in that: The sub-aperture image data of different viewing angles include a -135° sub-aperture image, a -45° sub-aperture image, a central sub-aperture image, a 45° sub-aperture image, and a 135° sub-aperture image.

3. The face recognition method based on light field camera according to claim 1, characterized in that: The acquiring three-dimensional data of the face region of interest from the light field image data based on the candidate face region of interest specifically includes: Transforming the candidate face region of interest in the central sub-aperture coordinate system into the light field coordinate system to obtain the light field region of interest; The light field region of interest is processed to obtain three-dimensional data of the face region of interest.

4. The face recognition method based on a light field camera according to any one of claims 1 to 3, characterized in that: Before extracting sub-aperture image data of different viewing angles from the light field image data, the method further includes: The obtained light field image data is subjected to distortion correction using the center point distortion correction parameter.

5. The face recognition method based on light field camera according to claim 4, characterized in that: The process of obtaining the center point distortion correction parameters is as follows: Shooting a white scene with the light field camera to obtain white image data; Locating and obtaining the center point coordinate data of each microlens from the white image data; The center point distortion correction parameters are calculated based on the center point coordinate data of each microlens, the distortion model and the energy equation.

6. A face recognition system based on a light field camera, characterized in that: The system comprises: The light field camera is configured to capture a scene image to obtain light field image data; An extraction module is configured to directly extract sub-aperture image data of different viewing angles from the light field image data in a fixed angle dimension manner; The first processing module is configured to detect and process the sub-aperture image data of different viewing angles to obtain the candidate human face region of interest and two-dimensional human face feature data; An acquisition module is configured to acquire three-dimensional data of a human face region of interest from the light field image data based on the candidate human face region of interest; A second processing module is configured to process the three-dimensional data of the face region of interest according to the two-dimensional face feature data to obtain a three-dimensional face feature vector for face recognition; The second processing module is specifically configured to calculate and process the three-dimensional data of the face region of interest according to the two-dimensional face feature data to obtain three-dimensional feature point coordinate data, three-dimensional feature point normals, three-dimensional feature point first-order gradient data and / or three-dimensional feature point second-order gradient data; Using a preset three-dimensional facial feature model to process three-dimensional feature point coordinate data, three-dimensional feature point normals, three-dimensional feature point first-order gradient data and / or three-dimensional feature point second-order gradient data to obtain the three-dimensional spatial facial feature vector; The first processing module is specifically configured to use a preset two-dimensional face detection model to detect and process sub-aperture image data of different viewing angles to obtain multiple face rectangular areas; merge the multiple face rectangular areas to obtain the candidate face region of interest; use a preset two-dimensional face feature algorithm library to extract features of the candidate face region of interest to obtain corresponding two-dimensional face feature data.

7. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the steps in the face recognition method based on a light field camera described in any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps in the face recognition method based on a light field camera described in any one of claims 1 to 5 are implemented.

Images