Face pose detection method and system based on light field camera

Abstract

The invention discloses a face pose detection method and system based on a light field camera. The method comprises the following steps: acquiring a scene image by using the light field camera to obtain light field image data; acquiring three-dimensional data of a human face region of interest from the light field image data; performing detection processing on the three-dimensional data of the region of interest of the human face to obtain information of a plurality of three-dimensional feature points of the human face; and substituting the information of the plurality of face three-dimensional feature points into the face attitude calculation model for solving to obtain a face attitude matrix. According to the light field camera-based face pose detection method, detection is carried out based on the multi-view image, the influence of shielding, illumination change and pose change on detection can be effectively reduced, high environmental adaptability is achieved, and the accuracy of face pose detection can be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and more specifically, to a method and system for detecting facial posture based on a light field camera. Background Art

[0002] Face pose detection based on light field cameras is an important research direction in the field of computer vision. It aims to determine the direction and angle of the face in an image or video, and has broad application prospects in many fields such as security monitoring, human-computer interaction, and autonomous driving.

[0003] At present, the research on face pose detection algorithms based on light field cameras is mainly based on two-dimensional face images, including: feature point detection algorithms, direct regression algorithms based on deep learning, and template matching algorithms. In the process of face pose detection based on light field cameras in real scenes, due to the influence of lighting changes, occlusion, and pose changes, the use of the above detection algorithms is not ideal.

[0004] In summary, at this stage, there is an urgent need to develop a face posture detection method and system based on a light field camera to solve the above-mentioned technical problems. Summary of the invention

[0005] One purpose of the present application is to provide a new technical solution for a method and system for detecting facial posture based on a light field camera.

[0006] According to a first aspect of the present application, a method for detecting face posture based on a light field camera is provided, the method comprising: Using a light field camera to collect scene images to obtain light field image data; Acquire three-dimensional data of a human face region of interest from the light field image data; Detecting and processing the three-dimensional data of the face region of interest to obtain information of a plurality of three-dimensional feature points of the face; The information of multiple facial 3D feature points is brought into the facial posture calculation model for solution to obtain the facial posture matrix.

[0007] According to the method of the first aspect of the present application, the three-dimensional feature point information of the face includes feature point numbers and feature point normals; wherein different feature point numbers represent different positions of the face.

[0008] According to the method of the first aspect of the present application, the face posture calculation model is expressed as: (1) in, Represents the normal of the feature point numbered i, where i can be 1, 2, 3, ... n. represents the rotation matrix, represents the translation vector, represents the angle; The rotation matrix and the translation matrix are obtained by solving the problem that the normal direction of the n feature points is consistent with the Z-axis direction of the light field coordinate system to form the face posture matrix.

[0009] According to the method of the first aspect of the present application, obtaining three-dimensional data of a human face region of interest from the light field image data specifically includes: Extracting sub-aperture image data of different viewing angles from the light field image data; Detect and process the sub-aperture image data of different viewing angles to obtain the candidate face region of interest; Based on the candidate human face region of interest, three-dimensional data of the human face region of interest is acquired from the light field image data.

[0010] According to the method of the first aspect of the present application, detection and processing are performed on sub-aperture image data of different viewing angles to obtain candidate face regions of interest, specifically including: Using a preset two-dimensional face detection model to detect and process sub-aperture image data of different viewing angles to obtain multiple rectangular face regions; A plurality of face rectangular regions are merged to obtain the candidate face region of interest.

[0011] According to the method of the first aspect of the present application, the acquiring three-dimensional data of the face region of interest from the light field image data based on the candidate face region of interest specifically includes: Transforming the candidate face region of interest in the central sub-aperture coordinate system into the light field coordinate system to obtain the light field region of interest; The light field region of interest is processed to obtain three-dimensional data of the face region of interest.

[0012] According to the method of the first aspect of the present application, after merging the multiple rectangular face regions to obtain the candidate face region of interest, the method further includes: Using a preset two-dimensional face feature algorithm library to extract features from the candidate face region of interest to obtain a plurality of two-dimensional face feature data; The detecting and processing of the three-dimensional data of the face region of interest to obtain multiple three-dimensional face feature point information is specifically: processing the three-dimensional data of the face region of interest based on multiple two-dimensional face feature data to obtain multiple three-dimensional face feature point information.

[0013] According to a second aspect of the present application, a face posture detection system based on a light field camera is provided, the system comprising: The light field camera is configured to capture a scene image to obtain light field image data; An acquisition module is configured to acquire three-dimensional data of a human face region of interest from the light field image data; A processing module is configured to detect and process the three-dimensional data of the face region of interest to obtain information of a plurality of three-dimensional feature points of the face; The solution module is configured to bring information of multiple three-dimensional facial feature points into the facial posture calculation model for solution to obtain a facial posture matrix.

[0014] According to a third aspect of the present application, an electronic device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps in the method for detecting facial posture based on a light field camera as described in the first aspect of the present application are implemented.

[0015] According to a fourth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the method for face posture detection based on a light field camera as described in the first aspect of the present application are implemented.

[0016] According to an embodiment disclosed in the present application, a method and system for detecting face posture based on a light field camera of the present application has the following beneficial effects: The face posture detection method based on a light field camera of the present application is based on multi-view images collected by the light field camera. Since the occluded area may be visible in some viewing angles, the problem of the face being unable to be detected due to partial occlusion can be solved. In addition, the shadow distribution under different viewing angles is different and the face posture is different. Through multi-view integration, the influence of illumination changes and posture changes on the accuracy of face detection can be reduced. Therefore, the face posture detection method based on a light field camera of the present application has strong environmental adaptability and can effectively improve the accuracy of face posture detection.

[0017] Other features and advantages of the present application will become apparent from the following detailed description of exemplary embodiments of the present application with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the application and, together with the description, serve to explain the principles of the application.

[0019] Figure 1 A schematic diagram of a flow chart of a method for detecting facial posture based on a light field camera according to an embodiment; Figure 2 Schematic diagram of the process of obtaining the region of interest for the candidate face; Figure 3This is a schematic diagram of the process of obtaining the three-dimensional feature point information of the face; Figure 4 Schematic diagram of light field image distortion; Figure 5 Schematic diagram of light field image distortion correction, where 5(a) is the image before distortion correction, and 5(b) is the image after distortion correction; Figure 6 Schematic diagram of vignetting correction for light field images, where 6(a) is the image before vignetting correction, and 5(b) is the image after vignetting correction; Figure 7 Schematic diagram of the solved 3D data of the face region of interest, where 7(a) is the frontal face pose and 7(b) is the side face pose; Figure 8 Schematic diagram of extracted 3D facial feature points, where 8(a) is the frontal face pose and 8(b) is the side face pose; Fig. 9 This is a schematic diagram of the normal lines of the three-dimensional feature points of a human face; Fig.10 A structural schematic diagram of a face posture detection system based on a light field camera provided according to an embodiment; Fig.11 A schematic diagram of an electronic device. DETAILED DESCRIPTION

[0020] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present application.

[0021] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present application, its application, or uses.

[0022] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0023] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0024] Embodiment 1: At present, the research on face pose detection algorithms based on light field cameras is mainly based on two-dimensional face images, including: feature point detection algorithms, direct regression algorithms based on deep learning, and template matching algorithms. In the process of face pose detection based on light field cameras in real scenes, due to the influence of lighting changes, occlusion, and pose changes, the use of the above detection algorithms is not ideal.

[0025] In order to solve the above technical problems, the present application embodiment provides a face posture detection method based on a light field camera, see Figure 1 As shown, the method includes: Step S1: using a light field camera to capture a scene image to obtain light field image data.

[0026] In this embodiment, a light field camera is used to simultaneously capture scene images from multiple perspectives through a single exposure.

[0027] It should be noted that the light field camera in this embodiment may be a microlens array light field camera (for example, Lytro Illum, Raytrix R5), a camera array light field camera, or a coded mask light field camera, and this application does not make any specific limitations on this.

[0028] Preferably, a microlens array light field camera is used in this embodiment.

[0029] It should be noted that the scene images collected in this embodiment refer to images of scenes that require face posture detection based on a light field camera in specific circumstances, such as security monitoring scenes, human-computer interaction scenes, or autonomous driving scenes, etc. This application does not make specific limitations on this.

[0030] Step S2: Acquire three-dimensional data of the face region of interest from the light field image data.

[0031] It should be noted that, in order to improve the image quality and ensure the accuracy of face posture detection based on the light field camera in this embodiment, a series of processing is first performed on each frame of light field image data collected, specifically including: Bayer conversion, which is used to obtain color light field image data; filtering processing, which is used to filter and denoise the image data and remove high-frequency noise in low-texture areas; vignetting correction, which is used to remove dark corners of the image data and compensate for lens edges.

[0032] Optionally, step S2 in the face posture detection method based on a light field camera in this embodiment specifically includes: Step S21: extracting sub-aperture image data of different viewing angles from the light field image data; Step S22: performing detection processing on the sub-aperture image data of different viewing angles to obtain a candidate face region of interest; Step S23: acquiring three-dimensional data of the human face region of interest from the light field image data based on the candidate human face region of interest.

[0033] It should be noted that it is difficult to formally represent four-dimensional light field data in a three-dimensional world. Usually, the light field is visualized by fixing any two dimensions of the dual-plane light field model and displaying two-dimensional slices. In this embodiment, light field data in the form of a sub-aperture image array can be obtained by fixing the angle dimension. For example, for a microlens array light field camera: the pixel values ​​at the center of each microlens are extracted in sequence and reassembled into a new image in sequence, and the central sub-aperture image is obtained; similarly, all microlenses are traversed in sequence, and the coordinates of a certain angle are inferred based on the coordinate points of the center of the microlens, and the corresponding pixel values ​​are extracted, and a new image can be reassembled, which is the sub-aperture image of the angle.

[0034] It should be noted that in order to ensure the accuracy of face posture detection based on the light field camera while improving the data processing speed, in this embodiment, sub-aperture images of several angles including the central viewing angle can be selected for extraction, for example: central sub-aperture image, 30° sub-aperture image, 45° sub-aperture image, 60° sub-aperture image, 135° sub-aperture image, -30° sub-aperture image, -45° sub-aperture image, -60° sub-aperture image and -135° sub-aperture image, etc., and this application does not make any specific limitations on this.

[0035] Preferably, the sub-aperture image data of different viewing angles in the face posture detection method based on a light field camera of this embodiment includes a -135° sub-aperture image, a -45° sub-aperture image, a center sub-aperture image, a 45° sub-aperture image and a 135° sub-aperture image.

[0036] In this embodiment, by extracting sub-aperture image data of different viewing angles, the missing of two-dimensional face image data caused by factors such as illumination changes, posture changes, and occlusion in uncontrolled scenes is avoided.

[0037] It should be noted that the sub-aperture images of different viewing angles extracted in step S21 in this embodiment are all two-dimensional images, and a traditional image processing algorithm or a pre-trained neural network model can be used to detect the face region of interest, and this application does not make specific limitations on this. The candidate face region of interest obtained in this embodiment can be a two-dimensional rectangular frame, and of course it can also be a framed region of other shapes, which will not be listed one by one in this application.

[0038] Optionally, step S22 in the face posture detection method based on a light field camera in this embodiment specifically includes: Using a preset two-dimensional face detection model to detect and process sub-aperture image data of different viewing angles to obtain multiple rectangular face regions; Multiple face rectangular regions are merged to obtain candidate face regions of interest.

[0039] It should be noted that, in this embodiment, the preset two-dimensional face detection model may be a pre-trained ‌YOLO model (You Only Look Once), such as YOLOv1, YOLOv2, YOLOv3 or YOLOv5, and this application does not make any specific limitation on this.

[0040] In this embodiment, multiple rectangular face regions may be merged, and their union may be taken to obtain a candidate face region of interest.

[0041] Optionally, in the face posture detection method based on a light field camera of this embodiment, after merging multiple face rectangular regions to obtain the candidate face region of interest, the method further includes: The preset two-dimensional face feature algorithm library is used to extract features of the candidate face interest region to obtain multiple two-dimensional face feature data.

[0042] It should be noted that the preset two-dimensional face feature algorithm library in this embodiment may be a dlib algorithm library, and of course may be other algorithm libraries, which are not listed one by one in this application.

[0043] For an example, see Figure 2 As shown in the figure, firstly, the YOLO model is used to perform face detection on the -135° sub-aperture image, -45° sub-aperture image, center sub-aperture image, 45° sub-aperture image and 135° sub-aperture image, respectively, to obtain the corresponding five rectangular face regions; then, the five rectangular face regions are merged to obtain the candidate face region of interest (i.e., face ROI region), and finally, the dlib algorithm library is used to extract features from the candidate face region of interest to obtain two-dimensional face feature data.

[0044] In the case of occlusion of the face area, multiple face rectangular areas can be obtained through sub-aperture images of multiple perspectives. Each face rectangular area contains different face areas due to different perspectives. By merging multiple face rectangular areas, they can complement each other and reduce or eliminate the impact of occlusion. In the case of changes in illumination of the face area, multiple face rectangular areas can be obtained through sub-aperture images of multiple perspectives. Each face rectangular area contains different shadow areas of the face area due to different perspectives. By merging multiple face rectangular areas, they can complement each other and reduce or eliminate the impact of shadows.

[0045] Step S23 of this embodiment is to calculate the corresponding three-dimensional data of the human face region of interest (ie, three-dimensional point cloud data) from the light field image data based on the candidate human face region of interest.

[0046] Optionally, step S23 in the face posture detection method based on a light field camera of this embodiment specifically includes: transforming the candidate face region of interest in the central sub-aperture coordinate system to the light field coordinate system to obtain the light field region of interest; The light field region of interest is processed to obtain three-dimensional data of the face region of interest.

[0047] Step S3: Detect and process the three-dimensional data of the face region of interest to obtain information of multiple three-dimensional feature points of the face.

[0048] It should be noted that, in this embodiment, the three-dimensional data of the face region of interest can be processed by a pre-trained three-dimensional face detection model to obtain information of multiple three-dimensional feature points of the face.

[0049] Optionally, step S3 in the face posture detection method based on light field camera of this embodiment is specifically: processing the three-dimensional data of the face region of interest according to multiple two-dimensional face feature data to obtain multiple three-dimensional face feature point information.

[0050] It should be noted that in step S3 of this embodiment, the two-dimensional facial feature data and the three-dimensional data of the facial area of ​​interest can be processed by a pre-trained 3D face model to directly obtain the three-dimensional facial feature point information; it is also possible to first quickly determine the corresponding three-dimensional facial feature point coordinate data in the three-dimensional data of the facial area of ​​interest based on the two-dimensional facial feature data, and then use a traditional facial feature processing algorithm or a pre-trained neural network model to process the three-dimensional facial feature point coordinate data to obtain the final three-dimensional facial feature point information.

[0051] Preferably, step S3 in the face posture detection method based on a light field camera of this embodiment specifically includes: Calculating and processing the three-dimensional data of the face region of interest based on the two-dimensional face feature data to obtain three-dimensional feature point coordinate data, three-dimensional feature point normals, three-dimensional feature point first-order gradient data and / or three-dimensional feature point second-order gradient data; A preset three-dimensional facial feature model is used to process three-dimensional feature point coordinate data, three-dimensional feature point normals, three-dimensional feature point first-order gradient data and / or three-dimensional feature point second-order gradient data to obtain three-dimensional facial feature point information.

[0052] It should be noted that the characteristic of light field is that it not only collects information on light intensity, but also collects information on the direction of light. Three-dimensional information can be obtained through ray tracing. The extraction of three-dimensional facial features can provide richer information than two-dimensional facial features, thus making face detection results more reliable.

[0053] It should be noted that the preset three-dimensional facial feature model in this embodiment can be an ANN model, and of course it can also be other models, which are not listed one by one in this application.

[0054] For an example, see Figure 3 As shown, firstly, based on the two-dimensional facial feature data (i.e., 2D facial feature points), the three-dimensional feature point coordinate data (i.e., the feature point 3D coordinates) is extracted from the three-dimensional data (i.e., 3D data) of the facial region of interest; then, a simple gradient calculation can be performed to obtain the first-order gradient data of the three-dimensional feature points (i.e., the first-order gradient of the feature points) and the second-order gradient data of the three-dimensional feature points (i.e., the first-order gradient of the feature points), and the three-dimensional feature point normal can also be calculated; finally, the above data is input into the ANN model for fitting processing to obtain the three-dimensional feature point information of the face (i.e., the 3D facial feature vector).

[0055] Step S4: Bringing information of multiple facial three-dimensional feature points into the facial posture calculation model for solving to obtain a facial posture matrix.

[0056] Optionally, in the face posture detection method based on a light field camera in this embodiment, the three-dimensional feature point information of the face includes feature point numbers and feature point normals; wherein different feature point numbers represent different positions of the face.

[0057] Optionally, the face posture calculation model in the face posture detection method based on the light field camera in this embodiment is expressed as: (1) in, Represents the normal of the feature point numbered i, where i can be 1, 2, 3, ... n. represents the rotation matrix, represents the translation vector, represents the angle; By solving the problem that the normal direction of n feature points is consistent with the Z-axis direction of the light field coordinate system, the rotation matrix and translation matrix are obtained to form the face posture matrix.

[0058] Optionally, before step S1, the method for detecting facial posture based on a light field camera in this embodiment further includes: Step S0: performing distortion correction on the acquired light field image data using the center point distortion correction parameters.

[0059] It should be noted that in this embodiment, for the microlens array light field camera, the central point distortion correction parameters are used to perform distortion correction on the acquired light field image data.

[0060] Preferably, the process of acquiring the center point distortion correction parameters in the face posture detection method based on the light field camera of this embodiment is: White image data is obtained by photographing a white scene with a light field camera; Locate and obtain the center point coordinate data of each microlens from the white image data; The center point distortion correction parameters are calculated based on the center point coordinate data of each microlens, the distortion model and the energy equation.

[0061] Optionally, the distortion model in the face posture detection method based on the light field camera of this embodiment is expressed as: (2) (3) The energy equation E is expressed as: (4) Among them, x, y are the horizontal and vertical coordinates of the original image; x d , y d is the horizontal coordinate and vertical coordinate of the center point of the microlens in the original image; u , y u is the horizontal coordinate and vertical coordinate of the center point of the microlens after correction; k1, k2 are radial distortion coefficients; p1, p2 are tangential distortion coefficients; r is the distance from the original pixel to the distortion center; x mla , y mla are the standard abscissa and ordinate of the center point of the microlens.

[0062] It should be noted that traditional image correction performs distortion correction by photographing a calibration plate, and the general reprojection error can reach 0.2 pixels. If the magnification of the distance is taken into account, the correction accuracy is generally tens or hundreds of microns. In this embodiment, the center point of the white image is used to correct the light field image, and the ultra-high precision of microlens processing is utilized. Its accuracy is generally at the nanometer level, so it can significantly improve the distortion correction accuracy.

[0063] The following is a specific implementation example of a microlens array light field camera to specifically illustrate the face posture detection method based on a light field camera of this embodiment: 1) Calculation of center point distortion correction parameters; Capture white images: A light field camera captures a white scene, such as a piece of white paper or a white wall, to obtain a white image with relatively uniform brightness.

[0064] Center point extraction: Locate the center point of each microlens by finding the circular area in the white image.

[0065] Calculate distortion correction parameters: Due to camera distortion, the arrangement of the center points of the white image is not consistent with the arrangement of the center points of the microlenses. Figure 4As shown, green is the center point of the microlens, and red is the center point of the white image detection. Due to lens distortion, the center point deviates significantly at the four corners far from the center. According to the above distortion models (2), (3) and the constructed energy equation (4), the center point distortion correction parameters k1, k2, p1, p2 are obtained through a nonlinear optimization algorithm. The center point distortion correction parameters are applied to the original light field image to obtain distortion-free light field data.

[0066] 2) Obtain light field data: Bayer conversion: Each frame of light field data collected is first converted by Bayer to obtain a color image; Filtering: Use bilateral filtering algorithm to remove high-frequency noise in low-texture areas of light field images; Distortion correction: The light field image (such as Figure 5 a) is used to perform distortion correction, and the corrected image is a distortion-free image (as shown in Figure 5 b); Vignetting Correction: Distortion-free images (such as Figure 6 a) is vignetted to remove dark corners and compensate for lens edges (as shown in Figure 6 b).

[0067] 3) Image data preprocessing: By detecting sub-aperture images of different viewing angles, the candidate face regions of interest and two-dimensional face feature data are obtained. Preprocessing can eliminate most of the data that does not contain the face area, thereby saving solution time and achieving a higher calculation frame rate.

[0068] 4) 3D solution and face detection: Region transformation: transform the candidate face region of interest from the central sub-aperture coordinate system to the original light field coordinate system; Light field solution: solve the 3D data of the candidate face interest area corresponding to the original light field, that is, obtain the 3D data of the face interest area, such as Figure 7 As shown; 3D face detection and feature extraction: The 3D data of the face area of ​​interest and the 2D face feature data are used as input, and the 3D feature point information of the face (i.e. the 3D face feature vector) is obtained after being processed by the 3D face model. Figure 8 As shown, the three-dimensional face feature vector is a multidimensional vector representing face features in 3D space and is used to identify different faces.

[0069] 5) Face pose calculation Calculate the feature point normal based on the 3D feature point information of the face, such as Fig. 9As shown. The feature point information includes the feature point number and the feature point normal. Different feature point numbers represent different positions of the face. According to the above face posture calculation model, that is, the posture is calculated by formula (1), and the direction of the normal of the n feature points is consistent with the Z axis direction to obtain the face posture matrix.

[0070] The face posture detection method based on the light field camera of the embodiment of the present application first uses the light field camera to collect scene images from multiple perspectives at the same time through a single exposure; then obtains the three-dimensional data of the face interest area from the light field image data of the multi-perspective; then detects and processes the three-dimensional data of the face interest area to obtain multiple three-dimensional feature point information of the face; finally, the multiple three-dimensional feature point information of the face is brought into the face posture calculation model for solution to obtain the face posture matrix. The face posture detection method based on the light field camera of the embodiment of the present application is based on the multi-perspective images collected by the light field camera. Since the occluded area may be visible in some perspectives, the problem of the face being unable to be detected due to partial occlusion can be solved. In addition, the shadow distribution under different perspectives is different and the face posture is different. Through multi-perspective integration, the influence of illumination changes and posture changes on face detection can be reduced; the accuracy of face posture detection is improved through the three-dimensional feature point information of the face. In summary, the face posture detection method based on the light field camera of the embodiment of the present application has strong environmental adaptability and can effectively improve the accuracy of face posture detection.

[0071] Embodiment 2: The present application embodiment provides a face posture detection system based on a light field camera, see Fig.10 As shown, the system 1 comprises: The light field camera 10 is configured to capture scene images to obtain light field image data; The acquisition module 20 is configured to acquire three-dimensional data of a human face region of interest from the light field image data; The processing module 30 is configured to detect and process the three-dimensional data of the face region of interest to obtain information of a plurality of three-dimensional feature points of the face; The solving module 40 is configured to bring information of multiple three-dimensional facial feature points into the facial posture calculation model for solving to obtain a facial posture matrix.

[0072] Optionally, the three-dimensional feature point information of the face in the face posture detection system based on the light field camera of this embodiment includes feature point numbers and feature point normals; wherein different feature point numbers represent different positions of the face.

[0073] Optionally, the face posture calculation model in the face posture detection system based on the light field camera of this embodiment is expressed as: (1) in, Represents the normal of the feature point numbered i, where the value of i is 1, 2, 3, ... n, i represents the rotation matrix, represents the translation vector, represents the angle; By solving the problem that the normal direction of n feature points is consistent with the Z-axis direction of the light field coordinate system, the rotation matrix and translation matrix are obtained to form the face posture matrix.

[0074] Optionally, the acquisition module 20 in the face posture detection system based on the light field camera of this embodiment specifically includes: An extraction unit is configured to extract sub-aperture image data of different viewing angles from the light field image data; A first processing unit is configured to detect and process the sub-aperture image data of different viewing angles to obtain a candidate face region of interest; The acquisition unit is configured to acquire three-dimensional data of the human face region of interest from the light field image data based on the candidate human face region of interest.

[0075] Preferably, the sub-aperture image data of different viewing angles in the face posture detection system based on the light field camera of this embodiment includes a -135° sub-aperture image, a -45° sub-aperture image, a center sub-aperture image, a 45° sub-aperture image and a 135° sub-aperture image.

[0076] Optionally, the first processing unit in the face posture detection system based on the light field camera of this embodiment is specifically configured to: use a preset two-dimensional face detection model to detect and process sub-aperture image data of different viewing angles to obtain multiple face rectangular areas; merge the multiple face rectangular areas to obtain a candidate face region of interest.

[0077] Optionally, the acquisition unit in the face posture detection system based on the light field camera of this embodiment is specifically configured to: transform the candidate face region of interest in the central sub-aperture coordinate system to the light field coordinate system to obtain the light field region of interest; and perform solution processing on the light field region of interest to obtain three-dimensional data of the face region of interest.

[0078] Optionally, the first processing unit in the face gesture detection system based on the light field camera of this embodiment is further configured to: extract features of the candidate face interest region using a preset two-dimensional face feature algorithm library to obtain a plurality of two-dimensional face feature data; The processing module 30 is specifically configured to process the three-dimensional data of the face region of interest according to the multiple two-dimensional face feature data to obtain multiple three-dimensional face feature point information.

[0079] Optionally, the processing module 30 in the face posture detection system based on the light field camera of this embodiment specifically includes: The second processing unit is configured to calculate and process the three-dimensional data of the face region of interest according to the two-dimensional face feature data to obtain three-dimensional feature point coordinate data, three-dimensional feature point normals, three-dimensional feature point first-order gradient data and / or three-dimensional feature point second-order gradient data; The third processing unit is configured to use a preset three-dimensional facial feature model to process the three-dimensional feature point coordinate data, the three-dimensional feature point normal, the three-dimensional feature point first-order gradient data and / or the three-dimensional feature point second-order gradient data to obtain the three-dimensional feature point information of the face.

[0080] Optionally, the face posture detection system 1 based on a light field camera of this embodiment further includes: a correction module configured to perform distortion correction on the acquired light field image data using a center point distortion correction parameter.

[0081] Optionally, the face posture detection system 1 based on the light field camera of this embodiment also includes a correction parameter acquisition module, which is configured to: obtain white image data by shooting a white scene with a light field camera; obtain the center point coordinate data of each microlens from the white image data; and obtain the center point distortion correction parameter based on the center point coordinate data of each microlens, the distortion model and the energy equation.

[0082] It is not difficult to find that the face posture detection system based on the light field camera of this embodiment is a system embodiment corresponding to the first embodiment, and this embodiment can be implemented in conjunction with the first embodiment. The relevant technical details mentioned in the first embodiment are still valid in this embodiment, and in order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the first embodiment.

[0083] It is worth mentioning that all modules involved in this embodiment are logic modules. In practical applications, a logic unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, in order to highlight the innovative part of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed by this application, but this does not mean that there are no other units in this embodiment.

[0084] Embodiment three: The present application discloses an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of a method for detecting human face posture based on a light field camera in any one of the first embodiments disclosed in the present application are implemented.

[0085] Fig.11 is a structural diagram of an electronic device according to an embodiment of the present application, such as Fig.11As shown, the electronic device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, near field communication (NFC) or other technologies. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the housing of the electronic device, or an external keyboard, touchpad or mouse, etc.

[0086] Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a structural diagram of the part related to the technical solution of the present disclosure, and does not constitute a limitation on the electronic device to which the technical solution of the present application is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0087] Embodiment 4: The embodiment of the present application discloses a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any one of the face posture detection methods based on a light field camera in the first embodiment disclosed in the present application are implemented.

[0088] Please note that the technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. The above embodiments only express several implementation methods of the present application, and their descriptions are relatively specific and detailed, but they cannot be understood as limiting the scope of the patent application. It should be pointed out that for ordinary technicians in this field, without departing from the concept of the present application, several variations and improvements can be made, which all belong to the scope of protection of the present application. Therefore, the scope of protection of the patent application of this application shall be based on the attached claims.

[0089] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

[0090] Although some specific embodiments of the present application have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are only for illustration, not for limiting the scope of the present application. It should be understood by those skilled in the art that the above embodiments may be modified without departing from the scope and spirit of the present application. The scope of the present application is defined by the appended claims.

Claims

1. A method for detecting face posture based on a light field camera, characterized in that: The method comprises: Using a light field camera to collect scene images to obtain light field image data; Acquire three-dimensional data of a human face region of interest from the light field image data; Detecting and processing the three-dimensional data of the face region of interest to obtain information of a plurality of three-dimensional feature points of the face; The information of multiple facial 3D feature points is brought into the facial posture calculation model for solution to obtain the facial posture matrix.

2. The method for detecting face posture based on a light field camera according to claim 1, characterized in that: The three-dimensional feature point information of the face includes feature point numbers and feature point normals; wherein different feature point numbers represent different positions of the face.

3. The method for detecting face posture based on a light field camera according to claim 2, characterized in that: The face posture calculation model is expressed as: （1） in, Represents the normal of the feature point numbered i, where i can be 1, 2, 3, ... n. represents the rotation matrix, represents the translation vector, represents the angle; The rotation matrix and the translation matrix are obtained by solving the problem that the normal direction of the n feature points is consistent with the Z-axis direction of the light field coordinate system to form the face posture matrix.

4. The method for detecting face posture based on a light field camera according to claim 1, characterized in that: The step of acquiring the three-dimensional data of the face region of interest from the light field image data specifically includes: Extracting sub-aperture image data of different viewing angles from the light field image data; Detect and process the sub-aperture image data of different viewing angles to obtain the candidate face region of interest; Based on the candidate human face region of interest, three-dimensional data of the human face region of interest is acquired from the light field image data.

5. The method for detecting face posture based on a light field camera according to claim 4, characterized in that: The sub-aperture image data of different viewing angles are detected and processed to obtain the candidate face region of interest, including: Using a preset two-dimensional face detection model to detect and process sub-aperture image data of different viewing angles to obtain multiple rectangular face regions; A plurality of face rectangular regions are merged to obtain the candidate face region of interest.

6. The method for detecting face posture based on a light field camera according to claim 4, characterized in that: The acquiring three-dimensional data of the face region of interest from the light field image data based on the candidate face region of interest specifically includes: Transforming the candidate face region of interest in the central sub-aperture coordinate system into the light field coordinate system to obtain the light field region of interest; The light field region of interest is processed to obtain three-dimensional data of the face region of interest.

7. The method for detecting face posture based on a light field camera according to claim 5, characterized in that: After merging the plurality of face rectangular regions to obtain the candidate face region of interest, the method further includes: Using a preset two-dimensional face feature algorithm library to extract features from the candidate face region of interest to obtain a plurality of two-dimensional face feature data; The detecting and processing of the three-dimensional data of the face region of interest to obtain multiple three-dimensional face feature point information is specifically: processing the three-dimensional data of the face region of interest based on multiple two-dimensional face feature data to obtain multiple three-dimensional face feature point information.

8. A face posture detection system based on a light field camera, characterized in that: The system comprises: The light field camera is configured to capture a scene image to obtain light field image data; An acquisition module is configured to acquire three-dimensional data of a human face region of interest from the light field image data; A processing module is configured to detect and process the three-dimensional data of the face region of interest to obtain information of a plurality of three-dimensional feature points of the face; The solution module is configured to bring information of multiple three-dimensional facial feature points into the facial posture calculation model for solution to obtain a facial posture matrix.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the steps in the face posture detection method based on a light field camera described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps in the method for detecting facial posture based on a light field camera according to any one of claims 1 to 7 are implemented.

Images