Face data processing method, device, electronic device and storage medium

Abstract

In the embodiments of the present application, a method, device, electronic device and storage medium for processing facial data are provided to solve the problem that facial recognition in the related art is easily affected by the environment and has low efficiency. In the embodiments of the present application, three-dimensional point cloud data of the face of the target object is collected for facial recognition, and the three-dimensional features of the face can be obtained. Then the three-dimensional point cloud data is mapped to a two-dimensional plane, and a plane image of the target object can be obtained. Afterwards, facial recognition is performed based on the plane image. In the embodiments of the present application, the extracted two-dimensional plane image is converted from the three-dimensional point cloud data, is not affected by ambient light, and can also extract the facial features. In the face recognition stage, face recognition through plane images can also improve the efficiency of face recognition. Therefore, in the embodiments of the present application, a face recognition method that can well balance efficiency and accuracy is provided.

Description

Background Art

[0002] The development of face recognition technology has brought a lot of convenience to people's life and study. Face recognition technology can be used to complete corresponding business in many scenarios, such as attendance clocking in, smart door lock control and security monitoring.

[0003] However, the accuracy of face recognition is easily affected by the environment and various factors. For example, in a dim environment, if the face image is not collected clearly, it will lead to the inability to recognize the face, or if the facial appearance features of people are changed due to factors such as makeup, it is easy to cause recognition failure. Therefore, how to further improve the accuracy of face recognition still needs to be solved. Summary of the invention

[0004] The purpose of this application is to provide a method, device, electronic device and storage medium for processing face data, which are used to solve the problem in related technologies that the accuracy of face recognition needs to be further improved.

[0005] In a first aspect, an embodiment of the present application provides a method for processing face data, including:

[0006] Obtaining three-dimensional point cloud data of the target object's face;

[0007] Mapping the three-dimensional point cloud data to a two-dimensional plane to obtain a plane image of the target object;

[0008] Perform face recognition on the target object based on the planar image.

[0009] In some embodiments, performing face recognition on the target object based on the planar image includes:

[0010] Performing facial region recognition from the planar image to obtain facial information, wherein the facial information includes the facial region and the positions of facial key points;

[0011] The face information is used to perform face classification processing to determine whether the target object is a known object.

[0012] In some embodiments, mapping the three-dimensional point cloud data to a two-dimensional plane to obtain a planar image of the target object includes:

[0013] Dividing the three-dimensional point cloud data into a plurality of grids;

[0014] The three-dimensional point cloud data in each of the grids is normalized to a specified value space to obtain the plane image.

[0015] In some embodiments, normalizing the three-dimensional point cloud data in each of the grids to a specified value space includes:

[0016] For each of the grids, execute:

[0017] Assigning a weight to the grid based on the position information and height information of the point cloud data within the grid, wherein the weight has a positive correlation with the height information, and the weight is higher when the position of the point cloud data is closer to the central area;

[0018] The weights of the grid are normalized to obtain pixel points corresponding to the grid in the plane image.

[0019] In some embodiments, dividing the three-dimensional point cloud data into a plurality of grids includes:

[0020] The three-dimensional point cloud data is divided in length and width directions according to the size of each grid to obtain the multiple grids.

[0021] In some embodiments, the step of performing facial region recognition from the planar image to obtain facial information includes:

[0022] Using a first neural network to identify a plurality of first candidate face frames from the planar image;

[0023] Using a second neural network to score each of the first candidate human face frames to obtain a score value for each of the first candidate human face frames; and, based on the score value, selecting a second candidate human face frame from the plurality of first candidate human face frames;

[0024] A third neural network is used to process and analyze the sub-image corresponding to the second candidate face frame to obtain face information.

[0025] In some embodiments, the using the face information to perform face classification processing to determine whether the target object is a known object includes:

[0026] Extracting features of the target object from the plane image according to the face area and the positions of the key points of the face in the face information to obtain the face features of the target object;

[0027] Performing classification and recognition based on the facial features to obtain the facial category of the target object;

[0028] If the face category is a known target object, determining that the target object is a known object;

[0029] If the face category is an unknown target object, then the target object is determined to be an unknown object.

[0030] In some embodiments, the method further comprises:

[0031] If the target object is a known object, the smart door lock is controlled to perform an unlocking operation.

[0032] In a second aspect, the present application further provides a facial data processing device, the device comprising:

[0033] A point cloud data acquisition module is used to acquire three-dimensional point cloud data of the face of the target object;

[0034] A plane image acquisition module, used for mapping the three-dimensional point cloud data to a two-dimensional plane to obtain a plane image of the target object;

[0035] A face recognition module is used to perform face recognition on the target object based on the plane image.

[0036] In some embodiments, the face recognition module is used to:

[0037] Performing facial region recognition from the planar image to obtain facial information, wherein the facial information includes the facial region and the positions of facial key points;

[0038] The face information is used to perform face classification processing to determine whether the target object is a known object.

[0039] In some embodiments, the planar image acquisition module is used to:

[0040] Dividing the three-dimensional point cloud data into a plurality of grids;

[0041] The three-dimensional point cloud data in each of the grids is normalized to a specified value space to obtain the plane image.

[0042] In some embodiments, the three-dimensional point cloud data in each of the grids is normalized to a specified value space, and the plane image acquisition module is used to:

[0043] For each of the grids, execute:

[0044] Assigning a weight to the grid based on the position information and height information of the point cloud data within the grid, wherein the weight has a positive correlation with the height information, and the weight is higher when the position of the point cloud data is closer to the central area;

[0045] The weights of the grid are normalized to obtain pixel points corresponding to the grid in the plane image.

[0046] In some embodiments, the three-dimensional point cloud data is divided into a plurality of grids, and the plane image acquisition module is used to: divide the three-dimensional point cloud data in length and width directions according to the size of each grid to obtain the plurality of grids.

[0047] In some embodiments, when performing face area recognition from the planar image to obtain face information, the face recognition module is used to:

[0048] Using a first neural network to identify a plurality of first candidate face frames from the planar image;

[0049] Using a second neural network to score each of the first candidate human face frames to obtain a score value for each of the first candidate human face frames; and, based on the score value, selecting a second candidate human face frame from the plurality of first candidate human face frames;

[0050] A third neural network is used to process and analyze the sub-image corresponding to the second candidate face frame to obtain face information.

[0051] In some embodiments, when performing face classification processing using the face information to determine whether the target object is a known object, the face recognition module is used to:

[0052] Extracting features of the target object from the plane image according to the face area and the positions of the key points of the face in the face information to obtain the face features of the target object;

[0053] Performing classification and recognition based on the facial features to obtain the facial category of the target object;

[0054] If the face category is a known target object, determining that the target object is a known object;

[0055] If the face category is an unknown target object, then the target object is determined to be an unknown object.

[0056] In some embodiments, the apparatus further comprises:

[0057] The door lock control module is used to control the smart door lock to perform an unlocking operation if the target object is a known object.

[0058] In the third aspect, another embodiment of the present application also provides an electronic device, comprising at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any facial data processing method provided in the embodiment of the present application.

[0059] In a fourth aspect, another embodiment of the present application further provides a computer storage medium, wherein the computer storage medium stores a computer program, and the computer program is used to enable a computer to execute any facial data processing method in the embodiments of the present application.

[0060] In an embodiment of the present application, the three-dimensional point cloud data of the target object's face is collected for face recognition, and the three-dimensional features of the face can be obtained. The three-dimensional point cloud data is then mapped to a two-dimensional plane, and a plane image of the target object can be obtained. Afterwards, face recognition is performed based on the plane image. In an embodiment of the present application, the extracted two-dimensional plane image is converted from the three-dimensional point cloud data, is not affected by ambient light, and can also extract the facial features. In the face recognition stage, face recognition through plane images can also improve the efficiency of face recognition. Therefore, an embodiment of the present application provides a face recognition method that can well balance efficiency and accuracy.

[0061] In the embodiments of the present application, other features and advantages of the present application will be described in the subsequent description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. Obviously, the drawings introduced below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0063] Figure 1 is a schematic diagram of an application environment according to an embodiment of the present application;

[0064] Figure 2 A schematic diagram of a process flow for processing face data according to an embodiment of the present application;

[0065] Figure 3a A diagram of a door lock system according to an embodiment of the present application;

[0066] Figure 3b A schematic diagram of a process flow for processing face data according to an embodiment of the present application;

[0067] Figure 4 A schematic diagram of a device for processing face data according to an embodiment of the present application;

[0068] Figure 5 is a schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0069] In order to enable ordinary persons in the art to better understand the technical solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0070] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0071] In view of the fact that face recognition in related technologies is easily affected by the environment and the makeup factors of the identified object, a method for processing face data is provided in an embodiment of the present application to improve the accuracy of face recognition.

[0072] In this application, the acquisition and use of facial data strictly comply with relevant laws and regulations. Before collecting facial data, users have been clearly informed of the purpose, method, and scope of data collection. Users can understand the specific content and use of data collection by reading the detailed privacy policy and user agreement. After fully understanding the relevant information, users must voluntarily authorize the collection and use of their facial data by clicking the "Agree" button or other clear methods. The use of facial data is limited to the scope of user authorization and is mainly used for legitimate purposes such as identity authentication and personalized services. Facial data will not be used for any illegal or unauthorized activities. The use of all facial data complies with national and regional laws and regulations.

[0073] In an embodiment of the present application, the three-dimensional point cloud data of the target object's face is collected for face recognition, and the three-dimensional features of the face can be obtained. The three-dimensional point cloud data is then mapped to a two-dimensional plane, and a plane image of the target object can be obtained. Afterwards, face recognition is performed based on the plane image. In an embodiment of the present application, the extracted two-dimensional plane image is converted from the three-dimensional point cloud data, is not affected by ambient light, and can also extract the facial features. In the face recognition stage, face recognition through plane images can also improve the efficiency of face recognition. Therefore, an embodiment of the present application provides a face recognition method that can well balance efficiency and accuracy.

[0074] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0075] Taking smart door lock control as an example, the processing method of face data in the embodiment of the present application is explained. Traditional smart door lock control may include fingerprint unlocking and password unlocking. However, these unlocking methods are not suitable for the elderly and children. Face recognition can unlock without the user performing any operation, so it has greater potential for popularization. However, due to the influence of factors such as light and makeup, the low accuracy of face recognition affects the application and popularization of smart door locks. Therefore, the face data processing method provided by the implementation of this application can take into account both efficiency and accuracy, and when applied in smart door locks, face recognition can be conveniently performed to control smart door locks.

[0076] In the embodiments of the present application, it is not only suitable for smart door lock control, but can be used in any application scenario that requires face recognition.

[0077] Figure 1 It is a schematic diagram of an application scenario according to an embodiment of the present application.

[0078] like Figure 1 As shown, the application scenario may include, for example, a storage system 10, a server 20, and a terminal device 30 (for example, 30_1 and 30_2 or 30_N). The terminal device 30 may be used for any suitable electronic device for network access, including but not limited to a smart door lock, a computer, a laptop, a smart phone, a tablet computer, a smart watch, a smart bracelet, or other types of terminals. The storage system 10 is capable of storing feature information of known objects, etc. Of course, the feature information of each user in the smart door lock can be stored in the smart door lock to facilitate door lock control. The server 20 is used to implement interaction with the terminal device 30 and return it to the terminal device 30.

[0079] During implementation, taking a smart door lock as an example, the smart door lock can collect 3D point cloud data of a face, and the smart door lock can analyze the collected 3D point cloud data to obtain a face recognition result, or the smart door lock can send the 3D point cloud data to the server 20, and the server 20 performs face recognition on the 3D point cloud data, and returns the recognition result to the smart door lock. If a known object is recognized, that is, an object that can open the smart door lock is recognized, the smart door lock is controlled to unlock, otherwise the smart door lock may prompt that it cannot be unlocked.

[0080] It should be noted that in the embodiment of the present application, the processing of facial data can be performed by the terminal device 30, or by the server 20, or partly by the terminal device 30 and partly by the server 20, and the present application does not limit this.

[0081] The terminal devices 30 (for example, 30_1 and 30_2 or 30_N) can also communicate with each other via the network 40. For example, the smart door lock can prompt the bound mobile terminal when the door is successfully unlocked, and the unlocking record of the smart door lock can also be exported and viewed on the mobile terminal.

[0082] The network 40 may be a network used for information transmission in a broad sense, and may include one or more communication networks, such as a wireless communication network, the Internet, a private area network, a local area network, a metropolitan area network, a wide area network, or a cellular data network.

[0083] The description in this application only details a single server or terminal device, but those skilled in the art should understand that the single server 20, terminal device 30 and storage system 10 shown are intended to indicate that the technical solution of this application involves the operation of the terminal device, server and storage system. The single terminal device and the single server and storage system are described in detail at least for the convenience of explanation, and do not imply any limitation on the number, type or location of the terminal devices and servers. It should be noted that adding additional modules to the illustrated environment or removing individual modules from it does not change the underlying concepts of the example embodiments of this application. In addition, although for the convenience of explanation, Figure 1 A bidirectional arrow from the storage system 10 to the server 20 is shown in the figure, but those skilled in the art will appreciate that the sending and receiving of the above data may also be achieved through the network 40.

[0084] The server 20 may be a single server, a server cluster consisting of several servers, or a cloud computing center. The server 20 may be an independent physical server, or a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0085] like Figure 2 As shown in the figure, a method for processing face data provided in an embodiment of the present application may include the following steps:

[0086] A three-dimensional image is a special form of information expression that expresses data in three dimensions in space. Compared with two-dimensional images, three-dimensional images can achieve natural object-background decoupling with the help of information in the third dimension. In addition, for visual measurement, the two-dimensional information of an object often changes with the projection method, but its three-dimensional features have better uniformity for different measurement methods. The representation forms of three-dimensional images include: depth map (expressing the distance between the object and the camera in grayscale), geometric model, point cloud model (all reverse engineering equipment samples the object into a point cloud). It can be seen that point cloud data is the most common and basic three-dimensional model. The point cloud model is often obtained directly by measurement, and each point corresponds to a measurement point without other processing methods, so it contains the largest amount of information. Therefore, in an embodiment of the present application, in step 201, the three-dimensional point cloud data of the face of the target object is obtained. In this way, the influence of ambient light, makeup, etc. on face recognition is avoided.

[0087] Furthermore, in the embodiment of the present application, in order to improve the speed of face recognition, in step 202, the three-dimensional point cloud data is mapped to a two-dimensional plane to obtain a plane image of the target object. Then, in step 203, face recognition is performed on the target object based on the plane image.

[0088] The types of 3D point cloud data include XYZ, XYZC, XYZN, XYZG and many other types. All types include XYZ spatial information. In the 3D point cloud data of a face, XY can represent the plane of the front face, and Z is the height information, indicating the three-dimensional degree of the facial features. For example, the height of the nose tip.

[0089] In order to map the three-dimensional point cloud data to the two-dimensional plane simply and effectively, a rasterization method can be used to obtain a plane image. For example, the three-dimensional point cloud data can be divided into multiple grids; then, the three-dimensional point cloud data in each grid is normalized to a specified value space to obtain the plane image.

[0090] For example, among all the three-dimensional point cloud data of a human face, the three-dimensional point cloud data is divided in length and width directions according to the size of each grid to obtain the multiple grids.

[0091] For example, select the maximum and minimum values ​​of the X-axis and Y-axis, and then select the appropriate pixel interval according to the actual situation (it can be set to about 0.1~0.4). Finally, divide the difference between the maximum and minimum values ​​of the X-axis by the pixel interval to get the number of pixels in the X direction. Similarly, divide the difference between the maximum and minimum values ​​of the Y-axis by the pixel interval to get the number of pixels in the Y direction. Then the degree and length of the plane image can be determined.

[0092] Then, each point in the grid is projected onto a two-dimensional plane to obtain a pixel value, which can improve the mapping from three-dimensional space to two-dimensional space and obtain a planar image of the face.

[0093] By mapping the three-dimensional spatial features of the facial features into a plane as described above, the plane includes the shapes of the facial features, which is sufficient for subsequent face recognition.

[0094] In some embodiments, in order to accurately extract irrelevant information of the face and retain it in a two-dimensional plane. In the embodiment of the present application, the projection from the three-dimensional space to the two-dimensional plane can be realized based on weight allocation. For example, in the three-dimensional point cloud data of the face, the nose tip is the highest point of the face, and the number of point clouds in the area where the nose tip is located is the densest, so the weight of this position is the largest to highlight the features of the nose tip. In addition, in the three-dimensional face data, the nose tip is mostly in the central area. When confirming the weight, it can be first determined based on the Z value of the point (i.e., the point height). The higher the height, the more point clouds there are in the corresponding area, and the corresponding weight of the point is also greater. For points on the same plane (i.e., when Z is equal), it is considered that the closer to the center area, the greater the weight. Therefore, the higher a point is (i.e., the larger the Z value), the greater the weight. When the height is the same, the closer to the center area of ​​the image, the greater the weight. That is, the weight has a positive correlation with the height information, and the weight is higher when the position of the point cloud data is closer to the center area. Based on this, in an embodiment of the present application, the following can be performed on each of the grids: a weight is assigned to the grid based on the position information and height information of the point cloud data within the grid, and then the weight of the grid is normalized to obtain the pixel points corresponding to the grid in the plane image.

[0095] For example, if the pixel value in the plane image is required to range from 0 to 255, then the maximum and minimum weights of all grids are taken during normalization. If the weight of the i-th grid is, then the pixel value of the i-th grid after normalization is the calculation result shown in formula (1):

[0096]

[0097] In order to further improve the accuracy of face recognition, in the embodiment of the present application, face area recognition can be performed from the plane image during face recognition, so as to accurately locate the face area and obtain face information, wherein the face information can at least include the face area and the position of the key points of the face; the key point position can be a small number of positions, such as the positions of two eyes, the tip of the nose and the mouth. Then, the face information is used to perform face classification processing to determine whether the target object is a known object. That is, in the embodiment of the present application, the face recognition adopts a classification recognition method, that is, the input information is classified and processed, and face recognition can be achieved without marking the three-dimensional key points of the face.

[0098] During implementation, the face area can be located based on the MTCNN model, and then InceptionV3 can be used for face classification.

[0099] If face area recognition is performed from the planar image, obtaining face information can be implemented as follows:

[0100] Step A1: using a first neural network to identify a plurality of first candidate face frames from the plane image;

[0101] Among them, a plurality of candidate face frames are initially screened out in the first neural network and then screened using the non-maximum suppression method.

[0102] Step A2: using a second neural network to score each of the first candidate human face frames to obtain a score value of each of the first candidate human face frames; and selecting a second candidate human face frame from the plurality of first candidate human face frames based on the score value;

[0103] Step A3: using a third neural network to process and analyze the sub-image corresponding to the second candidate face frame to obtain face information.

[0104] For example, a planar image is input to the P-Net network, which outputs a series of face boundary candidate frames. The non-maximum suppression method is used to remove some invalid candidate frames, and the remaining candidate frames are the first candidate face frames of the final output of P-Net.

[0105] Then, the plane image is captured based on the first face candidate frame output by P-Net, and the captured sub-image is input into R-Net for further screening. The probability of each input sub-image being a face is calculated as the score value of whether it is a face. The first face candidate frame whose score does not meet the standard is eliminated to obtain the second face candidate frame, so as to achieve the purpose of further screening.

[0106] Finally, the same method as R-Net is used to capture the second face candidate frame from the image to obtain a new sub-image, which is then input into the O-Net network for final screening, and the final output is the face area and key point positions.

[0107] During face recognition, the target object can be feature extracted from the plane image based on the face area and the position of the face key points in the face information to obtain the face features of the target object; then classification and recognition are performed based on the face features to obtain the face category of the target object; if the face category is a known target object, the target object is determined to be a known object; if the face category is an unknown target object, the target object is determined to be an unknown object.

[0108] In the embodiment of the present application, the face recognition result can be applied to the control of the anti-theft door smart door lock. The known object can be a legal object that is allowed to open the smart door lock. Therefore, if the target object is a known object, the smart door lock is controlled to perform the unlocking operation.

[0109] Continuing with the example of smart door lock control, in the embodiment of the present application, Figure 3a This is a schematic diagram of a three-dimensional face recognition anti-theft door lock control system, which includes a face recognition module and a door lock control module. The face recognition module consists of a face information collector and a processor, and the door lock control module consists of a controller and a door lock switch. In actual operation, the user collects facial information through the face information collector, sends the collected facial information to the processor for processing, obtains the recognition result through the face recognition algorithm, and then sends the obtained recognition result to the door lock controller. If the user is a legitimate user, the door lock can be unlocked, that is, the unlocking is successful; on the contrary, if the user is an illegal user, the door lock cannot be unlocked, that is, the unlocking fails.

[0110] The face recognition algorithm used by anti-theft doors is divided into two parts: face detection and face recognition. Figure 3b As shown, in step 301, the user's face three-dimensional point cloud data is collected, and then in step 302, the three-dimensional point cloud data is mapped to a two-dimensional plane to obtain a plane image. In step 303, in the face detection stage, face detection is implemented based on the MTCNN model, which consists of three networks: P-Net, R-Net and O-Net. First, the acquired user three-dimensional point cloud data is preprocessed, and the three-dimensional face data is projected to a two-dimensional plane to obtain a plane image, and then the plane image is input into P-Net to generate a large number of face boundary candidate regions; then the P-Net output result is input into the R-Net network to screen the candidate regions and exclude invalid regions; finally, the result is input into O-Net to further screen the candidate regions and determine the face boundary and the position of facial key points. This process maps the input 3D face data to a 2D plane. After mapping the 3D face data to a 2D plane, the face data mapped to the 2D plane is input to P-Net for training. The data is further trained through the R-Net and O-Net networks to determine the facial feature points. This method only requires the position of the face frame to be calibrated, and there is no need to manually calibrate the 3D face feature points, so it is relatively simple to implement. And the mapped face information is used as the network input, which reduces the network training time and improves the recognition speed.

[0111] In step 304, in the face recognition stage, the Inception V3 neural network model may be used to further identify the three-dimensional face information that has undergone face detection to determine whether the user is a legitimate user.

[0112] It is necessary to establish a face recognition dataset based on the face data previously provided by the user, which contains the face information of legitimate users to facilitate face recognition.

[0113] In actual use, the user's three-dimensional facial information collected by the camera is processed by the detection algorithm and input into the model for judgment, and the facial recognition result is output.

[0114] In step 305, if the user is identified as a legitimate user, the smart door card is controlled to unlock the door.

[0115] Of course, when an illegal user is identified, the smart door lock can be controlled to send a prompt message.

[0116] like Figure 4 As shown, based on the same inventive concept, a facial data processing device 400 is proposed, comprising:

[0117] Point cloud data acquisition module 401, used to acquire three-dimensional point cloud data of the face of the target object;

[0118] A plane image acquisition module 402 is used to map the three-dimensional point cloud data to a two-dimensional plane to obtain a plane image of the target object;

[0119] The face recognition module 403 is used to perform face recognition on the target object based on the planar image.

[0120] In some embodiments, the face recognition module is used to:

[0121] Performing facial region recognition from the planar image to obtain facial information, wherein the facial information includes the facial region and the positions of facial key points;

[0122] The face information is used to perform face classification processing to determine whether the target object is a known object.

[0123] In some embodiments, the planar image acquisition module is used to:

[0124] Dividing the three-dimensional point cloud data into a plurality of grids;

[0125] The three-dimensional point cloud data in each of the grids is normalized to a specified value space to obtain the plane image.

[0126] In some embodiments, the three-dimensional point cloud data in each of the grids is normalized to a specified value space, and the plane image acquisition module is used to:

[0127] For each of the grids, execute:

[0128] Assigning a weight to the grid based on the position information and height information of the point cloud data within the grid, wherein the weight has a positive correlation with the height information, and the weight is higher when the position of the point cloud data is closer to the central area;

[0129] The weights of the grid are normalized to obtain pixel points corresponding to the grid in the plane image.

[0130] In some embodiments, the three-dimensional point cloud data is divided into a plurality of grids, and the plane image acquisition module is used to: divide the three-dimensional point cloud data in length and width directions according to the size of each grid to obtain the plurality of grids.

[0131] In some embodiments, when performing face area recognition from the planar image to obtain face information, the face recognition module is used to:

[0132] Using a first neural network to identify a plurality of first candidate face frames from the planar image;

[0133] Using a second neural network to score each of the first candidate human face frames to obtain a score value for each of the first candidate human face frames; and, based on the score value, selecting a second candidate human face frame from the plurality of first candidate human face frames;

[0134] A third neural network is used to process and analyze the sub-image corresponding to the second candidate face frame to obtain face information.

[0135] In some embodiments, when performing face classification processing using the face information to determine whether the target object is a known object, the face recognition module is used to:

[0136] Extracting features of the target object from the plane image according to the face area and the positions of the key points of the face in the face information to obtain the face features of the target object;

[0137] Performing classification and recognition based on the facial features to obtain the facial category of the target object;

[0138] If the face category is a known target object, determining that the target object is a known object;

[0139] If the face category is an unknown target object, then the target object is determined to be an unknown object.

[0140] In some embodiments, the apparatus further comprises:

[0141] The door lock control module is used to control the smart door lock to perform an unlocking operation if the target object is a known object.

[0142] The implementation and beneficial effects of each operation in the facial data processing device can be found in the description of the method above, which will not be repeated here.

[0143] After introducing the method and device for processing face data according to an exemplary embodiment of the present application, next, an electronic device according to another exemplary embodiment of the present application is introduced.

[0144] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0145] In some possible implementations, the electronic device according to the present application may include at least one processor and at least one memory. The memory stores program code, and when the program code is executed by the processor, the processor executes the steps of the image scaling method according to various exemplary implementations of the present application described above in this specification. For example, the processor may execute the steps in the method for processing face data.

[0146] Refer to the following Figure 5 The electronic device 130 according to this embodiment of the present application is described. Figure 5 The electronic device 130 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0147] like Figure 5 As shown, the electronic device 130 is in the form of a general electronic device. The components of the electronic device 130 may include but are not limited to: the at least one processor 131, the at least one memory 132, and a bus 133 connecting different system components (including the memory 132 and the processor 131).

[0148] Bus 133 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.

[0149] The memory 132 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 1321 and / or a cache memory 1322 , and may further include a read-only memory (ROM) 1323 .

[0150] The memory 132 may also include a program / utility 1325 having a set (at least one) of program modules 1324, such program modules 1324 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0151] The electronic device 130 may also communicate with one or more external devices 134 (e.g., keyboards, pointing devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 130, and / or communicate with any device that enables the electronic device 130 to communicate with one or more other electronic devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 135. In addition, the electronic device 130 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) through a network adapter 136. As shown, the network adapter 136 communicates with other modules for the electronic device 130 through a bus 133. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 130, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0152] In some possible implementations, various aspects of the facial data processing method provided by the present application may also be implemented in the form of a program product, which includes program code. When the program product runs on a computer device, the program code is used to enable the computer device to execute the steps of the facial data processing method according to various exemplary implementations of the present application described above in this specification.

[0153] The program product may adopt any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0154] The program product for image zooming of the embodiment of the present application can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on an electronic device. However, the program product of the present application is not limited to this. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0155] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein readable program code is carried. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium other than a readable storage medium, which may transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0156] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0157] The program code for performing the operation of the present application can be written in any combination of one or more programming languages, including object-oriented programming languages, such as Java, C++, etc., and also conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user electronic device, partially on the user device, as an independent software package, partially on the user electronic device and partially on the remote electronic device, or completely on the remote electronic device or server. In the case of a remote electronic device, the remote electronic device can be connected to the user electronic device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external electronic device (for example, using an Internet service provider to connect through the Internet).

[0158] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0159] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0160] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0161] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable image scaling device to produce a machine, so that the instructions executed by the processor of the computer or other programmable image scaling device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0162] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable image scaling device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, the instruction device being implemented in the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0163] These computer program instructions may also be loaded onto a computer or other programmable image scaling device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0164] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0165] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for processing face data, characterized in that: The method comprises: Obtaining three-dimensional point cloud data of the target object's face; The three-dimensional point cloud data is divided into a plurality of grids, and the three-dimensional point cloud data in each of the grids is normalized to a specified value space to obtain a plane image; wherein the three-dimensional point cloud data in each of the grids is normalized to a specified value space, including: performing for each of the grids: assigning a weight to the grid based on the position information and height information of the point cloud data in the grid, normalizing the weight of the grid, and obtaining a pixel point corresponding to the grid in the plane image; the weight has a positive correlation with the height information, and the weight is higher when the position of the point cloud data is closer to the central area; Using a first neural network to identify a plurality of first candidate face frames from the planar image; Using a second neural network to score each of the first candidate human face frames to obtain a score value for each of the first candidate human face frames; and, based on the score value, selecting a second candidate human face frame from the plurality of first candidate human face frames; Using a third neural network to process and analyze the sub-image corresponding to the second candidate face frame to obtain face information, wherein the face information includes a face area and positions of key points of the face; The face information is used to perform face classification processing to determine whether the target object is a known object.

2. The method according to claim 1, characterized in that The step of dividing the three-dimensional point cloud data into a plurality of grids comprises: The three-dimensional point cloud data is divided in length and width directions according to the size of each grid to obtain the multiple grids.

3. The method according to claim 1, characterized in that The using the face information to perform face classification processing to determine whether the target object is a known object includes: Extracting features of the target object from the plane image according to the face area and the positions of the key points of the face in the face information to obtain the face features of the target object; Performing classification and recognition based on the facial features to obtain the facial category of the target object; If the face category is a known target object, determining that the target object is a known object; If the face category is an unknown target object, then the target object is determined to be an unknown object.

4. The method according to claim 1, characterized in that: The method further comprises: If the target object is a known object, the smart door lock is controlled to perform an unlocking operation.

5. A facial data processing device, characterized in that: The device comprises: A point cloud data acquisition module is used to acquire three-dimensional point cloud data of the face of the target object; A plane image acquisition module, used for dividing the three-dimensional point cloud data into a plurality of grids, and normalizing the three-dimensional point cloud data in each of the grids to a specified value space to obtain a plane image; wherein, normalizing the three-dimensional point cloud data in each of the grids to a specified value space comprises: performing for each of the grids: assigning a weight to the grid based on the position information and height information of the point cloud data in the grid, normalizing the weight of the grid, and obtaining a pixel point corresponding to the grid in the plane image; the weight has a positive correlation with the height information, and the weight is higher when the position of the point cloud data is closer to the central area; A face recognition module is used to use a first neural network to identify multiple first candidate face frames from the plane image; use a second neural network to score each of the first candidate face frames to obtain a score value for each of the first candidate face frames; and, based on the score value, screen out a second candidate face frame from the multiple first candidate face frames; use a third neural network to process and analyze a sub-image corresponding to the second candidate face frame to obtain face information, wherein the face information includes a face area and a position of a face key point; and use the face information to perform face classification processing to determine whether the target object is a known object.

6. An electronic device, characterized in that: The invention comprises at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method according to any one of claims 1 to 4.

7. A computer storage medium, characterized in that: The computer storage medium stores a computer program, and the computer program is used to make a computer execute the method according to any one of claims 1 to 4.

Images