A vehicle-mounted face recognition system based on an RGBD depth camera

Abstract

The application discloses a kind of vehicle-mounted face recognition systems based on RGBD depth camera, including vehicle-mounted face area snapshot interception, angle orientation acquisition and face feature matching;Face feature is extracted using convolutional neural network and compared with information in standard database, and then face recognition is realized.Vehicle-mounted face snapshot color camera and depth camera are set in vehicle, and the direction of vehicle-mounted face snapshot camera module can be rotated, and vehicle terminal module is connected with remote server through communication module;Remote server is connected with terminal through network, the face snapshot camera module can be real-time snapshot, obtain face image and intercept, and then transmit to remote server through communication module, then database search and relevant feature analysis processing are carried out, and then the identification result is returned to terminal and displayed by remote server, since depth information is introduced on the basis of 2D image, it has higher security and robustness.

Description

TECHNICAL FIELD

[0001] The application relates to a vehicle-mounted face recognition system, in particular to a vehicle-mounted face recognition system based on an RGBD depth camera, and belongs to the technical field of face recognition. BACKGROUND

[0002] With the significant increase in the per capita number of vehicles in society, the problem of autonomous safety and prevention of the vehicle body and the driver has attracted more attention. From convenient travel to intelligent travel, the ADAS (Advanced Driving Assistance System) has always shown its intelligent and friendly characteristics. Among them, the efficient, convenient and reliable biometric identification technology represented by face recognition has become an important part of the intelligent vehicle security technology. Unlike other biometric identification requirements, face recognition technology can achieve identity authentication and unlocking, in-vehicle online payment and other functions by using only the vehicle-mounted camera and related modules; its deployment is convenient, the effect is excellent, and the cost is low, and many other advantages make face recognition technology stand out in the field of autonomous safety and prevention of the vehicle body and the driver.

[0003] Traditional face recognition algorithms are applied to many public places such as stations for face verification, which requires the posture of the identified person to be constrained; however, various uncontrollable factors in the actual application environment can seriously affect the key features of the face, making it more difficult to recognize the face. Therefore, in order to ensure the recognition efficiency, the face recognition seen in daily life requires active cooperation of the personnel, thereby reducing the influence of uncertain interference factors. However, in the vehicle environment, the state of the driver cannot be effectively constrained, and the images collected by the vehicle-mounted camera contain many problem factors such as light and posture, which makes it difficult for the traditional face recognition algorithm to realize safety and prevention in the vehicle environment.

[0004] At the same time, traditional two-dimensional face recognition still faces great challenges in biometric identification technology, because factors such as human head posture, lighting, facial expression, facial obstruction and decoration, facial aging, and inter-class similarity cause great changes in the appearance of the face, thereby reducing the face recognition rate. Among all possible changes, the greatest change in facial appearance comes from three-dimensional head rotation.

[0005] Although early 3D scanners and depth sensors have good accuracy, they are too expensive to be used in ordinary applications, and the three-dimensional image acquisition speed is slow, so most face recognition is achieved using two-dimensional images. With the gradual deepening of researchers, two-dimensional face recognition methods have shown more and more disadvantages, and have more limiting factors, which have also led to the lack of recognition accuracy, and two-dimensional images can only contain a limited amount of face information.

[0006] With the gradual reduction in the price of sensors such as Kinect v2, it is possible to integrate RGB-D sensors into ordinary notebook computers, tablet computers and even smart terminal devices. Through it, two-dimensional color images (RGB) and three-dimensional depth images (Depth) can be collected, wherein the depth image can provide depth information of the face, and the outstanding spatiality, three-dimensional structure and other characteristics of the three-dimensional face make the three-dimensional face have illumination and posture robustness in the recognition process, and can significantly solve the contour blur problem in the posture. Can more completely and accurately express the face state, so this way has natural advantages, can maintain stronger robustness under the influence of factors such as illumination, posture, expression and occlusion, and has greater potential in improving face recognition accuracy and face anti-counterfeiting.

[0007] Depth image: the depth image refers to an image expressing the distance characteristics between the geometric surface of an object and a specific plane. That is, in a three-dimensional coordinate space, the projection distance between the three-dimensional point cloud and the specific plane is calculated, and the distance is used as the pixel value of the point to generate a specific image, which is a way to express three-dimensional space structure information in the form of a two-dimensional image. The depth image reduces the dimension of the image data, and can be directly used as the input of a two-dimensional convolutional neural network, and is an ideal data expression form for the fusion of two-dimensional face recognition technology and depth information. SUMMARY

[0008] The purpose of the present application is to provide a vehicle-mounted face recognition system based on an RGBD depth camera, which can be more robust and secure for in-vehicle personnel identification tasks using multi-source information.

[0009] The present application realizes the above-mentioned purpose through the following technical scheme: a vehicle-mounted face recognition system based on an RGBD depth camera, comprising

[0010] A terminal is arranged in the vehicle, and comprises a camera device and a display device, the camera device of the terminal adopts a color camera and a high-precision depth camera to collect data in real time, the camera device comprises a central processor and a collection camera module, a snapshot camera module, a communication module and a Linux system capable of independent operation, which are all electrically connected with the central processor, a pre-trained lightweight deep neural network is run on the terminal to preprocess and transmit the original collected color image and depth image, and the display device is used to display the identification information publicly;

[0011] A remote server connected to the terminal through a network, the remote server comprising a face recognition network and a system face database, the face recognition network being a pre-trained model, using a deep learning method to extract face features, and then comparing with the face information in the system face database, if consistent, the information is sent to the terminal and displayed;

[0012] The system operation comprises the following steps:

[0013] I. Establish a three-dimensional face database containing depth information on the remote server; train the proposed first convolutional neural network model to obtain the angle and orientation of the actual image captured by the vehicle-mounted terminal and other three-dimensional spatial information;

[0014] II. Establish a trained first convolutional neural network model; import the angle and orientation information obtained by the trained first convolutional neural network model into the three-dimensional face database, and obtain the standard face three-dimensional feature map containing depth information with the same perspective, angle and orientation in the database;

[0015] III. Establish a trained second convolutional neural network; input the obtained standard face three-dimensional feature map and the depth map and color map transmitted by the terminal into the second convolutional neural network at the same time, and judge whether to pass the face verification according to the similarity obtained by the second convolutional neural network.

[0016] IV. Transmit the detected face identity to the terminal and display.

[0017] As a further scheme of the present application: the establishment of the three-dimensional face database specifically comprises:

[0018] High-precision three-dimensional face models are obtained by means of high-precision modular cameras, such as the currently popular three-dimensional face modeling cameras Artec Eva, EinScan-Pro, etc., and saved to the server database.

[0019] The lightweight neural network running on the terminal device is mainly constructed by an anchor-free convolutional network;

[0020] All label images and depth images containing different angles and orientations of human faces and labeled face regions are divided into training sets and test sets.

[0021] As a further scheme of the present application: before the training set image data is input into the convolutional neural network, the input data needs to be normalized to reduce the influence of environmental interference and noise on recognition.

[0022] As a further scheme of the present application: the training of the convolutional neural network model specifically comprises:

[0023] initializing weights of the network;

[0024] The input data is forwarded through the convolutional layer, the normalization layer, the pooling layer and the global pooling layer to obtain output values, such as the position and size of the bounding box and the predicted category.

[0025] The error between the output value of the network and the target value label is calculated.

[0026] The error is transmitted back to the network, and the errors of the global pooling layer, the pooling layer and the convolutional layer are sequentially calculated. The error of each layer can be understood as the total error of the network, that is, how much weight the network should bear. The network weight is updated, and the cycle is repeated for multiple rounds. When the error is equal to or less than the expected value or reaches the specified number of times, the training is ended.

[0027] Finally, the network generates the position and size of the bounding box in the color image.

[0028] As a further scheme of the application: the first convolutional neural network running on the remote server is composed of a convolutional neural network containing a residual module, and the final output is the stereo orientation information of the image transmitted by the terminal.

[0029] The second convolutional network running on the remote server is composed of a deep network containing an Inception module and a channel self-attention module, which is used to extract face features and output the similarity of the color image and depth image transmitted by the terminal compared with the specified angle and orientation in the database, for judging whether to pass the verification.

[0030] As a further scheme of the application: the collected face photo and depth image are transmitted to the remote server by using the trained lightweight neural network model. The remote server first runs the first convolutional neural network to obtain the specific orientation information of the photographed image, and transmits the orientation information to the three-dimensional face database to obtain the standard face feature map containing depth information under the photographed angle from the database. Then, the standard face feature map and the face photo and depth photo collected by the terminal are transmitted to the second convolutional neural network at the same time to calculate the actual similarity, so as to obtain the identity information of the face.

[0031] The color camera and the depth camera of the vehicle-mounted terminal collect the depth image and the color image containing the face, and run the lightweight neural network on the terminal to crop the face part in the image and the corresponding area in the depth image.

[0032] The network module is transmitted to the remote server.

[0033] The intercepted color image and the depth image are transmitted to a first convolutional neural network first, the stereo azimuth information of the image is extracted, the information is introduced into a three-dimensional face database, and a standard face feature depth image under the angle information is obtained;

[0034] The standard face feature depth image and the feature depth image collected by the terminal are transmitted to a second convolutional neural network to obtain a similarity, and whether the face verification is passed is judged according to the similarity.

[0035] The face verification information is transmitted back to the terminal and displayed.

[0036] The beneficial effects of the present application are:

[0037] The three-dimensional face data used contains the face set feature information in the three-dimensional space, which is proved to be more robust under the action of factors such as illumination and angle. The shape features extracted from the depth image obtained by the RGBD camera have illumination independence. The powerful feature extraction and description ability of the convolutional neural network is used to extract the face features and then compare them with the information in the standard database, and then realize the recognition of the face. Compared with a two-dimensional image alone, it can capture more spatial structure information, depth information and texture information. At the same time, due to the introduction of the azimuth angle information in the three-dimensional database, the precision decline phenomenon caused by the change of the posture can be overcome to a certain extent. The vehicle-mounted face snapshot color camera and the depth camera are arranged in the vehicle, and the direction of the vehicle-mounted face snapshot camera module can be rotated. The vehicle-mounted terminal module and the remote server are connected through a communication module. The remote server is connected with the terminal through a 5G network, so that the face snapshot camera module can be used for real-time snapshot, face image acquisition and interception, and the communication module is used for transmitting the face image to the remote server. Then the database is searched and the related feature analysis is processed, and the recognition result is returned to the terminal and displayed through the remote server. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 It is a composition module diagram of the vehicle-mounted face recognition system based on the RGBD depth camera of the present application.

[0039] Figure 2 It is a face recognition process schematic diagram of the present application.

[0040] Figure 3 It is a first convolutional network residual module drawing of the present application.

[0041] Figure 4 It is a second convolutional network channel attention mechanism drawing of the present application. DETAILED DESCRIPTION

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] Example 1

[0044] like Figures 1 to 4 As shown, an in-vehicle face recognition system based on an RGBD depth camera includes...

[0045] The terminal, installed inside a vehicle, includes a camera device and a display device. The camera device uses a color camera and a high-precision depth camera to acquire data in real time. The camera device includes a central processing unit and an acquisition camera module, a snapshot camera module, a communication module, and a Linux system that are all electrically connected to the central processing unit. A pre-trained lightweight deep neural network runs on the terminal to preprocess and transmit the original acquired color and depth images. The display device is used to display the recognition information publicly.

[0046] A remote server is connected to the terminal via a network. The remote server includes a face recognition network and a system face database. The face recognition network is a pre-trained model that uses deep learning to extract facial features and then compares them with the facial information in the system face database. If they match, the information is sent to the terminal and displayed.

[0047] The system operation includes the following steps:

[0048] 1. Establish a 3D face database containing depth information running on a remote server; perform deep learning training on the proposed first convolutional neural network model to obtain the angle and orientation of the actual images captured by the vehicle terminal, and other 3D spatial information.

[0049] 2. Establish a well-trained first convolutional neural network model; import the angle and orientation information obtained from the well-trained first convolutional neural network model into a 3D face database, and obtain standard 3D face feature maps containing depth information with the same viewpoint, angle and orientation from the database.

[0050] 3. Establish a trained second convolutional neural network; input the obtained standard face 3D feature map and the depth map and color map transmitted by the terminal into the second convolutional neural network at the same time, and determine whether the face verification is passed based on the similarity obtained by the second convolutional neural network.

[0051] IV. Transmitting the detected face identity to the terminal and displaying.

[0052] In the embodiment of the present application, the establishment of the three-dimensional face database specifically comprises:

[0053] A high-precision three-dimensional face model is obtained by means of a high-precision modular camera, such as the popular three-dimensional face modeling camera Artec Eva, EinScan-Pro, etc. in the market at present, and is saved into a server database.

[0054] The lightweight neural network running on the terminal device is mainly constructed by an anchor-free convolutional network;

[0055] All label images and depth images containing different angles and orientations of human faces and labeled face regions are divided into a training set and a test set.

[0056] In the embodiment of the present application, before the training set image data is input into the convolutional neural network, the input data needs to be normalized to reduce the influence of environmental interference and noise on recognition.

[0057] In the embodiment of the present application, the training of the convolutional neural network model specifically comprises:

[0058] The network is initialized in weight;

[0059] The input data is forward propagated through a convolutional layer, a normalization layer, a pooling layer and a global pooling layer to obtain output values such as the position and size of a bounding box and the predicted category;

[0060] The error between the output value of the network and the target value label is calculated;

[0061] The error is transmitted back to the network, and the errors of the global pooling layer, the pooling layer and the convolutional layer are sequentially calculated. The error of each layer can be understood as the total error of the network, and the network should bear how much weight. The network weight is updated, and the cycle is repeated for multiple rounds. When the error is equal to or less than the expected value or reaches a specified number of times, the training is ended.

[0062] Finally, the network generates the position and size of the bounding box in the color image.

[0063] Embodiment two

[0064] As shown in Figures 1 to 4 A vehicle-mounted face recognition system based on an RGBD depth camera, comprising

[0065] The terminal is arranged in a vehicle and comprises a camera and a display device, the camera of the terminal adopts a color camera and a high-precision depth camera to collect data in real time, the camera comprises a central processor and an acquisition camera module, a snapshot camera module, a communication module and a Linux system capable of independent operation, which are electrically connected with the central processor, a pre-trained lightweight deep neural network is run on the terminal to pre-process and transmit original collected color images and depth images, and the display device is used for displaying identification information.

[0066] The remote server is connected with the terminal through a network, and the remote server comprises a face recognition network and a system face database, the face recognition network is a pre-trained model, and a face feature is extracted by using a deep learning method, and then compared with face information in the system face database, if consistent, the information is sent to the terminal and displayed.

[0067] The system comprises the following steps:

[0068] I. A three-dimensional face database containing depth information on the remote server is established; a first convolutional neural network model is trained by depth learning to obtain the angle and orientation of the actual image of the vehicle-mounted terminal and other three-dimensional space information;

[0069] II. A trained first convolutional neural network model is established; the angle and orientation information obtained by the trained first convolutional neural network model is imported into the three-dimensional face database, and a standard three-dimensional face feature map containing depth information with the same perspective, angle and orientation is obtained in the database;

[0070] III. A trained second convolutional neural network is established; the obtained standard three-dimensional face feature map and the depth map and color map transmitted by the terminal are simultaneously input into the second convolutional neural network, and whether the face verification is passed is judged according to the similarity obtained by the second convolutional neural network.

[0071] IV. The detected face identity is transmitted to the terminal and displayed.

[0072] In the embodiment of the application, the first convolutional neural network running on the remote server is composed of a convolutional neural network containing a residual module, and the final output is the three-dimensional orientation information of the image transmitted by the terminal;

[0073] The second convolutional network running on the remote server terminal is composed of a deep network containing an Inception module and a channel self-attention module, which is used for extracting face features and outputting the similarity of the color map and the depth map transmitted by the terminal compared with the specified angle and orientation in the database, and is used for judging whether the verification is passed.

[0074] In the embodiment of the present application, the collected face photo and depth image are transmitted to a remote server by using the trained lightweight neural network model. The remote server first runs a first convolutional neural network to obtain specific orientation information of the photographed image, and transmits the orientation information to a three-dimensional face database. The standard face feature map containing depth information under the photographed angle is obtained from the database, and then the standard face feature map is transmitted to a second convolutional neural network together with the face photo and depth photo collected by the terminal. The actual similarity is calculated to obtain the identity information of the face. Specifically:

[0075] The color camera and the depth camera of the vehicle-mounted terminal collect the depth map and the color map containing the face, and run the lightweight neural network on the terminal to intercept the face part in the image and the corresponding area in the depth map.

[0076] The network module is used to transmit the collected information to the remote server.

[0077] The intercepted color map and depth map are first transmitted to the first convolutional neural network to extract the stereoscopic orientation angle information of the image, and the information is imported into the three-dimensional face database to obtain the standard face feature depth map under the angle information.

[0078] The standard face feature depth map and the feature depth image collected by the terminal are simultaneously transmitted to the second convolutional neural network to obtain the similarity, and whether the face verification is passed is determined according to the level of the similarity.

[0079] The face verification information is transmitted back to the terminal and displayed.

[0080] The actual implementation steps are as follows:

[0081] I. The vehicle-mounted terminal module starts running self-checking and connects with the remote server. When face detection is needed, the RGBD module is started and the lightweight neural network located on the terminal is run to complete the interception of the corresponding area of the RGB map and the depth map.

[0082] II. The vehicle-mounted terminal transmits the intercepted corresponding area to the remote server through the network communication module.

[0083] III. After receiving the picture information from the terminal, the remote server runs the first convolutional neural network to obtain the corresponding orientation angle information, and transmits the spatial information into the three-dimensional database.

[0084] IV. After receiving the orientation information, the three-dimensional database exports the standard three-dimensional face picture feature information under the corresponding spatial information.

[0085] Five, the database derived information and the information transmitted by the terminal are input into the second convolutional neural network at the same time, the second convolutional neural network extracts features and then outputs the similarity of the two information sources, and the identity information of the to-be-identified person is confirmed according to the similarity.

[0086] Six, the server returns the recognition result to the terminal.

[0087] Seven, the terminal displays the final recognition result.

[0088] Working principle: the present application comprises three links of vehicle-mounted face region snapshot interception, angle and direction acquisition and face feature matching, face features are extracted by using a convolutional neural network, and the extracted face features are compared with information in a standard database, so that face recognition is realized.

[0089] It will be obvious to a person skilled in the art that the application is not limited to the details of the above-described exemplary embodiments, and the application can be implemented in other specific forms without departing from the spirit or essential characteristics of the application. Therefore, the embodiments should be regarded as exemplary and non-limiting, and the scope of the application is defined by the appended claims rather than the above description, and all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the application. Any reference signs in the claims should not be regarded as limiting the claims involved.

[0090] In addition, it should be understood that, although the present specification is described in terms of embodiments, not every embodiment contains only one independent technical solution, and the description of the specification is only for the sake of clarity, and those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can be combined appropriately to form other embodiments that those skilled in the art can understand.

Claims

1. A vehicle-mounted face recognition system based on an RGBD depth camera, characterized in that: include The terminal includes a camera device and a display device. The camera device of the terminal uses a color camera and a high-precision depth camera to collect data in real time. The camera device includes a central processing unit and a data acquisition module, a snapshot module, a communication module, and a Linux system that are all electrically connected to the central processing unit. A pre-trained lightweight deep neural network runs on the terminal to preprocess and transmit the original acquired color and depth images. The display device is used to display the recognition information publicly. A remote server is connected to the terminal via a network. The remote server includes a face recognition network and a system face database. The face recognition network is a pre-trained model that uses deep learning to extract facial features and then compares them with the facial information in the system face database. If they match, the information is sent to the terminal and displayed. The operation of the vehicle-mounted facial recognition system includes the following steps: 1) Establish a 3D face database containing depth information running on a remote server; perform deep learning training on the proposed first convolutional neural network model to obtain the angle and orientation of the actual images captured by the vehicle terminal, and other 3D spatial information. 2) Establish a trained first convolutional neural network model; import the angle and orientation information obtained from the trained first convolutional neural network model into a 3D face database, and obtain standard 3D face feature maps containing depth information with the same viewpoint, angle and orientation from the database; 3) Establish a trained second convolutional neural network; input the obtained standard face 3D feature map and the depth map and color map transmitted by the terminal into the second convolutional neural network at the same time, and determine whether the face verification is passed based on the similarity obtained by the second convolutional neural network. 4) Transmit the detected facial identity to the terminal and display it; The establishment of the three-dimensional face database specifically includes the following steps: 1) Obtain high-precision 3D face models by using high-precision module cameras; 2) Lightweight neural networks running on terminal devices are mainly constructed from anchor-free convolutional networks; 3) All labeled images and depth images containing faces at different angles and orientations, with the face regions marked, are divided into training and testing sets; Before the training set image data is input into the convolutional neural network, the input data needs to be normalized to reduce the impact of environmental interference and noise on recognition. The training of the first convolutional neural network model and the second convolutional neural network specifically includes: Initialize the weights of the network; The input data is passed through a convolutional layer, a normalization layer, a pooling layer, and a global pooling layer before the output value is obtained. Calculate the error between the network's output value and the target label value; The error is fed back into the network, and the errors of the global pooling layer, pooling layer, and convolutional layer are calculated in turn. The error of each layer is the total error of the network, and the weight that the network should bear is determined. The network weights are then updated. This process is repeated multiple times. When the error is equal to or less than the expected value or reaches the specified number of iterations, the training ends. Ultimately, the network generates the border position and size in the color image; The first convolutional neural network running on the remote server is composed of a convolutional neural network containing residual modules, and the final output is the stereo orientation information of the image transmitted by the terminal. The second convolutional network running on the remote server consists of a deep network containing an Inception module and a channel self-attention module. It is used to extract facial features and output the similarity between the color image and depth image transmitted by the terminal and the specified angle and orientation in the database, which is used to determine whether the verification is successful.

2. The vehicle-mounted face recognition system according to claim 1, characterized in that: Before the training set image data is input into the convolutional neural network, the input data needs to be normalized to reduce the impact of environmental interference and noise on recognition.

3. The vehicle-mounted face recognition system based on an RGBD depth camera according to claim 1, characterized in that: The trained lightweight neural network model transmits the acquired face photos and depth images to a remote server. The remote server first runs a first convolutional neural network to obtain the specific orientation information of the captured images and transmits this orientation information to a 3D face database. From the database, it obtains a standard face feature map containing depth information at the captured angle. Then, the standard face feature map and the face photos and depth images acquired by the terminal are simultaneously transmitted to a second convolutional neural network to calculate the actual similarity, thereby obtaining the face's identity information; specifically: The vehicle-mounted terminal's color camera and depth camera capture depth and color images containing faces, and run a lightweight neural network on the terminal to crop the face portion of the image and then crop the corresponding region in the depth image. Transmitted to a remote server via the network module; The captured color image and depth map are first transmitted to the first convolutional neural network to extract the stereo orientation angle information of the image, and then the information is imported into the three-dimensional face database to obtain the standard face feature depth map under the angle information. The standard facial feature depth map and the feature depth image collected by the terminal are simultaneously transmitted to the second convolutional neural network to obtain the similarity. The similarity is used to determine whether the face verification is successful. The facial verification information is transmitted back to the terminal and displayed.

Images