A face recognition method for heterogeneous video surveillance based on edge computing

Abstract

The present invention relates to a heterogeneous video surveillance face recognition method based on edge computing, including edge device face detection, face alignment and backend service face image feature extraction and cloud database face feature comparison. By deploying a custom face recognition network model on the edge computing device, the key points of the face are detected and obtained, and then the face alignment operation is performed on the edge device and sent to the cloud server. The cloud server obtains the aligned face image and edge device ID, extracts face features, and compares the obtained face features with the face features in the big data system to perform face recognition and information retrieval. The present invention reduces the network bandwidth pressure by detecting and pushing the key points of the face at the edge end, while releasing the computing resources of the server to the maximum extent, focusing on the face feature extraction function; the cloud enables multiple video surveillance devices to be interconnected, and can achieve complete face recognition and information retrieval traceability.

Description

Technical Field

[0001] The present invention relates to the field of video surveillance and security technology, and in particular to a heterogeneous video surveillance face recognition method based on edge computing. Background Art

[0002] The uniqueness of human face and its unreproducibility provide conditions for identity authentication. Therefore, face recognition is an important means of identity authentication. At present, face recognition is widely used.

[0003] In real life, the conventional face recognition method is to save one or a limited number of face images, and confirm the identity through a face recognition method with low real-time performance, which is mainly used for personal access control systems. In the field of security, adding real-time face recognition to real-time video surveillance systems for face recognition and face information retrieval is not popular. Taking the community security system as an example, the access control system provides basic card number tracking, fingerprint recognition or ID card confirmation for people entering and leaving the community, and the video surveillance system records and saves the movement of people in the community in the form of video, but does not realize resource sharing and information exchange between systems, which makes it difficult to achieve true visitor information tracking and tracing. Therefore, it is very difficult to save the flow of people in the community and retrieve it when needed.

[0004] In order to solve this problem, a cloud database is needed to save historical data and face capture information, but the traditional stand-alone face recognition system cannot provide the function of saving a large amount of data. At the same time, it is difficult to deploy high-precision face recognition algorithms due to the limitation of stand-alone equipment resources. It can only be used for close-range face recognition, such as access control. Since the surveillance video shooting is relatively far away from the access control, and a database is needed to save a large amount of face data and capture information to provide retrieval functions, the security system corresponding to video surveillance has posed challenges to the complexity of face recognition algorithms and system network communications compared to traditional access control. Summary of the invention

[0005] The purpose of this application is to provide a heterogeneous video surveillance face recognition method based on edge computing to solve the problem that the existing technology cannot effectively identify in the video surveillance security system scenario.

[0006] To achieve the above object, the present invention proposes a heterogeneous video surveillance face recognition method based on edge computing, comprising the following steps:

[0007] (1) Obtain the original image video stream;

[0008] (2) Perform image decoding on the edge computing device to obtain the original color image in real time;

[0009] (3) After the video is decoded, the color image is sent to the custom face detection network on the edge computing device to extract several key points of the face;

[0010] (4) Aligning the facial image based on several key points of the face on the edge computing device;

[0011] (5) Push the aligned face image and device ID information to the cloud;

[0012] (6) The cloud obtains the aligned face image and edge device ID and performs face feature extraction;

[0013] (7) Compare the acquired facial features with the facial features in the big data system to perform face recognition and information retrieval.

[0014] Furthermore, taking the RGB color image as input, the following formula is used to map the pixel values ​​from the interval [0, 255] to the interval [0, 1] before feeding the image into the custom face detection network:

[0015] new_pixel = pixel / 255

[0016] In the above formula, pixel represents the original pixel value, and new_pixel represents the mapped pixel value.

[0017] Furthermore, the custom face detection network uses a 1x1 convolution, a 3x3 grouped convolution and a 1x1 convolution as the smallest composite convolution module, and the Eltwise network layer introduces a residual network structure for the entire feature extraction network part, with a 1x1 convolution as the last output layer of the network.

[0018] Furthermore, the dimensions of the output results of the custom face detection network are all multiples of 16, and a set of 16-dimensional floating-point arrays represents a face detection result, which are: the left boundary of the face frame, the upper boundary of the face frame, the width of the face frame, the height of the face frame, the horizontal coordinates of the five key points of the face, the vertical coordinates of the five key points of the face, the probability that the frame is the foreground, and the probability that the frame is the face; for the final output result of the custom face detection network, the output result value representing the probability and the output result value directly representing the position of the point are mapped to the [0, 1] interval using logistic regression, as shown in the following formula:

[0019]

[0020] Among them, x represents the direct result value of the network, and y represents the result value after mapping.

[0021] The output values ​​representing the width and height of the face frame need to be mapped to the real number space using an exponential function, as shown in the following formula:

[0022] y=e x

[0023] Among them, x represents the direct result value of the network, and y represents the result value after mapping.

[0024] Furthermore, the custom face detection network model uses the following loss function during training:

[0025]

[0026] Among them, x, y, w, h, tx i ,ty i , obj, c represents the left boundary, upper boundary, width, height, horizontal and vertical coordinates of the five key points, foreground probability, and face confidence probability of the annotation box that best matches the prediction box; px, py, pw, ph, ptx i , pty i , pobj, pc represent the left boundary, upper boundary, width, height, horizontal and vertical coordinates of the five key points, foreground probability, and face confidence probability of the prediction box.

[0027] Furthermore, in step (4), the face is aligned to a standard pose face with a size of 112x96 pixels, and the alignment coordinates used are

[0028] Furthermore, in step (5), the aligned face image is encoded in base64 encoding and pushed to the cloud in json data format.

[0029] Furthermore, in step (6), the aligned face image is mirrored, and the original aligned image and the mirrored aligned image are used as input to extract 1024-dimensional face features.

[0030] Furthermore, in step (7), the cosine calculation formula is used to calculate the similarity to recognize the face, as shown in the following formula:

[0031]

[0032] Among them, A and B represent facial feature vectors, and the result interval of cos(A, B) is [0,1] corresponding to [0%,100%].

[0033] The present invention is aimed at the scene of video surveillance security system. Compared with the traditional access control scene, the shooting distance of this scene is relatively far. The traditional method is limited by the equipment and cannot deploy high-precision and complex face recognition algorithms, resulting in extremely low long-distance face recognition rate. At the same time, the traditional method does not provide a retrieval function, and unrecognizable faces will be abandoned and forgotten, and face retrieval and tracking based on similarity will not be performed. The method proposed by the present invention can combine the advantages of edge computing and cloud services to provide a face recognition system for video surveillance security. Compared with traditional methods, more complex and efficient face recognition algorithms can be deployed to identify faces in surveillance videos; by detecting and pushing the key points of the face at the edge, the network bandwidth pressure can be reduced, and the computing resources of the server can be released to the greatest extent, focusing on the face feature extraction function, and providing the possibility of deploying more complex and accurate face recognition methods; in addition, cloud services provide the possibility of network interconnection and information storage and retrieval, realize the interconnection of multiple video surveillance devices, and realize complete face recognition and information retrieval traceability. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a flow chart of the method of the present invention;

[0035] Figure 2 This is a diagram of the convolutional layer combination of the custom face detection network;

[0036] Figure 3 It is the Eltwise operator of the custom face detection network;

[0037] Figure 4 It is a schematic diagram of the detection module structure of the custom face detection network;

[0038] Figure 5 This is the effect diagram of face detection results;

[0039] Figure 6 This is the effect diagram of face image alignment operation;

[0040] Figure 7 It is a mirrored face alignment image;

[0041] Figure 8 It is a demonstration of face recognition and capture results. DETAILED DESCRIPTION

[0042] The present invention discloses a heterogeneous video surveillance face recognition method based on edge computing, including an edge computing device, a face recognition backend server (belonging to a cloud server) and a cloud database. By deploying a custom face recognition network model on the edge computing device, five key points of the face are detected and obtained (as other implementation methods, the number of key points can be designed as needed), and then the face alignment operation is performed on the edge device and sent to the cloud server. The cloud server obtains the aligned face image and edge device ID, extracts face features, and compares the obtained face features with the face features in the big data system to perform face recognition and information retrieval.

[0043] like Figure 1 As shown, the method mainly includes edge device face detection, face alignment and backend service face image feature extraction and cloud database face feature comparison. Specifically, it includes the following steps:

[0044] (1) Obtain the original image video stream;

[0045] (2) Perform image decoding on the edge computing device to obtain the original color image in real time;

[0046] (3) After the video is decoded, the color image is sent to a custom face detection network to extract the five key points of the face;

[0047] (4) Align the facial image based on the five key points of the face on the edge computing device;

[0048] (5) Push the aligned face image and device ID information to the cloud;

[0049] (6) The cloud obtains the aligned face image and edge device ID and performs face feature extraction;

[0050] (7) Compare the acquired facial features with the facial features in the big data system to perform face recognition and information retrieval.

[0051] Finally, the cloud server can send the recognition result to the corresponding terminal; or respond according to the request of the terminal.

[0052] In the steps (1) and (2), the monitoring video stream is used as the input source of the system, and the video stream is decoded by hardware decoding to obtain an RGB color image.

[0053] In step (3), the custom face detection network used is Figure 2 The convolutional layer combination shown is a 1x1 convolution, a 3x3 grouped convolution, and a 1x1 convolution as the smallest composite convolution module, combined with Figure 3The Eltwise network layer shown introduces a residual network structure for the entire feature extraction network. Finally, use Figure 4 The module structure shown is used as the detector of the network, with 1x1 convolution as the last output layer of the network. The specific network structure parameters are shown in Table 1.

[0054] The custom face detection network takes an RGB color image as input and uses the following formula to map pixel values ​​from the [0, 255] interval to the [0, 1] interval before feeding the image into the network:

[0055] new_pixel = pixel / 255

[0056] Among them, pixel represents the original pixel value, and new_pixel represents the mapped pixel value.

[0057] The custom face detection network has three outputs, indicating face detection at three different feature scales. The dimension of each output result is a multiple of 16. A set of 16-dimensional floating-point arrays represents a face detection result, which are: the left boundary of the face box, the upper boundary of the face box, the width of the face box, the height of the face box, the horizontal and vertical coordinates of the five key points of the face, the probability that the box is the foreground, and the probability that the box is a face.

[0058] For the final output of the network, both the output value representing the probability and the output value directly representing the position of the point need to be mapped to the [0, 1] interval using logistic regression, as shown in the following formula:

[0059]

[0060] Among them, x represents the direct result value of the network, and y represents the result value after mapping.

[0061] The output values ​​representing the width and height of the face frame need to be mapped to the real number space using an exponential function, as shown in the following formula:

[0062] y=e x

[0063] Among them, x represents the direct result value of the network, and y represents the result value after mapping.

[0064] The final mapping result of probability and width and height is the actual predicted value of probability and width and height. The mapping result of point position indicates the position of the point in the original RGB image at the scale of the original image, so the horizontal and vertical coordinates of the position are in the interval [0, 1]. The position coordinates of the point can be obtained by multiplying the size of the original image.

[0065] The network model uses the following loss function during training:

[0066]

[0067] Among them, x, y, w, h, tx i ,ty i , obj, c represents the left boundary, upper boundary, width, height, horizontal and vertical coordinates of the five key points, foreground probability, and face confidence probability of the annotation box that best matches the prediction box; px, py, pw, ph, ptx i , pty i , pobj, pc represent the left boundary, upper boundary, width, height, horizontal and vertical coordinates of the five key points, foreground probability, and face confidence probability of the prediction box.

[0068] In step (4), after the face detection network obtains the five key points of the face, it is necessary to align the face to a standard posture face with a size of 112x96 pixels. The alignment coordinates used are Alignment effect Figure 5 After calculating the radial change matrix between the five points obtained by detection and the standard five points, the face radial transformation is performed, and then the face area of ​​112x96 pixels is cropped, as shown in Figure 6 shown.

[0069] In the step (5), the aligned face image is encoded in the form of base64 encoding and pushed to the cloud together with the ID information of the edge computing device in the json data format.

[0070] In step (6), the face recognition backend server obtains the aligned face image base64 encoding and edge computing device ID information, decodes the base64 encoded image to obtain an RGB image, and then mirrors the image to obtain another RGB image, such as Figure 7 As shown in the figure, the InsightFace facial feature extraction network with an input size of 2 x 3 x 112 x 96 is used, and the original aligned image and the mirrored aligned image are used as input to extract 1024-dimensional facial features. The network output is a 1024-dimensional floating-point array.

[0071] In step (7), the similarity between the currently acquired facial feature vector and the facial feature vector stored in the data is calculated using the cosine calculation formula, as shown in the following formula:

[0072]

[0073] Among them, A and B represent facial feature vectors, and the result interval of cos(A, B) is [0,1] corresponding to [0%,100%].

[0074] A similarity threshold can be specified to classify the recognized faces into recognized and unrecognized, and store them in different databases for different retrieval scenarios. Among them, the recognition results above the threshold can be used for identity confirmation and identity image expansion, while the recognition results below the threshold are saved in another database for information recording and later retrieval, such as Figure 8 shown.

[0075] The present invention is aimed at the scene of video surveillance security system. Compared with the traditional access control scene, the shooting distance of this scene is relatively far. The traditional method is limited by the equipment and cannot deploy high-precision and complex face recognition algorithms, resulting in extremely low long-distance face recognition rate. At the same time, the traditional method does not provide a retrieval function, and unrecognizable faces will be abandoned and forgotten, and face retrieval and tracking based on similarity will not be performed. The method proposed by the present invention can combine the advantages of edge computing and cloud services to provide a face recognition system for video surveillance security. Compared with traditional methods, more complex and efficient face recognition algorithms can be deployed to identify faces in surveillance videos; by detecting and pushing the key points of the face at the edge, the network bandwidth pressure can be reduced, and the computing resources of the server can be released to the greatest extent, focusing on the face feature extraction function, and providing the possibility of deploying more complex and accurate face recognition methods; in addition, cloud services provide the possibility of network interconnection and information storage and retrieval, realize the interconnection of multiple video surveillance devices, and realize complete face recognition and information retrieval traceability.

[0076] Table 1 Custom face detection network structure parameter table

[0077]

[0078]

[0079]

Claims

1. A heterogeneous video surveillance face recognition method based on edge computing, characterized in that: The following steps are involved: (1) Obtain the original image video stream; (2) Perform image decoding on the edge computing device to obtain the original color image in real time; (3) After the video is decoded, the color image is sent to the custom face detection network on the edge computing device to extract several key points of the face; (4) Aligning the facial image based on several key points of the face on the edge computing device; (5) Push the aligned face image and device ID information to the cloud; (6) The cloud obtains the aligned face image and edge device ID and performs face feature extraction; (7) Compare the acquired facial features with the facial features in the big data system to perform face recognition and information retrieval; The dimensions of the output results of the custom face detection network are all multiples of 16. A set of 16-dimensional floating-point arrays represents a face detection result, which are: the left boundary of the face frame, the upper boundary of the face frame, the width of the face frame, the height of the face frame, the horizontal coordinates of the five key points of the face, the vertical coordinates of the five key points of the face, the probability that the frame is the foreground, and the probability that the frame is a face; For the final output result of the custom face detection network, both the output result value representing the probability and the output result value directly representing the position of the point are mapped to the [0,1] interval using logistic regression, as shown in the following formula: Among them, x represents the direct result value of the network, and y represents the result value after mapping; The output values ​​representing the width and height of the face frame need to be mapped to the real number space using an exponential function, as shown in the following formula: and=e x Among them, x represents the direct result value of the network, and y represents the result value after mapping.

2. The method for face recognition in heterogeneous video surveillance based on edge computing according to claim 1 is characterized in that: Taking an RGB color image as input, the following formula is used to map pixel values ​​from the [0,255] interval to the [0,1] interval before feeding the image into the custom face detection network: new_pixel = pixel / 255 In the above formula, pixel represents the original pixel value, and new_pixel represents the mapped pixel value.

3. The method for face recognition in heterogeneous video surveillance based on edge computing according to claim 1 is characterized in that: The custom face detection network uses a 1x1 convolution, a 3x3 grouped convolution and a 1x1 convolution as the smallest composite convolution module. At the same time, the Eltwise network layer introduces a residual network structure for the entire feature extraction network part, with a 1x1 convolution as the last output layer of the network.

4. The method for face recognition in heterogeneous video surveillance based on edge computing according to claim 1 is characterized in that: The custom face detection network model uses the following loss function during training: Among them, x, y, w, h, tx i ,ty i , obj, c represents the left boundary, upper boundary, width, height, horizontal and vertical coordinates of the five key points, foreground probability, and face confidence probability of the annotation box that best matches the prediction box; px, py, pw, ph, ptxi, ptyi, pobj, pc represent the left boundary, upper boundary, width, height, horizontal and vertical coordinates of the five key points, foreground probability, and face confidence probability of the prediction box.

5. The method for face recognition in heterogeneous video surveillance based on edge computing according to claim 1 is characterized in that: In step (4), the face is aligned to a standard pose face with a size of 112x96 pixels, and the alignment coordinates used are 6. The method for face recognition in heterogeneous video surveillance based on edge computing according to claim 1 is characterized in that: In step (5), the aligned face image is encoded in base64 and pushed to the cloud in json data format.

7. The method for face recognition in heterogeneous video surveillance based on edge computing according to claim 1 is characterized in that: In step (6), the aligned face image is mirrored, and the original aligned image and the mirrored aligned image are used as input to extract 1024-dimensional face features.

8. The method for face recognition in heterogeneous video surveillance based on edge computing according to claim 1, characterized in that: In step (7), the cosine calculation formula is used to calculate the similarity to recognize the face, as shown in the following formula: Among them, A and B represent facial feature vectors, and the result interval of cos(A, B) is [0,1] corresponding to [0%,100%].

Images