A detection and recognition method for document images

Abstract

This specification provides a method for detecting and recognizing ID images, relating to the field of image processing technology. The method involves obtaining training images to be detected and recognized, dividing the training images into a training sample set and a test sample set; then constructing a deep neural network structure for detecting and recognizing the ID images to be detected; inputting the obtained training sample set into the data input layer of the constructed deep neural network structure for deep neural network training to obtain a detection and recognition model; and finally, inputting the obtained test sample set into the detection and recognition model to detect and recognize the ID images to be detected, which contain 68 facial feature points, 3 facial posture angles, and 1 eyeglass wearing state. The method implements hybrid estimation, boasting high speed and accuracy. It also simultaneously recognizes a user's facial feature points, facial posture angles, and eyeglass wearing state, and can provide reminders to the user, thereby obtaining a photo that meets the ID requirements when the user takes the photo.

Description

Technical Field

[0001] One or more embodiments of this specification relate to the field of image processing technology, and in particular, to a method for detecting and identifying document images. Background Art

[0002] With the development of artificial intelligence and information technology, a self-service photo-taking device specifically for ID photos has been developed. This device allows people to take, automatically crop, assess quality, pay for, and upload photos for their ID cards, driver's licenses, and entry-exit documents to the public security network. This not only provides convenience for the public, saves time, and reduces the workload of public security departments, but also involves technologies such as automatic portrait cropping, facial feature point estimation, facial pose estimation, and image quality assessment. When using the self-service photo-taking device, artificial intelligence technology is used to automatically estimate facial pose, providing real-time reminders to adjust sitting and head posture to capture qualified ID photos. Facial feature point estimation can effectively assist in cropping photos that meet Ministry of Public Security standards. Furthermore, ID photos generally do not allow users to wear glasses, so automatic recognition of whether or not they are wearing glasses is also required.

[0003] There are many deep-learning-based facial landmark estimation algorithms available. One proposed a multi-task learning facial landmark detection algorithm (Zhanpeng Zhang, Ping Luo, Chen Change Loy, Xiaoou Tang. Facial Landmark Detection by Deep Multi-task Learning, in Proceedings of European Conference on Computer Vision (ECCV), 2014). However, this algorithm only extracts five key points: the left and right eye centers, the left and right corners of the mouth, and the tip of the nose. Another proposed a progressive representation learning network structure for extracting five facial features (Lingbo Liu, Guanbin Li, Yuan Xie, Yizhou Yu, Qing Wang, and Liang Lin, “Facial Landmark Machines: A Backbone-Branches Architecture with Progressive Representation Learning.”, IEEE Transactions on Mulitmedia, 2019); 68 facial landmarks are a popular facial landmark annotation method and are widely researched and applied, as they provide richer information. Currently, traditional methods have implemented 68 landmark detection, such as the dlib library (http: / / dlib.net / face_landmark_detection_ex.cpp.html).

[0004] Face pose estimation is to determine the angle information of the face orientation given a face image. The face pose can be determined by the rotation angle relative to the X, Y, and Z coordinate axes. The deflection of these three coordinate axes is called Pitch (up and down flip), Yaw (left and right rotation), and Roll (left and right tilt). At present, the main methods of face pose estimation are: (1) methods based on random forest pattern recognition; (2) methods based on convolutional neural networks; (3) methods based on geometric calculations; (4) methods based on particle swarm filtering and optimization; (5) methods based on target tracking. Sichuan University proposed a face pose estimation based on deep learning (Computer Technology and Development, 10 Oct, 2016, Chengdu, China). This work was carried out on the CAS-PEAL dataset. The facial area of ​​the detected person was extracted and the facial area was input into the network. When different face poses are used, the face angle is also different, and the neural network can map the samples to different spaces, thereby realizing the face pose estimation algorithm. Changchun University of Technology proposed a multi-view face pose estimation algorithm (Journal of Changchun University, January 2004, Changchun, China). This algorithm uses a facial feature extraction algorithm based on kernel principal component analysis (KPCA) and combines it with independent subspace analysis (ISA) observation subspace learning. The resulting ISA face pose estimation algorithm uses KPCA analysis, which primarily analyzes facial features and facial images to form a set of feature vectors. Finally, the ISA analysis forms a feature subspace, enabling face pose estimation without increasing complexity while achieving relatively high accuracy. Lex Fridman et al. (IET Computer Vision, 2016, vol. 10, no. 4, pp. 308-313) proposed estimating facial pose using a geometric method based on the spatial location of the coordinates of facial feature points. The CDF method was then used to detect pupil angle. Finally, the head pose and pupil angle were combined, and a random forest prediction model was trained to implement a face pose estimation algorithm. Dornaika et al. (IEEE Transactions on Circuits and Systems for Video Technology, 2006, vol. 6, no. 9, pp. 1107-1124) used an adaptive appearance model through particle filtering to track the intrinsic connection between three-dimensional facial pose and local motion of internal facial features for face pose estimation.POSIT is a point-to-point matching algorithm that uses 2D image feature points and 3D image key feature points to estimate facial pose. This method involves finding the corresponding 2D feature points on the 3D face model's feature points, calculating the rotation, projection, and translation matrices between the two, and estimating the 3D face's orientation from the 2D image feature point positions.

[0005] It can be seen that the existing traditional estimation algorithms only use multiple different algorithms to estimate facial feature points and facial postures, which are not only inaccurate but also slow, affecting the user interaction experience, and do not have the function of identifying glasses. Therefore, providing a detection and recognition method for document images is a technical problem that needs to be solved urgently by those skilled in the art. Summary of the Invention

[0006] In view of this, the purpose of one or more embodiments of this specification is to propose a detection and recognition method for ID images to solve the problem that it is impossible to simultaneously detect and recognize 68 facial feature points, 3 facial posture angles and 1 glasses wearing status when processing ID images.

[0007] Based on the above objectives, one or more embodiments of this specification provide a method for detecting and identifying document images, including:

[0008] Step 1: Obtain training images to be detected and identified, and divide the training images into a training sample set and a test sample set; the training images are ID images that have been annotated with facial feature points, facial posture angles, and glasses wearing status;

[0009] Step 2: Construct a deep neural network structure to detect and identify the document image to be detected;

[0010] The deep neural network structure includes a data input layer, a first residual downsampling layer, a second residual downsampling layer, a third residual downsampling layer, a fourth residual downsampling and global pooling layer, a fifth residual downsampling and global pooling layer, a sixth residual downsampling and global pooling layer, a fully connected layer and an output layer, and the output layer is connected to the hybrid loss function;

[0011] Step 3: Input the training sample set obtained in step 1 into the data input layer of the deep neural network structure constructed in step 2 to perform deep neural network training to obtain a detection and recognition model;

[0012] The loss function during training adopts the hybrid loss function in step 2. The output of the hybrid loss function is fed back to the network, and the network is trained iteratively to minimize the hybrid loss function.

[0013] Step 4: Input the test sample set obtained in step 1 into the detection and recognition model to realize the detection and recognition of the document image to be tested containing 68 facial feature points, 3 facial posture angles and 1 glasses wearing state.

[0014] Preferably, in step 3, the training sample set obtained in step 1 is input into the data input layer of the deep neural network structure constructed in step 2 to perform deep neural network training to obtain a detection and recognition model, including:

[0015] Step 3.1: Perform residual downsampling three times on the training sample set input to the deep neural network to obtain the first feature map;

[0016] Step 3.2: Divide the first feature map obtained after step 3.1 into two paths. The first path continues to perform residual downsampling processing to obtain the second feature map; the second path performs global pooling processing to obtain the third feature map;

[0017] Step 3.3: Divide the second feature map obtained after step 3.2 into two paths. The first path continues to perform residual downsampling processing to obtain the fourth feature map; the second path performs global pooling processing to obtain the fifth feature map;

[0018] Step 3.4: Divide the fourth feature map obtained after processing in step 3.3 into two paths. The first path continues to perform residual downsampling processing to obtain the sixth feature map; the second path performs global pooling processing to obtain the seventh feature map;

[0019] Step 3.5: Copy and connect the pooled features in the third feature map, the fifth feature map, and the seventh feature map obtained by the three global pooling processes to obtain a multi-scale output feature vector in a fully connected manner.

[0020] Preferably, the residual downsampling process includes using three downsampling methods: average pooling, maximum pooling, and convolution, and adding the results of the three downsampling methods to obtain a residual downsampling structure.

[0021] Preferably, the hybrid loss function includes:

[0022]

[0023] in, is the mean square error loss function of 68 facial feature points, is the mean square error loss function of the three face pose angles, is the cross entropy loss function of the single-channel glasses wearing state, λ1, λ2 and λ3 are and The weighting coefficient of

[0024] The mean square error loss function of 68 facial feature points is defined as follows:

[0025]

[0026] in, and To predict the coordinates of the i-th facial feature point in the k-th face image, and is the coordinate of the i-th facial feature point in the k-th face image manually labeled, and N is the total number of face images in the training sample set;

[0027] The mean square error loss function of the three face pose angles is defined as follows:

[0028]

[0029] in, and are the predicted value and labeled value of the pose angle pitch of the k-th face image, and are the predicted value and the labeled value of the posture angle yaw of the k-th face image, and are the predicted value and labeled value of the posture angle roll of the k-th face image respectively;

[0030] The cross entropy loss function of single-channel glasses wearing state is defined as follows:

[0031] When the predicted probability of wearing glasses is greater than or equal to 50%, that is, The loss function is defined as follows:

[0032]

[0033] in, and The predicted probability and labeling probability of glasses wearing in the k-th face image are respectively Can only take 0 and 1;

[0034] When the predicted probability of wearing glasses is less than 50%, that is, The loss function is defined as follows:

[0035]

[0036] The characteristic function is introduced to combine formulas (4) and (5). The characteristic function is defined as follows:

[0037]

[0038] Therefore, the final single-channel glasses wearing state cross entropy loss function is defined as:

[0039]

[0040] Preferably, the size of the training image is 512x512x3, which becomes 256x256x16 after the first residual downsampling process, becomes 128x128x32 after the second residual downsampling process, and becomes a first feature map of size 64x64x48 after the third residual downsampling process; the size of the second feature map is 32x32x64, the size of the third feature map is 1x1x64, the size of the fourth feature map is 16x16x64, the size of the fifth feature map is 1x1x64, the size of the sixth feature map is 8x8x96, the size of the seventh feature map is 1x1x96, the feature vector after copying and connecting is 224x1 dimension, and the dimension of the output feature vector is 140x1 dimension.

[0041] As can be seen from the foregoing, one or more embodiments of this specification provide a method for detecting and recognizing ID images. The method first obtains a training image to be detected and recognized, divides the training image into a training sample set and a test sample set, then constructs a deep neural network structure for detecting and recognizing the ID image to be detected. The obtained training sample set is then input into the data input layer of the constructed deep neural network structure to train the deep neural network and obtain a detection and recognition model. Finally, the obtained test sample set is input into the detection and recognition model to detect and recognize the ID image to be detected, which includes 68 facial feature points, 3 facial pose angles, and 1 eyewear wearing state. The present invention utilizes a deep neural network to implement hybrid estimation, resulting in fast speed and high accuracy. The method also simultaneously recognizes a user's facial feature points, facial pose angles, and eyewear wearing state, enabling user notification to ensure that a photo that meets the ID requirements is taken. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only one or more embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 A flowchart of a method for detecting and identifying document images according to one or more embodiments of this specification;

[0044] Figure 2 This is a schematic diagram of the distribution of 68 facial feature points in a method for detecting and recognizing ID images in one or more embodiments of this specification;

[0045] Figure 3 A schematic diagram of a deep neural network structure in a method for detecting and recognizing document images according to one or more embodiments of this specification;

[0046] Figure 4 A schematic diagram of global pooling of multi-scale feature maps in a method for detecting and recognizing document images according to one or more embodiments of this specification;

[0047] Figure 5 This is a schematic diagram of a residual downsampling structure in a method for detecting and recognizing document images in one or more embodiments of this specification. DETAILED DESCRIPTION

[0048] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0049] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in one or more embodiments of this specification should have the usual meanings understood by people with ordinary skills in the field to which this disclosure belongs. The "first", "second" and similar words used in one or more embodiments of this specification do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative position relationships. When the absolute position of the object being described changes, the relative position relationship may also change accordingly.

[0050] like Figure 1 A flowchart of a method for detecting and identifying document images according to one or more embodiments of this specification; Figure 2 This is a schematic diagram of the distribution of 68 facial feature points in a method for detecting and recognizing ID images in one or more embodiments of this specification; Figure 3 A schematic diagram of a deep neural network structure in a method for detecting and recognizing document images according to one or more embodiments of this specification; Figure 4 A schematic diagram of global pooling of multi-scale feature maps in a method for detecting and recognizing document images according to one or more embodiments of this specification; Figure 5 This is a schematic diagram of a residual downsampling structure in a method for detecting and recognizing document images in one or more embodiments of this specification.

[0051] Facial feature point detection has important applications in face comparison, deformation, and beautification. Currently, many face databases often provide annotations of face frame positions and 68 facial feature point coordinates, such as Figure 2 The 68 facial feature points can provide key information about the distance from the face outline to the edge of the photo and the top of the head for background removal and cropping of second-generation ID card photos, driver's license photos, and entry-exit documents, so that ID photos that meet the standards of the Ministry of Public Security can be cropped.

[0052] When taking ID photos, users are required to sit upright, with their heads neither tilted nor rotated, nor pitched forward or backward, in compliance with Ministry of Public Security standards. These requirements can be assessed through facial pose. Facial pose can be measured using rotation angles relative to the horizontal X axis, vertical Y axis, and depth axis Z. The deflection angles of these three coordinate axes are called pitch (up and down), yaw (left and right rotation), and roll (left and right tilt).

[0053] In addition, the Ministry of Public Security's ID photo standards generally do not allow wearing glasses, so it is necessary to detect the wearing status of glasses.

[0054] Therefore, it is necessary to estimate the coordinates of 68 facial feature points, facial posture angles, and glasses wearing status. Estimating the coordinates of the 68 facial feature points and the facial posture angles is essentially a regression problem, while the glasses wearing status can be considered an image classification problem. Therefore, this problem is a hybrid recognition problem that combines regression and classification. To this end, the present invention provides a detection and recognition method for document images based on an end-to-end deep neural network, which is used to simultaneously detect 68 facial feature points, three facial posture angles, and one glasses wearing status in the document image to be detected.

[0055] like Figure 1 As shown, the embodiment of this specification provides a method for detecting and identifying document images, including:

[0056] Step 1: Obtain a training image to be detected and identified, and divide the training image into a training sample set and a test sample set; the training image is an ID image to be detected including facial feature points, facial posture angles, and glasses wearing status.

[0057] Step 2: Construct a deep neural network structure to detect and identify the document image to be detected.

[0058] The deep neural network structure includes a data input layer, a first residual downsampling layer, a second residual downsampling layer, a third residual downsampling layer, a fourth residual downsampling and global pooling layer, a fifth residual downsampling and global pooling layer, a sixth residual downsampling and global pooling layer, a fully connected layer and an output layer, and the output layer is connected to the hybrid loss function.

[0059] Step 3: Input the training sample set obtained in step 1 into the data input layer of the deep neural network structure constructed in step 2 to perform deep neural network training and obtain a detection and recognition model.

[0060] The loss function during training uses the hybrid loss function in step 2. The output of the hybrid loss function is fed back to the network, and the network is trained iteratively to minimize the hybrid loss function.

[0061] Step 4: Input the test sample set obtained in step 1 into the detection and recognition model to realize the detection and recognition of the document image to be tested containing 68 facial feature points, 3 facial posture angles and 1 glasses wearing state.

[0062] The embodiments of this specification provide a method for detecting and recognizing ID images. The method first obtains a training image to be detected and recognized, and divides the training image into a training sample set and a test sample set. A deep neural network structure is then constructed for detecting and recognizing the ID image to be detected. The obtained training sample set is then input into the data input layer of the constructed deep neural network structure for deep neural network training to obtain a detection and recognition model. Finally, the obtained test sample set is input into the detection and recognition model to detect and recognize the ID image to be detected, which includes 68 facial feature points, 3 facial posture angles, and 1 eyeglass wearing state. The present invention utilizes a deep neural network to implement hybrid estimation, resulting in high speed and accuracy. The method also simultaneously recognizes a user's facial feature points, facial posture angles, and eyeglass wearing state, and can provide a user with a reminder to obtain a photo that meets the ID requirements when the user takes the photo.

[0063] The embodiments of this specification also provide a method for detecting and identifying document images, including:

[0064] Step 1: Obtain a training image to be detected and identified, and divide the training image into a training sample set and a test sample set; the training image is an ID image to be detected including facial feature points, facial posture angles, and glasses wearing status.

[0065] Step 2: Construct a deep neural network structure to detect and identify the document image to be detected.

[0066] The deep neural network structure includes a data input layer, a first residual downsampling layer, a second residual downsampling layer, a third residual downsampling layer, a fourth residual downsampling and global pooling layer, a fifth residual downsampling and global pooling layer, a sixth residual downsampling and global pooling layer, a fully connected layer and an output layer, and the output layer is connected to the hybrid loss function.

[0067] Step 3: Input the training sample set obtained in step 1 into the data input layer of the deep neural network structure constructed in step 2 to perform deep neural network training and obtain a detection and recognition model.

[0068] Wherein, in said step 3, the training sample set obtained in step 1 is input into the data input layer of the deep neural network structure constructed in step 2 to perform deep neural network training to obtain a detection and recognition model, including:

[0069] Step 3.1: Perform residual downsampling three times on the training sample set input to the deep neural network to obtain the first feature map;

[0070] Step 3.2: Divide the first feature map obtained after step 3.1 into two paths. The first path continues to perform residual downsampling processing to obtain the second feature map; the second path performs global pooling processing to obtain the third feature map;

[0071] Step 3.3: Divide the second feature map obtained after step 3.2 into two paths. The first path continues to perform residual downsampling processing to obtain the fourth feature map; the second path performs global pooling processing to obtain the fifth feature map;

[0072] Step 3.4: Divide the fourth feature map obtained after processing in step 3.3 into two paths. The first path continues to perform residual downsampling processing to obtain the sixth feature map; the second path performs global pooling processing to obtain the seventh feature map;

[0073] Step 3.5: Copy and connect the pooled features in the third feature map, the fifth feature map, and the seventh feature map obtained by the three global pooling processes to obtain a multi-scale output feature vector in a fully connected manner.

[0074] The residual downsampling process includes using three downsampling methods: average pooling, maximum pooling and convolution, and adding the results of the three downsampling methods to obtain a residual downsampling structure.

[0075] The loss function during training uses the hybrid loss function in step 2. The output of the hybrid loss function is fed back to the network, and the network is trained iteratively to minimize the hybrid loss function.

[0076] Wherein, the hybrid loss function includes:

[0077]

[0078] in, is the mean square error loss function of 68 facial feature points, is the mean square error loss function of the three face pose angles, is the cross entropy loss function of the single-channel glasses wearing state, λ1, λ2 and λ3 are and The weighting coefficient of

[0079] The mean square error loss function of 68 facial feature points is defined as follows:

[0080]

[0081] in, and To predict the coordinates of the i-th facial feature point of the k-th face image, the coordinates of the i-th facial feature point of the k-th face image are predicted to be and is the coordinates of the i-th facial feature point in the k-th face image manually annotated, that is, the coordinates of the i-th facial feature point in the k-th face image manually annotated N is the total number of face images in the training sample set;

[0082] The mean square error loss function of the three face pose angles is defined as follows:

[0083]

[0084] in, and are the predicted value and labeled value of the pose angle pitch of the k-th face image, and are the predicted value and the labeled value of the posture angle yaw of the k-th face image, and are the predicted value and labeled value of the posture angle roll of the k-th face image respectively;

[0085] The cross entropy loss function of single-channel glasses wearing state is defined as follows:

[0086] When the predicted probability of wearing glasses is greater than or equal to 50%, that is, The loss function is defined as follows:

[0087]

[0088] in, and The predicted probability and labeling probability of glasses wearing in the k-th face image are respectively Can only take 0 and 1;

[0089] When the predicted probability of wearing glasses is less than 50%, that is, The loss function is defined as follows:

[0090]

[0091] The characteristic function is introduced to combine formulas (4) and (5). The characteristic function is defined as follows:

[0092]

[0093] Therefore, the final single-channel glasses wearing state cross entropy loss function is defined as:

[0094]

[0095] Step 4: Input the test sample set obtained in step 1 into the detection and recognition model to realize the detection and recognition of the document image to be tested containing 68 facial feature points, 3 facial posture angles and 1 glasses wearing state.

[0096] Among them, the size of the training image is 512x512x3, and after the first residual downsampling process, the size becomes 256x256x16, after the second residual downsampling process, the size becomes 128x128x32, and after the third residual downsampling process, it becomes a first feature map of size 64x64x48; the size of the second feature map is 32x32x64, the size of the third feature map is 1x1x64, the size of the fourth feature map is 16x16x64, the size of the fifth feature map is 1x1x64, the size of the sixth feature map is 8x8x96, and the size of the seventh feature map is 1x1x96. The feature vector after copying and connecting is 224x1 dimension, and the dimension of the output feature vector is 140x1 dimension.

[0097] The embodiments of this specification provide a method for detecting and recognizing ID images. The method first obtains a training image to be detected and recognized, and divides the training image into a training sample set and a test sample set. A deep neural network structure is then constructed for detecting and recognizing the ID image to be detected. The obtained training sample set is then input into the data input layer of the constructed deep neural network structure for deep neural network training to obtain a detection and recognition model. Finally, the obtained test sample set is input into the detection and recognition model to detect and recognize the ID image to be detected, which includes 68 facial feature points, 3 facial posture angles, and 1 eyeglass wearing state. The present invention utilizes a deep neural network to implement hybrid estimation, resulting in high speed and accuracy. The method also simultaneously recognizes a user's facial feature points, facial posture angles, and eyeglass wearing state, and can provide a user with a reminder to obtain a photo that meets the ID requirements when the user takes the photo.

[0098] like Figure 3As shown, the input image size of the training sample set of the deep neural network structure is 512×512×3, and the input image of the training sample set may include being obtained through a camera; after the first residual downsampling, it becomes 256×256×16, after the second residual downsampling, it becomes 128×128×32, and after the third residual downsampling, it becomes 64×64×48; starting from the fourth residual downsampling, the feature map is divided into two paths, one path continues to downsample, and the other path is globally pooled and then copied for connection; after the fourth residual downsampling, the feature map becomes 32×32×64, and the feature map after global pooling is 1×1×64; after the fifth residual downsampling, the feature map becomes 16×16×64, and the feature map after global pooling is 1×1×64; after the sixth residual downsampling, the feature map becomes 8×8×96, and the feature map after global pooling is 1×1×96; After copying and connecting the 3-way pooled features, a feature vector of (64+64+96)x1=224x1 dimension is obtained.

[0099] The use of three-way pooling of features at different scales is primarily to enhance scale invariance, thereby improving estimation and prediction accuracy. Since 68 facial feature points, three facial pose angles, and one eyeglass wearing state need to be estimated, the dimension of the output feature vector of the fully connected deep neural network should be 68 × 2 + 3 + 1 = 140. To reduce network parameters and computational complexity, this deep neural network uses one fully connected layer.

[0100] The method provided by the present invention can estimate and predict multiple different targets at one time, such as 68 facial feature points, 3 facial posture angles and 1 glasses wearing probability value; at the same time, it adopts residual downsampling, global pooling and other methods to realize a specialized end-to-end multi-target estimation network.

[0101] At the same time, in order to obtain a robust estimation deep neural network, the present invention designs special structures such as a residual structure, a multi-scale feature structure, and a skip-layer copy connection. Figure 5 As shown, the patent of the present invention adopts three downsampling methods: average pooling (avg_pool), maximum pooling (max_pool) and convolution (con) after input, batch normalization is performed after convolution downsampling, and then the results of the three downsampling methods are superimposed to output the residual downsampling structure; average pooling obtains feature mean information, while maximum pooling obtains the maximum feature value signal. These two pooling methods obtain different nonlinear information, which is very useful for classification and regression; however, the pooling operation is a non-learnable downsampling method; and the convolution downsampling method can obtain learning parameters, thereby realizing a learnable downsampling operation and realizing more flexible and data-dependent downsampling; the residual downsampling composed of these three methods greatly enhances the robustness and discrimination ability of the features.

[0102] like Figure 4 As shown in the figure, the multi-scale feature of the present invention is mainly reflected in the global pooling operation on the feature maps of different scales, and then the pooled features at different scales are concatenated, so that the multi-scale output feature vector is obtained; the replication and connection of pooled features at different scales embodies the idea of ​​skipping and crossing layers of features, which can greatly enhance the robustness of features.

[0103] The method provided by the present invention greatly enhances the prediction accuracy of the network and accelerates the network training speed; at the same time, the multi-scale pooling features and their cross-layer connections significantly enhance the scale invariance of the deep neural network, and can estimate facial feature points of different sizes, facial posture angles and glasses wearing status. At the same time, the combination of mean square error, cross entropy mixed loss function and single-channel eye wear cross entropy loss is used to unify the mixed loss function, accelerate the training speed, improve the prediction accuracy, and achieve end-to-end estimation of 68 feature points, 3 posture angles and 1 glasses wearing probability. The present invention uses a deep neural network to achieve hybrid estimation with high speed and high accuracy. At the same time, it can identify the user's facial feature points, facial posture angles and glasses wearing status, and can remind the user to obtain a document photo that meets the requirements when the user takes a photo.

[0104] It is understood that the method of the present invention can be executed by any device, equipment, platform, or equipment cluster with computing and processing capabilities, and produce beneficial effects corresponding to the system.

[0105] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Based on the concept of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of different aspects of one or more embodiments of the present specification as described above, which are not provided in detail for the sake of simplicity.

[0106] In addition, to simplify the description and discussion, and so as not to obscure one or more embodiments of the present specification, well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, devices may be shown in block diagram form to avoid obscuring one or more embodiments of the present specification, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which one or more embodiments of the present specification will be implemented (i.e., these details should be fully within the purview of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present disclosure, it will be apparent to those skilled in the art that one or more embodiments of the present specification may be implemented without these specific details or with variations in these specific details. Accordingly, these descriptions should be considered illustrative rather than restrictive.

[0107] Although the present disclosure has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0108] The one or more embodiments of this specification are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of this specification shall be included within the scope of protection of this disclosure.

Claims

1. A method for detecting and identifying document images, characterized in that: include: Step 1: Obtain training images to be detected and identified, and divide the training images into a training sample set and a test sample set; the training images are ID images that have been annotated with facial feature points, facial posture angles, and glasses wearing status; Step 2: Construct a deep neural network structure to detect and identify the document image to be detected; The deep neural network structure includes a data input layer, a first residual downsampling layer, a second residual downsampling layer, a third residual downsampling layer, a fourth residual downsampling and global pooling layer, a fifth residual downsampling and global pooling layer, a sixth residual downsampling and global pooling layer, a fully connected layer and an output layer, and the output layer is connected to the hybrid loss function; Step 3: Input the training sample set obtained in step 1 into the data input layer of the deep neural network structure constructed in step 2 to perform deep neural network training to obtain a detection and recognition model; The loss function during training adopts the hybrid loss function in step 2. The output of the hybrid loss function is fed back to the network, and the network is trained iteratively to minimize the hybrid loss function. The acquired detection and recognition model includes: Step 3.1: Perform residual downsampling three times on the training sample set input to the deep neural network to obtain the first feature map; Step 3.2: Divide the first feature map obtained after step 3.1 into two paths. The first path continues to perform residual downsampling processing to obtain the second feature map; the second path performs global pooling processing to obtain the third feature map; Step 3.3: Divide the second feature map obtained after step 3.2 into two paths. The first path continues to perform residual downsampling processing to obtain the fourth feature map; the second path performs global pooling processing to obtain the fifth feature map; Step 3.4: Divide the fourth feature map obtained after processing in step 3.3 into two paths. The first path continues to perform residual downsampling processing to obtain the sixth feature map; the second path performs global pooling processing to obtain the seventh feature map; Step 3.5: Copy and connect the pooled features in the third feature map, the fifth feature map, and the seventh feature map obtained by the three global pooling processes to obtain a multi-scale output feature vector in a fully connected manner; Step 4: Input the test sample set obtained in step 1 into the detection and recognition model to realize the detection and recognition of the document image to be tested containing 68 facial feature points, 3 facial posture angles and 1 glasses wearing state.

2. The method for detecting and identifying document images according to claim 1, characterized in that: The residual downsampling process includes using three downsampling methods: average pooling, maximum pooling and convolution, and adding the results of the three downsampling methods to obtain a residual downsampling structure.

3. The method for detecting and identifying document images according to claim 2, characterized in that: The hybrid loss function includes: l=λ1l1+λ2l2+λ3l3 (1) Among them, l1 is the mean square error loss function of 68 facial feature points, l2 is the mean square error loss function of 3 facial posture angles, l3 is the cross entropy loss function of single-channel glasses wearing state, λ1, λ2 and λ3 are the weighting coefficients of l1, l2 and l3 respectively; The mean square error loss function of 68 facial feature points is defined as follows: in, and To predict the coordinates of the i-th facial feature point in the k-th face image, and is the coordinate of the i-th facial feature point in the k-th face image manually labeled, and N is the total number of face images in the training sample set; The mean square error loss function of the three face pose angles is defined as follows: in, and are the predicted value and labeled value of the pose angle pitch of the k-th face image, and are the predicted value and the labeled value of the posture angle yaw of the k-th face image, and are the predicted value and labeled value of the posture angle roll of the k-th face image respectively; The cross entropy loss function of single-channel glasses wearing state is defined as follows: When the predicted probability of wearing glasses is greater than or equal to 50%, that is, The loss function is defined as follows: in, and The predicted probability and labeling probability of glasses wearing in the k-th face image are respectively Can only take 0 and 1; When the predicted probability of wearing glasses is less than 50%, that is, The loss function is defined as follows: The characteristic function is introduced to combine formulas (4) and (5). The characteristic function is defined as follows: Therefore, the final single-channel glasses wearing state cross entropy loss function is defined as:

4. The method for detecting and identifying document images according to claim 3, wherein: The size of the training image is 512x512x3. After the first residual downsampling process, the size becomes 256x256x16, after the second residual downsampling process, the size becomes 128x128x32, and after the third residual downsampling process, it becomes a first feature map of size 64x64x48; the size of the second feature map is 32x32x64, the size of the third feature map is 1x1x64, the size of the fourth feature map is 16x16x64, the size of the fifth feature map is 1x1x64, the size of the sixth feature map is 8x8x96, the size of the seventh feature map is 1x1x96, the feature vector after copying and connecting is 224x1 dimension, and the dimension of the output feature vector is 140x1 dimension.

Images