Deep learning method and system for identifying global race
Through the optimization of deep learning methods and CAFormer network model, combined with manual annotation and full connection layer design, the problem of uneven recognition capabilities in the race classification method is solved, more accurate face detection and race classification are achieved, and more Accurate identification solution.
Patent Information
- Application Number
- CN202411445894.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-16
- Publication Date
- 2025-01-14
AI Technical Summary
The existing race classification method is not accurate enough to extract the characteristics of different races, resulting in classification errors, and the training data may be biased. The number of samples in some races is too large or too small, resulting in the uneven recognition ability of the model for different races.
Deep learning methods, including face detection, manual annotation and model training, use CAFormer network model, train the model through manual annotation data, and add a new fully connected layer to the model to improve recognition accuracy, including Li near-ReLU -Dropout and Li near-Sigmoid layers to prevent overfitting and deploy the final model on a server or mobile device for face race recognition.
It realizes more accurate face detection and race classification, reduces the situation of missed detection and classification errors, and provides more accurate and effective race recognition solutions for various fields.
Smart Images
Figure CN119314216A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision and artificial intelligence technology, and in particular to a deep learning method and system for identifying global human races. Background Art
[0002] In terms of racial classification, existing methods may not accurately extract features of different races, resulting in classification errors. The training data of some methods may be biased, for example, the number of samples of certain races in the collected data set is too large or too small, resulting in uneven recognition of different races by the model. Summary of the invention
[0003] The purpose of the present invention is to provide a deep learning method and system for identifying global ethnic groups, thereby solving the above-mentioned problems existing in the prior art.
[0004] In order to achieve the above object, the technical solution adopted by the present invention is as follows:
[0005] A deep learning method for identifying global ethnic groups includes the following steps:
[0006] S1. Face detection: for the collected video files, perform face detection on each frame of each video file, crop the detected face area, and save it as a face image file;
[0007] S2, manual labeling: manually label the face image files to classify the face images into ethnic groups;
[0008] S3. Model training: Use manually annotated face image files to train, verify and test the ethnic classification model built based on the CAFormer network model to obtain the ethnic classification model with the best performance.
[0009] Preferably, step S1 specifically includes the following contents:
[0010] S11, select a suitable face detection algorithm;
[0011] S12, for each video file, reading the video picture frame by frame;
[0012] S13, inputting each frame of image into a face detection algorithm, and preprocessing each frame of image;
[0013] S14, the face detection algorithm uses a shallow convolutional neural network to quickly scan the input image and output a series of candidate windows that may contain faces;
[0014] S15, make a more accurate judgment on each candidate window, remove the misdetected windows, and then use a deep convolutional neural network to output the bounding box coordinates of the face and the positions of the five facial key points;
[0015] S16. According to the bounding box coordinates, a face region is captured from the original image and saved as a separate face image file.
[0016] Preferably, step S2 specifically includes that a labeling staff checks the face image files one by one, determines the race of the person in the face image according to the skin color and facial features of the person in the face image, and classifies each face image into a race category label for labeling;
[0017] The racial categories include white race, black race, yellow race, and brown race.
[0018] Preferably, step S3 specifically includes the following contents:
[0019] S31, sorting and dividing the manually annotated face image files into a training set, a validation set, and a test set;
[0020] S32, input the training data set into the ethnic classification model, use the back propagation algorithm and the optimization algorithm to update the parameters of the model, in each iteration the model makes a prediction based on the input data, and then calculates the loss function between the prediction result and the true annotation; then the gradient of the loss function to the model parameters is calculated by the back propagation algorithm, and then the optimization algorithm is used to update the model parameters according to the gradient, so that the loss function is gradually reduced;
[0021] S33. During the training process, the validation set is used regularly to verify and evaluate the model. The performance and generalization ability of the model can be understood by calculating the evaluation indicators on the validation set. If the performance of the model on the validation set no longer improves, training is stopped to prevent overfitting.
[0022] S34. After the training is completed, use the test set to evaluate multiple trained models and select the model with the best performance as the final ethnic classification model.
[0023] Preferably, the ethnic classification model uses the CAFormer network model as a basic model, cancels the original classification layer, and adds a new fully connected layer; the new connection layer is composed of four groups of Li near-ReLU-Dropout and one group of Li near-Si gmoid;
[0024] Li near-ReLU-Dropout consists of Li near layer, ReLU activation function and Dropout regularization;
[0025] The Linear layer multiplies the input data by the weight matrix and adds the bias term to perform a linear transformation.
[0026] The linearly transformed data is processed through the ReLU activation function to increase the model's ability to express nonlinearity, allowing the model to learn more complex features;
[0027] Dropout regularization randomly sets the output of some neurons to 0 during training, so that the model cannot be overly dependent on certain neurons and prevents the model from overfitting.
[0028] Li near-Si gmoid is composed of Li near layer and Sigmoid activation function;
[0029] The sigmoid function compresses the output value of the Line layer between 0 and 1 and normalizes the model output so that the output result represents the probability of each category.
[0030] Preferably, after step S3, the method further includes:
[0031] S4. Model weight deployment: Save the weight parameters of the final race classification model and store it in a specific file format; according to the needs of actual applications, deploy the final race classification model on a server or mobile device to achieve face race recognition.
[0032] Preferably, after step S4, the method further includes:
[0033] S5. Model reasoning: Use the final race classification model to perform race recognition on face images.
[0034] Preferably, step S5 specifically includes the following contents:
[0035] S51, the user submits a picture with one or more faces, where the picture is a real-time photo taken by a camera or a picture file selected from an album;
[0036] S52, using a face detection model to perform face detection on the image submitted by the user, and determining the number and positions of faces in the image;
[0037] S53, inputting the image of the detected face into the final race classification model to determine the race to which each face belongs;
[0038] S54. The model outputs the classification results, which include the number of faces contained in the image, the location coordinates of each face, and the ethnic category to which it belongs. The results will be displayed to the user in a visual manner.
[0039] Preferably, before step S1, the step further includes:
[0040] S0. Video data collection: Filter video files from various countries from the material library. The video files need to contain people of different races or skin colors from multiple countries.
[0041] The present invention also aims to provide a deep learning system for identifying global ethnic groups, the system can implement the above-mentioned method, and the system includes:
[0042] Video data acquisition module: Filter video files from various countries from the material library. The video files need to contain people of different races or skin colors from multiple countries.
[0043] Face detection module: for the collected video files, perform face detection on each frame of each video file, crop the detected face area, and save it as a face image file;
[0044] Manual labeling module: manually label the face image files to classify the face images into ethnic groups;
[0045] Model training module: Use manually annotated face image files to train, verify and test the ethnic classification model built based on the CAFormer network model to obtain the ethnic classification model with the best performance;
[0046] Model weight deployment module: saves the weight parameters of the final ethnic classification model and stores them in a specific file format; deploys the final ethnic classification model on a server or mobile device according to actual application requirements to achieve face and ethnic recognition;
[0047] Model reasoning module: Use the final race classification model to perform race recognition on face images.
[0048] The beneficial effects of the present invention are: compared with the existing technology, the present invention can more accurately detect faces and classify races through advanced algorithms and new fully connected layer designs, reduce false detections and missed detections and classification errors, and provide more accurate and effective race recognition solutions for various fields. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] Figure 1 is a flow chart of a method in an embodiment of the present invention;
[0050] Figure 2 is a flow chart of face detection in an embodiment of the present invention;
[0051] Figure 3 It is a flowchart of model training in an embodiment of the present invention. DETAILED DESCRIPTION
[0052] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation methods described herein are only used to explain the present invention and are not used to limit the present invention.
[0053] like Figure 1 As shown, in this embodiment, a deep learning method for identifying global ethnic groups is provided. The method starts from the collection of video materials of ethnic groups from multiple countries, and is deployed and inferred after accurate face detection, detailed manual annotation and specific neural network training. The method includes the following parts:
[0054] 1. Video Data Collection
[0055] Filter video files from various countries from the material library. The video files need to have people of different races or skin colors from different countries. The specific content includes the following:
[0056] 1.1. First, determine the screening criteria. According to the common regional distribution of different races, determine some keywords for the main video that may contain multiple races, such as "international sports events", "world tourist attractions", etc. At the same time, check the classification labels of the material library to find classifications related to different countries and regions.
[0057] 1.2. Browse the videos in the library one by one, and observe whether the characters in the video have different skin color characteristics, such as white, black, yellow, brown, etc. through quick preview and other methods.
[0058] 1.3. Select the videos that meet the requirements and prepare for the subsequent face detection steps.
[0059] 2. Face Detection
[0060] For the collected video files, face detection is performed on each frame of each video file, and the detected face area is cropped and saved as a face image file. Figure 2 As shown, it specifically includes the following contents:
[0061] 2.1. Select a suitable face detection algorithm; such as face detection algorithms based on deep learning, such as MTCNN (Multi-task Cascaded Convolutional Networks).
[0062] 2.2. For each video file, read the video frame by frame;
[0063] 2.3. Input each frame of image into the face detection algorithm and preprocess each frame of image, including resizing, normalization and other operations.
[0064] 2.4. The face detection algorithm uses a shallow convolutional neural network to quickly scan the input image and output a series of candidate windows that may contain faces.
[0065] 2.5. Accurately adjust and output the results of 2.4, make more accurate judgments on each candidate window, remove misdetected windows, and then use a deep convolutional neural network to output the bounding box coordinates of the face and the positions of the five facial key points.
[0066] 2.6. Based on the bounding box coordinates, capture the face area from the original image and save it as a separate face image file.
[0067] 3. Manual Labeling
[0068] Manually annotate face image files to classify the face images into different ethnic groups.
[0069] Specifically: the labeling personnel check the face image files one by one, judge the race of the people in the face images according to the skin color, facial features, etc., and classify each face image into a race category label for labeling; the race categories include white, black, yellow, and brown.
[0070] 4. Model Training
[0071] The human race classification model built based on the CAFormer network model is trained, verified and tested using manually annotated face image files to obtain the human race classification model with the best performance. Figure 3 As shown, it specifically includes the following contents:
[0072] 4.1. Data preparation: The manually annotated face image data is sorted and divided into training set, validation set and test set. The training set is used for model training, the validation set is used to adjust the model hyperparameters and evaluate the model performance during the training process, and the test set is used to evaluate the final performance of the model after the model training is completed.
[0073] 4.2. Model network adjustment: Using the CAFormer network model as the basic model, the original classification layer was abolished, and a new fully connected layer was added to build a human race classification model. This new connection layer consists of four sets of Line-ReLU-Dropout and one set of Line-Sigmoid, which expands the 768-length matrix output by the original model to 2048 and then compresses it into one output.
[0074] Linite-ReLU-Dropout consists of Linite layer, ReLU activation function and Dropout regularization. Linear-Sigmoid consists of Linite layer and Sigmoid activation function.
[0075] (1) Linear layer: Linear transformation layer, which multiplies the input data by the weight matrix and adds the bias term to perform a linear transformation. This is to extract and combine features of the input data and prepare for subsequent nonlinear transformation.
[0076] (2) ReLU activation function: After linear transformation, the data is processed by the ReLU function to increase the model's nonlinear expression ability, allowing the model to learn more complex features.
[0077] Dropout regularization: During the training process, the outputs of some neurons are randomly set to 0, so that the model does not rely too much on certain neurons, preventing the model from overfitting and improving the generalization ability and robustness of the model.
[0078] (4) The sigmod function compresses the upper layer output value between 0 and 1 and normalizes the model output so that the output result represents the probability of each category.
[0079] 4.3. Training process: Input the training set data into the adjusted model, and use the back propagation algorithm and optimization algorithm to update the model parameters. In each iteration, the model makes predictions based on the input data, and then calculates the loss function between the prediction results and the true annotations. Then, the gradient of the loss function to the model parameters is calculated through the back propagation algorithm, and then the optimization algorithm is used to update the model parameters according to the gradient, so that the loss function gradually decreases.
[0080] 4.4. Verification and evaluation: During the training process, the validation set data is used regularly to verify and evaluate the model. By calculating indicators such as the loss function and accuracy on the validation set, the performance and generalization ability of the model can be understood. If the performance of the model on the validation set no longer improves, it may be necessary to stop training to prevent overfitting.
[0081] 4.4 Model selection: After training is completed, use the test set data to evaluate multiple trained models and select the model with the best performance as the final ethnic classification model.
[0082] 5. Model Weight Deployment
[0083] The weight parameters of the final ethnic classification model are saved and stored in a specific file format; according to the actual application requirements, the final ethnic classification model is deployed on a server or mobile device to achieve face ethnic recognition. Specifically, it includes the following contents:
[0084] 5.1. Save the trained model weight parameters and store them in a specific file format.
[0085] 5.2. Choose the appropriate deployment method based on the actual application needs. For example, you can deploy the model on a server and provide services through an API interface; or deploy the model on a mobile device to implement local face and ethnicity recognition.
[0086] (1) Server deployment: The model is deployed on the server and services are provided through the API interface. Users can send requests to the server. After receiving the request, the server uses the deployed model to process the input data and returns the result. This method is suitable for scenarios that require processing large amounts of data and high concurrent requests.
[0087] (2) Mobile device deployment: The model is deployed on mobile devices, such as smartphones and tablets. A dedicated mobile deep learning framework can be used to convert the model into a format suitable for mobile devices and perform inference on the mobile device. This method is suitable for scenarios where real-time face and ethnicity recognition is required on mobile devices.
[0088] 5.3. Ensure that the deployment environment has the required software and hardware conditions, such as installing the corresponding deep learning framework and configuring sufficient computing resources.
[0089] 6. Model Reasoning
[0090] Use the final ethnic classification model to perform ethnic recognition on face images. The specific contents include the following:
[0091] 6.1. Users submit pictures with one or more faces, which can be real-time photos taken by a camera or picture files selected from the album.
[0092] 6.2. The face detection model first performs face detection on the input image. Similar to the face detection step in the training process, the algorithm searches for the location of the face in the input image and outputs the bounding box coordinates of the face to determine the number and location of the face.
[0093] 6.3. For each detected face, input it into the trained race classification model. The model uses the trained neural network to perform race classification. It first extracts and transforms features through a series of convolutional layers and fully connected layers, and then outputs the race category probability through the final classification layer. The race of the face is determined based on the category with the highest probability.
[0094] 6.4. The output results include the number of faces, the position coordinates of each face, and the race category. The results will be displayed to the user in a visual way. The position coordinates can be represented by the coordinates of the upper left corner and lower right corner of the bounding box, and the race category is output in the form of a specific label, such as "white race", "black race", etc.
[0095] In this embodiment, a deep learning system for identifying global ethnic groups is also provided. The system can implement the above-mentioned method. The system includes:
[0096] (1) Video data acquisition module: Filter video files from various countries from the material library. The video files need to contain people of different races or skin colors from multiple countries.
[0097] (2) Face detection module: For the collected video files, face detection is performed on each frame of each video file, and the detected face area is cropped and saved as a face image file.
[0098] (3) Manual labeling module: Manually label the face image files to classify the face images into ethnic groups.
[0099] (4) Model training module: Use manually annotated face image files to train, verify and test the ethnic classification model built based on the CAFormer network model to obtain the ethnic classification model with the best performance.
[0100] (5) Model weight deployment module: save the weight parameters of the final ethnic classification model and store them in a specific file format; deploy the final ethnic classification model on a server or mobile device according to the actual application requirements to achieve face ethnic recognition.
[0101] (6) Model reasoning module: Use the final race classification model to perform race recognition on face images.
[0102] By adopting the above technical solution disclosed in the present invention, the following beneficial effects are obtained:
[0103] The present invention provides a deep learning method and system for identifying global ethnic groups. Compared with the existing technology, the present invention can more accurately detect faces and classify ethnic groups through advanced algorithms and new fully connected layer designs, reduce false detections, missed detections and classification errors, and provide more accurate and effective ethnic group recognition solutions for various fields.
[0104] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be considered as the scope of protection of the present invention.
Claims
1. A deep learning method for identifying global ethnic groups, characterized by: The following steps are included: S1. Face detection: for the collected video files, perform face detection on each frame of each video file, crop the detected face area, and save it as a face image file; S2, manual labeling: manually label the face image files to classify the face images into ethnic groups; S3. Model training: Use manually annotated face image files to train, verify and test the ethnic classification model built based on the CAFormer network model to obtain the ethnic classification model with the best performance.
2. The deep learning method for identifying global ethnic groups according to claim 1, characterized in that: Step S1 specifically includes the following contents: S11, select a suitable face detection algorithm; S12, for each video file, reading the video picture frame by frame; S13, inputting each frame of image into a face detection algorithm, and preprocessing each frame of image; S14, the face detection algorithm uses a shallow convolutional neural network to quickly scan the input image and output a series of candidate windows that may contain faces; S15, make a more accurate judgment on each candidate window, remove the misdetected windows, and then use a deep convolutional neural network to output the bounding box coordinates of the face and the positions of the five facial key points; S16. According to the bounding box coordinates, a face region is captured from the original image and saved as a separate face image file.
3. The deep learning method for identifying global ethnic groups according to claim 1, characterized in that: Step S2 specifically includes: a labeler checks the face image files one by one, determines the race of the person in the face image according to the skin color and facial features of the person in the face image, and labels each face image with a race category label; The racial categories include white race, black race, yellow race, and brown race.
4. The deep learning method for identifying global ethnic groups according to claim 1, characterized in that: Step S3 specifically includes the following contents: S31, sorting and dividing the manually annotated face image files into a training set, a validation set, and a test set; S32, input the training data set into the ethnic classification model, use the back propagation algorithm and the optimization algorithm to update the parameters of the model, in each iteration the model makes a prediction based on the input data, and then calculates the loss function between the prediction result and the true annotation; then the gradient of the loss function to the model parameters is calculated by the back propagation algorithm, and then the optimization algorithm is used to update the model parameters according to the gradient, so that the loss function is gradually reduced; S33. During the training process, the validation set is used regularly to verify and evaluate the model. The performance and generalization ability of the model can be understood by calculating the evaluation indicators on the validation set. If the performance of the model on the validation set no longer improves, training is stopped to prevent overfitting. S34. After the training is completed, use the test set to evaluate multiple trained models and select the model with the best performance as the final ethnic classification model.
5. The deep learning method for identifying global ethnic groups according to claim 4, characterized in that: The ethnic classification model uses the CAFormer network model as the basic model, abolishes the original classification layer, and adds a new fully connected layer; the new connection layer consists of four groups of Linear-ReLU-Dropout and one group of Linear-Sigmoid; Linear-ReLU-Dropout consists of a Linear layer, a ReLU activation function, and Dropout regularization; The Linear layer multiplies the input data by the weight matrix and adds the bias term to perform a linear transformation on it. The linearly transformed data is processed through the ReLU activation function to increase the model's ability to express nonlinearity, allowing the model to learn more complex features; Dropout regularization randomly sets the output of some neurons to 0 during training, so that the model cannot be overly dependent on certain neurons and prevents the model from overfitting. Linear-Sigmoid consists of a Linear layer and a Sigmoid activation function; The sigmoid function compresses the output value of the Linear layer between 0 and 1 and normalizes the model output so that the output result represents the probability of each category.
6. The deep learning method for identifying global ethnic groups according to claim 1, characterized in that: Step S3 also includes: S4, model weight deployment: save the weight parameters of the final ethnic classification model and store them in a specific file format; According to the actual application requirements, the final ethnic classification model is deployed on the server or mobile device to realize face ethnic recognition.
7. The deep learning method for identifying global ethnic groups according to claim 6, characterized in that: Step S4 also includes: S5. Model reasoning: Use the final race classification model to perform race recognition on face images.
8. The deep learning method for identifying global ethnic groups according to claim 7, characterized in that: Step S5 specifically includes the following contents: S51, the user submits a picture with one or more faces, where the picture is a real-time photo taken by a camera or a picture file selected from an album; S52, using a face detection model to perform face detection on the image submitted by the user, and determining the number and positions of faces in the image; S53, inputting the image of the detected face into the final race classification model to determine the race to which each face belongs; S54. The model outputs the classification results, which include the number of faces contained in the image, the location coordinates of each face, and the ethnic category to which it belongs. The results will be displayed to the user in a visual manner.
9. The deep learning method for identifying global ethnic groups according to claim 1, characterized in that: Before step S1, the method further includes: S0. Video data collection: Filter video files from various countries from the material library. The video files need to contain people of different races or skin colors from multiple countries.
10. A deep learning system for identifying global ethnicity, characterized by: The system can implement the method described in any one of claims 1 to 9, and the system includes: Video data acquisition module: Filter video files from various countries from the material library. The video files need to contain people of different races or skin colors from multiple countries. Face detection module: for the collected video files, perform face detection on each frame of each video file, crop the detected face area, and save it as a face image file; Manual labeling module: manually label the face image files to classify the face images into ethnic groups; Model training module: Use manually annotated face image files to train, verify and test the ethnic classification model built based on the CAFormer network model to obtain the ethnic classification model with the best performance; Model weight deployment module: saves the weight parameters of the final ethnic classification model and stores them in a specific file format; deploys the final ethnic classification model on a server or mobile device according to actual application requirements to achieve face and ethnic recognition; Model reasoning module: Use the final race classification model to perform race recognition on face images.