A facial key point recognition method based on multi-visual depth information

Abstract

The present invention provides a method for recognizing key points of faces based on multi-visual depth information, and performs feature fusion of RGB image features and multi-visual depth image features to construct multi-visual depth image fusion features, and inputs the multi-visual depth image fusion features into a preset ResNet model for feature training and learning, and constructs a multi-visual depth detection model corresponding to the recognition of key points of faces, and the multi-visual depth detection model recognizes and outputs the recognition results of key points of faces on face images. On the basis of the existing key point recognition model, the depth information of the image can be added for retraining and optimization to achieve higher accuracy of recognition of key points of faces. Based on the ResNet model structure, the present invention increases depth data and optimizes the network structure to make it more suitable for key point recognition tasks through depth data. The addition of depth information can provide additional three-dimensional spatial features and enhance the recognition ability of the network.

Description

Technical Field

[0001] The present invention relates to the field of intelligent recognition technology, and in particular to a method and device for recognizing key points of a face based on multi-visual depth information, an electronic device and a computer-readable storage medium. Background Art

[0002] Among the current traditional technologies, the common facial key point recognition technologies are:

[0003] Based on traditional algorithms, such as Active Shape Model (ASM), an average shape model is constructed through a large number of annotated face images based on statistical shape models. During detection, the input image is matched with the average shape model. Problems: Insufficient robustness: sensitive to lighting changes, occlusion, posture changes, etc., and the detection effect is unstable;

[0004] Based on neural network algorithms, such as Convolutional Neural Networks (CNNs), CNNs automatically extract image features through a hierarchical structure and can capture image information at different levels, thus performing well in key point detection. However, a large amount of labeled data is required for training, and the model design and training process are complex, requiring careful adjustment of parameters and structure.

[0005] When designing skin morphology, it is often necessary to design and identify some special facial key points, such as the nose tip. However, in the identification of the nose tip, the RGB features of the nose tip are not obvious enough. The above traditional technologies are usually not accurate enough (the accuracy is not high enough) to identify these facial key feature points. The lack of deep information features of facial key points leads to greater errors in the identified facial key points, and there are many differences between the model and reality. Summary of the invention

[0006] In order to solve the technical problems existing in the prior art, the present invention provides the following technical solutions:

[0007] On the one hand, a method for recognizing key points of a face based on multi-visual depth information is provided, the method being implemented by an electronic device, the method comprising:

[0008] S1. Collect big image data of facial key points, including RGB data and multi-vision depth scanning data;

[0009] S2, performing feature engineering on the image big data of the facial key points, respectively extracting RGB image features and multi-vision depth image features corresponding to the RGB data and the multi-vision depth scanning data;

[0010] S3, feature fusion of the RGB image features and the multi-vision depth image features to construct a multi-vision depth image fusion feature, and input the multi-vision depth image fusion feature into a preset ResNet model to perform feature training and learning, and construct a multi-vision depth detection model corresponding to the recognition of face key points;

[0011] S4. Collect a facial image of the user and input it into the multi-vision depth detection model, and the multi-vision depth detection model recognizes and outputs the recognition result of facial key points on the facial image.

[0012] Furthermore, in step S1, the multi-vision depth scanning data includes scanning data under at least one of the following visions:

[0013] The surrounding environment data of facial key points;

[0014] Axial depth data of facial key points in XYZ three-dimensional space;

[0015] or,

[0016] Point cloud data of facial key points.

[0017] Further, S2, feature engineering is performed on the image big data of the facial key points, and RGB image features and multi-visual depth image features corresponding to the RGB data and the multi-visual depth scanning data are extracted respectively, including:

[0018] Performing image preprocessing on the RGB data and the multi-vision depth scan data respectively;

[0019] Using a ResNet model to extract RGB image features of the RGB data;

[0020] Using a CNN model, sequentially extracting depth image features of the depth scan data under each vision in the multi-vision depth scan data, and obtaining a depth scan feature set composed of the depth image features under each vision;

[0021] The RGB image features and the depth scan feature set are saved separately to a background database.

[0022] Further, S3, the RGB image features and the multi-vision depth image features are feature-fused to construct a multi-vision depth image fusion feature, and the multi-vision depth image fusion feature is input into a preset ResNet model for feature training and learning to construct a multi-vision depth detection model corresponding to the recognition of face key points, including:

[0023] Traversing the depth scanning feature set, and sequentially extracting the depth image features under each visual sense in the depth scanning feature set;

[0024] Input the extracted depth image features under each visual condition and the RGB image features into the deep convolutional layer of the CNN model;

[0025] On the deep convolution layer, by convolutionally fusing the corresponding feature elements on the key points of the face, the deep image features and the RGB image features are fused on the convolution layer to obtain the convolution fusion features under each visual sense:

[0026] According to the above method, the multi-vision depth image fusion feature composed of the convolution fusion features under each vision is obtained, that is, RGBD data;

[0027] The RGBD data is preprocessed, and the preprocessed RGBD data is input into a preset ResNet model, feature training and learning of the multi-visual depth image fusion features in the RGBD data are performed based on the ResNet network, and the multi-visual depth detection model is constructed and generated.

[0028] Further, S3, the RGB image features and the multi-vision depth image features are feature-fused to construct RGBD data consisting of multi-vision depth image fusion features, and the RGBD data is input into a preset ResNet model to perform feature training and learning, and a multi-vision depth detection model corresponding to the recognition of face key points is constructed, including:

[0029] Traversing the depth scanning feature set, and sequentially extracting the depth image features under each visual sense in the depth scanning feature set;

[0030] Input the extracted depth image features under each visual condition and the RGB image features into the deep convolutional layer of the CNN model;

[0031] On the deep convolution layer, by convolutionally fusing the corresponding feature elements on the key points of the face, the deep image features and the RGB image features are fused on the convolution layer to obtain the convolution fusion features under the current vision:

[0032] According to the above steps, the depth image feature under the next vision is subjected to feature fusion on the convolution layer by combining the feature elements on the depth image feature under the next vision with the conveyed convolution fusion feature under the current vision to obtain the convolution fusion feature under the next vision:

[0033] According to the above method, the depth image features under the remaining vision are sequentially fused to obtain the final convolution fusion features;

[0034] Preprocessing the final convolution fusion features, inputting the preprocessed final convolution fusion features into a preset ResNet model, and performing feature training and learning on the final convolution fusion features based on the ResNet network;

[0035] The final convolutional fusion features are preprocessed, and the preprocessed final convolutional fusion features are input into a preset ResNet model, and feature training and learning of the final convolutional fusion features are performed based on the ResNet network to construct and generate the multi-visual depth detection model.

[0036] On the other hand, a face key point recognition device based on multi-visual depth information is provided, and the face key point recognition device based on multi-visual depth information is used to implement the face key point recognition method based on multi-visual depth information, and the device includes:

[0037] An image acquisition module is used to acquire a user's face image and input it into a face key point recognition module;

[0038] A facial key point recognition module, used to recognize and output the recognition result of the facial key points on the facial image using a pre-trained multi-vision depth detection model;

[0039] The image acquisition module is communicatively connected with the facial key point recognition module.

[0040] On the other hand, an electronic device is provided, comprising: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned methods for facial key point recognition based on multi-visual depth information is implemented.

[0041] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned methods for facial key point recognition based on multi-visual depth information.

[0042] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0043] The present invention collects image big data of facial key points, including RGB data and multi-visual depth scanning data; performs feature engineering on the image big data of facial key points, extracts RGB image features and multi-visual depth image features corresponding to RGB data and multi-visual depth scanning data respectively; performs feature fusion on the RGB image features and the multi-visual depth image features to construct multi-visual depth image fusion features, and inputs the multi-visual depth image fusion features into a preset ResNet model for feature training and learning, constructs a multi-visual depth detection model corresponding to the recognition of facial key points, and the multi-visual depth detection model recognizes and outputs the recognition results of facial key points on the facial image. On the basis of the existing key point recognition model, the depth information of the image can be added for retraining and optimization to achieve higher facial key point recognition accuracy.

[0044] Based on the ResNet model structure, the present invention increases depth data and optimizes the network structure to make it more suitable for key point recognition tasks through depth data. The addition of depth information can provide additional three-dimensional spatial features and enhance the recognition ability of the network.

[0045] The present invention can provide the following improvements, thereby further improving the accuracy of the model:

[0046] 1. Supplement visual information and improve robustness: Depth information combined with RGB images can provide richer environmental information and help the model identify key points more accurately in complex scenes. Depth information can reduce the impact of lighting changes, shadows, and color similarities on the model, making the model more robust in different environments. The geometric features provided by depth data are more stable and less susceptible to external interference, thereby improving the overall performance of the model.

[0047] 2. Multimodal fusion: Fusion of depth information and RGB image features can improve the richness and differentiation of feature expression. Through the synergy of multimodal data, the model can more comprehensively understand and analyze the input data and improve the accuracy of key point recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0049] Figure 1 This is a flow chart of a method for recognizing key points of a face based on multi-visual depth information provided by an embodiment of the present invention;

[0050] Figure 2 This is a schematic diagram of a ResNet model training process provided by an embodiment of the present invention;

[0051] Figure 3 It is a fusion training intent of a convolution fusion feature provided by an embodiment of the present invention;

[0052] Figure 4 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0054] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0055] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0056] In the embodiments of the present invention, sometimes the subscripts such as W 1 It may be mistakenly written as a non-subscript form such as W1. When the difference is not emphasized, the meanings they express are the same.

[0057] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0058] The key points of the face of the present invention, in addition to the nose tip point, can also be the eye corner point, mouth corner point, eyebrow peak point, the highest point of the cheekbone, etc.

[0059] The embodiment of the present invention provides a method for recognizing key points of a face based on multi-visual depth information, which can be implemented by an electronic device, which can be a terminal or a server. Figure 1 The flowchart of the method for recognizing key points of faces based on multi-visual depth information is shown in FIG. 1 , and the processing flow of the method may include the following steps:

[0060] S1. Collect big image data of facial key points, including RGB data and multi-vision depth scanning data;

[0061] S2, performing feature engineering on the image big data of the facial key points, respectively extracting RGB image features and multi-vision depth image features corresponding to the RGB data and the multi-vision depth scanning data;

[0062] S3, feature fusion of the RGB image features and the multi-vision depth image features to construct a multi-vision depth image fusion feature, and input the multi-vision depth image fusion feature into a preset ResNet model to perform feature training and learning, and construct a multi-vision depth detection model corresponding to the recognition of face key points;

[0063] S4. Collect a facial image of the user and input it into the multi-vision depth detection model, and the multi-vision depth detection model recognizes and outputs the recognition result of facial key points on the facial image.

[0064] Based on the 2D key point algorithm model ResNet, the present invention further improves the key point recognition accuracy by adding depth data. By combining depth information with RGB images, richer environmental information can be provided to help the model more accurately identify key points in complex scenes. Depth information can reduce the impact of lighting changes, shadows and color similarities on the model, making the model more robust in different environments. The geometric features provided by depth data are more stable and less susceptible to external interference, thereby improving the overall performance of the model.

[0065] When it is implemented, the recognition result of facial key points not only needs to identify its three-dimensional spatial coordinates, but also needs to identify the depth information of its key points, such as spatial depth information under three axes, point cloud information, surrounding environment including illumination, color, shadow and other depth color feature information, etc. By collecting and fusing multi-visual depth features to train the model, the model can be used to further identify and output multi-dimensional feature data of key points on the face image, output facial key points more quickly and conveniently, improve the efficiency of face shape design, and strengthen and optimize the face shape design effect.

[0066] During implementation, ROI construction can also be performed on the part of the image where the key points of the face are located, ROI area annotation can be performed, and the depth information of the surrounding environment and depth feature extraction can be performed on the ROI area where the key points of the face are located, so that the model can recognize the depth information of the part where the key points of the face are located and output the results, so that the recognition accuracy of the key points of the face and the environmental features are further optimized. The depth image of the ROI area is obtained by scanning to extract features (lighting, color, shadows, stripes, scars, etc.) on the depth image, so as to extract the skin morphology depth features of the ROI area of ​​the user's face key points, so that the extracted face image is more in line with the actual situation of the user and improves practicality. ROI construction can be automatically constructed or manually constructed.

[0067] Furthermore, in step S1, the multi-vision depth scanning data includes scanning data under at least one of the following visions:

[0068] The surrounding environment data of facial key points;

[0069] Axial depth data of facial key points in XYZ three-dimensional space;

[0070] or,

[0071] Point cloud data of facial key points.

[0072] There are several ways to collect RGB data of facial key points:

[0073] Use RGB camera to collect: directly obtain the image data of the face through the RGB camera, and then perform image processing to extract key point information;

[0074] Depth camera combined with RGB camera acquisition: The depth camera can obtain the depth information of the face, and combined with the image data collected by the RGB camera, it can extract key points more accurately;

[0075] 3D scanner acquisition: Use a 3D scanner to obtain a three-dimensional model of the face and then extract key point information.

[0076] The multi-vision depth scanning data and the surrounding environment data of the key points of the face can be collected by the depth camera to collect depth images such as lighting, color, and shadow; the axial depth data of the key points of the face in the XYZ three-dimensional space can be collected by a three-axis camera to collect depth images, which can provide depth information in the three axes of XYZ. Combined with the multi-vision depth scanning technology, the illumination, color, shadow and other environmental data around the key points of the face in the three axes of XYZ can be obtained. These axial depth data can accurately reflect the position of the key points of the face in the three-dimensional space; the point cloud data of the key points of the face can be collected by a point cloud camera. The three-dimensional point cloud of the key points of the face is represented by the point cloud. Each point has three coordinate values ​​(X, Y, Z) to represent its position in the three-dimensional space. However, the model design and training of point cloud data will be more complicated and require higher computing power.

[0077] Further, S2, feature engineering is performed on the image big data of the facial key points, and RGB image features and multi-visual depth image features corresponding to the RGB data and the multi-visual depth scanning data are extracted respectively, including:

[0078] Performing image preprocessing on the RGB data and the multi-vision depth scan data respectively;

[0079] Using a ResNet model to extract RGB image features of the RGB data;

[0080] Using a CNN model, sequentially extracting depth image features of the depth scan data under each vision in the multi-vision depth scan data, and obtaining a depth scan feature set composed of the depth image features under each vision;

[0081] The RGB image features and the depth scan feature set are saved separately to a background database.

[0082] For the RGB data and the multi-visual depth scanning data, the present invention adopts the ResNet model and the CNN model for feature extraction respectively, and uses the advantages of each model to extract features of the corresponding image data.

[0083] 1. The steps to extract RGB image features using the ResNet model are as follows:

[0084] Load RGB image data;

[0085] Preprocess the image, including normalization, resizing, etc.;

[0086] Input the preprocessed image into the ResNet model;

[0087] Extract features through multiple convolutional layers and residual blocks of the ResNet model;

[0088] Get the feature vector before the last fully connected layer of the model;

[0089] The output feature vector is used as the feature representation of the image.

[0090] advantage:

[0091] The residual structure can effectively alleviate the gradient vanishing problem in deep neural network training;

[0092] Improved the convergence speed and accuracy of the model;

[0093] By combining multiple residual blocks, a deep network can be constructed to improve feature extraction capabilities;

[0094] It is suitable for a variety of image classification and recognition tasks and has good generalization ability.

[0095] 2. The steps to extract deep image features using the CNN model are as follows:

[0096] Prepare the dataset: collect deep scan images and perform preprocessing, such as normalization and cropping;

[0097] Design the convolutional neural network structure: select appropriate convolutional layers, pooling layers, and fully connected layers;

[0098] Initialize network parameters: use a pre-trained model or randomly initialize network weights;

[0099] Training network: Use the training data set to train the convolutional neural network and adjust the network parameters to minimize the loss function;

[0100] Verify network performance: Use the validation dataset to evaluate network performance and adjust hyperparameters to optimize the model;

[0101] Test network: Use the test data set to conduct a final test on the network to evaluate the generalization ability of the model;

[0102] Feature extraction: Use the trained convolutional neural network to extract features from the deep scan image;

[0103] Feature post-processing: normalize and reduce the dimension of the extracted features for use in subsequent tasks.

[0104] advantage:

[0105] Strong feature extraction capability: CNN can automatically learn the hierarchical features in images and extract deep features.

[0106] Good generalization ability: The trained CNN model has good generalization ability for new image data.

[0107] High real-time performance: The CNN model can quickly process image data and achieve real-time feature extraction under hardware acceleration, such as GPU.

[0108] Reduced manual intervention: The CNN model automatically learns features, reducing the need for manual feature design.

[0109] The specific feature extraction methods of the ResNet model and the CNN model can be understood in combination with the model architectures of the two models, and will not be described in detail in this embodiment.

[0110] The present invention fuses the features extracted by the two model recognitions to train a new multi-vision depth detection model. The feature fusion method adopts deep learning model fusion: the two-dimensional features and three-dimensional features are input into different network branches using a neural network, and then fused at a high level. The two-dimensional features and three-dimensional features are respectively input into two independent network branches for feature extraction, and then the features of the two branches are fused at a higher level of the network (such as after a deep convolutional layer or before a fully connected layer) by splicing, element addition, element multiplication, or using a custom fusion layer, thereby realizing the comprehensive utilization of multimodal features.

[0111] The present invention provides two feature fusion training methods:

[0112] Method 1:

[0113] like Figure 3 As shown, further, S3, the RGB image features and the multi-vision depth image features are feature-fused to construct a multi-vision depth image fusion feature, and the multi-vision depth image fusion feature is input into a preset ResNet model for feature training and learning, and a multi-vision depth detection model corresponding to the recognition of face key points is constructed, including:

[0114] Traversing the depth scanning feature set, and sequentially extracting the depth image features under each visual sense in the depth scanning feature set;

[0115] Input the extracted depth image features under each visual condition and the RGB image features into the deep convolutional layer of the CNN model;

[0116] On the deep convolution layer, by convolutionally fusing the corresponding feature elements on the key points of the face, the deep image features and the RGB image features are fused on the convolution layer to obtain the convolution fusion features under each visual sense:

[0117] According to the above method, the multi-vision depth image fusion feature composed of the convolution fusion features under each vision is obtained, that is, RGBD data;

[0118] The RGBD data is preprocessed, and the preprocessed RGBD data is input into a preset ResNet model, feature training and learning of the multi-visual depth image fusion features in the RGBD data are performed based on the ResNet network, and the multi-visual depth detection model is constructed and generated.

[0119] This method is to convolve and fuse the deep image features of each vision or each dimension with the RGB image features to obtain the convolution fusion features of each dimension or each vision, and then fuse the features of other visions or dimensions to obtain a feature set composed of convolution comprehensive features of several dimensions or visions, that is, RGBD data. After these feature sets are pre-processed by feature sorting, cleaning or feature enhancement, they are input into the ResNet network for model feature training and learning to build a multi-vision depth detection model.

[0120] The feature fusion on the convolutional layer includes:

[0121] Determine the fused depth image layer;

[0122] Preprocess each layer of images, such as normalization;

[0123] Use splicing or feature mapping methods to fuse image features at different depths;

[0124] The fused features are further extracted and optimized through the convolution layer;

[0125] Send the optimized features to the subsequent network layer for further processing;

[0126] Evaluate the fused image features and adjust the fusion strategy based on the evaluation results;

[0127] Repeat steps 2-6 until you achieve a satisfactory blending effect.

[0128] In the above model training process, multiple fusion features can also be divided into training sets and verification sets, and the training set can be used for model feature training and learning, and the verification set can be used to verify the performance of the multi-vision depth detection model. If the verification passes, the model is deployed to the background for application; if it fails, it is retrained.

[0129] The model performance verification method during the model training process has the following verification schemes:

[0130] Accuracy

[0131] Recall

[0132] Precision

[0133] F1 Score

[0134] AUC (Area Under the ROC Curve).

[0135] The specific model verification can be completed by the administrator.

[0136] Method 2:

[0137] Further, S3, the RGB image features and the multi-vision depth image features are feature-fused to construct RGBD data consisting of multi-vision depth image fusion features, and the RGBD data is input into a preset ResNet model to perform feature training and learning, and a multi-vision depth detection model corresponding to the recognition of face key points is constructed, including:

[0138] Traversing the depth scanning feature set, and sequentially extracting the depth image features under each visual sense in the depth scanning feature set;

[0139] Input the extracted depth image features under each visual condition and the RGB image features into the deep convolutional layer of the CNN model;

[0140] On the deep convolution layer, by convolutionally fusing the corresponding feature elements on the key points of the face, the deep image features and the RGB image features are fused on the convolution layer to obtain the convolution fusion features under the current vision:

[0141] According to the above steps, the depth image feature under the next vision is subjected to feature fusion on the convolution layer by combining the feature elements on the depth image feature under the next vision with the conveyed convolution fusion feature under the current vision to obtain the convolution fusion feature under the next vision:

[0142] According to the above method, the depth image features under the remaining vision are sequentially fused to obtain the final convolution fusion features;

[0143] Preprocessing the final convolution fusion features, inputting the preprocessed final convolution fusion features into a preset ResNet model, and performing feature training and learning on the final convolution fusion features based on the ResNet network;

[0144] The final convolutional fusion features are preprocessed, and the preprocessed final convolutional fusion features are input into a preset ResNet model, and feature training and learning of the final convolutional fusion features are performed based on the ResNet network to construct and generate the multi-visual depth detection model.

[0145] This method adopts another feature fusion method. First, the deep image features of each vision or each dimension are convoluted and fused with the RGB image features to obtain the convolution fusion features of the current vision. Then, the deep image features of the next dimension or the next vision are convoluted and fused with the convolution fusion features obtained by the previous fusion, which is equivalent to performing a composite feature fusion. Through continuous convolution features, the deep fusion is continued on the convolution fusion features obtained by the first fusion until the final convolution fusion features are obtained, and then the final convolution fusion features are trained and learned by the model. Through this method, there is no need to perform convolution fusion in batches, but to perform superposition fusion of deep features, so that the final generated convolution fusion features have rich fusion feature information. When training, the model only needs to learn the final convolution fusion features, so the model training cycle can be reduced to improve the training speed.

[0146] The above two methods can also be cross-trained. For example, the features obtained by fusion construction of method 1 are used for model training, and the final convolution fusion features generated by method 2 are used to verify the performance of the model generated by method 1. If the verification is passed, the final multi-visual depth detection model is generated. The features of method 2 can also be used for model training, and the features of method 1 are used to verify the performance of the model constructed by method 2. If the verification is passed, the multi-visual depth detection model is finally obtained. Or after the model is trained using the features of method 1, the model is trained again using the final convolution fusion features obtained by method 2, so that the model can comprehensively learn the multi-visual depth image features extracted and generated by the two methods, and then randomly extract a number of features from the features generated by the two methods, and perform performance verification on the model generated by the final training. Through cross-training, the comprehensive recognition ability of the model can be improved.

[0147] On the other hand, a face key point recognition device based on multi-visual depth information is provided, and the face key point recognition device based on multi-visual depth information is used to implement the face key point recognition method based on multi-visual depth information, and the device includes:

[0148] An image acquisition module is used to acquire a user's face image and input it into a face key point recognition module;

[0149] A facial key point recognition module, used to recognize and output the recognition result of the facial key points on the facial image using a pre-trained multi-vision depth detection model;

[0150] The image acquisition module is communicatively connected with the facial key point recognition module.

[0151] Please understand and implement the implementation principles of the above methods. This article will not go into details. The deployment and application of the model can be done by the administrator through background deployment and model parameter adjustment.

[0152] Figure 4 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Figure 4 As shown, the electronic device may include the above Figure 3 The illustrated apparatus for recognizing facial key points based on multi-visual depth information. Optionally, the electronic device 410 may include a first processor 2001 .

[0153] Optionally, the electronic device 410 may further include a memory 2002 and a transceiver 2003 .

[0154] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0155] Combine the following Figure 4 The components of the electronic device 410 are described in detail:

[0156] The first processor 2001 is the control center of the electronic device 410, and may be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or may be application specific integrated circuits (ASICs), or may be one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (field programmable gate arrays, FPGAs).

[0157] Optionally, the first processor 2001 can execute various functions of the electronic device 410 by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002 .

[0158] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 4 CPU0 and CPU1 are shown in FIG.

[0159] In a specific implementation, as an embodiment, the electronic device 410 may also include multiple processors, such as Figure 4The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0160] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0161] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently and access the first processor 2001 through the interface circuit ( Figure 4 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0162] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0163] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 4 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0164] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently and communicate with the first processor 2001 through the interface circuit ( Figure 4 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0165] It should be noted that Figure 4 The structure of the electronic device 410 shown in the figure does not constitute a limitation on the router, and the actual knowledge structure recognition device may include more or fewer components than those shown in the figure, or combine certain components, or arrange the components differently.

[0166] In addition, the technical effects of the electronic device 410 can refer to the technical effects of the facial key point recognition method based on multi-visual depth information described in the above method embodiment, and will not be repeated here.

[0167] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0168] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0169] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0170] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0171] In the present invention, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0172] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0173] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0174] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0175] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0176] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0177] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0178] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0179] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for recognizing facial key points based on multi-visual depth information, characterized in that: The method comprises: S1. Collect big image data of facial key points, including RGB data and multi-vision depth scanning data; S2, performing feature engineering on the image big data of the facial key points, respectively extracting RGB image features and multi-vision depth image features corresponding to the RGB data and the multi-vision depth scanning data; S3, feature fusion of the RGB image features and the multi-vision depth image features to construct a multi-vision depth image fusion feature, and input the multi-vision depth image fusion feature into a preset ResNet model to perform feature training and learning, and construct a multi-vision depth detection model corresponding to the recognition of face key points, including: Traversing the depth scanning feature set, and sequentially extracting the depth image features under each visual sense in the depth scanning feature set; Input the extracted depth image features under each visual condition and the RGB image features into the deep convolutional layer of the CNN model; On the deep convolution layer, by convolutionally fusing the corresponding feature elements on the key points of the face, the deep image features and the RGB image features are fused on the convolution layer to obtain the convolution fusion features under each visual sense: According to the above method, the multi-vision depth image fusion feature composed of the convolution fusion features under each vision is obtained, that is, RGBD data; Preprocessing the RGBD data, and inputting the preprocessed RGBD data into a preset ResNet model, performing feature training and learning on the multi-vision depth image fusion features in the RGBD data based on the ResNet network, and constructing and generating the multi-vision depth detection model; S4. Collect a facial image of the user and input it into the multi-vision depth detection model, and the multi-vision depth detection model recognizes and outputs the recognition result of facial key points on the facial image.

2. The method for recognizing facial key points based on multi-visual depth information according to claim 1, characterized in that: In step S1, the multi-vision depth scanning data includes scanning data under at least one of the following visions: The surrounding environment data of facial key points; Axial depth data of facial key points in XYZ three-dimensional space; or, Point cloud data of facial key points.

3. The method for recognizing facial key points based on multi-visual depth information according to claim 1, characterized in that: S2. Perform feature engineering on the image big data of the facial key points, and extract RGB image features and multi-vision depth image features corresponding to the RGB data and the multi-vision depth scan data, respectively, including: Performing image preprocessing on the RGB data and the multi-vision depth scan data respectively; Using a ResNet model to extract RGB image features of the RGB data; Using a CNN model, sequentially extracting depth image features of the depth scan data under each vision in the multi-vision depth scan data, and obtaining a depth scan feature set composed of the depth image features under each vision; The RGB image features and the depth scan feature set are saved separately to a background database.

4. The method for recognizing facial key points based on multi-visual depth information according to claim 3, characterized in that: S3, feature fusion of the RGB image features and the multi-vision depth image features, constructing RGBD data consisting of multi-vision depth image fusion features, and inputting the RGBD data into a preset ResNet model to perform feature training and learning, and constructing a multi-vision depth detection model corresponding to the recognition of face key points, including: Traversing the depth scanning feature set, and sequentially extracting the depth image features under each visual sense in the depth scanning feature set; Input the extracted depth image features under each visual condition and the RGB image features into the deep convolutional layer of the CNN model; On the deep convolution layer, by convolutionally fusing the corresponding feature elements on the key points of the face, the deep image features and the RGB image features are fused on the convolution layer to obtain the convolution fusion features under the current vision: According to the above steps, the depth image feature under the next vision is subjected to feature fusion on the convolution layer by combining the feature elements on the depth image feature under the next vision with the conveyed convolution fusion feature under the current vision to obtain the convolution fusion feature under the next vision: According to the above method, the depth image features under the remaining vision are sequentially fused to obtain the final convolution fusion features; Preprocessing the final convolution fusion features, inputting the preprocessed final convolution fusion features into a preset ResNet model, and performing feature training and learning on the final convolution fusion features based on the ResNet network; The final convolutional fusion features are preprocessed, and the preprocessed final convolutional fusion features are input into a preset ResNet model. Feature training and learning of the final convolutional fusion features are performed based on the ResNet network to construct and generate the multi-visual depth detection model.

5. A facial key point recognition device based on multi-visual depth information, the facial key point recognition device based on multi-visual depth information is used to implement the facial key point recognition method based on multi-visual depth information as claimed in any one of claims 1 to 4, characterized in that: The device comprises: An image acquisition module is used to acquire a user's face image and input it into a face key point recognition module; A facial key point recognition module, used to recognize and output the recognition result of the facial key points on the facial image using a pre-trained multi-vision depth detection model; The image acquisition module is communicatively connected with the facial key point recognition module.

6. An electronic device, characterized in that: The electronic device comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 4 is implemented.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 4.

Images