Image recognition model training method, image recognition method, device and electronic equipment

Abstract

Embodiments of the present application provide an image recognition model training method, an image recognition method, an image recognition device and an electronic device, and relate to the technical field of intelligent security. The image recognition model training method comprises: obtaining a first sample image containing a real biological feature, a second sample image containing a photo of the real biological feature, a third sample image containing a biological feature prosthesis, and a classification label indicating whether an object in each sample image is a real biological feature; training a classification model using the aforementioned sample images and respective classification labels to obtain a trained feature extraction network; training a first model branch using the first sample image, the second sample image and respective classification labels; training a second model branch using the first sample image, the third sample image and respective classification labels to obtain a trained image recognition model. Based on the trained image recognition model, the real biological feature, the photo of the real biological feature and the biological feature prosthesis are distinguished.

Description

Technical Field

[0001] This application relates to the field of intelligent security technology, and in particular to an image recognition model training method, an image recognition method, a device, and an electronic device. Background Technology

[0002] In the field of intelligent security, to improve the security of designated areas (such as schools, companies, etc.), users can be authenticated before entering the designated area; after successful authentication, the access control system (such as door locks, turnstiles, etc.) for the designated area can be activated, allowing the user to enter. For example, an image acquisition device can be used to capture an image of the user's palm, and the user's identity can be authenticated by detecting the palm veins in the image.

[0003] However, malicious attackers can use photos of a user's hand or a prosthetic hand for authentication to impersonate the user and open access control systems. To improve the security of designated areas, there is an urgent need for an image recognition method to distinguish between images containing a user's hand, images containing photos of a user's hand, and images containing a prosthetic hand. Summary of the Invention

[0004] The purpose of this application is to provide an image recognition model training method, an image recognition method, a device, and an electronic device to accurately distinguish between real biometric features and photographs of real biometric features or biometric prostheses. The specific technical solution is as follows:

[0005] In a first aspect of this application, an image recognition model training method is provided, the method comprising:

[0006] The system acquires a first sample image containing real biometric features, a second sample image containing a photograph of real biometric features, a third sample image containing a biometric prosthesis, and a classification label indicating whether the object in each sample image is a real biometric feature.

[0007] Using the first sample image, the second sample image, the third sample image, and their respective classification labels, a classification model containing a feature extraction network and a first classification network is trained to obtain a trained feature extraction network.

[0008] Using the first sample image, the second sample image, and their respective classification labels, the first model branch of the image recognition model with the initial structure is trained, and using the first sample image, the third sample image, and their respective classification labels, the second model branch of the image recognition model is trained, thus obtaining a trained image recognition model.

[0009] The first model branch includes a trained feature extraction network and a second classification network; the second model branch includes a trained feature extraction network and a third classification network; during the training of the first model branch and the second model branch, the parameters of the trained feature extraction network are frozen.

[0010] Optional, biometric feature is the palm;

[0011] For any sample image, when processing the sample image using a feature extraction network, the input data for the feature extraction network is obtained through the following steps:

[0012] Detect the position of the palm and the position of the palm area in the sample image;

[0013] Based on the palm position, extract the palm image region occupied by the palm from the sample image;

[0014] Based on the location of the palm region, extract the palm image region occupied by the palm from the sample image;

[0015] The extracted palm image region and palm center image region are fused to obtain the input data for the feature extraction network.

[0016] Optionally, the location of the palm region in the sample image is detected, including:

[0017] Detect key points in the palm of the hand in the sample image;

[0018] Determine the midpoint of the first line connecting the detected key points at the base of the index finger and the key points at the base of the little finger, and use it as the first midpoint.

[0019] Determine the midpoint of the line connecting the first midpoint and the second line connecting the detected key point on the palm root, and use it as the second midpoint;

[0020] Extract a square image region centered at the second midpoint and with the first connecting line as its edge from the sample image, and use it as the palm image region.

[0021] Optionally, any sample image can be an image acquired under infrared light source illumination.

[0022] In a second aspect of this application, an image recognition method is provided, the method comprising:

[0023] Acquire the image to be recognized;

[0024] The trained image recognition model is used to process the image to be recognized to obtain the recognition results of the second classification network and the third classification network in the image recognition model; wherein, the image recognition model is trained based on any of the image recognition model training methods described in the first aspect above;

[0025] Based on the recognition results of the second classification network and the third classification network, it is determined whether the object in the image to be identified has real biological characteristics.

[0026] Optional, biometric feature is the palm;

[0027] The process of using a trained image recognition model to process the image to be recognized, and obtaining the recognition results of the second and third classification networks in the image recognition model, includes:

[0028] Detect the position of the palm and the position of the palm area in the image to be identified;

[0029] Based on the palm position, extract the palm image region occupied by the palm from the image to be identified;

[0030] Based on the location of the palm region, extract the palm image region occupied by the palm from the image to be identified;

[0031] The extracted palm image region and the center of the palm image region are fused to obtain the fusion result;

[0032] The fusion result is processed using the feature extraction network, the second classification network, and the third classification network in the trained image recognition model to obtain the recognition results of the second classification network and the third classification network, respectively.

[0033] Optionally, detecting the location of the palm region in the image to be identified includes:

[0034] Detect key points in the palm of the hand in the image to be identified;

[0035] Determine the midpoint of the third line connecting the detected key points at the base of the index finger and the key points at the base of the little finger, and use this as the third midpoint.

[0036] The midpoint of the line connecting the third midpoint and the fourth line connecting the detected key point of the palm root is determined as the fourth midpoint.

[0037] Extract a square image region centered at the fourth midpoint and with the third connecting line as its edge from the image to be identified, and use it as the palm image region.

[0038] Optionally, based on the recognition results of the second classification network and the third classification network, determining whether the object in the image to be identified is a genuine biometric feature includes:

[0039] If the recognition result of the second classification network indicates that the object in the image to be identified is a real biometric feature, and the recognition result of the third classification network indicates that the object in the image to be identified is a real biometric feature, then the object in the image to be identified is determined to be a real biometric feature.

[0040] Otherwise, it is determined that the object in the image to be identified does not possess genuine biometric features.

[0041] Optionally, the image to be identified is an image acquired under infrared light source illumination.

[0042] In a third aspect of this application, an image recognition model training apparatus is provided, the apparatus comprising:

[0043] The first acquisition module is used to acquire a first sample image containing real biometric features, a second sample image containing a photograph containing real biometric features, a third sample image containing a biometric prosthesis, and a classification label indicating whether the object in each sample image has real biometric features.

[0044] The first training module is used to train a classification model containing a feature extraction network and a first classification network using the first sample image, the second sample image, the third sample image and their respective classification labels, so as to obtain a trained feature extraction network.

[0045] The second training module is used to train the first model branch of the image recognition model with the first sample image, the second sample image and their respective classification labels, and to train the second model branch of the image recognition model with the first sample image, the third sample image and their respective classification labels, so as to obtain the trained image recognition model.

[0046] The first model branch includes a trained feature extraction network and a second classification network; the second model branch includes a trained feature extraction network and a third classification network; during the training of the first model branch and the second model branch, the parameters of the trained feature extraction network are frozen.

[0047] Optionally, the biometric feature is the palm; for any sample image, when processing the sample image using a feature extraction network, the input data for the feature extraction network is obtained through the following steps:

[0048] Detect the position of the palm and the position of the palm area in the sample image;

[0049] Based on the palm position, extract the palm image region occupied by the palm from the sample image;

[0050] Based on the location of the palm region, extract the palm image region occupied by the palm from the sample image;

[0051] The extracted palm image region and palm center image region are fused to obtain the input data for the feature extraction network.

[0052] Optionally, the location of the palm region in the sample image is detected, including:

[0053] Detect key points in the palm of the hand in the sample image;

[0054] Determine the midpoint of the first line connecting the detected key points at the base of the index finger and the key points at the base of the little finger, and use it as the first midpoint.

[0055] Determine the midpoint of the line connecting the first midpoint and the second line connecting the detected key point on the palm root, and use it as the second midpoint;

[0056] Extract a square image region centered at the second midpoint and with the first connecting line as its edge from the sample image, and use it as the palm image region.

[0057] Optionally, any sample image can be an image acquired under infrared light source illumination.

[0058] In a fourth aspect of this application, an image recognition apparatus is provided, the apparatus comprising:

[0059] The second acquisition module is used to acquire the image to be recognized;

[0060] The recognition module is used to process the image to be recognized using a trained image recognition model to obtain the recognition results of the second classification network and the third classification network in the image recognition model; wherein, the image recognition model is trained based on any of the image recognition model training methods described in the first aspect above;

[0061] The result determination module is used to determine whether the object in the image to be identified is a real biometric feature based on the recognition results of the second classification network and the third classification network.

[0062] Optional, biometric feature is the palm;

[0063] The identification module is specifically used for:

[0064] Detect the position of the palm and the position of the palm area in the image to be identified;

[0065] Based on the palm position, extract the palm image region occupied by the palm from the image to be identified;

[0066] Based on the location of the palm region, extract the palm image region occupied by the palm from the image to be identified;

[0067] The extracted palm image region and the center of the palm image region are fused to obtain the fusion result;

[0068] The fusion result is processed using the feature extraction network, the second classification network, and the third classification network in the trained image recognition model to obtain the recognition results of the second classification network and the third classification network, respectively.

[0069] Optionally, the identification module is specifically used for:

[0070] Detect key points in the palm of the hand in the image to be identified;

[0071] Determine the midpoint of the third line connecting the detected key points at the base of the index finger and the key points at the base of the little finger, and use this as the third midpoint.

[0072] The midpoint of the line connecting the third midpoint and the fourth line connecting the detected key point of the palm root is determined as the fourth midpoint.

[0073] Extract a square image region centered at the fourth midpoint and with the third connecting line as its edge from the image to be identified, and use it as the palm image region.

[0074] Optionally, the result determination module is specifically used for:

[0075] If the recognition result of the second classification network indicates that the object in the image to be identified is a real biometric feature, and the recognition result of the third classification network indicates that the object in the image to be identified is a real biometric feature, then the object in the image to be identified is determined to be a real biometric feature.

[0076] Otherwise, it is determined that the object in the image to be identified does not possess genuine biometric features.

[0077] Optionally, the image to be identified is an image acquired under infrared light source illumination.

[0078] In a fifth aspect of the embodiments of this application, an electronic device is provided, comprising: a memory for storing a computer program; and a processor for executing the program stored in the memory to implement the image recognition model training method described in any of the first aspects above, or to implement the image recognition method described in any of the second aspects above.

[0079] In a sixth aspect of the embodiments of this application, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements the image recognition model training method described in any of the first aspects above, or implements the image recognition method described in any of the second aspects above.

[0080] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to execute the image recognition model training method described in any of the first aspects above, or to implement the image recognition method described in any of the second aspects above.

[0081] This application provides an image recognition model training method, which involves acquiring a first sample image containing real biometric features, a second sample image containing real biometric features, a third sample image containing biometric prostheses, and classification labels indicating whether the object in each sample image is a real biometric feature. Using the first, second, and third sample images and their respective classification labels, a classification model comprising a feature extraction network and a first classification network is trained to obtain a trained feature extraction network. Using the first, second, and third sample images and their respective classification labels, a first model branch of the initial structured image recognition model is trained, and using the first, third, and respective classification labels, a second model branch of the image recognition model is trained to obtain a trained image recognition model. The first model branch includes the trained feature extraction network and the second classification network; the second model branch includes the trained feature extraction network and the third classification network. During the training of the first and second model branches, the parameters of the trained feature extraction network are frozen.

[0082] Based on the above processing, the classification model is first trained using the three types of sample images and their corresponding classification labels to obtain a trained feature extraction network. That is, a feature extraction network that has learned the features of each of the three types of samples is first trained. Subsequently, while keeping the parameters of the trained feature extraction network frozen, the two model branches are trained using this network. This avoids the second and third classification networks affecting the training process of the feature extraction network, allowing the feature extraction network to learn the features of real biometrics, photos of real biometrics, and biometric prostheses. Consequently, the image features extracted using the trained feature extraction network are more accurate, and the image recognition results obtained based on more accurate image features are also more accurate.

[0083] Furthermore, the trained image recognition model comprises two branches: the first branch includes a trained feature extraction network and a second classification network, specifically designed to distinguish between a first sample image containing genuine biometric features and a second sample image containing photographs of genuine biometric features; the second branch includes a trained feature extraction network and a third classification network, specifically designed to distinguish between a first sample image containing genuine biometric features and a third sample image containing biometric prostheses. Subsequently, during image recognition, the first branch is used to identify whether the object in the image to be recognized is a genuine biometric feature or a photograph of a genuine biometric feature; the second branch is used to identify whether the object is a genuine biometric feature or a biometric prosthesis. Compared to using a single classification network to identify genuine biometric features, photographs of genuine biometric features, and biometric prostheses, this approach allows each model branch to focus on recognizing its corresponding object type, reducing interference between the recognition process for photographs of genuine biometric features and the recognition process for biometric prostheses. Therefore, the accuracy of the object type recognition result obtained by each model branch is higher. Furthermore, the final recognition result obtained by combining the recognition results of the two model branches is also highly accurate, and can accurately distinguish between real biometric features and photos of real biometric features and biometric prostheses.

[0084] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0085] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0086] Figure 1 A flowchart of an image recognition model training method provided in an embodiment of this application;

[0087] Figure 2 A schematic diagram of key points in the palm provided in an embodiment of this application;

[0088] Figure 3 A flowchart of an image recognition method provided in an embodiment of this application;

[0089] Figure 4 A structural diagram of an image recognition model training device provided in an embodiment of this application;

[0090] Figure 5 A structural diagram of an image recognition device provided in an embodiment of this application;

[0091] Figure 6 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0092] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0093] In the field of intelligent security, in order to improve the security of designated areas (such as schools, companies, etc.), users can be authenticated before entering the designated area; after the user's identity is authenticated, the access control (such as door lock, turnstile, etc.) of the designated area can be opened, and then the user can enter the designated area through the access control.

[0094] For example, access control systems can be based on palm vein recognition for identity authentication. Correspondingly, access control systems can include palm vein smart locks, palm vein doorbells, palm vein turnstiles, etc. Palm vein smart locks can use image acquisition devices to capture images and then authenticate users by detecting palm veins in the captured images.

[0095] However, malicious attackers can use photos of a user's hand or a prosthetic hand for authentication to impersonate the user and open access control systems. To improve the security of designated areas, there is an urgent need for an image recognition method to distinguish between images containing a user's hand, images containing photos of a user's hand, and images containing a prosthetic hand. The process of a malicious attacker impersonating a user to open access control systems can be called an anti-spoofing attack or a spoofing attack; distinguishing between images containing a user's hand, images containing photos of a user's hand, and images containing a prosthetic hand can be called anti-spoofing or attack protection.

[0096] To address the aforementioned issues, this application provides an image recognition model training method. This method is applied to a training device, such as a server. After obtaining a trained image recognition model according to the image recognition model training method provided in this application, the training device can send the trained image recognition model to an access control system (such as a door lock or turnstile). The access control system can use the received image recognition model to perform image recognition on the acquired image to be recognized, and then perform corresponding processing based on the image recognition result. If the image recognition result indicates that the object in the image to be recognized has genuine biometric features, the access control system can open, allowing the user undergoing authentication to pass through. Alternatively, after acquiring the image to be recognized, the access control system can send the image to be recognized to the training device. The training device performs image recognition based on the trained image recognition model, and after obtaining an image recognition result indicating whether the object in the image to be recognized has genuine biometric features, it sends the image recognition result to the access control system, enabling the access control system to perform subsequent processing based on the received image recognition result. Based on the extracted image features with higher accuracy, and by combining the accurate recognition results of the first and second model branches in the image recognition model, a more accurate final recognition result can be obtained, enabling accurate differentiation between real biometric features and photos of real biometric features and biometric prostheses.

[0097] See Figure 1 , Figure 1 This is a first flowchart of an image recognition model training method provided in an embodiment of this application. The method may include the following steps:

[0098] S101: Obtain a first sample image containing real biometric features, a second sample image containing a photograph of real biometric features, a third sample image containing a biometric prosthesis, and a classification label indicating whether the object in each sample image is a real biometric feature.

[0099] S102: Using the first sample image, the second sample image, the third sample image, and their respective classification labels, train the classification model containing the feature extraction network and the first classification network to obtain the trained feature extraction network.

[0100] S103: Using the first sample image, the second sample image, and their respective classification labels, train the first model branch of the image recognition model with the initial structure, and use the first sample image, the third sample image, and their respective classification labels to train the second model branch of the image recognition model, thus obtaining the trained image recognition model.

[0101] The first model branch includes a trained feature extraction network and a second classification network; the second model branch includes a trained feature extraction network and a third classification network; during the training of the first and second model branches, the parameters of the trained feature extraction network are frozen.

[0102] Based on the above processing, the classification model is first trained using the three types of sample images and their corresponding classification labels to obtain a trained feature extraction network. That is, a feature extraction network that has learned the features of each of the three types of samples is first trained. Subsequently, while keeping the parameters of the trained feature extraction network frozen, the two model branches are trained using this network. This avoids the second and third classification networks affecting the training process of the feature extraction network, allowing the feature extraction network to learn the features of real biometrics, photos of real biometrics, and biometric prostheses. Consequently, the image features extracted using the trained feature extraction network are more accurate, and the image recognition results obtained based on more accurate image features are also more accurate.

[0103] Furthermore, the trained image recognition model comprises two branches: the first branch includes a trained feature extraction network and a second classification network, specifically designed to distinguish between a first sample image containing genuine biometric features and a second sample image containing photographs of genuine biometric features; the second branch includes a trained feature extraction network and a third classification network, specifically designed to distinguish between a first sample image containing genuine biometric features and a third sample image containing biometric prostheses. Subsequently, during image recognition, the first branch is used to identify whether the object in the image to be recognized is a genuine biometric feature or a photograph of a genuine biometric feature; the second branch is used to identify whether the object is a genuine biometric feature or a biometric prosthesis. Compared to using a single classification network to identify genuine biometric features, photographs of genuine biometric features, and biometric prostheses, this approach allows each model branch to focus on recognizing its corresponding object type, reducing interference between the recognition process for photographs of genuine biometric features and the recognition process for biometric prostheses. Therefore, the accuracy of the object type recognition result obtained by each model branch is higher. Furthermore, the final recognition result obtained by combining the recognition results of the two model branches is also highly accurate, and can accurately distinguish between real biometric features and photos of real biometric features and biometric prostheses.

[0104] Regarding step S101, biometric features are features that can uniquely identify a person's identity, such as including at least one of a person's palm, fingers, wrist, iris, and face. This application does not specifically limit this.

[0105] Real biometrics refers to a person's biological characteristics. A photograph of a real biometric is a photo obtained by capturing images of real biometric features. A biometric prosthesis is a 3D prosthesis manufactured using prosthesis manufacturing technology (such as 3D printing). In real-world scenarios, the first sample image containing real biometrics, the second sample image containing a photograph of real biometrics, and the third sample image containing a biometric prosthesis all contain the same type of biometric feature. For example, in the case of a hand as a biometric feature, an image of a person's hand is captured using an image acquisition device as the first sample image; an image of the photograph containing the hand is captured to obtain the second sample image; a model of the hand is created, and this model is 3D printed to obtain a 3D prosthesis of the hand; finally, an image of the 3D prosthesis of the hand is captured to obtain the third sample image.

[0106] In one approach, the classification label corresponding to a sample image indicates whether the object in that sample image possesses genuine biological characteristics or not. That is, the classification label corresponding to the first sample image indicates that the object in the first sample image possesses genuine biological characteristics; for any sample image between the second and third sample images, the classification label corresponding to that sample image indicates that the object in that sample image does not possess genuine biological characteristics. Accordingly, the subsequent first classification network can be a binary classification network.

[0107] In another approach, the classification label corresponding to a sample image indicates whether the object in that sample image is a genuine biometric feature, a photograph of a genuine biometric feature, or a biometric prosthesis. That is, the classification label corresponding to the first sample image indicates that the object in the first sample image is a genuine biometric feature, the classification label corresponding to the second sample image indicates that the object in the second sample image is a photograph of a genuine biometric feature, and the classification label corresponding to the third sample image indicates that the object in the third sample image is a biometric prosthesis. Accordingly, the subsequent first classification network can be a three-class classification network.

[0108] For step S102, the classification model includes a feature extraction network and a first classification network connected in series. The output data of the feature extraction network can be used as the input data of the first classification network.

[0109] Feature extraction networks are used to extract features from their input data, such as VGG (Visual Geometry Group Network) and Transformer networks. In practical smart security applications, it may be necessary to run image recognition models on devices with limited hardware performance (such as door locks, doorbells, and turnstiles) to determine whether to open the access control system based on the image recognition results output by the model. Therefore, in real-world scenarios, to enable image recognition models to run on these devices with limited hardware performance, the feature extraction network (which can be called the backbone network) can be a lightweight feature extraction network, such as MobileNet v2.

[0110] For any one of the first, second, and third sample images, the training device can use the feature extraction network of the initial structure to extract the image features of that sample image.

[0111] In one approach, the training device can directly input the sample image into the feature extraction network of the initial structure, and the feature extraction network will extract features from the sample image to obtain the image features of the sample image.

[0112] In another approach, the training device can acquire input data for the feature extraction network based on the sample image, and then use the feature extraction network to extract features from the input data to obtain the image features of the sample image.

[0113] For example, the training device can use a depth map transformation algorithm to convert the sample image into a depth map; based on Fourier transform, it can convert the sample image into a frequency domain map; the original sample image, the converted depth map, and the converted frequency domain map are mapped to the same dimension, and then image fusion is performed based on element-wise addition to obtain the input data for the feature extraction network. The aforementioned depth map transformation algorithm can be monocular depth estimation, semi-global block matching, etc.

[0114] For example, the training device can use a sliding window technique to sample the sample image, and the resulting sampling can be called multi-scale information. Then, the multi-scale information is combined (e.g., by stitching together the multi-scale information) to obtain the input data for the feature extraction network.

[0115] For example, when acquiring the sample image, a speckle pattern of biological features can also be collected based on a structured light device; the speckle pattern and the sample image are then fused to obtain the input data for the feature extraction network.

[0116] For example, when acquiring the sample image, sample images containing other biometric features can also be acquired; by combining the sample image and the sample images containing other biometric features, the input data for the feature extraction network is obtained. For example, the biometric feature contained in the sample image can be a palm, and other biometric features can be the aforementioned face, fingers, etc., and this application does not specifically limit this.

[0117] The first classification network is used to classify the output data of the feature extraction network to obtain the corresponding first prediction result. As mentioned above, the first classification network can be a binary classification network or a tri-class classification network, and the specific choice can be determined based on the method of setting the classification labels. The first classification network can be called the Classifier Head A, such as a linear classification layer, or an MLP (Multi-Layer Perceptron), etc., and this application does not limit it in this way.

[0118] After the training device extracts image features from the sample image using a feature extraction network, it can then classify these features using a first classification network to obtain a first prediction result from the classification model, indicating whether the object in the sample image is a genuine biological feature. If the first classification network is a binary classification network, the first prediction result can indicate either that the object in the sample image is a genuine biological feature, or that the object in the sample image is not a genuine biological feature. If the first classification network is a ternary classification network, the first prediction result can indicate either that the object in the sample image is a genuine biological feature, or that the object in the sample image is a photograph of a genuine biological feature, or that the object in the sample image is a biometric prosthesis. Furthermore, based on the first difference between the first prediction result and the classification label, the training device can adjust the model parameters of the classification model until the classification model converges, resulting in a trained classification model. At this point, the feature extraction network in the trained classification model is considered the trained feature extraction network. This adjustment process can be called the backbone training process.

[0119] For example, the training device can use a first loss function to calculate a first loss function value based on a first difference, and then adjust the model parameters of the classification model based on the first loss function value until a first convergence condition is met, at which point the classification model converges, resulting in a trained feature extraction network. The first loss function can be Softmax Cross-Entropy Loss (exponentially normalized cross-entropy loss), MSE (Mean Squared Error Loss), etc.; the first convergence condition can be that the model parameters of the classification model have been adjusted a first number of times, or that the obtained first loss function value is less than a first threshold.

[0120] Furthermore, the training device adjusts the model parameters of the classification model based on the aforementioned first loss function value, enabling the classification model to learn the image features of the first sample image. Similarly, adjusting the model parameters based on the second and third sample images allows the classification model to learn the image features of the second and third sample images. This allows the classification model to learn the image features of the aforementioned first, second, and third sample images. Subsequently, based on the trained feature extraction network within the trained classification model, features that reflect the objects in the image can be extracted, i.e., accurate image features can be extracted.

[0121] Regarding step S103, the image recognition model includes a first model branch and a second model branch. The first model branch includes a trained feature extraction network and a second classification network; the second model branch includes a trained feature extraction network and a third branch network. Both the second and third classification networks are binary classification networks. When the first classification network is a binary classification network, the network structures of the first, second, and third classification networks can be the same, such as all being linear classification layers. When the first classification model is a three-class classification network, the network structures of the three classification networks can be different, such as the first classification network being an MLP, and the second and third classification networks being linear classification layers.

[0122] In practical scenarios, to simplify the model structure, the trained feature extraction networks in the first and second model branches can be the same network. That is, the second and third classification networks are two parallel branches used to process the output data of the trained feature extraction network respectively. In this case, the second classification network can be called classification branch head B1, and the third classification network can be called classification branch head B2. The image recognition model includes a backbone network (i.e., the trained feature extraction network) and parallel classification branch heads B1 and B2. The output data of the backbone network is the input data of classification branch heads B1 and B2.

[0123] Since the trained feature extraction network has learned the features of real biological features, real biological photos, and the features of biological prostheses, in order to avoid interference from the training process of the first and second model branches on the features already learned by the feature extraction network, which would reduce the accuracy of the image features extracted by the subsequent feature extraction network, the parameters of the trained feature extraction network need to be kept frozen during the training process of the first and second model branches.

[0124] For the first model branch, after obtaining the trained feature extraction network according to the aforementioned step S102, the training device extracts image features from any sample image in the first and second sample images using the trained feature extraction network. Then, it uses the second classification network to classify the image features extracted by the trained feature extraction network, obtaining a second prediction result. Based on the second difference between the second prediction result and the classification label corresponding to the sample image, the model parameters of the second classification network are adjusted until the first model branch converges, resulting in a trained second classification network. For example, the training device calculates a second loss function value based on the second difference using a second loss function, and then adjusts the model parameters of the second classification network based on the second loss function value until the second convergence condition is met, and the first model branch converges, resulting in a trained second classification network. The second loss function can be the aforementioned Softmax Cross-Entropy Loss, MSE, etc.; the second loss function and the aforementioned first loss function can be the same or different; the second convergence condition can be that the model parameters of the second classification network have been adjusted a second number of times, or that the obtained second loss function value is less than a second threshold.

[0125] For the second model branch, after obtaining the trained feature extraction network according to step S102, the training device extracts image features from any sample image between the first and third sample images using the trained feature extraction network. Then, it uses the third classification network to classify the image features extracted by the trained feature extraction network, obtaining a third prediction result. Based on the third difference between the third prediction result and the classification label corresponding to the sample image, the model parameters of the third classification network are adjusted until the second model branch converges, resulting in a trained third classification network. For example, the training device calculates a third loss function value based on the third difference using a third loss function, and then adjusts the model parameters of the third classification network based on the third loss function value until the third convergence condition is met, at which point the second model branch converges, resulting in a trained third classification network. The third loss function can be the aforementioned Softmax Cross-Entropy Loss, MSE, etc.; the third loss function can be the same as or different from the aforementioned first and second loss functions; the third convergence condition can be that the model parameters of the third classification network have been adjusted a third number of times, or that the obtained third loss function value is less than a third threshold.

[0126] The training device can utilize the pre-trained feature extraction network in the image recognition model to extract image features from the first sample image, the second sample image, and the third sample image, respectively. Then, based on the image features of the first and second sample images, the first model branch is trained (this can be called the first branch training process); based on the image features of the first and third sample images, the second model branch is trained (this can be called the second branch training process). There is no temporal order between the first and second branch training processes; they can be performed simultaneously or sequentially, and this invention does not limit this. The first and second branch training processes can be collectively referred to as the branch training process.

[0127] In some embodiments, the biometric feature is the palm of the hand.

[0128] Accordingly, for any sample image, when processing the sample image using a feature extraction network, the input data of the feature extraction network is obtained through the following steps:

[0129] Step A1: Detect the position of the palm and the position of the palm area in the sample image.

[0130] Step A2: Extract the palm image region occupied by the palm from the sample image according to the palm position.

[0131] Step A3: Extract the palm image region occupied by the palm from the sample image according to the position of the palm region.

[0132] Step A4: Fuse the extracted palm image region and palm center image region to obtain input data for inputting to the feature extraction network.

[0133] In the embodiments of this application, during the aforementioned training of the classification model, the input data of the feature extraction network of the initial structure can be obtained based on the above steps; during the aforementioned training of each model branch based on the trained feature extraction network, the input data of the trained feature extraction network can also be obtained based on the above steps.

[0134] For any given sample image, after acquiring the sample image, the training device can extract the palm image region and the palm center image region from the sample image according to the aforementioned steps A1-A3; fuse the extracted palm image region and palm center image region, and use the fusion result as input data to the feature extraction network, and perform the aforementioned backbone training process based on the obtained fusion result to obtain the trained classification model.

[0135] Furthermore, the training device can utilize the trained feature extraction network to extract features from the fusion result of the palm image region and the palm center image region. For example, the trained feature extraction network can be used directly to extract features from the fusion result obtained during the training of the aforementioned classification model; or, the training device can re-acquire the fusion result according to the aforementioned steps A1-A4, and use the trained feature extraction network to extract features from the re-acquired fusion result.

[0136] If the sample image is the first sample image or the second sample image, the training device can perform the first branch training process based on the aforementioned fusion result to train the first model branch; if the sample image is the first sample image or the third sample image, the training device can perform the second branch training process based on the aforementioned fusion result to train the second model branch and obtain the trained image recognition model.

[0137] In practical scenarios, a fusion network can be connected before the feature extraction network to fuse the aforementioned palm image region and palm center image region. Correspondingly, the classification model also includes a fusion network connected to the input of the feature extraction network, and the first and second model branches also include fusion networks connected to the input of the feature extraction network. In other words, the input data for the classification model and the image recognition model can be the aforementioned palm image region and palm center image region.

[0138] During the aforementioned backbone training process, after the training device executes steps A1-A3, it can input the extracted palm image region and palm center image region into the fusion network in the classification model. The fusion network in the classification model fuses the palm image region and palm center image region to obtain the fusion result. Then, using the feature extraction network connected to the fusion network in the classification model, the feature of the fusion result output by the fusion network is extracted to obtain the image features of the sample image.

[0139] During the aforementioned branch training process, after the training device executes steps A1-A3, it can input the extracted palm image region and palm center image region into the fusion network in the image recognition model. The fusion network in the image recognition model fuses the palm image region and palm center image region to obtain the fusion result. Then, using the trained feature extraction network connected to the fusion network in the image recognition model, the fusion result output by the fusion network is used to extract features to obtain the image features of the sample image.

[0140] In the case of a hand as a biometric feature, the training device integrates the global contour features and local texture details of the hand. From the two complementary dimensions of global and local, it jointly determines the input data of the feature extraction network. This reduces the computational load of obtaining the input data of the feature extraction network while avoiding a decrease in the accuracy of the output data of the feature extraction network, thereby improving training efficiency.

[0141] Specifically, in order to obtain more accurate global contour features of the palm and reduce interference from the background area other than the image area occupied by the palm, the training device can extract the palm image area occupied by the palm from the sample image; then the palm image area includes the complete palm, and correspondingly, the palm image area can provide global contour features of the palm.

[0142] Because the palm area occupies a larger region than the fingertips and wrist, it provides richer and more easily identifiable local texture details. Therefore, the training device can extract the palm area from the sample image, providing local texture details of the hand image. If the sample image was captured under visible light, these texture details could be palm prints; if the sample image was captured under infrared light, these texture details could be palm veins.

[0143] The training device can detect the position of the palm and the palm region in the sample image. For example, the palm position can be detected based on a detection algorithm, and the palm region position can be detected based on a palm detection model. The detection algorithm could be MediaPipeHands (palm detection); the palm detection model could be a CNN-based model trained on manually labeled palm regions. The position of the palm can be determined using the bounding box (e.g., the box corresponding to the smallest bounding rectangle of the palm) in the sample image; the position of the palm region can be determined using the bounding box (e.g., the box corresponding to the largest inscribed rectangle of the palm). The position of the bounding box can be represented using the coordinates of its center point, length, and width; alternatively, the position can be represented using the coordinates of its four vertices. This application does not limit this representation.

[0144] Based on the hand's position, the training device can extract the palm image region from the sample image; based on the palm's position, it can extract the palm image region. Since the palm and palm image regions can be fused using a fusion network, and in real-world scenarios, square images are typically processed by the network, for each image region to be extracted, the training device can crop (extract) a square region from the sample image using the center point of the corresponding detection box as the center and the maximum of the length and width of the detection box as the side length. Alternatively, the training device can directly extract the image region within the detection box corresponding to the image region from the sample image and then fill the image region within the detection box with pixels of a specified pixel value to form a square region.

[0145] In some embodiments, to further reduce the computational cost of acquiring the palm image region and improve model training efficiency, step A3 may include the following steps:

[0146] Step B1: Detect key points in the palm of the hand in the sample image.

[0147] Step B2: Determine the midpoint of the first line connecting the detected key points at the base of the index finger and the key points at the base of the little finger, and use this as the first midpoint.

[0148] Step B3: Determine the midpoint of the second line connecting the first midpoint and the detected key point on the palm root, and use it as the second midpoint.

[0149] Step B4: Extract a square image region from the sample image, centered at the second midpoint and with the first connecting line as its edge, as the palm image region.

[0150] In this embodiment, the training device can detect key points in the palm of the sample image based on a detection algorithm. For example, the detection algorithm could be the aforementioned MediaPipe Hands. See also [example description missing]. Figure 2 , Figure 2 This is a schematic diagram of key points in the palm of the hand provided in an embodiment of this application. Figure 2 A black dot represents a detected key point; key points in the palm can include key points on each finger and the palm base key point. A finger can have four key points: one at the fingertip, one at the base, and one at each finger joint; the key point at the base of the finger is called the finger base key point. Thus, 21 key points can be obtained for a single palm.

[0151] Based on the position of the palm image area in the palm, it can be known that the palm image area is usually located below the line connecting the key point at the base of the index finger and the key point at the base of the little finger (i.e., the first line), and the length of the first line is usually no greater than the distance from the key point at the base of the palm to the first line.

[0152] If the line connecting other finger root key points (such as the line connecting the little finger root key point and the middle finger root key point) is used as the first connecting line, the area below this line may include part of the index finger. If the distance from the palm root key point to the first connecting line is used as the side length of the palm image region to be extracted, the extracted palm image region may include other image regions in the sample image besides the palm, which will lead to low accuracy of the obtained palm image region. Therefore, in order to improve the accuracy of the palm image region, the training device determines the midpoint of the first connecting line between the detected index finger root key point and the little finger root key point as the first midpoint; then it determines the midpoint of the second connecting line between the first midpoint and the detected palm root key point as the second midpoint. With the second midpoint as the center and the first connecting line as the side, a square region can be extracted from the sample image, which is the palm image region occupied by the palm.

[0153] In some embodiments, before the training device extracts the palm image region from the sample image, it can also perform image rotation processing on the sample image until the aforementioned first connecting line is horizontal. Thus, when subsequently extracting the palm image region, only simple addition and subtraction operations based on the length of the first connecting line and the coordinates of its two endpoints are needed to determine the coordinates of the other vertices of the palm image region. Furthermore, subsequent palm image region extraction only needs to be based on horizontal and vertical lines, eliminating the need to calculate the coordinates of other vertices based on complex functional relationships or to extract the palm image region based on diagonal lines, thereby simplifying the subsequent extraction process.

[0154] After the training device acquires the palm image region and the palm center image region, it can fuse the extracted palm image region and palm center image region. For example, the fusion network can map the palm image region and palm center image region to the same dimension that the feature extraction network can process, and then add the mapping results element-wise to obtain the fused result, which is used as input data to the feature extraction network. Subsequently, the feature extraction network extracts features based on the input fusion result to obtain the image features of the sample image; based on the image features of the sample image, the corresponding model branch can be trained. Since the trained feature extraction network can already extract accurate image features from the output data of the fusion network, during the training of the model branches, to avoid interference from the training process of the first and second model branches on the image feature extraction process of the aforementioned feature extraction network, the model parameters of the aforementioned fusion network can also be frozen.

[0155] Based on the above processing, when the biometric feature is the palm, the palm image area is larger than that of the fingertips and wrist, thus providing richer and more easily identifiable local texture details. Therefore, by providing these local texture details based on the palm image area and the global contour features of the palm image area, the input data of the feature extraction network is enriched from two complementary dimensions: global and local. This enhances the richness (or information richness) of the image features extracted by the feature extraction network. This approach avoids reducing the accuracy of the feature extraction network's output data while reducing the computational cost of acquiring the input data, thereby improving training efficiency.

[0156] Furthermore, compared to other methods of obtaining input data for feature extraction networks, the above-mentioned method provided in this application embodiment has a simple calculation process, does not require additional complex operations such as generating depth maps and frequency domain maps, and does not require obtaining other information based on additional devices (such as the aforementioned structured light devices or devices for collecting other biometric features), which can reduce costs.

[0157] In some embodiments, any sample image is an image acquired under infrared light source illumination. For example, an infrared light source with an emission wavelength in the range of [850 nm, 940 nm] can be used to illuminate the field of view of an IR (Infrared Radiation) camera; subsequently, the IR camera can acquire images of objects within its field of view, obtaining sample images acquired under infrared light source illumination. In this case, the aforementioned palm image region can be referred to as the palm vein texture region. Compared to images acquired under visible light source illumination, providing palmprint information through the palm image region is advantageous because palm veins are located under the skin, and their complex and unique structure makes them difficult to replicate or forge; furthermore, blood flows within real biometric veins, while photos of real biometric features or biometric prostheses rarely exhibit such blood flow. Therefore, subsequent image recognition based on the palm vein texture region can further improve security in real-world scenarios.

[0158] This application also provides an image recognition method applied to a recognition device. The recognition device and the aforementioned training device can be the same device, or they can be different devices; this application does not limit this.

[0159] As in the previous example, after acquiring the image to be recognized, the access control system can send the image to the recognition device, which then performs image recognition based on a trained image recognition model. Alternatively, in the case where the access control system uses an image recognition model to perform image recognition on the acquired image to be recognized, the access control system itself is the recognition device.

[0160] In real-world scenarios, the recognition device can also be a mobile device used by the user. For example, a user can use a mobile device for authentication. If it is necessary to exit the lock screen state of the mobile device, the user can be authenticated through facial recognition. In this scenario, after the training device obtains a trained image recognition model according to the image recognition model training method provided in any of the foregoing embodiments, it can send the trained image recognition model to the mobile device. The mobile device can then perform image recognition on the acquired image to be recognized based on the received image recognition model to determine whether it is necessary to exit the lock screen state based on the image recognition result.

[0161] See Figure 3 , Figure 3 A flowchart of an image recognition method provided in this application embodiment, the method may include the following steps:

[0162] S301: Acquire the image to be recognized.

[0163] S302: Use the trained image recognition model to process the image to be recognized, and obtain the recognition results of the second and third classification networks in the image recognition model.

[0164] The image recognition model is trained using any of the image recognition model training methods described in the foregoing embodiments.

[0165] S303: Based on the recognition results of the second and third classification networks, determine whether the object in the image to be identified is a real biological feature.

[0166] Based on the above processing, since the image recognition model is trained using any of the image recognition model training methods described in the preceding embodiments, the image recognition model includes a feature extraction network capable of extracting image features with higher accuracy, and a model branch focused on recognizing the type of its corresponding object. Thus, the image features extracted using the feature extraction network are more accurate, and the image recognition result obtained based on these more accurate image features will also be more accurate. Each model branch focuses on recognizing the type of its corresponding object, reducing the mutual interference between the recognition process for photos of real biometric features and the recognition process for biometric prostheses. Therefore, the recognition result obtained by each model branch for the type of object corresponding to that model branch is more accurate. Furthermore, the final recognition result obtained by combining the recognition results of the two model branches is also more accurate, enabling accurate differentiation between real biometric features and photos of real biometric features or biometric prostheses.

[0167] For step S301, the image to be identified is the image in which the object to be identified is a real biological feature.

[0168] In real-world scenarios, after training an image recognition model based on the aforementioned sample images, the recognition device can deploy this model. The recognition device can integrate image acquisition equipment, a light source, and other components. The light source can emit light into the acquisition area of ​​the image acquisition device; the image acquisition device can acquire an image when it detects an object within its acquisition area. For example, a biometric feature could be a hand; an access control system could provide a placement area for a user's hand, and the image acquisition device's acquisition area could cover this area, with the light source emitting light into it. The user can place their hand in this area, and when the image acquisition device detects an object (i.e., the user's hand) within its acquisition area, it can acquire an image of that area, obtaining the image to be recognized; correspondingly, the acquired image will contain the user's hand.

[0169] Similarly, a malicious attacker could place a photograph of a hand (which could be called a 2D paper hand) in the designated area. In this case, the image to be identified by the image acquisition device would contain the photograph of the hand.

[0170] Similarly, a malicious attacker can place a prosthetic hand (which can be called a 3D hand mold) made based on prosthetic manufacturing technology in the placement area. In this case, the image to be identified by the image acquisition device will contain the prosthetic hand.

[0171] For steps S302 and S303, the trained image recognition model includes a first model branch and a second model branch. The first model branch includes a trained feature extraction network and a second classification network; the second model branch includes a trained feature extraction network and a third branch network. In practical scenarios, to simplify the model structure, the trained feature extraction network in the first and second model branches can be the same network, that is, the second and third classification networks are two parallel branches used to process the output data of the trained feature extraction network respectively.

[0172] After extracting image features from the image to be identified using a trained feature extraction network, the extracted image features can be classified using a second classification network and a third classification network, respectively, to obtain the recognition result of the second classification network (which can be called the second recognition result) and the recognition result of the third classification network (which can be called the third recognition result).

[0173] The second recognition result indicates that the object in the image to be recognized is a genuine biometric feature, or that the object in the image to be recognized is a photograph of a genuine biometric feature. The third recognition result indicates that the object in the image to be recognized is a genuine biometric feature, or that the object in the image to be recognized is a biometric prosthesis. Furthermore, the recognition device can combine the second and third recognition results to jointly determine whether the object in the image to be recognized is a genuine biometric feature.

[0174] In real-world scenarios, the image recognition process described above for determining whether an object in an image possesses genuine biometric features can also be the reasoning process of a trained image recognition model.

[0175] In some embodiments, the aforementioned step S303 may include the following steps: if the recognition result of the second classification network indicates that the object in the image to be recognized is a real biometric feature, and the recognition result of the third classification network indicates that the object in the image to be recognized is a real biometric feature, then determine that the object in the image to be recognized is a real biometric feature; otherwise, determine that the object in the image to be recognized is not a real biometric feature.

[0176] In this embodiment, the recognition result of a classification network can be the predicted probability (also known as confidence level) that an object in the image to be identified is a real biometric feature. If the probability is greater than the threshold corresponding to the classification network, the recognition result can be considered to indicate that the object in the image to be identified is a real biometric feature; correspondingly, if the probability is not greater than the threshold corresponding to the classification network, the recognition result can be considered to indicate that the object in the image to be identified is not a real biometric feature, such as a photograph or biometric prosthesis indicating that the object in the image to be identified is a real biometric feature.

[0177] If the second recognition result is greater than the threshold corresponding to the second classification network (which can be denoted as threshold_2D), and the third recognition result is greater than the threshold corresponding to the third classification network (which can be denoted as threshold_3D), then the object in the image to be recognized can be determined to be a real biological feature.

[0178] Similarly, if the second recognition result is not greater than threshold_2D, it can be determined that the object in the image to be recognized is a photo of real biometric features. For example, if the biometric feature is a palm, the image recognition model can output the recognition result of the attack of 2D paper palm (i.e., the aforementioned photo of real biometric features) in the image to be recognized.

[0179] If the third recognition result is not greater than threshold_3D, then the object in the image to be recognized can be determined to be a biometric prosthetic. For example, if the biometric is a hand, the image recognition model can output the recognition result that there is a 3D hand model (i.e. the aforementioned biometric prosthetic) attack in the image to be recognized.

[0180] Based on the above processing, the first model branch is used to identify whether the object in the image to be identified is a real biometric feature or a photograph of a real biometric feature; the second model branch is used to identify whether the object is a real biometric feature or a biometric prosthesis. Compared to using a single classification network to identify real biometric features, photographs of real biometric features, and biometric prostheses, this allows each model branch to focus on identifying its corresponding object type, reducing the mutual interference between the identification process for photographs of real biometric features and the identification process for biometric prostheses. Therefore, the accuracy of the identification result for the object type corresponding to each model branch is higher. Furthermore, the accuracy of the final identification result obtained by combining the identification results of the two model branches is also high, which can accurately distinguish between real biometric features, photographs of real biometric features, and biometric prostheses, ultimately improving the overall anti-counterfeiting accuracy.

[0181] In some embodiments, the biometric feature is the palm of the hand.

[0182] The aforementioned step S302 may include the following steps:

[0183] Step C1: Detect the position of the palm and the position of the palm area in the image to be identified.

[0184] Step C2: Extract the palm image region occupied by the palm from the image to be recognized according to the palm position.

[0185] Step C3: Extract the palm image region occupied by the palm from the image to be recognized according to the position of the palm region.

[0186] Step C4: Fuse the extracted palm image region and the palm center image region to obtain the fusion result.

[0187] Step C5: Process the fusion result using the feature extraction network, second classification network, and third classification network in the trained image recognition model to obtain the recognition results of the second classification network and the third classification network, respectively.

[0188] In this embodiment, when the biometric feature is a palm, the recognition device can extract the palm image region from the image to be recognized; the extracted palm image region provides the global contour features of the palm. Since the local texture details provided by the palm image region, occupied by the center of the palm, are easier to recognize and richer, the recognition device can extract the palm image region from the image to be recognized, providing local texture details of the palm image. By combining the global contour features and local texture details of the palm, the input data of the feature extraction network is determined from two complementary dimensions: global and local. This avoids reducing the accuracy of the output data of the feature extraction network while reducing the computational load of acquiring the input data of the feature extraction network, thereby improving image recognition efficiency. The recognition device performs steps C1-C3 in a manner similar to the training device performing steps A1-A3, and can be referred to the relevant descriptions in the foregoing embodiments, which will not be repeated here.

[0189] In some embodiments, to further reduce the computational cost of acquiring the palm image region and improve image recognition training efficiency, step C3 may include the following steps:

[0190] Step D1: Detect key points in the palm of the hand in the image to be identified.

[0191] Step D2: Determine the midpoint of the third line connecting the detected key points at the base of the index finger and the key points at the base of the little finger, and use this as the third midpoint.

[0192] Step D3: Determine the midpoint of the line connecting the third midpoint and the fourth line connecting the detected palm root key point, and use it as the fourth midpoint.

[0193] Step D4: Extract a square image region from the image to be recognized, centered at the fourth midpoint and with the third connecting line as its edge, as the palm image region.

[0194] In this embodiment, the method by which the recognition device detects key points in the palm of the hand in the image to be recognized is similar to the method by which the aforementioned training device detects key points in the palm of the hand in the sample image. Please refer to the relevant description in the foregoing embodiments, which will not be repeated here.

[0195] Key points in the palm of the image to be recognized include key points on each finger and key points at the base of the palm. To improve the accuracy of the palm image region, the recognition device determines the midpoint of the third line connecting the detected key points at the base of the index finger and the base of the little finger as the third midpoint; then it determines the midpoint of the fourth line connecting the third midpoint and the detected key points at the base of the palm as the fourth midpoint. Using the fourth midpoint as the center and the third line as the edge, a square region can be extracted from the image to be recognized. This square region is the palm image region occupied by the center of the hand.

[0196] In some embodiments, before extracting the palm image region from the image to be recognized, the recognition device may perform image rotation processing on the image to be recognized until the aforementioned third line is horizontal. In this way, when extracting the palm image region subsequently, it is not necessary to calculate the coordinates of each vertex of the palm image region based on complex functional relationships, nor is it necessary to extract the palm image region based on diagonal lines, which can simplify the subsequent extraction process.

[0197] In a real-world scenario, the feature extraction network can be preceded by a network layer (i.e., the aforementioned fusion network) for performing step C4. The recognition device can perform steps C1-C3 to obtain the palm image region and the palm center image region, and then input these regions into the trained image recognition model. The fusion network then fuses the extracted palm image region and palm center image region. Next, the feature extraction network connected to the fusion network extracts features from the fusion result to obtain the image features of the image to be recognized. Finally, the second and third classification networks connected to the feature extraction network classify the image features of the image to be recognized, respectively, to obtain the recognition results of the second and third classification networks.

[0198] In some embodiments, where the aforementioned sample image is an image acquired under infrared light source illumination, the image to be identified is also an image acquired under infrared light source illumination. For example, an infrared light source with an emission wavelength in the range of [850 nm, 940 nm] can be used to illuminate the field of view of the IR camera; subsequently, the IR camera can acquire images of objects within its field of view, obtaining the image to be identified acquired under infrared light source illumination. In this case, the aforementioned palm image region can be referred to as the palm vein texture region. Compared to images acquired under visible light source illumination, providing palmprint information through the palm image region and performing subsequent image recognition based on the palm vein texture region can further improve the security of real-world scenarios.

[0199] In the technical solution of this application, the acquisition, storage, use, processing, transmission, provision and disclosure of various sample images related to biometrics are all carried out with the user's authorization.

[0200] It should be noted that the sample images related to biometrics in this embodiment are not sample images containing the biometrics of a specific user, and do not reflect the personal information of a specific user.

[0201] It should be noted that in this embodiment, all sample images related to biometrics are from publicly available datasets. The training device can obtain the aforementioned sample images and classification labels indicating whether the objects in each sample image possess genuine biometric features from the publicly available datasets.

[0202] Based on the same inventive concept as the image recognition model training method described above, this application also provides an image recognition model training apparatus. See also... Figure 4 , Figure 4 A structural diagram of an image recognition model training device provided in this application embodiment, the device comprising:

[0203] The first acquisition module 401 is used to acquire a first sample image containing real biometric features, a second sample image containing a photograph containing real biometric features, a third sample image containing a biometric prosthesis, and a classification label indicating whether the object in each sample image is a real biometric feature.

[0204] The first training module 402 is used to train a classification model containing a feature extraction network and a first classification network using the first sample image, the second sample image, the third sample image and their respective classification labels, so as to obtain a trained feature extraction network.

[0205] The second training module 403 is used to train the first model branch of the image recognition model with the first sample image, the second sample image and their respective classification labels, and to train the second model branch of the image recognition model with the first sample image, the third sample image and their respective classification labels, so as to obtain the trained image recognition model.

[0206] The first model branch includes a trained feature extraction network and a second classification network; the second model branch includes a trained feature extraction network and a third classification network; during the training of the first model branch and the second model branch, the parameters of the trained feature extraction network are frozen.

[0207] Optionally, the biometric feature is the palm; for any sample image, when processing the sample image using a feature extraction network, the input data for the feature extraction network is obtained through the following steps:

[0208] Detect the position of the palm and the position of the palm area in the sample image;

[0209] Based on the palm position, extract the palm image region occupied by the palm from the sample image;

[0210] Based on the location of the palm region, extract the palm image region occupied by the palm from the sample image;

[0211] The extracted palm image region and palm center image region are fused to obtain the input data for the feature extraction network.

[0212] Optionally, the location of the palm region in the sample image is detected, including:

[0213] Detect key points in the palm of the hand in the sample image;

[0214] Determine the midpoint of the first line connecting the detected key points at the base of the index finger and the key points at the base of the little finger, and use it as the first midpoint.

[0215] Determine the midpoint of the line connecting the first midpoint and the second line connecting the detected key point on the palm root, and use it as the second midpoint;

[0216] Extract a square image region centered at the second midpoint and with the first connecting line as its edge from the sample image, and use it as the palm image region.

[0217] Optionally, any sample image can be an image acquired under infrared light source illumination.

[0218] Based on the same inventive concept as the image recognition method described above, this application also provides an image recognition device. See also... Figure 5 , Figure 5 A structural diagram of an image recognition device provided in an embodiment of this application is shown. The device includes:

[0219] The second acquisition module 501 is used to acquire the image to be recognized;

[0220] The recognition module 502 is used to process the image to be recognized using a trained image recognition model to obtain the recognition results of the second classification network and the third classification network in the image recognition model; wherein, the image recognition model is trained based on any of the image recognition model training methods described in the first aspect above;

[0221] The result determination module 503 is used to determine whether the object in the image to be identified is a real biological feature based on the recognition results of the second classification network and the third classification network respectively.

[0222] Optional, biometric feature is the palm;

[0223] The identification module 502 is specifically used for:

[0224] Detect the position of the palm and the position of the palm area in the image to be identified;

[0225] Based on the palm position, extract the palm image region occupied by the palm from the image to be identified;

[0226] Based on the location of the palm region, extract the palm image region occupied by the palm from the image to be identified;

[0227] The extracted palm image region and the center of the palm image region are fused to obtain the fusion result;

[0228] The fusion result is processed using the feature extraction network, the second classification network, and the third classification network in the trained image recognition model to obtain the recognition results of the second classification network and the third classification network, respectively.

[0229] Optionally, the identification module 502 is specifically used for:

[0230] Detect key points in the palm of the hand in the image to be identified;

[0231] Determine the midpoint of the third line connecting the detected key points at the base of the index finger and the key points at the base of the little finger, and use this as the third midpoint.

[0232] The midpoint of the line connecting the third midpoint and the fourth line connecting the detected key point of the palm root is determined as the fourth midpoint.

[0233] Extract a square image region centered at the fourth midpoint and with the third connecting line as its edge from the image to be identified, and use it as the palm image region.

[0234] Optionally, the result determination module 503 is specifically used for:

[0235] If the recognition result of the second classification network indicates that the object in the image to be identified is a real biometric feature, and the recognition result of the third classification network indicates that the object in the image to be identified is a real biometric feature, then the object in the image to be identified is determined to be a real biometric feature.

[0236] Otherwise, it is determined that the object in the image to be identified does not possess genuine biometric features.

[0237] Optionally, the image to be identified is an image acquired under infrared light source illumination.

[0238] This application also provides an electronic device, such as... Figure 6 As shown, it includes: a memory 601 for storing computer programs; and a processor 602 for executing the program stored in the memory 601 to implement the image recognition model training method steps described in any of the above embodiments, or to implement the image recognition method steps described in any of the above embodiments.

[0239] Furthermore, the aforementioned electronic device may also include a communication bus and / or a communication interface, with the processor 602, communication interface, and memory 601 communicating with each other via the communication bus.

[0240] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0241] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0242] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0243] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0244] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described image recognition model training methods, or implements the steps of any of the above-described image recognition methods.

[0245] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the image recognition model training methods in the above embodiments, or to implement any of the image recognition methods in the above embodiments.

[0246] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a solid-state drive (SSD), etc.

[0247] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0248] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, electronic devices, computer-readable storage media, and computer program products are basically similar to the method embodiments, and therefore the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0249] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A method for training an image recognition model, characterized in that, The method includes: Acquire a first sample image containing real biometric features, a second sample image containing a photograph of real biometric features, a third sample image containing a biometric prosthesis, and a classification label indicating whether the object in each sample image is a real biometric feature; Using the first sample image, the second sample image, the third sample image, and their respective classification labels, a classification model containing a feature extraction network and a first classification network is trained to obtain a trained feature extraction network. Using the first sample image, the second sample image, and their respective classification labels, the first model branch of the image recognition model with the initial structure is trained, and using the first sample image, the third sample image, and their respective classification labels, the second model branch of the image recognition model is trained, thus obtaining a trained image recognition model. The first model branch includes a trained feature extraction network and a second classification network; the second model branch includes a trained feature extraction network and a third classification network; during the training of the first model branch and the second model branch, the parameters of the trained feature extraction network are frozen.

2. The method according to claim 1, characterized in that, Biological characteristic: palm; For any sample image, when processing the sample image using a feature extraction network, the input data for the feature extraction network is obtained through the following steps: Detect the position of the palm and the position of the palm area in the sample image; Based on the palm position, extract the palm image region occupied by the palm from the sample image; Based on the location of the palm region, extract the palm image region occupied by the palm from the sample image; The extracted palm image region and palm center image region are fused to obtain the input data for the feature extraction network.

3. The method according to claim 2, characterized in that, Detecting the location of the palm region in the sample image includes: Detect key points in the palm of the hand in the sample image; Determine the midpoint of the first line connecting the detected key points at the base of the index finger and the key points at the base of the little finger, and use it as the first midpoint. Determine the midpoint of the line connecting the first midpoint and the second line connecting the detected key point on the palm root, and use it as the second midpoint; Extract a square image region centered at the second midpoint and with the first connecting line as its edge from the sample image, and use it as the palm image region.

4. The method according to any one of claims 1-3, characterized in that, Each sample image is an image acquired under infrared light source illumination.

5. An image recognition method, characterized in that, The method includes: Acquire the image to be recognized; The image to be recognized is processed using a trained image recognition model to obtain the recognition results of the second classification network and the third classification network in the image recognition model; wherein, the image recognition model is trained based on the image recognition model training method according to any one of claims 1-4; Based on the recognition results of the second classification network and the third classification network, it is determined whether the object in the image to be identified has real biological characteristics.

6. The method according to claim 5, characterized in that, Biological characteristic: palm; The process of using a trained image recognition model to process the image to be recognized, and obtaining the recognition results of the second and third classification networks in the image recognition model, includes: Detect the position of the palm and the position of the palm area in the image to be identified; Based on the palm position, extract the palm image region occupied by the palm from the image to be identified; Based on the location of the palm region, extract the palm image region occupied by the palm from the image to be identified; The extracted palm image region and the center of the palm image region are fused to obtain the fusion result; The fusion result is processed using the feature extraction network, the second classification network, and the third classification network in the trained image recognition model to obtain the recognition results of the second classification network and the third classification network, respectively.

7. The method according to claim 6, characterized in that, Detecting the location of the palm region in the image to be identified includes: Detect key points in the palm of the hand in the image to be identified; Determine the midpoint of the third line connecting the detected key points at the base of the index finger and the key points at the base of the little finger, and use this as the third midpoint. The midpoint of the line connecting the third midpoint and the fourth line connecting the detected key point of the palm root is determined as the fourth midpoint. Extract a square image region centered at the fourth midpoint and with the third connecting line as its edge from the image to be identified, and use it as the palm image region.

8. The method according to claim 5, characterized in that, Based on the recognition results of the second classification network and the third classification network, determining whether the object in the image to be identified is a genuine biometric feature includes: If the recognition result of the second classification network indicates that the object in the image to be identified is a real biometric feature, and the recognition result of the third classification network indicates that the object in the image to be identified is a real biometric feature, then the object in the image to be identified is determined to be a real biometric feature. Otherwise, it is determined that the object in the image to be identified does not possess genuine biometric features.

9. The method according to any one of claims 5-8, characterized in that, The image to be identified is an image acquired under infrared light source illumination.

10. An image recognition model training device, characterized in that, The device includes: The first acquisition module is used to acquire a first sample image containing real biometric features, a second sample image containing a photograph containing real biometric features, a third sample image containing a biometric prosthesis, and a classification label indicating whether the object in each sample image has real biometric features. The first training module is used to train a classification model containing a feature extraction network and a first classification network using the first sample image, the second sample image, the third sample image and their respective classification labels, so as to obtain a trained feature extraction network. The second training module is used to train the first model branch of the image recognition model with the first sample image, the second sample image and their respective classification labels, and to train the second model branch of the image recognition model with the first sample image, the third sample image and their respective classification labels, so as to obtain the trained image recognition model. The first model branch includes a trained feature extraction network and a second classification network; the second model branch includes a trained feature extraction network and a third classification network; during the training of the first model branch and the second model branch, the parameters of the trained feature extraction network are frozen.

11. The apparatus according to claim 10, characterized in that, Biological characteristic: palm; For any sample image, when processing the sample image using a feature extraction network, the input data for the feature extraction network is obtained through the following steps: Detect the position of the palm and the position of the palm area in the sample image; Based on the palm position, extract the palm image region occupied by the palm from the sample image; Based on the location of the palm region, extract the palm image region occupied by the palm from the sample image; The extracted palm image region and palm center image region are fused to obtain the input data for inputting to the feature extraction network; Detecting the location of the palm region in the sample image includes: Detect key points in the palm of the hand in the sample image; Determine the midpoint of the first line connecting the detected key points at the base of the index finger and the key points at the base of the little finger, and use it as the first midpoint. Determine the midpoint of the line connecting the first midpoint and the second line connecting the detected key point on the palm root, and use it as the second midpoint; Extract a square image region centered at the second midpoint and with the first connecting line as its edge from the sample image, and use it as the palm image region; Each sample image is an image acquired under infrared light source illumination.

12. An image recognition device, characterized in that, The device includes: The second acquisition module is used to acquire the image to be recognized; The recognition module is used to process the image to be recognized using a trained image recognition model to obtain the recognition results of the second classification network and the third classification network in the image recognition model; wherein, the image recognition model is trained based on the image recognition model training method according to any one of claims 1-4; The result determination module is used to determine whether the object in the image to be identified is a real biometric feature based on the recognition results of the second classification network and the third classification network.

13. The apparatus according to claim 12, characterized in that, Biological characteristic: palm; The identification module is specifically used for: Detect the position of the palm and the position of the palm area in the image to be identified; Based on the palm position, extract the palm image region occupied by the palm from the image to be identified; Based on the location of the palm region, extract the palm image region occupied by the palm from the image to be identified; The extracted palm image region and the center of the palm image region are fused to obtain the fusion result; The fusion result is processed using the feature extraction network, the second classification network, and the third classification network in the trained image recognition model to obtain the recognition results of the second classification network and the third classification network, respectively. The identification module is specifically used for: Detect key points in the palm of the hand in the image to be identified; Determine the midpoint of the third line connecting the detected key points at the base of the index finger and the key points at the base of the little finger, and use this as the third midpoint. The midpoint of the line connecting the third midpoint and the fourth line connecting the detected key point of the palm root is determined as the fourth midpoint. Extract a square image region centered at the fourth midpoint and with the third connecting line as its edge from the image to be identified, and use it as the palm image region. The result determination module is specifically used for: If the recognition result of the second classification network indicates that the object in the image to be identified is a real biometric feature, and the recognition result of the third classification network indicates that the object in the image to be identified is a real biometric feature, then the object in the image to be identified is determined to be a real biometric feature. Otherwise, it is determined that the object in the image to be identified does not possess genuine biometric features; The image to be identified is an image acquired under infrared light source illumination.

14. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method of any one of claims 1-4, or implements the method of any one of claims 5-9.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1-4, or implements the method of any one of claims 5-9.

16. A computer program product, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method of any one of claims 1-4, or to implement the method of any one of claims 5-9.

Images