Deep learning hand foreign object recognition and palm and back of hand same person discrimination method and system

Abstract

The application discloses a deep learning hand foreign matter identification and palm-back-of-hand same person discrimination method and system, and the method comprises the following steps: face recognition is used to verify the identity of the morning inspection personnel, and corresponding identity information is obtained; a pre-registered palm mask and back-of-hand mask are called; hand key points are detected, and if the hand key points do not fall into the palm mask, adjustment is prompted, and if the hand key points fall into the palm mask, palm image and coordinates are stored; a flipped frame and a back-of-hand image are used for dynamic continuity verification, and if the hand key points fall into the back-of-hand mask, the image and coordinates are stored; the image and coordinates are cut to obtain a hand foreign matter identification result through foreign matter detection fusion; a skeletal proportion feature and a relative difference value are calculated to obtain a skeletal proportion similarity score; dynamic continuity verification and score weighting fusion are used to determine whether the palm and back of hand are the same person by taking passing as a necessary condition and combining a comprehensive score threshold value; through the deep learning foreign matter detection of sub-image cutting, the fusion determination of the skeletal proportion feature and the dynamic continuity verification, the hand foreign matter identification and the palm-back-of-hand same person discrimination are realized.

Description

Technical Field

[0001] This invention relates to the field of identity verification and security technology, and more specifically to a method and system for deep learning-based foreign object recognition of hands and identification of the same person's palm and back of the hand. Background Technology

[0002] Food safety management in primary and secondary school canteens is crucial to the physical and mental health of students, and the morning hand checks of canteen staff directly impact the food safety of teachers and students. By conducting morning hand checks for each meal, we ensure the health of the hands of those involved in food processing, thereby guaranteeing the safety of the food preparation process.

[0003] With the development of technology, some intelligent systems and devices have gradually emerged, such as those that automatically identify foreign objects or scars on hands based on artificial intelligence and can upload detailed morning inspection information to the backend system with one click. However, the existing intelligent systems are relatively simple in function, simply emphasizing whether they have the function of identifying foreign objects on hands, but lacking a mechanism to guarantee the effectiveness of the identification. In recent years, biometric identity verification technology has been gradually applied to morning inspection scenarios, but existing technologies have obvious limitations: ① Single-modal hand recognition schemes (only palm or back of hand) are easily affected by posture and lighting, and cannot solve the problem of fraudulent inspection behavior using "not one's own hand". Attackers can use an image of their own palm to pass verification, and then replace it with someone else's back of hand to perform the actual operation. ② Existing palm-back consistency discrimination schemes mostly rely on skin color similarity or vein texture matching. Skin color features are easily affected by the complex lighting environment of the cafeteria, leading to misjudgment; vein recognition requires near-infrared imaging equipment, which is costly and has special hardware requirements, making it unsuitable for large-scale deployment in cafeterias.

[0004] Therefore, there is an urgent need for a non-contact method for identifying the same person by palm and back of hand, which is adapted to the characteristics of the canteen environment, has low cost, strong anti-counterfeiting capabilities, and has the function of identifying foreign objects on the hand. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for identifying foreign objects in the hand and determining whether the palm and back of the hand belong to the same person through deep learning. By using deep learning foreign object detection through sub-image segmentation and the fusion judgment of skeletal proportion features and dynamic continuity verification, the invention achieves identification of foreign objects in the hand and determination of whether the palm and back of the hand belong to the same person.

[0006] To achieve the above objectives, the present invention provides the following technical solution: A deep learning-based method for foreign object recognition in the hand and for determining if the palm and back of the hand belong to the same person includes the following steps: S1. Verify the identity of the morning check-up personnel through facial recognition and obtain the corresponding identity information; S2. Retrieve the pre-registered hand mask of the corresponding morning check-up personnel based on the identity information; the hand mask includes: a palm mask and a back of hand mask; S3. Determine if a hand exists in the detection area. If so, extract the key points of the hand and determine if the key points fall within the palm mask area. If not, output an adjustment prompt until the requirements are met. If the key points of the hand fall within the palm mask area, save the palm image and the coordinates of the key points. S4. Acquire continuous video frames and images of the back of the hand during the flipping process, and perform dynamic continuity verification on the continuous video frames during the flipping process; and determine whether the key points of the back of the hand fall within the back of the hand mask. If so, save the back of the hand image and the coordinates of the key points of the back of the hand. S5. Perform image preprocessing and segmentation on the saved palm and back of hand images and corresponding key point coordinates, perform foreign object detection on the segmented images, and fuse the foreign object detection results to obtain the hand foreign object recognition result. S6. Based on the saved key point coordinates of the palm and back of the hand, calculate the skeletal proportion features of the palm and back of the hand, calculate the relative difference of the corresponding skeletal proportion features of the palm and back of the hand, and obtain the skeletal proportion similarity score. S7. The results of dynamic continuity verification are weighted and fused with the skeletal proportion similarity score. With passing dynamic continuity verification as a necessary condition, and combined with the comprehensive score threshold, the results are determined to determine whether the palm and the back of the hand belong to the same person and output.

[0007] Furthermore, in S2, the process of creating the hand mask includes: Initial images of the palm and back of the hand were acquired separately, and pure hand region images were obtained through image binarization and hole filling. A morphological dilation algorithm is used to expand the pure hand region, and the outer contour coordinates of the expanded pure hand region image are extracted to generate corresponding palm and back of hand masks; wherein the dilation threshold of the morphological dilation algorithm can be customized.

[0008] Furthermore, in S4, the dynamic continuity verification process includes: state sequence integrity verification, time window verification, spatial location continuity verification, and skin feature continuity verification; The state sequence integrity verification is used to detect whether the palm state, transition state and back of hand state appear sequentially in consecutive video frames. The time window verification is used to determine whether the time interval between flipping the palm state to the back of the hand state is within a preset time threshold. The spatial position continuity verification is used to track the movement trajectory of the center point of the hand region and determine whether the displacement of the hand region in adjacent frames exceeds a preset displacement threshold. The skin feature continuity verification is used to calculate the cosine similarity of the color histograms of the hand regions in adjacent frames, and to determine whether the minimum similarity is lower than the first preset similarity threshold or whether the average similarity is lower than the second preset similarity threshold.

[0009] Furthermore, in step S5, the image preprocessing and segmentation process includes: The saved palm and back of hand images are binarized and hole-filled to remove background interference, resulting in a pure hand region image. The pure hand region image is segmented into 6 sub-images based on the coordinates of key points on the palm and back of the hand; the 6 sub-images include: thumb sub-image, index finger sub-image, middle finger sub-image, ring finger sub-image, little finger sub-image, and palm image; An open object detection algorithm is used to identify whether there are foreign objects in each sub-image, and the identification results of each sub-image are fused to obtain the foreign object identification result of the hand.

[0010] Furthermore, in S6, the skeletal proportion features include: the length-to-width ratio of the palm, the relative length ratio of the index finger, the relative length ratio of the middle finger, the relative length ratio of the ring finger, the relative length ratio of the little finger, and the relative length ratio of the thumb; The calculation process of the bone proportion similarity score includes: calculating the relative difference between the corresponding bone proportion features of the palm side and the back side of the hand; if all relative differences do not exceed the preset difference threshold, the bone proportion similarity score is obtained through bone proportion consistency constraints.

[0011] Furthermore, in S7, the formula for weighted fusion of the dynamic continuity verification result and the bone proportion similarity score is as follows: ; in, Scoring is based on the similarity of skeletal proportions. The score is for dynamic continuity verification, with 1 for passing and 0 for failing. Scoring based on skeletal proportions; weighting coefficients , .

[0012] Furthermore, in S7, the comprehensive score threshold supports adaptive adjustment, specifically as follows: For first-time registered users, multiple sets of palm-back images are collected to calculate the mean and standard deviation of skeletal proportion features and establish a personalized benchmark. During real-time discrimination, the comprehensive score threshold is adjusted to a strict mode threshold, a standard mode threshold, or a lenient mode threshold based on the range of the standard deviation of the user's historical detected bone ratio differences.

[0013] This invention also provides a system for performing deep learning methods for foreign object recognition in the hand and for determining whether the palm and back of the hand belong to the same person, comprising: The identity verification module is used to verify the identity of morning check-up personnel through facial recognition and obtain the corresponding identity information; The mask retrieval module is used to retrieve the pre-registered hand mask of the corresponding person based on the identity information; the hand mask includes: a palm mask and a back of hand mask; The hand localization and key point detection module is used to determine whether a hand exists in the detection area. If so, it extracts the key points of the hand and determines whether the key points fall within the palm mask area. If not, it outputs a posture adjustment prompt until the requirements are met. If the key points of the hand fall within the palm mask area, it saves the palm image and the coordinates of the key points of the palm. The dynamic continuity verification module is used to acquire continuous video frames and images of the back of the hand during the flipping process, and to perform dynamic continuity verification on the continuous video frames during the flipping process; and to determine whether the key points of the back of the hand fall within the back of the hand mask. If so, the back of the hand image and the coordinates of the key points of the back of the hand are saved. The foreign object recognition module is used to perform image preprocessing and segmentation on the saved palm and back of hand images and corresponding key point coordinates, and to perform foreign object detection on the segmented images and fuse the results to obtain the hand foreign object recognition result. The same person identification module is used to calculate the skeletal proportion features of the palm and back of the hand based on the saved key point coordinates of the palm and back of the hand, calculate the relative difference of the corresponding skeletal proportion features of the palm and back of the hand, and obtain the skeletal proportion similarity score; and weightedly fuse the result score of dynamic continuity verification with the skeletal proportion similarity score, and determine whether the palm and back of the hand belong to the same person by combining the comprehensive score threshold, with passing dynamic continuity verification as a necessary condition.

[0014] The present invention also provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements a deep learning method for identifying foreign objects in the hand and determining whether the palm and back of the hand belong to the same person.

[0015] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements a deep learning method for identifying foreign objects in the hand and determining whether the palm and back of the hand belong to the same person.

[0016] According to specific embodiments provided by the present invention, the present invention has the following technical effects compared to the prior art: This invention combines facial identity verification with a pre-registration mask to achieve precise binding of personnel identity and standardized constraints on the hand region, effectively improving the positioning accuracy and anti-interference capability of key hand point extraction. It employs a sub-image segmentation and result fusion strategy based on key point coordinates to overcome the vulnerability of overall image detection to background noise interference, significantly improving the fine-grainedness and accuracy of foreign object recognition on the hand. By weightedly fusing dynamic continuity verification of the flipping process with bio-similarity calculation based on skeletal proportion features, it not only overcomes the matching difficulties caused by differences in appearance between the palm and back of the hand by utilizing the physiological stability of skeletal proportion features, but also uses passing dynamic continuity verification as a necessary condition. This ensures recognition efficiency while improving the anti-counterfeiting robustness and reliability of identifying the same person, perfectly meeting the stringent requirements of scenarios with high authenticity demands, such as morning health checks. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0018] The following description, in conjunction with the accompanying drawings, further illustrates the deep learning method and system for identifying foreign objects in the hand and determining whether the palm and back of the hand belong to the same person. Figure 1 This is a flowchart of the morning hand inspection process in Embodiment 1 of the present invention; Figure 2 This is a flowchart of the hand mask creation process in Embodiment 1 of the present invention; Figure 3 This is a flowchart of the hand positioning process in Embodiment 1 of the present invention; Figure 4 This is a flowchart of the hand foreign object recognition process in Embodiment 1 of the present invention; Figure 5 This is a schematic diagram showing the numbering of key hand points in Embodiment 1 of the present invention; Figure 6 This is a flowchart of the process for identifying the same person by their palm and hand in Embodiment 1 of this invention; Figure 7 Here are example diagrams of 21 key points of the hand; Figure 8 This is an example of cutting the hand into 6 sub-images. Detailed Implementation

[0019] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0020] To better understand the purpose, structure, and function of this invention, the invention will be described in further detail below with reference to the accompanying drawings.

[0021] Example 1 like Figure 1 As shown, this invention provides a deep learning method for identifying foreign objects in the hand and determining if the palm and back of the hand belong to the same person, comprising the following steps: S1. Verify the identity of the morning check-up personnel through facial recognition and obtain the corresponding identity information; In this specific embodiment, facial recognition is used to log into the morning inspection system and obtain the staff's identity information for subsequent hand morning inspection information association.

[0022] S2. Retrieve the pre-registered hand mask of the corresponding person based on the identity information; the hand mask includes: palm mask and back of hand mask; In step S2, the process of creating the hand mask includes: Initial images of the palm and back of the hand were acquired separately, and pure hand region images were obtained through image binarization and hole filling. A morphological dilation algorithm is used to expand the pure hand region, and the outer contour coordinates of the expanded pure hand region image are extracted to generate corresponding palm and back of hand masks; wherein the dilation threshold of the morphological dilation algorithm can be customized.

[0023] Specifically, in this embodiment: for users registering for the first time, multiple sets of palm-back images are collected to calculate the mean and standard deviation of skeletal proportion features and establish a personalized benchmark. During real-time discrimination, the final judgment threshold is adjusted to a strict mode threshold, a standard mode threshold, or a lenient mode threshold based on the range of the standard deviation of the user's historical detected bone ratio differences.

[0024] The system uses facial recognition to first confirm the identity information of the person currently undergoing morning checks. Then, it automatically retrieves the palm and back of the hand mask from the person's registration and guides the user to correctly place their hands in the morning check camera area, improving the accuracy of key hand feature points.

[0025] Mask creation is divided into palm masks and back-of-hand masks. Upon entering the mask creation page, first place both hands in the center area of ​​the device, directly below the camera. After capturing the image, image binarization and hole filling are used to obtain a pure hand area image. A morphological dilation algorithm is then used to expand this pure hand area image, creating palm masks for both hands. Next, the user is prompted to flip their palms to show the backs of their hands, and a mask for the backs of the hands is created similarly. Both palm and back-of-hand masks have an adjustable dilation threshold, allowing for a certain degree of misalignment tolerance during morning inspection recognition.

[0026] S3. Determine if a hand exists in the detection area. If so, extract the key points of the hand and determine if the key points fall within the palm mask area. If not, output a posture adjustment prompt until the requirements are met. If the key points of the hand fall within the palm mask area, save the palm image and the coordinates of the key points of the palm. S4. Acquire continuous video frames and images of the back of the hand during the flipping process, and perform dynamic continuity verification on the continuous video frames during the flipping process; and determine whether the key points of the back of the hand fall within the back of the hand mask. If so, save the back of the hand image and the coordinates of the key points of the back of the hand. In this specific embodiment, the palm is inserted into the area of ​​the hand inspection camera. The hand key point detection algorithm detects and determines in real time whether all 21 key points are within the mask area. If so, an image of the palm is captured and a voice prompt is given to rotate the palm. An image of the back of the hand is also captured to obtain a palm-back image group. At the same time, during the palm rotation process, it is determined in real time whether the dynamic continuity is met.

[0027] For each group of collected palm-back images, a hand key point detection algorithm is used to determine the consistency of bone proportions. The dynamic continuity results are then fused to determine whether the hand being inspected in the morning has a risk of being replaced. If so, the risk information, the image, and detailed information about its key points are recorded.

[0028] In S4, the dynamic continuity verification process includes: state sequence integrity verification, time window verification, spatial location continuity verification, and skin feature continuity verification. The state sequence integrity verification is used to detect whether the palm state, transition state and back of hand state appear sequentially in consecutive video frames. The time window verification is used to determine whether the time interval from the palm state to the back of the hand state is within a preset time threshold. The spatial position continuity verification is used to track the movement trajectory of the center point of the hand area and determine whether the displacement of adjacent frames exceeds a preset displacement threshold. The skin feature continuity verification is used to calculate the cosine similarity of the color histograms of the hand regions in adjacent frames, and to determine whether the minimum similarity is lower than the first preset similarity threshold or whether the average similarity is lower than the second preset similarity threshold.

[0029] This embodiment specifically involves: entering the morning check page and starting the hand check. Extend both hands, palms up, into the palm mask area. An open object detection algorithm is used to determine if a hand exists in the mask area. If so, a keypoint detection algorithm is used to obtain the hand's skeletal points. Then, it is determined whether the hand skeletal points fall within the palm mask area. If not, a voice prompt is given to adjust the hand posture until the requirements are met. If the current hand posture meets the requirements, the image and skeletal point coordinates are saved. A voice prompt is given to flip the palm, ensuring the hand does not leave the camera's field of view during the flipping process. Similarly, a keypoint detection algorithm is used to obtain the hand's skeletal points, and then it is determined whether the skeletal points fall within the back of the hand mask area. If not, a voice prompt is given to adjust the hand posture until the requirements are met. If the current hand posture meets the requirements, the image and skeletal point coordinates are saved.

[0030] S5. Perform image segmentation on the saved palm and back of hand images and corresponding key point coordinates, and perform foreign object detection and fusion on the segmented images to obtain the hand foreign object recognition result. In step S5, the image segmentation process includes: Based on the coordinates of key points on the palm and back of the hand, the saved images of the palm and back of the hand are segmented into 6 sub-images; the 6 sub-images include: thumb sub-image, index finger sub-image, middle finger sub-image, ring finger sub-image, little finger sub-image, and palm image; An open object detection algorithm is used to identify whether there are foreign objects in each sub-image, and the identification results of each sub-image are fused to obtain the foreign object identification result of the hand.

[0031] This embodiment specifically involves: performing image binarization, hole filling, and background interference removal on the saved images of the palm and back of the hand to obtain the pure hand region. Based on the image and skeletal point coordinates, the palm and back of the hand regions are then segmented as follows: (1) Images of the palm: thumb, index finger, middle finger, ring finger, little finger, and palm. (2) Images of the back of the hand: thumb, index finger, middle finger, ring finger, little finger, and back of the hand; Segmenting the palm and back of the hand images into six sub-images enriches the image scale of the detection model and improves the detection effect of small foreign objects. The open object detection algorithm YOLOE is used to identify whether each sub-image of the palm and back of the hand contains foreign objects such as scratches, nail polish, or dirt. The identification results from the palm and back of the hand images are then fused. If the foreign object detection score exceeds a set threshold, an alarm is triggered in real time, and the abnormal identification result is pushed to the relevant person in charge for manual verification to ensure the reliability of the identification result.

[0032] Using a deep learning object detection algorithm, the image group of the palm-back of the hand is divided into 6 sub-images for foreign object detection. Then, the results of the 6 sub-images are fused back into the original image, and the foreign object information is recorded in detail.

[0033] The information on identifying the same person by palm and back of hand, along with the information on foreign objects in the hand, is merged and recorded in the morning inspection system. For cases with abnormal morning inspection risks, the information is promptly pushed to relevant reviewers for manual verification, as shown in Table 1.

[0034] Table 1

[0035] S6. Based on the saved key point coordinates of the palm and back of the hand, calculate the skeletal proportion features of the palm and back of the hand, calculate the relative difference of the corresponding skeletal proportion features of the palm and back of the hand, and obtain the skeletal proportion similarity score. In S6, the skeletal proportion features include: the length-to-width ratio of the palm, the relative length ratio of the index finger, the relative length ratio of the middle finger, the relative length ratio of the ring finger, the relative length ratio of the little finger, and the relative length ratio of the thumb; The calculation process of the bone proportion similarity score includes: calculating the relative difference between the corresponding bone proportion features of the palm side and the back side of the hand; if all relative differences do not exceed the preset difference threshold, the bone proportion similarity score is obtained through bone proportion consistency constraints.

[0036] This embodiment includes: ① Dynamic continuity verification The system performs real-time analysis of continuous video frames recorded during hand localization to verify whether the hand-to-back flipping process conforms to the natural continuity constraint, including: state sequence integrity verification, time window verification, spatial position continuity verification, and skin feature continuity verification. If the hand disappears, jumps in position, or undergoes abrupt changes in features during the flipping process, a substitution attack is identified, and the discrimination process is terminated. ② Skeletal proportion consistency judgment Skeletal proportion features were extracted from the preprocessed palm and back of hand images, respectively, and a deep learning-based hand keypoint detection algorithm was used to achieve accurate localization and measurement. The key point detection algorithm outputs the pixel coordinates and confidence scores of 21 key points of the hand bones. The key points include: the center point of the wrist joint, the thumb bone chain (4 points), the index finger bone chain (4 points), the middle finger bone chain (4 points), the ring finger bone chain (4 points), and the little finger bone chain (4 points). The skeletal proportion features are calculated based on the coordinates of the detected key points, including: the length-to-width ratio of the palm, the relative length ratio of the index finger, the relative length ratio of the middle finger, the relative length ratio of the ring finger, the relative length ratio of the little finger, and the relative length ratio of the thumb; the relative difference between the corresponding skeletal proportion features on the palm side and the back side of the hand is calculated, and if all proportion differences do not exceed the preset threshold, the skeletal proportion consistency constraint is passed. S7. The result score of dynamic continuity verification is weighted and fused with the score of skeletal proportion similarity. With passing dynamic continuity verification as a necessary condition, combined with the comprehensive score threshold, the result is determined to determine whether the palm and the back of the hand belong to the same person and the result is output.

[0037] In S7, the weighted fusion process is as follows: Based on the dynamic continuity verification results and the skeleton proportion consistency judgment results, the final similarity score is calculated using a weighted fusion method: ; in, The score is for dynamic continuity verification; a score of 1 indicates a pass and a score of 0 indicates a failure. Scoring is based on skeletal proportion similarity; the weighting coefficients satisfy... , .

[0038] If the following conditions are met, the hand is determined to be from the same person and can pass the morning inspection; otherwise, it is determined to be a substitute inspection and an alarm is triggered.

[0039] ; ; ,in This is the final threshold for determination.

[0040] Example 2 1. Create a hand mask When a user first uses this invention's system, they need to register and input their hand information. A deep learning keypoint detection algorithm is used to extract 21 skeletal points from the palm and back of the hand. Image binarization is then used to remove the background, extract the pure hand connected components, and fill any holes in these components before performing an overall dilation operation. The dilation threshold can be set in the system settings. The outer contour coordinates of the dilated image are extracted to create a hand mask. Mask information is shown in Table 2.

[0041] Table 2

[0042] The personalized hand mask not only provides a quick and easy way to identify and replace objects, but more importantly, it guides users to position their hands correctly, preventing excessive changes in hand posture and improving the accuracy of identifying key hand points and foreign objects.

[0043] 2. Hand key point detection After undergoing facial recognition to verify their identity, morning check-up personnel are directed to a hand inspection page. The page automatically retrieves the personnel's hand mask from the database. Employing a deep object detection algorithm, the system first analyzes the images captured by the morning check-up camera in real time to determine if a hand is present. If not, a voice prompt instructs the user to place their hand within the hand mask area as required. If a hand is detected, a deep learning keypoint detection algorithm extracts 21 key points from the hand and analyzes whether these points fall within the current hand mask area. If not, a voice prompt instructs the user to adjust their hand posture, and an arrow symbol on the device interface indicates the correct direction for adjustment until the desired posture is achieved. If the posture meets the requirements, the palm image and keypoint coordinates are saved, and a voice prompt instructs the user to flip their palm to show the back of their hand, saving the image frame of the palm flipped to the back of the hand in real time. Similarly, the system checks if the back of the hand posture meets the requirements, saving images and keypoint coordinates of compliant hands.

[0044] 3. Foreign object detection in the hand Image binarization, hole filling, and background interference removal are performed on the saved palm and back of the hand images respectively to obtain the pure hand region. To enrich the image scale of the detection model and improve the detection effect of small foreign objects, this invention divides the pure hand region of the palm and back of the hand into sub-images based on the key point coordinates of the palm and back of the hand, namely: thumb (2,3,4), index finger (5,6,7,8), middle finger (9,10,11,12), ring finger (13,14,15,16), little finger (17,18,19,20), and palm (0,1,2,5,9,13,17).

[0045] This invention uses the open-source object detection algorithm YOLOE to detect foreign objects on the hands. A basic detection model is trained based on a production sample library, capable of detecting items such as nail polish, dirt, scratches, bandages, and tattoos. When encountering a foreign object category not trained on the model, the user manually selects the object area on the system's recognition page, adds a name (e.g., "ring"), and saves the selection. Subsequent encounters with rings will automatically recognize them as foreign objects, significantly improving the scalability and flexibility of the foreign object recognition system.

[0046] 4. Identifying the same person by examining the palm and back of the hand. To improve the safety and reliability of morning hand inspections, this invention proposes a mechanism for determining whether the palm and back of the hand belong to the same person. Based on the dynamic continuity verification of the palm and back of the hand, the consistency judgment of bone proportions, and feature fusion, the mechanism comprehensively determines whether they belong to the same person, greatly reducing the phenomenon of substitute inspections and fraud in morning hand inspections.

[0047] (1) Dynamic continuity verification. To prevent hand replacement attacks during morning hand inspection, the continuous video frames of the hand positioning process are analyzed in real time to verify the natural continuity of the flipping process.

[0048] (2) Verification of the integrity of the state sequence. The detection video frame sequence must show the palm state, transition state, and back of hand state in sequence. If any state is missing, it is judged as abnormal and "there may be a replacement risk".

[0049] (3) Time window verification. The time interval between the palm state and the final back of the hand state must be within 0.5 seconds to 2 seconds. If the time is too short, an automatic prompt will be displayed that "there may be a risk of pre-recorded video attack". If the time is too long, a prompt will be displayed that "there is a risk of mid-term replacement".

[0050] (4) Verification of spatial position continuity. Using a target tracking algorithm, when the target detection algorithm detects a hand, it continuously tracks the movement trajectory of the center point of the hand area and calculates the displacement of adjacent frames. If the displacement of any adjacent frame exceeds 50 pixels (about 1 / 10 of the screen width), it is judged as a position jump, and a message "there may be a risk of replacement" is displayed.

[0051] (5) Verification of skin feature continuity. Although the skin color of the hand is easily affected by lighting, the lighting conditions of adjacent frames are relatively stable. Calculate the cosine similarity of the color histograms of the hand regions in adjacent frames. If the minimum similarity is less than 0.5 or the average similarity is less than 0.8, it is judged as a feature cliff, indicating "there may be a risk of replacement".

[0052] During the hand inspection, if any of the following conditions are not met: state sequence integrity, time window, spatial location continuity, or skin feature continuity, a "potential replacement risk" message will be displayed. The risk result will be recorded in the current personnel's morning inspection information and copied to the relevant person in charge for secondary verification.

[0053] (6) Skeletal proportion consistency judgment Skeletal proportion features were extracted from the preprocessed palm and back of hand images, respectively. A deep learning-based keypoint detection algorithm was used for accurate localization and measurement.

[0054] 1) Key point detection The hand keypoint algorithm detects and outputs the pixel coordinates of 21 skeletal keypoints. and confidence level .

[0055] The 21 key points follow the standard hand skeletal architecture model, and their numbering and anatomical location definitions are shown in Table 3.

[0056] Table 3

[0057] 2) Post-processing of key points (a) Confidence screening: Set a threshold Remove Low-confidence points are marked as missing states; (b) Timing consistency verification: Kalman filtering is applied to smooth the detection results of multiple consecutive frames to eliminate single-frame jitter noise; (c) Cross-view alignment: A normalized coordinate system is established with key point 0 as the common origin and key point 9 as the Y-axis reference.

[0058] 3) Calculation of skeletal proportion features Based on the detected keypoint coordinates, the following skeletal proportion features were calculated, as shown in Table 4.

[0059] Table 4

[0060] 4) Consistency verification Calculate the relative differences in the proportional features of the corresponding bones on the palm side and the back side of the hand: ; Table 5

[0061] ; Table 6

[0062] Set preset threshold .like And all Then, the constraint is achieved through the consistency of bone proportions.

[0063] 5) Skeletal proportion similarity score ; If the maximum difference exceeds ,but The cutoff value is 0.

[0064] 6) Handling of missing key points, see Table 7.

[0065] Table 7

[0066] (7) Feature fusion discrimination Based on the dynamic continuity verification results and the skeleton proportion consistency judgment results, the final similarity score is calculated using an adaptive weighted fusion method, as shown in Table 8.

[0067] Table 8

[0068] in, .

[0069] The weighted fusion formula is: ( , ) (8) Consistency judgment rules If the hands are determined to belong to the same person, they are allowed to pass the morning inspection, but the following conditions must be met simultaneously: 1) Necessary conditions (hard constraints) (Must pass to prevent proxy testing attacks) (Minimum score for bone proportion) 2) Overall score The final threshold for determination 82.

[0070] Substitution detection judgment and response: If any of the above necessary conditions are not met, or If the identification is found to be fraudulent or the identity is inconsistent, the following response will be triggered: (a) Immediately trigger an audio-visual warning and announce "Detection abnormal, please re-detect"; (b) Save the current palm image, back of hand image, and key point information of the hand as evidence; (c) Record the reasons for failure (dynamic failure, skeletal failure, low overall score); (d) If a single person fails to perform the test three times in a row, the device will be locked and the relevant person in charge will be automatically notified for manual review.

[0071] 3) Adaptive threshold adjustment, see Table 9.

[0072] For first-time registered users, three pairs of palm-back images are collected, and the mean of the skeletal proportion features is calculated. and standard deviation Establish a personalized benchmark for each user. Real-time judgment: (a) Skeletal proportion difference: ; (b) If the user's historical detection variance This raises the overall threshold to 0.91; (c) If variance The overall threshold was reduced to 0.76.

[0073] Table 9

[0074] in, For the user's 10 most recent successful tests The standard deviation of the value.

[0075] Example 3 This embodiment also provides a deep learning system for identifying foreign objects in the hand and determining if the palm and back of the hand belong to the same person, used to execute the deep learning method for identifying foreign objects in the hand and determining if the palm and back of the hand belong to the same person in Embodiment 1, including: The identity verification module is used to verify the identity of morning check-up personnel through facial recognition and obtain the corresponding identity information; The mask retrieval module is used to retrieve the pre-registered hand mask of the corresponding person based on the identity information; the hand mask includes: a palm mask and a back of hand mask; The hand localization and key point detection module is used to determine whether a hand exists in the detection area. If so, it extracts the key points of the hand and determines whether the key points fall within the palm mask area. If not, it outputs a posture adjustment prompt until the requirements are met. If the key points of the hand fall within the palm mask area, it saves the palm image and the coordinates of the key points of the palm. The dynamic continuity verification module is used to acquire continuous video frames and images of the back of the hand during the flipping process, and to perform dynamic continuity verification on the continuous video frames during the flipping process; and to determine whether the key points of the back of the hand fall within the back of the hand mask. If so, the back of the hand image and the coordinates of the key points of the back of the hand are saved. The foreign object recognition module is used to perform image preprocessing and segmentation on the saved palm and back of hand images and corresponding key point coordinates, and to perform foreign object detection on the segmented images and fuse the results to obtain the hand foreign object recognition result. The same person identification module is used to calculate the skeletal proportion features of the palm and back of the hand based on the saved key point coordinates of the palm and back of the hand, calculate the relative difference of the corresponding skeletal proportion features of the palm and back of the hand, and obtain the skeletal proportion similarity score; and weightedly fuse the result score of dynamic continuity verification with the skeletal proportion similarity score, and determine whether the palm and back of the hand belong to the same person by combining the comprehensive score threshold, with passing dynamic continuity verification as a necessary condition.

[0076] This embodiment also provides an electronic device, including a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, a deep learning method for identifying foreign objects in the hand and determining whether the palm and back of the hand belong to the same person is implemented.

[0077] This embodiment also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements a deep learning method for identifying foreign objects in the hand and determining if the palm and back of the hand belong to the same person.

[0078] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A deep learning method for foreign object recognition in the hand and for determining if the palm and back of the hand belong to the same person, characterized in that, Includes the following steps: S1. Verify the identity of the morning check-up personnel through facial recognition and obtain the corresponding identity information; S2. Retrieve the pre-registered hand mask of the corresponding morning check-up personnel based on the identity information; the hand mask includes: a palm mask and a back of hand mask; S3. Determine if a hand exists in the detection area. If so, extract the key points of the hand and determine if the key points fall within the palm mask area. If not, output an adjustment prompt until the requirements are met. If the key points of the hand fall within the palm mask area, save the palm image and the coordinates of the key points. S4. Acquire continuous video frames and images of the back of the hand during the flipping process, and perform dynamic continuity verification on the continuous video frames during the flipping process; and determine whether the key points of the back of the hand fall within the back of the hand mask. If so, save the back of the hand image and the coordinates of the key points of the back of the hand. S5. Perform image preprocessing and segmentation on the saved palm and back of hand images and corresponding key point coordinates, perform foreign object detection on the segmented images, and fuse the foreign object detection results to obtain the hand foreign object recognition result. S6. Based on the saved key point coordinates of the palm and back of the hand, calculate the skeletal proportion features of the palm and back of the hand, calculate the relative difference of the corresponding skeletal proportion features of the palm and back of the hand, and obtain the skeletal proportion similarity score. S7. The results of dynamic continuity verification are weighted and fused with the skeletal proportion similarity score. With passing dynamic continuity verification as a necessary condition, and combined with the comprehensive score threshold, the results are determined to determine whether the palm and the back of the hand belong to the same person and output.

2. The deep learning method for foreign object recognition in the hand and for determining whether the palm and back of the hand belong to the same person, as described in claim 1, is characterized in that... In step S2, the process of creating the hand mask includes: Initial images of the palm and back of the hand were acquired separately, and pure hand region images were obtained through image binarization and hole filling. A morphological dilation algorithm is used to expand the pure hand region, and the outer contour coordinates of the expanded pure hand region image are extracted to generate corresponding palm and back of hand masks; wherein the dilation threshold of the morphological dilation algorithm can be customized.

3. The deep learning method for foreign object recognition in the hand and for determining whether the palm and back of the hand belong to the same person, as described in claim 1, is characterized in that... In S4, the dynamic continuity verification process includes: state sequence integrity verification, time window verification, spatial location continuity verification, and skin feature continuity verification. The state sequence integrity verification is used to detect whether the palm state, transition state and back of hand state appear sequentially in consecutive video frames. The time window verification is used to determine whether the time interval between flipping the palm state to the back of the hand state is within a preset time threshold. The spatial position continuity verification is used to track the movement trajectory of the center point of the hand region and determine whether the displacement of the hand region in adjacent frames exceeds a preset displacement threshold. The skin feature continuity verification is used to calculate the cosine similarity of the color histograms of the hand regions in adjacent frames, and to determine whether the minimum similarity is lower than the first preset similarity threshold or whether the average similarity is lower than the second preset similarity threshold.

4. The deep learning method for foreign object recognition in the hand and for determining whether the palm and back of the hand belong to the same person, as described in claim 1, is characterized in that... In step S5, the image preprocessing and segmentation process includes: The saved palm and back of hand images are binarized and hole-filled to remove background interference, resulting in a pure hand region image. The pure hand region image is segmented into 6 sub-images based on the coordinates of key points on the palm and back of the hand; the 6 sub-images include: thumb sub-image, index finger sub-image, middle finger sub-image, ring finger sub-image, little finger sub-image, and palm image; An open object detection algorithm is used to identify whether there are foreign objects in each sub-image, and the identification results of each sub-image are fused to obtain the foreign object identification result of the hand.

5. The deep learning method for foreign object recognition in the hand and the method for determining whether the palm and back of the hand belong to the same person, as described in claim 1, is characterized in that... In S6, the skeletal proportion features include: the length-to-width ratio of the palm, the relative length ratio of the index finger, the relative length ratio of the middle finger, the relative length ratio of the ring finger, the relative length ratio of the little finger, and the relative length ratio of the thumb; The calculation process of the bone proportion similarity score includes: calculating the relative difference between the corresponding bone proportion features of the palm side and the back side of the hand; if all relative differences do not exceed the preset difference threshold, the bone proportion similarity score is obtained through bone proportion consistency constraints.

6. The deep learning method for foreign object recognition in the hand and for determining whether the palm and back of the hand belong to the same person, as described in claim 1, is characterized in that... In S7, the formula for weighted fusion of the dynamic continuity verification result and the bone proportion similarity score is as follows: ; in, Scoring is based on the similarity of skeletal proportions. The score is for dynamic continuity verification, with 1 for passing and 0 for failing. Scoring based on skeletal proportions; weighting coefficients , .

7. The deep learning method for foreign object recognition in the hand and for determining whether the palm and back of the hand belong to the same person, as described in claim 1, is characterized in that... In S7, the comprehensive score threshold supports adaptive adjustment, specifically as follows: For first-time registered users, multiple sets of palm-back images are collected to calculate the mean and standard deviation of skeletal proportion features and establish a personalized benchmark. During real-time discrimination, the comprehensive score threshold is adjusted to a strict mode threshold, a standard mode threshold, or a lenient mode threshold based on the range of the standard deviation of the user's historical detected bone ratio differences.

8. A deep learning system for identifying foreign objects in the hand and determining if the palm and back of the hand belong to the same person, used to execute the deep learning method for identifying foreign objects in the hand and determining if the palm and back of the hand belong to the same person as described in any one of claims 1-7, characterized in that, include: The identity verification module is used to verify the identity of morning check-up personnel through facial recognition and obtain the corresponding identity information; The mask retrieval module is used to retrieve the pre-registered hand mask of the corresponding morning check-up personnel based on the identity information; the hand mask includes: a palm mask and a back of hand mask; The hand localization and key point detection module is used to determine whether a hand exists in the detection area. If so, it extracts the key points of the hand and determines whether the key points fall within the palm mask area. If not, it outputs an adjustment prompt until the requirements are met. If the key points of the hand fall within the palm mask area, it saves the palm image and the coordinates of the key points of the palm. The dynamic continuity verification module is used to acquire continuous video frames and images of the back of the hand during the flipping process, and to perform dynamic continuity verification on the continuous video frames during the flipping process; and to determine whether the key points of the back of the hand fall within the back of the hand mask. If so, the back of the hand image and the coordinates of the key points of the back of the hand are saved. The foreign object recognition module is used to perform image preprocessing and segmentation on the saved palm and back of hand images and corresponding key point coordinates, and to perform foreign object detection on the segmented images and fuse the results to obtain the hand foreign object recognition result. The same person identification module is used to calculate the skeletal proportion features of the palm and back of the hand based on the saved key point coordinates of the palm and back of the hand, calculate the relative difference of the corresponding skeletal proportion features of the palm and back of the hand, and obtain the skeletal proportion similarity score; and weightedly fuse the result score of dynamic continuity verification with the skeletal proportion similarity score, and determine whether the palm and back of the hand belong to the same person by combining the comprehensive score threshold, with passing dynamic continuity verification as a necessary condition.

9. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the deep learning method for foreign object recognition in the hand and the method for determining whether the palm and back of the hand belong to the same person, as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The medium stores a computer program, which, when executed by a processor, implements the deep learning method for identifying foreign objects in the hand and determining if the palm and back of the hand belong to the same person, as described in any one of claims 1-7.

Images