A method and apparatus for matching a portrait recognition feature
By detecting facial and human features in multi-person images, and using binary human body masks to calculate key point coverage and matrix matching algorithms, the problem of inaccurate facial and human feature matching in multi-person images is solved, achieving accurate feature association and stable beautification effects.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHENZHEN YANYU TECHNOLOGY CO LTD
- Filing Date
- 2026-02-10
- Publication Date
- 2026-05-29
AI Technical Summary
Existing multi-person image processing solutions cannot automatically and accurately match facial features with corresponding human body features, resulting in incorrect application of beauty effects or filters being applied to the wrong portraits.
By detecting facial and human features in the image, the key point coverage is calculated using a pre-processed binary human mask, and the human body to which each face belongs is determined based on a matrix matching algorithm, generating complete feature data.
It achieves automatic and accurate matching of facial and human features in multi-person images, avoiding filter crosstalk and effect misalignment, and improving the accuracy and user experience of beauty recognition applications.
Smart Images
Figure CN122116418A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and more specifically, to a method and apparatus for matching facial recognition features. Background Technology
[0002] With the rapid development of computer vision and artificial intelligence technologies, image recognition and processing have been widely applied in social media, camera applications, and beauty cameras. Current technologies typically detect and recognize faces and bodies separately in images. For example, face detection algorithms can obtain the location and key features of faces in an image, while body detection or segmentation algorithms can obtain the regions and key points of bodies in an image. These algorithms work well in scenarios involving single portraits, and many beauty applications can perform skin smoothing, whitening, and other processing on individual faces.
[0003] However, in images containing multiple people, existing processing solutions typically detect faces and bodies separately, but lack a direct mechanism to link the two. This can lead to confusion in the processing of multi-person images: face beautification effects may be applied to the wrong bodies, or body filters may be mistakenly applied to the wrong faces.
[0004] Therefore, there is a need to design a method and device for matching facial features and corresponding human body features in multi-person scenarios. Summary of the Invention
[0005] The purpose of this invention is to provide a method and apparatus for matching facial features to solve the problem that existing image processing schemes containing multiple images cannot automatically and accurately match facial features with corresponding human body features.
[0006] To achieve the above objectives, the present invention adopts the following technical solution: A method for matching facial recognition features, comprising: Acquire the image to be processed, and perform face and body detection and feature extraction on the image; The obtained facial features are matched with human body features to determine the human body to which each face belongs; wherein, the matching of the obtained facial features with human body features includes: Each human body mask is preprocessed and converted into a binary format; For each face, calculate the proportion of its key points that fall into each human body mask area; The proportions of all key points falling into each human body mask area are organized into a matrix R[j,i]; where matrix R[j,i] represents the proportion of key points falling into the j-th face relative to the i-th human body; Optimal matching calculation is performed based on matrix R[j,i]. Based on the results of the matching algorithm, the body to which each face belongs is determined; The matching results are integrated to generate complete feature data for each detected person.
[0007] In one embodiment, acquiring the image to be processed and performing face and body detection and feature extraction on the image includes: The pre-trained AI model is used to detect faces in images, and the location region of each detected face and the coordinates of multiple key points of that face are obtained. Human detection algorithms are used to detect the regions of each human body in the image; segmentation algorithms are used to obtain the human body mask corresponding to each portrait.
[0008] In one embodiment, the step of using a human detection algorithm to detect the region of each human body in the image and using a segmentation algorithm to obtain a human body mask corresponding to each portrait includes: Human body detection algorithms are used to obtain the bounding box or key point information of each human body. A segmentation probability map is generated by applying a semantic segmentation-based deep learning model to each portrait region; Finally, a binary human mask image is generated through thresholding.
[0009] In one embodiment, before calculating the proportion of key points falling into each human face mask area for each face, the method further includes: Perform a morphological dilation operation on the human body mask by a certain number of pixels.
[0010] In one embodiment, calculating the proportion of key points falling into each human body mask area includes: Let the set of key points for the j-th face be . The i-th human body mask is (For a two-dimensional binary image), define the function I(p,M) to indicate whether point p falls within the mask M; then the coverage ratio R[j,i] of the face on the mask can be expressed as: , Where n is the total number of facial landmarks; k = 1, 2, ..., n.
[0011] In one embodiment, the optimal matching calculation based on matrix R[j,i] includes: Transform the scaling matrix R[j,i] into the cost matrix C[j,i], where C[j,i] = 1 - R[j,i]. Expand C[j,i] to a size The matrix, with an extra column for "virtual matching" items, has a cost set to a constant c_null; The Hungarian algorithm is applied to the cost matrix C[j,i] to find the minimum cost match by constructing a feasible labeling system and iteratively augmenting the path; The optimal matching pair (j,i) is obtained, which means that the j-th face matches the i-th person. If a dummy item is matched, it means that no match is found.
[0012] In one embodiment, determining the human body to which each face belongs based on the results of the matching algorithm includes: For successfully matched facial and body features, the facial features are bound together with the corresponding body features as the image features of the same person. For cases where no match is found, the unmatched facial or human features are either retained or specially marked: if a human feature does not match any facial feature, it is considered that the facial feature of that human feature has not been detected in the image, and the human feature is still retained as an independent individual; if a facial feature does not match a human feature, it can be considered that only the facial feature was detected but the human feature was not detected, and it is also retained or specially marked.
[0013] In one embodiment, integrating the matching results to generate complete feature data for each detected person includes: For each matched face and body, the facial and body features of the image are combined to form a complete description. Output a list of human features, where each element corresponds to a human in the image.
[0014] In one embodiment, the list of human features includes a set of facial key points, a set of human body key points, a human body mask, and other derived features.
[0015] A facial recognition feature matching device, comprising: The feature detection and extraction module is used to acquire the image to be processed and to detect and extract features of faces and bodies from the image. The feature matching module is used to match the obtained facial features with human body features to determine the human body to which each face belongs. The process involves matching the obtained facial features with human body features, including: Each human body mask is preprocessed and converted into a binary format; For each face, calculate the proportion of its key points that fall into each human body mask area; The proportions of all key points falling into each human body mask area are organized into a matrix R[j,i]; where matrix R[j,i] represents the proportion of key points falling into the j-th face relative to the i-th human body; Optimal matching calculation is performed based on matrix R[j,i]. Based on the results of the matching algorithm, the body to which each face belongs is determined; The feature integration and output module is used to integrate the matching results and generate complete feature data for each detected person.
[0016] As can be seen from the above technical solution, the present invention has at least the following advantages and positive effects compared with the prior art: An embodiment of the present invention provides a method and apparatus for matching facial recognition features, comprising acquiring an image to be processed, detecting and extracting features of faces and bodies from the image; matching the obtained facial features with body features to determine the body to which each face belongs; and integrating the matching results to generate complete feature data for each detected person. The present invention utilizes the spatial relationship between facial key points and body regions for matching, automatically matching each facial feature to the corresponding body feature in the same image, achieving correct correspondence between facial features and body features for each person in the image. This allows subsequent beautification algorithms to apply effects to each individual portrait, eliminating filter crosstalk or effect misalignment caused by matching errors, and significantly improving the accuracy and user experience of multi-person image processing, especially beautification recognition applications. Attached Figure Description
[0017] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.
[0018] Figure 1 This is a flowchart illustrating a facial recognition feature matching method in one embodiment of the present invention; Figure 2 This is a schematic diagram of the process of obtaining the image to be processed in step S1 of the present invention, and detecting and extracting features of the face and human body in the image. Figure 3 This is a schematic diagram of the process of obtaining the human body mask corresponding to each portrait using a segmentation algorithm in step S120 of the present invention. Figure 4 This is a schematic diagram of the process of matching the obtained facial features with human body features in step S2 of the present invention. Figure 5 This is a flowchart illustrating the optimal matching calculation based on matrix R[j,i] in step S240 of the present invention. Figure 6 This is a schematic diagram of the process of determining the human body to which each face belongs based on the result of the matching algorithm in step S250 of the present invention. Figure 7 This is a flowchart illustrating the process of integrating the matching results to generate complete feature data for each detected person in step S3 of the present invention. Figure 8 This is a schematic diagram of the structure of the facial recognition feature matching device of the present invention.
[0019] The annotations in the attached figures are explained as follows: 210. Feature detection and extraction module; 220. Feature matching module; 230. Feature integration and output module. Detailed Implementation
[0020] To more clearly explain the purpose, technical solutions, and advantages of the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein. On the contrary, these embodiments are provided so that the present invention will be more comprehensive and complete, and fully convey the concept of the exemplary embodiments to those skilled in the art.
[0021] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a full understanding of embodiments of the invention. However, those skilled in the art will recognize that the technical solutions of the invention can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of the invention.
[0022] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.
[0023] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.
[0024] The present invention will now be described in detail with reference to specific embodiments: Example 1
[0025] This invention provides a method for matching facial recognition features, with reference to... Figure 1 , Figure 1 This is a flowchart illustrating the facial recognition feature matching method of the present invention.
[0026] Depend on Figure 1 It can be seen that a facial recognition feature matching method includes the following steps: S1. Obtain the image to be processed, and perform face and human body detection and feature extraction on the image.
[0027] Since faces and bodies belong to the same image, facial features and body features in the same image are detected and extracted separately to prepare for subsequent facial and body feature matching.
[0028] Depend on Figure 2 It can be seen that acquiring the image to be processed and performing face and body detection and feature extraction on the image includes the following steps: S110. Face Detection and Keypoint Extraction: A pre-trained AI model is used to detect faces in images, obtaining the location region of each detected face and the coordinates of multiple keypoints. These keypoints accurately depict the shape and features of the face.
[0029] It should be noted that the location area of a face includes, but is not limited to, the bounding box of the face.
[0030] It should be noted that key point coordinates include, but are not limited to, feature points such as eyes, nose tip, and corners of mouth.
[0031] S120. Human Detection and Segmentation: Using human detection algorithms, detect the region of each human body in the image; using segmentation algorithms, obtain the human body mask corresponding to each portrait.
[0032] Preferably, a human body mask corresponding to each person is obtained through a segmentation algorithm, and the pixel value of the mask indicates whether the pixel belongs to that person. At the same time, the skeletal key point information of each human body, as well as possible pose or category information, are extracted.
[0033] It should be noted that key skeletal information for the human body includes, but is not limited to, locations such as the shoulders, elbows, and knees.
[0034] It should be noted that posture or category information includes, but is not limited to, standing, sitting, and other postures.
[0035] Depend on Figure 3It can be seen that, using human detection algorithms, the region of each human body is detected in the image; using segmentation algorithms, the human body mask corresponding to each portrait is obtained, including the following steps: S121. Use human body detection algorithms to obtain the bounding box or key point information of each human body.
[0036] S122. Apply a semantic segmentation-based deep learning model to each portrait region to generate a segmentation probability map.
[0037] S123. Finally, a binary human body mask image is generated through thresholding.
[0038] After completing step S1 and acquiring the image to be processed, and performing face and body detection and feature extraction on the image, the method of the present invention will obtain: a set of key points for each face, and a binary mask and key points for each body, etc., to prepare for subsequent matching of face and body features.
[0039] S2. Match the obtained facial features with human body features to determine the human body to which each face belongs.
[0040] Based on the facial and body features obtained from the above steps, the facial and body features of people belonging to the same portrait are matched to obtain the facial and body features of that portrait.
[0041] Depend on Figure 4 It is known that matching the obtained facial features with human body features to determine the human body to which each face belongs includes the following steps: S210. Preprocessing human body masks: Preprocess each human body mask and convert it into a binary format.
[0042] S220. If necessary, a morphological dilation operation of a certain number of pixels can be performed on the human body mask to expand the coverage area, thereby tolerating detection errors and boundary misalignment. This step yields a clear human body region after processing.
[0043] It should be noted that in this embodiment, the morphological dilation operation is set to dilate by 3 pixels. It is understood that in other embodiments, the morphological dilation operation can also dilate by 2 or 4 pixels; this invention does not specifically limit the number of pixels dilated.
[0044] S230. For each face, calculate the proportion of its key points falling into each human body mask area.
[0045] Specifically, for a set of key points of a face, compare them one by one with the mask of each human body: count the percentage of key points of the face that are located within the mask of the human body.
[0046] It should be noted that only key points of the upper half of the face can be selected for calculation to avoid key points such as the chin being mistakenly placed in the area of other people due to their proximity. When most of the key points of a person's face are located within a certain human body mask, the face can be considered to belong to that human body.
[0047] Let the set of key points for the j-th face be . The i-th human body mask is (For a two-dimensional binary image), define the function I(p, M) to indicate whether point p falls within the mask M; then the coverage ratio R[j,i] of the face on the mask can be expressed as: , Where n is the total number of facial landmarks; k = 1, 2, ..., n.
[0048] S240. Organize the proportions of all key points falling into each human body mask area into a matrix R[j,i]; where matrix R[j,i] represents the proportion of key points falling into the j-th face relative to the i-th human body.
[0049] By organizing the proportions of key points falling into each human body mask area into a matrix R[j,i], it is possible to intuitively compare the matching degree between each face and each human body.
[0050] S250. Optimal matching calculation is performed based on matrix R[j,i].
[0051] Depend on Figure 5 It can be seen that the optimal matching calculation based on matrix R[j,i] includes the following steps: S251. Convert the scaling matrix R[j,i] into the cost matrix C[j,i], where C[j,i]=1-R[j,i].
[0052] S252, Expand C[j,i] into a size The matrix has an extra column for "virtual matching" items, with a cost set to the constant c_null.
[0053] S253. Perform the Hungarian algorithm on the cost matrix C[j,i] to find the minimum cost match by constructing a feasible labeling system and iteratively augmenting the path.
[0054] S254. Obtain the optimal matching pair (j,i), which means that the j-th face matches the i-th person. If a virtual item is matched, it means that no match is found.
[0055] The algorithm converts proportions into costs for matching, and a threshold can be set. For example, a match is considered valid only if the keypoint coverage of a facial feature on a body feature is not less than 25%; otherwise, it is considered a no-match. After solving, several pairs will be obtained ( , ), indicating that the j-th face is matched with the i-th person.
[0056] S260. Based on the results of the matching algorithm, determine the human body to which each face belongs.
[0057] Depend on Figure 6 As can be seen, determining the human body to which each face belongs based on the matching algorithm involves the following steps: S261. For successfully matched facial features and body features, bind the facial features together with the corresponding body features as the image features of the same person.
[0058] S262. In the case of failure to match, the unmatched facial features or human features shall be retained or specially marked: if a human feature does not match any facial feature, it shall be considered that the facial feature of that human feature has not been detected in the image, and the human feature shall still be retained as an independent individual; if a facial feature does not match a human feature, it may be regarded as only the facial feature was detected but the human feature was not detected, and it shall also be retained or specially marked.
[0059] S3. Integrate the matching results to generate complete feature data for each detected person.
[0060] Depend on Figure 7 As can be seen, integrating the matching results to generate complete feature data for each detected person involves the following steps: S310. For each matched face and body, combine the facial features and body features of the image to form a complete description.
[0061] By grouping facial landmark data and body landmark data under the same person information and attaching a body mask region for that person, it is possible to further calculate or extract other feature information of that person. For example, the matching information for person A includes: facial landmarks, extended facial feature vectors, body landmark locations and visibility, corresponding body mask images, pose categories, skin region images, etc.
[0062] S320. Output a list of character features, where each element corresponds to a character in the image.
[0063] It should be noted that the list of human features includes a set of facial key points, a set of human body key points, a human body mask, and other derived features.
[0064] It should be noted that other derived features include, but are not limited to, each person's skin region image, hair region image, body pose classification results, etc.
[0065] For portraits lacking facial features, the facial feature field can be empty; for portraits with only a face but no matching body features, the body feature field should be empty. This list of human features will serve as the basis for subsequent beautification processing or analysis decisions.
[0066] In summary, the facial recognition feature matching method described in this application can significantly improve the accuracy and user experience of multi-person image processing, especially beauty recognition applications.
[0067] First, this invention utilizes the spatial relationship between facial key points and human body regions for matching, automatically matching each facial feature to the corresponding human body feature in the same image. This achieves a correct correspondence between the facial and human body features of each person in the image, allowing subsequent beautification algorithms to apply effects to each individual portrait. This eliminates filter crosstalk or effect misalignment caused by matching errors, significantly improving the accuracy and user experience of multi-person image processing, especially in beautification recognition applications.
[0068] Secondly, the method of this invention is robust to complex scenes. When multiple people are partially occluded or close together, they can be distinguished by key point ratio matching, greatly reducing the probability of false matching. Compared with traditional simple methods, both accuracy and reliability are improved, ensuring that subsequent portrait analysis or enhancement functions can be based on accurate feature association.
[0069] Furthermore, the solution provided by this invention is computationally efficient and suitable for real-time applications. The matching process is mainly based on key point geometric calculations and matrix optimization, requiring relatively little computation. It can be completed quickly in environments with limited hardware conditions, such as mobile devices. Even in scenarios such as real-time previews with mobile phone beauty cameras and multi-person video conferencing filters, this invention can still run smoothly, providing users with a stable user experience. Example 2
[0070] refer to Figure 8 This invention provides a facial recognition feature matching device. Figure 8 This is a schematic diagram of the structure of the facial recognition feature matching device of the present invention.
[0071] Depend on Figure 8 It is known that a facial recognition feature matching device includes: a feature detection and extraction module 210, a feature matching module 220, and a feature integration and output module 230.
[0072] The feature detection and extraction module 210 is used to acquire the image to be processed and to detect and extract features of faces and bodies from the image.
[0073] The feature matching module 220 is used to match the obtained facial features with human body features to determine the human body to which each face belongs.
[0074] The feature integration and output module 230 is used to integrate the matching results and generate complete feature data for each detected person.
[0075] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the claims. It should be understood that the invention is not limited to the precise structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.
Claims
1. A method for matching facial recognition features, characterized in that, include: Acquire the image to be processed, and perform face and body detection and feature extraction on the image; The obtained facial features are matched with human body features to determine the human body to which each face belongs; wherein, the matching of the obtained facial features with human body features includes: Each human body mask is preprocessed and converted into a binary format; For each face, calculate the proportion of its key points that fall into each human body mask area; The proportions of all key points falling into each human body mask area are organized into a matrix R[j,i]; where matrix R[j,i] represents the proportion of key points falling into the j-th face relative to the i-th human body; Optimal matching calculation is performed based on matrix R[j,i]. Based on the results of the matching algorithm, the body to which each face belongs is determined; The matching results are integrated to generate complete feature data for each detected person.
2. The facial recognition feature matching method according to claim 1, characterized in that, The process of acquiring the image to be processed and performing face and body detection and feature extraction on the image includes: The pre-trained AI model is used to detect faces in images, and the location region of each detected face and the coordinates of multiple key points of that face are obtained. Human detection algorithms are used to detect the regions of each human body in the image; segmentation algorithms are used to obtain the human body mask corresponding to each portrait.
3. The facial recognition feature matching method according to claim 2, characterized in that, The algorithm uses human detection to detect the region of each human body in the image; The human body mask for each portrait is obtained using a segmentation algorithm, including: Human body detection algorithms are used to obtain the bounding box or key point information of each human body. A segmentation probability map is generated by applying a semantic segmentation-based deep learning model to each portrait region; Finally, a binary human mask image is generated through thresholding.
4. The facial recognition feature matching method according to claim 1, characterized in that, Before calculating the proportion of key points falling into each human body mask area for each face, the process also includes: Perform a morphological dilation operation on the human body mask by a certain number of pixels.
5. The facial recognition feature matching method according to claim 1, characterized in that, Calculate the proportion of key points falling into each human body mask area, including: Let the set of key points for the j-th face be . The i-th human body mask is (For a two-dimensional binary image), define the function I(p,M) to indicate whether point p falls within the mask M; then the coverage ratio R[j,i] of the face on the mask can be expressed as: ， Where n is the total number of facial landmarks; k = 1, 2, ..., n.
6. The facial recognition feature matching method according to claim 1, characterized in that, The optimal matching calculation based on matrix R[j,i] includes: Transform the scaling matrix R[j,i] into the cost matrix C[j,i], where C[j,i] = 1 - R[j,i]. Expand C[j,i] to a size The matrix, with an extra column for "virtual matching" items, has a cost set to a constant c_null; The Hungarian algorithm is applied to the cost matrix C[j,i] to find the minimum cost match by constructing a feasible labeling system and iteratively augmenting the path; The optimal matching pair (j,i) is obtained, which means that the j-th face matches the i-th person. If a dummy item is matched, it means that no match is found.
7. The facial recognition feature matching method according to claim 1, characterized in that, The step of determining the human body to which each face belongs based on the results of the matching algorithm includes: For successfully matched facial and body features, the facial features are bound together with the corresponding body features as the image features of the same person. For cases where no match is found, the unmatched facial or human features are either retained or specially marked: if a human feature does not match any facial feature, it is considered that the facial feature of that human feature has not been detected in the image, and the human feature is still retained as an independent individual; if a facial feature does not match a human feature, it can be considered that only the facial feature was detected but the human feature was not detected, and it is also retained or specially marked.
8. The facial recognition feature matching method according to claim 1, characterized in that, The process of integrating the matching results to generate complete feature data for each detected person includes: For each matched face and body, the facial and body features of the image are combined to form a complete description. Output a list of human features, where each element corresponds to a human in the image.
9. The facial recognition feature matching method according to claim 8, characterized in that, The list of human features includes a set of facial key points, a set of human body key points, a human body mask, and other derived features.
10. A facial recognition feature matching device, characterized in that, include: The feature detection and extraction module is used to acquire the image to be processed and to detect and extract features of faces and bodies from the image. The feature matching module is used to match the obtained facial features with human body features to determine the human body to which each face belongs. The process involves matching the obtained facial features with human body features, including: Each human body mask is preprocessed and converted into a binary format; For each face, calculate the proportion of its key points that fall into each human body mask area; The proportions of all key points falling into each human body mask area are organized into a matrix R[j,i]; where matrix R[j,i] represents the proportion of key points falling into the j-th face relative to the i-th human body; Optimal matching calculation is performed based on matrix R[j,i]. Based on the results of the matching algorithm, the body to which each face belongs is determined; The feature integration and output module is used to integrate the matching results and generate complete feature data for each detected person.