A face recognition method and system based on a face multi-label algorithm

Abstract

The application discloses a face recognition method and system based on a face multi-label algorithm, belongs to the technical field of face recognition, and is used for solving the technical problem of low recognition accuracy. The method comprises the following steps: firstly, key points are obtained by scanning a current face image based on a sliding window, and a global coverage path of face features is constructed; secondly, a face region is globally scanned according to the path, and an image global label containing action and static attribute labels is obtained by matching the face region with preset label images; thirdly, action label corresponding key points are determined, and a local ROI region is determined according to the neighborhood of the action label; then, the local ROI region of multiple frames of images is subjected to multi-layer feature resampling, and the action label is corrected according to the local action label of the sampling result; finally, the corrected action label and the static attribute label are matched with a preset label result knowledge graph, and face recognition of the current face image is completed.

Description

Technical Field

[0001] This invention relates to the field of facial recognition technology, and in particular to a facial recognition method and system based on a multi-label facial recognition algorithm. Background Technology

[0002] With increasing urban security demands, security monitoring systems are being widely deployed. In complex scenarios, it's not enough to simply identify faces; information such as facial expressions, actions, and attributes is also needed to quickly determine if any security risks exist. In practical applications, the facial image data collected is highly complex and diverse. Changes in facial expressions and the range of motion can all interfere with facial recognition and motion analysis.

[0003] Currently, to improve the accuracy of facial recognition, a sliding window scanning method is typically used to initially locate the facial region, followed by key point detection to extract facial features. However, this method requires the sliding window to traverse a large number of invalid regions, which is unsuitable for scenarios such as vehicle fatigue warning where high recognition efficiency is required. Furthermore, existing systems often perform similarity matching through global feature embedding and pre-built label libraries, or directly output attribute labels through multi-task networks. However, these methods are generally based on static labels for recognition, which cannot capture instantaneous actions, resulting in insufficient accuracy in emotion recognition. Moreover, if action labels are added, action labels and static attribute labels are processed independently, lacking cross-modal correlation. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a face recognition method and system based on a multi-label face algorithm.

[0005] The present invention adopts the following technical solution:

[0006] On one hand, the present invention provides a face recognition method based on a multi-label face algorithm, the method comprising:

[0007] A preliminary scan is performed based on a sliding window corresponding to the current face image to obtain facial key points, and a global facial feature coverage path is constructed based on the facial key points.

[0008] A global scan of the current face region is performed based on the global coverage path of the face features, and the scan results are matched with preset label images to obtain the global image label corresponding to the current face image; wherein, the global image label includes: action label and static attribute label;

[0009] Determine the specified facial key points corresponding to the action labels, and determine the local ROI region of each specified facial key point in the current face image based on the neighborhood corresponding to each specified facial key point.

[0010] Based on the multiple frames corresponding to the current face image, multi-layer feature resampling is performed on the local ROI region to correct the action label according to the local action label corresponding to the sampling result;

[0011] The corrected action labels are matched with the static attribute labels and the preset label result knowledge graph to achieve face recognition of the current face image.

[0012] Optionally, in one or more embodiments of this specification, a preliminary scan is performed based on a sliding window corresponding to the current face image to obtain facial key points, and a global facial feature coverage path is constructed based on the facial key points, specifically including:

[0013] Based on the image data of the current face image and the recognition scenario, the size range of the sliding window is determined so as to perform a preliminary scan of the current face image according to the sliding window of each size range and obtain the face key points;

[0014] Based on the physiological structural and functional features corresponding to each of the facial key points, each of the facial key points is classified to obtain multiple sets of facial key point sets.

[0015] Based on the matching relationship between each set of facial key points and the preset physiological cognitive order, the set path order corresponding to each set of facial key points is determined;

[0016] Based on the spatial positional relationship of each facial key point within each set of facial key points, the local paths between each facial key point within each set of facial key points are determined.

[0017] Determine whether the set of facial key points corresponding to the local path corresponds to a high dynamic structure. If so, expand the branches of the local path based on the preset values ​​corresponding to the high dynamic structure to obtain the expanded local path.

[0018] The initial global coverage path for facial features is determined based on the set path order and the local and expanded local paths between each facial key point in the set of facial key points.

[0019] The initial global coverage path of facial features is smoothed to obtain a global coverage path of facial features.

[0020] Optionally, in one or more embodiments of this specification, the step of determining the size range of the sliding window based on the image data of the current face image and the recognition scene, so as to perform a preliminary scan of the current face image according to the sliding window of each size range to obtain facial key points, specifically includes:

[0021] The image attributes of the current face image are obtained, and the image attribute label corresponding to the current face image is determined based on the image data. The historical face images corresponding to the recognition scene of the current face image and the image attribute label are obtained. The image attributes include: illumination intensity, average face proportion, and background complexity.

[0022] Based on the sliding window size corresponding to historical face images, determine the initial size range of the sliding window corresponding to the current face image, and expand the initial size range based on the image resolution corresponding to the current face image to obtain the size range of the sliding window;

[0023] Based on the image size and resolution of the current face image, the corresponding number of image layers is determined, and the corresponding scaling factor is determined based on the preset recognition requirement information, so as to determine the number of sampleable layers of the current face image according to the number of image layers and the scaling factor;

[0024] Based on the scaling factor and the size range of the sliding window, determine the current size range of the sliding window for each sampleable layer of the current face image;

[0025] Sliding detection is performed on the current face image at each sampleable layer using a sliding window within the current size range to obtain facial key points.

[0026] Optionally, in one or more embodiments of this specification, a global scan of the current face region is performed according to the global coverage path of the face features, so as to match the scan results with a preset label image to obtain the global image label corresponding to the current face image, specifically including:

[0027] A global scan of the current face region is performed based on the global coverage path of the facial features to obtain basic data of each key point in the current face region and key point displacement data between adjacent frames based on the scan results; wherein, the basic data includes: pixel values ​​and position data;

[0028] Based on the basic data of each key point in the current face region, the static attribute features of the current face region are determined, and based on the key point displacement data between adjacent frames, the motion features of the current face region are determined.

[0029] The static attribute label corresponding to the static attribute feature is determined by performing similarity matching between the static attribute feature and the facial geometric features corresponding to each label in the preset static label image.

[0030] The action label corresponding to the action feature is determined by performing similarity matching between the action feature and the action trajectory features corresponding to each label in the preset action label image.

[0031] Optionally, in one or more embodiments of this specification, determining the specified facial key points corresponding to the action tags, and determining the local ROI region of each specified facial key point in the current face image based on the neighborhood corresponding to each specified facial key point, specifically includes:

[0032] Based on the action features corresponding to the action tags, the facial key points corresponding to the action tags are determined as designated facial key points;

[0033] Using each specified facial key point as the center, obtain the neighborhood corresponding to each specified facial key point;

[0034] Determine whether the neighborhoods corresponding to each of the specified facial key points overlap;

[0035] If there is overlap and the overlap ratio is greater than a preset ratio, the neighborhood is merged to obtain the processed neighborhood as the local ROI region of the current face image.

[0036] Optionally, in one or more embodiments of this specification, multi-layer feature resampling is performed on the local ROI region based on multiple frames corresponding to the current face image, specifically including:

[0037] Based on the positions of key points in the multi-frame images corresponding to the current face image and the positions of key points in the local ROI region, align the spatial positions of the multi-frame images and the local ROI region in the time series;

[0038] Determine the dynamic local ROI region corresponding to the key points of the local ROI region in the aligned multi-frame images;

[0039] The dynamic local ROI region is input into a preset CNN model to obtain multi-layer features output by different network layers of the preset CNN model;

[0040] The spatial resolution corresponding to the multi-layer features is unified, and the sampling result is obtained by fusing the unified multi-layer features.

[0041] Optionally, in one or more embodiments of this specification, the action label is modified based on the local action label corresponding to the sampling result, specifically including:

[0042] The sampling results are input into the classifier of the preset CNN model to obtain the local action labels corresponding to the sampling results;

[0043] The local action labels of the multi-frame images are compared with the action labels to determine whether there is a conflict between the local action labels and the action labels;

[0044] If so, the action tag is overwritten and modified based on the local action tag to obtain the corrected action tag.

[0045] Optionally, in one or more embodiments of this specification, before matching the modified action label with the static attribute label and the preset label result knowledge graph to achieve face recognition of the current face image, the method further includes:

[0046] Based on existing face databases and behavior databases, structured labeled data is obtained, and unstructured data is obtained by extracting action descriptions and action attribute association rules from multiple public texts.

[0047] Entity extraction is performed on the unstructured data to obtain key entities and attribute information; wherein, the key entities include: action subject and action object;

[0048] Based on the relationships between the key entities, an initial label result knowledge graph is constructed, and the attribute information is added to the initial label result knowledge graph;

[0049] Align the entities corresponding to the structured annotation data with the initial label result knowledge graph to expand the initial label result knowledge graph based on the structured annotation data, thereby obtaining a preset label result knowledge graph.

[0050] Optionally, in one or more embodiments of this specification, the modified action label is matched with the static attribute label and the preset label result knowledge graph to achieve face recognition of the current face image, specifically including:

[0051] The encoding format of the pre-set tag result knowledge graph is determined, and the modified action tags and static attribute tags are standardized based on the encoding format to obtain processed action tags and processed static attribute tags.

[0052] In the pre-set tag result knowledge graph, search for entities and relationships corresponding to the processed action tags and processed static attribute tags to obtain multiple matching results;

[0053] Obtain the similarity corresponding to each of the matching results, sort the matching results based on the similarity, and take the matching result with the highest score as the face recognition result of the current face image.

[0054] On the other hand, the present invention also provides a face recognition system based on a multi-label face algorithm, the system comprising:

[0055] The construction unit is used to perform a preliminary scan based on a sliding window corresponding to the current face image, obtain facial key points, and construct a global coverage path for facial features based on the facial key points;

[0056] The acquisition unit is configured to perform a global scan of the current face region based on the global coverage path of the face features, and to match the scan results with a preset label image to obtain the global image label corresponding to the current face image; wherein, the global image label includes: action label and static attribute label;

[0057] The determining unit is used to determine the specified facial key points corresponding to the action label, so as to determine the local ROI region of each specified facial key point in the current face image according to the grid region corresponding to each specified facial key point.

[0058] The correction unit is used to perform multi-layer feature resampling on the local ROI region based on the multi-frame images corresponding to the current face image, so as to correct the action label according to the local action label corresponding to the sampling result;

[0059] The recognition unit is used to match the corrected action label with the static attribute label and the preset label result knowledge graph to achieve face recognition of the current face image.

[0060] The above-described at least one technical solution used in the embodiments of this specification can achieve the following beneficial effects:

[0061] By initially scanning facial key points using a sliding window and constructing a global coverage path, comprehensive facial features, both overall and detailed, can be captured, avoiding the omission of important information and improving the accuracy of feature extraction. Global scanning based on the global coverage path and matching it with pre-labeled images yields global image labels containing action and static attribute labels, accurately describing the features and attributes of the facial image. Determining the specific facial key points corresponding to the action labels and their surrounding local ROI regions allows focusing on key areas, reducing interference from irrelevant information, and improving the targeting and efficiency of recognition. Multi-layer feature resampling of the local ROI regions and subsequent correction of action labels further refines and optimizes the description of facial actions, improving the accuracy of action recognition. Matching the corrected action and static attribute labels with a knowledge graph of pre-labeled results integrates multiple information sources to achieve facial recognition, improving accuracy and reliability while enhancing adaptability to complex scenarios. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0063] Figure 1 A flowchart of a face recognition method based on a multi-label face algorithm provided by the present invention;

[0064] Figure 2 This is a schematic diagram of the structure of a face recognition system based on a multi-label face algorithm provided by the present invention. Detailed Implementation

[0065] This specification provides a face recognition method based on a multi-label face algorithm.

[0066] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0067] The method of the present invention will now be described in detail with reference to the accompanying drawings.

[0068] Figure 1 This invention provides a schematic flowchart of a face recognition method based on a multi-label face algorithm, as shown below. Figure 1 As shown, the method of the present invention includes at least the following execution steps:

[0069] S101: Perform a preliminary scan based on the sliding window corresponding to the current face image to obtain facial key points, and construct a global facial feature coverage path based on the facial key points.

[0070] In real-world scenarios, such as fatigue driving detection, the driving environment is complex and ever-changing. The dashboard, seats, and other elements inside the vehicle create complex backgrounds, often interfering with the facial images captured by the camera. Traditional methods, such as using a sliding window to initially locate the face region and then extracting facial features through key point detection, are inefficient, especially in complex backgrounds or multi-person scenarios, because the sliding window needs to traverse a large number of invalid areas. Furthermore, it struggles to accurately locate faces in such complex environments, leading to feature extraction errors. Therefore, to address this issue, the embodiments in this specification perform a preliminary scan based on a sliding window corresponding to the current facial image to obtain facial key points, such as those for the eyes, nose, and mouth. Constructing a global facial feature coverage path based on these key points connects the scattered key points, forming a complete path that comprehensively describes the facial features. This path not only includes the location information of each key point, but also reflects the spatial relationship and relative position between them. In the process of facial recognition in the scenario of fatigue driving, it can help to fully capture the overall features and details of the face. For example, it can judge whether there are signs of fatigue such as drooping eyelids and frequent blinking by the position changes of key points of the eyes, and identify whether there is a yawning action based on the state of key points of the mouth, which greatly improves the accuracy and stability of fatigue driving recognition.

[0071] Specifically, in one or more embodiments of this specification, a preliminary scan is performed based on a sliding window corresponding to the current face image to obtain facial key points, and a global facial feature coverage path is constructed based on the facial key points. The process includes the following steps:

[0072] First, based on the image data of the current face image and the recognition scenario, the size range of the sliding window is determined. This allows for a preliminary scan of the current face image using sliding windows within each size range to obtain facial key points. Specifically, the size range of the sliding window is determined based on the image data and the recognition scenario. For example, in images with strong lighting, simple backgrounds, and a large proportion of the face, a larger sliding window can be selected to quickly scan the face area; while in images with dim lighting, complex backgrounds, and a small proportion of the face, a smaller sliding window is needed for a more detailed search of the face, ensuring no key information is missed. After determining the size range, a preliminary scan of the current face image can be performed using sliding windows within each size range to obtain facial key points. Based on the physiological structural features corresponding to these key points, such as their location in different parts of the face (eyes, nose, mouth, etc.) and functional features (e.g., key points related to the eyes are used to express gaze, and key points related to the mouth are used for speech expressions), the acquired facial key points are classified into multiple sets. For example, all key points related to the eyes are grouped into one group, and key points related to the mouth into another.

[0073] Then, based on the matching relationship between each set of facial key points and the preset physiological cognitive order, the sequence of the set paths corresponding to the set of facial key points is determined. It should be noted that the preset physiological cognitive order refers to the scanning priority rules predefined based on the anatomical structure of the face, functional importance, and human visual cognitive habits. Next, for each set of facial key points, local paths are determined based on the spatial relationships between the key points within the set. For example, in the set of key points related to the eyes, the connection order and method between key points such as the corners of the eyes and the eyeballs are determined; such local paths reflect the relative positional relationships of the key points in this set. Then, it is determined whether the set of facial key points corresponding to the local path corresponds to a high-dynamic structure. If so, the branches of the local path are expanded based on the preset values ​​corresponding to the high-dynamic structure to obtain the expanded local path. It can be understood that high-dynamic structures correspond to parts with large amplitude or frequent movements, such as the mouth whose shape changes greatly under different expressions like speaking and smiling. Expanding the branches of the local path can more accurately describe its dynamic changes, allowing the path to more accurately reflect the feature changes of these parts. Then, based on the sequence of the set paths and the local paths and expanded local paths between each facial keypoint in the set, an initial global facial feature coverage path is determined. This initial global facial feature coverage path is then smoothed to obtain the final global facial feature coverage path.

[0074] The process described above determines the sliding window size range based on image data and the recognition scene, effectively handling various complex image environments. Regardless of changes in lighting, background, or face proportions, the method can accurately capture facial key points by adjusting the window size, ensuring its versatility and stability. Furthermore, by classifying facial key points and constructing paths based on physiological cognitive order and spatial relationships, while also considering the extension of highly dynamic structures, the method comprehensively and meticulously describes facial features. It not only covers static facial structural features but also fully considers dynamic changes, enabling the constructed paths to more realistically reflect the entirety of the face.

[0075] Furthermore, in one or more embodiments of this specification, based on the image data of the current face image and the recognition scene, the size range of the sliding window is determined, so as to perform a preliminary scan of the current face image according to the sliding window of each size range to obtain facial key points, specifically including:

[0076] First, the image attributes of the current face image are acquired to determine the corresponding image attribute labels. Simultaneously, historical face images corresponding to the current face image's recognition scene and image attribute labels are acquired. It should be noted that image attributes include: illumination intensity, average face proportion, and background complexity. Then, based on the sliding window sizes corresponding to historical face images, the initial size range of the sliding window corresponding to the current face image is determined. This initial size range is then expanded based on the image resolution of the current face image to obtain the sliding window's size range. In other words, since different image resolutions affect the actual number of pixels and the size of the area covered by the window, the initial size range needs to be expanded in conjunction with the current face image's resolution to obtain a sliding window size range more suitable for the current image. For example, high-resolution images may require larger windows to cover sufficient feature areas, while low-resolution images require relatively smaller windows to avoid oversampling. Next, based on the current face image's image size and resolution, the corresponding number of image layers is determined, and a scaling factor is determined based on preset recognition requirements. Thus, the number of sampleable layers for the current face image is determined based on the number of image layers and the scaling factor. Based on the scaling factor and the size range of the sliding window, the current size range of the sliding window for each sampleable layer of the current face image is determined. Since the image resolution differs at different sampling layers, the size of the sliding window needs to be adjusted accordingly to ensure that the image can be effectively scanned and key information acquired at each sampling layer. Using the sliding window with the determined size range, sliding detection is performed on the current face image at each sampleable layer to obtain facial landmarks.

[0077] The process described above, by combining image attributes, the recognition scene, and historical images to determine the range of sliding window sizes, fully considers the characteristics of the current image and past experience, making the sliding window size more suitable for the current face image. Using a sliding window of appropriate size for detection in sampleable layers of different resolutions can take into account both the overall features and local details of the image, acquiring facial key points more comprehensively and accurately, thereby improving the accuracy of facial key point detection and providing a more reliable data foundation for subsequent tasks such as face recognition. Furthermore, determining the scaling factor and the number of sampleable layers based on pre-defined recognition requirements allows for the rational allocation of computational resources while meeting recognition accuracy requirements.

[0078] S102: Perform a global scan of the current face region according to the global coverage path of the face features, and match the scan results with a preset label image to obtain the global image label corresponding to the current face image; wherein, the global image label includes: action label and static attribute label.

[0079] Traditional face recognition technologies rely on global feature embedding and similarity matching with pre-set label libraries, or directly output attribute labels through multi-task networks. However, these methods typically rely on static labels for recognition, which cannot capture instantaneous actions, leading to insufficient accuracy in emotion recognition. Therefore, this embodiment aims to achieve a more comprehensive description of faces and overcome the shortcomings of traditional technologies in information acquisition. This embodiment performs a global scan of the current face region based on a global coverage path of face features. The scan results are then matched with pre-set label images to obtain the corresponding global image label. It should be noted that the global image label includes action labels and static attribute labels. This process, through global scanning and matching with pre-set label images, simultaneously acquires both action and static attribute labels, achieving a more comprehensive description of the face and overcoming the shortcomings of traditional technologies in information acquisition. Utilizing a global coverage path for global scanning allows for more comprehensive capture of facial features in various states. Combined with pre-set label image matching, this effectively addresses face recognition in complex scenarios, improving the adaptability and accuracy of the face recognition system in complex environments.

[0080] Specifically, in one or more embodiments of this specification, a global scan of the current face region is performed based on the global coverage path of facial features, and the scan results are matched with a preset label image to obtain the global image label corresponding to the current face image. The process includes the following steps:

[0081] The system utilizes a pre-constructed global facial feature coverage path to perform a comprehensive scan of the current facial region. Based on the scan results, it obtains basic data of each key point in the current facial region, as well as key point displacement data between adjacent frames. It's important to note that the basic data includes pixel values ​​and positional data. In other words, the scanning process acquires two crucial types of data for each key point in the current facial region. One type is the basic data, including pixel values ​​and positional data. Pixel values ​​reflect the image color information of the key point's location, while positional data precisely records the key point's coordinates in the image. This basic data describes the features of the facial key points from a static perspective. The other type is the key point displacement data between adjacent frames, which records the positional changes of the facial key points in the current frame relative to the previous frame in the video stream, reflecting the dynamic features of the face. After acquiring the basic data of each key point in the current face region and the key point displacement data between adjacent frames, the static attribute features of the current face region, such as gender, approximate age range, and skin color, are determined based on the basic data of each key point. The motion features of the current face region, such as smiling, nodding, and turning the head, are determined based on the key point displacement data between adjacent frames. The extracted static attribute features are then matched with the facial geometric features corresponding to each label in a pre-set static label image to find the pre-set label with the highest similarity, thus determining the static attribute label corresponding to the current face's static attribute features. Finally, the determined motion features are matched with the motion features corresponding to each label in a pre-set motion label image to find the best-matching pre-set motion label, thereby determining the motion label corresponding to the current face's motion features.

[0082] The above steps not only focus on the static features of the face but also acquire dynamic action features through the displacement data of key points in adjacent frames. This comprehensively covers both the static attributes and action information of the face, providing rich data support for subsequent face recognition and analysis. Compared to traditional recognition methods based solely on static facial features, this approach can more comprehensively describe the facial state. By combining static attribute features and action features for matching, facial information can be confirmed from multiple perspectives. Even if some static features are missing due to occlusion or other reasons, action features can still assist in recognition, thereby improving the accuracy and stability of face recognition in complex scenes. Matching static attribute features and action features separately allows for more refined and accurate feature-based matching. Different features have their corresponding matching strategies and label libraries, enabling optimization for different types of information, avoiding interference between different types of features, and improving the reliability and credibility of the matching.

[0083] S103: Determine the specified facial key points corresponding to the action labels, so as to determine the local ROI region of each specified facial key point in the current face image based on the neighborhood corresponding to each specified facial key point.

[0084] Traditional face recognition and action analysis often treat the entire face as the analysis object, ignoring the characteristics of specific key regions corresponding to different actions, resulting in low analysis accuracy. Therefore, to improve the accuracy of recognition, the embodiments of this specification determine the specified facial key points corresponding to the action labels, and then determine the local ROI region of each specified facial key point in the current face image based on the neighborhood corresponding to each specified facial key point.

[0085] Specifically, in one or more embodiments of this specification, determining the specified facial key points corresponding to the action tags, and determining the local ROI region of each specified facial key point in the current face image based on the neighborhood corresponding to each specified facial key point, specifically includes:

[0086] Different action labels correspond to specific facial action features. Therefore, based on the action features corresponding to the action labels, facial keypoints corresponding to the action labels are determined as designated facial keypoints. Then, with each designated facial keypoint as the center, its corresponding neighborhood is obtained. This neighborhood contains the surrounding pixel information of the keypoint, and its size and shape can be set according to actual needs. For example, it can be set as a circular area with a certain radius centered on the keypoint, or a square area with fixed side lengths. The purpose of obtaining the neighborhood is to collect surrounding information closely related to the keypoint. This information helps to more comprehensively understand the changes in the keypoint and the performance of actions in local areas. After determining the neighborhood corresponding to each designated facial keypoint, it is checked whether there are overlapping parts between these neighborhoods, and the proportion of the overlapping area is calculated. When overlapping neighborhoods are found and the overlap ratio is greater than a preset ratio, these neighborhoods are merged to obtain the processed neighborhood as the local ROI region of the current face image.

[0087] The above process, by determining specific facial key points and their neighborhoods based on action features, can accurately locate regions closely related to specific actions, avoiding indiscriminate processing of the entire face image and allowing the analysis to focus more on the key parts where the action occurs. When determining local ROI regions, overlapping neighborhoods are merged, avoiding multiple processing of repetitive information. This not only reduces the amount of data processing and computational complexity but also avoids the accumulation of errors that may result from repeatedly processing overlapping areas, improving the system's operating efficiency and the reliability of the analysis results.

[0088] S104: Based on the multi-frame images corresponding to the current face image, perform multi-layer feature resampling on the local ROI region to correct the action label according to the local action label corresponding to the sampling result.

[0089] The action information obtained from a single frame image is limited and cannot fully and accurately reflect the real action. Therefore, to improve recognition accuracy, multi-layer feature resampling is performed on the local ROI region based on multiple frames corresponding to the current face image, thereby correcting the action label based on the local action label corresponding to the sampling result. Specifically, in one or more embodiments of this specification, multi-layer feature resampling of the local ROI region based on multiple frames corresponding to the current face image specifically includes:

[0090] Because faces may move and rotate slightly within a series of video frames, to ensure temporal and spatial consistency between multi-frame images and local areas of interest (ROIs) and to allow subsequent analysis to be based on a unified coordinate system, the spatial positions of multi-frame images and local ROIs are aligned along the temporal dimension by comparing the positions of keypoints in the multi-frame images with those in the local ROIs. After spatial alignment, the corresponding dynamic local ROIs in each frame are determined based on the parts in the multi-frame images that correspond to the keypoints of the local ROIs. Since facial poses and expressions may change across multiple frames, the local ROIs will also change dynamically. Therefore, it is necessary to accurately locate the regions corresponding to the keypoints of the initial local ROIs in each frame, forming dynamic local ROIs. These dynamic local ROIs reflect the changes in local facial features at different times. The determined dynamic local ROIs are then input into a pre-defined convolutional neural network (CNN) model. Different network layers of the CNN model extract features at different levels and abstractions, resulting in multi-layered features output from different network layers of the CNN model. Since the multi-layer features output by different network layers may have different spatial resolutions, in order to effectively fuse these features, it is necessary to unify their spatial resolutions so as to obtain the sampling results by fusing the unified multi-layer features.

[0091] In this process, spatial alignment ensures temporal and spatial consistency between multiple image frames and local Regions of Interest (ROIs), enabling subsequently extracted features to accurately reflect local changes in the face at different times. The setting of dynamic local ROIs allows for focus on key local areas at different times, improving the ability to capture dynamic changes. Furthermore, by unifying spatial resolution and fusing features, these different levels of feature information can be integrated to form a more representative sampling result. This comprehensive feature set better reflects the overall features and changes of the face, contributing to improved performance in subsequent recognition. By processing multiple image frames and fusing multi-layer features, features at different levels can complement each other, making subsequent face recognition more accurate.

[0092] Furthermore, in one or more embodiments of this specification, the action label is modified based on the local action label corresponding to the sampling result, specifically including:

[0093] The sampling results obtained after multi-layer feature resampling are input into the classifier of a pre-set CNN model. Based on the classifier's analysis and matching, local action labels corresponding to the sampling results are obtained. Since the initially determined action labels are obtained from some basic feature analysis in previous processing steps, they may not be accurate or comprehensive enough. Therefore, the local action labels from multiple frames are compared with the actual action labels to determine if there are any conflicts. For example, if the initial action label is judged as "normal expression," while the local action label shows "slight frown," this indicates a conflict. If a conflict is found between the local action labels and the actual action labels during the comparison, the initial action labels are overwritten and modified using the local action labels. It can be understood that the local action labels are obtained based on more refined multi-frame image multi-layer feature resampling and classifier judgment, containing richer and more accurate information. Through overwriting and modification, corrected action labels are obtained, making them more consistent with the actual facial movements.

[0094] S105: Match the corrected action label with the static attribute label and the preset label result knowledge graph to achieve face recognition of the current face image.

[0095] Traditional facial recognition primarily relies on static facial features, such as the shape and position of facial features. However, faces undergo rich dynamic changes in different scenarios, which can significantly alter static features, thus reducing recognition accuracy. Therefore, this application combines modified action labels and static attribute labels for matching to achieve facial recognition of the current face image. This allows for a multi-dimensional description of facial features, increasing the basis for recognition. It overcomes the shortcomings of relying solely on static feature recognition and improves recognition accuracy.

[0096] Furthermore, in one or more embodiments of this specification, before matching the modified action labels and static attribute labels with a preset label result knowledge graph to achieve face recognition of the current face image, the method further includes the following process:

[0097] First, structured labeled data is obtained from existing face and behavior databases. Then, action descriptions and action attribute association rules are extracted from multiple public texts to obtain unstructured data. It's understandable that public texts may contain various descriptions of actions; by analyzing these texts, the relationships between actions and related attributes can be extracted. For example, the action of "smiling" may be associated with the attribute of "friendly." Next, entity extraction is performed on the unstructured data to identify key entities and attribute information. Key entities include the action subject and the action object. Then, an initial labeling result knowledge graph is constructed based on the key entities and their relationships. The extracted attribute information is added to the initial labeling result knowledge graph to enrich and specify it. The entities corresponding to the structured labeled data are aligned with the initial labeling result knowledge graph. In this way, the rich information in the structured labeled data is used to expand the initial knowledge graph, resulting in a pre-defined labeling result knowledge graph.

[0098] This process integrates structured and unstructured data, fully leveraging the advantages of data from different sources. Structured labeled data provides accurate and standardized information, while unstructured data contains richer and more diverse textual descriptions and potential association rules. The combination of the two allows the knowledge graph to cover more comprehensive information. Through entity extraction and relationship construction, key knowledge can be effectively extracted from complex data and represented intuitively and clearly in the form of a knowledge graph. Furthermore, the approach of first constructing an initial knowledge graph and then expanding it by aligning it with structured labeled data gives the knowledge graph strong scalability and flexibility.

[0099] Specifically, in one or more embodiments of this specification, the modified action label is matched with the static attribute label and the preset label result knowledge graph to achieve face recognition of the current face image, specifically including:

[0100] The encoding format of the pre-set label result knowledge graph is determined, and the corrected action labels and static attribute labels are standardized based on the encoding format to obtain processed action labels and processed static attribute labels. After standardization, entities and relationships corresponding to the processed action labels and static attribute labels are searched in the pre-set label result knowledge graph to obtain multiple matching results. For each of the found matching results, the similarity score is calculated, and the matching results are sorted based on the similarity score. The matching result with the highest score is taken as the face recognition result of the current face image.

[0101] By defining and standardizing the encoding format, the data compatibility and consistency between the corrected action labels and static attribute labels and the pre-defined label results knowledge graph are ensured. This helps improve the accuracy and efficiency of data matching, avoids errors or omissions caused by inconsistent data formats, and enables the effective integration and utilization of data from different sources within the knowledge graph framework. Calculating and ranking the similarity of the matching results provides a quantitative basis for the recognition results, avoids the uncertainty of subjective judgment, and further improves the accuracy of recognition.

[0102] Based on the same inventive concept, this invention also provides a face recognition system based on a multi-label face algorithm, the structure of which is as follows: Figure 2 As shown.

[0103] Figure 2 This is a schematic diagram of the structure of a face recognition system based on a multi-label face algorithm provided by the present invention. Figure 2 As shown, the system in this invention specifically includes:

[0104] The construction unit 201 is used to perform a preliminary scan based on a sliding window corresponding to the current face image to obtain facial key points, and to construct a global coverage path for facial features based on the facial key points;

[0105] The acquisition unit 202 is used to perform a global scan of the current face region according to the global coverage path of the face features, so as to match the scan result with a preset label image to obtain the global image label corresponding to the current face image; wherein, the global image label includes: action label and static attribute label;

[0106] The determining unit 203 is used to determine the specified facial key points corresponding to the action label, so as to determine the local ROI region of each specified facial key point in the current face image according to the grid region corresponding to each specified facial key point.

[0107] The correction unit 204 is used to perform multi-layer feature resampling on the local ROI region based on the multi-frame images corresponding to the current face image, so as to correct the action label according to the local action label corresponding to the sampling result;

[0108] The recognition unit 205 is used to match the corrected action label with the static attribute label and the preset label result knowledge graph to realize face recognition of the current face image.

[0109] The various embodiments in this invention are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0110] The device and method provided by this invention are one-to-one correspondences. Therefore, the device also has similar beneficial technical effects as its corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the device will not be repeated here.

[0111] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media containing computer-usable program code.

[0112] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0113] The above are merely embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of protection of the present invention.

Claims

1. A face recognition method based on a multi-label face algorithm, characterized in that, The method includes: A preliminary scan is performed based on a sliding window corresponding to the current face image to obtain facial key points, and a global facial feature coverage path is constructed based on the facial key points. The current face image is globally scanned according to the global coverage path of the facial features, and the scan results are matched with preset label images to obtain the global image label corresponding to the current face image; wherein, the global image label includes: action label and static attribute label; the scan result includes: basic data of each key point of the current face image obtained during the scan, and key point displacement data between adjacent frames; the basic data includes: pixel value and position data; the key point displacement data between adjacent frames represents the position change of the face key points in the current frame relative to the previous frame; Determine the specified facial key points corresponding to the action labels, and determine the local ROI region of each specified facial key point in the current face image based on the neighborhood corresponding to each specified facial key point. Based on the multiple frames corresponding to the current face image, multi-layer feature resampling is performed on the local ROI region to correct the action label according to the local action label corresponding to the sampling result; The corrected action labels are matched with the static attribute labels and the preset label result knowledge graph to achieve face recognition of the current face image; A preliminary scan is performed based on a sliding window corresponding to the current face image to obtain facial key points. A global facial feature coverage path is then constructed based on these key points, specifically including: Based on the image data of the current face image and the recognition scenario, the size range of the sliding window is determined so as to perform a preliminary scan of the current face image according to the sliding window of each size range and obtain the face key points; Based on the physiological structural and functional features corresponding to each of the facial key points, each of the facial key points is classified to obtain multiple sets of facial key point sets. Based on the matching relationship between each set of facial key points and the preset physiological cognitive order, the set path order corresponding to each set of facial key points is determined; wherein, the preset physiological cognitive order refers to the scanning priority rule predefined according to the anatomical structure of the face, the functional importance, and human visual cognitive habits. Based on the spatial positional relationship of each facial key point within each set of facial key points, the local paths between each facial key point within the set of facial key points are determined. Determine whether the set of facial key points corresponding to the local path corresponds to a high dynamic structure. If so, expand the branches of the local path based on the preset values ​​corresponding to the high dynamic structure to obtain the expanded local path. Based on the set path order and the local paths and expanded local paths between each of the facial key points in the set of facial key points, the initial global coverage path of facial features is determined; wherein, the high dynamic structure corresponds to the part with large movement amplitude or frequent movement. The initial global coverage path of facial features is smoothed to obtain a global coverage path of facial features.

2. The face recognition method based on a multi-label face algorithm according to claim 1, characterized in that, Based on the image data of the current face image and the recognition scene, the size range of the sliding window is determined, and a preliminary scan of the current face image is performed according to the sliding window of each size range to obtain facial key points, specifically including: The image attributes of the current face image are obtained, and the image attribute label corresponding to the current face image is determined based on the image data. The historical face images corresponding to the recognition scene of the current face image and the image attribute label are obtained. The image attributes include: illumination intensity, average face proportion, and background complexity. Based on the sliding window size corresponding to historical face images, determine the initial size range of the sliding window corresponding to the current face image, and expand the initial size range based on the image resolution corresponding to the current face image to obtain the size range of the sliding window; Based on the image size and resolution of the current face image, the corresponding number of image layers is determined, and the corresponding scaling factor is determined based on the preset recognition requirement information, so as to determine the number of sampleable layers of the current face image according to the number of image layers and the scaling factor; Based on the scaling factor and the size range of the sliding window, determine the current size range of the sliding window for each sampleable layer of the current face image; Sliding detection is performed on the current face image at each sampleable layer using a sliding window within the current size range to obtain facial key points.

3. The face recognition method based on a multi-label face algorithm according to claim 1, characterized in that, A global scan of the current face image is performed based on the global coverage path of the facial features. The scan results are then matched with preset label images to obtain the global image label corresponding to the current face image. Specifically, this includes: The current face image is scanned globally based on the global coverage path of the face features, so as to obtain the basic data of each key point of the current face image and the key point displacement data between adjacent frames based on the scanning results; Based on the basic data of each key point in the current face image, the static attribute features of the current face image are determined, and based on the key point displacement data between adjacent frames, the motion features of the current face image are determined. The static attribute label corresponding to the static attribute feature is determined by performing similarity matching between the static attribute feature and the facial geometric features corresponding to each label in the preset static label image. The action label corresponding to the action feature is determined by performing similarity matching between the action feature and the action trajectory features corresponding to each label in the preset action label image.

4. The face recognition method based on a multi-label face algorithm according to claim 3, characterized in that, Determine the specified facial key points corresponding to the action labels, and based on the neighborhoods corresponding to each specified facial key point, determine the local ROI region of the current face image for each specified facial key point, specifically including: Based on the action features corresponding to the action tags, the facial key points corresponding to the action tags are determined as designated facial key points; Using each specified facial key point as the center, obtain the neighborhood corresponding to each specified facial key point; Determine whether the neighborhoods corresponding to each of the specified facial key points overlap; If there is overlap and the overlap ratio is greater than a preset ratio, the neighborhood is merged to obtain the processed neighborhood as the local ROI region of the current face image.

5. A face recognition method based on a multi-label face algorithm according to claim 1, characterized in that, Based on the multiple frames corresponding to the current face image, multi-layer feature resampling is performed on the local ROI region, specifically including: Based on the positions of key points in the multi-frame images corresponding to the current face image and the positions of key points in the local ROI region, align the spatial positions of the multi-frame images and the local ROI region in the time series; Determine the dynamic local ROI region corresponding to the key points of the local ROI region in the aligned multi-frame images; The dynamic local ROI region is input into a preset CNN model to obtain multi-layer features output by different network layers of the preset CNN model; The spatial resolution corresponding to the multi-layer features is unified, and the sampling result is obtained by fusing the unified multi-layer features.

6. A face recognition method based on a multi-label face algorithm according to claim 5, characterized in that, The action labels are corrected based on the local action labels corresponding to the sampling results, specifically including: The sampling results are input into the classifier of the preset CNN model to obtain the local action labels corresponding to the sampling results; The local action labels of the multi-frame images are compared with the action labels to determine whether there is a conflict between the local action labels and the action labels; If so, the action tag is overwritten and modified based on the local action tag to obtain the corrected action tag.

7. A face recognition method based on a multi-label face algorithm according to claim 1, characterized in that, Before matching the corrected action labels with the static attribute labels and the preset label result knowledge graph to achieve face recognition of the current face image, the method further includes: Based on existing face databases and behavior databases, structured labeled data is obtained, and unstructured data is obtained by extracting action descriptions and action attribute association rules from multiple public texts. Entity extraction is performed on the unstructured data to obtain key entities and attribute information; wherein, the key entities include: action subject and action object; Based on the relationships between the key entities, an initial label result knowledge graph is constructed, and the attribute information is added to the initial label result knowledge graph; Align the entities corresponding to the structured annotation data with the initial label result knowledge graph to expand the initial label result knowledge graph based on the structured annotation data, thereby obtaining a preset label result knowledge graph.

8. A face recognition method based on a multi-label face algorithm according to claim 1, characterized in that, The corrected action labels are matched with the static attribute labels and the preset label result knowledge graph to achieve face recognition of the current face image, specifically including: The encoding format of the pre-set tag result knowledge graph is determined, and the modified action tags and static attribute tags are standardized based on the encoding format to obtain processed action tags and processed static attribute tags. In the pre-set tag result knowledge graph, search for entities and relationships corresponding to the processed action tags and processed static attribute tags to obtain multiple matching results; Obtain the similarity corresponding to each of the matching results, sort the matching results based on the similarity, and take the matching result with the highest score as the face recognition result of the current face image.

9. A face recognition system based on a multi-label face algorithm, characterized in that, The system includes: The construction unit is used to perform a preliminary scan based on a sliding window corresponding to the current face image, obtain facial key points, and construct a global coverage path for facial features based on the facial key points; The acquisition unit is used to perform a global scan of the current face image according to the global coverage path of the face features, and to match the scan results with a preset label image to obtain the global image label corresponding to the current face image; wherein, the global image label includes: action label and static attribute label; The determining unit is used to determine the specified facial key points corresponding to the action label, so as to determine the local ROI region of each specified facial key point in the current face image according to the grid region corresponding to each specified facial key point. The correction unit is used to perform multi-layer feature resampling on the local ROI region based on the multi-frame images corresponding to the current face image, so as to correct the action label according to the local action label corresponding to the sampling result; The recognition unit is used to match the corrected action label with the static attribute label and the preset label result knowledge graph to achieve face recognition of the current face image; A preliminary scan is performed based on a sliding window corresponding to the current face image to obtain facial key points. A global facial feature coverage path is then constructed based on these key points, specifically including: Based on the image data of the current face image and the recognition scenario, the size range of the sliding window is determined so as to perform a preliminary scan of the current face image according to the sliding window of each size range and obtain the face key points; Based on the physiological structural and functional features corresponding to each of the facial key points, each of the facial key points is classified to obtain multiple sets of facial key point sets. Based on the matching relationship between each set of facial key points and the preset physiological cognitive order, the set path order corresponding to each set of facial key points is determined; wherein, the preset physiological cognitive order refers to the scanning priority rule predefined according to the anatomical structure of the face, the functional importance, and human visual cognitive habits. Based on the spatial positional relationship of each facial key point within each set of facial key points, the local paths between each facial key point within the set of facial key points are determined. Determine whether the set of facial key points corresponding to the local path corresponds to a high dynamic structure. If so, expand the branches of the local path based on the preset values ​​corresponding to the high dynamic structure to obtain the expanded local path. Based on the set path order and the local paths and expanded local paths between each facial key point in the set of facial key points, an initial global coverage path for facial features is determined; wherein, the high dynamic structure corresponds to the part with large movement amplitude or frequent movement; the initial global coverage path for facial features is smoothed to obtain a global coverage path for facial features.

Images