Stranger face recognition system and method, foot type humanoid robot and medium

Abstract

The invention discloses a stranger face recognition system and method, a foot-type humanoid robot and a medium, which are applied to the technical field of image processing, and adopt a multi-task face recognition model to perform face bounding box prediction and face key point extraction on an environment image to obtain a face key point set in each face bounding box; aligning the face key point set in each face bounding box to the standard template face image to obtain an undeformed face image; extracting a face feature vector of a to-be-recognized face from each non-deformation face image; calculating the similarity between the face feature vector of each face to be recognized and the face feature vector of each temporary reference face in a lightweight face database; and determining whether the to-be-recognized face is a stranger face based on the similarity. Thus, stranger face recognition can be achieved on the foot type humanoid robot, network delay caused by cloud comparison is avoided, real-time interaction response in a public environment is achieved, and the man-machine interaction efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a stranger face recognition system and method, a foot-type humanoid robot and a medium. BACKGROUND

[0002] With the rapid development of foot-type humanoid robots and embodied intelligence, foot-type humanoid robots in public environments are increasingly widely used in human-computer interaction, service guidance and safety monitoring and other tasks. As one of the core means of human-computer interaction, face recognition technology can detect and compare user face information to achieve identity recognition, behavior recording and personalized services based on identity.

[0003] In crowded and dynamically high environment public places such as shopping malls, stations and airports, foot-type humanoid robots need to achieve real-time face recognition and interaction in complex lighting, frequent personnel flow and multiple target simultaneous appearance and other complex scenes, which puts higher requirements on the robustness, power consumption and response delay of the face recognition system.

[0004] The existing face recognition system mostly adopts a cloud comparison method, and the foot-type humanoid robot uploads the collected face image to the cloud for comparison and verification. The cloud comparison process introduces network delay, which is difficult to meet the requirements of real-time interaction of the robot, thereby affecting the interaction efficiency of the robot in public places. SUMMARY

[0005] The present application provides a stranger face recognition system, method, foot-type humanoid robot and medium to solve the problem of low efficiency of stranger face recognition of foot-type humanoid robots in public environments. The technical solutions provided by the present application are as follows: On the one hand, the present application provides a stranger face recognition system deployed on a foot-type humanoid robot, comprising a video stream acquisition module, an image quality optimization module, a face feature extraction module and a face identity determination module. The video stream acquisition module is integrated on the body of the foot-type humanoid robot and is used to acquire environmental video streams in public environments. The image quality optimization module is connected with the video stream acquisition module and is used to use a lightweight quality repair model to repair the quality of the environmental images in the environmental video streams to obtain target environmental images. The face feature extraction module is connected with the image quality optimization module, and is configured to adopt a multi-task face recognition model to perform face bounding box prediction and face key point extraction on the target environment image to obtain a face bounding box set and a face key point set in each face bounding box in the face bounding box set; based on a standard template key point set, the face key point set in each face bounding box is aligned to a standard template face image to obtain a non-distortion face image corresponding to each face bounding box; a multi-task feature extraction model is adopted to extract a global face visual feature vector and a multi-dimensional face attribute feature vector of the face to be recognized from the non-distortion face image corresponding to each face bounding box as a face feature vector; The face identity determination module is connected with the face feature extraction module, and is configured to calculate the similarity between the face feature vector of the face to be recognized in each face bounding box and the face feature vector of each temporary reference face dynamically stored in the local lightweight face database, and determine whether the face to be recognized in each face bounding box is a stranger face based on the similarity; wherein the face feature vector of each temporary reference face dynamically stored in the local lightweight face database is the face feature vector of each stranger face recognized in a recent set time range.

[0006] Optionally, the image quality optimization module is configured to, for each frame of environment image in the environment video stream, calculate an image quality score of the environment image by using an image quality evaluation function; when the image quality score is less than a first quality threshold, discarding the environment image; when the image quality score is greater than or equal to the first quality threshold and less than a second quality threshold, repairing the motion blur, image noise and illumination anomaly of the environment image by using a lightweight quality repair model to obtain a target environment image; when the image quality score is greater than or equal to the second quality threshold, determining the environment image as the target environment image.

[0007] Optionally, the multi-task face recognition model comprises a first feature map extraction branch and a three-level detection head connected in parallel with the first feature map extraction branch; wherein the three-level detection head is a face two-classification detection branch, a face bounding box regression branch and a face key point regression branch. The face feature extraction module is configured to, after extracting first shared semantic feature maps from the target environment image through a first feature map extraction branch in the multi-task face recognition model, perform face binary classification detection, face bounding box detection, and face key point detection in parallel based on the first shared semantic feature maps through a face binary classification detection branch, a face bounding box regression branch, and a face key point regression branch in the multi-task face recognition model to obtain face binary classification detection results, face bounding box detection results, and face key point detection results; and perform post-processing on the face binary classification detection results, the face bounding box detection results, and the face key point detection results to obtain a face bounding box set corresponding to the target environment image and a face key point set in each face bounding box in the face bounding box set.

[0008] Optionally, the face feature extraction module is configured to, for each face bounding box, calculate an affine transformation matrix between the face key point set in the face bounding box and a standard template key point set, and align the face key point set in the face bounding box to a calibrated template face image based on the affine transformation matrix to obtain a non-distortion face image corresponding to the face bounding box.

[0009] Optionally, the multi-task feature extraction model includes a second feature map extraction branch and a two-level detection head connected in parallel to the second feature map extraction branch; the two-level detection head includes a global visual feature extraction branch and a multi-attribute feature extraction branch. The face feature extraction module is configured to, after extracting second shared semantic feature maps from the non-distortion face image corresponding to each face bounding box through a second feature map extraction branch in the multi-task feature extraction model, perform global visual feature extraction and multi-attribute feature extraction in parallel based on the second shared semantic feature maps corresponding to each face bounding box through a global visual feature extraction branch and a multi-attribute feature extraction branch in the multi-task feature extraction model to obtain a global face visual feature vector and a multi-dimensional face attribute feature vector corresponding to each face bounding box; and perform post-processing on the global face visual feature vector and the multi-dimensional face attribute feature vector corresponding to each face bounding box to obtain a face feature vector corresponding to each face bounding box.

[0010] Optionally, the face identity determination module is configured to calculate the Euclidean distance between the face feature vector of each face in the face bounding box to be recognized and the face feature vector of each temporary reference face dynamically stored in the local lightweight face database by using a similarity calculation method based on the Euclidean distance; for each face in the face bounding box to be recognized, if the Euclidean distance between the face feature vector of the face to be recognized and the face feature vector of any temporary reference face dynamically stored in the local lightweight face database is less than the distance threshold, it is determined that the face to be recognized and the temporary reference face are the same face, and if the Euclidean distance between the face feature vector of the face to be recognized and the face feature vector of all temporary reference faces dynamically stored in the local lightweight face database is greater than or equal to the distance threshold, it is determined that the face to be recognized is a stranger face.

[0011] Optionally, the stranger face recognition system provided by the present application further comprises: The data management module is connected with the face identity determination module and is configured to, for each face in the face bounding box to be recognized, if the face to be recognized and any temporary reference face in the local lightweight face database are the same face, update the recording time of any temporary reference face to the current time, if the face to be recognized is a stranger face, assign a unique temporary identity to the stranger face, store the stranger face as a temporary reference face together with the face feature vector and the unique temporary identity of the stranger face to the local lightweight face database, and take the current time as the recording time; and automatically delete the unique temporary identity and the face feature vector of the temporary reference face in the local lightweight face database whose recording duration exceeds the storage duration threshold.

[0012] On the other hand, the present application provides a stranger face recognition method applied to a legged humanoid robot, comprising: capturing an environment video stream in a public environment; using a lightweight quality restoration model to perform quality restoration on the environment images in the environment video stream to obtain target environment images; using a multi-task face recognition model to perform face bounding box prediction and face key point extraction on the target environment images to obtain a face bounding box set and a face key point set in each face bounding box in the face bounding box set; aligning the face key point set in each face bounding box to a standard template face image based on a standard template key point set to obtain a non-distorted face image corresponding to each face bounding box; using a multi-task feature extraction model to extract a global face visual feature vector and a multi-dimensional face attribute feature vector of each face to be recognized from the non-distorted face image corresponding to each face bounding box as a face feature vector; calculate the similarity between the face feature vector of the to-be-identified face in each face bounding box and the face feature vector of each temporary reference face dynamically stored in the local lightweight face database; determine whether the to-be-identified face in each face bounding box is a stranger face based on the similarity; wherein the face feature vector of each temporary reference face dynamically stored in the local lightweight face database is the face feature vector of each stranger face identified within a recently set time range.

[0013] In another aspect, the present application provides a legged humanoid robot comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the above-mentioned stranger face recognition method when executing the computer program.

[0014] In another aspect, the present application provides a computer readable storage medium, which stores computer instructions, wherein the computer instructions are executed by a processor to implement the above-mentioned stranger face recognition method.

[0015] The beneficial effects of the present application are as follows: The present application can utilize the image quality restoration of the lightweight quality restoration model to solve the problem of low-quality conditions such as overexposure or underexposure, blur, etc. caused by complex public environment lighting and motion blur caused by the walking of the legged humanoid robot, ensuring the reliability of subsequent face recognition. In addition, the present application can utilize the face bounding box recognition of the multi-task face recognition model, the face feature extraction of the multi-task feature extraction model and the stranger face discrimination of the local lightweight face database to realize stranger face recognition on the legged humanoid robot side, avoid network delay introduced by cloud comparison, and thus realize real-time interaction response in a public environment, improve the understanding and service ability of the legged humanoid robot to the interactive object, and significantly improve the human-computer interaction experience in public places.

[0016] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent to those skilled in the art from the description, or can be learned by practice of the present application. The objects and other advantages of the present application will be realized and attained by the structure particularly pointed out in the written description and the appended drawings. BRIEF DESCRIPTION OF DRAWINGS

[0017] The accompanying drawings, which are included to provide a further understanding of the present application and constitute a part of this application, illustrate embodiments of the present application and, together with the description, serve to explain the present application. In the drawings: Figure 1 It is a system framework schematic diagram of the stranger face recognition system in the present application; Figure 2 This is a schematic diagram outlining the process of the stranger face recognition method in this application; Figure 3 This is a schematic diagram of the hardware structure of the legged humanoid robot in this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and beneficial effects of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with the embodiments and accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0019] This application provides a stranger facial recognition system deployed on a legged humanoid robot. (See attached document.) Figure 1 As shown, the stranger face recognition system 100 provided in this application embodiment includes a video stream acquisition module 110, an image quality optimization module 120, a face feature extraction module 130, and a face identity determination module 140. The video stream acquisition module 110 is integrated into the body of the legged humanoid robot and is used to acquire environmental video streams in public environments. The image quality optimization module 120 is connected to the video stream acquisition module 110 and is used to perform quality restoration on the environmental image in the environmental video stream using a lightweight quality restoration model to obtain the target environmental image. The face feature extraction module 130, connected to the image quality optimization module 120, is used to perform face bounding box prediction and face key point extraction on the target environment image using a multi-task face recognition model to obtain a set of face bounding boxes and a set of face key points within each face bounding box in the set of face bounding boxes; based on the standard template key point set, the set of face key points within each face bounding box is aligned to the standard template face image to obtain an undistorted face image corresponding to each face bounding box; using a multi-task feature extraction model, the global face visual feature vector and multi-dimensional face attribute feature vector of the face to be recognized are extracted from the undistorted face image corresponding to each face bounding box as face feature vectors; The face identity determination module 140 is connected with the face feature extraction module 130, and is configured to calculate the similarity between the face feature vector of each face to be recognized in each face bounding box and the face feature vector of each temporary reference face dynamically stored in the local lightweight face database, and determine whether each face to be recognized in each face bounding box is a stranger face based on the similarity; wherein the face feature vector of each temporary reference face dynamically stored in the local lightweight face database is the face feature vector of each stranger face recognized in the recent set time range.

[0020] In the embodiment of the present application, the color camera on the body (for example, the chest) of the biped humanoid robot collects the environmental video stream in the public environment through the color camera during the walking process of the biped humanoid robot. The lightweight quality repair model, the multi-task face recognition model, the multi-task feature extraction model and the local lightweight face database are deployed at the end side of the biped humanoid robot; the image quality optimization module 120 uses the lightweight quality repair model to perform quality repair on the environmental images in the environmental video stream to obtain target environmental images; after the face feature extraction module 130 uses the multi-task face recognition model to perform face bounding box prediction and face key point extraction on the target environmental images to obtain a face bounding box set and a face key point set in each face bounding box in the face bounding box set, based on a standard template key point set, respectively aligns the face key point set in each face bounding box to a standard template face image to obtain a non-distortion face image corresponding to each face bounding box, and uses the multi-task feature extraction model to extract a global face visual feature vector and a multi-dimensional face attribute feature vector of each face to be recognized from the non-distortion face image corresponding to each face bounding box as a face feature vector; the face identity determination module 140 calculates the similarity between the face feature vector of each face to be recognized in each face bounding box and the face feature vector of each temporary reference face dynamically stored in the local lightweight face database, and determines whether each face to be recognized in each face bounding box is a stranger face based on the similarity. In this way, the image quality repair of the lightweight quality repair model can be used to solve the problem of low-quality conditions such as overexposure or underexposure, blur, etc. of the environmental images caused by complex public environment lighting and motion blur caused by the walking of the biped humanoid robot, ensuring the reliability of subsequent face recognition, and the face bounding box recognition of the multi-task face recognition model, the face feature extraction of the multi-task feature extraction model and the stranger face discrimination of the local lightweight face database can be used to realize stranger face recognition at the end side of the biped humanoid robot, thereby avoiding the network delay introduced by cloud comparison, realizing real-time interactive response in the public environment, improving the understanding and service ability of the biped humanoid robot to the interactive object, and significantly improving the human-computer interaction experience in public places.

[0021] In a possible implementation, the image quality optimization module 120 is configured to, for each frame of the environment image in the environment video stream, calculate an image quality score of the environment image by using an image quality evaluation function; discard the environment image when the image quality score is less than a first quality threshold; repair the motion blur, image noise and light abnormality of the environment image by using a lightweight quality repair model to obtain a target environment image when the image quality score is greater than or equal to the first quality threshold and less than a second quality threshold; and determine the environment image as the target environment image when the image quality score is greater than or equal to the second quality threshold.

[0022] In the embodiments of the present application, the image quality evaluation function is defined as:

[0023] wherein, I is the environment image; Q(I) is the image quality score of the environment image I; S(I) is an image sharpness index of the environment image I, used to quantify the sharpness of image details; C(I) is an image contrast index of the environment image I, used to measure the image gray scale distribution range; E(I) is an image exposure evaluation index of the environment image I, used to quantify the image brightness deviation; and α, β, γ are weighting coefficients.

[0024] The sharpness index S(I) is calculated in the following manner: the environment image I is converted to a luminance image Y∈ (H×W); a 3x3 convolution is performed on the luminance image Y by using horizontal and vertical Sobel operators Gx, Gy to obtain gradient images Gx(i,j), Gy(i,j); the gradient amplitude G(i,j)=|Gx(i,j)|+|Gy(i,j)| is calculated pixel by pixel; and the image sharpness index S(I) is obtained by averaging and normalizing the full-frame gradient amplitude. wherein, 510 is the theoretical maximum value of the gradient amplitude, and the normalized image sharpness index S(I)∈[0,1].

[0025] The contrast index C(I) is calculated in the following manner: the environment image I is converted to a luminance image Y∈ (H×W); the global mean value of the luminance image Y is calculated. The image contrast index C(I) is obtained by calculating the luminance standard deviation based on the global mean value and normalizing. wherein, the normalized image contrast index C(I)∈[0,1].

[0026] The exposure evaluation index E(I) is calculated in the following manner: the environment image I is converted to a luminance image Y∈ (H×W); the luminance image Y is quantized to a 64-bin histogram H[k], k=0…63; and an ideal Gaussian weight W[k] is constructed. , 31.5 is the center position of the 64-bin histogram (k = 0…63), i.e. the mean μ of the Gaussian distribution, corresponding to the middle gray level of brightness, 2 is a fixed constant to ensure the integral of the probability density function to be 1, and 8 is the standard deviation σ of the Gaussian distribution to control the weight drop speed, so that the brightness obtains the maximum weight near the center; the weighted histogram sum is calculated and normalized to obtain the image exposure index ; wherein the normalized image exposure index E(I) ∈ [0, 1], the larger the value, the more concentrated the brightness distribution is on the middle gray, and the better the exposure.

[0027] wherein α, β, γ are dynamically adjusted according to the lighting conditions of the public environment and the motion state of the biped humanoid robot, and specifically include: By means of the inertial measurement unit arranged in the body of the biped humanoid robot and the force sensor arranged on the sole, the motion acceleration time sequence signal of the biped humanoid robot is acquired in real time, and the motion impact spectrum energy in a predetermined frequency band is calculated to generate the image sharpness confidence at the current moment; By means of the ambient light sensor array arranged beside the color camera, the ambient illumination jump intensity and color temperature drift amount are acquired in real time to generate the image exposure confidence at the current moment; The image sharpness confidence and the image exposure confidence are input into the pre-trained lightweight mapping network, and the initial weighting coefficients α', β', γ' are output after Softmax normalization, wherein when it is detected that the face detection recall rate is lower than the recall rate threshold and the motion impact spectrum energy exceeds the energy threshold or the illumination jump intensity exceeds the intensity threshold, a gradient descent update is triggered once to make the network parameters of the lightweight mapping network adaptive to the new environment, and the initial weighting coefficients α', β', γ' are output again through the lightweight mapping network; The initial weighting coefficients α', β', γ' output by the lightweight mapping network are subjected to inertia damping filtering to obtain the final weighting coefficients α, β, γ to suppress the oscillation of the weighting coefficients, wherein α+β+γ=1.

[0028] Further, in the embodiment of the present application, the first quality threshold and the second quality threshold Two thresholds, low-quality images are directly discarded, i.e. the frame is skipped, to avoid recording low-quality faces or false recognition, medium-quality images are subjected to enhancement processing, a lightweight quality repair model trained based on the adversarial generation network technology is used to repair motion blur, noise, excessive brightness or darkness and other abnormal conditions to obtain a target environment image, and high-quality images are kept as they are and directly determined as target environment images. The processed target environment image set is represented as By image inpainting through a lightweight quality repair model, low-quality frames can be effectively filtered and medium-quality images can be improved, so that subsequent face detection and recognition remain stable and efficient in dynamic walking environments and complex lighting.

[0029] In a possible implementation, the multi-task face recognition model comprises a first feature map extraction branch, and a three-level detection head connected in parallel with the first feature map extraction branch; the three-level detection head is a face two-classification detection branch, a face bounding box regression branch, and a face key point regression branch. The face feature extraction module 130 is configured to perform feature extraction on the target environment image through the first feature map extraction branch in the multi-task face recognition model to obtain a first shared semantic feature map, and perform face two-classification detection, face bounding box detection, and face key point detection in parallel based on the first shared semantic feature map through the face two-classification detection branch, the face bounding box regression branch, and the face key point regression branch in the multi-task face recognition model to obtain a face two-classification detection result, a face bounding box detection result, and a face key point detection result; and perform post-processing on the face two-classification detection result, the face bounding box detection result, and the face key point detection result to obtain a face bounding box set corresponding to the target environment image and a face key point set in each face bounding box in the face bounding box set.

[0030] In the embodiments of the present application, on the basis of image quality optimization, the optimized target environment image set is subjected to face detection and key point extraction to realize real-time perception of faces of the humanoid robot in a public environment. The face bounding box set corresponding to the kth target environment image is represented as: , is the left upper corner coordinate of the detected ith face bounding box and the width and height , of the face bounding box, is a face bounding box detection function; the face key point set in the ith face bounding box in the face bounding box set corresponding to the kth target environment image is represented as: ; key point regression is realized by minimizing a mean square error loss function: , m is the total number of face key points in the ith face bounding box, is the predicted coordinate of the jth face key point, is the calibrated coordinate of the jth face key point, which is considered as a completely correct face key point position given by a calibration officer or a high-precision device during training or evaluation, and is used to compare and calculate the loss or accuracy with the network prediction value.

[0031] In a possible implementation, the face feature extraction module 130 is configured to, for each face bounding box, calculate an affine transformation matrix between a set of face key points in the face bounding box and a set of standard template key points, and align the set of face key points in the face bounding box to a calibrated template face image based on the affine transformation matrix to obtain a non-distortion face image corresponding to the face bounding box.

[0032] In the embodiments of the present application, to improve the face recognition accuracy, the set of face key points in the detected face bounding box needs to be calibrated, which aims to align the set of face key points in the face bounding box to a unified standard coordinate system, eliminate pose deviation, rotation and slight perspective distortion, so as to ensure the stability of subsequent feature extraction and comparison. Specifically, the calibration includes: aligning the set of face key points in the i-th face bounding box to the set of standard template key points based on an affine transformation or similarity transformation.

[0033] wherein, is a rotation matrix, is a scaling coefficient, is a translation vector, is a predicted coordinate of the face key point before calibration, is a standard coordinate of the standard template key point.

[0034] solving the affine transformation or similarity transformation by minimizing the key point alignment error to obtain an affine transformation matrix .

[0035] wherein, is a key point alignment error loss function, m is the total number of face key points in the i-th face bounding box, is a predicted coordinate of the j-th face key point, is a standard coordinate of the standard template key point corresponding to the j-th face key point.

[0036] aligning the i-th face bounding box image to a standard template face image of a standard size and a standard pose to obtain a calibrated non-distortion face image based on the affine transformation matrix .

[0037] wherein, is a non-distortion face image, is an affine transformation matrix,​​​ For the first Personal face bounding box image, T(·) is an image geometric transformation function (derivable, supporting back propagation) with bilinear interpolation as the sampling kernel.

[0038] In a possible implementation, the multi-task feature extraction model comprises a second feature map extraction branch, and a two-level detection head connected in parallel with the second feature map extraction branch; wherein the two-level detection head is a global visual feature extraction branch and a multi-attribute feature extraction branch; The face feature extraction module 130 is configured to: after the second shared semantic feature map corresponding to each face bounding box is obtained by performing feature extraction on the non-distortion face image corresponding to each face bounding box through the second feature map extraction branch in the multi-task feature extraction model, the global face visual feature vector and the multi-dimensional face attribute feature vector corresponding to each face bounding box are obtained by performing global visual feature extraction and multi-attribute feature extraction on the second shared semantic feature map corresponding to each face bounding box in parallel through the global visual feature extraction branch and the multi-attribute feature extraction branch in the multi-task feature extraction model; and the face feature vector corresponding to each face bounding box is obtained by performing post-processing on the global face visual feature vector and the multi-dimensional face attribute feature vector corresponding to each face bounding box.

[0039] In the embodiment of the present application, the target of the multi-task feature extraction model is to obtain the global face visual feature vector and the multi-dimensional face attribute feature vector corresponding to each face bounding box, which specifically comprises: After the second shared semantic feature map is obtained by performing feature extraction on the non-distortion face image corresponding to the i-th face bounding box through the second feature map extraction branch in the multi-task feature extraction model, the global face visual feature vector is extracted based on the second shared semantic feature map through the global visual feature extraction branch in the multi-task feature extraction model , ) represents the global visual feature extraction branch, is the second shared semantic feature map, is the feature dimension, is the global face visual feature vector of the i-th face bounding box, and the multi-dimensional face attribute feature vector is extracted in parallel based on the second shared semantic feature map through the multi-attribute feature extraction branch in the multi-task feature extraction model , is the age, is the gender, is the skin color, is the expression, is the eye state, is the mouth state, is the head pose, ​whether wearing glasses, whether wearing a mask. Multi-dimensional face attribute feature prediction is realized through classification and regression models. For example, age is realized through a regression function: ; binary or multi-class attributes such as gender, expression, whether wearing a mask, and whether wearing glasses are realized through a cross-entropy loss function: ; the final multi-task joint loss function is: , is the global face visual feature loss, is the jth face attribute feature loss, and is a dynamic weight that can be automatically learned through Uncertainty Weighting or GradNorm. Through the multi-task feature extraction model, comprehensive extraction of face features and attributes can be realized, ensuring that not only can strangers be recognized in a dynamic public environment, but also rich attribute information can be obtained.

[0040] In one possible implementation, the face identity determination module 140 is configured to calculate the Euclidean distance between the face feature vector of each face in each face bounding box to be recognized and the face feature vector of each temporary reference face dynamically stored in the local lightweight face database based on a Euclidean distance-based similarity calculation method; for each face in each face bounding box to be recognized, if the Euclidean distance between the face feature vector of the face to be recognized and the face feature vector of any temporary reference face dynamically stored in the local lightweight face database is less than the distance threshold, it is determined that the face to be recognized and any temporary reference face are the same face, and if the Euclidean distance between the face feature vector of the face to be recognized and the face feature vector of all temporary reference faces dynamically stored in the local lightweight face database is greater than or equal to the distance threshold, it is determined that the face to be recognized is a stranger.

[0041] In the embodiments of the present application, the local lightweight face database is stored in a structured manner, and each record contains a unique face number (i.e., a unique temporary identity), a global face visual feature vector, a multi-dimensional face attribute feature vector, and an entry time. As shown in Table 1, a local lightweight face database containing 12 fields is initialized, wherein the unique face number id is the unique index key, which is used to realize short-term, anonymous, and collision-free real-time interaction on the side of the biped robot, for example, when the same person enters the field of view multiple times within 5 minutes and is recognized, the same person is identified as the same person through id consistency, so that the human-machine interaction can be skipped, and repeated operations such as repeated greetings and repeated reminders to wear a mask can be avoided. Moreover, only the unique face number id is generated as the unique temporary identity, and no real identity information is involved, thereby reducing the risk of privacy leakage from the source. For ease of display, Table 1 only takes id=1 and id=2 as two example data.

[0042] Table 1.

[0043] wherein, a global face visual feature vector for recording a temporary reference face, an entry time for recording the temporary reference face, and other fields for recording multi-dimensional face attribute feature vectors of the temporary reference face, such as age, gender, skin color, expression, eye state, mouth state, head posture, whether wearing glasses, whether wearing a mask, and the like, so as to facilitate subsequent stranger face determination, short-time data management, and periodic cleaning.

[0044] In the embodiment of the present application, the face identity determination module 140 compares the face feature vector of the to-be-identified face in the face bounding box with the face feature vectors of each temporary reference face dynamically stored in the local lightweight face database, so as to realize stranger determination and identity matching. In a specific implementation, the feature storage, retrieval, and similarity calculation are realized based on SQL technology, and specifically include: comparing the face feature vector of the to-be-identified face in the i-th face bounding box with the face feature vectors of all temporary reference faces in the local lightweight face database, calculating the Euclidean distance one by one:

[0045] wherein, is the Euclidean distance, is the face feature vector of the to-be-identified face in the i-th face bounding box, is the face feature vector of the j-th temporary reference face in the local lightweight face database, and k is a feature dimension index, k=1, 2, 3, …, d (d=128 or 256).

[0046] If the Euclidean distance between the face feature vector of the j-th temporary reference face and the face feature vector of the to-be-identified face in the i-th face bounding box satisfies: then it is determined that the to-be-identified face in the i-th face bounding box and the temporary reference face with the unique face number (i.e., the unique temporary identity) of in the local lightweight face database are the same identity; if all Euclidean distances are greater than the distance threshold θ, it is determined that the to-be-identified face in the i-th face bounding box is a stranger face.

[0047] In one possible implementation, the stranger face recognition system 100 provided by the embodiment of the present application further includes: ​​​The data management module 150, connected to the face identity determination module 140, is used to, for each face bounding box, if the face to be identified is the same as any temporary reference face in the local lightweight face database, update the entry time of any temporary reference face to the current time; if the face to be identified is an unfamiliar face, assign a unique temporary identity to the unfamiliar face, store the unfamiliar face as a temporary reference face along with its face feature vector and unique temporary identity in the local lightweight face database, and use the current time as the entry time; and automatically delete the unique temporary identity and face feature vector of temporary reference faces in the local lightweight face database whose entry time exceeds the storage time threshold.

[0048] In this embodiment, if the face to be identified successfully matches any temporary reference face in the local lightweight face database, the recording time of that temporary reference face is determined based on its unique face number (i.e., unique temporary identifier). Updated to current time This process ensures that the local lightweight face database always reflects the most recent recognition event, thus enabling time-based dynamic management. Specifically, the data entry time update operation is represented as:

[0049] in, Indicates the entry time. The unique face number (i.e., unique temporary identity) represents any temporary reference face. This indicates the current time (system time).

[0050] For example: in a local lightweight face database The temporary baseline face was first entered on 2025-09-04 at 10:24:55. When the temporary baseline face is detected and recognized again on 2025-09-04 at 10:30:10, only record_time=2025-09-04 10:30:10 is updated, without adding a new temporary baseline face. Thus, by updating the entry time, the consistency and timeliness of the local lightweight face database can be maintained, providing a reliable basis for subsequent short-term management, activity statistics or trajectory tracking, while also reducing storage and computing pressure.

[0051] In this embodiment, when the face to be identified fails to match any of the temporary reference faces in the local lightweight face database, that is, when the Euclidean distance between the face to be identified and any of the temporary reference faces in the local lightweight face database is greater than a distance threshold... At this time, it is determined that the face to be identified is a stranger's face, and a unique face number is further assigned to the stranger's face as a unique temporary identity identifier, wherein the unique face number is generated in an order number increment manner, and when the unique face number reaches an upper limit (for example, 999), it is recycled from 0 and numbered through a slot detection. For example, if the recording time of the temporary reference face with the unique face number 0 is still within the active period of the storage duration threshold (such as 5 minutes), the number 0 is skipped and the number is increased until a unique face number that is not within the active period of the storage duration threshold (such as 5 minutes) and has been automatically deleted is found as the unique temporary identity identifier of the newly recorded stranger's face. Further, the stranger's face and its face feature vector, recording time and unique temporary identity identifier are stored in the local lightweight face database, wherein the storage of the stranger's face data includes: global face visual feature vector, multi-dimensional face attribute feature vector (such as age, gender, skin color, expression, eye state, mouth state, head posture, whether wearing glasses, whether wearing a mask, etc.), unique temporary identity identifier and recording time. For example, a stranger's face is detected at 2025-09-04 11:02:33, and a unique temporary identity identifier id=3 is assigned, and a temporary reference face and its related data are added to the local lightweight face database as shown in Table 2: Table 2.

[0052] In this way, by introducing the timing deletion and slot detection increment mechanism, the number of temporary reference faces in the local lightweight face database can be kept within a certain number range (such as 1000) at all times, and the numbering cycle will not cause conflicts, so that the local lightweight face database always only saves active and recently detected stranger's face data as temporary reference face data, thereby realizing real-time recording and effective management of temporary reference face data under the condition of limited storage space, and ensuring space controllability and high real-time performance.

[0053] In the embodiment of the application, the timing deletion strategy based on the recording time is to directly complete the expired data filtering and cleaning at the database level through SQL query, thereby significantly reducing the system burden. Specifically, the following operations are periodically performed, a storage duration threshold is set, and any temporary reference face and its global face visual feature vector, multi-dimensional face attribute feature vector and unique temporary identity identifier data whose storage time exceeds the storage duration threshold are determined to be expired and automatically cleaned up; specifically, the expiration determination condition is represented as: record_time(id)< τ wherein record_time(id) represents the recording time, A unique face number (i.e., a unique temporary identity) representing a temporary reference face. τ represents the current time, and τ represents the storage duration threshold.

[0054] For example, if the current time is If the record time is earlier than 2025-09-04 11:10:00, then all temporary reference faces, their global facial visual feature vectors, multi-dimensional facial attribute feature vectors, and unique temporary identity data that are earlier than 2025-09-04 11:05:00 will be deleted all at once, thus avoiding the redundant overhead of calculating the time difference one by one and improving the running efficiency.

[0055] In one possible implementation, the data management module 150 is also connected to the face feature extraction module 130, and is used to count the number of face bounding boxes in the set of face bounding boxes corresponding to the target environment image identified by the multi-task face recognition model, and count the number of faces to be recognized; based on the crowd density coefficients corresponding to different crowd density intervals, the crowd density coefficient corresponding to the crowd density interval in which the number of faces to be recognized is located is determined as the current crowd density coefficient, wherein the higher the crowd density interval, the larger the crowd density coefficient; and the storage duration threshold is adjusted based on the current crowd density coefficient; wherein the storage duration threshold τ_base is adjusted to τ_dynamic=τ_base×ρ, where ρ is the current crowd density coefficient.

[0056] For example: Define the crowd density coefficient ρ. Low crowd density range: Number of faces to be recognized < first density threshold (e.g., 5), corresponding to the first value of the crowd density coefficient (e.g., ρ=0.8); Medium crowd density range: First density threshold (e.g., 5) ≤ Number of faces to be recognized ≤ Second density threshold (e.g., 15), corresponding to the second value of the crowd density coefficient (e.g., ρ=1.0); High crowd density range: Number of faces to be recognized > Second density threshold (e.g., 15), corresponding to the second value of the crowd density coefficient (e.g., ρ=1.2). If τ_base is the storage time threshold = 5 min, then: Low crowd density range: τ_dynamic = 4 min (shorten storage time, reduce redundancy); Medium crowd density range: τ_dynamic = 5 min (maintain the original baseline, balance efficiency); High crowd density range: τ_dynamic = 6 min (extend storage time, avoid premature deletion of effective interactive data, such as data being cleaned up before the robot completes the guidance service).

[0057] In a possible implementation, the data management module 150 is configured to, when performing the timing deletion, call the real-time calculated τ_dynamic, and update the expiration determination condition to record_time(id) < T_now-τ_dynamic; meanwhile, when the crowd density switches from high to medium to low, automatically trigger a redundant data screening once to delete the expired data exceeding the current τ_dynamic.

[0058] Based on the above embodiments, the application provides a stranger face recognition method, which is applied to a legged humanoid robot, as shown in Figure 2 The general process of the stranger face recognition method provided by the application is as follows: Step 200: Collecting an environment video stream in a public environment.

[0059] Step 201: Using a lightweight quality repair model to perform quality repair on an environment image in the environment video stream to obtain a target environment image.

[0060] Step 202: Using a multi-task face recognition model to perform face bounding box prediction and face key point extraction on the target environment image to obtain a face bounding box set and a face key point set in each face bounding box in the face bounding box set.

[0061] Step 203: Based on a standard template key point set, respectively aligning each face key point set in each face bounding box to a standard template face image to obtain a non-distorted face image corresponding to each face bounding box.

[0062] Step 204: Using a multi-task feature extraction model to respectively extract a global face visual feature vector and a multi-dimensional face attribute feature vector of a to-be-recognized face from each non-distorted face image corresponding to each face bounding box as a face feature vector.

[0063] Step 205: Respective calculation of the similarity between the face feature vector of the to-be-recognized face in each face bounding box and the face feature vector of each temporary reference face dynamically stored in the local lightweight face database.

[0064] Step 206: Determining whether the to-be-recognized face in each face bounding box is a stranger face based on the similarity; wherein the face feature vector of each temporary reference face dynamically stored in the local lightweight face database is the face feature vector of each stranger face recognized within a set time range.

[0065] Step 207: For each face bounding box, if the face to be identified is determined to be a stranger face, a unique temporary identity is assigned to the stranger face, the stranger face is stored as a temporary reference face together with the face feature vector and the unique temporary identity in the local lightweight face database, and the current time is recorded as the entry time; if the face to be identified is determined to be the same person as any temporary reference face in the local lightweight face database, the entry time of any temporary reference face is updated to the current time.

[0066] Step 208: The unique temporary identity and face feature vector of the temporary reference face in the local lightweight face database whose entry duration exceeds the storage duration threshold are automatically deleted.

[0067] In the embodiments of the present application, by deploying a lightweight quality repair model, a multi-task face recognition model, a multi-task feature extraction model and a local lightweight face database on the end side of the legged humanoid robot, the image quality repair of the lightweight quality repair model can be used to solve the problem of low-quality conditions such as overexposure or underexposure, blur and the like caused by complex public environment lighting and motion blur caused by the walking of the legged humanoid robot, ensuring the reliability of subsequent face recognition. In addition, the face boundary box recognition of the multi-task face recognition model, the face feature extraction of the multi-task feature extraction model and the stranger face discrimination of the local lightweight face database can be used to realize stranger face recognition on the end side of the legged humanoid robot, avoid network delay introduced by cloud comparison, and thus realize real-time interactive response in a public environment, improve the understanding and service ability of the legged humanoid robot for interactive objects, and significantly improve the human-machine interaction experience in public places.

[0068] After introducing the stranger face recognition system and method provided by the embodiments of the present application, the legged humanoid robot provided by the embodiments of the present application is briefly introduced.

[0069] Referring to FIG. 3, Figure 3 The legged humanoid robot 300 provided by the embodiments of the present application at least includes a processor 301, a memory 302 and a computer program stored in the memory 302 and executable on the processor 301, and the processor 301 implements the stranger face recognition method provided by the embodiments of the present application when executing the computer program.

[0070] The legged humanoid robot 300 provided by the embodiments of the present application can further include a bus 303 connecting different components (including the processor 301 and the memory 302). Among them, the bus 303 represents one or more of several types of bus structures, including a memory bus, a peripheral bus, a local bus, etc.

[0071] The memory 302 can include a readable storage medium in the form of volatile memory, such as a random access memory (RAM) 3021 and / or cache memory 3022, and can further include a read-only memory (ROM) 3023. The memory 302 can also include program tools 3025 having a set of (at least one) program modules 3024, including but not limited to operating systems, one or more applications, other program modules, and program data, each of which can include implementation of a network environment, or some combination thereof.

[0072] The processor 301 can be one processing element or a collective term for a plurality of processing elements, for example, the processor 301 can be a central processing unit (CPU), or one or more integrated circuits configured to implement the stranger face recognition method provided by the embodiments of the present application. Specifically, the processor 301 can be a general-purpose processor, including but not limited to a CPU, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0073] The foot-shaped humanoid robot 300 can communicate with one or more external devices 304 (such as a keyboard, a remote controller, etc.), and can also communicate with one or more devices that enable a user to interact with the foot-shaped humanoid robot 300 (such as a mobile phone, a computer, etc.), and / or communicate with a device that enables the foot-shaped humanoid robot 300 to communicate with one or more other foot-shaped humanoid robots 300 (such as a router, a modem, etc.). Such communication can be carried out through an input / output (I / O) interface 305. In addition, the foot-shaped humanoid robot 300 can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN) and / or a public network such as the Internet) through a network adapter 306. As Figure 3 shown, the network adapter 306 communicates with other modules of the foot-shaped humanoid robot 300 through the bus 303. It should be understood that although Figure 3Other hardware and / or software modules can be used in conjunction with the bipedal humanoid robot 300, as shown, including, but not limited to, microcode, device drivers, redundant processors, external disk drive arrays, Redundant Arrays of Independent Disks (RAID) subsystems, tape drives, and data backup storage subsystems, etc.

[0074] It should be noted that, Figure 3 The bipedal humanoid robot 300 shown is merely an example and should not limit the function and use range of the embodiments of the present application.

[0075] The computer readable storage medium provided by the embodiments of the present application is introduced as follows. The computer readable storage medium provided by the embodiments of the present application stores computer instructions, which are executed by a processor to implement the stranger face recognition method provided by the embodiments of the present application. Specifically, the computer instructions can be built-in or installed in the processor, so that the processor can implement the stranger face recognition method provided by the embodiments of the present application by executing the built-in or installed computer instructions.

[0076] In addition, the stranger face recognition method provided by the embodiments of the present application can also be implemented as a computer program product, which includes program codes that implement the stranger face recognition method provided by the embodiments of the present application when running on a processor.

[0077] The computer program product provided by the embodiments of the present application can adopt one or more computer readable storage media, and the computer readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any suitable combination of the above. Specifically, more specific examples (non-exhaustive list) of the computer readable storage medium include an electrical connection with one or more conductive wires, a portable disk, a hard disk, RAM, ROM, Erasable Programmable Read Only Memory (EPROM), an optical fiber, a portable Compact Disc Read-Only Memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0078] The computer program product provided by the embodiments of the present application can adopt a CD-ROM and include program codes, and can also run on a bipedal humanoid robot. However, the computer program product provided by the embodiments of the present application is not limited to this. In the embodiments of the present application, the computer readable storage medium can be any tangible medium containing or storing program codes, which can be used or combined with an instruction execution system, device or component.

[0079] It should be noted that, although several units or sub-units of the apparatus are mentioned in the above detailed description, such a division is merely exemplary and not mandatory. Indeed, according to an embodiment of the application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into units embodied by several units.

[0080] Moreover, although the operations of the method(s) herein may be described in a particular, sequential order, this order is not meant to be a limitation and is not intended to imply that

[0081] Although the preferred embodiments of the application have been described, those skilled in the art will recognize that many modifications and variations of the preferred embodiments could be made without departing from the spirit and scope of the application. It is, therefore, desired to be protected in the appended claims the preferred embodiments as well as all such modifications and variations as fall within the scope of the application.

[0082] It will be apparent to those skilled in the art that various modifications and variations can be made to the present application without departing from the spirit or scope of the application. Thus, it is intended that the present application cover the modifications and variations of this application provided they come within the scope of the appended claims and their equivalents.

Claims

1. A stranger facial recognition system, characterized in that, Deployed on a legged humanoid robot, it includes a video stream acquisition module, an image quality optimization module, a face feature extraction module, and a face identity determination module; The video stream acquisition module is integrated into the body of the legged humanoid robot and is used to acquire environmental video streams in public environments. The image quality optimization module is connected to the video stream acquisition module and is used to perform quality restoration on the environmental image in the environmental video stream using a lightweight quality restoration model to obtain the target environmental image. The face feature extraction module is connected to the image quality optimization module and is used to use a multi-task face recognition model to predict face bounding boxes and extract face key points from the target environment image to obtain a set of face bounding boxes and a set of face key points within each face bounding box in the set of face bounding boxes. Based on the standard template key point set, the set of facial key points within each facial bounding box is aligned to the standard template facial image to obtain the undeformed facial image corresponding to each facial bounding box; a multi-task feature extraction model is used to extract the global facial visual feature vector and multi-dimensional facial attribute feature vector of the face to be identified from the undeformed facial image corresponding to each facial bounding box as facial feature vector. The face identity determination module, connected to the face feature extraction module, is used to calculate the similarity between the face feature vector of the face to be identified in each face bounding box and the face feature vector of each temporary reference face dynamically stored in the local lightweight face database, and to determine whether the face to be identified in each face bounding box is an unfamiliar face based on the similarity; wherein, the face feature vector of each temporary reference face dynamically stored in the local lightweight face database is the face feature vector of each unfamiliar face identified within the most recent set time range.

2. The stranger facial recognition system as described in claim 1, characterized in that, The image quality optimization module is used to calculate the image quality score of each frame of the environmental image in the environmental video stream using an image quality evaluation function; when the image quality score is less than a first quality threshold, the environmental image is discarded; when the image quality score is greater than or equal to the first quality threshold and less than a second quality threshold, the lightweight quality repair model is used to repair the motion blur, image noise, and lighting abnormalities of the environmental image to obtain the target environmental image; when the image quality score is greater than or equal to the second quality threshold, the environmental image is determined as the target environmental image.

3. The stranger facial recognition system as described in claim 1, characterized in that, The multi-task face recognition model includes a first feature map extraction branch and a three-level detection head connected in parallel with the first feature map extraction branch; wherein, the three-level detection head is a face binary classification detection branch, a face bounding box regression branch, and a face key point regression branch; The face feature extraction module is used to extract features from the target environment image through the first feature map extraction branch in the multi-task face recognition model to obtain a first shared semantic feature map. Then, based on the first shared semantic feature map, it performs face binary classification detection, face bounding box detection, and face key point detection in parallel through the face binary classification detection branch, the face bounding box regression branch, and the face key point regression branch in the multi-task face recognition model to obtain face binary classification detection results, face bounding box detection results, and face key point detection results. The module then performs post-processing on the face binary classification detection results, the face bounding box detection results, and the face key point detection results to obtain a set of face bounding boxes corresponding to the target environment image and a set of face key points within each face bounding box in the set of face bounding boxes.

4. The stranger facial recognition system as described in claim 1, characterized in that, The face feature extraction module is used to calculate the affine transformation matrix between the set of facial key points within the face bounding box and the set of standard template key points for each face bounding box, and then align the set of facial key points within the face bounding box to the calibration template face image based on the affine transformation matrix to obtain the undeformed face image corresponding to the face bounding box.

5. The stranger facial recognition system as described in claim 1, characterized in that, The multi-task feature extraction model includes a second feature map extraction branch and a secondary detection head connected in parallel with the second feature map extraction branch; wherein, the secondary detection head is a global visual feature extraction branch and a multi-attribute feature extraction branch; The face feature extraction module is used to extract features from the undeformed face image corresponding to each face bounding box through the second feature map extraction branch in the multi-task feature extraction model to obtain a second shared semantic feature map. Then, through the global visual feature extraction branch and the multi-attribute feature extraction branch in the multi-task feature extraction model, it performs global visual feature extraction and multi-attribute feature extraction in parallel based on the second shared semantic feature map corresponding to each face bounding box to obtain a global face visual feature vector and a multi-dimensional face attribute feature vector corresponding to each face bounding box. Finally, it performs post-processing on the global face visual feature vector and the multi-dimensional face attribute feature vector corresponding to each face bounding box to obtain a face feature vector corresponding to each face bounding box.

6. The stranger facial recognition system as described in claim 1, characterized in that, The face identification module is used to calculate the Euclidean distance between the face feature vector of the face to be identified in each face bounding box and the face feature vectors of each temporary reference face dynamically stored in the local lightweight face database, using a similarity calculation method based on Euclidean distance. For each face to be identified in the face bounding box, if the Euclidean distance between the face feature vector of the face to be identified and the face feature vector of any temporary reference face dynamically stored in the local lightweight face database is less than a distance threshold, then the face to be identified and any temporary reference face are determined to be the same face. If the Euclidean distance between the face feature vector of the face to be identified and the face feature vectors of all temporary reference faces dynamically stored in the local lightweight face database is greater than or equal to the distance threshold, then the face to be identified is determined to be a stranger.

7. The stranger face recognition system as described in any one of claims 1-5, characterized in that, Also includes: The data management module, connected to the face identity determination module, is used to, for each face to be identified within the face bounding box, if the face to be identified is the same as any temporary reference face in the local lightweight face database, update the entry time of any temporary reference face to the current time; if the face to be identified is an unfamiliar face, assign a unique temporary identity to the unfamiliar face, store the unfamiliar face as a temporary reference face along with its face feature vector and unique temporary identity in the local lightweight face database, and use the current time as the entry time; automatically delete the unique temporary identity and face feature vector of temporary reference faces in the local lightweight face database whose entry time exceeds the storage time threshold.

8. A method for recognizing faces of strangers, characterized in that, Applications in legged humanoid robots include: Acquire environmental video streams in public environments; A lightweight quality restoration model is used to restore the quality of environmental images in the environmental video stream to obtain the target environmental image. A multi-task face recognition model is used to predict face bounding boxes and extract face key points from the target environment image to obtain a set of face bounding boxes and a set of face key points within each face bounding box in the set of face bounding boxes. Based on the standard template key point set, the set of facial key points within each facial bounding box is aligned to the standard template facial image to obtain the undeformed facial image corresponding to each facial bounding box; A multi-task feature extraction model is adopted to extract the global face visual feature vector and the multi-dimensional face attribute feature vector of the face to be identified from the undeformed face image corresponding to each face bounding box as the face feature vector. Calculate the similarity between the facial feature vector of the face to be identified in each of the facial bounding boxes and the facial feature vectors of each temporary reference face dynamically stored in the local lightweight face database; Based on the similarity, it is determined whether the face to be identified in each face bounding box is a stranger face; wherein, the face feature vectors of each temporary reference face dynamically stored in the local lightweight face database are the face feature vectors of each stranger face identified within the most recent set time range.

9. A legged humanoid robot, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the stranger face recognition method as described in claim 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the stranger face recognition method as described in claim 8.

Images