User identification method for mobile camera robot and storage medium

Abstract

The application discloses a user identification method of a mobile camera robot and a storage medium, and the method comprises the following steps: the mobile camera robot collects a video stream and an audio stream in real time; video frames are obtained from the video stream periodically to perform face detection and body posture identification, and voice activity detection is performed on the audio stream; if a face is detected, a face feature vector is extracted, and a face candidate identity ID is determined; if the identified posture key points meet the full-body or half-body ReID gate condition, a full-body or half-body ReID feature vector is extracted, and a ReID candidate identity ID is determined; if an effective voice segment is detected, a voiceprint feature vector is extracted, and a voiceprint candidate identity ID is determined; and according to a preset priority order, a final identity ID is determined from the face candidate identity ID, the ReID candidate identity ID and the voiceprint candidate identity ID. The application can realize user identity identification when low-view shooting, the user's front face cannot be seen or the user does not speak actively.

Description

Technical Field

[0001] This invention relates to the field of identity recognition technology, and in particular to a user recognition method and storage medium for a mobile camera robot. Background Technology

[0002] Existing home companion robots or home patrol security robots typically employ a single facial recognition solution, or a simple combination of facial recognition and voiceprint recognition. The general process is as follows: a camera captures an image of the user's face, the system extracts facial features and compares them with a pre-registered facial database; or, upon detecting speech, the system extracts voiceprint features and compares them with a voiceprint database.

[0003] For traditional desktop or fixed cameras, the above solutions work when the user is facing the camera and actively speaking. However, in low-angle mobile camera robots, the camera is close to the ground, and the robot often captures the user's body, side profile, lower face, back view, or part of the torso at an upward angle when moving; users in home settings also typically do not actively speak to the robot. Therefore, relying solely on facial recognition or a loose combination of face and voiceprint can easily lead to problems such as invisible faces, unregistered voiceprints, uncertain voiceprint sources, and the inability to continuously replenish the feature database.

[0004] Person re-identification (ReID) can perform identity retrieval based on human appearance when the face is not visible. However, directly writing arbitrary human images into the identity database can easily contaminate the feature database due to occlusion, incomplete human figures, changes in perspective, or cropping errors. Voiceprint recognition can supplement identity information when it is difficult to see the face, but in a home environment, there may be interference such as television sounds, other family members talking, or distant voices. If the binding conditions between voiceprint segments and the current facial identity are not limited, incorrect binding may also occur. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a user identification method and storage medium for a mobile camera robot, which can realize user identification when shooting from a low angle, when the user's face is not visible, or when the user does not speak.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a user recognition method for a mobile camera robot, comprising: A mobile camera robot collects environmental data in real time, including video and audio streams. According to a preset processing cycle, video frames are obtained from the video stream, face detection and body pose recognition are performed on the video frames, and voice activity detection is performed on the audio stream. If a face is detected, the face feature vector is extracted, and the candidate face identity ID is determined based on the face feature vector and the face feature set corresponding to each existing identity ID in the local identity feature library. If the identified posture key points and their confidence levels meet the full-body pedestrian re-identification gating conditions or the half-body pedestrian re-identification gating conditions, then the full-body pedestrian re-identification feature vector or the half-body pedestrian re-identification feature vector is extracted, and based on the full-body pedestrian re-identification feature vector or the half-body pedestrian re-identification feature vector, as well as the full-body feature set or half-body feature set corresponding to each existing identity ID in the local identity feature database, the pedestrian re-identification candidate identity ID is determined. If a valid voice segment with a continuous duration longer than a preset duration is detected, the voiceprint feature vector of the valid voice segment is extracted, and a voiceprint candidate identity ID is determined based on the voiceprint feature vector and the voiceprint feature set corresponding to each existing identity ID in the local identity feature library. The final identity ID is determined from the candidate face identity ID, candidate pedestrian re-identification identity ID, and candidate voiceprint identity ID according to a preset priority order.

[0007] The present invention also proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described above.

[0008] The beneficial effects of this invention are as follows: By maintaining full-body ReID features and half-body ReID features separately, and treating lower-body features as part of the full-body features, it avoids simply discarding lower limb information and prevents the inclusion of independent lower-body features without stable identity binding criteria into the identity database. This reduces the risks of misbinding, misupdating, and misidentification in low-view, non-cooperative family scenarios. Priority-based decision-making adapts the system to low-view scenarios, with face candidate identity IDs having the highest priority, followed by pedestrian re-identification candidate identity IDs, and voiceprint candidate identity IDs having the lowest priority. This rule allows the robot to identify existing family members using ReID or voiceprint features even when it cannot see the user's face or when the user does not speak. Attached Figure Description

[0009] Figure 1 This is a flowchart of a user identification method for a mobile camera robot according to the present invention; Figure 2 This is a flowchart of the user identification method according to Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of the single-person face registration process according to Embodiment 1 of the present invention; Figure 4 This is a schematic diagram of the ReID feature recognition process according to Embodiment 1 of the present invention. Detailed Implementation

[0010] To explain in detail the technical content, objectives, and effects of the present invention, the following description is provided in conjunction with the embodiments and accompanying drawings.

[0011] Please refer to Figure 1 A user identification method for a mobile camera robot, comprising: A mobile camera robot collects environmental data in real time, including video streams collected by a wide-angle camera and audio streams collected by a microphone array. According to a preset processing cycle, video frames are obtained from the video stream, face detection and body pose recognition are performed on the video frames, and voice activity detection is performed on the audio stream. If a face is detected, the face feature vector is extracted, and the candidate face identity ID is determined based on the face feature vector and the face feature set corresponding to each existing identity ID in the local identity feature library. If the identified posture key points and their confidence levels meet the full-body pedestrian re-identification gating conditions or the half-body pedestrian re-identification gating conditions, then the full-body pedestrian re-identification feature vector or the half-body pedestrian re-identification feature vector is extracted, and based on the full-body pedestrian re-identification feature vector or the half-body pedestrian re-identification feature vector, as well as the full-body feature set or half-body feature set corresponding to each existing identity ID in the local identity feature database, the pedestrian re-identification candidate identity ID is determined. If a valid voice segment with a continuous duration longer than a preset duration is detected, the voiceprint feature vector of the valid voice segment is extracted, and a voiceprint candidate identity ID is determined based on the voiceprint feature vector and the voiceprint feature set corresponding to each existing identity ID in the local identity feature library. The final identity ID is determined from the candidate face identity ID, candidate pedestrian re-identification identity ID, and candidate voiceprint identity ID according to a preset priority order.

[0012] As can be seen from the above description, the beneficial effect of the present invention is that it can achieve user identification when shooting from a low angle, when the user's face is not visible, or when the user does not speak.

[0013] Further, determining the candidate identity ID based on the face feature vector and the set of face features corresponding to each existing identity ID in the local identity feature database includes: Calculate the similarity between the face feature vector and each face feature vector in the face feature set corresponding to each existing identity ID in the local identity feature database, and determine the maximum similarity between the face feature vector and each existing identity ID; If the maximum similarity between the facial feature vector and an existing identity ID is greater than a preset first recognition threshold, then the existing identity ID is used as a candidate facial identity ID.

[0014] Further, if the identified posture key points and their confidence levels satisfy the full-body pedestrian re-identification gating condition or the half-body pedestrian re-identification gating condition, then the full-body pedestrian re-identification feature vector or the half-body pedestrian re-identification feature vector is extracted, including: Obtain the pose key points obtained through body pose recognition and the confidence level of each pose key point; Determine whether the full-body pedestrian re-identification gating condition is met. The full-body pedestrian re-identification gating condition is that the acquired posture key points include full-body key points and the confidence of each full-body key point is greater than or equal to a preset first confidence threshold. The full-body key points include left shoulder, right shoulder, left hip, right hip, left knee, right knee, left ankle, and right ankle. If the conditions are met, then based on the aforementioned key points throughout the body, extract the full-body pedestrian re-identification feature vector; If not satisfied, then determine whether the half-body pedestrian re-identification gating condition is satisfied. The half-body pedestrian re-identification gating condition is that the acquired posture key points include half-body key points and the confidence of each half-body key point is greater than or equal to the preset first confidence threshold. The half-body key points include the left shoulder, right shoulder, left hip and right hip. If the conditions are met, then based on the aforementioned half-body key points, extract the half-body pedestrian re-identification feature vector.

[0015] As described above, low-angle mobile robots often only capture partial images of the human body. By requiring that the half-body have at least two shoulder and two hip keypoints, and the whole body have at least two shoulder, two hip, two knee, and two ankle keypoints, and requiring that the confidence of the keypoints be no less than a preset threshold, images with severe occlusion, incomplete human body cropping, or unreliable pose estimation can be filtered out, thereby reducing the contamination of the ReID library by low-quality human samples.

[0016] Further, the step of extracting the full-body pedestrian re-identification feature vector based on the full-body key points includes: The whole body region is obtained based on the bounding rectangle of the key points of the whole body and the preset first expansion height and first expansion width, and the whole body pedestrian re-identification feature vector is extracted based on the whole body region; the first expansion height and the first expansion width are determined according to the height and width of the bounding rectangle of the key points of the whole body, respectively. The step of extracting a half-body pedestrian re-identification feature vector based on the half-body key points includes: Based on the bounding rectangle of the half-body key points and the preset second external height and second external width, the half-body region is obtained, and the half-body pedestrian re-identification feature vector is extracted based on the half-body region. The second external height and second external width are determined according to the height and width of the bounding rectangle of the half-body key points, respectively.

[0017] As described above, the ReID features can be fully extracted based on the bounding rectangle of the key points in the cropping region that meets the conditions.

[0018] Further, the step of determining candidate identity IDs for pedestrian re-identification based on the full-body pedestrian re-identification feature vector or half-body pedestrian re-identification feature vector, and the full-body feature set or half-body feature set corresponding to each existing identity ID in the local identity feature database, includes: Calculate the similarity between the full-body pedestrian re-identification feature vector and each full-body pedestrian re-identification feature vector in the full-body feature set corresponding to each existing identity ID in the local identity feature database, and determine the maximum similarity between the full-body pedestrian re-identification feature vector and each existing identity ID; Alternatively, calculate the similarity between the half-body pedestrian re-identification feature vector and each half-body pedestrian re-identification feature vector in the half-body feature set corresponding to each existing identity ID in the local identity feature database, and determine the maximum similarity between the half-body pedestrian re-identification feature vector and each existing identity ID. If the maximum similarity between the full-body pedestrian re-identification feature vector and an existing identity ID is greater than a preset second identification threshold, or if the maximum similarity between the half-body pedestrian re-identification feature vector and an existing identity ID is greater than a preset second identification threshold, then the existing identity ID will be used as a candidate identity ID for pedestrian re-identification.

[0019] Further, determining the candidate identity ID based on the voiceprint feature vector and the voiceprint feature set corresponding to each existing identity ID in the local identity feature database includes: Calculate the similarity between the voiceprint feature vector and each voiceprint feature vector in the voiceprint feature set corresponding to each existing identity ID in the local identity feature database, and determine the maximum similarity between the voiceprint feature vector and each existing identity ID. If the maximum similarity between the voiceprint feature vector and an existing identity ID is greater than a preset third recognition threshold, then the existing identity ID will be used as a voiceprint candidate identity ID.

[0020] Further, determining the final identity ID from the candidate face IDs, candidate pedestrian re-identification IDs, and candidate voiceprint IDs according to a preset priority order includes: If the aforementioned candidate face ID exists, then the candidate face ID shall be used as the final identity ID; If the face candidate identity ID does not exist but the pedestrian re-identification candidate identity ID exists, then the pedestrian re-identification candidate identity ID shall be used as the final identity ID. If the face candidate identity ID and pedestrian re-identification candidate identity ID do not exist, but the voiceprint candidate identity ID does exist, then the voiceprint candidate identity ID will be used as the final identity ID.

[0021] As described above, when a face is present, the candidate face ID is given the highest priority. When a face is unavailable, ReID features are used to supplement the identity. Only when there is no face and no ReID candidate, and the VAD quality and sound source direction are reliable, can low-confidence voiceprint candidates be output. This rule allows the robot to identify existing family members using ReID or voiceprint features even when it cannot see the user's face or the user does not speak, while avoiding triggering registration or updates solely based on voiceprint feature recognition results.

[0022] Furthermore, the step of using the candidate face ID as the final identity ID if a candidate face ID exists further includes: Update the set of facial features corresponding to the final identity ID based on the facial feature vector; If there is a pedestrian re-identification candidate ID, and the pedestrian re-identification candidate ID is consistent with the face candidate ID, then the full-body feature set or half-body feature set corresponding to the final ID is updated according to the full-body pedestrian re-identification feature vector or the half-body pedestrian re-identification feature vector. If a candidate voiceprint identity ID exists and the candidate voiceprint identity ID is consistent with the candidate face identity ID, then the voiceprint feature set corresponding to the final identity ID is updated according to the voiceprint feature vector.

[0023] As described above, when a face is present, the candidate face ID is given the highest priority. When a face is unavailable, ReID features are used to supplement the identity. Only when there is no face and no ReID candidate, and the VAD quality and sound source direction are reliable, can low-confidence voiceprint candidates be output. This rule allows the robot to identify existing family members using ReID or voiceprint features even when it cannot see the user's face or the user does not speak, while avoiding triggering registration or updates solely based on voiceprint feature recognition results.

[0024] Further, updating the face feature set corresponding to the final identity ID based on the face feature vector includes: If the number of face feature vectors in the face feature set corresponding to the final identity ID does not reach the preset first capacity, then the face feature vectors are written into the face feature set corresponding to the final identity ID. If the number of face feature vectors in the face feature set corresponding to the final identity ID has reached the preset first capacity, then calculate the maximum similarity between the face feature vector and each face feature vector in the face feature set corresponding to the final identity ID. If the maximum similarity is greater than the preset update threshold, the face feature vector is written into the face feature set corresponding to the final identity ID, and the face feature vector written earliest in the face feature set corresponding to the final identity ID is deleted.

[0025] As described above, the high-threshold update mechanism separates "identification" and "updating," with the update threshold higher than the identification threshold. This reduces the probability of low-quality samples, samples from erroneous sources, or low-confidence samples entering the identity feature database. When the feature database is full, only the earliest written features of the same modality are replaced, maintaining a stable database capacity and allowing identity templates to be gradually updated according to changes in clothing, posture, and voice in a home setting.

[0026] Further, updating the full-body feature set or half-body feature set corresponding to the final identity ID based on the full-body pedestrian re-identification feature vector or the half-body pedestrian re-identification feature vector includes: If the number of full-body pedestrian re-identification feature vectors in the full-body feature set corresponding to the final identity ID does not reach the preset second capacity, then the full-body pedestrian re-identification feature vectors are written into the full-body feature set corresponding to the final identity ID. If the number of full-body pedestrian re-identification feature vectors in the full-body feature set corresponding to the final identity ID has reached the preset second capacity, then calculate the maximum similarity between the full-body pedestrian re-identification feature vector and each full-body pedestrian re-identification feature vector in the full-body feature set corresponding to the final identity ID. If the maximum similarity is greater than the preset update threshold, the full-body pedestrian re-identification feature vector is written into the full-body feature set corresponding to the final identity ID, and the full-body pedestrian re-identification feature vector with the earliest writing time in the full-body feature set corresponding to the final identity ID is deleted. or, If the number of half-body pedestrian re-identification feature vectors in the half-body feature set corresponding to the final identity ID does not reach the preset second capacity, then the half-body pedestrian re-identification feature vectors are written into the half-body feature set corresponding to the final identity ID. If the number of half-body pedestrian re-identification feature vectors in the half-body feature set corresponding to the final identity ID has reached the preset second capacity, then calculate the maximum similarity between the half-body pedestrian re-identification feature vector and each half-body pedestrian re-identification feature vector in the half-body feature set corresponding to the final identity ID. If the maximum similarity is greater than the preset update threshold, the half-body pedestrian re-identification feature vector is written into the half-body feature set corresponding to the final identity ID, and the half-body pedestrian re-identification feature vector with the earliest writing time in the half-body feature set corresponding to the final identity ID is deleted.

[0027] As described above, lower body information is limited to a portion of the full-body ReID features: the system extracts full-body ReID features and writes or updates the full-body ReID feature set only when keypoints for both shoulders, hips, knees, and ankles are complete and the confidence level meets the requirements; if only the keypoints for both shoulders and hips are met, half-body ReID features are extracted and compared only with the half-body ReID feature set. This approach avoids simply discarding lower limb information while preventing the inclusion of independent lower body features without stable identity binding criteria in the identity database, thereby reducing the risk of misbinding and misupdating in low-angle, non-cooperative family scenarios.

[0028] Further, updating the voiceprint feature set corresponding to the final identity ID based on the voiceprint feature vector includes: If the number of voiceprint feature vectors in the voiceprint feature set corresponding to the final identity ID does not reach the preset third capacity, then the voiceprint feature vectors are written into the voiceprint feature set corresponding to the final identity ID. If the number of voiceprint feature vectors in the voiceprint feature set corresponding to the final identity ID has reached the preset third capacity, then calculate the maximum similarity between the voiceprint feature vector and each voiceprint feature vector in the voiceprint feature set corresponding to the final identity ID. If the maximum similarity is greater than the preset update threshold, the voiceprint feature vector is written into the voiceprint feature set corresponding to the final identity ID, and the voiceprint feature vector with the earliest writing time in the voiceprint feature set corresponding to the final identity ID is deleted.

[0029] As described above, when a user only completes the initial face registration, the low-angle mobile camera robot can reliably bind the uncooperative human ReID features and voiceprint features collected during subsequent home patrols and companionship to the same face identity ID, and automatically update the feature library during the identity recognition process to avoid writing features from low-quality or erroneous sources into the identity library.

[0030] Furthermore, after the mobile camera robot collects environmental data in real time, it further includes: When a registration instruction is received, it is determined whether the single-person scene condition is met. The single-person scene condition includes that only one target person is detected within a preset video time window, and there are no other persons whose detection confidence exceeds a preset second confidence threshold and whose detection box area exceeds a preset area threshold. If the conditions are met, then n+1 video frames containing the face of the target person are continuously acquired to obtain a face image, where n is a preset first capacity; Facial landmarks are located for each face image, and facial correction is performed based on the facial landmarks. Based on each corrected face image, extract the corresponding face feature vector, and calculate the center feature based on the face feature vector; Calculate the similarity between each facial feature vector and the central feature, and take the n facial feature vectors with the highest similarity as the facial feature set of the target person; Calculate the similarity between each face feature vector in the face feature set of the target person and each face feature vector in the face feature set corresponding to each existing identity ID in the local identity feature database, and determine the maximum similarity between the target person and each existing identity ID; If the maximum similarity between the target person and an existing identity ID is greater than the preset registration deduplication threshold, then the target person is determined to have already registered. If the maximum similarity between the target person and each existing identity ID is less than or equal to the preset registration deduplication threshold, then the target person's identity ID is generated, and the target person's identity ID and facial feature set are written into the local identity feature database.

[0031] As described above, new users must complete face registration in a single-person scenario to ensure registration accuracy. By selecting the n face feature vectors with the highest similarity to the central feature from n+1 face feature vectors as the initial face feature set, it is possible to eliminate registration samples with large deviations by utilizing the consistency of continuous sampling without setting additional occlusion detection, liveness detection, or minimum face bounding box conditions.

[0032] Further, generating the target person's identity ID includes: The identity ID of the target person is generated based on the robot device identifier, registration timestamp, preset serial number and random salt value of the mobile camera robot. The preset serial number is either a locally incrementing serial number or a random serial number.

[0033] As described above, the uniqueness of the identity ID is guaranteed.

[0034] Further, if a face is detected, the step of extracting a face feature vector and determining a candidate face identity ID based on the face feature vector and the face feature set corresponding to each existing identity ID in the local identity feature database further includes: Determine whether the full-body pedestrian re-identification registration conditions are met. The full-body pedestrian re-identification registration conditions include that the full-body feature set corresponding to the face candidate identity ID is empty and meets the single-person scene condition and the full-body pedestrian re-identification gating condition. The full-body pedestrian re-identification gating condition is that the posture key points obtained through posture recognition include full-body key points and the confidence of each full-body key point is greater than or equal to a preset first confidence threshold. The full-body key points include left shoulder, right shoulder, left hip, right hip, left knee, right knee, left ankle, and right ankle. If satisfied, then based on the full-body key points, extract the full-body pedestrian re-identification feature vector and write the full-body pedestrian re-identification feature vector into the full-body feature set corresponding to the face candidate identity ID; If not, then determine whether the half-body pedestrian re-identification registration conditions are met. The half-body pedestrian re-identification registration conditions include that the half-body feature set corresponding to the face candidate identity ID is empty and meets the single-person scene condition and the half-body pedestrian re-identification gating condition. The half-body pedestrian re-identification gating condition is that the posture key points obtained through posture recognition include half-body key points and the confidence of each half-body key point is greater than or equal to the preset first confidence threshold. The half-body key points include the left shoulder, right shoulder, left hip and right hip. If the conditions are met, then based on the half-body key points, extract the half-body pedestrian re-identification feature vector, and write the half-body pedestrian re-identification feature vector into the half-body feature set corresponding to the face candidate identity ID.

[0035] As described above, ReID registration is subject to facial identity ID, single-person scenario, and full-body / half-body ReID gating conditions, which can reduce the risk of mistakenly binding other family members to the current identity ID.

[0036] Further, if a face is detected, the step of extracting a face feature vector and determining a candidate face identity ID based on the face feature vector and the face feature set corresponding to each existing identity ID in the local identity feature database further includes: The voiceprint registration window is determined based on the time of determination of the face candidate identity ID; Determine whether the voiceprint registration conditions are met. The voiceprint registration conditions include that the voiceprint feature set corresponding to the candidate face ID is empty, a valid voice segment with a continuous duration greater than a preset duration is detected in the voiceprint registration window, and the single-person scene condition and the sound source direction consistency condition are met. The sound source direction consistency condition is that the angle between the sound source direction estimated by the microphone array and the azimuth angle of the target image is less than or equal to a preset angle threshold. The azimuth angle of the target image is the azimuth angle of the target person relative to the optical axis of the camera.

[0037] If the conditions are met, then based on the valid speech segment, the voiceprint feature vector is extracted and written into the voiceprint feature set corresponding to the face candidate identity ID.

[0038] As described above, voiceprint registration is constrained by facial identity ID, time window, single-person scenario, and sound source direction, which can reduce the risk of mistakenly binding the sounds of other family members or the environment to the current identity ID.

[0039] The present invention also proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described above.

[0040] Example 1 Please refer to Figure 2-4 Embodiment 1 of the present invention is: a user identification method for a mobile camera robot, wherein the mobile camera robot can be an indoor mobile robot equipped with a wide-angle camera and a microphone array. A typical hardware configuration is that the camera is installed at a height similar to that of a robotic vacuum cleaner, and the camera captures environmental video from a low position with a wide field of view; the default video specifications are 720p and 30 frames per second. Audio is captured by a microphone array, which supports sound source localization; the specific sampling rate can be determined according to the hardware configuration.

[0041] This method is mainly used for family members to complete identity recognition, personalized companionship, home patrol security and identification of registered family members without obvious cooperative actions. It can also be extended to indoor service robots, elderly care robots, tour guide robots and other systems that need to identify people in low-angle or non-cooperative scenarios.

[0042] The system maintains its identity feature database locally, eliminating the need for cloud storage. Each identity ID corresponds to one identity feature record. In this embodiment, the record includes at least the following fields:

[0043] like Figure 2 As shown, this method includes the following steps: S1: A mobile camera robot collects environmental data in real time, including video and audio streams.

[0044] When the mobile camera robot moves along a preset cruise path or companion task, it captures video streams at a low angle and wide field of view through a wide-angle camera and audio streams through a microphone array. At the same time, it records the robot device identifier (robot_id), system time or frame timestamp (t), date, and current scene context (scene_context).

[0045] The default video capture specifications are 720p and 30 frames per second. The microphone array is used for voice activity detection and sound source localization; the specific sampling rate is determined by the hardware configuration.

[0046] S2: Upon receiving a registration instruction, perform single-person face registration and generate an identity ID.

[0047] Specifically, such as Figure 3 As shown, this step includes the following steps: S201: Upon receiving a registration instruction, determine whether the single-person scenario condition is met. If so, proceed to step S202. The single-person scenario condition includes detecting only one target person within a preset video time window, and there are no other persons whose detection confidence exceeds a preset second confidence threshold and whose detection box area exceeds a preset area threshold.

[0048] After receiving the registration instruction, the system enters single-person face registration mode. Registration mode requires that the current registration window maintain a single-person scene; that is, only one valid person target is detected within the preset video time window (e.g., a human detection bounding box or a pose instance count of 1), and there is no second person target whose detection confidence and image area both exceed preset thresholds. New users must complete face registration first. The system does not generate temporary IDs for unidentified individuals, nor does it create formal identity IDs if face registration is not completed.

[0049] The preset video time window can be determined based on the time the registration instruction is received, for example, within a few seconds after the registration instruction is received.

[0050] S202: Continuously acquire n+1 video frames containing the face of the target person to obtain a face image, where n is a preset first capacity. In this embodiment, n=4, that is, continuously acquire 5 video frames containing the face of the target person.

[0051] S203: Locate facial landmarks in each face image and perform facial correction based on these landmarks. For example, align landmarks such as the eyes, nose, and mouth to preset template positions.

[0052] S204: Extract the corresponding facial feature vectors for each corrected face image, and calculate the center feature based on the facial feature vectors.

[0053] In this embodiment, the system extracts five facial feature vectors from five corrected facial images and calculates the central features of the five facial feature vectors.

[0054] The central feature refers to the representative facial feature obtained by normalizing, averaging, and normalizing multiple continuously collected facial feature vectors during a single face registration process; or, selecting the feature with the highest average similarity to other feature vectors from the multiple facial feature vectors as the registration central feature. In other embodiments, the mean feature can also be calculated.

[0055] S205: Calculate the similarity between each facial feature vector and the central feature, and take the n facial feature vectors with the highest similarity as the facial feature set of the target person.

[0056] Since each identity ID's corresponding facial feature set stores a maximum of 4 features, the system calculates the similarity between each facial feature vector and the central feature, then removes the facial feature vector with the lowest similarity, and uses the other 4 facial feature vectors as the initial facial feature set for that identity ID.

[0057] This method can eliminate registered samples that deviate significantly from the target without setting additional conditions for occlusion detection, liveness detection, or minimum face bounding boxes, by utilizing the consistency of continuous sampling.

[0058] S206: Determine the maximum similarity between the target person and each existing identity ID.

[0059] In this embodiment, the preferred implementation of registration deduplication determination is feature-by-feature comparison, that is, calculating the similarity between the four facial feature vectors retained in step S205 and each facial feature vector in the facial feature set of each existing identity ID in the local identity feature database, and taking the maximum value among all pairwise similarities as the maximum similarity between the target person and the existing identity ID.

[0060] In some embodiments, the central feature can be used for rapid pre-screening or indexing to reduce the amount of retrieval. When the central feature pre-screening is successful, the maximum similarity obtained by comparing the four retained facial feature vectors with the facial features in the database one by one is still used as the final criterion for determining whether to register repeatedly.

[0061] S207: Determine whether the maximum similarity between the target person and an existing identity ID is greater than the preset registration deduplication threshold. If yes, proceed to step S208. If no, that is, the maximum similarity between the target person and each existing identity ID is not greater than the registration deduplication threshold, proceed to step S209.

[0062] In this embodiment, the deduplication threshold θ is registered. reg =0.5; to further reduce the risk of duplicate registration, it can also be set to 0.55.

[0063] S208: It is determined that the target person has already registered. The system merges the ID with the existing ID or prompts that the person is already registered, without creating a new ID.

[0064] S209: Generate the target person's identity ID based on the robot device identifier, registration timestamp, preset serial number and random salt value, and write the target person's identity ID and facial feature set into the local identity feature database.

[0065] When the above registration deduplication retrieval fails to find an existing identity ID, the identity ID generation module generates a new identity ID. Identity ID generation does not rely on facial feature summaries, age ranges, or gender attributes to avoid instability caused by changes in auxiliary metadata. In this embodiment, the identity ID can be generated as follows: identity_id = HASH(robot_id ||register_time || seq || random_salt); where robot_id is the robot device identifier, register_time is the registration timestamp, seq is a locally incrementing sequence number or a random sequence number, and random_salt is a locally generated random salt value.

[0066] After generation, the system writes the identity_id, along with four initial facial feature vectors, local registration time, and robot device identifier robot_id, into the local identity feature database.

[0067] In some embodiments, a multimodal large language model or attribute recognition model can be used as an optional auxiliary metadata generation module. Its input may include a cropped face image, a target torso image, date and time, robot device identifier, and scene context; the output may include age range, gender, appearance description, and scene label. The aforementioned auxiliary metadata is only written to the metadata field for recording purposes and does not participate in identity matching, candidate identity priority determination, or feature library update conditions.

[0068] S3: According to the preset processing cycle, obtain video frames from the video stream, perform face detection and body pose recognition on the video frames, and perform voice activity detection and sound source localization on the audio stream.

[0069] The system extracts frames from the video stream in each processing cycle and sends the video frames to the face detection, pose recognition and target image cropping process, and sends the audio stream to the speech activity detection and sound source localization process.

[0070] S4: Perform face detection on the video frame to check if a face is detected. If so, proceed to step S5.

[0071] S5: Extract the face feature vector, and determine the candidate face identity ID based on the face feature vector and the face feature set corresponding to each existing identity ID in the local identity feature library.

[0072] Specifically, after a face is detected, the current face feature vector is extracted and searched in the face feature set of the local identity feature database. That is, the similarity between this face feature vector and each face feature vector in the face feature set corresponding to each existing identity ID in the local identity feature database is calculated, and the maximum similarity between the current face feature vector and each existing identity ID is determined. Specifically, the maximum similarity between the current face feature vector and each face feature vector in the face feature set corresponding to an existing identity ID is taken as the maximum similarity between the current face feature vector and that existing identity ID.

[0073] If the maximum similarity between the current facial feature vector and an existing identity ID is greater than the preset first recognition threshold θ face If so, the existing identity ID is used as the candidate face identity ID (denoted as ID_face). In this embodiment, θ face =0.5. If a candidate face ID exists, it will have the highest priority in subsequent multimodal conflict determination.

[0074] Furthermore, if face detection fails, or if the maximum similarity between the current face feature vector and each existing identity ID is less than or equal to the first recognition threshold θ, face If not, the system will not output identity through facial recognition.

[0075] S6: Whether the pose key points and their confidence levels obtained by pose recognition of the video frame meet the full-body pedestrian re-identification gating conditions or the half-body pedestrian re-identification gating conditions. If so, proceed to step S7.

[0076] S7: Extract the full-body pedestrian re-identification feature vector or the half-body pedestrian re-identification feature vector, and determine the pedestrian re-identification candidate identity ID based on the full-body pedestrian re-identification feature vector or the half-body pedestrian re-identification feature vector, and the full-body feature set or half-body feature set corresponding to each existing identity ID in the local identity feature database.

[0077] Specifically, such as Figure 4 As shown, steps S6-S7 include the following steps: S601: Obtain the pose key points obtained through body pose recognition and the confidence level of each pose key point.

[0078] In this embodiment, the coordinates of COCO17 keypoints and the confidence level of each keypoint are obtained, using COCO17's starting number 0, where 0 is the nose, 5 is the left shoulder, 6 is the right shoulder, 11 is the left hip, 12 is the right hip, 13 is the left knee, 14 is the right knee, 15 is the left ankle, and 16 is the right ankle. A keypoint is considered usable only if it exists and its confidence level is not lower than 0.6.

[0079] S602: Determine whether the full-body pedestrian re-identification gating condition is met. If yes, proceed to step S603; otherwise, proceed to step S606.

[0080] The full-body pedestrian re-identification gating condition is that the acquired posture key points include all full-body key points and the confidence of all full-body key points is greater than or equal to a preset first confidence threshold (e.g., 0.6). The full-body key points include key points 5, 6, 11, 12, 13, 14, 15, and 16, namely, left shoulder, right shoulder, left hip, right hip, left knee, right knee, left ankle, and right ankle.

[0081] S603: Extract the full-body pedestrian re-identification feature vector based on the full-body key points.

[0082] Specifically, based on the bounding rectangle of the key points of the whole body (height H1, width W1), the whole body area is cropped by expanding outward by 0.2H1 in the vertical direction and 0.1W1 in the horizontal direction. Based on the whole body area, the whole body pedestrian re-identification feature vector (whole body ReID feature vector) is extracted.

[0083] S604: Calculate the similarity between the full-body pedestrian re-identification feature vector and each full-body pedestrian re-identification feature vector in the full-body feature set corresponding to each existing identity ID in the local identity feature database, and determine the maximum similarity between the full-body pedestrian re-identification feature vector and each existing identity ID.

[0084] Specifically, the maximum similarity between the current whole-body ReID feature vector and each whole-body ReID feature vector in the whole-body feature set corresponding to an existing identity ID is taken as the maximum similarity between the current whole-body ReID feature vector and the existing identity ID. This process is repeated to obtain the maximum similarity between the current whole-body ReID feature vector and each existing identity ID.

[0085] S605: Determine whether the maximum similarity between the full-body pedestrian re-identification feature vector and an existing identity ID is greater than a preset second identification threshold. If so, proceed to step S610. In this embodiment, the second identification threshold θ voice =0.6.

[0086] S606: Determine whether the half-body pedestrian re-identification gating condition is met. If yes, proceed to step S607. If no, skip the ReID feature extraction, comparison, seamless registration and update of the current frame, and do not generate ReID candidate identity IDs.

[0087] The half-body pedestrian re-identification gating condition is that the acquired posture key points include all half-body key points and the confidence of all half-body key points is greater than or equal to the preset first confidence threshold (such as 0.6). The half-body key points include key points 5, 6, 11, and 12, namely the left shoulder, right shoulder, left hip, and right hip.

[0088] S607: Extract the half-body pedestrian re-identification feature vector based on the half-body key points.

[0089] Specifically, based on the bounding rectangle of the half-body key points (height H2, width W2), the half-body region is cropped by expanding outward by 0.2H2 in the vertical direction and 0.1W2 in the horizontal direction. Then, the half-body pedestrian re-identification feature vector (half-body ReID feature vector) is extracted based on the half-body region.

[0090] S608: Calculate the similarity between the half-body pedestrian re-identification feature vector and each half-body pedestrian re-identification feature vector in the half-body feature set corresponding to each existing identity ID in the local identity feature database, and determine the maximum similarity between the half-body pedestrian re-identification feature vector and each existing identity ID.

[0091] Specifically, the maximum similarity between the current half-body ReID feature vector and each half-body ReID feature vector in the half-body feature set corresponding to an existing identity ID is taken as the maximum similarity between the current half-body ReID feature vector and the existing identity ID. This process is repeated to obtain the maximum similarity between the current half-body ReID feature vector and each existing identity ID.

[0092] S609: Determine whether the maximum similarity between the half-body pedestrian re-identification feature vector and an existing identity ID is greater than a preset second identification threshold. If so, proceed to step S610.

[0093] S610: Use the existing identity ID as a pedestrian re-identification candidate identity ID (ReID candidate identity ID, denoted as ID_reid).

[0094] In this embodiment, the half-body ReID feature vector represents the features of the upper body. Even if the system can obtain partial lower limb images, it does not extract independent lower body ReID features for identity recognition, seamless registration, or feature database updates. This is because this embodiment is designed for low-angle mobile camera robots. While these robots can easily capture images of legs or feet during home patrols, the reliability and discriminative power of the lower limb region alone in identity recognition are insufficient: lower limb images are easily affected by tables, chairs, sofas, door frames, bed edges, floor reflections, robot movement jitter, and image edge truncation, resulting in incomplete human body cropping. Therefore, writing features extracted solely from the lower body region into the identity database can easily lead to low-quality samples or samples from incorrect sources contaminating the identity feature database.

[0095] This embodiment limits the use of lower body information to a portion of the full-body ReID features: the system extracts full-body ReID features and writes or updates the full-body ReID feature set only when keypoints for both shoulders, hips, knees, and ankles are complete and the confidence level meets the requirements; if only the keypoints for both shoulders and hips are met, half-body ReID features are extracted and compared only with the half-body ReID feature set. This approach avoids simply discarding lower limb information while preventing the inclusion of independent lower body features without stable identity binding criteria in the identity database, thereby reducing the risk of misbinding and misupdating in low-angle, non-cooperative home scenarios.

[0096] S8: Perform voice activity detection on the audio stream to detect whether a valid voice segment with a continuous duration greater than a preset duration is detected. If so, proceed to step S9.

[0097] In this embodiment, the preset duration is 2 seconds.

[0098] S9: Extract the voiceprint feature vector of the effective speech segment, and determine the voiceprint candidate identity ID based on the voiceprint feature vector and the voiceprint feature set corresponding to each existing identity ID in the local identity feature library.

[0099] Specifically, after detecting a valid speech segment, a voiceprint feature vector is extracted and compared with the voiceprint feature set in the local identity feature database. That is, the similarity between the voiceprint feature vector and each voiceprint feature vector in the voiceprint feature set corresponding to each existing identity ID in the local identity feature database is calculated, and the maximum similarity between the voiceprint feature vector and each existing identity ID is determined. Specifically, the maximum similarity between the current voiceprint feature vector and each voiceprint feature vector in the voiceprint feature set corresponding to an existing identity ID is taken as the maximum similarity between the current voiceprint feature vector and that existing identity ID.

[0100] If the maximum similarity between the current voiceprint feature vector and an existing identity ID is greater than the preset third recognition threshold θ voice Then, the existing identity ID is used as a candidate voiceprint identity ID (denoted as ID_voice). In this embodiment, θ voice =0.5.

[0101] S10: Determine the final identity ID from the face candidate identity ID, pedestrian re-identification candidate identity ID and voiceprint candidate identity ID according to the preset priority order.

[0102] The system prioritizes candidate identities generated from face, ReID, and voiceprint. This prioritization is not a simple summation, but rather a combination of modal reliability and face anchoring principles.

[0103] Specifically, if the candidate face ID exists, it is used as the final identity ID. The feature library for the final identity ID can then be updated. If ID_reid or ID_voice is inconsistent with ID_face, the conflicting modality is not used for feature library updates.

[0104] If the face candidate ID does not exist but the pedestrian re-identification candidate ID (ReID candidate ID) does, then the pedestrian re-identification candidate ID will be used as the final ID. If a voiceprint candidate ID also exists but ID_voice and ID_reid are inconsistent, then ID_voice will prevail.

[0105] If neither the face candidate ID nor the pedestrian re-identification candidate ID exists, but the voiceprint candidate ID does, then the voiceprint candidate ID is used as the final identity ID. In this embodiment, when only a voiceprint candidate ID exists, and the VAD (Voice Activity Detection, used to determine whether there are continuous valid speech segments in the audio) quality is acceptable, and the sound source direction is reliable (i.e., when there is a candidate location for the current person, the sound source direction is consistent with the candidate location for that person), a low-confidence ID_voice can be output. However, no new identity ID is created based on the ID_voice, nor are any upper-layer actions such as seamless registration, feature library updates, or security alarms triggered. If the VAD quality is unacceptable, or the sound source direction is unreliable, then "unrecognized" is output.

[0106] If there are no valid candidates for any of the three modalities, i.e., no candidate identity ID for face, candidate identity ID for ReID, and candidate identity ID for voiceprint, then the output will be "unrecognized," no temporary ID will be generated, and no report will be submitted.

[0107] By using the above priority rules, existing identities can be output using ReID or voiceprint when faces are unavailable due to low viewpoints, while avoiding the use of conflicting modalities to pollute the identity feature database when there are multimodal candidate conflicts.

[0108] Furthermore, when the system obtains an identity ID through face comparison during routine recognition, the face feature set can also be updated. For example, after detecting the existence of a candidate face identity ID and using it as the final identity ID, the face feature set corresponding to the final identity ID is updated based on the currently extracted face feature vector. Specifically, if the face feature set corresponding to the final identity ID is not full, that is, the number of face feature vectors in the face feature set has not reached the preset first capacity (4 in this embodiment), then the current face feature vector is appended; if it is full, then the maximum similarity between the current face feature vector and each face feature vector in the face feature set corresponding to the final identity ID is calculated. If the maximum similarity is greater than a preset update threshold (e.g., 0.55), then the earliest written face feature vector is replaced with the current face feature vector; if the maximum similarity is not greater than the preset update threshold, then the face feature set corresponding to the final identity ID is not updated.

[0109] Furthermore, after determining the candidate face ID, it is checked whether the full-body feature set / half-body feature set corresponding to the candidate face ID is empty. If it is empty, it means that the full-body ReID feature / half-body ReID feature of the candidate face ID has not yet been registered. At this time, the ReID seamless registration process is initiated. ReID seamless registration is used to bind human appearance features to an identity ID that has been verified by face. In some embodiments, ReID seamless registration is also performed after successfully registering a new identity ID based on face features in step S2.

[0110] Specifically, the ReID seamless registration process includes the following steps: S1101: Determine whether the conditions for full-body pedestrian re-identification registration are met. If yes, proceed to step S1102; otherwise, proceed to step S1103.

[0111] The registration conditions for full-body pedestrian re-identification include an empty full-body feature set corresponding to the candidate face ID, and meeting both the single-person scene condition and the full-body pedestrian re-identification gating condition. The single-person scene condition can be referred to the description in step S201, and the full-body pedestrian re-identification gating condition can be referred to the description in step S602.

[0112] S1102: Based on the full-body key points, extract the full-body pedestrian re-identification feature vector and write the full-body pedestrian re-identification feature vector into the full-body feature set corresponding to the face candidate identity ID. The extraction process of the full-body ReID feature vector can be referred to the description in step S603.

[0113] S1103: Determine whether the conditions for half-body pedestrian re-identification registration are met. If yes, proceed to step S1104. If no, end the ReID non-contact registration process and do not register full-body ReID features / half-body ReID features.

[0114] The registration conditions for half-body pedestrian re-identification include an empty half-body feature set corresponding to the candidate face ID, and satisfaction of the single-person scene condition and the half-body pedestrian re-identification gating condition. The single-person scene condition can be referred to the description in step S201, and the half-body pedestrian re-identification gating condition can be referred to the description in step S606.

[0115] S1104: Based on the half-body key points, extract the half-body pedestrian re-identification feature vector, and write the half-body pedestrian re-identification feature vector into the half-body feature set corresponding to the face candidate identity ID. The extraction process of the half-body ReID feature vector can be referred to the description in step S607.

[0116] If the full-body feature set corresponding to the candidate face ID is not empty, and a candidate ReID ID has also been determined, and if the candidate ReID ID is consistent with the candidate face ID (i.e., a face can be detected in the current video frame, and pose key points can also be identified, and the ReID features corresponding to the face features and pose key points correspond to the same existing ID), then the full-body feature set or half-body feature set corresponding to the final ID is updated based on the currently extracted full-body ReID feature vector or half-body ReID feature vector.

[0117] Specifically, when updating the full-body feature set, if the full-body feature set corresponding to the final identity ID is not full, that is, the number of full-body ReID feature vectors in the full-body feature set has not reached the preset second capacity (5 in this embodiment), then the current full-body ReID feature vector is appended and written; if it is full, then the maximum similarity between the current full-body ReID feature vector and each full-body ReID feature vector in the full-body feature set corresponding to the final identity ID is calculated. If the maximum similarity is greater than the preset update threshold (e.g., 0.55), then the earliest written full-body ReID feature vector is replaced with the current full-body ReID feature vector. If the maximum similarity is not greater than the preset update threshold, then the full-body feature set corresponding to the final identity ID is not updated.

[0118] Similarly, when updating the half-body feature set, if the half-body feature set corresponding to the final identity ID is not full, that is, the number of half-body ReID feature vectors in the half-body feature set has not reached the preset second capacity (5 in this embodiment), then the current half-body ReID feature vector is appended; if it is full, then the maximum similarity between the current half-body ReID feature vector and each half-body ReID feature vector in the half-body feature set corresponding to the final identity ID is calculated. If the maximum similarity is greater than the preset update threshold (e.g., 0.55), then the earliest written half-body ReID feature vector is replaced with the current half-body ReID feature vector. If the maximum similarity is not greater than the preset update threshold, then the half-body feature set corresponding to the final identity ID is not updated.

[0119] In this embodiment, the full-body feature set and the half-body feature set are maintained separately and are not mixed with each other.

[0120] Furthermore, after determining the candidate face ID, it is determined whether the voiceprint feature set corresponding to the candidate face ID is empty. If it is empty, it means that the voiceprint features of the candidate face ID have not yet been registered. At this time, the voiceprint seamless registration process is initiated. Voiceprint seamless registration is used to bind the voice features to the identity ID that has been confirmed by face. In some embodiments, after successfully registering a new identity ID based on face features in step S2, voiceprint seamless registration is also initiated.

[0121] Specifically, the voiceprint-based contactless registration process includes the following steps: S1201: Determine the voiceprint registration window based on the time of determination of the face candidate identity ID.

[0122] Specifically, a voiceprint registration window is established centered on the face verification time t0, preferably within 3 seconds after t0. If pre-caching is required, the audio can be traced back 1 second.

[0123] S1202: Determine whether the voiceprint registration conditions are met. If yes, proceed to step S1203. If no, end the voiceprint non-intrusive registration process and do not register the voiceprint features.

[0124] The voiceprint registration conditions include an empty voiceprint feature set corresponding to the candidate face ID, detection of a valid voice segment with a continuous duration greater than a preset duration (e.g., 2 seconds) within the voiceprint registration window, and satisfaction of the single-person scene condition and the sound source direction consistency condition.

[0125] The single-person scene conditions can be referred to in step S201, that is, only one target person is detected within the preset video time window, and there are no other persons whose detection confidence exceeds the preset second confidence threshold and whose detection box area exceeds the preset area threshold.

[0126] The sound source direction consistency condition is that the angle between the sound source direction estimated by the microphone array and the azimuth angle of the target image is less than or equal to a preset angle threshold, wherein the azimuth angle of the target image is the azimuth angle of the target person relative to the optical axis of the camera.

[0127] In this embodiment, when determining the location of a target image, the system prioritizes using the target region where a currently confirmed face is located. If there is no valid face in the current frame, but a full-body or half-body region has been obtained based on the full-body or half-body pedestrian re-identification gating condition, then that full-body or half-body region is used. If none of the above regions are available, but a valid human detection box exists, then the human detection box is used.

[0128] Assume the x-coordinate of the left boundary of the target region used to determine the orientation of the target image is x. l The right boundary of the image has an x-coordinate of x. r The center coordinates u of the current target person in the horizontal direction of the image are calculated. c Assume the x-coordinate of the image center point is u. O The equivalent focal length of the camera in the horizontal direction is f. x Then, the target image azimuth angle of the current target person relative to the camera's optical axis is: α target =arctan((u c -u O ) / f x The target image orientation only requires the output of the horizontal angle α. target There is no need to calculate the target distance or three-dimensional spatial coordinates.

[0129] Assume the microphone array estimates the sound source direction as α. sound The angle difference between the direction of the sound source and the orientation of the target image is: Δα = |α sound -α target |

[0130] If the direction angle is represented by a circular angle, then Δα is the smaller of the two angles. This is true when: Δα ≤ θ sound If the direction of the sound source matches the orientation of the current target image, then the direction of the sound source is determined to be consistent with the orientation of the current target image; otherwise, the direction of the sound source is determined to be inconsistent with the orientation of the current target image. Wherein, θ sound The threshold value for the angle between the sound source direction can be set to 15° to 30° in this embodiment, preferably 20°.

[0131] S1203: Extract the voiceprint feature vector based on the effective speech segment, and write the voiceprint feature vector into the voiceprint feature set corresponding to the face candidate identity ID.

[0132] If the voiceprint feature set corresponding to the candidate face ID is not empty, and a candidate voiceprint ID has also been determined, and if the candidate voiceprint ID is consistent with the candidate face ID (i.e., a face can be detected in the current video frame, a valid speech segment can be detected in the current audio data, and the face features and voiceprint features correspond to the same existing ID), then the voiceprint feature set corresponding to the final ID is updated based on the currently extracted voiceprint feature vector. Specifically, if the voiceprint feature set corresponding to the final ID is not full (i.e., the number of voiceprint feature vectors in the voiceprint feature set has not reached the preset third capacity (4 in this embodiment), then the current voiceprint feature vector is appended; if it is full, then the maximum similarity between the current voiceprint feature vector and each voiceprint feature vector in the voiceprint feature set corresponding to the final ID is calculated. If the maximum similarity is greater than the preset update threshold (e.g., 0.55), then the earliest written voiceprint feature vector is replaced with the current voiceprint feature vector. If the maximum similarity is not greater than the preset update threshold, then the voiceprint feature set corresponding to the final ID is not updated.

[0133] In this embodiment, the identification threshold is used to determine whether identity candidates can be output, and the update threshold is used to determine whether updating the identity template is allowed. Setting the update threshold higher than the identification threshold can reduce the probability of low-quality samples, samples from erroneous sources, or low-confidence samples entering the identity feature database.

[0134] Furthermore, in this embodiment, cosine similarity is preferably used to compare each modal feature with the features in the database. For the current feature vector f and the feature vector g in the database, the cosine similarity is: sim(f,g)=(f·g) / (||f||·||g||), where f represents the currently extracted feature vector (face feature vector, full-body ReID feature vector, half-body ReID feature vector, or voiceprint feature vector), g represents the same modal feature vector in the local identity feature database, and ||f|| and ||g|| represent the vector norms, respectively. In actual implementation, the feature vectors can be normalized first, and then the vector inner product can be used as the cosine similarity.

[0135] This embodiment has the following technical effects: First, facial recognition reduces uncertainty in cross-modal registration. New users must first complete facial registration in a single-person scenario. ReID and voiceprint do not directly create new identity IDs, but are seamlessly bound to an existing facial identity ID. Therefore, the initial trusted source of the identity ID is clear.

[0136] Second, pose keypoint gating improves the quality of ReID features. Low-angle mobile robots often only capture partial images of the human body. By requiring at least two shoulder and two hip keypoints for the half-body and at least two shoulder, two hip, two knee, and two ankle keypoints for the full body, and requiring a keypoint confidence of no less than 0.6, images with severe occlusion, incomplete human body cropping, or unreliable pose estimation can be filtered out, reducing the contamination of the ReID database by low-quality human samples.

[0137] Third, voiceprint registration is constrained by face ID, time window, single-person scenario, and sound source direction. Voiceprint features are not written whenever voice is detected, but within a short time window after face ID has been confirmed. It also requires that the valid voice last for at least 2 seconds and that the sound source direction is consistent with the current single-person target, thereby reducing the risk of mistakenly binding the voices of other family members or the environment to the current identity ID.

[0138] Fourth, priority determination enables the system to adapt to low-view scenarios. When a face is present, the face candidate identity is given the highest priority; when a face is unavailable, ReID is used to supplement the identity; only when there is no face and no ReID candidate, and the VAD quality and sound source direction are reliable, can low-confidence voiceprint candidates be output. This rule allows the robot to identify existing family members using the already unconsciously bound ReID or voiceprint even when it cannot see the frontal face or the user does not speak, while avoiding the voiceprint result triggering registration or updates independently.

[0139] Fifth, the high-threshold update mechanism separates "identification" and "updating". The identification threshold is used to improve the recall capability of existing identities, while the update threshold is 0.05 higher than the identification threshold to control the refresh of identity templates. When the feature library is full, only the earliest written features of the same modality are replaced, which not only keeps the library capacity stable, but also allows the identity templates to be updated gradually with changes in clothing, posture and voice in the home scene.

[0140] Example 2 This embodiment is a specific application scenario of the above embodiment one, mainly including face anchoring registration, seamless binding of ReID and voiceprint, multimodal identity recognition, and automatic updating of the local identity feature database.

[0141] Scenario 1: Face-based registration during first-time use When a home user uses the robot for the first time, they send a registration command through the application or the robot's interface. The robot remains stationary or moves at low speed, and the wide-angle camera captures video frames within the registration window at 720p and 30 frames per second. Once the system detects only one clearly visible person within the registration window, it continuously captures five facial images, performs facial landmark localization and correction on each image, and extracts five facial features.

[0142] The system calculates the central feature of 5 facial features, removes the one with the lowest similarity to the central feature, and uses the remaining 4 as the initial facial feature set for the user. Subsequently, the system uses the central feature to search the local facial feature database; if the maximum similarity is greater than the registration deduplication threshold θ... reg If no match is found, the existing identity ID will be retrieved or a message indicating that the user has already registered will be displayed; no new identity ID will be created. If no existing identity is found, the identity ID generation module will generate an identity ID based on the robot device identifier, registration timestamp, local serial number, and random salt value, and write this identity ID and four initial facial features into the local identity feature library. Optional multimodal large language models or attribute recognition models will only output auxiliary metadata such as age range, appearance description, and scene labels. This auxiliary metadata will only be written to the metadata field and will not participate in identity matching or feature library updates.

[0143] Scenario 2: Seamless ReID Binding When Face is Visible After completing initial facial registration, the user stands or walks in front of the robot during family time. Due to its low-angle, upward-facing camera, the robot may only be able to consistently capture images of the user's upper body, but it can still detect a face within a certain processing cycle. The system first obtains an identity ID through facial feature comparison. For example, if the facial similarity is 0.62, which is greater than the facial recognition threshold of 0.5, the system confirms that the current person is identity ID_A.

[0144] Within the same single-person processing window, the system performs pose estimation. If keypoints 5, 6, 11, and 12 in COCO17 (i.e., left shoulder, right shoulder, left hip, and right hip) all exist and their confidence levels are not lower than 0.6, the system uses the circumscribed regions of these four keypoints as a basis, expanding outwards by 0.2H in the vertical direction and 0.1W in the horizontal direction, cropping the half-body region and extracting half-body ReID features. If the half-body ReID feature set for identity ID_A is not full, the half-body ReID feature is written into the set; if it is full, the maximum similarity between the feature and the existing half-body ReID features of identity ID_A is calculated. Only when the maximum similarity is greater than 0.65 is the current feature used to replace the earliest written half-body ReID feature.

[0145] If the conditions for full-body keypoints are met simultaneously, i.e. keypoints 5, 6, 11, 12, 13, 14, 15, and 16 are all complete and their confidence scores are not lower than 0.6, then the system will prioritize extracting full-body ReID features and writing or updating the full-body ReID feature set; half-body features correspond only to the half-body database, and full-body features correspond only to the full-body database, and the two are not mixed for comparison.

[0146] Scenario 3: Identity recognition in subsequent low-angle, faceless scenarios After the ReID seamless binding was completed in Scenario 2, the robot cruises along the living room, hallway, or bedroom doorway. Because the camera is close to the robot's height, the user may not be looking down at the camera in the current video frame, and the system may not be able to detect a valid face or the face similarity may not be greater than 0.5. At this time, the system does not create a new identity ID, but instead enters the ReID identity recognition process.

[0147] If the current human image meets the full-body keypoint gating condition, the system extracts full-body ReID features and compares them only with the full-body ReID feature set in the local identity feature library. If only the half-body keypoint gating condition is met, the system extracts half-body ReID features and compares them only with the half-body ReID feature set. If the half-body keypoint gating condition is not met, the system skips the extraction, comparison, seamless registration, and update of ReID features for the current frame and does not generate ReID candidate identities. If the maximum similarity is greater than 0.6, the corresponding existing identity ID is output. If no face, ReID, or voiceprint candidate identity meets the threshold, an unrecognized result is output. In this embodiment, no temporary ID is generated, and unrecognized personnel are not reported.

[0148] In this scenario, ReID recognition is used to output existing identities; to avoid erroneous source samples from faceless conditions contaminating the identity database, the new modality binding of ReID is still preferably performed within a single-person window where identity has already been confirmed by face.

[0149] Scenario 4: Seamless voiceprint binding after facial recognition After a family member completes facial recognition near the robot, the system records the facial identity confirmation time t0 and initiates the voiceprint registration window associated with that identity ID. Preferably, the voiceprint registration window is within 3 seconds after t0; if audio pre-buffering exists, it can trace back 1 second of audio. If the voice activity detection within this window obtains a continuous valid voice segment of no less than 2 seconds, and the current window is a single-person scene, and the angle between the sound source direction estimated by the microphone array and the current target image orientation is no greater than a preset angle threshold (preferably 20°), then the system extracts the voiceprint features and binds them to that identity ID.

[0150] If the voiceprint feature set for the identity ID is not full, it is directly appended; if it is full, the maximum similarity between the current voiceprint feature and the existing voiceprint feature set for the identity ID is calculated. Only when the maximum similarity is greater than 0.55 is the current voiceprint feature replaced with the earliest written voiceprint feature. If no valid voice is detected for 2 consecutive seconds, multiple obvious human targets appear, or the direction of the sound source is inconsistent with the current target's location, voiceprint registration and updating are not performed.

[0151] Scenario 5: Internal Identity Filtering in Home Security Patrols When the robot is in home patrol security mode, the identity recognition module can provide the upper-level patrol logic with an internal result of "identified family member" or "unidentified". For example, when the robot patrols through a corridor at night, it detects a single human target but does not detect a valid face; if the target meets the full-body ReID gating condition and the full-body ReID similarity is greater than 0.6, the identity recognition module outputs the corresponding family member identity ID, which is then recorded by the upper-level system as a known family member.

[0152] If a voiceprint candidate identity is detected simultaneously and it does not match the ReID candidate identity, the ReID candidate identity will be prioritized according to the priority rules of this scheme, and the use of conflicting voiceprint features to update the voiceprint database for that identity will be prohibited. If a face candidate identity is detected, the face candidate identity will be output with the highest priority, regardless of whether a ReID or voiceprint candidate identity exists; ReID or voiceprint features that conflict with the face candidate identity will not be written into the feature database. If all modalities fail to match, this identity recognition module will only output the unrecognized result and will not generate a temporary ID; whether to trigger an alarm, notification, or other security actions will be determined by the upper-layer business strategy.

[0153] Example 3 This embodiment is a computer-readable storage medium corresponding to the above embodiments, on which a computer program is stored. When the program is executed by a processor, it implements the various steps of the user recognition method for a mobile camera robot as described in the above embodiments, and can achieve the same technical effect, which will not be repeated here.

[0154] In summary, the user identification method and storage medium for a mobile camera robot provided by this invention ensures registration accuracy by requiring users to complete face registration in a single-person scenario. By selecting the n face feature vectors with the highest similarity to the central feature from n+1 face feature vectors as the initial face feature set, large deviations in registration samples can be eliminated through the consistency of continuous sampling without additional occlusion detection, liveness detection, or minimum face bounding box conditions. ReID registration is constrained by face identity ID, single-person scenario, and full-body / half-body ReID gating conditions, reducing the risk of mistakenly binding other family members to the current identity ID. Voiceprint registration is simultaneously constrained by face identity ID, time window, single-person scenario, and sound source direction, further reducing the risk of mistakenly binding other family members or environmental sounds to the current identity ID.

[0155] When performing user identification, if a face is present, the candidate face ID is given the highest priority. If a face is unavailable, ReID features are used to supplement the identity. Only when there is no face and no ReID candidate, and the VAD quality and sound source direction are reliable, can low-confidence voiceprint candidates be output. This rule allows the robot to identify existing family members using ReID or voiceprint features even when it cannot see the user's face or the user does not speak, while avoiding triggering registration or updates solely based on voiceprint recognition results. When the user has only completed initial face registration, the low-angle moving camera robot reliably binds uncooperatively collected human ReID and voiceprint features to the same face ID during subsequent home patrols and companionship, and automatically updates the feature library during the identification process, preventing low-quality or erroneous features from being written into the identity database.

[0156] Setting the update threshold higher than the recognition threshold reduces the probability of low-quality samples, samples from erroneous sources, or low-confidence samples entering the identity feature database. When the feature database is full, only the earliest written features of the same modality are replaced, maintaining a stable database size and allowing identity templates to be updated gradually with changes in clothing, posture, and voice in a home setting.

[0157] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent modifications made based on the content of the present invention specification and drawings, or direct or indirect applications in related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A user identification method for a mobile camera robot, characterized in that, include: A mobile camera robot collects environmental data in real time, including video and audio streams. According to a preset processing cycle, video frames are obtained from the video stream, face detection and body pose recognition are performed on the video frames, and voice activity detection is performed on the audio stream. If a face is detected, the face feature vector is extracted, and the candidate face identity ID is determined based on the face feature vector and the face feature set corresponding to each existing identity ID in the local identity feature library. If the identified posture key points and their confidence levels meet the full-body pedestrian re-identification gating conditions or the half-body pedestrian re-identification gating conditions, then the full-body pedestrian re-identification feature vector or the half-body pedestrian re-identification feature vector is extracted, and based on the full-body pedestrian re-identification feature vector or the half-body pedestrian re-identification feature vector, as well as the full-body feature set or half-body feature set corresponding to each existing identity ID in the local identity feature database, the pedestrian re-identification candidate identity ID is determined. If a valid voice segment with a continuous duration longer than a preset duration is detected, the voiceprint feature vector of the valid voice segment is extracted, and a voiceprint candidate identity ID is determined based on the voiceprint feature vector and the voiceprint feature set corresponding to each existing identity ID in the local identity feature library. The final identity ID is determined from the candidate face identity ID, candidate pedestrian re-identification identity ID, and candidate voiceprint identity ID according to a preset priority order.

2. The user identification method for a mobile camera robot according to claim 1, characterized in that, The step of determining candidate identity IDs based on the face feature vector and the set of face features corresponding to each existing identity ID in the local identity feature database includes: Calculate the similarity between the face feature vector and each face feature vector in the face feature set corresponding to each existing identity ID in the local identity feature database, and determine the maximum similarity between the face feature vector and each existing identity ID; If the maximum similarity between the facial feature vector and an existing identity ID is greater than a preset first recognition threshold, then the existing identity ID is used as a candidate facial identity ID.

3. The user identification method for a mobile camera robot according to claim 1, characterized in that, If the identified pose key points and their confidence levels satisfy the full-body pedestrian re-identification gating condition or the half-body pedestrian re-identification gating condition, then the full-body pedestrian re-identification feature vector or the half-body pedestrian re-identification feature vector is extracted, including: Obtain the pose key points obtained through body pose recognition and the confidence level of each pose key point; Determine whether the full-body pedestrian re-identification gating condition is met. The full-body pedestrian re-identification gating condition is that the acquired posture key points include full-body key points and the confidence of each full-body key point is greater than or equal to a preset first confidence threshold. The full-body key points include left shoulder, right shoulder, left hip, right hip, left knee, right knee, left ankle, and right ankle. If the conditions are met, then based on the aforementioned key points throughout the body, extract the full-body pedestrian re-identification feature vector; If not satisfied, then determine whether the half-body pedestrian re-identification gating condition is satisfied. The half-body pedestrian re-identification gating condition is that the acquired posture key points include half-body key points and the confidence of each half-body key point is greater than or equal to the preset first confidence threshold. The half-body key points include the left shoulder, right shoulder, left hip and right hip. If the conditions are met, then based on the aforementioned half-body key points, extract the half-body pedestrian re-identification feature vector.

4. The user identification method for a mobile camera robot according to claim 3, characterized in that, The step of extracting the full-body pedestrian re-identification feature vector based on the full-body key points includes: The whole body region is obtained based on the bounding rectangle of the key points of the whole body and the preset first expansion height and first expansion width, and the whole body pedestrian re-identification feature vector is extracted based on the whole body region; the first expansion height and the first expansion width are determined according to the height and width of the bounding rectangle of the key points of the whole body, respectively. The step of extracting a half-body pedestrian re-identification feature vector based on the half-body key points includes: Based on the bounding rectangle of the half-body key points and the preset second external height and second external width, the half-body region is obtained, and the half-body pedestrian re-identification feature vector is extracted based on the half-body region. The second external height and second external width are determined according to the height and width of the bounding rectangle of the half-body key points, respectively.

5. The user identification method for a mobile camera robot according to claim 1, characterized in that, The step of determining candidate identity IDs for pedestrian re-identification based on the full-body pedestrian re-identification feature vector or half-body pedestrian re-identification feature vector, and the full-body feature set or half-body feature set corresponding to each existing identity ID in the local identity feature database, includes: Calculate the similarity between the full-body pedestrian re-identification feature vector and each full-body pedestrian re-identification feature vector in the full-body feature set corresponding to each existing identity ID in the local identity feature database, and determine the maximum similarity between the full-body pedestrian re-identification feature vector and each existing identity ID; Alternatively, calculate the similarity between the half-body pedestrian re-identification feature vector and each half-body pedestrian re-identification feature vector in the half-body feature set corresponding to each existing identity ID in the local identity feature database, and determine the maximum similarity between the half-body pedestrian re-identification feature vector and each existing identity ID. If the maximum similarity between the full-body pedestrian re-identification feature vector and an existing identity ID is greater than a preset second identification threshold, or if the maximum similarity between the half-body pedestrian re-identification feature vector and an existing identity ID is greater than a preset second identification threshold, then the existing identity ID will be used as a candidate identity ID for pedestrian re-identification.

6. The user identification method for a mobile camera robot according to claim 1, characterized in that, The step of determining candidate identity IDs based on the voiceprint feature vector and the voiceprint feature set corresponding to each existing identity ID in the local identity feature database includes: Calculate the similarity between the voiceprint feature vector and each voiceprint feature vector in the voiceprint feature set corresponding to each existing identity ID in the local identity feature database, and determine the maximum similarity between the voiceprint feature vector and each existing identity ID. If the maximum similarity between the voiceprint feature vector and an existing identity ID is greater than a preset third recognition threshold, then the existing identity ID will be used as a voiceprint candidate identity ID.

7. The user identification method for a mobile camera robot according to claim 1, characterized in that, The step of determining the final identity ID from the candidate face IDs, candidate pedestrian re-identification IDs, and candidate voiceprint IDs according to a preset priority order includes: If the aforementioned candidate face ID exists, then the candidate face ID shall be used as the final identity ID; If the face candidate identity ID does not exist but the pedestrian re-identification candidate identity ID exists, then the pedestrian re-identification candidate identity ID shall be used as the final identity ID. If the face candidate identity ID and pedestrian re-identification candidate identity ID do not exist, but the voiceprint candidate identity ID does exist, then the voiceprint candidate identity ID will be used as the final identity ID.

8. The user identification method for a mobile camera robot according to claim 7, characterized in that, If the candidate face ID exists, then after using the candidate face ID as the final identity ID, the method further includes: Update the set of facial features corresponding to the final identity ID based on the facial feature vector; If there is a pedestrian re-identification candidate ID, and the pedestrian re-identification candidate ID is consistent with the face candidate ID, then the full-body feature set or half-body feature set corresponding to the final ID is updated according to the full-body pedestrian re-identification feature vector or the half-body pedestrian re-identification feature vector. If a candidate voiceprint identity ID exists and the candidate voiceprint identity ID is consistent with the candidate face identity ID, then the voiceprint feature set corresponding to the final identity ID is updated according to the voiceprint feature vector.

9. The user identification method for a mobile camera robot according to claim 8, characterized in that, The step of updating the face feature set corresponding to the final identity ID based on the face feature vector includes: If the number of face feature vectors in the face feature set corresponding to the final identity ID does not reach the preset first capacity, then the face feature vectors are written into the face feature set corresponding to the final identity ID. If the number of face feature vectors in the face feature set corresponding to the final identity ID has reached the preset first capacity, then calculate the maximum similarity between the face feature vector and each face feature vector in the face feature set corresponding to the final identity ID. If the maximum similarity is greater than the preset update threshold, the face feature vector is written into the face feature set corresponding to the final identity ID, and the face feature vector with the earliest writing time in the face feature set corresponding to the final identity ID is deleted.

10. The user identification method for a mobile camera robot according to claim 8, characterized in that, The step of updating the full-body feature set or half-body feature set corresponding to the final identity ID based on the full-body pedestrian re-identification feature vector or the half-body pedestrian re-identification feature vector includes: If the number of full-body pedestrian re-identification feature vectors in the full-body feature set corresponding to the final identity ID does not reach the preset second capacity, then the full-body pedestrian re-identification feature vectors are written into the full-body feature set corresponding to the final identity ID. If the number of full-body pedestrian re-identification feature vectors in the full-body feature set corresponding to the final identity ID has reached the preset second capacity, then calculate the maximum similarity between the full-body pedestrian re-identification feature vector and each full-body pedestrian re-identification feature vector in the full-body feature set corresponding to the final identity ID. If the maximum similarity is greater than the preset update threshold, the full-body pedestrian re-identification feature vector is written into the full-body feature set corresponding to the final identity ID, and the full-body pedestrian re-identification feature vector with the earliest writing time in the full-body feature set corresponding to the final identity ID is deleted. or, If the number of half-body pedestrian re-identification feature vectors in the half-body feature set corresponding to the final identity ID does not reach the preset second capacity, then the half-body pedestrian re-identification feature vectors are written into the half-body feature set corresponding to the final identity ID. If the number of half-body pedestrian re-identification feature vectors in the half-body feature set corresponding to the final identity ID has reached the preset second capacity, then calculate the maximum similarity between the half-body pedestrian re-identification feature vector and each half-body pedestrian re-identification feature vector in the half-body feature set corresponding to the final identity ID. If the maximum similarity is greater than the preset update threshold, the half-body pedestrian re-identification feature vector is written into the half-body feature set corresponding to the final identity ID, and the half-body pedestrian re-identification feature vector with the earliest writing time in the half-body feature set corresponding to the final identity ID is deleted.

11. The user identification method for a mobile camera robot according to claim 8, characterized in that, The step of updating the voiceprint feature set corresponding to the final identity ID based on the voiceprint feature vector includes: If the number of voiceprint feature vectors in the voiceprint feature set corresponding to the final identity ID does not reach the preset third capacity, then the voiceprint feature vectors are written into the voiceprint feature set corresponding to the final identity ID. If the number of voiceprint feature vectors in the voiceprint feature set corresponding to the final identity ID has reached the preset third capacity, then calculate the maximum similarity between the voiceprint feature vector and each voiceprint feature vector in the voiceprint feature set corresponding to the final identity ID. If the maximum similarity is greater than the preset update threshold, the voiceprint feature vector is written into the voiceprint feature set corresponding to the final identity ID, and the voiceprint feature vector with the earliest writing time in the voiceprint feature set corresponding to the final identity ID is deleted.

12. The user identification method for a mobile camera robot according to claim 1, characterized in that, After the mobile camera robot collects environmental data in real time, it further includes: When a registration instruction is received, it is determined whether the single-person scene condition is met. The single-person scene condition includes that only one target person is detected within a preset video time window, and there are no other persons whose detection confidence exceeds a preset second confidence threshold and whose detection box area exceeds a preset area threshold. If the conditions are met, then n+1 video frames containing the face of the target person are continuously acquired to obtain a face image, where n is a preset first capacity; Facial landmarks are located for each face image, and facial correction is performed based on the facial landmarks. Based on each corrected face image, extract the corresponding face feature vector, and calculate the center feature based on the face feature vector; Calculate the similarity between each facial feature vector and the central feature, and take the n facial feature vectors with the highest similarity as the facial feature set of the target person; Calculate the similarity between each face feature vector in the face feature set of the target person and each face feature vector in the face feature set corresponding to each existing identity ID in the local identity feature database, and determine the maximum similarity between the target person and each existing identity ID; If the maximum similarity between the target person and an existing identity ID is greater than the preset registration deduplication threshold, then the target person is determined to have already registered. If the maximum similarity between the target person and each existing identity ID is less than or equal to the preset registration deduplication threshold, then the target person's identity ID is generated, and the target person's identity ID and facial feature set are written into the local identity feature database.

13. The user identification method for a mobile camera robot according to claim 12, characterized in that, The generation of the target person's identity ID includes: The identity ID of the target person is generated based on the robot device identifier, registration timestamp, preset serial number and random salt value, wherein the preset serial number is a locally incrementing serial number or a random serial number.

14. The user identification method for a mobile camera robot according to claim 12, characterized in that, If a face is detected, the process involves extracting a face feature vector and determining a candidate face identity ID based on the face feature vector and the face feature set corresponding to each existing identity ID in the local identity feature database. The process further includes: Determine whether the full-body pedestrian re-identification registration conditions are met. The full-body pedestrian re-identification registration conditions include that the full-body feature set corresponding to the face candidate identity ID is empty and meets the single-person scene condition and the full-body pedestrian re-identification gating condition. The full-body pedestrian re-identification gating condition is that the posture key points obtained through posture recognition include full-body key points and the confidence of each full-body key point is greater than or equal to a preset first confidence threshold. The full-body key points include left shoulder, right shoulder, left hip, right hip, left knee, right knee, left ankle, and right ankle. If satisfied, then based on the full-body key points, extract the full-body pedestrian re-identification feature vector and write the full-body pedestrian re-identification feature vector into the full-body feature set corresponding to the face candidate identity ID; If not, then determine whether the half-body pedestrian re-identification registration conditions are met. The half-body pedestrian re-identification registration conditions include that the half-body feature set corresponding to the face candidate identity ID is empty and meets the single-person scene condition and the half-body pedestrian re-identification gating condition. The half-body pedestrian re-identification gating condition is that the posture key points obtained through posture recognition include half-body key points and the confidence of each half-body key point is greater than or equal to the preset first confidence threshold. The half-body key points include the left shoulder, right shoulder, left hip and right hip. If the conditions are met, then based on the half-body key points, extract the half-body pedestrian re-identification feature vector, and write the half-body pedestrian re-identification feature vector into the half-body feature set corresponding to the face candidate identity ID.

15. The user identification method for a mobile camera robot according to claim 12, characterized in that, If a face is detected, the process involves extracting a face feature vector and determining a candidate face identity ID based on the face feature vector and the face feature set corresponding to each existing identity ID in the local identity feature database. The process further includes: The voiceprint registration window is determined based on the time of determination of the face candidate identity ID; Determine whether the voiceprint registration conditions are met. The voiceprint registration conditions include that the voiceprint feature set corresponding to the face candidate identity ID is empty, and that a valid voice segment with a continuous duration greater than a preset duration is detected in the voiceprint registration window and that the single-person scene condition and the sound source direction consistency condition are met. The sound source direction consistency condition is that the angle between the sound source direction estimated by the microphone array and the azimuth angle of the target image is less than or equal to a preset angle threshold. The azimuth angle of the target image is the azimuth angle of the target person relative to the optical axis of the camera. If the conditions are met, then based on the valid speech segment, the voiceprint feature vector is extracted and written into the voiceprint feature set corresponding to the face candidate identity ID.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-15.

Images