A robot anti-occlusion person re-identification system based on multi-part feature fusion

Abstract

The application discloses a robot anti-occlusion personnel re-identification system based on multi-part feature fusion, and relates to artificial intelligence which proposes the scheme in view of the problems of insufficient accuracy and the like in the prior art.The system performs the following steps when running: multi-part feature target detection and boundary box matching;spatial position information of a human body and a face is extracted from a video stream input by a robot camera, and a corresponding physical attribution relationship is established, thereby providing a structured input for subsequent feature extraction and fusion; a unified feature extraction model is constructed; a human body image block and a face image block are taken as inputs, channel normalization preprocessing is performed through the same standard statistics; operation is performed based on a confidence level grading matching strategy; mixed feature fusion and ID bidirectional binding; and feature database dynamic management is performed based on a dynamic forgetting strategy and an ID automatic merging mechanism.The advantage lies in that modular design is adopted, five key technical steps work cooperatively, and high-precision, low-delay and adaptive continuous identity perception is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and particularly relates to a robot anti-occlusion person re-identification system based on multi-part feature fusion. BACKGROUND

[0002] With the deep integration of artificial intelligence and Internet of Things technology, home service robots are gradually evolving from tools that execute single instructions to intelligent home companions with situational awareness and proactive service capabilities. In core application scenarios such as fall detection, emotional companionship, and proactive health monitoring, high-robustness person re-identification (ReID) technology has become a key prerequisite for realizing personalized interaction and accurate early warning. Specifically, in the fall detection task, the robot not only needs to identify abnormal postures in real time, but also needs to accurately associate the behavior to a specific family member, thereby triggering a customized alarm process for their emergency contacts or medical institutions; in the emotional companionship scenario, identity information is the basis for activating personalized dialogue strategies, calling historical interaction memories, and providing emotion-adaptive responses; and in proactive health monitoring, whether physiological parameters (such as heart rate, respiratory rate, blood pressure, etc.) obtained through vision, millimeter wave radar, or multi-part feature sensor fusion, must be accurately bound to the specific user's identity to build a personalized health profile with tracking value, supporting chronic disease management or early risk warning.

[0003] However, unlike traditional monitoring scenarios with fixed perspectives and controlled lighting, home service robots operate in highly dynamic and unstructured environments: perspectives frequently change, lighting conditions are complex (including low light at night), backgrounds are cluttered, occlusions are common, and users dress casually and have free postures. These factors collectively constitute a highly challenging "uncontrolled home visual environment". Under this background, existing ReID technologies - whether single-part methods or preliminary multi-part feature fusion solutions - are difficult to balance accuracy, robustness, and resource efficiency at the deployment end, exposing significant application bottlenecks.

[0004] 1. Status of ReID technology based on human appearance features

[0005] Human appearance-based ReID methods primarily rely on deep neural networks to extract discriminative global and local visual features from pedestrian images to construct identity embedding vectors. Current mainstream architectures include deep convolutional neural networks (such as ResNet and OSNet) and Vision Transformers based on self-attention mechanisms. These often combine strategies such as part alignment, channel attention, and feature decoupling to learn features from multiple dimensions, including color distribution, texture details, and semantic regions, and optimize intra-class compactness and inter-class discriminancy in the feature space through metric learning. While these methods have achieved good performance on standard datasets such as Market-1501 and DukeMTMC-reID, they face fundamental challenges in home robot scenarios. The phenomenon of "changing clothes" is extremely common in the home environment; users may change pajamas, bathrobes, or loungewear in a short period, causing features dependent on clothing appearance to quickly become invalid in identity matching across time periods or even hours, forming a typical "clothing-changing ReID" problem. Furthermore, when multiple family members wear similar styles of clothing or are in highly textured backgrounds (such as printed sofas or striped sheets), the inter-class discriminancy of appearance features significantly decreases, leading to a higher false recognition rate. Even more serious is that service robots typically observe from a low angle (looking up or at eye level). When users are standing, facing away, bending over, or severely obstructed by furniture, the effective human body area is greatly reduced, and even a large amount of background noise is mixed in, which greatly reduces the reliability of the embedding vector generated by the feature extractor and seriously affects the accuracy of subsequent matching.

[0006] 2. Current Status of ReID Technology Based on Facial Recognition

[0007] ReID methods based on facial recognition utilize the highly stable biometric feature of the face for identity determination, possessing inherent clothing invariance and achieving extremely high recognition accuracy in cooperative scenarios (such as access control and mobile phone unlocking). However, in the actual deployment of mobile home service robots, this technology is subject to multiple physical and environmental limitations. First, limited by the robot's height (typically the camera is 30–80 cm above the ground), the robot often shoots users from a low angle, making it difficult to capture a complete, frontal facial image when the user is standing or moving normally; when the user is sideways, looking down, facing away, or sitting, the face may even be completely invisible. Second, home life is a typical non-cooperative scenario—users do not actively face the camera or remain still, but move freely in a natural state. Under these conditions, factors such as low light at night, motion blur, large changes in posture, or strong reflections can easily lead to severe degradation of facial image quality, resulting in feature extraction failure or extremely low confidence. More importantly, relying solely on facial features ignores rich contextual clues such as body shape, posture, and gait. Once the face becomes unavailable, the system falls into an "identity blind spot," unable to maintain continuous tracking and identity continuation of the target, making it difficult to support health monitoring or behavior analysis tasks that require long-term identity consistency.

[0008] 3. Current Technological Status of Existing Fusion Methods

[0009] To overcome the limitations of single-feature fusion, researchers have begun exploring multi-feature fusion strategies for human and facial features. However, current fusion schemes remain rudimentary in terms of system architecture, decision-making mechanisms, and engineering implementation. Most methods employ a "two-stream network" structure, deploying separate face recognition backbone networks and human ReID backbone networks, then simply concatenating, weighting, or shallowly fusioning the output features. This design not only doubles the number of model parameters and computational overhead, significantly increasing inference latency and energy consumption at the deployment end, but also severely restricts the scalability of the system on computationally limited embedded robot platforms. More importantly, existing fusion logic lacks a dynamic evaluation mechanism for the confidence of features from each part. For example, when a face is visible but severely blurred (low confidence) while a human body is clear but in a changing state (feature drift), the system cannot adaptively adjust the feature weights, easily leading to incorrect decisions. Regarding spatiotemporal alignment, existing methods mostly rely on the IoU (Intersection over Union) or centroid distance between face and body detection boxes for attribution matching. However, in family scenarios where limbs are intertwined or mutually occluded, misidentification can easily occur. Furthermore, almost all existing systems neglect the long-term evolution management of the feature library—historical features accumulate indiscriminately without dynamic updates or forgetting mechanisms based on time decay, matching frequency, or feature freshness. This leads to old features interfering with new matches, decreased retrieval efficiency, and historical contamination, further weakening the system's stability and accuracy in long-term operation.

[0010] In summary, the human re-identification technology for home service robots urgently needs to overcome the limitations of existing fragmented, crudely integrated, and static management systems. It is necessary to develop a new generation of ReID deployment framework that is lightweight, adaptive, features collaborative across multiple parts, and possesses long-term memory evolution capabilities, so as to achieve reliable, continuous, and personalized identity perception and service response in real home environments.

[0011] However, the most significant and critical technical problem is the lack of a dynamic evaluation and adaptive fusion mechanism for the confidence of features from multiple body parts. This results in a significant decrease in the accuracy of identity matching in home scenarios such as changes in user clothing, changes in perspective, or partial occlusion, making it impossible to maintain long-term identity consistency.

[0012] Existing people re-identification technologies generally remain at the static, fragmented model inference stage, lacking a backend management framework capable of intelligent perception and dynamic response in complex home environments. These systems typically rely on pre-set single facial or body feature models, making it difficult to autonomously adjust to dynamic factors such as changes in user clothing, fluctuations in lighting conditions, perspective changes, or multi-person interactions after deployment. Especially in critical application scenarios—such as the need for instantaneous identity association and alarm triggering in the event of a fall, or the need for long-term binding of physiological data to specific users for health monitoring—existing ReID systems cannot perform online optimization for identity decisions based on real-time multi-source sensor information (such as visual quality, posture visibility, and historical match confidence). This "one-time matching, static storage, and no feedback updates" working mode results in poor robustness and weak continuity in real-world home environments. There is an urgent need to overcome the limitations of existing fragmented, coarsely integrated, and statically managed systems, and to develop a lightweight, adaptive, multi-feature collaborative, and long-term memory evolution-enabled next-generation ReID deployment framework to achieve reliable and continuous identity perception in real-world home environments.

[0013] The following problems still exist:

[0014] (1) The multi-part feature fusion mechanism lacks dynamic decision-making capability. Although current multi-part feature ReID schemes attempt to combine facial and body features, the fusion logic is highly static, usually using fixed weight splicing or simple averaging, without establishing an evaluation mechanism for the real-time reliability of each part feature. For example, when the face is blurred due to low brightness, but the body posture is complete and the clothing has not changed, the system may still rely on low-quality facial features; conversely, when the user changes clothes but the front is clearly visible, it may misjudge due to body feature drift. This fusion method, which lacks adaptive weight adjustment and conflict resolution capabilities, makes the system prone to incorrect identity determination in typical home interference scenarios, seriously weakening service accuracy and user trust.

[0015] (2) Lack of Feature Database Management Mechanism: Existing systems generally accumulate and store all historically extracted identity features indiscriminately, lacking dynamic maintenance strategies based on time decay, matching frequency, or feature freshness. Over time, old clothing features from several months ago, as well as some low-quality features extracted under low signal-to-noise ratio conditions, continue to occupy database resources and interfere with the current matching process when the similarity between old and new features is close, forming a "historical pollution" effect. At the same time, the system cannot automatically update or discard outdated representations based on changes in user body shape (such as elderly people becoming thinner, children growing up), changes in hairstyle, or returning home after a long period of absence. This static and closed feature management model restricts the stability, scalability, and personalized service capabilities of ReID systems in long-term deployment in home environments.

[0016] (3) Redundant system architecture makes it difficult to meet the lightweight and real-time requirements of embedded platforms. To achieve multi-part feature recognition, the mainstream solution uses dual-stream independent networks to process face and body images separately, resulting in large model size, high computational overhead, and significant inference latency. For home service robots with limited computing power, memory, and power consumption, such a high-resource-consuming architecture is difficult to support the high-frequency, low-latency identity recognition requirements. Especially in emergency scenarios such as fall detection that require millisecond-level response, performance bottlenecks may directly affect user safety. In addition, the complex dual-stream structure also increases the engineering difficulty of model deployment, version updates, and troubleshooting, hindering the productization and large-scale application of the technology. Summary of the Invention

[0017] The purpose of this invention is to provide a robot anti-occlusion person re-identification system based on multi-part feature fusion, so as to solve the problems existing in the prior art.

[0018] The robot anti-occlusion person re-identification system based on multi-part feature fusion described in this invention performs the following steps during operation:

[0019] Step 1: Multi-part feature target detection and bounding box matching; extract the spatial location information of the human body and face from the video stream input by the robot camera, and establish the corresponding physical attribution relationship to provide structured input for subsequent feature extraction and fusion;

[0020] Step 2: Construction of a unified feature extraction model; using human body image patches and face image patches as input, channel normalization preprocessing is performed using the same standard statistics;

[0021] Step 3: Perform calculations based on a confidence-based hierarchical matching strategy;

[0022] Step 4: Hybrid Feature Fusion and Two-Way ID Binding;

[0023] Step 5: Dynamically manage the feature database based on a dynamic forgetting strategy and an automatic ID merging mechanism.

[0024] The robot anti-occlusion person re-identification system based on multi-part feature fusion described in this invention has the advantages of solving the core problems of existing technologies, such as poor robustness of identity recognition in dynamic and uncontrolled home environments, rigid fusion mechanisms, high resource consumption, and insufficient long-term operational stability. The system adopts a modular design and achieves high-precision, low-latency, and adaptive continuous identity perception through the collaborative work of five key technical steps.

[0025] The following significant technical effects were achieved:

[0026] (1) Significantly improve the robustness and accuracy of recognition in complex family scenarios: By integrating features of both face and body parts and using unified embedding space and batch inference optimization, the system can still output high confidence matching under typical family interferences such as occlusion (such as wearing a mask, facing away from the camera), changes in lighting, and diverse postures.

[0027] (2) Achieve seamless binding and consistency maintenance of cross-part identity: Based on the confidence-based hierarchical matching strategy and bidirectional hash mapping, the system realizes automatic, reliable and reversible binding of face ID and body ID. Even if the feature of a certain part is temporarily invalidated (such as when the user looks down and the face is not visible), the system can still maintain continuous identity tracking through another part, which solves the identity separation problem of "face recognition is A, body recognition is B" in the traditional solution.

[0028] (3) Effectively suppress the “catastrophic growth” of the feature library and ensure long-term operational stability: The high-confidence multi-match ID merging mechanism successfully solves the problem of “one person ID”; the noise ID removal mechanism automatically removes visitor or noise identities.

[0029] (4) Supporting the long-term reliable operation of upper-layer intelligent services: The above technologies together construct a self-maintaining, self-purifying, and highly consistent identity perception foundation, providing a stable identity basis for key services such as fall detection alarms, personalized interaction, and health behavior analysis. Users do not need to manually intervene in the feature library maintenance; the system can automatically complete identity updates, noise removal, and ID merging, supporting continuous operation for more than 30 days without restarting or resetting. Attached Figure Description

[0030] Figure 1 This is a flowchart illustrating the steps involved in running the system described in this invention.

[0031] Figure 2 This is a flowchart illustrating step three of the present invention. Detailed Implementation

[0032] like Figure 1 , Figure 2As shown, the robot anti-occlusion re-identification system based on multi-part feature fusion described in this invention is deployed for home service robots and performs the following steps during operation:

[0033] Step 1: Multi-part feature target detection and bounding box matching;

[0034] This step is the perception front-end of the entire system, responsible for extracting the spatial location information of the human body and face from the video stream input from the robot's camera, and establishing the physical attribution relationship between them, providing structured input for subsequent feature extraction and fusion. It includes the following sub-steps:

[0035] S11. Parallel detection of features across multiple locations;

[0036] For each frame of input image , Indicates the height of the image. The width represents the image width, and 3 represents the three color channels of the image. Human detection and face detection tasks are performed in parallel.

[0037] The human detection task employs the lightweight YOLOv8 model, which maintains high detection accuracy while exhibiting low computational complexity, making it suitable for embedded platforms. A human confidence threshold is set. Detected If a human body bounding box is generated, the output set of human body bounding boxes will be:

[0038] ;

[0039] Each human body bounding box This represents the coordinates and dimensions with the top-left corner as the origin. Represents the set of integers. The meanings of each component are as follows: It is the x-coordinate of the top left corner of the human body bounding box; It is the ordinate of the top left corner of the human body bounding box; It is the width of the human body bounding box; It is the width of the human body bounding box.

[0040] The face detection task employs the MediaPipe Face Detection model, developed by Google and optimized for mobile and edge devices, featuring high real-time performance and low power consumption. A face confidence threshold is set. Detected If the individual face bounding box is obtained, the output set of face bounding boxes will be:

[0041] ;

[0042] Each face bounding box This represents the coordinates and width / height with the top-left corner as the origin. The meanings of each component are as follows: It is the x-coordinate of the top left corner of the face bounding box; It is the ordinate of the top left corner of the face bounding box; It is the width of the face bounding box; It is the width of the face bounding box.

[0043] The two detection tasks run independently and do not interfere with each other, ensuring that if the features of one part fail (such as when the face is turned away and no face is visible), the other part can still provide effective perception.

[0044] S12. Run the cross-feature bounding box matching algorithm;

[0045] Traditional methods often use Intersection over Union (IoU) or Euclidean distance to associate faces with human bodies, but these methods are prone to mismatches in densely populated family settings (such as when family members are sitting together or children are climbing). To address this, this invention employs an asymmetric Intersection over Reference (IoR) algorithm, which is based on the prior knowledge that "a face must be completely located within the corresponding human body region" for calculation.

[0046] Define the face bounding box human body bounding box The asymmetric inclusion rate is:

[0047] ;

[0048] in, This represents the area of ​​the bounding box. Intersection region. The coordinates are calculated using the following formula:

[0049] ;

[0050] ;

[0051] ;

[0052] ;

[0053] If satisfied and Then the area of ​​intersection is:

[0054] ;

[0055] Otherwise, the intersection area is 0. Based on the above definition, construct a space of size... Asymmetric inclusion matrix , of which elements Filter out all that meet the criteria. Candidate bounding box pairs are selected, and a greedy matching strategy is employed: all valid candidate pairs are sorted in descending order of asymmetric containment rate, and the bounding box pairs with the highest asymmetric containment rate that are not matched on either side are locked in turn, thereby avoiding duplicate binding in multi-person scenarios. In particular, when multiple candidate pairs have the same asymmetric containment rate (e.g., a child's face is contained by both its own body and an adult's body bounding box), their intersection-union ratio (IoU) is further compared:

[0056] ;

[0057] The area of ​​the union region is calculated using the following formula:

[0058] ;

[0059] The pair with the larger intersection-union ratio (IUGR) is selected as the final matching result. All detected targets are divided into the following three states:

[0060] First state: Successfully associated face-body pair, indicating that the target has complete multi-part feature information;

[0061] Second state: Only human body detected (human body that did not successfully match the face, such as the user's back is facing away, at a distance, or with the head covered).

[0062] The third state: only faces were detected (faces that were not successfully paired with a human body, such as when the user's body is obscured by furniture and only the head is visible). This classification result will serve as the basis for subsequent feature extraction and fusion strategies, significantly improving the accuracy and robustness of multi-part feature data alignment.

[0063] Step 2: Construction of a unified feature extraction model;

[0064] This step aims to address the core issues of "system architecture redundancy and high resource consumption" in the background technology. Traditional multi-part feature ReID systems generally adopt a "dual-stream network" architecture, i.e., deploying separate face recognition backbone networks and body ReID backbone networks, resulting in a doubling of model parameters, high memory consumption, and large inference latency, making it difficult to run efficiently on embedded robot platforms such as Jetson. To address this, this invention designs a unified feature extraction model based on parameter sharing, which simultaneously encodes face and body images through a single neural network, significantly reducing computational overhead and improving throughput efficiency.

[0065] The unified feature extraction model uses Vision Transformer (ViT) as the unified backbone network. ViT is a deep neural network model based on a self-attention mechanism, which can effectively model long-distance dependencies in images and is suitable for simultaneously capturing local texture details of faces and global semantic structures of the human body.

[0066] The unified feature extraction model is configured to accept two different sizes of image patches as input:

[0067] Human body image patch: scaled to 256×128 pixels;

[0068] Face image patch: scaled to 112×112 pixels.

[0069] Despite the different input sizes, both undergo the same normalization preprocessing procedure to ensure consistent data distribution.

[0070] Channel normalization was performed using standard statistics from the ImageNet dataset:

[0071] mean ;

[0072] Standard deviation .

[0073] This standardization strategy can improve the model's generalization ability and reduce the interference of lighting and color deviations on feature extraction.

[0074] The unified feature extraction model ultimately outputs a 3072-dimensional L2-normalized feature vector. This high-dimensional embedding space exhibits good discriminative power, and after L2 normalization, the Euclidean distance between any two feature vectors is equivalent to the cosine distance, simplifying subsequent similarity calculations. ViT's self-attention mechanism naturally supports variable-length sequence inputs, thus eliminating the need to design dedicated subnetworks for different body parts. All weights in the Transformer encoding layer are fully shared between the face and body paths, achieving true parameter reuse.

[0075] The feature extraction process of the unified feature extraction model follows a standardized pipeline of "preprocessing → batch inference → postprocessing", including the following sub-steps:

[0076] S21. Tensor Quantization Preprocessing: Convert the original BGR format image to RGB and scale it to the target size using bilinear interpolation. Then, calculate the mean value. with standard deviation Perform channel-by-channel normalization to generate a four-dimensional tensor. ,in This represents the total number of regions of interest to be processed, i.e., the total number of region images containing human body bounding boxes and face bounding boxes.

[0077] S22. Batch Inference: Concatenate all regions of interest into a single batch processing tensor, complete all feature encoding through a single forward propagation using the GPU, and output the original feature set. ,in , It represents the set of real numbers.

[0078] S23.L2 normalization post-processing: Perform unit sphere projection on each feature vector, as shown in the following formula:

[0079] ;

[0080] in For numerically stable terms, to prevent division by zero errors. Feature set. Divided into human body parts and the face This will serve as the query vector for subsequent steps.

[0081] ;

[0082] ;

[0083] The normalized feature vectors lie on the unit hypersphere, which allows cosine similarity to be directly calculated through the vector inner product, thus improving retrieval efficiency.

[0084] This step achieves a high degree of resource efficiency: by replacing the traditional dual-stream convolutional neural network architecture with a single VIT backbone network that shares parameters, the system reduces the number of model parameters by approximately 50% and GPU memory usage by over 40%, significantly alleviating memory pressure on embedded platforms. Secondly, it improves inference efficiency: leveraging a dynamic batching mechanism, the system concatenates all regions of interest within the same frame into a single batch tensor, completing all feature encoding through a single GPU forward propagation. This fully releases the hardware's parallel computing capabilities, achieving a 3-5x improvement in feature extraction speed on the NVIDIA Jetson Orin platform, with end-to-end latency controlled within 80ms, meeting real-time interaction requirements. More importantly, since face and body features are generated from the same set of network weights, they naturally reside in the same high-dimensional embedding space, with semantically aligned distributions. This avoids cross-part feature offset issues, providing a solid and consistent semantic foundation for subsequent confidence-based hierarchical matching, bidirectional ID binding, and feature fusion, significantly improving the robustness and accuracy of multi-part feature identity decisions.

[0085] Step 3: Perform calculations based on a confidence-based hierarchical matching strategy;

[0086] This step aims to address the core issues in the background technology, such as "tracking interruption caused by hard threshold determination" and "lack of adaptability to feature drift." Traditional ReID systems use a single similarity threshold for identity determination. Once feature shifts occur due to changes in clothing, posture, or lighting interference, known users are incorrectly identified as strangers. Therefore, this invention provides a three-level confidence-driven hierarchical matching strategy, combining GPU parallel tensor retrieval and feature slot incremental update mechanisms to achieve efficient and robust identity association. Specifically, it includes the following sub-steps:

[0087] S31. Dynamic Feature Library Structure and Parallel Similarity Calculation; For human features, N is set as the number of currently registered identities, initially... Each new user Increment by one, A fixed number of feature slots are allocated to each ID. For the feature embedding dimension, the system organizes the global feature database into a dynamically expanding three-dimensional tensor. It is always placed in the GPU memory to support high-speed parallel computing. The calculation formula is:

[0088] ;

[0089] For each ID There are 1 feature slots, and unused slots are initialized to an all-zero vector. The used slots store valid feature vectors. The system synchronously maintains a matching counter tensor of the same shape. :

[0090] ;

[0091] in, Represents the set of non-negative integers. A counter records the number of times each feature slot is successfully matched. Human query features output by the unified feature extraction module. satisfy:

[0092] ;

[0093] The system performs fully connected cosine similarity calculations on the GPU to obtain the similarity tensor. As a confidence level:

[0094] ;

[0095] Calculate the similarity between all query features and all features under all IDs in one go, and take the maximum value in the feature dimension:

[0096] ;

[0097] in: Indexes representing query features ; This indicates the index for querying the ID. ; This represents the feature index of a specific ID being queried. .

[0098] Obtain the two-dimensional confidence matrix Its elements Indicates the first The query feature in the first The maximum confidence score matched among all features of an ID. Since empty features are zero vectors, the similarity is always 0, which does not affect the selection of the maximum value.

[0099] S32. Three-level confidence interval division and decision logic: For each query target, based on the first... Taking a single query target as an example, the system selects the largest response ID:

[0100] ;

[0101] ;

[0102] in, This indicates the ID index corresponding to the maximum response confidence of the query target. This represents the maximum confidence level of the query target.

[0103] Based on this result, the system presets two thresholds: a high confidence threshold. Low confidence threshold The matching results are divided into the following three intervals, and differential processing is performed to return the matching index. :

[0104] High confidence interval The judgment logic is to confirm a match with a known identity ID index. The system then takes the following action: locates the specific slot index that yields the maximum response. ;Execute the counter for the corresponding slot ; at index Identity ID Within each feature, according to Perform bubble sort to shift high-frequency features forward; return the identity ID index. As the final matching result, .

[0105] Medium confidence interval The judgment logic is as follows: if the highest similarity between the current query feature and any known identity ID is lower than the high confidence threshold but higher than the low confidence threshold, it indicates that there may be appearance changes or observation condition degradation. Then the system takes the following action: [The system then proceeds to the next step, using the index...] Identity ID Among the features, find the index of the first zero vector. If it exists, write the current query feature to the corresponding slot. and set If none of the slots have a zero vector, use the least access update strategy to overwrite the last slot. , and set Return the identity ID index. As the final matching result, .

[0106] Low confidence interval If the system determines that there is no valid match, it is considered a new user. The system then performs the following action: dynamically expands the feature library... and Join a new row of slots along the first dimension. and The shape becomes The new dimensions are all filled with zero vectors, and let Increment by one; the first slot of the new ID is written with the characteristic. ;set up Return the new ID index. As the final matching result, .

[0107] The confidence-based hierarchical matching strategy proposed in this step demonstrates significant overall performance advantages in a real-world home environment. Thanks to the all-GPU parallel tensor similarity calculation mechanism, the system can complete a global retrieval of the dynamically expanded feature library within milliseconds. Real-world testing on the NVIDIA Jetson Orin platform shows an end-to-end matching latency of less than 10ms, fully meeting the real-time identity perception requirements of home robots.

[0108] To verify the effectiveness of this mechanism, a deployment test was conducted in a real-world home environment containing three family members and running continuously for 14 days: the system was configured with a ViT backbone network (input size: human body 256×128, face 112×112), and each ID was assigned... Each feature slot, confidence threshold Experimental results show that the strategy maintains high robustness under complex interference—in typical scenarios such as users changing their home clothes daily, low-light conditions at night, and sitting / back-facing postures, the success rate of continuous identity tracking (without splits or interruptions) reaches 96.3%. Compared with traditional ReID methods that use fixed threshold matching (such as ResNet50 + a single cosine threshold of 0.7), the false recognition rate is reduced from 28.7% to 4.9%, with the most significant improvement in the "clothing change ReID" subtask.

[0109] Meanwhile, by independently maintaining matching counters and performing internal bubble sorting for each of the 10 feature slots under each identity ID, the system can precisely distinguish the validity of different appearances of the same user. Frequently occurring typical clothing or frontal poses naturally move to higher priority positions, while randomly captured low-quality or abnormal poses sink to the end due to fewer matching attempts, effectively avoiding interference from noisy features on high-confidence matches. Building on this, the incremental update mechanism in the medium-confidence interval enables the system to continuously learn: when a user changes clothes, hairstyle, or appears from a new angle, the system automatically writes this appearance as a new feature snapshot into the corresponding ID's empty slot. If the slot is full, it replaces the lowest-frequency feature, thus achieving adaptive evolution to long-term challenges such as "clothing-changing ReID." This mechanism not only significantly improves the robustness and continuity of identity recognition but also effectively prevents historical contamination through the natural elimination of low-frequency features, ensuring that the system automatically maintains itself over months or even years to maintain high accuracy and efficiency.

[0110] Step 4: Hybrid Feature Fusion and Two-Way ID Binding;

[0111] This step is the core of the system's cross-part identity consistency and conflict resolution, aiming to solve problems such as "coarse fusion logic," "weak conflict handling capability," and "fragile frame correlation" in the background technology. Existing methods often suffer from identity mismatch or tracking interruption when face and body features are inconsistent due to the lack of an effective arbitration mechanism. To address this, this invention designs a hybrid fusion mechanism based on a bidirectional ID mapping graph. By establishing a dynamic binding relationship between body ID and face ID and introducing a face-first conflict decision strategy, it ensures that the system can maintain highly robust identity continuity even when any part of the feature fails or conflicts. Specifically, it includes the following sub-steps:

[0112] S41. Construction of a two-way ID binding graph; the system maintains two hash mapping tables in memory, forming a two-way ID binding graph: hash mapping tables This is a one-way mapping dictionary from human ID to face ID, in the form of: Hash Map This is a reverse mapping dictionary from face ID to body ID, in the form of: .in It is a human ID. It's a facial recognition ID.

[0113] The two hash maps are dynamically updated during system operation and consistency is guaranteed by a mutex lock mechanism: at any given time, one Bind to at most one Conversely, when a new user is first identified and possesses both high-quality facial and body features, the system assigns a unique identifier to that user. and And register the binding relationship in both tables simultaneously.

[0114] S42. A differentiated fusion strategy based on detection state; before performing fusion, the system calls the confidence-level matching module separately for the multi-part feature input of each target to obtain mutually independent human body matching results and face matching results; the human body matching result is denoted as... ;in, This represents the queried human ID; This represents the maximum confidence level of the human body feature. The face matching result is denoted as... ;in; This indicates the face ID found in the query. It is the return to step three. A matching index for individual facial features; This represents the maximum confidence level for all facial features. It is the third step Maximum confidence level of individual face query features.

[0115] Subsequently, based on the target state output in step one, the system executes the following fusion logic:

[0116] For the first state, both human bodies and faces were successfully detected and associated; the query index for the associated human bodies and faces is... and The system is based on and The combination of these factors leads to four sub-cases;

[0117] The first seed case is a double confirmation: if and Then check and Has it been bound in the bidirectional graph? If they match, confirm the identity; if they conflict, prioritize trusting the facial ID. The human ID will be forcibly corrected using the following formula:

[0118] ;

[0119] .

[0120] The second seed scenario is face-dominant: if... and The system uses the face ID as the standard and writes the current human features into the human feature database corresponding to the current face ID in an incremental update manner with medium confidence. If the human features are subsequently matched with multiple identities, the ID merging mechanism in step five will be used for post-verification.

[0121] The third scenario is dominated by the human body: if and The system uses human ID as the standard, ignores low-confidence faces, and only updates according to the logic in step three. And use bubble sort to maintain identity continuity.

[0122] The fourth seed scenario is a double-blind scenario: if and The target is identified as entirely new. The system executes the following steps: concatenates the face and body feature databases along the first dimension to create a new row of slots; writes the current face and body features to their respective slots (slot 0); initializes the corresponding counters to 1; and then... and Register in China and Two-way binding.

[0123] In the second state, only a human body is detected; the system performs matching and updating operations solely based on human body features, without incorporating information about facial features. This strategy ensures that even when a user is in a situation where their back is turned, their head is down, or their face is obscured, the system can still maintain stable identity tracking based on the user's appearance, avoiding identity interruption due to the loss of a single feature.

[0124] In the third state—only a face is detected—the system uses facial features as the sole input and makes independent decisions strictly following the confidence-level matching strategy in step three. Regardless of whether the human body is visible, as long as the facial features meet the medium confidence level or higher, identity verification or incremental learning can be completed, effectively ensuring identity continuity in partially visible scenarios.

[0125] The hybrid feature fusion and two-way ID binding mechanism used in this step demonstrates excellent robustness and adaptability in real-world home environments. By constructing a two-way ID binding graph, the system achieves dynamic and consistent binding between human body identity and facial identity, fundamentally avoiding the "misattribution" error in multi-part feature association. In scenarios with conflicting identification of different body parts, the system adopts a "face-first" arbitration strategy—fully leveraging the advantage of the face as a highly stable biometric feature. Even when features drift due to changes in clothing, occlusion, or posture, the system can still confirm identity with high confidence and proactively incorporate the new human appearance into the feature library of the corresponding identity with medium confidence, achieving unsupervised incremental learning of changes in the user's daily attire. Simultaneously, this mechanism ensures identity continuity when any part is missing for an extended period: when the user is continuously facing away or in a low-angle blind spot, the system relies on human body features to maintain tracking; when only the head is visible or the body is obscured by furniture, it relies entirely on the face for independent decision-making, effectively covering various non-cooperative interaction scenarios in the home environment. Furthermore, the entire fusion process is seamlessly integrated with the dynamically expanded feature library tensor structure. All operations are based on GPU computation, eliminating the need for additional pointer management or cross-part synchronization overhead. This ensures both the logical reliability of identity binding and the computational efficiency and memory controllability of the embedded platform. Ultimately, this mechanism significantly improves the long-term identity perception stability of home service robots in complex and dynamic home environments, providing a solid identity foundation for core applications such as fall detection, health record construction, and emotional companionship.

[0126] Step 5: Dynamically manage the feature database based on a dynamic forgetting strategy and an automatic ID merging mechanism;

[0127] To address the "catastrophic growth" problem caused by the unlimited accumulation of features during long-term system operation and to prevent outdated features from interfering with current matching, this invention designs a lightweight, adaptive feature database lifecycle management mechanism. This mechanism comprises two core functions: (1) a dynamic forgetting strategy based on matching frequency and timestamps; and (2) an automatic ID merging mechanism for high-confidence matching conflicts. These two mechanisms work together to maintain the discriminative power and timeliness of the feature database under limited memory resources.

[0128] The dynamic forgetting mechanism specifically involves: for any query ID... After any update of the counter, if the total number of matches is ;in This represents the attenuation threshold, which can be configured according to requirements to perform integer-downward attenuation on all counters: ;wherein the attenuation factor Simulating the natural forgetting process of human memory, the system avoids outdated but frequently used features (such as winter clothes from a year ago) from holding high priority for extended periods. The system periodically performs garbage collection (default period is 1 hour, configurable), indexing any recorded ID. If the total number of matches satisfies If it is, it will be judged as a noise ID. This represents the threshold for identifying noise IDs, which can be configured according to requirements. Once an ID is identified as noise, the system performs a cleanup operation: from the feature tensor... and counter tensor Physical deletion index is ID data; from a bidirectional hash table and Remove all IDs involved. The entry is released; the corresponding ID index is released so that it can be reused for subsequent new user registrations.

[0129] During long-term system operation, due to changes in user attire, diverse postures, or incomplete feature coverage during early registration, the same family member may be incorrectly assigned multiple independent identity IDs (i.e., the "one person, multiple IDs" problem). To address this identity fragmentation phenomenon, this invention designs a lightweight automatic ID merging mechanism that decouples features across different body parts and triggers them independently. This mechanism independently monitors the facial feature database and the human body feature database. When a high-confidence multiple match is detected for any body part, the corresponding ID merging process is initiated, thereby maintaining the reliability of identity verification while ensuring recall rate.

[0130] The automatic ID merging mechanism is as follows: After the confidence-level matching in step three, the system separately compiles high-confidence matching ID sets for the current query target under both facial features and body features. (High-confidence facial ID set) , Indicates that the index is The face query features are indexed in the ID feature library. Confidence level: High-confidence ID set for humans , Indicates that the index is The human body query features are indexed in the ID feature database as follows: Confidence level: .in, A preset high confidence threshold is used. The merge operation is triggered when any of the following conditions are met: If If so, then merge the face ID space; If the condition is met, the human ID space will be merged. The two checks are independent of each other and can be triggered simultaneously or separately.

[0131] For sets All indexes Calculate its total matching frequency ;according to Sort in descending order to obtain an ordered list: Select As the primary ID; select As a secondary ID; This is considered a redundant ID.

[0132] exist Among the feature slots, the slot with the largest positioning counter. Extract the corresponding feature vectors ;calculate With the main ID Maximum similarity of all features ;like ,Will Write using the medium confidence incremental update strategy from step three. Remove the secondary ID and redundant ID from the feature tensor. and counter tensor Delete it; if the ID to be deleted is in or If a corresponding entry exists, it will be removed synchronously; otherwise, it will be skipped.

[0133] The dynamic feature database management mechanism proposed in this invention demonstrates significant technical advantages in ensuring long-term system stability and identity recognition accuracy. First, by introducing a dual-forgetting strategy based on matching frequency and timestamps, the system effectively suppresses the "catastrophic growth" problem, strictly controlling the feature database size to a constant level within limited embedded memory, thus avoiding increased retrieval latency and storage overflow risks caused by the infinite accumulation of historical features. Second, the global frequency decay mechanism simulates the natural forgetting process of human memory, gradually reducing the weight of outdated but frequently used features, thereby freeing up priority space for more recent and representative appearance snapshots, significantly improving the timeliness and discriminative power of the feature database. More importantly, this mechanism integrates ID merging logic under high-confidence multi-match conditions, precisely solving the problem of "one person, multiple IDs" caused by early misregistration or long-term offline absences.

[0134] For those skilled in the art, various other corresponding changes and modifications can be made based on the technical solutions and concepts described above, and all such changes and modifications should fall within the protection scope of the claims of this invention.

Claims

1. A robot anti-occlusion person re-identification system based on multi-part feature fusion, characterized in that, The following steps are executed at runtime: Step 1: Multi-part feature target detection and bounding box matching; extract the spatial location information of the human body and face from the video stream input by the robot camera, and establish the corresponding physical attribution relationship to provide structured input for subsequent feature extraction and fusion; Step 2: Construction of a unified feature extraction model; using human body image patches and face image patches as input, channel normalization preprocessing is performed using the same standard statistics; the unified feature extraction model uses Vision Transformer as a unified backbone network, models long-distance dependencies in images based on a self-attention mechanism, and simultaneously captures local texture details of the face and global semantic structure of the human body. Step 3: Perform calculations based on a confidence-based hierarchical matching strategy; Includes the following sub-steps: S31. Dynamic feature library structure and parallel similarity calculation; obtaining a two-dimensional confidence matrix. ,element Indicates the first The query feature in the first The highest confidence score matched among all features of an ID; S32. Three-level confidence interval division and decision logic: Divide the matching results into three confidence intervals, perform differential processing, and return the matching index. ; Step 4: Hybrid Feature Fusion and Two-Way ID Binding; Includes the following sub-steps: S41. Construction of a two-way ID binding graph; The system maintains two hash mapping tables in memory to form a two-way ID binding graph; S42. Differentiated fusion strategy based on detection state; Before fusion, the system calls the confidence-level matching module for the multi-part feature input of each target to obtain independent human body matching results and face matching results; Based on the target state output in step one, the system executes the fusion logic: For the first state, the associated query index for human body and face is: and The system is based on and The combination; For the second state, matching and update operations are performed based on human body features; For the third state, facial features are used as the sole input; Step 5: Dynamically manage the feature database based on a dynamic forgetting strategy and an automatic ID merging mechanism; The dynamic forgetting strategy specifically involves: for any query ID... After any update of the counter, if the total number of matches is ;in This represents the decay threshold, which is applied to all counters by rounding down to the nearest integer. ;wherein the attenuation factor The system periodically performs garbage collection on any recorded ID index. If the total number of matches satisfies If it is, it will be judged as a noise ID. Indicating a noise ID discrimination threshold, the system performs a cleanup operation: from the feature tensor and counter tensor Physical deletion index is ID data; from a bidirectional hash table and Remove all IDs involved. The entry; release the corresponding ID index.

2. The robot anti-occlusion person re-identification system based on multi-part feature fusion according to claim 1, characterized in that, Step one includes the following sub-steps: S11. Parallel detection of multiple feature parts; for each frame of input image , Indicates the height of the image. The width of the image is represented by 3, and the three color channels of the image are represented by 3. Human detection and face detection tasks are performed in parallel. S12. Run the cross-feature bounding box matching algorithm; define the face bounding box. human body bounding box The asymmetric inclusion rate is: ;in, Indicates the area of ​​the bounding box; Construct a size of Asymmetric inclusion matrix , of which elements ; Filter out all that meet the requirements Candidate bounding box pairs are selected, and a greedy matching strategy is adopted: all valid candidate pairs are sorted in descending order of asymmetric inclusion rate, and the bounding box pairs with the highest asymmetric inclusion rate and neither pair has been matched are locked in turn; when multiple candidate pairs have the same asymmetric inclusion rate, the intersection-union ratio (IoU) is compared; the pair with the larger IoU is selected as the final matching result; all detected targets are divided into the following three states: First state: Successfully associated face-body pair; Second state: Human body detected only; Third state: Only faces were detected.

3. The robot anti-occlusion person re-identification system based on multi-part feature fusion according to claim 2, characterized in that, Step two includes the following sub-steps: S21. Tensor quantization preprocessing: Convert the original BGR format image to RGB and scale it to the target size using bilinear interpolation; then, calculate the mean value... with standard deviation Perform channel-wise normalization to generate a four-dimensional tensor. ,in This represents the total number of regions of interest to be processed, i.e., the total number of region images containing human body bounding boxes and face bounding boxes. S22. Batch Inference: Concatenate all regions of interest into a single batch processing tensor, complete all feature encoding through a single forward propagation using the GPU, and output the original feature set. ,in , Represents the set of real numbers; S23.L2 normalization post-processing: Perform unit sphere projection on each feature vector, as shown in the following formula: ； in This is a numerically stable term to prevent division by zero errors; Feature set Divided into human body parts and the face This will serve as the query vector for subsequent steps. ； ； The normalized eigenvectors lie on the unit hypersphere.

4. The robot anti-occlusion person re-identification system based on multi-part feature fusion according to claim 3, characterized in that, In step three, for the high confidence interval The judgment logic is to confirm a match with a known identity ID index. The system then takes the following action: locates and obtains the specific slot index that yields the maximum response. ;Execute the counter for the corresponding slot ; at index Identity ID Within each feature, according to Perform bubble sort to shift high-frequency features forward; Return identity ID index As the final matching result, .

5. The robot anti-occlusion person re-identification system based on multi-part feature fusion according to claim 4, characterized in that, In step three, for the medium confidence interval The judgment logic is as follows: if the highest similarity between the current query feature and any known identity ID is lower than the high confidence threshold but higher than the low confidence threshold, then the system takes the following action: [The text abruptly ends here, likely due to an incomplete sentence or missing information.] Identity ID Among the features, find the index of the first zero vector. ; If it exists, write the current query feature to the corresponding slot. and set ; If none of the slots have a zero vector, use the least access update strategy to overwrite the last slot. , and set Return the identity ID index. As the final matching result, .

6. The robot anti-occlusion person re-identification system based on multi-part feature fusion according to claim 5, characterized in that, In step three, for the low confidence interval If the system determines that there is no valid match and the user is a new user, then the system will take the following actions: dynamically expand the feature library. and Join a new row of slots along the first dimension. and The shape becomes The new dimensions are all filled with zero vectors, and let Increment by one; the first slot of the new ID is written with the characteristic. ;set up Return the new ID index. As the final matching result, .

7. The robot anti-occlusion person re-identification system based on multi-part feature fusion according to claim 6, characterized in that, In step five, the automatic ID merging mechanism specifically involves: after the confidence-based hierarchical matching in step three, the system separately compiles high-confidence matching ID sets for the current query target under facial features and human body features; the high-confidence facial ID set... , Indicates that the index is The face query features are indexed in the ID feature library. Confidence level: Human high-confidence ID set , Indicates that the index is The human body query features are indexed in the ID feature database as follows: Confidence level: ;in, A preset high confidence threshold is set; the merging operation is triggered when any of the following conditions are met: If If so, then merge the face ID space; Then, the human ID space will be merged. For sets All indexes Calculate the total matching frequency ;according to Sort in descending order to obtain an ordered list: Select As the primary ID; select As a secondary ID; Consider it a redundant ID; exist Among the feature slots, the slot with the largest positioning counter. Extract the corresponding feature vectors ;calculate With the main ID Maximum similarity of all features ;like ,Will Write using the medium confidence incremental update strategy from step three. Remove secondary and redundant IDs from the feature tensor. and counter tensor Delete it; if the ID to be deleted is in or If a corresponding entry exists, it will be removed synchronously; otherwise, it will be skipped.

Images