Anti-occlusion group re-identification method and system based on multi-granularity cross-modal representation

Abstract

The invention discloses an anti-occlusion group re-identification method and system based on multi-granularity cross-modal representation, and the method comprises the steps: constructing a member set for members in images captured by different cameras, and carrying out the group matching; respectively marking a bounding box for each member, calculating the average spatial density of each member, establishing a local candidate subset, identifying sheltered and unsheltered members, and marking the sheltered and unsheltered members; for shielded members, dividing human body parts and key points, calculating the visibility of each part, and identifying shielded parts; a double-layer weighting strategy is designed, local part features of all parts of the members are obtained, the local part features are spliced, and enhanced member features are generated; and establishing an overall loss function to supervise model training, and completing re-identification of the occlusion group in the camera image. According to the method and the corresponding system, the positioning precision of the shielding group and the shielding part is improved, and the calculation overhead is remarkably reduced, so that quick reasoning can be realized under the condition that the precision is not influenced.

Description

Technical Field

[0001] This invention relates to a processing method for group pedestrian re-identification and the cross-field of cross-camera target association in intelligent security monitoring. Specifically, it relates to an anti-occlusion group re-identification method based on multi-granularity cross-modal representation, and also to an anti-occlusion group re-identification system based on multi-granularity cross-modal representation. Background Technology

[0002] Person re-identification (ReID) aims to identify individuals across different images using similarity metrics. Despite significant advancements in deep learning, existing ReID techniques struggle to meet practical application needs due to challenges such as occlusion, lighting variations, resolution differences, and complex interpersonal interactions. Therefore, group re-identification (GReID) has emerged as an important research direction, given that people often move in groups in real life. Unlike ReID, GReID requires not only matching individual pedestrians but also meticulously inferring the group's internal topology and membership relationships, presenting a novel and complex challenge.

[0003] Specifically, GReID faces the following unique challenges: (1) Changes in group members: Individuals may leave or join a group at any time. (2) Dynamic changes in group topology: Due to differences in movement speed and interaction among individuals, the group structure may change at any time. (3) Increased occlusion problem: Due to the close proximity of individuals within the same group, occlusion between people will pose a significant challenge to re-identification technology.

[0004] In 2009, Wei-Shi Zheng et al. first proposed the problem of cross-camera group pedestrian association and put forward a group representation and matching method based on dictionary learning, which uses the learned visual dictionary to encode group images. This pioneering work inspired initial interest in the field and spurred subsequent explorations: for example, Cai et al. proposed a group matching and context integration method based on covariance descriptors, Zheng et al. proposed a novel group representation method to handle relative position changes, and Zhu et al. proposed a group matching method based on saliency channels. With the development of deep learning, Lisanti et al. proposed the first deep learning-based group re-identification method. In 2022, Zhang et al. innovatively introduced the idea of ​​3D layout modeling.

[0005] In the same year, the groundbreaking SOT model established a feasible paradigm for uncertainty modeling within the GReID framework, which was later refined by UMSOT. However, this extension primarily focused on improving the uncertainty modeling paradigm itself, without providing specific solutions for key real-world perturbations such as clothing changes or severe occlusion. Therefore, this foundation inspired subsequent research aimed at enhancing the framework from different perspectives. These efforts mainly fall into two paths: one exploring cross-modal guidance, and the other focusing on feature-level robustness. Representing the first path, the GCUM model integrates the CLIP language model to generate uncertain group text descriptions and leverages this semantic prior to guide and reinforce visual feature learning, thereby handling different group structures. As a typical attempt at the second path, the PBSOT model introduces a layout-guided local feature sampling module that employs random feature masking for vulnerable individuals to mitigate the impact of misinformation and improve robustness in occluded scenarios. In summary, these works mark the maturity and diversity of uncertainty modeling frameworks in addressing the complex challenges of GReID. However, despite these advances, significant challenges remain in extreme real-world scenarios involving clothing variations, complex lighting changes, and severe occlusion. Specifically, two key limitations remain unresolved in addressing pedestrian occlusion: (1) Inaccurate occlusion localization: Current density-based estimations lack the precision to identify specific occluded body parts of a target individual. (2) Feature preservation dilemma: While popular random masking strategies improve robustness, they inevitably discard complete features, necessitating a trade-off between occlusion handling and feature integrity. Summary of the Invention

[0006] The purpose of this invention is to provide a method and system for anti-occlusion group re-identification based on multi-granularity cross-modal representation, so as to achieve accurate occlusion localization, reduce computational overhead, and improve the robustness of group re-identification in complex occlusion scenarios.

[0007] To achieve the above functions, this invention designs an anti-occlusion group re-identification method based on multi-granularity cross-modal representation, which executes the following steps S1-S5 to complete the re-identification of occlusion groups in the camera image:

[0008] Step S1: For any two different cameras, construct two sets of member sets for the members in the images captured by the two cameras respectively; for the two sets of member sets, perform group matching based on the GReID model to identify the same member in the two sets of images.

[0009] Step S2: Label the bounding box for each member in each image and extract the center point and spatial layout coordinates; further calculate the average spatial density of each member, establish a local candidate subset based on the average spatial density of the members, identify the occluded members and the unoccluded members in the local candidate subset, and label them respectively.

[0010] Step S3: Based on the HRNet model, for members who are occluded in the local candidate subset, divide the body parts and related key points according to the human body structure, calculate the visibility of each part based on the key points, and obtain the binary occlusion decision for each part based on the visibility of each part in order to identify the occluded parts of the members.

[0011] Step S4: Based on the member's image and the occluded parts of the identified member, generate a text description based on the CLIP language model, and then encode it into a semantic feature vector. Based on the semantic feature vector and the visual feature vector, design a two-layer weighting strategy to obtain the local features of each part of the member. Concatenate the local features to generate the enhanced member features.

[0012] Step S5: Establish member-level loss functions and group-level loss functions respectively. Based on the member-level loss functions and group-level loss functions, further establish the overall loss function, supervise the training of the GReID model, and complete the re-identification of occluded groups in the camera image based on the trained GReID model.

[0013] This invention also designs an anti-occlusion group re-identification system based on multi-granularity cross-modal representation, including a group matching module, an adaptive high-granularity occlusion processing module, an occlusion mitigation module based on complete features, and a loss function module, to implement the aforementioned anti-occlusion group re-identification method based on multi-granularity cross-modal representation:

[0014] Group matching module: For any two different cameras, construct two sets of member sets for the members in the images captured by the two cameras respectively; for each member in the two sets of images, perform group matching based on the GReID model to identify the same member in the two sets of images;

[0015] The adaptive occlusion high-granularity processing module: labels the bounding box for each member in each image and extracts the center point and spatial layout coordinates; further calculates the average spatial density of each member, establishes a local candidate subset based on the average spatial density of the members, identifies the occluded members and the unoccluded members in the local candidate subset, and labels them respectively.

[0016] Based on the HRNet model, for members who are occluded in a local candidate subset, the system divides the body into parts and related key points according to the human body structure, calculates the visibility of each part based on the key points, and obtains a binary occlusion decision for each part based on the visibility of each part in order to identify the occluded parts of the members.

[0017] The occlusion mitigation module based on complete features: Based on the member's image and the identified occluded parts of the member, a text description is generated based on the CLIP language model, and then encoded into a semantic feature vector. Based on the semantic feature vector and the visual feature vector, a two-layer weighting strategy is designed to obtain the local features of each part of the member. The local features are then concatenated to generate the enhanced member features.

[0018] Loss function module: Establish member-level loss function and group-level loss function respectively. Based on the member-level loss function and group-level loss function, further establish the overall loss function to supervise the training of GReID model. Based on the trained GReID model, complete the re-identification of occluded groups in camera images.

[0019] The present invention also designs a computer-readable storage medium storing a computer program / instruction thereon, which, when executed by a processor, implements the aforementioned anti-occlusion group re-identification method based on multi-granularity cross-modal representation.

[0020] The present invention also designs a computer program product, including a computer program / instruction, which, when executed by a processor, implements the aforementioned anti-occlusion group re-identification method based on multi-granularity cross-modal representation.

[0021] Beneficial effects: Compared with the prior art, the advantages of the present invention include:

[0022] 1. A multi-granularity cross-modal representation framework (MGCM) is proposed, which effectively solves the occlusion challenge in GReID by integrating an adaptive occlusion high-granularity processing (AOH) module and a full feature-based occlusion mitigation (CFOM) module;

[0023] 2. A pose-guided AOH module is proposed, which accurately locates partial-level occlusion by combining the visual geometric prior of the local population with the confidence threshold of key points, thereby effectively suppressing the error propagation of damaged features in the population representation;

[0024] 3. A CFOM module based on the CLIP language model is proposed. This module preserves the integrity of full-dimensional features through dynamic attention recalibration, which significantly enhances the robustness of the GReID model without affecting the integrity of features.

[0025] 4. The method designed in this invention can achieve advanced performance and competitive results on existing datasets such as CSG, RoadGroup, and DukeGroup datasets. Attached Figure Description

[0026] Figure 1 This is a schematic diagram illustrating key challenges in group re-identification according to an embodiment of the present invention;

[0027] Figure 2 This is an architecture diagram of an anti-occlusion group re-identification method based on multi-granularity cross-modal representation provided in an embodiment of the present invention;

[0028] Figure 3 This is a schematic diagram of an occlusion detection strategy provided according to an embodiment of the present invention;

[0029] Figure 4 This is a schematic diagram of an occlusion processing method based on the HRNet model provided in an embodiment of the present invention;

[0030] Figure 5 This is a schematic diagram of an occlusion processing method based on the CLIP language model provided in an embodiment of the present invention. Detailed Implementation

[0031] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0032] Group re-identification (GReID) aims to match group members from different camera perspectives and plays a crucial role in intelligent surveillance systems. However, the inherent social affinity within groups leads to tight clustering, resulting in frequent and severe mutual occlusion, significantly hindering accurate matching. Although existing GReID models have attempted to address the occlusion problem, they still struggle to adapt to the diversity and dynamics of occlusion in the real world. Figure 1 Each column illustrates a different challenge in group re-identification: the first column shows a new member joining (marked with a red triangle); the second column shows two members swapping positions (marked with a blue circular arrow); the third column shows a member being severely occluded (marked with a red border). To address these challenges, embodiments of the present invention provide an occlusion-resistant group re-identification method based on multi-granularity cross-modal representation, referring to... Figure 2 Perform the following steps S1-S5 to complete the re-identification of occluded groups in the camera image:

[0033] Step S1: For any two different cameras, construct two sets of member sets for the members in the images captured by the two cameras respectively; for the two sets of member sets, perform group matching based on the GReID model to identify the same member in the two sets of images.

[0034] The specific method is as follows:

[0035] For the membership sets of two image groups, the GReID model is used to identify the same group of members in both groups. The definition of the same group of members is: if the number of people in both groups is greater than or equal to 60% of the total number of people, they are considered to belong to the same group. The formula is expressed as follows:

[0036] ;

[0037] In the formula, Represents a set of members of a group of images. Represents a member set of another group of images. , express The j-th member, express Total number of members; , express The j-th member, express Total number of members; express and The similarity coefficient;

[0038] A group of members that satisfies the above formula is defined as the same group of members.

[0039] If no common members are found in the images captured by the two cameras, the GReID model cannot establish a valid member correspondence during the matching process. In this case, the GReID model will classify the two groups as different groups and will not force an incorrect match. In subsequent queries and verifications, if it is confirmed that the two groups indeed have no common members, this judgment will be recorded as a successful negative sample detection (i.e., correctly classifying them as not being the same group).

[0040] Step S2: Label the bounding box for each member in each image and extract the center point and spatial layout coordinates; further calculate the average spatial density of each member, establish a local candidate subset based on the average spatial density of the members, identify the occluded members and the unoccluded members in the local candidate subset, and label them respectively.

[0041] For all members in the image, the bounding box of each member is labeled. A 12-layer transformer model is used to crop the bounding box of each member and extract the visual feature vector to obtain the cropped image of a single member. A 2-layer transformer model is used to extract the group topology to obtain the group structure. The 2-layer transformer model works independently and uses pre-trained weights. Then, the following steps are performed:

[0042] Step S2.1: For each member in the image, calculate its spatial layout coordinates as follows:

[0043] For each member's bounding box (x, y, w, h), take the center point (x+w / 2, y+h / 2), where x represents the horizontal coordinate of the top-left corner of the bounding box, y represents the vertical coordinate of the top-left corner of the bounding box, w represents the width of the bounding box, and h represents the height of the bounding box. Divide the center point by the width and height of the entire image to obtain the spatial layout coordinates (avg) normalized to [0, 1]. x avg y );

[0044] Step S2.2: Receive the bounding box and spatial layout coordinates of each member, and obtain the average spatial density of the members by calculating the Euclidean distance between the centers of the bounding boxes of each member.

[0045] ;

[0046] in, and These represent the center point of the i-th member in group p, respectively. and coordinate, and These represent the center point of the j-th member in group p, respectively. and Coordinates; and This represents the total number of members. Indicates except the first The total number of members other than the given number of members; expression Calculate the first The member and the Euclidean distance between members; double summation iterative traversal of all unique pairs of individuals in the population; This indicates that the average value is calculated to obtain the average spatial density of the members under consideration;

[0047] Step S2.3: Select the member with the highest average spatial density and its adjacent left and right neighbors to form a local candidate subset. The specific formula is as follows:

[0048] ;

[0049] In the formula, Represents a local candidate subset. The bounding box representing the member with the highest average spatial density is used as the center candidate box. This refers to the bounding box immediately to the left of the center candidate box in the spatial arrangement of the group. This refers to the bounding box immediately to the right of the center candidate box in the spatial arrangement of the group. Union; set and It may be an empty set (when there are no left and right adjacent bounding boxes);

[0050] Step S2.4: Based on key visual geometric priors in monocular vision, utilize the depth-height correspondence in the horizontal monitoring scene: when the camera tilt angle is extremely small or non-tilted, the vertical coordinates of the object on the image plane ( The coordinates are directly related to the actual distance from the camera. Specifically, objects farther from the camera appear closer to the horizon (top of the image), while objects closer to the camera appear closer to the bottom of the image. In a small crowd, pedestrians closer to the camera may partially or completely obscure pedestrians behind them; therefore, the coordinates of the bottom midpoint of the bounding box are directly related to the actual distance from the camera. Coordinates can be used as an effective indicator for depth estimation because a larger y-coordinate value indicates that the distance to the camera is greater.

[0051] like Figure 3 As shown, firstly, the ground in all images is abstracted to establish the relationship between the distance between pedestrians and the camera and their vertical image coordinates. The ground in all images is abstracted to model the relationship between the distance between pedestrians and the camera and their vertical image coordinates. Pedestrians with the smallest y-values ​​are directly selected for feature extraction because they represent the closest individuals and are therefore the least likely to be occluded (shown in green in the example image), while the remaining pedestrians are processed later.

[0052] To identify occluded individuals, compare local candidate subsets. The bottom midpoint of the bounding box lies on the image plane. coordinate , minimum Members with the highest value are considered to be closest to the camera and have the lowest probability of being blocked. All remaining members are considered potentially occluded, and the geometric rules are formalized as follows:

[0053] ;

[0054] in, Indicates the occlusion candidate set, Indicates the first in the local candidate subset The bounding box of each member, express The bottom midpoint is on the image plane coordinate, , Includes all members from the local candidate subset coordinate;

[0055] Step S2.5: For the smallest value in step S2.4 For members of the value, directly output the visual feature vector; for local candidate subsets The other two members are labeled True and False respectively to indicate whether pose detection is needed, and are then input into the subsequent processing steps.

[0056] Step S3: Based on the HRNet model, for members who are occluded in the local candidate subset, divide the body parts and related key points according to the human body structure, calculate the visibility of each part based on the key points, and obtain the binary occlusion decision for each part based on the visibility of each part in order to identify the occluded parts of the members.

[0057] Even within uncertainty modeling paradigms, occlusion remains a fundamental challenge in individual and group re-identification. Step S3 proposes a high-granularity processing method based on the HRNet model, which can detect precise body keypoints for each group member. This achieves local-level visibility assessment and facilitates fine-grained feature recovery in occluded regions, thus moving away from reliance on random masking and towards geometry-based occlusion inference. The specific steps are as follows:

[0058] Step S3.1: Receive local candidate subsets The bounding boxes and labels of the other two members are also provided. For the members whose labels indicate the need for pose detection, the HRNet model outputs a set of keypoints, each corresponding to a preset position on the human body:

[0059] ;

[0060] In the formula, For the first key point in the key point group K There are 17 key points in the key point group K;

[0061] Each key point Each of these is a triple, in the following form:

[0062] ;

[0063] in, Indicates the first in the image The coordinates of the key points, where R represents a real number; Indicates the first Confidence level of each key point;

[0064] Step S3.2: Divide the 17 keypoints into eight parts, each part corresponding to a semantic region, and calculate the visibility score for each part as follows:

[0065] ;

[0066] In the formula, The visibility score for part m is represented. This represents a subset of key points for part m. Indicates the first Confidence level of each key point Key points representing part m, sub-key points Quantity;

[0067] Step S3.3: Convert the occlusion decision into a binary occlusion decision using a step function, as shown in the following equation:

[0068] ;

[0069] in, Indicates location The masking code, Indicates location Obscured Indicates location visible, Indicates the preset part Detection threshold;

[0070] Step S3.4: Output the confidence values ​​of 17 key points and the occlusion masks of 8 parts.

[0071] Reference Figure 4 The 17 key points mentioned are COCO key points, which are annotation data used in the field of computer vision to describe the location of key parts of the human body. Specifically, they include: nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle. Among them, the area formed by connecting the nose, left eye, right eye, left ear, and right ear is the head; the area formed by connecting the left shoulder and right shoulder is the neck; the area formed by connecting the left shoulder, right shoulder, left elbow, and right elbow is the upper torso; the area formed by connecting the left elbow, right elbow, left hip, and right hip is the middle of the torso; the area formed by connecting the left elbow, right elbow, left wrist, and right wrist is the arm; the area formed by connecting the left hip and right hip is the hip; the area formed by connecting the left hip, right hip, left knee, and right knee is the thigh; and the area formed by connecting the left knee, right knee, left ankle, and right ankle is the lower leg, forming a total of eight parts.

[0072] Step S4: Based on the member's image and the occluded parts of the identified member, generate a text description based on the CLIP language model, and then encode it into a semantic feature vector. Based on the semantic feature vector and the visual feature vector, design a two-layer weighting strategy to obtain the local features of each part of the member. Concatenate the local features to generate the enhanced member features.

[0073] Step S4 utilizes the CLIP language model to mitigate occlusion interference, while maintaining feature integrity through adaptive feature recalibration and semantic enhancement. The specific steps are as follows:

[0074] Step S4.1: Receive the cropped image of a single member, the corresponding spatial layout coordinates, the confidence of 17 key points, the group structure, and the occlusion mask of eight parts;

[0075] Step S4.2: Based on the CLIP language model, input the information received in step S4.1, generate text descriptions for each part, and then encode them into semantic feature vectors, expressed as follows:

[0076] ;

[0077] In the formula, Represents a set of descriptive information. This represents the text description generated by the CLIP language model based on the input information. The generated semantic feature vector represents five semantic dimensions: occlusion, spatial relationships, temporal context, social interaction, and group structure.

[0078] Step S4.3: Design a two-layer weighting strategy for visual feature vectors and semantic feature vectors as follows:

[0079] ;

[0080] In the formula, Indicates local features, Represents visual feature vectors. Represents semantic feature vectors; This represents the balancing weights, used to control the contribution of semantic feature vectors relative to visual feature vectors. , Indicates the benchmark weight. Indicates local weights;

[0081] Step S4.4: Reassemble the eight local features of the member generated in step S4.3 into the final enhanced member features.

[0082] Reference Figure 5Step S4 designs an occlusion processing method based on the CLIP language model. Using a text encoder, structured prompts are used to convert the occlusion information generated by the adaptive high-granularity occlusion module into semantic descriptions. These descriptions guide visual processing, enhancing the features of the bounding box-marked regions through cross-modal alignment. The red annotations in all images collectively demonstrate the process of recovering the occluded pedestrian's arm through a semantic guidance mechanism. The purple annotations show semantic compensation information.

[0083] Step S5: Establish member-level loss functions and group-level loss functions respectively. Based on the member-level loss functions and group-level loss functions, further establish the overall loss function, supervise the training of the GReID model, and complete the re-identification of occluded groups in the camera image based on the trained GReID model.

[0084] Perform the following steps S5.1-S5.4 to establish the member-level loss function; perform the following steps S5.5-S5.8 to establish the group-level loss function; and perform the following step S5.9 to establish the overall loss function, as detailed below:

[0085] Step S5.1: After obtaining the first-order labeled representation, it is input into the member feature transformation layer to obtain global and local features at the member level. To optimize the learning of the member-level global representation, the overall objective is formulated as a weighted sum of cross-entropy loss and triplet loss, and they are jointly optimized; the member-level cross-entropy loss function is constructed as follows:

[0086] ;

[0087] in, This represents the member-level cross-entropy loss function. Indicates the first The first group The real tags of each member include two categories: those that are obscured and those that are not. This represents the probability distribution of the predicted scores of the GReID model for all possible memberships. In other words, the GReID model needs to predict the most likely membership of the current member from all known memberships. N represents the total number of members; N represents the total number of groups.

[0088] Step S5.2: Construct the intra-triplet loss function for hard sample mining as follows:

[0089] ;

[0090] in, This represents the loss function within triples for hard sample mining. Represents Euclidean distance; symbol Indicates positive value operation; , , These represent the anchor sample, difficult positive sample, and difficult negative sample in the current batch, respectively. The anchor sample is the sample currently being monitored and used as the benchmark for comparison. The difficult positive sample belongs to the same member (same class) as the anchor sample, but is the most difficult to distinguish among all samples of the same class. The difficult negative sample belongs to a different member (different class) than the anchor sample, but is the most easily confused among all samples of different classes. Represents anchor sample global feature vectors Indicating difficult positive samples global feature vectors Indicating difficult negative samples The global feature vector is a fixed 768-dimensional numerical vector extracted by the transformer model. These are edge hyperparameters;

[0091] Step S5.3: For the learning of local feature representations of members, another triplet loss function is used for supervision, aiming to reduce the similarity between members of the same group and increase the difference between members of different groups; the overall loss function for hard sample mining triplet is constructed as follows:

[0092] ;

[0093] in, This represents the overall loss function for mining triples from hard samples. Represents anchor sample The A set of local features Indicating difficult positive samples The A set of local features Indicating difficult negative samples The There are 10 local feature groups; D represents the total number of local feature groups; a local feature group is a fixed 768-dimensional numerical vector that represents the features of a local region of the input image, which is extracted by the transformer model.

[0094] Step S5.4: Construct the member-level loss function as follows:

[0095] ; in, Represents the member-level loss function;

[0096] Step S5.5: Introduce second-order labels, which are obtained by converting first-order labels into a group-level feature representation capable of characterizing the entire group. The group-level objective function consists of two parts: group identity loss and triplet loss; the group-level cross-entropy loss function is constructed as follows:

[0097] ;

[0098] in, This represents the group-level cross-entropy loss function. Indicates the first The true labels of the n groups are represented in the dataset as follows: the true label of the nth group represents the true label of the nth group in the dataset. Each group sample is manually labeled with a unique and correct group identity ID; This represents the probability distribution of the predicted scores of the GReID model for all possible group identities. The possible group identities indicate which group the GReID model needs to predict from all known group identities, and it represents the probability distribution predicted by the model.

[0099] Step S5.6: Construct the group-level triplet loss function as follows:

[0100] ;

[0101] in, This represents the group-level triplet loss function. Indicates the first Anchor group, that is, the group sample currently used as the benchmark; Indicates and The corresponding difficult positive sample group, i.e., the anchor group They belong to the same group (have the same group ID), but are different from other groups in all positive samples. The sample group with the greatest feature differences (i.e. the most difficult to distinguish); Indicates and The corresponding difficult negative sample group, i.e., the anchor group Belonging to different groups (having different group IDs), but appearing in all negative samples as... The sample group with the most similar features (i.e. the most easily confused); Indicates anchor group The global feature vector; Indicating the difficult positive sample group global feature vectors Indicating the difficult negative sample group The global feature vector;

[0102] Step S5.7: Construct the triplet loss function for the global features of the population as follows:

[0103] ;

[0104] in, The triplet loss function represents the global characteristics of the population; Indicates anchor group The corresponding local feature group, Indicating the difficult positive sample group The corresponding local feature group, Indicating the difficult negative sample group The corresponding local feature group;

[0105] Step S5.8: Establish group-level loss function:

[0106] ; in, Represents the group-level loss function;

[0107] Step S5.9: Establish the overall loss function as follows:

[0108] ;

[0109] In the formula, Represents the overall loss function. Represents the group-level loss function. This represents the member-level loss function. This represents the relative weights, used to adjust the relative weights of group-level and member-level losses during the GReID model learning process, to ensure that the GReID model can simultaneously learn effective between-group discriminative features and within-group consistency features.

[0110] This invention also provides an anti-occlusion group re-identification system based on multi-granularity cross-modal representation, including a group matching module, an adaptive high-granularity occlusion processing module, an occlusion mitigation module based on complete features, and a loss function module, to implement the aforementioned anti-occlusion group re-identification method based on multi-granularity cross-modal representation.

[0111] Group matching module: For any two different cameras, construct two sets of member sets for the members in the images captured by the two cameras respectively; for each member in the two sets of images, perform group matching based on the GReID model to identify the same member in the two sets of images;

[0112] Adaptive Occlusion High-Granularity Processing (AOH) module: It annotates the bounding boxes for each member in each image and extracts the center point and spatial layout coordinates; it further calculates the average spatial density of each member, establishes a local candidate subset based on the average spatial density of the members, identifies the occluded members and the unoccluded members in the local candidate subset, and annotates them respectively.

[0113] Based on the HRNet model, for members who are occluded in a local candidate subset, the system divides the body into parts and related key points according to the human body structure, calculates the visibility of each part based on the key points, and obtains a binary occlusion decision for each part based on the visibility of each part in order to identify the occluded parts of the members.

[0114] The Complete Feature-Based Occlusion Mitigation (CFOM) module: Based on the member's image and the identified occluded parts of the member, it generates a text description based on the CLIP language model, and then encodes it into a semantic feature vector. Based on the semantic feature vector and the visual feature vector, it designs a two-layer weighting strategy to obtain local features of each part of the member. The local features are then concatenated to generate enhanced member features.

[0115] Loss function module: Establish member-level loss function and group-level loss function respectively. Based on the member-level loss function and group-level loss function, further establish the overall loss function to supervise the training of GReID model. Based on the trained GReID model, complete the re-identification of occluded groups in camera images.

[0116] This invention also provides a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements the aforementioned anti-occlusion group re-identification method based on multi-granularity cross-modal representation.

[0117] This invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the aforementioned anti-occlusion group re-identification method based on multi-granularity cross-modal representation.

[0118] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A method for anti-occlusion group re-identification based on multi-granularity cross-modal representation, characterized in that, Perform the following steps S1-S5 to complete the re-identification of occluded groups in the camera image: Step S1: For any two different cameras, construct two sets of member sets for the members in the images captured by the two cameras respectively; for the two sets of member sets, perform group matching based on the GReID model to identify the same member in the two sets of images. Step S2: Label the bounding box for each member in each image and extract the center point and spatial layout coordinates; further calculate the average spatial density of each member, establish a local candidate subset based on the average spatial density of the members, identify the occluded members and the unoccluded members in the local candidate subset, and label them respectively. Step S3: Based on the HRNet model, for members who are occluded in the local candidate subset, divide the body parts and related key points according to the human body structure, calculate the visibility of each part based on the key points, and obtain the binary occlusion decision for each part based on the visibility of each part in order to identify the occluded parts of the members. Step S4: Based on the member's image and the occluded parts of the identified member, generate a text description based on the CLIP language model, and then encode it into a semantic feature vector. Based on the semantic feature vector and the visual feature vector, design a two-layer weighting strategy to obtain the local features of each part of the member. Concatenate the local features to generate the enhanced member features. Step S5: Establish member-level loss functions and group-level loss functions respectively. Based on the member-level loss functions and group-level loss functions, further establish the overall loss function, supervise the training of the GReID model, and complete the re-identification of occluded groups in the camera image based on the trained GReID model.

2. The anti-occlusion group re-identification method based on multi-granularity cross-modal representation according to claim 1, characterized in that, The specific method for step S1 is as follows: For the membership sets of two sets of images, the same membership group in the two sets of images is identified based on the GReID model. The same membership group is defined as follows: ； In the formula, Represents a set of members of a group of images. Represents a member set of another group of images. , express The j-th member, express Total number of members; , express The j-th member, express Total number of members; express and The similarity coefficient; A group of members that satisfies the above formula is defined as the same group of members.

3. The anti-occlusion group re-identification method based on multi-granularity cross-modal representation according to claim 2, characterized in that, In step S2, for all members in the image, the bounding box of each member is labeled. A 12-layer transformer model is used to crop the bounding box of each member and extract the visual feature vector to obtain the cropped image of a single member. A 2-layer transformer model is used to extract the group topology to obtain the group structure. Then, the following steps are performed: Step S2.1: For each member in the image, calculate its spatial layout coordinates as follows: For each member's bounding box (x, y, w, h), take the center point (x+w / 2, y+h / 2), where x represents the horizontal coordinate of the top-left corner of the bounding box, y represents the vertical coordinate of the top-left corner of the bounding box, w represents the width of the bounding box, and h represents the height of the bounding box. Divide the center point by the width and height of the entire image to obtain the spatial layout coordinates (avg) normalized to [0, 1]. x avg y ); Step S2.2: Receive the bounding box and spatial layout coordinates of each member, and obtain the average spatial density of the members by calculating the Euclidean distance between the centers of the bounding boxes of each member. ； in, and These represent the center point of the i-th member in group p, respectively. and coordinate, and These represent the center point of the j-th member in group p, respectively. and Coordinates; and This represents the total number of members. Indicates except the first The total number of members other than the given number of members; expression Calculate the first The member and the Euclidean distance between members; This indicates the calculation of the average value; Step S2.3: Select the member with the highest average spatial density and its adjacent left and right neighbors to form a local candidate subset. The specific formula is as follows: ； In the formula, Represents a local candidate subset. The bounding box representing the member with the highest average spatial density is used as the center candidate box. This refers to the bounding box immediately to the left of the center candidate box in the spatial arrangement of the group. This refers to the bounding box immediately to the right of the center candidate box in the spatial arrangement of the group. Represents the union; Step S2.4: Compare local candidate subsets The bottom midpoint of the bounding box lies on the image plane. coordinate , minimum Members with the highest value are considered to be closest to the camera and have the lowest probability of being blocked. All remaining members are considered potentially occluded, and the geometric rules are formalized as follows: ； in, Indicates the occlusion candidate set, Indicates the first in the local candidate subset The bounding box of each member, express The bottom midpoint is on the image plane coordinate, , Includes all members from the local candidate subset coordinate; Step S2.5: For the smallest value in step S2.4 For members of the value, directly output the visual feature vector; for local candidate subsets The other two members are labeled True and False respectively to indicate whether pose detection is needed, and are then input into the subsequent processing steps.

4. The anti-occlusion group re-identification method based on multi-granularity cross-modal representation according to claim 3, characterized in that, The specific steps of step S3 are as follows: Step S3.1: Receive local candidate subsets The bounding boxes and labels of the other two members are also provided. For the members whose labels indicate the need for pose detection, the HRNet model outputs a set of keypoints, each corresponding to a preset position on the human body: ； In the formula, For the first key point in the key point group K There are 17 key points in the key point group K; Each key point Each of these is a triple, in the following form: ； in, Indicates the first in the image The coordinates of the key points, where R represents a real number; Indicates the first Confidence level of each key point; Step S3.2: Divide the 17 keypoints into eight parts, each part corresponding to a semantic region, and calculate the visibility score for each part as follows: ； In the formula, The visibility score for part m is represented. This represents a subset of key points for part m. Indicates the first Confidence level of each key point Key points representing part m, sub-key points Quantity; Step S3.3: Convert the occlusion decision into a binary occlusion decision using a step function, as shown in the following equation: ； in, Indicates location The masking code, Indicates location Obscured Indicates location visible, Indicates the preset part Detection threshold; Step S3.4: Output the confidence values ​​of 17 key points and the occlusion masks of 8 parts.

5. The anti-occlusion group re-identification method based on multi-granularity cross-modal representation according to claim 4, characterized in that, The 17 key points mentioned in step S3 include: nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle; among them, the area formed by connecting the nose, left eye, right eye, left ear, and right ear is the head; the area formed by connecting the left shoulder and right shoulder is the neck; the area formed by connecting the left shoulder, right shoulder, left elbow, and right elbow is the upper torso; the area formed by connecting the left elbow, right elbow, left hip, and right hip is the middle torso; the area formed by connecting the left elbow, right elbow, left wrist, and right wrist is the arm; the area formed by connecting the left hip and right hip is the hip; the area formed by connecting the left hip, right hip, left knee, and right knee is the thigh; and the area formed by connecting the left knee, right knee, left ankle, and right ankle is the calf; forming a total of eight parts.

6. The anti-occlusion group re-identification method based on multi-granularity cross-modal representation according to claim 5, characterized in that, The specific steps of step S4 are as follows: Step S4.1: Receive the cropped image of a single member, the corresponding spatial layout coordinates, the confidence of 17 key points, the group structure, and the occlusion mask of eight parts; Step S4.2: Based on the CLIP language model, input the information received in step S4.1, generate text descriptions for each part, and then encode them into semantic feature vectors, expressed as follows: ； In the formula, Represents a set of descriptive information. This represents the text description generated by the CLIP language model based on the input information. This represents the semantic feature vector generated by encoding. This indicates CLIP language model encoding processing; Step S4.3: Design a two-layer weighting strategy for visual feature vectors and semantic feature vectors as follows: ； In the formula, Indicates local features, Represents visual feature vectors. Represents semantic feature vectors; Indicates the balancing weights. , Indicates the benchmark weight. Indicates local weights; Step S4.4: Reassemble the eight local features of the member generated in step S4.3 into the final enhanced member features.

7. The anti-occlusion group re-identification method based on multi-granularity cross-modal representation according to claim 6, characterized in that, In step S5, execute steps S5.1-S5.4 to establish the member-level loss function, steps S5.5-S5.8 to establish the group-level loss function, and step S5.9 to establish the overall loss function, as detailed below: Step S5.1: Construct the member-level cross-entropy loss function as follows: ； in, This represents the member-level cross-entropy loss function. Indicates the first The first group The real labels of each member This represents the probability distribution of the predicted scores for all possible memberships by the GReID model; N represents the total number of members; N represents the total number of groups. Step S5.2: Construct the intra-triplet loss function for hard sample mining as follows: ； in, This represents the loss function within triples for hard sample mining. Represents Euclidean distance; symbol Indicates positive value operation; , , These represent anchor samples, difficult positive samples, and difficult negative samples in the current batch, respectively. Represents anchor sample global feature vectors Indicating difficult positive samples global feature vectors Indicating difficult negative samples The global feature vector; These are edge hyperparameters; Step S5.3: Construct the overall loss function for mining triples from hard samples as follows: ； in, This represents the overall loss function for mining triples from hard samples. Represents anchor sample The A set of local features Indicating difficult positive samples The A set of local features Indicating difficult negative samples The There are 1 local feature groups; D represents the total number of local feature groups; Step S5.4: Construct the member-level loss function as follows: ； in, Represents the member-level loss function; Step S5.5: Construct the population-level cross-entropy loss function as follows: ； in, This represents the group-level cross-entropy loss function. Indicates the first The true labels of each group This represents the probability distribution of the predicted scores of the GReID model for all possible group identities; Step S5.6: Construct the group-level triplet loss function as follows: ； in, This represents the group-level triplet loss function. This represents the i-th anchor point group; Indicates and The corresponding difficult positive sample group; Indicates and The corresponding difficult negative sample group; Indicates anchor group The global feature vector; Indicating the difficult positive sample group global feature vectors Indicating the difficult negative sample group The global feature vector; Step S5.7: Construct the triplet loss function for the global features of the population as follows: ； in, The triplet loss function represents the global characteristics of the population; Indicates anchor group The corresponding local feature group, Indicating the difficult positive sample group The corresponding local feature group, Indicating the difficult negative sample group The corresponding local feature group; Step S5.8: Establish group-level loss function: ； in, Represents the group-level loss function; Step S5.9: Establish the overall loss function as follows: ； In the formula, Represents the overall loss function. Represents the group-level loss function. This represents the member-level loss function. This indicates the relative weight.

8. A group re-identification system for resistance to occlusion based on multi-granularity cross-modal representation, characterized in that, The method includes a group matching module, an adaptive occlusion high-granularity processing module, an occlusion mitigation module based on complete features, and a loss function module, thereby implementing the occlusion-resistant group re-identification method based on multi-granularity cross-modal representation as described in any one of claims 1-7. Group matching module: For any two different cameras, construct two sets of member sets for the members in the images captured by the two cameras respectively; for each member in the two sets of images, perform group matching based on the GReID model to identify the same member in the two sets of images; The adaptive occlusion high-granularity processing module: labels the bounding box for each member in each image and extracts the center point and spatial layout coordinates; further calculates the average spatial density of each member, establishes a local candidate subset based on the average spatial density of the members, identifies the occluded members and the unoccluded members in the local candidate subset, and labels them respectively. Based on the HRNet model, for members who are occluded in a local candidate subset, the system divides the body into parts and related key points according to the human body structure, calculates the visibility of each part based on the key points, and obtains a binary occlusion decision for each part based on the visibility of each part in order to identify the occluded parts of the members. The occlusion mitigation module based on complete features: Based on the member's image and the identified occluded parts of the member, a text description is generated based on the CLIP language model, and then encoded into a semantic feature vector. Based on the semantic feature vector and the visual feature vector, a two-layer weighting strategy is designed to obtain the local features of each part of the member. The local features are then concatenated to generate the enhanced member features. Loss function module: Establish member-level loss function and group-level loss function respectively. Based on the member-level loss function and group-level loss function, further establish the overall loss function to supervise the training of GReID model. Based on the trained GReID model, complete the re-identification of occluded groups in camera images.

9. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instruction is executed by the processor, it implements the anti-occlusion group re-identification method based on multi-granularity cross-modal representation as described in any one of claims 1-7.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the anti-occlusion group re-identification method based on multi-granularity cross-modal representation as described in any one of claims 1-7.

Images