Model training method, costume-changing humanoid re-identification method, device, equipment and medium

Abstract

This application provides a model training method, a method, apparatus, device, and medium for re-identifying human figures in different costumes, relating to the field of data processing technology. The model training method includes: acquiring sample data pairs; the sample data pairs include: an image containing a human figure and corresponding descriptive text, the descriptive text being used to describe the human figure region in the image, excluding clothing; inputting the sample data pairs into a target feature extraction model to be trained, so that the target feature extraction model generates fused features of the sample data pairs; calculating the loss based on the features extracted by each network of the target feature extraction model to obtain the current target loss; in response to determining that a preset convergence condition is not met based on the current target loss, adjusting the network parameters of the target feature extraction model, and returning to the step of acquiring sample data pairs. This application can improve the accuracy of features used for re-identifying human figures in different costumes.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to model training methods, methods, devices, equipment and media for re-identifying human figures in different costumes. Background Technology

[0002] Cloth-Changing Person Re-Identification (CC ReID) is a technology that can still match the same person from different cameras or scenes after a person changes their clothes, hairstyle, or other appearance.

[0003] Currently, multimodal models are typically trained using images. After the multimodal model is trained, descriptive text corresponding to the image to be analyzed is generated. The image and descriptive text are encoded separately, and the encoded image features and text features are fused. The fused features are then used for human re-identification in costume.

[0004] However, the descriptive text generated by the multimodal model contains descriptions unrelated to the person, such as "white blanket," and descriptions of clothing, such as "black trousers," which are negative features for human re-identification in disguise. Furthermore, the fused features obtained by the multimodal model contain irrelevant and negative features, and the fused features used for human re-identification in disguise rely solely on encoded image and text features, resulting in insufficient feature reliance. Consequently, the accuracy of the fused features output by the multimodal model for human re-identification in disguise is not high.

[0005] Therefore, improving the accuracy of features used for humanoid re-identification during costume changes is an urgent problem to be solved. Summary of the Invention

[0006] The purpose of this application is to provide a model training method, a method, apparatus, device, and medium for re-identifying human figures in different costumes, thereby improving the accuracy of features used for re-identifying human figures in different costumes. The specific technical solution is as follows:

[0007] In a first aspect, embodiments of this application provide a model training method, the method comprising:

[0008] Obtain sample data pairs; wherein, the sample data pairs include: an image containing a human figure and a corresponding descriptive text for the image, the descriptive text for the image being used to describe the human figure region in the image, excluding clothing;

[0009] The sample data pairs are input into a target feature extraction model to be trained, so that the target feature extraction model generates fused features of the sample data pairs. The target feature extraction model includes a multimodal network, a feature extraction network, and a feature fusion network. The multimodal network is used to encode the image and descriptive text in the sample data pairs respectively, and fuse the encoded image features and text features to obtain a first feature of the sample data pairs. The feature extraction network is used to extract features from the first feature in multiple dimensions to obtain a second feature of the sample data pairs in multiple dimensions, including at least: human attributes, human detection, and clothing mask. The feature fusion network is used to fuse the second feature in the human attribute dimension and the second feature in the human detection dimension, and fuse the fused feature with the second feature in the clothing mask dimension to extract the fused feature.

[0010] Based on the features extracted by each network of the target feature extraction model, loss calculation is performed to obtain the current target loss;

[0011] In response to the determination that the preset convergence condition is not met based on the current target loss, the network parameters of the target feature extraction model are adjusted, and the process returns to the step of obtaining sample data pairs.

[0012] Secondly, embodiments of this application provide a method for re-identifying a human figure in different costumes, the method comprising:

[0013] Obtain the image to be identified and the corresponding descriptive text; wherein, the descriptive text corresponding to the image to be identified is used to describe the human-shaped region in the image to be identified, excluding clothing;

[0014] The image to be identified and its corresponding descriptive text are input into the target feature extraction model to obtain fused features; wherein, the target feature extraction model is trained according to the model training method.

[0015] Based on the fusion features, perform a re-identification process for the humanoid figure after costume change.

[0016] Thirdly, embodiments of this application provide a model training apparatus, the apparatus comprising:

[0017] The first acquisition module is used to acquire sample data pairs; wherein, the sample data pairs include: an image containing a human figure and a corresponding descriptive text for the image, wherein the descriptive text for the image is used to describe the human figure region in the image, excluding clothing.

[0018] A first input module is used to input the sample data pair into a target feature extraction model to be trained, so that the target feature extraction model generates a fused feature of the sample data pair. The target feature extraction model includes a multimodal network, a feature extraction network, and a feature fusion network. The multimodal network is used to encode the image and descriptive text in the sample data pair respectively, and fuse the encoded image features and text features to obtain a first feature of the sample data pair. The feature extraction network is used to extract features from the first feature in multiple dimensions to obtain a second feature of the sample data pair in multiple dimensions, including at least: human attributes, human detection, and clothing mask. The feature fusion network is used to fuse the second feature of the human attribute dimension and the second feature of the human detection dimension, and fuse the fused feature with the second feature of the clothing mask dimension to extract the fused feature.

[0019] The calculation module is used to perform loss calculation based on the features extracted by each network of the target feature extraction model to obtain the current target loss;

[0020] The adjustment module is used to adjust the network parameters of the target feature extraction model in response to the determination that the preset convergence condition is not met based on the current target loss, and then return to the first acquisition module.

[0021] Fourthly, embodiments of this application provide a humanoid re-identification device for clothing changes, the device comprising:

[0022] The second acquisition module is used to acquire the image to be identified and the descriptive text corresponding to the image to be identified; wherein, the descriptive text corresponding to the image to be identified is used to describe the human-shaped region in the image to be identified, excluding clothing.

[0023] The second input module is used to input the image to be identified and the corresponding descriptive text into the target feature extraction model to obtain fused features; wherein, the target feature extraction model is trained according to the model training method;

[0024] The processing module is used to perform humanoid re-identification processing based on the fused features.

[0025] Fifthly, embodiments of this application provide an electronic device, including:

[0026] Memory, used to store computer programs;

[0027] When executing a program stored in memory, the processor implements either the model training method or the costume-changing humanoid re-identification method.

[0028] Sixthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the aforementioned model training methods, or the method for re-identifying a humanoid figure in disguise.

[0029] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to execute any of the model training methods described above, or the method for re-identifying a humanoid figure in disguise.

[0030] Beneficial effects of the embodiments in this application:

[0031] The model training method provided in this application first obtains sample data pairs. The sample data pairs include an image containing a human figure and corresponding descriptive text. The descriptive text corresponding to the image is used to describe the human figure region in the image except for clothing, that is, it does not include descriptions unrelated to the human figure or descriptions that do not include clothing. The sample data is used to train a target feature extraction model. The features extracted by the trained target feature extraction model do not contain irrelevant features or negative features, thereby improving the accuracy of the features for re-identifying human figures in different outfits.

[0032] The target feature extraction model in this application includes a multimodal network, a feature extraction network, and a feature fusion network. The multimodal network encodes the image and descriptive text in the sample data pair separately, and fuses the encoded image features and text features to obtain a first feature. The feature extraction network extracts features from the first feature in multiple dimensions, including human attributes, human detection, and clothing masking, to obtain a second feature in multiple dimensions. The feature fusion network first fuses the second feature in the human attribute dimension and the second feature in the human detection dimension, and then fuses the fused feature with the second feature in the clothing masking dimension to obtain a fused feature. During training, the current target loss is calculated based on the features extracted by each network of the target feature extraction model. If the current target loss does not meet the preset convergence condition, the network parameters of the target feature extraction model are adjusted, and sample data pairs are obtained for retraining.

[0033] The target feature extraction model can extract features from the first feature in multiple dimensions, and the feature fusion network can fuse features based on the second feature in multiple dimensions. Furthermore, by using the second feature of the clothing mask dimension, the negative influence of clothing features is further reduced, resulting in higher accuracy of the fused features extracted by the trained target feature extraction model. Therefore, this application can improve the accuracy of features used for human figure re-identification in clothing changes.

[0034] Furthermore, in the human figure re-identification method for clothing changes provided in this application embodiment, the descriptive text corresponding to the image to be identified does not contain descriptions unrelated to human figures or descriptions of clothing. The target feature extraction model obtained through the above model training method can extract features from the image to be identified and the corresponding descriptive text in multiple dimensions, and further reduce the influence of negative features such as clothing by using clothing masks. The accuracy of the obtained fused features is higher. Therefore, using the fused features for human figure re-identification processing for clothing changes can improve the accuracy of human figure re-identification for clothing changes.

[0035] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0037] Figure 1 A schematic flowchart illustrating a model training method provided in an embodiment of this application;

[0038] Figure 2 A schematic diagram of a target feature extraction model provided in an embodiment of this application;

[0039] Figure 3 A schematic diagram illustrating the feature extraction and loss calculation process for the humanoid attribute dimension provided in this application embodiment;

[0040] Figure 4 A schematic diagram illustrating the feature extraction and loss calculation process for humanoid detection dimensions provided in this application embodiment;

[0041] Figure 5 A schematic diagram illustrating the feature extraction and loss calculation process for the clothing mask dimension provided in this application embodiment;

[0042] Figure 6 A flowchart illustrating a method for re-identifying a human figure in clothing changes, provided in an embodiment of this application;

[0043] Figure 7 This is a schematic diagram of the structure of a model training device provided in an embodiment of this application;

[0044] Figure 8 This is a schematic diagram of the structure of a humanoid re-identification device for clothing replacement provided in an embodiment of this application;

[0045] Figure 9This is a block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0046] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0047] Currently, multimodal models are typically trained using contrastive learning. During training, the multimodal model generates descriptive text corresponding to the image, and uses a vision encoder and a text encoder to encode the image and descriptive text respectively, generating fixed-dimensional feature vectors. Then, the text and image features are fused, and the fused features are used to calculate the contrastive loss, aligning the text and image features so that the multimodal model learns the text attributes on the image. The success of this method relies on two key prerequisites: First, data-driven approaches. For example, CLIP (Contrastive Language–Image Pre-training) used 400 million image-text pairs during training, primarily sourced from open datasets such as the internet, covering various visual scenes. The text data included image descriptive data (such as descriptive text), labels (ground truth labels), and titles. Second, large models are used; for example, ViT-L / 14 + Transformer has approximately 140M parameters, making it a large-scale model. Among them, ViT-L / 14 is a large visual Transformer (ViT) variant based on the Transformer encoder. Its core is to divide the image into 14×14 pixel patches and model global semantics through a 24-layer Transformer encoder. It is often used as a visual encoder (image encoder) for multimodal models such as CLIP, and has both strong feature extraction and cross-modal alignment capabilities.

[0048] For example, for any given image, the descriptive text obtained through a multimodal model might be: "A slightly overweight middle-aged person with short black hair, wearing black glasses and black dress pants, is standing on the floor, and there is a blanket beside them." As you can see, the descriptive text describes the image as a whole. However, these descriptions have both advantages and disadvantages in portraying a human figure. For instance, features such as "slightly overweight," "middle-aged," "short black hair," and "black glasses" are essential for training the task of re-identifying a person in disguise (i.e., training a model to extract features for this task). At the same time, features obtained from native multimodal training may also mislead the re-identification of a person in disguise. For example, the descriptive text might include "blanket," which is irrelevant to the person themselves. Furthermore, information about the shooting angle might also be negative features for re-identifying a person in disguise, because the re-identification task itself needs to cover different angles and scenes. It is important to note that black dress pants are a negative feature in the human re-identification task, and the influence of clothing needs to be removed.

[0049] Furthermore, existing multimodal models have a large number of parameters, making them extremely difficult to deploy in resource-constrained scenarios. Specifically, multimodal models possess a certain degree of generalization capability, which is often based on a large number of model parameters, such as 140M. Deploying such large-scale models in the cloud is difficult enough, let alone deploying them on edge devices. For example, a device might have only 0.5T or 1.0T of computing power, and considering that the device may also require real-time response (with high requirements for reaction speed), the processing time for a single algorithm is typically 70ms or even 50ms. This necessitates that the model's parameters be only a few megabytes (usually no more than 2M). Therefore, it is generally impossible to directly deploy a general-purpose multimodal model on an edge device without any pruning or miniaturization.

[0050] In addition, the fusion features extracted by current multimodal models have the following drawbacks:

[0051] 1. The input of a multimodal model is only the image itself. Even when attributes are involved, they are encoded in a way similar to one-hot encoding. Overall, the image representation is not as detailed as the text representation of a multimodal model (i.e., image features are not as accurate as text features). As a result, the learned human re-identification features (i.e., the fused features output by the multimodal model) are not accurate enough.

[0052] 2. Multimodal models are based on frameworks similar to human body shape and clothing mask. The training of human body shape and clothing mask is often done in stages, which will cause the accumulation of errors in multiple stages, ultimately leading to insufficient accuracy of multimodal models.

[0053] In addition to this, human re-identification after costume change also requires some key features, such as human posture.

[0054] The general features obtained by the multimodal models trained in the existing technology include irrelevant features and negative features, which are not completely equivalent to the features required for the re-identification of human figures in costume, and the features relied upon are insufficient. In order to improve the accuracy of features, the features required for the re-identification of human figures in costume should include some general features extracted by the multimodal model (such as hair and body shape), and also include key features related to the scene (such as human pose).

[0055] As can be seen above, there are many challenges in human re-identification under different clothing conditions. For example, clothing is a key feature in human re-identification under different clothing conditions, but it needs to be robust to different clothing (i.e., the accuracy should not decrease due to different clothes). Therefore, it is necessary to remove the influence of clothing. In addition, it is also necessary to incorporate some key features that can characterize the human body, such as the face, the shape of the person, and the posture. This allows for the integration of more supervised information (such as posture) into the feature learning process, resulting in a more powerful feature representation for human re-identification under different clothing conditions (i.e., higher accuracy of the features for human re-identification under different clothing conditions).

[0056] Based on this, embodiments of this application provide a model training method, a method, apparatus, device, and medium for re-identifying human figures in different costumes, in order to improve the accuracy of features used for re-identifying human figures in different costumes.

[0057] The following section first introduces a model training method provided in an embodiment of this application.

[0058] The model training method provided in this application can be applied to electronic devices, such as mobile phones and computers; this application does not limit the specific form of the electronic device. Furthermore, the electronic device can call the target feature extraction model. For example, the target feature extraction model can be deployed locally on the electronic device or on other devices communicating with the electronic device; the electronic device can simply call the target feature extraction model. Moreover, the electronic device used in the training process of the target feature extraction model can be the same as or different from the electronic device used in the human re-identification process; this application does not limit this.

[0059] In addition, the model training method provided in this application can be applied to the task of re-identifying human figures in disguise. For example, for an image containing human figures, in order to perform re-identification of human figures in disguise on the image, a scenario is to train a model to extract human figure features (features used for re-identification of human figures in disguise) in the image; or, in order to extract features used to train the re-identification model of human figures in disguise (i.e., the features can be used to train the re-identification model of human figures in disguise), this application does not limit this.

[0060] This application provides a model training method, including:

[0061] Obtain sample data pairs; wherein, the sample data pairs include: an image containing a human figure and a corresponding descriptive text for the image, the descriptive text for the image being used to describe the human figure region in the image, excluding clothing;

[0062] The sample data pairs are input into a target feature extraction model to be trained, so that the target feature extraction model generates fused features of the sample data pairs. The target feature extraction model includes a multimodal network, a feature extraction network, and a feature fusion network. The multimodal network is used to encode the image and descriptive text in the sample data pairs respectively, and fuse the encoded image features and text features to obtain a first feature of the sample data pairs. The feature extraction network is used to extract features from the first feature in multiple dimensions to obtain a second feature of the sample data pairs in multiple dimensions, including at least: human attributes, human detection, and clothing mask. The feature fusion network is used to fuse the second feature in the human attribute dimension and the second feature in the human detection dimension, and fuse the fused feature with the second feature in the clothing mask dimension to extract the fused feature.

[0063] Based on the features extracted by each network of the target feature extraction model, loss calculation is performed to obtain the current target loss;

[0064] In response to the determination that the preset convergence condition is not met based on the current target loss, the network parameters of the target feature extraction model are adjusted, and the process returns to the step of obtaining sample data pairs.

[0065] The model training method provided in this application first obtains sample data pairs. The sample data pairs include an image containing a human figure and corresponding descriptive text. The descriptive text corresponding to the image is used to describe the human figure region in the image except for clothing, that is, it does not include descriptions unrelated to the human figure or descriptions that do not include clothing. The sample data is used to train a target feature extraction model. The features extracted by the trained target feature extraction model do not contain irrelevant features or negative features, thereby improving the accuracy of the features for re-identifying human figures in different outfits.

[0066] The target feature extraction model in this application includes a multimodal network, a feature extraction network, and a feature fusion network. The multimodal network encodes the image and descriptive text in the sample data pair separately, and fuses the encoded image features and text features to obtain a first feature. The feature extraction network extracts features from the first feature in multiple dimensions, including human attributes, human detection, and clothing masking, to obtain a second feature in multiple dimensions. The feature fusion network first fuses the second feature in the human attribute dimension and the second feature in the human detection dimension, and then fuses the fused feature with the second feature in the clothing masking dimension to obtain a fused feature. During training, the current target loss is calculated based on the features extracted by each network of the target feature extraction model. If the current target loss does not meet the preset convergence condition, the network parameters of the target feature extraction model are adjusted, and sample data pairs are obtained for retraining.

[0067] The target feature extraction model can extract features from the first feature in multiple dimensions, and the feature fusion network can fuse features based on the second feature in multiple dimensions. Furthermore, by using the second feature of the clothing mask dimension, the negative influence of clothing features is further reduced, resulting in higher accuracy of the fused features extracted by the trained target feature extraction model. Therefore, this application can improve the accuracy of features used for human figure re-identification in clothing changes.

[0068] The following description, in conjunction with the accompanying drawings, provides an exemplary model training method according to an embodiment of this application.

[0069] like Figure 1 As shown in the embodiments of this application, a model training method may include the following steps:

[0070] S101: Obtain sample data pairs;

[0071] The sample data includes: an image containing a human figure and the corresponding descriptive text for the image, wherein the descriptive text is used to describe the human figure area in the image, excluding clothing.

[0072] When training a target feature extraction model, sample data pairs for model training can be obtained first. Considering that existing multimodal model training descriptions contain descriptions unrelated to human figures (such as blankets) and descriptions containing clothing, the sample data pairs in this application include: an image containing a human figure and the corresponding description text; the description text corresponding to the image is used to describe the human figure region in the image and does not contain descriptions of clothing, i.e., it does not contain descriptions unrelated to human figures or descriptions of clothing, thus providing a foundation for improving the accuracy of features used for human figure re-identification in clothing changes.

[0073] For example, in one implementation, the sample data pair is obtained by: processing an image containing a human figure using an existing multimodal model to obtain a descriptive text for the image; and removing descriptions of clothing and descriptions unrelated to the human figure from the descriptive text to obtain the descriptive text corresponding to the image. Alternatively, the descriptive text corresponding to the image can be pre-generated manually, for example, manually describing the human figure region in the image excluding clothing to obtain the descriptive text corresponding to the image. Then, a sample data pair containing the image and its corresponding descriptive text is constructed.

[0074] Sample data pairs can be pre-built and stored (e.g., stored in a database, or stored locally on the electronic device or in another device communicating with the electronic device), and the electronic device can access the sample data pairs.

[0075] It should be noted that the above methods for generating the descriptive text corresponding to the image and for obtaining sample data pairs are merely examples and should not constitute a limitation on this application.

[0076] S102: Input the sample data pairs into the target feature extraction model to be trained, so that the target feature extraction model generates fused features of the sample data pairs;

[0077] The target feature extraction model includes a multimodal network, a feature extraction network, and a feature fusion network. The multimodal network encodes the image and descriptive text in the sample data pair separately, and fuses the encoded image features and text features to obtain the first feature of the sample data pair. The feature extraction network extracts features from the first feature in multiple dimensions to obtain the second feature of the sample data pair in multiple dimensions, including at least: human attributes, human detection, and clothing mask. The feature fusion network fuses the second feature of the human attribute dimension and the second feature of the human detection dimension, and then fuses the fused feature with the second feature of the clothing mask dimension to extract the fused feature.

[0078] In this embodiment of the application, the target feature extraction model includes a multimodal network, a feature extraction network, and a feature fusion network. During the training process, sample data pairs can be input into the target feature extraction model to be trained. After passing through the multimodal network, feature extraction network, and feature fusion network of the target feature extraction model, the fused features of the sample data pairs are obtained.

[0079] In this process, the multimodal network can encode the image and descriptive text in the sample data pair separately, and then fuse the encoded image features and text features to obtain the first feature of the sample data pair. The first feature of the sample data pair is the feature after visual-language fusion. This process allows the semantic prior of the text to provide accurate semantic constraints and anchors for the subsequent feature extraction network to extract features from the image, avoiding problems such as background interference, ambiguity, and poor feature generalization that are prone to occur in the pure image (visual) extraction process. At the same time, it allows the human-shaped attributes (such as posture and body shape) of the text description to directly participate in feature modeling, improving the semantic consistency and task relevance of the obtained fused features.

[0080] The feature extraction network can extract features from the first feature in multiple dimensions to obtain multiple dimensions of second features for the sample data pair; namely, second features in the human figure attribute dimension, second features in the human figure detection dimension, and second features in the clothing mask dimension. The second features in the human figure attribute dimension can include descriptive features of human figure attributes such as hair, glasses, watch, age, and body shape. The second features in the human figure detection dimension can include features about the human bounding box and key points (such as classification and regression features). The second features in the clothing mask dimension are pixel-level semantic segmentation mask features. In the human figure region of the image, it accurately segments the pixel range corresponding to the clothing and assigns an independent mask label to this region (distinguishing it from human skin, hair, facial features, background, etc. at the pixel level). In other words, it "draws a precise pixel circle" for the clothing of the person in the image; the circled area is the effective range of the clothing mask (e.g., ...). Figure 2 The feature map extracted by the clothing mask feature extractor is shown in the image.

[0081] The feature fusion module can first fuse the second feature of the human figure attribute dimension and the second feature of the human figure detection dimension (e.g., by direct concatenation), and then fuse the fused feature with the second feature of the clothing mask dimension (e.g., by bitwise multiplication) to obtain the fused feature of the sample data pair.

[0082] The specific implementation process of each network and the details of its multiple dimensions will be described in detail in subsequent embodiments, and will not be repeated here.

[0083] S103: Based on the features extracted by each network of the target feature extraction model, calculate the loss to obtain the current target loss;

[0084] It should be noted that the multimodal network in this embodiment can directly use existing text encoders (for encoding text), image encoders (for encoding images), and visual-language fusion units (for fusing text features and image features); or, an existing text encoder can be used to train the image encoder and visual-language fusion unit to obtain the multimodal network. Furthermore, the multimodal network can be pre-trained, and the specific training process of the image encoder and visual-language fusion unit can be similar to existing technologies, which will not be elaborated here. That is, the training process for the target feature extraction model in this embodiment is equivalent to training the feature extraction network and feature fusion network of the target feature extraction model (or, the multimodal network can be trained simultaneously with the feature extraction network and feature fusion network, etc.).

[0085] After obtaining the fused features, loss calculation can be performed based on the features extracted by each network of the target feature extraction model. For example, in one implementation, if the feature extraction network and feature fusion network of the target feature extraction model are trained simultaneously, loss calculation can be performed based on the fused features, text features, and second features of multiple dimensions to obtain the current target loss. Specifically, the loss of the feature fusion network is calculated based on the fused features, and the loss of the feature extraction network is calculated based on the text features and the second features of multiple dimensions. Combining the losses of the feature extraction network and the feature fusion network yields the target loss. For instance, the target loss can be obtained by weighted fusion of the losses of the feature extraction network and the feature fusion network according to preset weight coefficients for the feature extraction network and the feature fusion network.

[0086] In the case of training both the feature extraction network and the feature fusion network simultaneously, the calculation methods for the loss of the feature extraction network and the loss of the feature fusion network are similar to those for the case when the training is divided into two stages. For details, please refer to the subsequent embodiments, which will not be elaborated here.

[0087] For example, in another implementation, the training process of the target feature extraction model in this application can be divided into two stages: the stage of training the feature extraction network and the stage of training the feature fusion network after the feature extraction network has been trained.

[0088] Optionally, based on the features extracted by each network of the target feature extraction model, loss calculation is performed to obtain the current target loss, including:

[0089] If the network parameters of the feature extraction network are not fully trained, loss calculation is performed based on text features and the second features of these multiple dimensions to obtain the current target loss; or,

[0090] Once the network parameters of the feature extraction network have been trained, loss calculation is performed based on the fused features to obtain the current target loss.

[0091] For the two-stage training process, if the network parameters of the feature extraction network are not fully trained, the current target loss, i.e. the network loss of the feature extraction network, can be calculated based on the text features and the second features of these multiple dimensions. If the network parameters of the feature extraction network are fully trained, the current target loss, i.e. the loss of the feature fusion network, can be calculated based on the fused features.

[0092] Whether the network parameters of the feature extraction network have been trained can be determined by calculating the loss of the feature extraction network. If the loss of the feature extraction network is less than the loss threshold set for the feature extraction network, then the network parameters of the feature extraction network have been trained. At this time, the current target loss can be calculated based on the fused features. If the loss of the feature extraction network is greater than the loss threshold, then the feature extraction network has not been trained. The loss of the feature extraction network (calculated based on text features and second features of multiple dimensions) can be directly used as the current target loss.

[0093] It should be noted that the specific calculation process of the current target loss will be described in detail in subsequent embodiments, and will not be repeated here.

[0094] In this embodiment, the training process of the target feature extraction model can be further divided into two stages. Adaptively, if the network parameters of the feature extraction network are not fully trained, the network loss of the feature extraction network can be calculated as the current target loss; if the network parameters of the feature extraction network are fully trained, the network loss of the feature fusion network can be calculated as the current target loss. This allows for accurate calculation of the current target loss of the target feature extraction model, providing a basis for subsequent precise adjustment of the network parameters (feature extraction network or feature fusion network).

[0095] It should be emphasized that the feature extraction network and feature fusion network in the embodiments of this application can be trained simultaneously (the target loss is the loss of both networks, and the parameter adjustment is also to adjust the parameters of both networks simultaneously), or they can be trained in stages. Specifically, the training method can be flexibly set according to factors such as the amount of sample data pairs, the difficulty of the costume-changing human re-identification task, and the different stages of model optimization.

[0096] S104: In response to determining that the preset convergence condition is not met based on the current target loss, adjust the network parameters of the target feature extraction model and return to the step of obtaining sample data pairs.

[0097] If the current target loss determines that the preset convergence condition is not met, the network parameters of the target feature extraction model are adjusted. For example, in one implementation, if the training process of the target feature extraction model is not divided into stages, i.e., the feature extraction network and the feature fusion network are trained simultaneously, then adjusting the network parameters of the target feature extraction model means adjusting the parameters of both the feature extraction network and the feature fusion network (assuming that the multimodal network has been trained; or, if the image encoder and visual-language fusion unit of the multimodal network have not been trained, their parameters can also be adjusted simultaneously).

[0098] For example, in another implementation, where the training process is divided into two phases, in response to determining that a preset convergence condition is not met based on the current target loss, the network parameters of the target feature extraction model are adjusted, including:

[0099] If the network parameters of the feature extraction network are not fully trained, adjust the network parameters of the feature extraction network.

[0100] or,

[0101] If the network parameters of the feature extraction network have been trained, adjust the network parameters of the feature fusion network.

[0102] Accordingly, the method also includes:

[0103] If the network parameters of the feature extraction network are not fully trained, in response to determining that the preset convergence condition is met based on the current target loss, the network parameters of the feature extraction network are determined to be fully trained, and the process returns to the step of obtaining sample data pairs.

[0104] If the training process of the target feature extraction model is divided into two stages, the adjustment of the network parameters of the target feature extraction network can be divided into two cases: The first case is when the network parameters of the feature extraction network are not fully trained (i.e., the current target loss is the network loss of the feature extraction network, and this network loss is greater than the loss threshold set for the feature extraction network). In this case, the network parameters of the feature extraction network can be adjusted (the feature fusion network can be trained after the feature extraction network is fully trained, or its network parameters can be adjusted). The second case is when the network parameters of the feature extraction network are fully trained (i.e., the network parameters of the feature extraction network are fully trained, the current target loss is the network loss of the feature fusion network, and this network loss is greater than the loss threshold set for the feature fusion network). In this case, the network parameters of the feature extraction network are the fully trained network parameters. The fully trained feature extraction network is used to continue training the feature fusion network, and the network parameters of the feature fusion network are adjusted.

[0105] In addition, if the network parameters of the feature fusion network are not fully trained, but the predetermined convergence condition is met based on the current target loss, since the current target loss is the network loss of the feature fusion network, it means that the network parameters of the feature fusion network are fully trained. We can then return to the step of obtaining sample data pairs and use the trained feature fusion network to process the sample data pairs to continue training the feature fusion network.

[0106] In this embodiment, if the training process of the target feature extraction model is divided into two stages, then if the preset convergence condition is not met based on the current target loss, the network parameters of the feature extraction network or the network parameters of the feature fusion network are accurately adjusted according to whether the network parameters of the feature extraction network have been trained, so as to train the feature extraction network and the feature fusion network in a targeted and accurate manner.

[0107] In the technical solution of this application, the acquisition, storage, use, processing, transmission, provision and disclosure of sample data are all carried out with the user's authorization.

[0108] It should be noted that the images in the sample data pairs of this embodiment do not reflect the personal information of a specific user. The images in the sample data pairs of this embodiment are from publicly available datasets.

[0109] The model training method provided in this application first obtains sample data pairs. The sample data pairs include an image containing a human figure and corresponding descriptive text. The descriptive text corresponding to the image is used to describe the human figure region in the image except for clothing, that is, it does not include descriptions unrelated to the human figure or descriptions that do not include clothing. The sample data is used to train a target feature extraction model. The features extracted by the trained target feature extraction model do not contain irrelevant features or negative features, thereby improving the accuracy of the features for re-identifying human figures in different outfits.

[0110] The target feature extraction model in this application includes a multimodal network, a feature extraction network, and a feature fusion network. The multimodal network encodes the image and descriptive text in the sample data pair separately, and fuses the encoded image features and text features to obtain a first feature. The feature extraction network extracts features from the first feature in multiple dimensions, including human attributes, human detection, and clothing masking, to obtain a second feature in multiple dimensions. The feature fusion network first fuses the second feature in the human attribute dimension and the second feature in the human detection dimension, and then fuses the fused feature with the second feature in the clothing masking dimension to obtain a fused feature. During training, the current target loss is calculated based on the features extracted by each network of the target feature extraction model. If the current target loss does not meet the preset convergence condition, the network parameters of the target feature extraction model are adjusted, and sample data pairs are obtained for retraining.

[0111] The target feature extraction model can extract features from the first feature in multiple dimensions, and the feature fusion network can fuse features based on the second feature in multiple dimensions. Furthermore, by using the second feature of the clothing mask dimension, the negative influence of clothing features is further reduced, resulting in higher accuracy of the fused features extracted by the trained target feature extraction model. Therefore, this application can improve the accuracy of features used for human figure re-identification in clothing changes.

[0112] To facilitate understanding, the following is a general introduction to a model training method provided in the embodiments of this application.

[0113] like Figure 2 As shown, the target feature extraction model provided in this application embodiment may include: a multimodal network, a feature extraction network, and a feature fusion network;

[0114] The image and its corresponding descriptive text are encoded using an image encoder and a text encoder, respectively, to obtain image features and text features. These features are then fused using visual language to obtain fused image features (i.e., the first feature). The descriptive text may include: "short black hair; wearing black glasses; a watch on his left wrist; middle-aged man; slightly overweight;". The extracted text features may include: "short black hair; black glasses; a watch; middle-aged man; slightly overweight;". The image encoder can be a CNN backbone network; the image features can be multi-scale features (i.e., features extracted from multiple scales of the image). The first feature can be represented using (N, C, H, W). N (Batch Size): Batch size, representing the number of sample data pairs being processed (e.g., processing 8 sample data pairs at a time, N=8); C (Channel): Number of channels, representing the dimension of the feature (e.g., a convolutional layer (Conv layer) in a visual-language fusion processor outputs 64 channels, so C=64; the original RGB image has C=3); H (Height): Height of the feature map of the first feature, W (Width): Width of the feature map of the first feature, both representing the spatial dimension of the first feature. Text features can be represented by (N, K, embed), where K represents the types of human attributes (e.g., human attributes include five types: hair, glasses, watch, age, and body shape, so K=5).

[0115] The feature extraction network includes a human attribute feature extractor, a human detection feature extractor, and a clothing mask feature extractor, which are used to extract second features in the dimensions of human attributes, human detection, and clothing mask, respectively. The second features in the human attribute dimension can be compared with text features to calculate a contrast loss. For example, the second features in the human attribute dimension include second features of hair attributes, glasses attributes, etc., and are compared with text features describing hair, glasses, etc., etc., to calculate contrast losses for hair attributes, glasses attributes, etc., respectively. Additionally, the second features in the human attribute dimension can be input into the corresponding classifier to obtain classification results, and the classification results are used to calculate classification losses against pre-labeled results. The second features in the human detection dimension can be input into the corresponding classifier or regressor, and the results are used to calculate classification or regression losses. The features in the clothing mask dimension are used to calculate losses against the ground truth of the clothing mask. In this context, the second feature of a specific human attribute in the human attribute dimension can be represented by (N, embedded_X_cls, H, W), where X represents a specific human attribute, such as hair for the hair attribute. The second feature in the human detection dimension can include features of the bounding box used for classification (N, embedded_box_cls, H, W), features of the bounding box used for regression analysis (N, 4×reg_max, H, W), features of the human keypoint used for classification (N, embedded_keypoint_cls, H, W), and features of the human keypoint used for regression analysis (N, embedded_keypoint_reg, H, W). The second feature in the clothing mask dimension can be represented by (N, embedded_cls_cls, H, W). `cls` represents Classification; `reg` represents Regression; `keypoint` represents the human keypoint; and `box` represents the bounding box.

[0116] The feature fusion network can first fuse the second feature of the human body attribute dimension and the second feature of the human body detection dimension ("+" representation concatenation) to obtain the fused feature; the fused feature is then fused with the second feature of the clothing mask dimension ("×" representation multiplication), and then passed through a CNN to obtain the fused feature. The fused feature can be used to calculate the loss of the feature fusion network, such as ID loss and triplet loss.

[0117] As can be seen, the target feature extraction model in this embodiment is trained on a multimodal basis. By mining detailed descriptions of human figures (descriptive text, excluding descriptions unrelated to human figures and descriptions excluding clothing), the feature accuracy of re-identification of human figures in different outfits is improved.

[0118] Unlike existing multimodal models (such as CLIP), this embodiment first removes features from the descriptive text that are irrelevant to the re-identification of the human figure in costume, such as the blanket next to the person and the shooting angle of the image, with a focus on removing clothing descriptions; secondly, it borrows from traditional methods for re-identifying human figures in costume and incorporates key features that better characterize the human figure, such as... Figure 2 Features for human detection have been added, specifically including human bounding boxes and key human features. These key features are fused and used for human re-identification after clothing change. To further eliminate the impact of clothing on the human re-identification task, the target feature extraction model in this embodiment adds a clothing mask feature extractor to extract a second feature of the clothing mask dimension. This second feature of the clothing mask dimension is used in the feature fusion process to eliminate the impact of clothing on the overall task.

[0119] For the fusion features, this application adopts an end-to-end task-oriented training method. Current methods mostly use the method of training a separate model for each key feature to extract key features. This method is not an end-to-end training process, the training steps are more cumbersome, and the lack of task orientation makes the fusion of each key feature different, resulting in low accuracy of the final fusion features.

[0120] For example, in one implementation, loss calculation is performed based on fused features to obtain the current target loss, including:

[0121] The humanoid ID classifier is used to classify the fusion features of the sample data pairs to obtain the humanoid ID to which the humanoid in the image belongs in the sample data pair.

[0122] Based on the obtained human ID and the ground truth human ID of the image in the sample data pair, the loss is calculated to obtain the ID loss;

[0123] The triplet loss is calculated based on the fusion features of the sample data pairs, as well as the fusion features of positive samples and negative samples in other sample data pairs corresponding to the sample data pairs.

[0124] Based on the obtained ID loss and triplet loss, determine the current target loss.

[0125] Once the network parameters of the feature extraction network have been trained, the network parameters are fixed to the trained parameters (i.e., the feature extraction network is frozen). The trained feature extraction network is used to extract features from the sample data pairs, and then fused through a feature fusion network to obtain rich human-shaped feature profiles. These profiles are then further abstracted through a CNN neural network to obtain the final fused features.

[0126] When calculating the current network loss of the feature fusion network based on the fusion features, considering that the fusion features of the same person need to be similar, this embodiment of the application uses ID loss and triplet loss to characterize the network loss of the feature fusion network (for example, the ID loss and triplet loss can be weighted and fused). Specifically, the fusion features of the sample data pair are first classified by a human ID classifier to obtain the human ID to which the human in the image of the sample data pair belongs; based on the human ID and the ground truth value of the human ID labeled for the image (e.g., human 1, which can be pre-labeled), the contrastive loss is calculated to obtain the ID loss.

[0127] Furthermore, based on the fusion features of the sample data pairs, and the fusion features belonging to positive samples (i.e., fusion features of the same humanoid ID) and fusion features belonging to negative samples (i.e., fusion features of different humanoid IDs) in other sample data pairs (other sample data pairs in the same batch) corresponding to the sample data pairs, a triplet loss is calculated. Specifically, the triplet loss ensures that the fusion feature satisfies the following distance metric: d(a, p) + margin < d(a, n), where a represents the current fusion feature, p is the fusion feature of positive samples (fusion features of other images belonging to the same ID), and n is the fusion feature of negative samples (fusion features of other images belonging to different IDs). This ensures that the fusion features of different IDs satisfy a certain distance constraint. Margin can be understood as a boundary or interval, a manually set hyperparameter (usually a positive number, such as 0.2 or 0.5), whose core function is to force a sufficient "gap" between the distance of positive sample pairs and the distance of negative sample pairs.

[0128] In this embodiment, ID loss ensures that within the same batch size (i.e., pairs of sample data fed into the model for training in the same batch), the same ID is considered to be of the same class, making the fused features belonging to the same ID more compact. For triplet loss, similarity is calculated pairwise for each fused feature within the same batch size, ensuring that the distance metric satisfies the above formula. By determining the network loss of the feature fusion network through triplet loss and ID loss, it is ensured that fused features belonging to the same humanoid ID are considered to be of the same class. Simultaneously, there is sufficient distance metric between fused features of different humanoid IDs, thus ensuring the accuracy of the fused features.

[0129] Optionally, in another embodiment of this application, loss calculation is performed based on the text features and the second features of the multiple dimensions to obtain the current target loss, including:

[0130] The loss is calculated based on the difference between the text features and the second feature of the humanoid attribute dimension to obtain the first type of loss. The loss is also calculated based on the difference between the task result of the first type of task determined by the second feature of the humanoid attribute dimension and the corresponding annotation result to obtain the second type of loss.

[0131] Based on the difference between the task results of the second type of task determined by the second feature of the human figure detection dimension and the corresponding labeled results, the loss is calculated to obtain the third type of loss;

[0132] The fourth type of loss is obtained by calculating the loss based on the difference between the second feature of the clothing mask dimension and the clothing mask annotation results of the image in the sample data pair;

[0133] Calculate the current target loss based on the first type of loss, the second type of loss, the third type of loss, and the fourth type of loss;

[0134] The first type of task is a classification task about humanoid attributes, and the method of determining the task result of the first type of task through the second feature of the humanoid attribute dimension includes: inputting the second feature of the humanoid attribute dimension into the task processing network used to perform the first type of task to obtain the task result of the first type of task;

[0135] The second type of task includes classification tasks or regression tasks. The method of determining the task result of the second type of task through the second feature of the human detection dimension includes: inputting the second feature of the human detection dimension into the task processing network used to perform the second type of task to obtain the task result of the second type of task.

[0136] based on Figure 2 In the embodiment shown, the method for calculating the network loss of the feature extraction network based on text features and multiple dimensions of second features in this application is as follows:

[0137] First, a contrastive loss is calculated based on the difference between the text features and the second features of the human-shaped attribute dimension, resulting in the first type of loss. Then, a classification loss is calculated based on the difference between the task result of the first type of task determined by the second features of the human-shaped attribute dimension and the corresponding labeled result, resulting in the second type of loss. The first type of task is a classification task about human-shaped attributes, and its result is obtained by inputting the second features of the human-shaped attributes into the task processing network used to perform the first type of task. For example, the task processing network for the first type of task can be a classifier corresponding to the human-shaped attributes (e.g., a classifier used to classify hair attributes).

[0138] In addition, a third type of loss can be calculated based on the difference between the task results of the second type of task determined by the second feature of the human detection dimension and the corresponding annotation results. The corresponding annotation results of the second type of task can be matched with the specific task content, which will be described in detail in subsequent embodiments and will not be elaborated here. Furthermore, a fourth type of loss (such as calculating BCE Loss (Binary Cross Entropy Loss)) can be calculated based on the difference between the second feature of the clothing mask dimension and the clothing mask annotation results of the image in the sample data pair (which can be obtained by downsampling the original mask of the image).

[0139] After obtaining the losses for each dimension, the current network loss of the feature extraction network can be calculated based on the first, second, third, and fourth type losses to obtain the current target loss. For example, the first, second, third, and fourth type losses can be weighted and fused according to their corresponding weight coefficients (the weight coefficients can be set according to requirements or actual conditions) to obtain the current target loss.

[0140] In this embodiment, when the network parameters of the feature extraction network are not fully trained, the loss can be calculated separately for each dimension, and the current target loss can be calculated based on the loss of each dimension. The loss of the human figure attribute dimension includes the contrast loss with text features and the classification loss for the task results of the first type of task; the loss of the human figure detection dimension includes the loss between the task results of the second type of task and the corresponding annotation results; the loss of the clothing mask dimension includes the loss between the second feature of the clothing mask dimension and the clothing mask annotation result of the image; in this way, the obtained target loss can accurately represent the loss of each dimension of the feature extraction network, ensuring the accuracy of the feature extraction network training.

[0141] In addition, the above methods also include:

[0142] The fifth type of loss is calculated based on the difference between the target intermediate features and the features of a predetermined category in the text description; where the target intermediate features are: intermediate features about the predetermined category generated during the process of determining the task result of the specified second type of task through the second features of the human detection dimension;

[0143] The second type of task is specified as a category task;

[0144] Based on the first type of loss, the second type of loss, the third type of loss, and the fourth type of loss, calculate the current target loss, including:

[0145] Calculate the current target loss based on the first type of loss, the second type of loss, the third type of loss, the fourth type of loss, and the fifth type of loss.

[0146] The second type of task for the second feature of the human detection dimension can be a classification task. In determining the result of this classification task, intermediate features for a predetermined category can be generated, such as features for human bounding box classification. These intermediate features are used for classification through the classification network of the second type of task (e.g., classifying the human figure as a middle-aged man). At this point, the difference between the intermediate features and the features of the predetermined category in the text description can also be calculated to obtain the fifth type of loss. The fifth type of loss can also be applied to the calculation of the target loss, and its calculation process is similar (e.g., weighted fusion).

[0147] It should be noted that the predetermined category is a certain attribute, such as hair, glasses, watch, age or body shape. The second type of task can include the human bounding box classification task. The target intermediate feature is the feature used for human bounding box classification. This feature can be compared with the text features of attributes such as hair, glasses, watch, age and body shape in the text description to calculate the loss, and obtain the fifth type of loss. The specific calculation process can be similar to the existing technology, and will not be elaborated here.

[0148] In this embodiment, the second type of task can be a classification task. During the execution of the second type of task, intermediate target features for classification can be generated. These intermediate target features can be compared with features of a predetermined type in the text description to calculate the loss, resulting in a fifth type of loss. The fifth type of loss can also be applied to the current network loss calculation process of the feature extraction network. In this way, the dimension on which the loss of the feature extraction network depends can be further increased, and the accuracy of the feature extraction network training can be improved by calculating the network loss of the feature extraction network in this way.

[0149] Optionally, in another embodiment of this application, the feature extraction network includes multiple feature extractors, each of which is used to extract a feature in one dimension;

[0150] The features of the humanoid attribute dimension include: various descriptive features about humanoid attributes;

[0151] The features of the human detection dimension include: at least one feature about the human bounding box and at least one feature about the human key points; at least one feature about the human bounding box includes: features about the human bounding box used for classification and / or features used for regression analysis; at least one feature about the human key points includes: features about the human key points used for classification and / or features used for regression analysis.

[0152] The loss calculation based on the difference between the text features and the second feature of the humanoid attribute dimension yields a first type of loss, including:

[0153] For each descriptive feature about humanoid attributes, a loss is calculated based on the difference between the descriptive feature and the content of the description of humanoid attributes in the text features, to obtain the first type of loss corresponding to the descriptive feature about humanoid attributes.

[0154] The loss calculation, based on the difference between the task result of the first type of task determined by the second feature of the humanoid attribute dimension and the corresponding labeled result, yields the second type of loss, including:

[0155] For each descriptive feature about humanoid attributes, based on the difference between the task result of the first task determined by the descriptive feature about humanoid attributes and the corresponding labeled result, loss is calculated to obtain the second type of loss corresponding to the descriptive feature about humanoid attributes.

[0156] The third type of loss is obtained by calculating the difference between the task results of the second type of task determined by the second feature of the human figure detection dimension and the corresponding labeled results, including:

[0157] For each feature in the human detection dimension, loss is calculated based on the difference between the task result of the corresponding second type of task determined by the feature and the corresponding annotation result, to obtain the third type of loss corresponding to the feature.

[0158] In the embodiments of this application, such as Figure 2 As shown, the feature extraction network can include multiple feature extractors, each extracting a feature in one dimension; the features of the human-shaped attribute can include various descriptive features about human-shaped attributes, such as hair, glasses, watch, age, and body shape. Each attribute has a corresponding first-class task, and... Figure 2 The feature extractor for the human-shaped attribute dimension can be understood as a feature extractor that includes multiple attributes such as hair, glasses, watch, age, and body shape; and each attribute can have a corresponding task processing network for performing the first type of task (e.g., the feature extractor for the hair attribute is connected to the corresponding hair attribute classifier), and has a corresponding annotation result (e.g., the corresponding annotation result for the hair attribute is the result represented by hair, such as: short black hair).

[0159] Features in the human detection dimension can include features for classification and / or regression analysis of human bounding boxes; and features for classification and / or regression analysis of human keypoints; that is, the feature extractor for the human detection dimension can include: an extractor for extracting features for classification of human bounding boxes, an extractor for extracting features for regression analysis of human bounding boxes, an extractor for extracting features for classification of human keypoints, and an extractor for extracting features for regression analysis of human keypoints (i.e. Figure 4 The human bounding box classifier, human bounding box regression extractor, keypoint classification extractor, and keypoint regression extractor are described in the text.

[0160] It should be noted that the features used for classification are discrete or probabilistic features, used to determine category affiliation through classification tasks; while the features used for regression analysis are continuous numerical features, used to determine specific locations or sizes (or their deviations) through regression tasks. Furthermore, the classification and regression tasks in this application are only performed when determining the network loss of the feature extraction network. After the target feature extraction model has been trained, if the model is used to extract fusion features for humanoid re-identification in costume, then the classification and regression tasks do not need to be performed.

[0161] Adaptively, the first type of loss can be understood as the contrast loss between each descriptive feature of a human-shaped attribute and the corresponding content about that human-shaped attribute in the text features (e.g., the contrast loss between the descriptive feature of the hair attribute and the content about hair in the text features; the contrast loss between the descriptive feature of the glasses attribute and the content about glasses in the text features, etc.). The second type of loss is the classification loss between the task result of the first task determined by each descriptive feature of a human-shaped attribute and the corresponding labeled result (e.g., the task result of the first task determined by the hair attribute is short black hair, and the corresponding labeled result is short black hair; the classification loss is calculated for both; the same applies to attributes such as glasses).

[0162] It should be noted that the first type of loss calculated for various descriptive features of humanoid attributes can be weighted and fused to obtain the first type of loss for the humanoid attribute dimension; and the second type of loss for the humanoid attribute dimension can be obtained in a similar manner.

[0163] The third type of loss is calculated for each feature in the human detection dimension, based on the difference between the task result of the corresponding second-type task determined by that feature and the corresponding labeled result. For example, for a feature related to human bounding boxes used for classification, the task result of the corresponding second-type task (human bounding box classification task) is determined by that feature, i.e., the classification result of the human bounding box, and the corresponding labeled result is the ground truth value of the labeled human bounding box (e.g., middle-aged man). The classification loss is calculated between the two to obtain the third type of loss for the feature related to human bounding boxes used for classification. Similarly, for a feature related to human bounding boxes used for regression analysis, the task result of the corresponding second-type task (human bounding box regression task) is determined by that feature, i.e., the regression result of the human bounding box (i.e., the position of the human bounding box in the image), and the corresponding labeled result is the actual position of the labeled human bounding box. The classification loss is calculated between the two to obtain the third type of loss for the feature related to human bounding boxes used for regression analysis.

[0164] For the features used for classification of human keypoints, the task result of the corresponding second type of task (human keypoint classification task) is determined by these features, which is the classification result of the human keypoints (i.e., the category to which each keypoint belongs). The corresponding annotation result is the location of the annotated human keypoints. The classification loss is calculated between the two to obtain the third type of loss of the features used for classification of human keypoints. For the features used for regression analysis of human keypoints, the task result of the corresponding second type of task (human keypoint regression task) is determined by these features, which is the regression result of the human keypoints (i.e., the location of each keypoint in the image). The classification loss is calculated between the two to obtain the third type of loss of the features used for regression analysis of human keypoints.

[0165] In this embodiment of the application, the human key points may include 17 key points: nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle. Using the features of human key points for classification, the task result of the second type of task can be: [1,1,1,0,1,1,1,1,1,1,1,1,1,1,1,1,0,1], where 1 indicates that the key point at the corresponding position is visible and 0 indicates that it is not visible. This task result indicates that the key points of the left ear and the left ankle are not visible, while the other human key points are visible.

[0166] In this embodiment, the feature extraction network may include multiple feature extractors, each of which extracts a feature in one dimension. Furthermore, the features in the human-shape attribute dimension may include various descriptive features about human-shape attributes, and the feature extractor in the human-shape attribute dimension can be understood as a feature extractor containing multiple human-shape attributes. The features in the human-shape detection dimension may include features for human-shape bounding boxes used for classification and / or features for regression analysis, as well as features for human-shape keypoints used for classification and / or features for regression analysis. The feature extractor in the human-shape detection dimension may include these four types of feature extractors. Each human-shape attribute can correspond to the task result of a first task, and each feature in the human-shape detection dimension can correspond to the task result of a second task. Thus, the first type of loss and the second type of loss are calculated for multiple human-shape attributes. The third type of loss is calculated based on the classification and regression tasks of human-shape bounding boxes and human-shape keypoints, making the network loss of the feature extraction network richer and the features that the feature extraction network can extract richer, providing a foundation for improving the accuracy of fusion features used for human-shape re-identification in costume changes.

[0167] The following is an exemplary description of a model training method provided by an embodiment of this application.

[0168] First, train the multimodal network:

[0169] Step 1: Multimodal Feature Extraction. This application is based on a general multimodal model training paradigm, such as... Figure 2 As shown, the image modality is encoded by the image encoder, and the text modality is encoded by the text encoder. The descriptive text needs to be segmented into words before being input into the text encoder, forming individual nouns or phrases. Then, it is processed by the CLIP Text Model (i.e., the text model in CLIP; this application directly uses the text encoder in the existing multimodal model) to generate fixed text feature text embeddings.

[0170] Step 2: Multimodal Feature Fusion. When fusing image and text features, FPN (Feature Pyramid Networks) is usually used. However, FPN has the disadvantages of unclear feature fusion and incomplete feature characterization. YOLOv5 adopts the PAFPN (Path Aggregation Feature Pyramid Network) structure, which has better feature fusion. PAFPN has an additional rightmost top-to-bottom path compared to FPN, making the feature representation more specific.

[0171] In the fusion of text features and image features, this application adopts a text-guided approach (i.e., text is used as a guide, for example, as a prompt word) to make the fusion of image features and text features more specific, and the feature similarity between similar regions (features of regions in an image) and corresponding text (text features) is higher.

[0172] For the human-like attribute dimension in the feature extraction network, the process is as follows: Figure 3 As shown:

[0173] Based on the image features and text features corresponding to the image and descriptive text, a fused image feature (i.e., the first feature) is obtained through visual-language fusion. Taking hair attribute as an example, the hair attribute feature extractor extracts hair attribute features from the image. The hair attribute feature is compared with the text features related to hair attribute in the text features to calculate the hair attribute contrast loss (i.e., the first type loss of hair attribute). After the hair attribute feature is processed by CNN, hair attribute classification is performed to obtain the hair attribute classification result (e.g., short black hair). The classification result is compared with the annotation result for hair attribute (i.e., the ground truth value of the image for hair attribute) to calculate the classification loss (i.e., the second type loss of hair attribute).

[0174] Among them, the second type of feature of the human figure attribute dimension is the feature that is beneficial to the re-identification of human figures after changing clothes by the fusion image feature mined by the multimodal network. This part takes the hair attribute as an example for illustrative introduction. Other human figure attributes can be calculated in a similar way for the first type of loss and the second type of loss of the human figure attribute. Then, based on the first type of loss and the second type of loss of each human figure attribute, the first type of loss and the second type of loss of the human figure attribute dimension are obtained.

[0175] In this embodiment, in addition to the comparison loss between humanoid attribute features and text features obtained by the multimodal network (the descriptive features of each humanoid attribute are compared with the content of the description of that humanoid attribute in the corresponding text features), this application adds a classification loss for humanoid attributes. This can provide some recognition ability for negative samples and increase the generalization of the model.

[0176] For the human detection dimension in the feature extraction network, the process is as follows: Figure 4 As shown:

[0177] For the fused image features (i.e., the first feature) obtained through a multimodal network, Figure 4(The text is omitted here). The corresponding features (features for classification and regression analysis of human bounding boxes, and features for classification and regression analysis of human keypoints) are extracted using a human bounding box classifier, a human bounding box regression extractor, a keypoint classification extractor, and a keypoint regression extractor, respectively. Specifically, the human bounding box classification features are compared with the descriptions of each human attribute in the text features to calculate the human bounding box classification contrast loss. The human bounding box classification features are classified using the corresponding classifier, and the classification result (e.g., middle-aged man) is compared with the corresponding annotation result (i.e., the pre-annotated ground truth value for the human figure in the image) to calculate the classification loss, resulting in the human bounding box classification loss. The human bounding box regression analysis features are regressed using the corresponding regressor, and the regression result (predicted human bounding box position) is compared with the corresponding annotation result (actual human bounding box position) to calculate the regression loss, resulting in the human bounding box regression loss. Features used for classification of human keypoints are classified using corresponding classifiers. The classification results (e.g., [1,1,1,0,1,1,1,1,1,1,1,1,1,1,1,0,1]) are compared with the corresponding annotation results (the results showing whether each human keypoint is actually visible) to calculate the classification loss of the human keypoints. Features used for regression analysis of human keypoints are regressed using corresponding regressors. The regression results (the predicted positions of each keypoint) are compared with the corresponding annotation results (the actual positions of each human keypoint) to calculate the regression loss of the human keypoints.

[0178] The human detection in this application mainly focuses on learning human bounding boxes and human key points. The loss mainly comes from five dimensions: human bounding box classification loss, human bounding box regression loss, human key point classification loss, human key point regression loss, and human classification comparison loss.

[0179] For the clothing mask dimension in the feature extraction network, the process is as follows: Figure 5 As shown:

[0180] The second feature (e.g., clothing mask dimension) is extracted from the fused image features using a clothing mask feature extractor. Figure 5 The mask image indicated by the rightward arrow); the second feature of the clothing mask dimension and the clothing mask annotation results of the image (e.g. Figure 5 The loss is calculated by performing a loss calculation (such as calculating BCE Loss) on the mask image indicated by the down arrow in the middle, resulting in the fourth type of loss.

[0181] The clothing mask dimension mainly involves learning the clothing mask of the humanoid figure. During the training phase, the labeled clothing mask and the extracted second feature map of the clothing mask dimension are used to calculate the loss. After the feature extraction network is trained, in the prediction phase, the extracted second feature of the clothing mask dimension is multiplied bit by bit with the feature fused with other dimensions. The resulting fused feature is used for learning the humanoid figure re-identification task or for humanoid figure re-identification processing (for details, please refer to the implementation example of humanoid figure re-identification, which will not be elaborated here).

[0182] Training the feature fusion network:

[0183] After the multimodal network and feature extraction network are trained, the second features of the human attributes dimension (features of human attributes such as hair / glasses / watches / age / body shape, etc.) and the second features of the human detection dimension (human bounding box classification and regression features, human key point classification and regression features) obtained by the trained multimodal network and feature extraction network are fused (such as splicing) to obtain the fused feature F. Then, the second feature mask of the clothing mask dimension is applied to the fused feature to obtain (F×(1 - mask)), which is the Reid (ReID, Person Re-identification) feature that eliminates the influence of clothing. Finally, this feature is used for learning the human re-identification task in clothing change.

[0184] This application presents an end-to-end key feature extraction and fusion framework, encompassing both key feature extraction and the final generation of Reid features. Therefore, it employs a two-stage training approach:

[0185] Phase 1: Key Feature Learning Phase

[0186] Learning to extract human-like attribute dimensions (hair / glasses / watch / age / body shape / clothing); generating corresponding representation vectors for each human-like attribute. These vectors are used for both comparative loss with text features and separate classification, increasing the robustness of the model.

[0187] Learning human detection dimension feature extraction (human bounding box classification and regression features, human key point classification and regression features).

[0188] Learning the clothing mask dimension feature extraction: The second feature of the clothing mask dimension, namely a pred_mask (H / 4, W / 4, where H and W represent the dimensions of the original feature, i.e., width and height), is used to calculate the BCE loss along with gt_mask (the clothing mask annotation result, which can be obtained by downsampling the original mask of the image). In the prediction stage, the mask generated based on the second feature of the clothing mask dimension is used to mask the fused features, removing the influence of clothing on human re-identification features and the human re-identification task.

[0189] Phase 2: Humanoid Reid Feature Fusion Phase

[0190] First, the network parameters of the feature extraction network are frozen; then, the features extracted by the trained feature extraction network are fused to obtain a rich depiction of basic human features.

[0191] The basic human-shaped features are abstracted using a CNN neural network to obtain the final human-shaped feature vector (fused features). The fused features are derived from two supervisions: the ID loss and the triplet loss of the fused features.

[0192] After being characterized by ID loss and triplet loss (i.e., the feature fusion network is trained by using this loss), the fused features extracted by the trained feature fusion network are the features extracted in this application for human re-identification in costume.

[0193] This application integrates a text encoder from a multimodal model with an image encoder, and extracts key features across multiple dimensions through visual-language fusion. These features are then applied to the process of re-identifying a human figure in costume, improving the accuracy of the extracted features. Furthermore, unlike existing large-scale multimodal models, this application directly reparameterizes the text encoder, eliminating the need for training. Other networks or modules have fewer parameters, typically only 20M, allowing the lightweight target feature extraction model to be deployed on the device. This end-to-end training mechanism further enhances the accuracy of the final costume-re-identification task.

[0194] The target feature extraction model in this application embodiment has stronger accuracy in re-identifying human figures in different outfits: the feature extraction module can extract general features (second features in the human figure attribute dimension) and key features for re-identifying human figures in different outfits (second features in the human figure detection dimension and second features in the clothing mask dimension), with more feature dependencies and removal of human figure-irrelevant features and clothing features, which can greatly improve the feature accuracy of re-identifying human figures in different outfits, especially when the human figures look very similar, it can often accurately achieve the task of re-identifying human figures in different outfits through key features. Furthermore, due to the lightweight model training and fewer model parameters, the deployment cost of the target feature extraction model is lower, especially in some edge scenarios.

[0195] Based on the above method embodiments, this application also provides a method for re-identifying a human figure in different outfits, such as... Figure 6 As shown, the method includes:

[0196] S601: Obtain the image to be recognized and the corresponding descriptive text of the image to be recognized; wherein, the descriptive text corresponding to the image to be recognized is used to describe the human-shaped region in the image to be recognized, excluding clothing;

[0197] S602: Input the image to be identified and the corresponding descriptive text into the target feature extraction model to obtain fused features; wherein, the target feature extraction model is trained according to the above model training method;

[0198] S603: Perform human figure re-identification processing based on fusion features.

[0199] In this embodiment, during the re-identification of a human figure in disguise, the descriptive text corresponding to the image to be identified is only used to describe the human figure region in the image, excluding clothing; descriptions unrelated to the human figure and descriptions of clothing are removed. A target feature extraction model trained using the aforementioned model training method is used to obtain a fusion feature of the image to be identified and its corresponding descriptive text. This fusion feature is obtained by fusing features from multiple dimensions and does not include descriptions unrelated to the human figure or descriptions of clothing. Using this fusion feature for re-identification of a human figure in disguise improves the accuracy of the re-identification process.

[0200] The specific process of humanoid re-identification after costume change can be similar to existing technologies and will not be elaborated here.

[0201] The method for re-identifying a human figure in disguise provided in this application embodiment does not include descriptions unrelated to human figures or descriptions of clothing in the descriptive text corresponding to the image to be identified. The target feature extraction model obtained through the above model training method can extract features from the image to be identified and the corresponding descriptive text in multiple dimensions, and further reduce the influence of negative features such as clothing by using clothing masks. The accuracy of the obtained fused features is higher. Therefore, using the fused features for re-identifying a human figure in disguise can improve the accuracy of re-identifying a human figure in disguise.

[0202] Based on the above method embodiments, this application also provides a model training apparatus, such as... Figure 7 As shown, the device includes:

[0203] The first acquisition module 710 is used to acquire sample data pairs; wherein, the sample data pairs include: an image containing a human figure and a corresponding descriptive text for the image, wherein the descriptive text for the image is used to describe the human figure region in the image, excluding clothing.

[0204] The first input module 720 is used to input the sample data pair into the target feature extraction model to be trained, so that the target feature extraction model generates fused features of the sample data pair; the target feature extraction model includes: a multimodal network, a feature extraction network, and a feature fusion network; the multimodal network is used to encode the image and descriptive text in the sample data pair respectively, and fuse the encoded image features and text features to obtain a first feature of the sample data pair; the feature extraction network is used to extract features from the first feature in multiple dimensions to obtain a second feature of the sample data pair in multiple dimensions, the multiple dimensions including at least: human attributes, human detection, and clothing mask; the feature fusion network is used to fuse the second feature of the human attribute dimension and the second feature of the human detection dimension, and fuse the fused feature with the second feature of the clothing mask dimension to extract the fused feature;

[0205] The calculation module 730 is used to perform loss calculation based on the features extracted by each network of the target feature extraction model to obtain the current target loss;

[0206] The adjustment module 740 is used to adjust the network parameters of the target feature extraction model in response to determining that the preset convergence condition is not met based on the current target loss, and then return to the first acquisition module.

[0207] Optionally, the computing module includes:

[0208] The first calculation submodule is used to calculate the current target loss based on the text features and the second features of the multiple dimensions if the network parameters of the feature extraction network have not been trained.

[0209] The second calculation submodule is used to calculate the loss based on the fused features after the network parameters of the feature extraction network have been trained, so as to obtain the current target loss.

[0210] The adjustment module is specifically used for:

[0211] If the network parameters of the feature extraction network have not been fully trained, the network parameters of the feature extraction network shall be adjusted.

[0212] or,

[0213] If the network parameters of the feature extraction network have been trained, the network parameters of the feature fusion network are adjusted.

[0214] Accordingly, the device further includes a determination module, used for:

[0215] If the network parameters of the feature extraction network have not been trained, in response to determining that the preset convergence condition is met based on the current target loss, it is determined that the network parameters of the feature extraction network have been trained, and the process returns to the step of obtaining sample data pairs.

[0216] Optionally, the first computing submodule includes:

[0217] The first calculation unit is used to perform loss calculation based on the difference between the text features and the second features of the humanoid attribute dimension to obtain a first type of loss, and to perform loss calculation based on the difference between the task result of the first type of task determined by the second features of the humanoid attribute dimension and the corresponding annotation result to obtain a second type of loss.

[0218] The second calculation unit is used to calculate the loss based on the difference between the task results of the second type of task determined by the second feature of the human detection dimension and the corresponding labeled results, so as to obtain the third type of loss.

[0219] The third calculation unit is used to perform loss calculation based on the difference between the second feature of the clothing mask dimension and the clothing mask annotation result of the image in the sample data pair, and obtain the fourth type of loss.

[0220] The fourth calculation unit is used to calculate the current target loss based on the first type of loss, the second type of loss, the third type of loss, and the fourth type of loss;

[0221] The first type of task is a classification task about humanoid attributes, and the method of determining the task result of the first type of task through the second feature of the humanoid attribute dimension includes: inputting the second feature of the humanoid attribute dimension into a task processing network for performing the first type of task to obtain the task result of the first type of task;

[0222] The second type of task includes classification tasks or regression tasks, and the method of determining the task result of the second type of task through the second feature of the human figure detection dimension includes: inputting the second feature of the human figure detection dimension into the task processing network used to perform the second type of task to obtain the task result of the second type of task.

[0223] Optionally, the device further includes a fifth computing unit, for:

[0224] A fifth type of loss is calculated based on the difference between the target intermediate features and the features of a predetermined category in the text description; wherein, the target intermediate features are: intermediate features about the predetermined category generated during the process of determining the task result of the specified second type of task through the second features of the human detection dimension;

[0225] The type of the specified second category of tasks is a category task;

[0226] The fourth calculation unit is specifically used for:

[0227] Calculate the current target loss based on the first type of loss, the second type of loss, the third type of loss, the fourth type of loss, and the fifth type of loss.

[0228] Optionally, the feature extraction network includes multiple feature extractors, each of which is used to extract a feature in one dimension;

[0229] The features of the humanoid attribute dimension include: various descriptive features about humanoid attributes;

[0230] The features of the human detection dimension include: at least one feature about the human bounding box and at least one feature about the human key points; at least one feature about the human bounding box includes: features about the human bounding box used for classification and / or features used for regression analysis; at least one feature about the human key points includes: features about the human key points used for classification and / or features used for regression analysis.

[0231] The first computing unit is specifically used for:

[0232] For each descriptive feature about humanoid attributes, a loss is calculated based on the difference between the descriptive feature and the content of the description of humanoid attributes in the text features, to obtain the first type of loss corresponding to the descriptive feature about humanoid attributes.

[0233] For each descriptive feature about humanoid attributes, based on the difference between the task result of the first task determined by the descriptive feature about humanoid attributes and the corresponding labeled result, loss is calculated to obtain the second type of loss corresponding to the descriptive feature about humanoid attributes.

[0234] The second computing unit is specifically used for:

[0235] For each feature in the human detection dimension, loss is calculated based on the difference between the task result of the corresponding second type of task determined by the feature and the corresponding annotation result, to obtain the third type of loss corresponding to the feature.

[0236] Optionally, the second calculation submodule is specifically used for:

[0237] The fusion features of the sample data pairs are classified using a humanoid ID classifier to obtain the humanoid ID to which the humanoid in the image belongs in the sample data pair.

[0238] Based on the obtained humanoid ID and the ground truth humanoid ID of the image in the sample data pair, the loss is calculated to obtain the ID loss;

[0239] Based on the fusion features of the sample data pair, and the fusion features of positive samples and negative samples in other sample data pairs corresponding to the sample data pair, the triplet loss is calculated.

[0240] Based on the obtained ID loss and triplet loss, determine the current target loss.

[0241] Based on the above method embodiments, this application also provides a human figure re-identification device for clothing changes, such as... Figure 8 As shown, the device includes:

[0242] The second acquisition module 810 is used to acquire an image to be identified and a corresponding descriptive text for the image to be identified; wherein, the descriptive text corresponding to the image to be identified is used to describe the human-shaped region in the image to be identified, excluding clothing.

[0243] The second input module 820 is used to input the image to be identified and the corresponding descriptive text into the target feature extraction model to obtain fused features; wherein, the target feature extraction model is trained according to the model training method;

[0244] The processing module 830 is used to perform humanoid re-identification processing based on the fused features.

[0245] This application also provides an electronic device, such as... Figure 9 As shown, it includes:

[0246] Memory 901 is used to store computer programs;

[0247] The processor 902, when executing the program stored in the memory 901, implements any of the aforementioned model training methods, or the humanoid costume re-identification method.

[0248] Furthermore, the aforementioned electronic device may also include a communication bus and / or a communication interface, with the processor 902, communication interface, and memory 901 communicating with each other via the communication bus.

[0249] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0250] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0251] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0252] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0253] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described model training method or the humanoid costume re-identification method.

[0254] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the model training methods described in the above embodiments, or the humanoid costume re-identification method.

[0255] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a solid-state drive (SSD), etc.

[0256] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0257] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0258] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A model training method, characterized in that, The method includes: Obtain sample data pairs; wherein, the sample data pairs include: an image containing a human figure and a corresponding descriptive text for the image, the descriptive text for the image being used to describe the human figure region in the image, excluding clothing; The sample data pairs are input into a target feature extraction model to be trained, so that the target feature extraction model generates fused features of the sample data pairs. The target feature extraction model includes a multimodal network, a feature extraction network, and a feature fusion network. The multimodal network is used to encode the image and descriptive text in the sample data pairs respectively, and fuse the encoded image features and text features to obtain a first feature of the sample data pairs. The feature extraction network is used to extract features from the first feature in multiple dimensions to obtain a second feature of the sample data pairs in multiple dimensions, including at least: human attributes, human detection, and clothing mask. The feature fusion network is used to fuse the second feature in the human attribute dimension and the second feature in the human detection dimension, and fuse the fused feature with the second feature in the clothing mask dimension to extract the fused feature. Based on the features extracted by each network of the target feature extraction model, loss calculation is performed to obtain the current target loss; In response to the determination that the preset convergence condition is not met based on the current target loss, the network parameters of the target feature extraction model are adjusted, and the process returns to the step of obtaining sample data pairs.

2. The method according to claim 1, characterized in that, The step of calculating the current target loss based on the features extracted by each network of the target feature extraction model includes: If the network parameters of the feature extraction network are not fully trained, loss calculation is performed based on the text features and the second features of the multiple dimensions to obtain the current target loss; or, Once the network parameters of the feature extraction network have been trained, loss calculation is performed based on the fused features to obtain the current target loss. The step of adjusting the network parameters of the target feature extraction model in response to determining that the preset convergence condition is not met based on the current target loss includes: If the network parameters of the feature extraction network have not been fully trained, the network parameters of the feature extraction network shall be adjusted. or, If the network parameters of the feature extraction network have been trained, the network parameters of the feature fusion network are adjusted. Accordingly, the method further includes: If the network parameters of the feature extraction network have not been trained, in response to determining that the preset convergence condition is met based on the current target loss, it is determined that the network parameters of the feature extraction network have been trained, and the process returns to the step of obtaining sample data pairs.

3. The method according to claim 2, characterized in that, The loss calculation based on the text features and the second features of the multiple dimensions to obtain the current target loss includes: Based on the difference between the text features and the second feature of the humanoid attribute dimension, a loss calculation is performed to obtain the first type of loss. Based on the difference between the task result of the first type of task determined by the second feature of the humanoid attribute dimension and the corresponding annotation result, a loss calculation is performed to obtain the second type of loss. Based on the difference between the task results of the second type of task determined by the second feature of the human figure detection dimension and the corresponding labeled results, the loss is calculated to obtain the third type of loss; Based on the difference between the second feature of the clothing mask dimension and the clothing mask annotation results of the image in the sample data pair, loss calculation is performed to obtain the fourth type of loss; Calculate the current target loss based on the first type of loss, the second type of loss, the third type of loss, and the fourth type of loss; The first type of task is a classification task about humanoid attributes, and the method of determining the task result of the first type of task through the second feature of the humanoid attribute dimension includes: inputting the second feature of the humanoid attribute dimension into a task processing network for performing the first type of task to obtain the task result of the first type of task; The second type of task includes classification tasks or regression tasks, and the method of determining the task result of the second type of task through the second feature of the human figure detection dimension includes: inputting the second feature of the human figure detection dimension into the task processing network used to perform the second type of task to obtain the task result of the second type of task.

4. The method according to claim 3, characterized in that, The method further includes: A fifth type of loss is calculated based on the difference between the target intermediate features and the features of a predetermined category in the text description; wherein, the target intermediate features are: intermediate features about the predetermined category generated during the process of determining the task result of the specified second type of task through the second features of the human detection dimension; The type of the specified second category of tasks is a category task; The calculation of the current target loss based on the first type of loss, the second type of loss, the third type of loss, and the fourth type of loss includes: Calculate the current target loss based on the first type of loss, the second type of loss, the third type of loss, the fourth type of loss, and the fifth type of loss.

5. The method according to claim 3, characterized in that, The feature extraction network includes multiple feature extractors, each of which is used to extract a feature in one dimension. The features of the humanoid attribute dimension include: various descriptive features about humanoid attributes; The features of the human detection dimension include: at least one feature about the human bounding box and at least one feature about the human key points; at least one feature about the human bounding box includes: features about the human bounding box used for classification and / or features used for regression analysis; at least one feature about the human key points includes: features about the human key points used for classification and / or features used for regression analysis. The loss calculation based on the difference between the text features and the second feature of the humanoid attribute dimension yields a first type of loss, including: For each descriptive feature about humanoid attributes, a loss is calculated based on the difference between the descriptive feature and the content of the description of humanoid attributes in the text features, to obtain the first type of loss corresponding to the descriptive feature about humanoid attributes. The loss calculation, based on the difference between the task result of the first type of task determined by the second feature of the humanoid attribute dimension and the corresponding labeled result, yields the second type of loss, including: For each descriptive feature about humanoid attributes, based on the difference between the task result of the first task determined by the descriptive feature about humanoid attributes and the corresponding labeled result, loss is calculated to obtain the second type of loss corresponding to the descriptive feature about humanoid attributes. The third type of loss is obtained by calculating the difference between the task results of the second type of task determined by the second feature of the human figure detection dimension and the corresponding labeled results, including: For each feature in the human detection dimension, loss is calculated based on the difference between the task result of the corresponding second type of task determined by the feature and the corresponding annotation result, to obtain the third type of loss corresponding to the feature.

6. The method according to claim 2, characterized in that, The loss calculation based on the fused features to obtain the current target loss includes: The fusion features of the sample data pairs are classified using a humanoid ID classifier to obtain the humanoid ID to which the humanoid in the image belongs in the sample data pair. Based on the obtained humanoid ID and the ground truth humanoid ID of the image in the sample data pair, the loss is calculated to obtain the ID loss; Based on the fusion features of the sample data pair, and the fusion features of positive samples and negative samples in other sample data pairs corresponding to the sample data pair, the triplet loss is calculated. Based on the obtained ID loss and triplet loss, determine the current target loss.

7. A method for re-identifying a humanoid figure after costume changes, characterized in that, The method includes: Obtain the image to be identified and the corresponding descriptive text; wherein, the descriptive text corresponding to the image to be identified is used to describe the human-shaped region in the image to be identified, excluding clothing; The image to be identified and the corresponding descriptive text are input into the target feature extraction model to obtain fused features; wherein, the target feature extraction model is trained according to any one of the methods described in claims 1-6; Based on the fusion features, perform a re-identification process for the humanoid figure after costume change.

8. A model training device, characterized in that, The device includes: The first acquisition module is used to acquire sample data pairs; wherein, the sample data pairs include: an image containing a human figure and a corresponding descriptive text for the image, the descriptive text for the image being used to describe the human figure region in the image, excluding clothing; A first input module is used to input the sample data pair into a target feature extraction model to be trained, so that the target feature extraction model generates a fused feature of the sample data pair. The target feature extraction model includes a multimodal network, a feature extraction network, and a feature fusion network. The multimodal network is used to encode the image and descriptive text in the sample data pair respectively, and fuse the encoded image features and text features to obtain a first feature of the sample data pair. The feature extraction network is used to extract features from the first feature in multiple dimensions to obtain a second feature of the sample data pair in multiple dimensions, including at least: human attributes, human detection, and clothing mask. The feature fusion network is used to fuse the second feature of the human attribute dimension and the second feature of the human detection dimension, and fuse the fused feature with the second feature of the clothing mask dimension to extract the fused feature. The calculation module is used to perform loss calculation based on the features extracted by each network of the target feature extraction model to obtain the current target loss; The adjustment module is used to adjust the network parameters of the target feature extraction model in response to the determination that the preset convergence condition is not met based on the current target loss, and then return to the first acquisition module.

9. A humanoid re-identification device for changing clothes, characterized in that, The device includes: The second acquisition module is used to acquire the image to be identified and the descriptive text corresponding to the image to be identified; wherein, the descriptive text corresponding to the image to be identified is used to describe the human-shaped region in the image to be identified, excluding clothing. The second input module is used to input the image to be identified and the corresponding descriptive text into the target feature extraction model to obtain fused features; wherein, the target feature extraction model is trained by the method according to any one of claims 1-6; The processing module is used to perform humanoid re-identification processing based on the fused features.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method of any one of claims 1-6, or the method of claim 7.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1-6, or the method of claim 7.

Images