Unsupervised training method, device and medium for re-identification model

Abstract

The present application provides an unsupervised training method, device and medium for a re-identification model, which is used to solve the problem of insufficient perception of local features of the human body and loss of human structure information in the model during unsupervised training. The method includes: based on a first sample, respectively encoding the first visible view, the mask view and the second visible view obtained after masking different body parts of the person image to obtain a first feature, a mask feature and a second feature, determining a reconstruction loss based on a reconstructed image and a person image determined based on the first feature, determining a predicted alignment loss based on a third feature and a mask feature determined based on the first feature and a mask query vector, determining a multi-view alignment loss based on the first feature and the second feature, and updating network parameters based on the multi-view alignment loss, the predicted alignment loss and the reconstruction loss; using the trained encoder as a feature extraction module in the re-identification model to obtain a re-identification model for pedestrian recognition.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to an unsupervised training method, device and medium for a re-identification model. Background Art

[0002] Traditional supervised person re-identification methods cannot do without a large amount of high-quality labeled data, but data labeling requires a lot of manpower and time. Among the existing unsupervised training methods, one is to label global tokens and local tokens by random cropping, so that the model can focus on the global and local features of pedestrians at the same time, thereby improving the accuracy of the person re-identification model in pedestrian recognition tasks. However, the local view obtained by random cropping may destroy the human body structure information, which is not conducive to capturing the detailed features of the human body, and may cause the loss of human body structure information. Summary of the invention

[0003] The present application provides an unsupervised training method, device and medium for a re-identification model, which are used to solve the problems of insufficient perception of local features of the human body and loss of human body structure information in the unsupervised training model.

[0004] In a first aspect, an embodiment of the present application provides an unsupervised training method for a re-identification model, comprising:

[0005] The following operations are performed for each first sample in the first training sample set: based on the semantic label in the first sample, randomly masking different body parts in the person image in the first sample to obtain a first visible view and a second visible view;

[0006] The first visible view and the second visible view are respectively encoded by an encoder in an unsupervised reconstruction model to obtain a first feature and a second feature, and a person is reconstructed according to the first feature to obtain a reconstructed image;

[0007] Determining a multi-view alignment loss value based on the first feature and the second feature; and determining a reconstruction loss value based on the reconstructed image and the person image;

[0008] Determine a total loss value based on the multi-view alignment loss value and the reconstruction loss value, and update the network parameters of the unsupervised reconstruction model by the total loss value to obtain a trained unsupervised reconstruction model;

[0009] The encoder in the trained unsupervised reconstruction model is used as the feature extraction module in the re-identification model to obtain the re-identification model for pedestrian recognition.

[0010] In a possible implementation, the method further includes:

[0011] Determining a mask view corresponding to the first visible view;

[0012] Encoding the mask view to obtain a mask feature through an encoder in the unsupervised reconstruction model;

[0013] Performing feature extraction on the first feature and a preset mask feature vector to obtain a third feature, and determining a predicted alignment loss value based on the third feature and the mask feature;

[0014] Determining a total loss value based on the multi-view alignment loss value and the reconstruction loss value includes:

[0015] The multi-view alignment loss value, the reconstruction loss value and the prediction alignment loss value are weighted to obtain the total loss value.

[0016] In a possible implementation, reconstructing the person according to the first feature to obtain a reconstructed image includes:

[0017] Performing feature extraction on the first feature and a preset mask feature vector to obtain a third feature;

[0018] The third feature is decoded by a decoder in the unsupervised reconstruction model to obtain a reconstructed image.

[0019] In a possible implementation manner, the extracting the first feature and the preset mask feature vector to obtain the third feature includes:

[0020] inputting the first feature and the mask feature vector into a mask predictor in the unsupervised reconstruction model;

[0021] Feature extraction is performed through the multi-layer feature extraction layers in the mask predictor to obtain the third feature.

[0022] In a possible implementation, the step of randomly masking different body parts in the person image in the first sample based on the semantic label in the first sample includes:

[0023] Dividing the character image into a plurality of sub-regions;

[0024] Masking the person image based on the M sub-regions where the semantic label of the first body part in the first sample is located to obtain a first visible view;

[0025] The person image is masked based on the K sub-regions where the semantic label of the second body part in the first sample is located to obtain a second visible view.

[0026] In a possible implementation, before randomly masking different body parts in the person image in the first sample based on the semantic label in the first sample, the method further includes:

[0027] Performing data enhancement on the person images in the first sample to obtain a first view and a second view from different perspectives;

[0028] The step of randomly masking different body parts in the person image in the first sample based on the semantic label in the first sample comprises:

[0029] Masking the first view based on the area where the semantic label of the first body part in the first sample is located to obtain a first visible view;

[0030] The second view is masked based on the area where the semantic label of the second body part in the first sample is located to obtain a second visible view.

[0031] In a possible implementation, after using the encoder in the trained unsupervised reconstruction model as a feature extraction module in the re-identification model, the method further includes:

[0032] The network parameters of the re-identification model are updated based on the second training sample set to obtain a trained re-identification model, wherein the second sample set includes multiple second samples, the second samples include pedestrian images and category labels, and the category labels are used to characterize different pedestrians.

[0033] In a possible implementation manner, updating the network parameters of the re-identification model based on the second training sample set includes:

[0034] Recognize the pedestrian image based on the re-recognition model to obtain pedestrian features and predicted recognition labels of the pedestrian image;

[0035] Determine the category loss based on the pedestrian features of the pedestrian image, the first target features and the second target features, wherein the first target features are obtained after the re-identification model performs feature extraction on the first target image, the second target features are obtained after the re-identification model performs feature extraction on the second target image, the first target image is a pedestrian image in the second training sample set, whose category label is the predicted identification label, and the second target image is a pedestrian image in the second training sample set, whose category label is different from the predicted identification label;

[0036] Determining a re-identification loss based on the predicted identification label and the category label of the pedestrian image;

[0037] A network parameter of the re-identification model is updated based on the category loss and the re-identification loss.

[0038] In a possible implementation manner, the first training sample set is obtained in the following manner:

[0039] Acquire an image data set, wherein the image data set includes a plurality of person images;

[0040] For each of the person images, body part segmentation is performed using a human body semantic segmentation model to obtain semantic labels of the body parts of the person image;

[0041] For each of the character images, the character image and the corresponding semantic label are used as a first sample to obtain a first training sample set.

[0042] In a second aspect, an embodiment of the present application provides an unsupervised training device for a re-identification model, comprising:

[0043] A processing module, configured to perform the following operations for each first sample in the first training sample set: based on the semantic label in the first sample, randomly mask different body parts in the person image in the first sample to obtain a first visible view and a second visible view;

[0044] The first visible view and the second visible view are respectively encoded by an encoder in an unsupervised reconstruction model to obtain a first feature and a second feature, and a person is reconstructed according to the first feature to obtain a reconstructed image;

[0045] Determining a multi-view alignment loss value based on the first feature and the second feature; and determining a reconstruction loss value based on the reconstructed image and the person image;

[0046] An adjustment module, configured to determine a total loss value based on the multi-view alignment loss value and the reconstruction loss value, and update a network parameter of the unsupervised reconstruction model by using the total loss value to obtain a trained unsupervised reconstruction model;

[0047] The processing module is also used to use the encoder in the trained unsupervised reconstruction model as a feature extraction module in the re-identification model to obtain the re-identification model for pedestrian recognition.

[0048] In a third aspect, an embodiment of the present application provides an execution device, including:

[0049] A memory for storing program instructions;

[0050] The processor is used to obtain the program instructions in the memory and execute the method described in the first aspect and different implementation methods of the first aspect according to the obtained program instructions.

[0051] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium includes computer program instructions. When the computer program instructions are executed by a computer, the method described in the first aspect and different implementation methods of the first aspect are implemented.

[0052] The beneficial effects of this application are as follows:

[0053] In the present application, when performing unsupervised training, the random mask human body part strategy is used for masking, which can well protect the human body related information from being destroyed. In addition, by constructing a multi-view loss error to compare the structure of different views, the encoder can learn the global features and human body feature invariance under different perspectives. In addition, the present application randomly selects human body parts for masking, and the visible views that are not blocked are responsible for providing rich human body structure information, helping the decoder to reconstruct the mask area, so that the encoder can learn as much as possible about the connection between human body structures in the pre-training stage. The present application uses masked image reconstruction to allow the encoder to pay more attention to the local features of the human body, thereby improving the accuracy of re-identification. In addition, the present application uses a human semantic segmentation model to perform high-quality segmentation of the human body structure, which provides a basis for the unsupervised reconstruction model to mask body parts, so as to facilitate more accurate masking of unsupervised tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0055] Figure 1A A schematic diagram of an application scenario provided for an embodiment of the present application;

[0056] Figure 1B A schematic diagram of a server structure provided in an embodiment of the present application;

[0057] Figure 2 A flowchart of an unsupervised training method for a re-identification model provided in an embodiment of the present application;

[0058] Figure 3 A schematic diagram of an unsupervised training architecture provided in an embodiment of the present application;

[0059] Figure 4 A schematic diagram of unsupervised pre-training provided in an embodiment of the present application;

[0060] Figure 5A schematic diagram of an unsupervised training device for a re-identification model provided in an embodiment of the present application;

[0061] Figure 6 A schematic diagram of an execution device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0063] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0064] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0065] Person re-identification technology is an important research direction in the field of computer vision, and it has a wide range of applications in security, transportation, and surveillance. However, traditional supervised person re-identification methods cannot do without a large amount of high-quality labeled data, which is difficult to obtain in practical applications. Self-supervised learning methods can automatically learn feature representations from a large amount of unlabeled data, providing a new idea for solving this problem. Its combination with visual Transformer has also received more and more attention. Therefore, studying the pre-training method of person re-identification based on self-supervised learning can not only reduce the model's dependence on labeled data and reduce data costs, but also improve the generalization ability and robustness of the model, which has important application value. Among the existing unsupervised training methods, one is to label global tokens and local tokens by random cropping, so that the model can focus on the global and local features of pedestrians at the same time, thereby improving the accuracy of the person re-identification model in pedestrian recognition tasks. However, the local view obtained by random cropping may destroy the human body structure information, which is not conducive to capturing the detailed features of the human body, and may cause the loss of human body structure information.

[0066] In response to the above problems, an embodiment of the present application provides an unsupervised training method for a re-identification model, which segments the body parts of a person image through a human semantic segmentation model, and determines the semantic labels of the body parts, so as to provide data support when masking the human body parts during the training phase. Furthermore, when performing unsupervised training, the use of a random mask human body part strategy for masking can well protect human body related information from being destroyed. In addition, by constructing a multi-view loss error structure that compares different views, the encoder can learn global features and human body feature invariance under different perspectives. In addition, the present application randomly selects human body parts for masking, and the visible views that are not obscured are responsible for providing rich human body structure information, helping the decoder to reconstruct the mask area, so that the encoder can learn as much as possible about the connection between human body structures during the pre-training phase. The use of masked image reconstruction in the present application allows the encoder to pay more attention to local features of the human body, thereby improving the accuracy of re-identification.

[0067] The unsupervised training method of the re-identification model provided in the embodiment of the present application can be implemented by an execution device. In some embodiments, the execution device can be an electronic device, and the electronic device can be implemented by one or more servers. Figure 1A Take a server 100 as an example. Figure 1AAs shown, it is a schematic diagram of a possible application scenario provided by an embodiment of the present application, including a server 100 and an acquisition device 200. The server 100 can be implemented by a physical server or a virtual server. The server can be implemented by a single server, or by a server cluster composed of multiple servers, and the unsupervised training method of the re-identification model provided in the present application can be implemented by a single server or a server cluster. The acquisition device 200 is a device with an image acquisition function, including an electric police device, an electronic monitoring device, a surveillance camera, a video recorder, a terminal device with an image acquisition function (such as a notebook, a computer, a mobile phone, a TV), etc. The acquisition device 200 can send the acquired image to the server 100 via the network. Optionally, the server 100 can be connected to the terminal device 300, receive the unsupervised training task of the re-identification model sent by the terminal device 300, and perform unsupervised training according to the received image sent by the acquisition device 200. In some scenarios, the server 100 can send the trained re-identification model to the terminal device 300. The terminal device 300 can be a TV, a mobile phone, a tablet computer, a personal computer, etc.

[0068] As an example, see Figure 1B As shown, the server 100 may include a processor 110, a communication interface 120, and a memory 130. Of course, the server 100 may also include other components. Figure 1B Not shown.

[0069] The communication interface 120 is used to communicate with the acquisition device 200 and the terminal device 300, to receive images sent by the acquisition device 200, or to receive unsupervised training tasks of re-identification models sent by the terminal device 300, or to send trained re-identification models to the terminal device 300.

[0070] In the embodiment of the present application, the processor 110 may be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiment of the present application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0071] The processor 110 is the control center of the server 100. It connects various parts of the entire server 100 using various interfaces and routes, and executes various functions of the server 100 and processes data by running or executing software programs / or modules stored in the memory 130, and calling data stored in the memory 130. Optionally, the processor 110 may include one or more processing units. The processor 110 may be, for example, a control component such as a processor, a microprocessor, a controller, and may be, for example, a general-purpose central processing unit (CPU), a general-purpose processor, a digital signal processing (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof.

[0072] The memory 130 can be used to store software programs and modules. The processor 110 executes various functional applications and data processing by running the software programs and modules stored in the memory 130. The memory 130 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to business processing, etc. The memory 130, as a non-volatile computer-readable storage medium, may be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 130 may include at least one type of storage medium, for example, a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read only memory (Prom), a read only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic memory, a disk, an optical disk, and the like. The memory 130 is any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 130 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0073] In some other embodiments, the execution device may be a terminal device. In some scenarios, the terminal device may receive images sent by the acquisition device and perform unsupervised training of the re-identification model based on the images. The terminal device may include a display device, which may be a liquid crystal display, an organic light-emitting semiconductor (OLED) display, a projection display device, etc., and this application does not specifically limit this.

[0074] It should be noted that the above Figure 1A and Figure 1B The structure shown is only an example and is not limited to this embodiment of the present application.

[0075] The present application embodiment provides an unsupervised training method for a re-identification model. Figure 2 The process of the unsupervised training method of the re-identification model is exemplarily shown, and the process can be performed by the unsupervised training device of the re-identification model, and the device can be located at Figure 1B The server 100 shown may be, for example, a processor 110 or a server 100. The unsupervised training device of the re-identification model may also be located in the terminal device. The specific process is as follows:

[0076] Step S201 , performing the following operations for each first sample in the first training sample set: based on the semantic label in the first sample, randomly masking different body parts in the person image in the first sample to obtain a first visible view and a second visible view.

[0077] In some embodiments, the first training sample set may be obtained in the following manner:

[0078] An image data set is obtained, where the image data set includes multiple person images; for each person image, body part segmentation is performed using a human body semantic segmentation model to obtain semantic labels of the body parts of the person image;

[0079] For each person image, the person image and the corresponding semantic label are used as the first sample to obtain a first training sample set.

[0080] As an example, in order to achieve efficient human body part segmentation, a SAM model with language capabilities can be used, and the corresponding semantic annotations can be obtained by inputting language instructions containing specified human body parts. Specifically, the character image X and the language instructions related to the human body parts can be input into the SAM model to obtain the semantic segmentation labels of the human body parts. Among them, the human body parts can include six parts: "head, upper body, left arm, right arm, left leg and right leg". At present, the human body parts can also be divided into more or fewer parts, and this application does not make specific limitations on this.

[0081] For example, a language instruction for a human body part is randomly selected from multiple language instructions for human body parts each time. Among them, i is the index of the human body part, and the character image X and the language instruction Input into the SAM model to obtain the language instructions The corresponding semantic tags can be expressed as:

[0082]

[0083] in, represents a SAM model with language functions, Represents the language instructions in the character image X output by the SAM model The corresponding semantic label.

[0084] Furthermore, for each person image X, the person image X and the corresponding semantic label are used as the first sample to obtain a first training sample set.

[0085] In some embodiments, based on the semantic labels in the first sample, randomly masking different body parts in the person image in the first sample can be achieved by:

[0086] Divide the person image into multiple sub-regions;

[0087] Based on the M sub-regions where the semantic label of the first body part in the first sample is located, the person image is masked to obtain a first visible view; based on the K sub-regions where the semantic label of the second body part in the first sample is located, the person image is masked to obtain a second visible view.

[0088] Specifically, the person image can be divided into multiple sub-regions according to the downsampling ratio of the encoder in the unsupervised reconstruction model, and then multiple sub-regions where body parts are located are selected for masking to obtain a visible view.

[0089] As an example, assuming that the first body part can be the left arm and the second body part can be the right leg, the character image is divided into N sub-regions according to the downsampling rate of the encoder, the character image is masked based on the M sub-regions where the left arm is located to obtain a first visible view, and the character image is masked based on the K sub-regions where the right leg is located to obtain a second visible view.

[0090] In some scenarios, after dividing the person image into N sub-regions, the multiple sub-regions where the body parts are located can be masked according to the mask ratio. For example, if the mask ratio is R, then J sub-regions where the body parts are located can be masked, where J≤N×R.

[0091] In some embodiments, before randomly masking different body parts in the person image in the first sample based on the semantic label in the first sample, the person image in the first sample may be data enhanced to obtain a first view and a second view from different perspectives. Further, when randomly masking different body parts in the person image in the first sample based on the semantic label in the first sample, the first view may be masked based on the area where the semantic label of the first body part in the first sample is located to obtain a first visible view, and the second view may be masked based on the area where the semantic label of the second body part in the first sample is located to obtain a second visible view.

[0092] As an example, we can perform two random data augmentations on the person image based on the semantic label to obtain view 1 and view Figure 2 , then mask the multiple sub-regions where the semantic labels of the first body part are located in view 1 and separate the masked regions to obtain visible view 1. Figure 2 Mask the multiple sub-regions where the semantic labels of the second body part are located and separate the masked regions to obtain visible visual Figure 2 .

[0093] Step S202 , respectively encode the first visible view and the second visible view through an encoder in the unsupervised reconstruction model to obtain a first feature and a second feature, and reconstruct the person according to the first feature to obtain a reconstructed image.

[0094] In some embodiments, when reconstructing a person based on a first feature to obtain a reconstructed image, it can be achieved in the following way: extracting features from the first feature and a preset mask feature vector to obtain a third feature; and then decoding the third feature through a decoder in an unsupervised reconstruction model to obtain a reconstructed image.

[0095] Specifically, the first feature and the mask feature vector may be input into a mask predictor, and feature extraction may be performed through multiple feature extraction layers to obtain the third feature. The feature extraction layer may be implemented by a Transformer.

[0096] Step S203: determining a multi-view alignment loss value based on the first feature and the second feature; and determining a reconstruction loss value based on the reconstructed image and the person image.

[0097] In some embodiments, the multi-view alignment loss value may be determined by a multi-view alignment loss function, and the multi-view alignment loss satisfies the condition shown in the following formula:

[0098]

[0099] in, represents the multi-view alignment loss function, Represent the first feature and the second feature respectively, B represents the number of samples, and t represents the weight hyperparameter.

[0100] In some embodiments, the reconstruction loss value may be determined by a reconstruction loss function, and the reconstruction loss satisfies the conditions shown in the following formula:

[0101]

[0102] in, Represent the reconstructed image and the character image respectively, represents the reconstruction loss function.

[0103] Step S204, determining a total loss value based on the multi-view alignment loss value and the reconstruction loss value, and updating the network parameters of the unsupervised reconstruction model according to the total loss value to obtain a trained unsupervised reconstruction model.

[0104] Step S205, using the encoder in the trained unsupervised reconstruction model as a feature extraction module in the re-identification model to obtain a re-identification model for pedestrian recognition.

[0105] In some embodiments, the encoder in the trained unsupervised reconstruction model can be used as a feature extractor in the re-identification model, the features output by the encoder are used to represent the pedestrian features, and a batch normalization layer and a classification head are added after the encoder to obtain a re-identification model for pedestrian recognition.

[0106] In some embodiments, after the encoder in the trained unsupervised reconstruction model is used as a feature extraction module in the re-identification model, the network parameters of the re-identification model can be updated based on the second training sample set to obtain a trained re-identification model. The second sample set includes multiple second samples, the second samples include pedestrian images and category labels, and the category labels are used to characterize different pedestrians.

[0107] Specifically, pedestrian images can be identified based on the re-identification model to obtain pedestrian features and predicted identification labels of the pedestrian images. Based on the pedestrian features of the pedestrian image, the first target feature and the second target feature, the category loss is determined, and the re-identification loss is determined based on the predicted identification label and the category label of the pedestrian image, and then the network parameters of the re-identification model are updated through the category loss and the re-identification loss. Among them, the first target feature is obtained after the re-identification model extracts features from the first target image, the second target feature is obtained after the re-identification model extracts features from the second target image, the first target image is a pedestrian image in the second training sample set, and its category label is the predicted identification label, and the second target image is a pedestrian image in the second training sample set, and its category label is different from the predicted identification label;

[0108] In some embodiments, the re-identification loss satisfies the following formula:

[0109]

[0110] in, represents the re-identification loss, Used to indicate the predicted identification label, Used to represent category labels.

[0111] In some embodiments, the category loss satisfies the following formula:

[0112]

[0113] in, represents the feature distance calculation function, Indicates pedestrian characteristics. Representation and Features of pedestrians belonging to the same category, Representation and Characteristics of pedestrians belonging to different categories, Represented as boundary values ​​to prevent outliers from interfering with training.

[0114] In some embodiments, the mask view corresponding to the first visible view can also be determined, and the mask view can be encoded by an encoder in the unsupervised reconstruction model to obtain a mask feature. Furthermore, the first feature and the preset mask feature vector can be feature extracted based on the mask predictor to obtain a third feature, and the predicted alignment loss value can be determined based on the third feature and the mask feature. At this time, when determining the total loss value based on the multi-view alignment loss value and the reconstruction loss value, the multi-view alignment loss value, the reconstruction loss value and the predicted alignment loss value can be weighted to obtain the total loss value.

[0115] In some embodiments, the predicted alignment loss value may be determined by a predicted alignment loss function, and the predicted alignment loss function satisfies the conditions shown in the following formula:

[0116]

[0117] in, represents the prediction alignment loss function, represents the third feature of the mask predictor output, Features representing the mask view output by the encoder.

[0118] In some embodiments, the total loss satisfies the following formula:

[0119]

[0120] in, represents the weight value of multi-view alignment loss, represents the weight value of the prediction alignment loss, Represents the weight value of the reconstruction loss.

[0121] The unsupervised training method of the re-identification model proposed in this application consists of three stages. In the first stage, SAM is used to obtain the fine-grained segmentation results of the human body parts of the training data to construct an unsupervised training data set. The second stage uses the training data with semantic labels for unsupervised training. In order to make full use of the prior information of the human body structure, this application randomly selects human body parts for masking. The unobstructed image blocks are responsible for providing rich human body structure information to help the decoder reconstruct the masked area, so that the encoder can learn as much as possible about the connection between human body structures during the pre-training stage. The third stage is fine-tuning training and testing after unsupervised pre-training. This application uses the original visual Transformer as a feature extractor to perform pedestrian re-identification tasks based on the weight parameters learned in the unsupervised pre-training.

[0122] This application uses the large image visual segmentation model SAM and unsupervised learning technology to improve the visual encoder's perception of human body structure and local information, thereby enhancing its accuracy and generalization in pedestrian re-identification tasks. Among them, the SAM model is responsible for high-quality segmentation of human body structure to facilitate more accurate mask image modeling for unsupervised tasks. In the unsupervised pre-training part, this application uses a random mask human body part strategy to replace the completely random masking method to avoid the destruction and loss of human body related information. Using mask image modeling instead of contrastive learning as a self-supervisory framework allows the encoder to pay more attention to local human features. At the same time, in order to enable the encoder to learn the global features and human feature invariance under different perspectives, the contrastive learning multi-view structure is integrated on the basis of the mask image modeling framework. In the fine-tuning stage, the encoder learned by unsupervised pre-training is used as a feature extractor, and training and testing are carried out in accordance with the conventional pedestrian re-identification paradigm. The CSL Token of the visual Transformer (ViT) is used as a human feature to avoid introducing additional calculations.

[0123] like Figure 3 , which is a schematic diagram of the unsupervised training method of the re-identification model, including training data preprocessing, unsupervised pre-training, and re-identification model fine-tuning.

[0124] Among them, the goal of the training data preprocessing stage is to label the unlabeled pedestrian images with labels related to human body parts. This is to provide a basis for masking human body parts in the unsupervised pre-training stage and has nothing to do with the training process of the encoder.

[0125] In the unsupervised pre-training stage, the unsupervised pre-training framework mainly consists of human structure mask strategy, encoder, mask predictor, decoder, multi-view alignment loss and reconstruction loss. The encoder uses visual Transformer, the mask predictor consists of multiple Transformers, and the decoder is ViT similar to the encoder.

[0126] First, two random data augmentations are performed on the training images and their labels to obtain view 1 and view 2. Figure 2 , and then use the human body structure mask strategy to mask view 1, and obtain the visible view 1 and mask view 1 corresponding to view 1, as well as the Figure 2 Mask to get the visual Figure 2 The corresponding visible Figure 2 .

[0127] Then the visible view 1 and mask view 1 corresponding to view 1, as well as the view Figure 2 The corresponding visible Figure 2 Input the weight-shared encoder to obtain the corresponding features, where visible view 1 and visible view Figure 2The corresponding features are used to calculate the multi-view alignment loss The multi-view alignment loss guides the encoder to learn the invariance of human structure and improves its discriminative ability.

[0128] Then, the features of visible view 1 and the mask feature vector (mask query vector) are input into the mask predictor together. The mask predictor is responsible for feature extraction of the mask view and is predicted by the alignment loss The mask predictor can better separate the encoder and decoder from the task of feature extraction and mask reconstruction, allowing them to focus on feature extraction or mask reconstruction.

[0129] Finally, the features output by the mask predictor are input into the decoder to reconstruct the image. The reconstruction is supervised by the reconstruction loss. In order to better reconstruct the masked human body parts, the reconstruction loss Use mean squared error loss.

[0130] Then, the network parameters are updated through the total loss determined by multi-view alignment loss, prediction alignment loss and reconstruction loss.

[0131] In the fine-tuning stage of the re-identification model, the encoder trained in the unsupervised pre-training stage is used as the feature extractor for the re-identification task, the CLS Token output by the encoder is used to represent the pedestrian features, and the batch normalization BN layer and classification head are added after the encoder according to the traditional ReID model.

[0132] In the fine-tuning training phase, category loss (triplet loss) is used and re-identification loss (classification loss) , the re-ID model is fine-tuned by two losses.

[0133] In some embodiments, it is also possible to obtain Figure 2 Determined visible Figure 2 Mask view Figure 2 ,At this time, the two sets of visible views and mask views are as follows Figure 4 Further, based on the same process, the visual Figure 2 The corresponding prediction alignment loss and reconstruction loss, and then when determining the total loss, the multi-view alignment loss, the prediction alignment loss and reconstruction loss corresponding to view 1, and the view Figure 2 The corresponding prediction alignment loss and reconstruction loss are weighted to obtain the total loss, and then the network parameters are adjusted through the total loss.

[0134] Based on the same technical concept, see Figure 5As shown, an unsupervised training device 500 for a re-identification model is provided in an embodiment of the present application. The device 500 can perform any step in the unsupervised training method for a re-identification model, and will not be described again to avoid repetition. The device 500 includes a processing module 501 and an adjustment module 502.

[0135] The processing module 501 is configured to perform the following operations for each first sample in the first training sample set: based on the semantic label in the first sample, randomly mask different body parts in the person image in the first sample to obtain a first visible view and a second visible view;

[0136] The first visible view and the second visible view are respectively encoded by an encoder in an unsupervised reconstruction model to obtain a first feature and a second feature, and a person is reconstructed according to the first feature to obtain a reconstructed image;

[0137] Determining a multi-view alignment loss value based on the first feature and the second feature; and determining a reconstruction loss value based on the reconstructed image and the person image;

[0138] An adjustment module 502 is used to determine a total loss value based on the multi-view alignment loss value and the reconstruction loss value, and update the network parameters of the unsupervised reconstruction model according to the total loss value to obtain a trained unsupervised reconstruction model;

[0139] The processing module 501 is further used to use the encoder in the trained unsupervised reconstruction model as a feature extraction module in the re-identification model to obtain a re-identification model for pedestrian recognition.

[0140] In some embodiments, the processing module 501 is further used to: determine a mask view corresponding to the first visible view; encode the mask view to obtain a mask feature through an encoder in the unsupervised reconstruction model; extract features from the first feature and a preset mask feature vector to obtain a third feature, and determine a predicted alignment loss value based on the third feature and the mask feature;

[0141] The processing module 501, when determining the total loss value based on the multi-view alignment loss value and the reconstruction loss value, is specifically configured to:

[0142] The multi-view alignment loss value, the reconstruction loss value and the prediction alignment loss value are weighted to obtain the total loss value.

[0143] In some embodiments, when reconstructing a person according to the first feature to obtain a reconstructed image, the processing module 501 is specifically configured to:

[0144] Performing feature extraction on the first feature and a preset mask feature vector to obtain a third feature;

[0145] The third feature is decoded by a decoder in the unsupervised reconstruction model to obtain a reconstructed image.

[0146] In some embodiments, when the processing module 501 extracts the first feature and the preset mask feature vector to obtain the third feature, it is specifically used to:

[0147] inputting the first feature and the mask feature vector into a mask predictor in the unsupervised reconstruction model;

[0148] Feature extraction is performed through the multi-layer feature extraction layers in the mask predictor to obtain the third feature.

[0149] In some embodiments, when randomly masking different body parts in the person image in the first sample based on the semantic label in the first sample, the processing module 501 is specifically configured to:

[0150] Dividing the character image into a plurality of sub-regions;

[0151] Masking the person image based on the M sub-regions where the semantic label of the first body part in the first sample is located to obtain a first visible view;

[0152] The person image is masked based on the K sub-regions where the semantic label of the second body part in the first sample is located to obtain a second visible view.

[0153] In some embodiments, the processing module 501, before randomly masking different body parts in the person image in the first sample based on the semantic label in the first sample, is further configured to:

[0154] Performing data enhancement on the person images in the first sample to obtain a first view and a second view from different perspectives;

[0155] The step of randomly masking different body parts in the person image in the first sample based on the semantic label in the first sample comprises:

[0156] Masking the first view based on the area where the semantic label of the first body part in the first sample is located to obtain a first visible view;

[0157] The second view is masked based on the area where the semantic label of the second body part in the first sample is located to obtain a second visible view.

[0158] In some embodiments, after using the encoder in the trained unsupervised reconstruction model as a feature extraction module in the re-identification model, the adjustment module 502 is further used to:

[0159] The network parameters of the re-identification model are updated based on the second training sample set to obtain a trained re-identification model, wherein the second sample set includes multiple second samples, the second samples include pedestrian images and category labels, and the category labels are used to characterize different pedestrians.

[0160] In some embodiments, when the adjustment module 502 updates the network parameters of the re-identification model based on the second training sample set, it is specifically used to:

[0161] Recognize the pedestrian image based on the re-recognition model to obtain pedestrian features and predicted recognition labels of the pedestrian image;

[0162] Determine the category loss based on the pedestrian features of the pedestrian image, the first target features and the second target features, wherein the first target features are obtained after the re-identification model performs feature extraction on the first target image, the second target features are obtained after the re-identification model performs feature extraction on the second target image, the first target image is a pedestrian image in the second training sample set, whose category label is the predicted identification label, and the second target image is a pedestrian image in the second training sample set, whose category label is different from the predicted identification label;

[0163] Determining a re-identification loss based on the predicted identification label and the category label of the pedestrian image;

[0164] A network parameter of the re-identification model is updated based on the category loss and the re-identification loss.

[0165] In some embodiments, the first training sample set is obtained by:

[0166] Acquire an image data set, wherein the image data set includes a plurality of person images;

[0167] For each of the person images, body part segmentation is performed using a human body semantic segmentation model to obtain semantic labels of the body parts of the person image;

[0168] For each of the character images, the character image and the corresponding semantic label are used as a first sample to obtain a first training sample set.

[0169] Based on the same technical concept, the embodiment of the present application provides an execution device 600, which can implement any step of the unsupervised training method of the re-identification model discussed above, please refer to Figure 6The device includes a memory 601 and a processor 602 .

[0170] The memory 601 is used to store program instructions;

[0171] The processor 602 is used to call the program instructions stored in the memory and execute the unsupervised training method of the re-identification model according to the obtained program.

[0172] In the embodiment of the present application, the processor 602 may be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiment of the present application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0173] The memory 601 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 601 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, and the like. The memory 601 is any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 601 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0174] Based on the same technical concept, an embodiment of the present application provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on a computer, the computer executes each step in the unsupervised training method of the above-mentioned re-identification model.

[0175] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0176] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0177] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0178] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0179] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. An unsupervised training method for a re-identification model, characterized in that: include: The following operations are performed for each first sample in the first training sample set: based on the semantic label in the first sample, different body parts in the person image in the first sample are randomly masked to obtain a first visible view and a second visible view; wherein the first visible view is an image obtained by masking the area where the first body part in the person image is located, and the second visible view is an image obtained by masking the area where the second body part in the person image is located; The first visible view and the second visible view are respectively encoded by an encoder in an unsupervised reconstruction model to obtain a first feature and a second feature, and a person is reconstructed according to the first feature to obtain a reconstructed image; Determining a multi-view alignment loss value based on the first feature and the second feature; and determining a reconstruction loss value based on the reconstructed image and the person image; Determine a total loss value based on the multi-view alignment loss value and the reconstruction loss value, and update the network parameters of the unsupervised reconstruction model by the total loss value to obtain a trained unsupervised reconstruction model; The encoder in the trained unsupervised reconstruction model is used as the feature extraction module in the re-identification model to obtain the re-identification model for pedestrian recognition.

2. The method according to claim 1, characterized in that The method further comprises: Determining a mask view corresponding to the first visible view; Encoding the mask view to obtain a mask feature through an encoder in the unsupervised reconstruction model; Performing feature extraction on the first feature and a preset mask feature vector to obtain a third feature, and determining a predicted alignment loss value based on the third feature and the mask feature; Determining a total loss value based on the multi-view alignment loss value and the reconstruction loss value includes: The multi-view alignment loss value, the reconstruction loss value and the prediction alignment loss value are weighted to obtain the total loss value.

3. The method according to claim 1, characterized in that The step of reconstructing the person according to the first feature to obtain a reconstructed image includes: Performing feature extraction on the first feature and a preset mask feature vector to obtain a third feature; The third feature is decoded by a decoder in the unsupervised reconstruction model to obtain a reconstructed image.

4. The method according to claim 2 or 3, characterized in that The extracting the first feature and the preset mask feature vector to obtain the third feature includes: inputting the first feature and the mask feature vector into a mask predictor in the unsupervised reconstruction model; Feature extraction is performed through the multi-layer feature extraction layers in the mask predictor to obtain the third feature.

5. The method according to any one of claims 1 to 3, characterized in that: The step of randomly masking different body parts in the person image in the first sample based on the semantic label in the first sample includes: Dividing the character image into a plurality of sub-regions; Masking the person image based on the M sub-regions where the semantic label of the first body part in the first sample is located to obtain a first visible view; The person image is masked based on the K sub-regions where the semantic label of the second body part in the first sample is located to obtain a second visible view.

6. The method according to any one of claims 1 to 3, characterized in that: Before randomly masking different body parts in the person image in the first sample based on the semantic label in the first sample, the method further includes: Performing data enhancement on the person images in the first sample to obtain a first view and a second view from different perspectives; The step of randomly masking different body parts in the person image in the first sample based on the semantic label in the first sample includes: Masking the first view based on the area where the semantic label of the first body part in the first sample is located to obtain a first visible view; The second view is masked based on the area where the semantic label of the second body part in the first sample is located to obtain a second visible view.

7. The method according to any one of claims 1 to 3, characterized in that: After using the encoder in the trained unsupervised reconstruction model as a feature extraction module in the re-identification model, the method further includes: The network parameters of the re-identification model are updated based on the second training sample set to obtain a trained re-identification model, wherein the second sample set includes multiple second samples, the second samples include pedestrian images and category labels, and the category labels are used to characterize different pedestrians.

8. The method according to claim 7, characterized in that The updating of network parameters of the re-identification model based on the second training sample set includes: Recognize the pedestrian image based on the re-recognition model to obtain pedestrian features and predicted recognition labels of the pedestrian image; Determine the category loss based on the pedestrian features of the pedestrian image, the first target features and the second target features, wherein the first target features are obtained after the re-identification model performs feature extraction on the first target image, the second target features are obtained after the re-identification model performs feature extraction on the second target image, the first target image is a pedestrian image in the second training sample set, whose category label is the predicted identification label, and the second target image is a pedestrian image in the second training sample set, whose category label is different from the predicted identification label; Determining a re-identification loss based on the predicted identification label and the category label of the pedestrian image; A network parameter of the re-identification model is updated based on the category loss and the re-identification loss.

9. The method according to any one of claims 1 to 3, characterized in that: The first training sample set is obtained by: Acquire an image data set, wherein the image data set includes a plurality of person images; For each of the person images, body part segmentation is performed using a human body semantic segmentation model to obtain semantic labels of the body parts of the person image; For each of the character images, the character image and the corresponding semantic label are used as a first sample to obtain a first training sample set.

10. An unsupervised training device for a re-identification model, characterized in that: include: a processing module, configured to perform the following operations for each first sample in the first training sample set: based on the semantic label in the first sample, randomly mask different body parts in the person image in the first sample to obtain a first visible view and a second visible view; wherein the first visible view is an image obtained by masking the area where the first body part in the person image is located, and the second visible view is an image obtained by masking the area where the second body part in the person image is located; The first visible view and the second visible view are respectively encoded by an encoder in an unsupervised reconstruction model to obtain a first feature and a second feature, and a person is reconstructed according to the first feature to obtain a reconstructed image; Determining a multi-view alignment loss value based on the first feature and the second feature; and determining a reconstruction loss value based on the reconstructed image and the person image; An adjustment module, configured to determine a total loss value based on the multi-view alignment loss value and the reconstruction loss value, and update a network parameter of the unsupervised reconstruction model by using the total loss value to obtain a trained unsupervised reconstruction model; The processing module is also used to use the encoder in the trained unsupervised reconstruction model as a feature extraction module in the re-identification model to obtain the re-identification model for pedestrian recognition.

11. An execution device, characterized in that: include: A memory for storing program instructions; A processor, configured to obtain program instructions in the memory and execute the method according to any one of claims 1 to 9 according to the obtained program instructions.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium comprises computer program instructions, and when the computer program instructions are executed by a computer, the method according to any one of claims 1 to 9 is implemented.

Images