Pedestrian re-identification method, device, electronic device and readable storage medium

Abstract

The embodiments of the present application provide a pedestrian re-identification method, device, electronic device and readable storage medium, which relate to the field of computer vision technology. By obtaining an image to be identified containing a portrait, the image to be identified is input into a re-identification model for re-identification to obtain a re-identification result. Among them, the re-identification model uses the portrait image as a training sample to train a pre-constructed re-identification network including an appearance sub-network and an appearance sub-network. The trained appearance sub-network improves the recognition accuracy and recognition efficiency of pedestrian re-identification.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a pedestrian re-identification method, device, electronic device and readable storage medium. Background Art

[0002] Person re-identification, also known as pedestrian re-identification, is a technology that uses computer vision technology to determine whether a specific pedestrian exists in an image or video sequence. It is widely considered to be a sub-problem of image retrieval. Given a monitored pedestrian image, retrieve the pedestrian image across devices. It aims to make up for the visual limitations of fixed cameras and can be combined with pedestrian detection / pedestrian tracking technology. It can be widely used in intelligent video surveillance, intelligent security and other fields.

[0003] However, traditional search algorithms are difficult to implement on mobile devices such as mobile phones due to the huge network and huge amount of data. At the same time, traditional search algorithms are highly dependent on clothing features. When the target person changes clothes, there is often a problem of low recognition accuracy. Summary of the invention

[0004] In view of this, the present application provides a pedestrian re-identification method, device, electronic device and readable storage medium to solve the above-mentioned problems.

[0005] In a first aspect, the present application provides a method for pedestrian re-identification, the method comprising:

[0006] Obtain an image to be recognized containing a human portrait;

[0007] Inputting the image to be identified into a re-identification model for re-identification to obtain a re-identification result;

[0008] The re-identification model uses portrait images as training samples to train a pre-constructed re-identification network including an outline sub-network and an appearance sub-network to obtain a trained appearance sub-network.

[0009] In an optional embodiment, the re-identification model is trained by the following steps:

[0010] Acquire a portrait image and a target image, wherein the target image includes a pedestrian identification mark;

[0011] Using the portrait image as a training sample, using the target image as a label, and using a pre-constructed loss function to train the re-identification network to obtain a trained re-identification network;

[0012] The appearance subnetwork in the trained re-ID network is used as the re-ID model.

[0013] In an optional embodiment, the shape sub-network includes a segmentation sub-network and a shape feature sub-network; the step of using the portrait image as a training sample, using the target image as a label, and using a pre-constructed loss function to train the re-identification network to obtain the trained re-identification network includes:

[0014] Inputting the portrait image into the segmentation sub-network, and using the segmentation sub-network to perform mask extraction on the portrait image to obtain an original mask image;

[0015] Processing the original mask image using the shape feature sub-network to obtain a fine-grained mask image and a shape feature image;

[0016] Inputting the portrait image into the appearance sub-network, and using the appearance sub-network to extract features of the portrait image to obtain an appearance feature image;

[0017] Calculating a loss value of the loss function according to the original mask image, the fine-grained mask image, the appearance feature image, the shape feature image and the target image;

[0018] The parameters of the re-identification network are updated according to the loss value until the loss value meets a preset condition, thereby obtaining a trained re-identification network so that the shape sub-network and the appearance sub-network can learn from each other.

[0019] In an optional embodiment, the shape feature subnetwork includes a fine-grained subnetwork and a feature subnetwork; the step of using the shape feature subnetwork to process the original mask image to obtain a fine-grained mask image and a shape feature image includes:

[0020] Inputting the original mask image into the fine-grained sub-network, and using the fine-grained sub-network to perform fine-grained mask extraction on the original mask image to obtain a fine-grained mask image;

[0021] Extracting skeleton point features from the portrait image to obtain a skeleton point feature image;

[0022] Performing clustering processing on the skeleton point feature image to obtain a cluster label of a fine-grained mask image corresponding to the skeleton point feature image, wherein the cluster label represents a posture category of the fine-grained mask image;

[0023] According to the clustering label of the fine-grained mask image, the fine-grained mask image is input into the feature sub-network, and the feature sub-network is used to extract features of the fine-grained mask image to obtain an appearance feature image.

[0024] In an optional implementation, the feature sub-network includes a plurality of feature sub-branches, and the clustering labels correspond one-to-one to the feature sub-branches;

[0025] The step of inputting the fine-grained mask image into the feature sub-network according to the clustering label of the fine-grained mask image comprises:

[0026] According to the clustering label of the fine-grained mask image, the fine-grained mask image is input into a feature sub-branch corresponding to the clustering label.

[0027] In an optional embodiment, the loss function includes a shape feature loss function, an appearance feature loss function, a logical interaction loss function, and a feature interaction loss function;

[0028] The step of calculating the loss value of the loss function according to the original mask image, the fine-grained mask image, the appearance feature image, the shape feature image and the target image comprises:

[0029] Calculating a first output value of the shape feature loss function using the original mask image, the fine-grained mask image, the shape feature image and the target image;

[0030] Calculating a second output value of the appearance feature loss function using the appearance feature image and the target image;

[0031] Obtaining an appearance feature map output by each feature layer included in the appearance sub-network and an appearance feature map output by each feature layer included in the appearance sub-network, and calculating a third output value of the feature interaction loss function using each of the appearance feature maps and the appearance feature maps;

[0032] Obtaining a first value of a logistic regression layer included in the appearance sub-network and a second value of a logistic regression layer included in the shape sub-network, and calculating a fourth output value of the logistic interaction loss function using the first value and the second value;

[0033] Calculating a weighted sum of the first output value, the second output value, the fourth output value, and the third output value;

[0034] The weighted sum is used as the loss value of the loss function.

[0035] In an optional embodiment, the step of updating the parameters of the re-identification network according to the loss value until the loss value satisfies a preset condition to obtain the trained re-identification network includes:

[0036] Determine whether the loss value is less than a preset threshold;

[0037] If yes, then stop updating the parameters of the re-identification network to obtain a trained re-identification network;

[0038] If not, the parameters of the re-identification network are updated, and the above steps are repeated until the loss value is less than the preset threshold, thereby obtaining a trained re-identification network.

[0039] In a second aspect, the present application provides a pedestrian re-identification device, the device comprising:

[0040] An acquisition module, used for acquiring an image to be recognized containing a human portrait;

[0041] A recognition module, used for inputting the image to be recognized into a re-recognition model for re-recognition to obtain a re-recognition result;

[0042] The re-identification model uses portrait images as training samples to train a pre-constructed re-identification network including an outline sub-network and an appearance sub-network to obtain a trained appearance sub-network.

[0043] In a third aspect, the present application provides an electronic device, comprising a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and the processor executes the machine-readable instructions to perform the steps of the pedestrian re-identification method described in any one of the aforementioned implementations.

[0044] In a fourth aspect, the present application provides a readable storage medium, wherein the readable storage medium stores a computer program, and when the computer program is executed, the steps of the pedestrian re-identification method described in any one of the aforementioned implementations are implemented.

[0045] The embodiment of the present application provides a pedestrian re-identification method, device, electronic device and readable storage medium, which obtains a re-identification result by obtaining an image to be identified containing a human portrait, inputting the image to be identified into a re-identification model for re-identification. The re-identification model uses the human portrait image as a training sample, trains a pre-constructed re-identification network including an appearance sub-network and an appearance sub-network, and obtains a trained appearance sub-network, which improves the recognition accuracy and recognition efficiency of pedestrian re-identification.

[0046] In order to make the above-mentioned objects, features and advantages of the present application more obvious and understandable, some examples are given below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 A structural block diagram of an electronic device provided in an embodiment of the present application.

[0049] Figure 2 This is one of the flowcharts of the pedestrian re-identification method provided in the embodiment of the present application.

[0050] Figure 3 A schematic diagram of re-identification provided in an embodiment of the present application.

[0051] Figure 4 The second flowchart of a re-identification method provided in an embodiment of the present application.

[0052] Figure 5 A schematic diagram of the network structure of a re-identification network shown in an embodiment of the present application.

[0053] Figure 6 A comparison chart of re-identification effects provided in an embodiment of the present application.

[0054] Figure 7 A functional schematic diagram of a pedestrian re-identification device provided in an embodiment of the present application.

[0055] Icon: 100 - electronic device; 110 - memory; 120 - processor; 130 - pedestrian re-identification device; 131 - acquisition module; 132 - identification module; 140 - communication unit. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0057] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0058] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0059] In addition, the terms “first”, “second”, etc., if used, are merely used to distinguish between the descriptions and should not be understood as indicating or implying relative importance.

[0060] It should be noted that, in the absence of conflict, the features in the embodiments of the present application may be combined with each other.

[0061] As introduced in the background technology, person re-identification, also known as pedestrian re-identification, is a technology that uses computer vision technology to determine whether a specific pedestrian exists in an image or video sequence. It is widely considered to be a sub-problem of image retrieval. Given a monitored pedestrian image, retrieve the pedestrian image across devices. It aims to make up for the visual limitations of fixed cameras and can be combined with pedestrian detection / pedestrian tracking technology. It can be widely used in intelligent video surveillance, intelligent security and other fields.

[0062] With the rapid development of deep learning, the development of person re-identification has also been very rapid in recent years. In some commonly used data sets, the algorithm has shown a trend of surpassing the human level. However, for person re-identification technology to move from academic research to mature applications, there are still some problems that need to be solved. One of them is that the existing algorithms are based on an assumption that the clothes of the person will not change, and this assumption is different from the actual situation. In reality, the same person appearing in different cameras is likely to wear different clothes, and this possibility becomes greater and greater as time goes by. At the same time, due to the high requirements of the hardware equipment that deep learning relies on, it is difficult to realize huge data calculations on ordinary mobile devices.

[0063] In other words, the inventors have found that traditional search algorithms are difficult to implement on mobile devices such as mobile phones due to the huge network and huge amount of data. At the same time, traditional search algorithms are highly dependent on clothing features, and often have low recognition accuracy when the target person changes clothes.

[0064] In view of this, the embodiments of the present application provide a pedestrian re-identification method, device, electronic device and readable storage medium. The method trains and learns the re-identification network including the shape sub-network and the appearance sub-network, and learns the appearance features and shape features of the image from each other to improve the accuracy of re-identification. At the same time, when in use, only the appearance sub-network obtained after training can be deployed on a mobile device, which speeds up the re-identification rate.

[0065] It should be noted that the defects existing in the solutions in the above-mentioned prior art are the results obtained by the applicant after practice and careful research. Therefore, the discovery process of the above-mentioned problems and the solutions proposed in the embodiments of the present application below for the above-mentioned problems should be the contributions made by the applicant to the present application during the application process.

[0066] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and key points in the embodiments can be combined with each other.

[0067] Please refer to Figure 1 , Figure 1 This is a structural block diagram of an electronic device 100 provided in an embodiment of the present application. The device may include a processor 120, a memory 110, a pedestrian re-identification device 130 and a communication unit 140. The memory 110 stores machine-readable instructions executable by the processor 120. When the electronic device 100 is running, the processor 120 and the memory 110 communicate through a bus, and the processor 120 executes the machine-readable instructions and executes the pedestrian re-identification method.

[0068] The memory 110 , the processor 120 , and the communication unit 140 are electrically connected to each other directly or indirectly to achieve signal transmission or interaction.

[0069] For example, these components may be electrically connected to each other via one or more communication buses or signal lines. The pedestrian re-identification device 130 includes at least one software function module that can be stored in the memory 110 in the form of software or firmware. The processor 120 is used to execute the executable module stored in the memory 110, such as the software function module or computer program included in the pedestrian re-identification device 130.

[0070] Among them, the memory 110 can be, but is not limited to, random access memory (Random Access Memory, RAM), read only memory (Read Only Memory, ROM), programmable read-only memory (Programmable Read-Only Memory, PROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, EPROM), electrically erasable read-only memory (Electric Erasable Programmable Read-Only Memory, EEPROM), etc.

[0071] The processor 120 may be an integrated circuit chip with signal processing capability. The processor 120 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), and the like.

[0072] It can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. It can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application. The general processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0073] In the embodiment of the present application, the memory 110 is used to store the program, and the processor 120 is used to execute the program after receiving the execution instruction. The process definition method disclosed in any implementation of the embodiment of the present application can be applied to the processor 120, or implemented by the processor 120.

[0074] The communication unit 140 is used to establish a communication connection between the electronic device 100 and other electronic devices through a network, and to send and receive data through the network.

[0075] In some embodiments, the network may be any type of wired or wireless network, or a combination thereof. By way of example only, the network may include a wired network, a wireless network, a fiber optic network, a telecommunications network, an intranet, the Internet, a local area network (LAN), a wide area network (WAN), a wireless local area network (WLAN), a metropolitan area network (MAN), a wide area network (WAN), a public switched telephone network (PSTN), a Bluetooth network, a ZigBee network, or a near field communication (NFC) network, or any combination thereof.

[0076] In the embodiment of the present application, the electronic device 100 may be, but is not limited to, a smart phone, a personal computer, a tablet computer, or other device with processing capabilities.

[0077] Understandably, Figure 1 The structure shown is for illustration only. The electronic device 100 may also have Figure 1More or fewer components than shown, or with Figure 1 Different configurations are shown. Figure 1 The components shown may be implemented in hardware, software or a combination thereof.

[0078] The following is based on Figure 1 The structural diagram of the electronic device 100 illustrates in detail the steps of the pedestrian re-identification method provided in the embodiment of the present application.

[0079] Please refer to Figure 2 , Figure 2 This is one of the flowcharts of the pedestrian re-identification method provided in the embodiment of the present application.

[0080] Step S1, obtaining an image to be recognized containing a human portrait.

[0081] Step S2: input the image to be identified into the re-identification model for re-identification to obtain a re-identification result.

[0082] Among them, the re-identification model uses portrait images as training samples to train a pre-built re-identification network including an appearance sub-network and an appearance sub-network, and obtains a trained appearance sub-network.

[0083] The image to be identified can be obtained by photographing the current electronic device through a camera, or by taking a screenshot from a video shot by the current electronic device, or can be pre-stored in a memory by the current electronic device through a network and retrieved from the memory when needed.

[0084] For further information, please refer to Figure 3 , Figure 3 A schematic diagram of re-identification provided in an embodiment of the present application. Step S2, inputting the image to be identified into the re-identification model for re-identification, and obtaining the re-identification result can be achieved in the following manner:

[0085] The image to be identified is input into the re-identification model for re-identification, and the image features and pedestrian information of the image to be identified are obtained. The image features and pedestrian information are compared with the pedestrian identities and image features in the retrieval library for similarity and sorted to obtain the re-identification result.

[0086] like Figure 3 As shown in , in the re-identification results obtained by the above method, the images with higher similarity are ranked higher.

[0087] As a possible implementation scenario, the pedestrian re-identification method provided in the embodiment of the present application can be used in ordinary image retrieval. For example, the user can input a picture of a host or celebrity of interest into a corresponding packaged application, use the re-identification model to perform re-identification, and obtain the re-identification result, so as to obtain more pictures or video information of the corresponding host or celebrity.

[0088] As another possible implementation scenario, the pedestrian re-identification method provided in the embodiment of the present application can also be used in the tracking of target objects in the security field, for example, the tracking of suspects. The user can also input the image of the target object of interest into the corresponding packaged application, use the re-identification model to perform re-identification, and obtain the re-identification result, and then obtain more images or video information of the corresponding target object taken by other camera devices, thereby achieving tracking.

[0089] The embodiment of the present application trains a pre-constructed re-identification network including a shape sub-network and an appearance sub-network. When the training is completed, only the trained appearance sub-network is retained, and the trained appearance sub-network is used for re-identification. In this way, on the one hand, by combining the two networks for training at the same time during the training phase, the appearance features and shape features of the portrait can be comprehensively extracted, more portrait features can be retained, and the accuracy of re-identification can be improved. On the other hand, when it is actually used and deployed, the trained shape sub-network is deleted, and only the trained appearance sub-network with a relatively simple structure is used as the re-identification model for re-identification, which can further improve the efficiency of re-identification of the image to be identified.

[0090] It is understandable that the above-mentioned re-identification model can be pre-trained in other electronic devices and then migrated to the current electronic device, or it can be pre-trained in the current electronic device and stored.

[0091] In an optional implementation, please refer to Figure 4 , Figure 4 The second flowchart of a re-identification method provided in an embodiment of the present application is as follows. The re-identification model is trained through the following steps S100-S300:

[0092] Step S100, obtaining a portrait image and a target image, wherein the target image includes a pedestrian identification.

[0093] The portrait image may come from an existing dataset on the Internet, such as DukeMTMC-reID dataset, DukeMTMC-attribute dataset, Market-1501 dataset, DG-Market dataset, Market1501-attribute dataset, CUHK03 dataset, etc. At the same time, the portrait image may be an RGB type image.

[0094] Step S200, taking the portrait image as a training sample, taking the target image as a label, and using a pre-constructed loss function to train the re-identification network to obtain a trained re-identification network.

[0095] Please refer to Figure 5 , Figure 5 A schematic diagram of the network structure of a re-identification network shown in an embodiment of the present application. As shown in the figure, the shape sub-network includes a segmentation sub-network and a shape feature sub-network. The shape feature sub-network includes a fine-grained sub-network and a feature sub-network.

[0096] The portrait image is used as a training sample, the target image is used as a label, and the re-identification network is trained using a pre-constructed loss function. The trained re-identification network can be obtained by the following steps S200-1 to S200-5:

[0097] Step S200 - 1 , inputting the portrait image into the segmentation sub-network, and using the segmentation sub-network to perform mask extraction on the portrait image to obtain an original mask image.

[0098] Optionally, the segmentation subnetwork may be a pre-trained segmentation model, and the segmentation subnetwork is used to extract the mask of the pedestrian in the portrait image, that is, the original mask image.

[0099] However, the original mask image extracted by the segmentation sub-network has low accuracy and is not enough to learn discriminative features in the scene of clothing changes. Therefore, a target image including pedestrian identification is used, and pedestrian identification and posture category are used as guidance information. Then, the subsequent appearance feature sub-network is trained and learned to guide the process of mask image generation, so that it changes from coarse-grained to fine-grained. The process is as follows:

[0100] Step S200-2, using the shape feature sub-network to process the original mask image to obtain a fine-grained mask image and a shape feature image.

[0101] Optionally, step S200-2 may be processed in the following manner to obtain a fine-grained mask image and an appearance feature image:

[0102] The original mask image is input into the fine-grained sub-network, and the fine-grained sub-network is used to perform fine-grained mask extraction on the original mask image to obtain a fine-grained mask image.

[0103] The skeleton point features of the portrait image are extracted to obtain a skeleton point feature image.

[0104] The skeleton point feature image is clustered to obtain cluster labels of the fine-grained mask image corresponding to the skeleton point feature image, wherein the cluster labels represent the posture category of the fine-grained mask image.

[0105] According to the clustering label of the fine-grained mask image, the fine-grained mask image is input into the feature sub-network, and the feature sub-network is used to extract features of the fine-grained mask image to obtain an appearance feature image.

[0106] Among them, the above-mentioned posture categories can be pre-set according to one's own needs. For example, posture categories such as "squatting", "standing", and "walking" are set according to the posture of the human body, and a corresponding cluster label is assigned to each posture category. For example, the cluster label corresponding to the posture category "squatting" is 1, the cluster label corresponding to the posture category "standing" is 2, and the cluster label corresponding to the posture category "walking" can be 3.

[0107] In this way, by using the pedestrian identity and posture category as guiding information, and then training and learning by the subsequent appearance feature sub-network, the process of mask image generation is guided, so that the original mask image is changed from coarse-grained to fine-grained, and a fine-grained mask image is obtained. According to the posture category of the human body in the image, the corresponding fine-grained mask image is input into the corresponding feature sub-network for fine-grained feature extraction, thereby obtaining more accurate feature information, which is conducive to improving the accuracy of re-identification.

[0108] Please read next Figure 5 In an optional implementation, the feature sub-network includes multiple feature sub-branches, and the clustering labels correspond one-to-one to the feature sub-branches.

[0109] Take the shape feature subnetwork as ResNet50 network, and the posture categories include three categories as an example. Since the ResNet50 network includes four layers. Optionally, the embodiment of the present application copies the fourth layer of the shape feature subnetwork three times to obtain three feature sub-branches, each branch corresponding to a posture category.

[0110] After the clustering labels of the fine-grained mask image are obtained, the fine-grained mask image is input into a feature sub-branch corresponding to the clustering labels according to the clustering labels of the fine-grained mask image.

[0111] As an implementation method, the fine-grained mask image and the input matching of the feature sub-branch can be achieved by the following formula, which is as follows:

[0112]

[0113] in, represents the i-th feature sub-branch of the fourth layer, (X s，3 ) represents the output features of the third layer, b represents the number of feature sub-branches, here is 3, l k Represents the selection function, which is 1 only when the cluster label is k, and 0 otherwise.

[0114] For example, there are three cluster labels, corresponding to the three feature sub-branches of the fourth layer of the ResNet50 network. kIndicates how to select the corresponding feature sub-branch according to the clustering label. The following is a detailed description of how to select the corresponding fine-grained mask image input value and the feature sub-branch corresponding to the clustering label based on the formula:

[0115] Assumption 1: If the cluster label of a fine-grained mask image is 1, then when i=1, l k is 1, i=2 or 3, l k is 0. At this time:

[0116]

[0117] That is, the output features of the third layer of the shape feature sub-network (the fine-grained mask image) enter the first feature sub-branch of the shape feature sub-network.

[0118] Assumption 2: If the cluster label of a fine-grained mask image is 2, then when i=2, l k is 1, i=1 or 3, l k is 0. At this time:

[0119]

[0120] That is, the output features of the third layer of the shape feature sub-network (the fine-grained mask image) enter the second feature sub-branch of the shape feature sub-network.

[0121] Assumption 3: If the cluster label of a fine-grained mask image is 3, then when i=3, l k is 1, i=1 or 2, l k is 0. At this time:

[0122]

[0123] That is, the output features of the third layer of the shape feature sub-network (the fine-grained mask image) enter the third feature sub-branch of the shape feature sub-network.

[0124] As another implementation, cluster labels and feature sub-branches may be matched in advance to generate a matching correspondence relationship. When inputting a match, the feature sub-branch corresponding to the cluster label may be searched according to the matching correspondence relationship.

[0125] In this way, by setting multiple feature sub-branches corresponding to posture categories, different fine-grained mask images are guided by the posture class to enter different feature sub-branches for different feature extractions, which is beneficial for the model to adjust parameters and learn according to different postures, thereby extracting finer-grained appearance features.

[0126] It is not difficult to understand that the above steps learn the appearance features in the appearance sub-network through the fine-grained mask image, and there is other information unrelated to clothes in the general image, such as the appearance features of the face and hair, which need to be learned and acquired. It is difficult to obtain such information through the appearance features learned through the above fine-grained mask image. It is difficult to obtain appearance features in the colored RGB image (i.e., the portrait image used as the training sample), but it is easier to obtain appearance features such as the face and hair. Therefore, the embodiment of the present application proposes that complementary features can be obtained from the fine-grained mask image and the RGB image at the same time. Please refer to Figure 5 , which is elaborated in detail below:

[0127] Step S200 - 3 , inputting the portrait image into the appearance sub-network, and using the appearance sub-network to extract features of the portrait image to obtain an appearance feature image.

[0128] Step S200 - 4 , calculating the loss value of the loss function according to the original mask image, the fine-grained mask image, the appearance feature image, the shape feature image and the target image.

[0129] In an optional embodiment, the loss function includes a shape feature loss function, an appearance feature loss function, a logical interaction loss function, and a feature interaction loss function.

[0130] The first output value of the shape feature loss function is calculated using the original mask image, the fine-grained mask image, the shape feature image and the target image.

[0131] The second output value of the appearance feature loss function is calculated using the appearance feature image and the target image.

[0132] Obtain the appearance feature map output by each feature layer included in the appearance sub-network, and the shape feature map output by each feature layer included in the shape sub-network, and use each appearance feature map and shape feature map to calculate the third output value of the feature interaction loss function.

[0133] Obtain a first value of the logistic regression layer included in the appearance sub-network and a second value of the logistic regression layer included in the shape sub-network, and use the first value and the second value to calculate a fourth output value of the logistic interaction loss function.

[0134] A weighted sum of the first output value, the second output value, the fourth output value, and the third output value is calculated.

[0135] The weighted sum is used as the loss value of the loss function.

[0136] Among them, the shape feature loss function is:

[0137] L M =L G +L R

[0138] in:

[0139]

[0140] is the cross entropy loss function applied in the shape subnetwork, is the triplet loss function applied in the shape sub-network. Θ m , Θ s are the parameters of the segmentation sub-network and the appearance feature sub-network respectively.

[0141]

[0142] M is a fine-grained mask image, is the original mask image.

[0143] Through the above L G The loss function can supervise the accuracy of the shape features learned by the shape sub-network. At the same time, since the pre-trained segmentation sub-network also contains some prior shape knowledge, this knowledge may be lost in the process of using pedestrian identification guidance. Therefore, the embodiment of the present application also proposes a method for retaining the knowledge in the segmentation sub-network. R As shown in the loss function, the difference between the generated fine-grained mask image and the original mask image extracted by the segmentation sub-network cannot be too large. R Loss function, based on L R The loss function can retain the knowledge in the segmentation sub-network, so that the subsequent fine-grained mask image still retains valid information.

[0144] Furthermore, the appearance feature loss function is:

[0145]

[0146] is the cross entropy loss function applied in the appearance sub-network, is the triplet loss function applied in the appearance sub-network.

[0147] Through the above appearance feature loss function, the appearance features learned by the appearance sub-network can be supervised, thereby improving the accuracy of re-identification.

[0148] Furthermore, since the appearance sub-network can learn the appearance features of the image and the shape sub-network can learn the shape features of the image, in order to learn each other's knowledge, the embodiment of the present application proposes to learn the knowledge of the intermediate layer and the logistic regression layer included in the appearance sub-network from the knowledge of the intermediate layer and the logistic regression layer included in the shape sub-network.

[0149] For mutual learning in the middle layer, the following feature interaction loss function can be used for supervised learning:

[0150]

[0151] Among them, l is the number of network layers, Θ a is the parameter of the appearance sub-network, Θ m is the parameter of the segmentation sub-network, Θ s are the parameters of the appearance feature sub-network.

[0152]

[0153] in, is the i-th shape feature map of the d-th layer in the shape subnetwork, It is the j-th shape feature map of the d-th layer in the shape sub-network.

[0154]

[0155] in, is the i-th appearance feature map of the d-th layer in the appearance subnetwork, It is the jth appearance feature map of the dth layer in the shape sub-network.

[0156] For mutual learning of logistic regression layers, supervised learning can be performed using the following logistic interaction loss function:

[0157] L KL (Θ a , Θ m , Θ s )=D KL (P a ||P s )+D KL (P s ||P a )

[0158] Among them, P s Represents the output value of the logistic regression layer of the shape subnetwork, P a Represents the output value of the logistic regression layer of the appearance subnetwork.

[0159] Based on the logical interaction loss function and the feature interaction loss function, the mutual learning between the appearance sub-network and the shape sub-network can be supervised to achieve fine-grained learning of multiple features, so that each sub-network can learn the appearance features and shape features separately. On the one hand, in the subsequent deployment learning, only the appearance sub-network with a relatively simple structure can be deployed for re-identification. On the other hand, the accuracy of re-identification can be improved.

[0160] That is to say, the preset loss function L can be expressed as follows:

[0161] L=L M +L R +λ SL L SL +λ KL L kL

[0162] Among them, L M is the shape feature loss function, L R is the appearance feature loss function, λ SI and KL is a weight parameter, usually a constant, L SL is the feature interaction loss function, λ KL is the logistic interaction loss function.

[0163] Step S200-5, updating the parameters of the re-identification network according to the loss value until the loss value meets the preset conditions, thereby obtaining a trained re-identification network so that the shape sub-network and the appearance sub-network can learn from each other.

[0164] In an optional implementation, the step of updating the parameters of the re-identification network according to the loss value until the loss value satisfies a preset condition, and obtaining the trained re-identification network includes:

[0165] Determine whether the loss value is less than the preset threshold.

[0166] If yes, stop updating the parameters of the re-identification network and obtain the trained re-identification network;

[0167] If not, update the parameters of the re-identification network and repeat the above steps until the loss value is less than the preset threshold to obtain the trained re-identification network.

[0168] According to the loss function, the target image is compared with other feature maps (such as the original mask image, the fine-grained mask image, and the shape feature image) to calculate the loss value of the loss function, and the re-identification network is repeatedly trained and the parameters of the re-identification network are updated until the loss value is less than the preset threshold. The trained re-identification model is obtained to improve the accuracy of re-identification.

[0169] Step S300: Using the trained appearance sub-network in the re-identification network as a re-identification model.

[0170] Please refer to Figure 6 , Figure 6 A comparison diagram of re-identification effects provided by an embodiment of the present application. The leftmost image in the figure is the image to be identified, the upper right image is the re-identification result obtained by re-identification using a traditional pedestrian re-identification algorithm, and the lower right image is the re-identification result obtained by re-identification using the pedestrian re-identification method provided by the present application.

[0171] Each re-identification result includes five recognition targets obtained after sorting from large to small according to similarity, namely rank 1 to rank 5, where the border color is darker, indicating that the rank is an incorrectly recognized target, and the border color is lighter, indicating that the rank is a correctly recognized target. It can be seen from the figure that the first three recognition targets and the last recognition target in the re-identification results obtained after re-identification by the traditional pedestrian re-identification algorithm are all recognized incorrectly. However, the first three recognition targets in the re-identification results obtained after re-identification using the pedestrian re-identification method provided in the embodiment of the present application are all recognized correctly. It can be seen from the comparison that the re-identification results obtained after re-identification using the pedestrian re-identification method provided in the embodiment of the present application are not affected by the pedestrian changing clothes or hairstyle, and the re-identification accuracy is higher.

[0172] In this way, by training the pre-built re-identification network including the shape sub-network and the appearance sub-network, only the trained appearance sub-network is retained when it is used after training, and the trained appearance sub-network is used for re-identification, and the appearance features and shape features of the portrait are comprehensively extracted, so that more portrait features are retained and the accuracy of re-identification is improved. When actually deployed, the trained shape sub-network is deleted, and only the trained appearance sub-network with a relatively simple structure is used as the re-identification model for re-identification, which further improves the efficiency of re-identification of the image to be identified.

[0173] Based on the same invention concept, please refer to Figure 7 , Figure 7 This is a functional schematic diagram of a pedestrian re-identification device provided in an embodiment of the present application. Figure 2 The pedestrian re-identification method shown corresponds to a pedestrian re-identification device 130, the device comprising:

[0174] The acquisition module 131 is used to acquire an image to be recognized that contains a human portrait.

[0175] The recognition module 132 is used to input the image to be recognized into the re-recognition model for re-recognition to obtain a re-recognition result.

[0176] Among them, the re-identification model uses portrait images as training samples to train a pre-built re-identification network including an appearance sub-network and an appearance sub-network, and obtains a trained appearance sub-network.

[0177] Since the principle of solving the problem by the device in the embodiment of the present application is similar to the above-mentioned pedestrian re-identification method in the embodiment of the present application, the implementation principle of the device can refer to the implementation principle of the method, and the repeated parts will not be repeated.

[0178] An embodiment of the present application also provides a readable storage medium, in which a computer program is stored. When the computer program is executed, the above-mentioned pedestrian re-identification method is implemented.

[0179] In summary, the embodiments of the present application provide a method, device, electronic device and readable storage medium for pedestrian re-identification, which obtains a re-identification result by acquiring an image to be identified containing a human portrait and inputting the image to be identified into a re-identification model for re-identification. The re-identification model uses the human portrait image as a training sample to train a pre-constructed re-identification network including an outer shape sub-network and an appearance sub-network, and the trained appearance sub-network improves the recognition accuracy and recognition efficiency of pedestrian re-identification.

[0180] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A pedestrian re-identification method, characterized in that: The method comprises: Obtain an image to be recognized containing a human portrait; Inputting the image to be identified into a re-identification model for re-identification to obtain a re-identification result; The re-identification model is trained by the following steps: Acquire a portrait image and a target image, wherein the target image includes a pedestrian identification mark; Using the portrait image as a training sample and the target image as a label, and using a pre-constructed loss function to train a pre-constructed re-identification network including an appearance sub-network and an appearance sub-network, to obtain a trained re-identification network; Use the appearance sub-network in the trained re-ID network as the re-ID model; The shape sub-network includes a segmentation sub-network and a shape feature sub-network; the steps of using the portrait image as a training sample, using the target image as a label, and using a pre-constructed loss function to train the pre-constructed re-identification network including the shape sub-network and the appearance sub-network to obtain the trained re-identification network include: Inputting the portrait image into the segmentation sub-network, and using the segmentation sub-network to perform mask extraction on the portrait image to obtain an original mask image; Processing the original mask image using the shape feature sub-network to obtain a fine-grained mask image and a shape feature image; Inputting the portrait image into the appearance sub-network, and using the appearance sub-network to extract features of the portrait image to obtain an appearance feature image; Calculating a loss value of the loss function according to the original mask image, the fine-grained mask image, the appearance feature image, the shape feature image and the target image; The parameters of the re-identification network are updated according to the loss value until the loss value meets a preset condition, thereby obtaining a trained re-identification network so that the shape sub-network and the appearance sub-network can learn from each other.

2. The pedestrian re-identification method according to claim 1, characterized in that: The shape feature subnetwork includes a fine-grained subnetwork and a feature subnetwork; the step of using the shape feature subnetwork to process the original mask image to obtain a fine-grained mask image and a shape feature image includes: Inputting the original mask image into the fine-grained sub-network, and using the fine-grained sub-network to perform fine-grained mask extraction on the original mask image to obtain a fine-grained mask image; Extracting skeleton point features from the portrait image to obtain a skeleton point feature image; Performing clustering processing on the skeleton point feature image to obtain a cluster label of a fine-grained mask image corresponding to the skeleton point feature image, wherein the cluster label represents a posture category of the fine-grained mask image; According to the clustering label of the fine-grained mask image, the fine-grained mask image is input into the feature sub-network, and the feature sub-network is used to extract features of the fine-grained mask image to obtain an appearance feature image.

3. The pedestrian re-identification method according to claim 2, characterized in that: The feature sub-network includes a plurality of feature sub-branches, and the clustering labels correspond one to one to the feature sub-branches; The step of inputting the fine-grained mask image into the feature sub-network according to the clustering label of the fine-grained mask image comprises: According to the clustering label of the fine-grained mask image, the fine-grained mask image is input into a feature sub-branch corresponding to the clustering label.

4. The pedestrian re-identification method according to claim 1, characterized in that: The loss functions include shape feature loss function, appearance feature loss function, logical interaction loss function and feature interaction loss function; The step of calculating the loss value of the loss function according to the original mask image, the fine-grained mask image, the appearance feature image, the shape feature image and the target image comprises: Calculating a first output value of the shape feature loss function using the original mask image, the fine-grained mask image, the shape feature image and the target image; Calculating a second output value of the appearance feature loss function using the appearance feature image and the target image; Obtaining an appearance feature map output by each feature layer included in the appearance sub-network and an appearance feature map output by each feature layer included in the appearance sub-network, and calculating a third output value of the feature interaction loss function using each of the appearance feature maps and the appearance feature maps; Obtaining a first value of a logistic regression layer included in the appearance sub-network and a second value of a logistic regression layer included in the shape sub-network, and calculating a fourth output value of the logistic interaction loss function using the first value and the second value; Calculating a weighted sum of the first output value, the second output value, the fourth output value, and the third output value; The weighted sum is used as the loss value of the loss function.

5. The pedestrian re-identification method according to claim 4, characterized in that: The step of updating the parameters of the re-identification network according to the loss value until the loss value satisfies a preset condition to obtain a trained re-identification network comprises: Determine whether the loss value is less than a preset threshold; If yes, then stop updating the parameters of the re-identification network to obtain a trained re-identification network; If not, the parameters of the re-identification network are updated, and the above steps are repeated until the loss value is less than the preset threshold, thereby obtaining a trained re-identification network.

6. A pedestrian re-identification device, characterized in that: The device comprises: An acquisition module, used for acquiring an image to be recognized containing a human portrait; A recognition module, used for inputting the image to be recognized into a re-recognition model for re-recognition to obtain a re-recognition result; The re-identification model is trained by the following steps: Acquire a portrait image and a target image, wherein the target image includes a pedestrian identification mark; Using the portrait image as a training sample and the target image as a label, and using a pre-constructed loss function to train a pre-constructed re-identification network including an appearance sub-network and an appearance sub-network, to obtain a trained re-identification network; Use the appearance sub-network in the trained re-ID network as the re-ID model; The shape sub-network includes a segmentation sub-network and a shape feature sub-network; the steps of using the portrait image as a training sample, using the target image as a label, and using a pre-constructed loss function to train the pre-constructed re-identification network including the shape sub-network and the appearance sub-network to obtain the trained re-identification network include: Inputting the portrait image into the segmentation sub-network, and using the segmentation sub-network to perform mask extraction on the portrait image to obtain an original mask image; Processing the original mask image using the shape feature sub-network to obtain a fine-grained mask image and a shape feature image; Inputting the portrait image into the appearance sub-network, and using the appearance sub-network to extract features of the portrait image to obtain an appearance feature image; Calculating a loss value of the loss function according to the original mask image, the fine-grained mask image, the appearance feature image, the shape feature image and the target image; The parameters of the re-identification network are updated according to the loss value until the loss value meets a preset condition, thereby obtaining a trained re-identification network so that the shape sub-network and the appearance sub-network can learn from each other.

7. An electronic device, characterized in that: The electronic device includes a processor, a memory and a bus, the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and the processor executes the machine-readable instructions to perform the steps of the pedestrian re-identification method described in any one of claims 1-5.

8. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed, the steps of the pedestrian re-identification method according to any one of claims 1 to 5 are implemented.

Images