A method, device, equipment and storage medium for target re-identification

By decomposing the target image into multiple channel images and matching features, the problem of low recognition accuracy caused by modal differences in cross-modal pedestrian re-identification is solved, and the recognition accuracy is significantly improved.

CN112257619BActive Publication Date: 2024-11-05XIAN JINGNENG FUTURE TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202011168154.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-10-27
Publication Date
2024-11-05
Estimated Expiration
2040-10-27

AI Technical Summary

Technical Problem

In cross-modal pedestrian re-identification, visible light and infrared images have low recognition accuracy due to modal differences in feature space.

Method used

By determining the number of color channels of the image to be identified, the target image is decomposed into multiple channel images that are the same as the image to be identified, and feature extraction and matching are performed to reduce modal differences caused by channel inconsistency and improve recognition accuracy.

Benefits of technology

It significantly improves the accuracy of cross-modal target re-identification, overcomes the recognition error caused by channel inconsistency, and improves the accuracy of the recognition results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112257619B_ABST
    Figure CN112257619B_ABST
Patent Text Reader

Abstract

The embodiments of the present application provide a target re-identification method, apparatus, device and storage medium, wherein the method comprises: determining the number of color channels of an acquired image to be identified; decomposing a target image including a target object into a plurality of channel images having the same number of color channels as the image to be identified, to obtain a channel image set; performing feature extraction on the channel images in the channel image set and the image to be identified, to obtain a channel image feature set and image to be identified features, respectively; and determining the similarity between the object to be identified in the image to be identified and the target object based on the channel image feature set and the image to be identified features, to obtain a recognition result.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and relate to but are not limited to a target re-identification method, device, equipment and storage medium. Background Art

[0002] In the cross-modal (e.g., visible light to infrared light) pedestrian re-identification method, it is implemented by directly matching the visible light (Visual Identification System, VIS) image with the IR image. In the implementation process, due to the large modality difference between VIS images and infrared (Infrared, IR) images in the feature space, it is not easy to cluster the two types of images, which affects the accuracy of pedestrian re-identification of IR images. Summary of the invention

[0003] The embodiment of the present application provides a technical solution for target re-identification.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The present invention provides a method for target re-identification, the method comprising:

[0006] Determine the number of color channels of the acquired image to be identified;

[0007] Decomposing a target image including a target object into a plurality of channel images having the same number of color channels as the image to be identified, to obtain a channel image set;

[0008] Extracting features of the channel images in the channel image set and the image to be identified, and obtaining a channel image feature set and features of the image to be identified respectively;

[0009] Based on the channel image feature set and the features of the image to be identified, the similarity between the object to be identified in the image to be identified and the target object is determined to obtain a recognition result.

[0010] The present application provides a target re-identification device, the device comprising:

[0011] A first determination module, used to determine the number of color channels of the acquired image to be identified;

[0012] A first decomposition module is used to decompose a target image including a target object into a plurality of channel images having the same number of color channels as the number of color channels of the image to be identified, to obtain a channel image set;

[0013] A first extraction module is used to extract features from the channel images in the channel image set and the image to be identified, and obtain a channel image feature set and features of the image to be identified respectively;

[0014] The second determination module is used to determine the similarity between the object to be identified in the image to be identified and the target object based on the channel image feature set and the image feature to be identified, so as to obtain a recognition result.

[0015] Correspondingly, an embodiment of the present application provides a target re-identification device, which includes a memory and a processor, wherein the memory stores computer-executable instructions, and when the processor runs the computer-executable instructions on the memory, the steps of the method described above can be implemented.

[0016] An embodiment of the present application provides a computer-readable storage medium, on which computer-executable instructions are stored. After the computer-executable instructions are executed, the steps of the method described above can be implemented.

[0017] The embodiments of the present application provide a target re-identification method, apparatus, device and storage medium. First, by determining the number of color channels of the image to be identified, the target image is decomposed into multiple channel images with the same number of channels as the image to be identified. In this way, the difference between modalities caused by channel inconsistency between images can be significantly reduced; then, the image features of the multiple channel images are matched with the features of the image to be identified to determine the similarity between the object to be identified and the target object in the image to be identified, so as to identify whether the object to be identified is the target object; in this way, the accuracy of cross-modal target re-identification can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a schematic diagram of the implementation process of the target re-identification method of the embodiment of the present application;

[0019] Figure 2 A schematic diagram of another implementation flow of the target re-identification method provided in an embodiment of the present application;

[0020] Figure 3 A schematic diagram of another implementation flow of the target re-identification method provided in other embodiments of the present application;

[0021] Figure 4 A schematic diagram of another implementation flow of the target re-identification method provided in an embodiment of the present application;

[0022] Figure 5 A schematic diagram of the implementation framework of the target re-identification method provided in the embodiment of the present application;

[0023] Figure 6This is a schematic diagram of the structure of the target re-identification device according to an embodiment of the present application;

[0024] Figure 7 This is a schematic diagram of the composition structure of the target re-identification device according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the specific technical solution of the invention will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application, but are not used to limit the scope of the present application.

[0026] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0027] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0029] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0030] 1) Pedestrian re-identification: It is a technology that uses computer vision technology to determine whether a specific pedestrian exists in an image or video sequence. Given a surveillance target image, retrieve the target image across devices. It aims to make up for the visual limitations of fixed cameras and can be combined with pedestrian detection / pedestrian tracking technology. It can be widely used in intelligent video surveillance, intelligent security and other fields.

[0031] 2) Cross-modality: In other embodiments, modal content (text, image, video, audio, sensor data, etc.) is used for cross-modal learning to perform joint feature learning and cross-modal relationship modeling, aiming to effectively utilize the correlation of different modal contents to optimize system performance. In the embodiment of the present application, cross-modality can be understood as from visible light images to near-infrared images.

[0032] The following describes an exemplary application of the target re-identification device provided in the embodiment of the present application. The device provided in the embodiment of the present application can be implemented as various types of user terminals such as a laptop computer with image acquisition function, a tablet computer, a desktop computer, a camera, a mobile device (for example, a personal digital assistant, a dedicated messaging device, a portable gaming device), etc., and can also be implemented as a server. The following describes an exemplary application when the device is implemented as a terminal or a server.

[0033] The method can be applied to a target re-identification device. The functions implemented by the method can be implemented by a processor in the target re-identification device calling a program code. Of course, the program code can be stored in a computer-readable storage medium. It can be seen that the target re-identification device at least includes a processor and a storage medium.

[0034] Figure 1 Schematic diagram of the implementation process of the target re-identification method of the present application embodiment. Figure 1 As shown, combined Figure 1 The steps shown are explained:

[0035] Step S101, determining the number of color channels of the acquired image to be recognized.

[0036] In some embodiments, the image to be identified can be a set of images stored in a local terminal (such as a mobile phone, tablet computer, laptop computer, etc.); any frame image in the video can also be used as the image to be identified, and a facial area image can also be detected from any frame image in the video, and the facial area image can be used as the image to be identified. The image to be identified can be a single-channel color image, such as a near-infrared image or a grayscale image, etc., or a multi-channel color image, such as a visible light image, etc. If the image to be identified is a single-channel color image, then the number of color channels of the image to be identified is 1. If the image to be identified is a red, yellow, and blue (RGB) image, then the color channels are R channel, G channel, and B channel, respectively, that is, the number of color channels is 3. In the embodiment of the present application, the image to be identified is a near-infrared image as an example for explanation, then the color channel of the image to be identified is a near-infrared channel, and the number of color channels is 1.

[0037] Step S102 , decomposing a target image including a target object into a plurality of channel images having the same number of color channels as the number of color channels of the image to be identified, to obtain a channel image set.

[0038] In some embodiments, the target image is an image including a target object, and may be an image including a target object captured in a visible light environment. The target object may be a movable object such as a pedestrian, a vehicle, or an animal. If the number of color channels of the image to be identified is 1, the target image is decomposed into multiple single-channel images. If the number of color channels of the image to be identified is M, the target image is decomposed into multiple M-channel images, where M is an integer greater than 1. For example, the target image is an image including a target pedestrian captured in a visible light environment. Taking the image to be identified as a near-infrared image as an example, since the near-infrared image is a single-channel image, that is, the number of color channels is 1, the target image is decoupled into a single-channel image with a color channel of 1. In some possible implementations, the target image can be decoupled into three single-channel images of R, G, and B, and the obtained R channel image, G channel image, and B channel image form a channel image set. The target image can also be decoupled into other color spaces, for example, decoupled into three single-channel images of hue, saturation, and value (HSV), and the obtained H channel image, S channel image, and V channel image form a channel image set, or decoupled into three single-channel images of Y, Cb, and Cr (where Y is the brightness component of the color, and Cb and Cr are the concentration offset components of blue and red, respectively), and the obtained Y channel image, Cb channel image, and Cr channel image form a channel image set. That is, the number of channels of the channel images in the channel image set is the same as the number of channels of the image to be identified. For example, if the image to be identified is a near-infrared image, then the channel images in the channel set are all single-channel images. In this way, the target image under visible light is decoupled to obtain multiple channel images with the same number of color channels as the image to be identified, thereby overcoming the huge difference caused by the inconsistency of channels between near-infrared images and visible light images during training, and thus obtaining more accurate recognition results.

[0039] Step S103, performing feature extraction on the channel images in the channel image set and the image to be identified, to obtain a channel image feature set and features of the image to be identified, respectively.

[0040] In some embodiments, feature extraction is performed on each channel image in the channel set, and feature extraction is performed on the image to be identified at the same time, to obtain channel image features of each channel image and image features to be identified of the image to be identified. In some possible implementations, the features of the image to be identified may be features obtained by extracting features from one image to be identified, or may be features obtained by extracting features from multiple images to be identified, including the following two cases:

[0041] Case 1: Perform feature extraction on the image to be identified to obtain the image features of a single image to be identified.

[0042] For example, first, each channel image is input into a neural network of different branches for feature extraction, and the channel image features of each channel image are obtained to obtain a channel image feature set. Then, any branch neural network is used to extract features of the image to be identified, and the image features to be identified of a single image to be identified are obtained.

[0043] Case 2: A channel image and an image to be identified are input together into the same branch of the neural network for feature extraction (different channel images are input into different branches of the neural network), and the channel image features of each channel image and the image features to be identified output by each branch of the neural network are obtained, thereby obtaining a channel image feature set and multiple image features to be identified.

[0044] Step S104, based on the channel image feature set and the features of the image to be identified, determining the similarity between the object to be identified in the image to be identified and the target object to obtain a recognition result.

[0045] In some embodiments, based on the channel image feature set and the features of the image to be identified, the similarity between the image to be identified and the target image is determined, thereby determining the similarity between the object to be identified and the target object in the image to be identified. In a specific example, if the pedestrian in the target image is re-identified, that is, the target object is the target pedestrian, the similarity between the pedestrian in the image to be identified and the target pedestrian is determined by determining the similarity between the target image and the image to be identified. If the similarity is greater than or equal to the similarity threshold, the recognition result is that the pedestrian in the image to be identified is the target pedestrian. If the similarity is less than the similarity threshold, the recognition result is that the pedestrian in the image to be identified is not the target pedestrian.

[0046] In an embodiment of the present application, first, by determining the number of color channels of the image to be identified, the target image is decomposed into multiple channel images with the same number of channels as the image to be identified. In this way, the inter-modal differences caused by channel inconsistency between images can be significantly reduced; then, the image features of the multiple channel images are matched with the features of the image to be identified to determine the similarity between the object to be identified and the target object in the image to be identified, so as to identify whether the object to be identified is the target object; in this way, the accuracy of cross-modal target re-identification can be improved.

[0047] In some embodiments, the image features of multiple channel images are fused and the similarity between the fused features is calculated to determine the similarity between the image to be identified and the target image. That is, step 104 can be implemented by the following steps:

[0048] Step S141, fusing the channel image features in the channel image feature set to obtain a first fused feature.

[0049] In some embodiments, multiple channel image features are spliced ​​in length to obtain a one-dimensional vector, and the one-dimensional vector is used as the first fusion feature. For example, the target image is decoupled into three single-channel images of R, G, and B, and features of the three single-channel images of R, G, and B are extracted to obtain R channel image features, G channel image features, and B channel image features. These three features are spliced ​​in length in the order of R channel image features and G channel image features, and then in the order of B channel image features, to obtain a one-dimensional vector, that is, to obtain the first fusion feature. In other embodiments, these three features can also be spliced ​​in length in the order of R channel image features and B channel image features, and then in the order of G channel image features, to obtain a one-dimensional vector, that is, to obtain the first fusion feature; these three features can also be spliced ​​in length in the order of B channel image features and G channel image features, and then in the order of R channel image features, to obtain a one-dimensional vector, that is, to obtain the first fusion feature.

[0050] Step S142, based on the first fusion feature and the feature of the image to be identified, determine the similarity between the object to be identified and the target object to obtain a recognition result.

[0051] In some embodiments, if the image feature to be identified is a feature of a single image to be identified, then the similarity between the first fused feature and the image feature to be identified is calculated to determine the similarity between the object to be identified and the target object. If the image feature to be identified is a feature of multiple images to be identified, then the features of the multiple images to be identified are first fused to obtain a second fused feature; then, the similarity between the object to be identified and the target object is determined by calculating the similarity between the first fused feature and the second fused feature; thereby determining whether the object to be identified is the same object as the target object.

[0052] In an embodiment of the present application, by fusing image features of multiple channel images and calculating the similarity between the fused features to determine the similarity between the image to be identified and the target image, the accuracy of cross-modal target re-identification is further improved.

[0053] In some embodiments, when the image to be identified is a color single-channel image, decoupling the target image into the color single-channel image, that is, step S102 can be implemented by the following steps:

[0054] Step S121, determining the number of color channels of the target image.

[0055] Here, if the image to be identified is a single-channel color image, such as a near-infrared image, then the number of color channels of the target image is first determined; for example, the target image is an RGB image collected in a natural environment, and the color channels are R channel, G channel, and B channel, so the number of color channels is 3.

[0056] Step S122: based on the number of color channels of the target image, decompose the target image into channel images having a single color channel to obtain the channel image set.

[0057] Here, after the number of color channels of the target image is determined, the target image is decoupled based on the number of color channels, and the target image is decoupled into single-channel images of the number of color channels, thereby obtaining a channel image set.

[0058] In an embodiment of the present application, when the image to be identified is a single-channel image, a single-channel image is obtained by decoupling the R, G, and B channels of the target image, and a three-branch convolutional neural network is used to match the single-channel image to be identified, which significantly reduces the difference between modalities and improves the accuracy of cross-modal target re-identification.

[0059] In some embodiments, for each channel image in the channel image set, a channel image is combined with an image to be identified and input into a branch of a neural network for feature extraction, thereby obtaining a channel image feature set and image features to be identified of multiple images to be identified, that is, step S103 can be implemented by the following steps, see Figure 2 , Figure 2 Another implementation flow diagram of the target re-identification method provided in the embodiment of the present application, combined with Figure 1 and 2 The following instructions are given:

[0060] Step S201, determining the number N of channel images in the channel image set.

[0061] Here, N is an integer greater than 0. If the image to be identified is a single-channel image, the number of channel images in the channel set is the same as the number of color channels of the target image. If the image to be identified is a non-single-channel image, the number of channel images in the channel set is less than the number of color channels of the target image.

[0062] Step S202, obtaining a neural network with N branches.

[0063] Here, a trained neural network with N branches is obtained. Taking the image to be recognized as a near-infrared image and the target image as an RGB image as an example, the RGB image is decomposed into three single-channel images of R, G, and B, and a trained convolutional neural network with three branches is obtained.

[0064] Step S203, inputting the ith channel image and the image to be identified into the ith branch of the neural network for feature extraction to obtain features of the ith channel image and features of the ith image to be identified.

[0065] Here, i is an integer greater than 0 and less than or equal to N. A channel image is combined with an image to be identified and input into a branch of a neural network for feature extraction to obtain the features of the channel image and the features of the image to be identified output by the branch. For example, the first channel image and the image to be identified are input into the first branch of the neural network for feature extraction to obtain the features of the first channel image and the features of the image to be identified of the image to be identified, and the second channel image and the image to be identified are input into the second branch of the neural network to obtain the features of the second channel image and the features of the image to be identified of the image to be identified.

[0066] In some possible implementations, the training process of the neural network is as follows:

[0067] First, a sample label of the target object in the target image and an object label of the object to be identified in the image to be identified are obtained.

[0068] Here, the sample label of the target object is a label that can uniquely identify the target object, such as the identity document (ID) of the target object. The object label of the object to be identified is a label that can uniquely identify the object to be identified, such as the identity document (ID) of the object to be identified.

[0069] Secondly, the channel images in the channel image set are input into the neural network to be trained for feature extraction to obtain a candidate channel image feature set.

[0070] Here, the neural network to be trained is a convolutional neural network to be trained with multiple branches, wherein the number of branches is the same as the number of channel images; each channel image is input into each branch of the neural network to be trained for feature extraction, and the channel image features output by each branch are obtained to obtain a candidate channel image feature set.

[0071] Again, based on the sample labels, the classification loss of the candidate channel image features is determined.

[0072] Here, for each candidate channel image feature in the candidate channel image feature set, the classification loss of the candidate channel image feature is obtained by determining the difference between the sample label and each candidate channel image feature. For example, an additional fully connected layer is used to realize the classification of the feature.

[0073] Thirdly, based on the first fused feature and the second fused feature, the candidate similarity between the target image and the image to be identified is determined.

[0074] Here, the cosine distance or the Euclidean distance between the first fused feature and the second fused feature is determined to determine the candidate similarity between the target image and the image to be identified.

[0075] Again, based on the sample label and the object label of the object to be identified, the triplet loss of the candidate similarity is determined.

[0076] Here, the triplet loss of candidate similarity is determined by determining the difference between the sample label and the object label. In an embodiment of the present application, the triplet loss is used to distinguish between the object to be identified and the target object. For the set triplet (reference sample, positive sample, negative sample, Anchor, Positive, Negative) (where the reference sample and the positive sample are different samples of the same type, and the positive sample and the negative sample are different samples), the triplet loss attempts to learn a feature space so that the reference samples of the same category in the space are closer to the positive samples, and the reference samples of different categories are farther from the negative samples. In this way, for a given target image and an image to be identified, it can be determined whether the objects in the two images belong to the same category of labels.

[0077] Finally, the classification loss and the triplet loss are used to adjust the network parameters of the neural network to be trained, so that the classification loss of the candidate channel image features and the triplet loss of the candidate similarity output by the adjusted neural network to be trained meet the convergence conditions, thereby obtaining the neural network.

[0078] Here, by determining the gradient of the triplet loss and the gradient of the classification loss, the gradient is fed back to the input end of the neural network to be trained to adjust the network parameters of the neural network to be trained, for example, adjusting the weight value and offset and other parameters of the neural network to be trained, so as to obtain a trained neural network.

[0079] In an embodiment of the present application, the neural network to be trained includes two parts: the first part is used to classify the channel image, and the second part is used to calculate the similarity between the target image and the image to be identified. In the first part, after the channel image is input into the neural network, the classification loss is used to adjust the network parameters used for classification for the output candidate channel image features; in the second part, the first fusion feature and the second fusion feature are input into the neural network, the candidate similarity between the target image and the image to be identified is output, and the network parameters for similarity calculation are adjusted using the triplet loss; in this way, the network parameters for implementing the corresponding functions are adjusted in a targeted manner based on different loss functions, so that the recognition results of the trained neural network can be more accurate and more robust.

[0080] Step S204, inputting the N channel images and the N to-be-recognized images into the N branches of the neural network in a one-to-one correspondence to perform feature extraction, thereby obtaining N channel image features and N to-be-recognized image features.

[0081] Here, if there are N channel images in the channel image set, each of the N channel images is matched with an image to be identified and input into a branch of the neural network for feature extraction, that is, the input in one branch is a channel image and an image to be identified, so as to obtain N channel image features and N image features to be identified output by N branches of the neural network. In a specific example, taking the image to be identified as a near-infrared image and the target image as an RGB image as an example, the RGB image is decoupled into three single-channel images of R, G, and B, and the neural network is a convolutional neural network including three branches, and the R channel image and the image to be identified are input into the first branch of the neural network for feature extraction, and the R channel image features and the first image to be identified features are obtained; the G channel image and the image to be identified are input into the second branch of the neural network for feature extraction, and the G channel image features and the second image to be identified features are obtained; the B channel image and the image to be identified are input into the third branch of the neural network for feature extraction, and the B channel image features and the third image to be identified features are obtained.

[0082] After obtaining N channel image features and N to-be-recognized image features through the above steps S201 to S204, the similarity between the to-be-recognized object in the to-be-recognized image and the target object can be determined through the following steps:

[0083] In the first step, the N channel image features in the channel image feature set are concatenated in length to obtain the first fusion feature.

[0084] In some possible implementations, these N channel image features are spliced ​​in length in the order from 1 to N to obtain a one-dimensional vector, that is, to obtain the first fusion feature. Alternatively, they can be spliced ​​in length in any order from 1 to N to obtain the first fusion feature. In a specific example, taking the image to be identified as a near-infrared image and the target image as a visible light image as an example, the target image is decoupled into three single-channel images H, S, and V. The three single-channel images H, S, and V are extracted one by one through the three branches of the neural network to obtain H channel image features, S channel image features, and V channel image features. These three features are spliced ​​in length in the order of H channel image features and S channel image features, and then spliced ​​in the order of V channel image features to obtain a one-dimensional vector, that is, to obtain the first fusion feature. In other embodiments, the three features may be spliced ​​in length according to the order of H channel image features and V channel image features, and then spliced ​​in the order of S channel image features to obtain a one-dimensional vector, that is, to obtain the first fusion feature; the three features may be spliced ​​in length according to the order of V channel image features and S channel image features, and then spliced ​​in the order of H channel image features to obtain a one-dimensional vector, that is, to obtain the first fusion feature.

[0085] In the second step, the N image features to be identified are concatenated in length to obtain the second fusion feature.

[0086] Here, the N image features to be identified output by the N branches of the neural network can be spliced ​​in length in any order from 1 to N to obtain a one-dimensional vector, that is, the second fusion feature. For example, the target image is decoupled into three single-channel images H, S, and V. The three single-channel images H, S, and V are respectively combined with an image to be identified and input into the three branches of the neural network to obtain three image features to be identified. The three image features to be identified are spliced ​​in length to obtain the second fusion feature.

[0087] The third step is to determine the similarity between the object to be identified and the target object based on the first fusion feature and the second fusion feature to obtain a recognition result.

[0088] Here, by calculating the similarity between the first fusion feature and the second fusion feature, it is determined whether the object to be identified is the same object as the target object.

[0089] The first to third steps mentioned above provide a method for determining the similarity between the object to be identified and the target object. In this method, the channel image features extracted by the multi-branch neural network and the image features to be identified are fused respectively, and the fused first fusion features and the second fusion features are used for matching, thereby improving the matching accuracy.

[0090] In some embodiments, after fusing N channel image features and N to-be-recognized image features to obtain a first fused feature and a second fused feature, the similarity between the object in the to-be-recognized image and the target object can be calculated by the following process:

[0091] First, based on the first fused feature and the second fused feature, a feature distance is determined to obtain a similarity between the first fused feature and the second fused feature.

[0092] Here, the feature distance may be a cosine distance or a Euclidean distance, and the similarity between the first fused feature and the second fused feature is determined by determining the cosine distance or the Euclidean distance between the first fused feature and the second fused feature.

[0093] Then, based on the similarity between the first fused feature and the second fused feature, the sorting position of the image to be identified in the test image library is determined.

[0094] Here, first, the similarity between the second fusion feature and the first fusion feature of each different image to be identified in the test image library is obtained; then, multiple candidate similarities greater than or equal to the similarity threshold are selected from the obtained multiple similarities; finally, the multiple candidate similarities are sorted in descending order, and the sorting result is determined as the sorting position of the image to be identified in the test image library. The test image library includes a large number of different images to be identified. For example, the target object is a pedestrian, and the test image library can be multiple frames of images captured from a road monitoring video.

[0095] Then, a matching relationship between the sorting position and a preset condition is determined.

[0096] Here, the preset condition is that when the similarity is greater than or equal to the preset threshold, the sequence number corresponding to the ranking position is the first preset percentage (for example, 10%) of the total number. The matching relationship between the ranking position and the preset condition includes: the ranking position meets the preset condition, and the ranking position does not meet the preset condition.

[0097] Finally, based on the matching relationship between the sorting position and the preset condition, the recognition result is determined.

[0098] Here, based on the matching relationship between the sorting position and the preset condition, the recognition result is determined, including the following two cases:

[0099] First, if the sorting position satisfies a preset condition, it is determined that the recognition result is that the object to be recognized and the target object are the same object.

[0100] Second, if the sorting position does not meet the preset conditions, it is determined that the recognition result is that the object to be recognized and the target object are not the same object. If the sorting position does not meet the preset conditions, it means that the similarity between the first fusion feature and the second fusion feature is small, which means that the possibility that the object to be recognized and the target object are the same object is low. In this way, by fusing the features extracted from multiple channel images through a convolutional neural network and then matching them with the features of the image to be recognized, the accuracy of cross-modal target re-recognition is further improved.

[0101] Below, an exemplary application of an embodiment of the present application in an actual application scenario will be described, taking the re-identification of target pedestrians in visible light target images and pedestrians to be identified in infrared images, and cross-modal target re-identification based on a channel decoupling network as an example for explanation.

[0102] Figure 3 Another implementation flow diagram of the target re-identification method provided in other embodiments of the present application, combined with Figure 3 The steps shown are explained below:

[0103] Step S301a, input the VIS image into the convolutional neural network 302.

[0104] Here, the VIS image corresponds to the target image in the above-described embodiment.

[0105] Step S301b, input the IR image into the convolutional neural network 302.

[0106] In some possible implementations, the IR image corresponds to the image to be identified in the above embodiment. Step S301a and step S301b may be performed simultaneously, that is, the VIS image and the IR image are simultaneously input into the trained convolutional neural network 302 for feature extraction.

[0107] Step S303a, extracting features from the VIS image to obtain VIS image features.

[0108] Step S303b, extracting features from the IR image to obtain IR image features.

[0109] In some possible implementations, step S303a and step S303b may be performed simultaneously, that is, in the trained convolutional neural network 302, feature extraction is performed on the VIS image and the IR image at the same time, thereby obtaining VIS image features and IR image features.

[0110] Step S304, performing similarity measurement on the VIS image features and the IR image features to obtain a recognition result.

[0111] In some possible implementations, the similarity of the VIS image features and the IR image features is measured. If the similarity is greater than or equal to a preset similarity threshold, it means that the similarity of the IR image and the VIS image features is high, that is, it means that the probability that the pedestrian in the IR image and the pedestrian in the VIS image are the same person is high, and the IR image is used as the image in which the pedestrian is identified, so that the recognition result is that the IR image and the VIS image are similar images. If the similarity is less than the preset similarity threshold, it means that the similarity of the IR image and the VIS image features is low, that is, it means that the probability that the pedestrian in the IR image and the pedestrian in the VIS image are the same person is low, and the IR image is used as the image in which the pedestrian is identified, so that the recognition result is that the IR image and the VIS image are dissimilar images.

[0112] The embodiment of the present application provides a method for target re-identification. First, the pedestrian graphic image under natural light is decoupled into three single-channel images of R, G, and B (or other color spaces, such as HSV and YCbCr, etc.); then, the three decomposed single-channel images are respectively sent to a three-branch neural network with a near-infrared image for end-to-end training; finally, the trained neural network is used for cross-modal target re-identification. In the embodiment of the present application, the visible light image is decoupled into three single channels of R, G, and B for illustration.

[0113] By decoupling the RGB image under visible light, three single-channel images are obtained, which are matched with the IR image respectively, and the channel alignment of the RGB image and the IR image is achieved; in this way, the problem of huge image differences caused by the inconsistency of the channels of near-infrared images and visible light images during training is solved, so that better training results can be obtained. On this basis, by fusing the features of the three single-branch networks, the accuracy of cross-modal target re-identification can be significantly improved.

[0114] Figure 4 A schematic diagram of another implementation flow of the target re-identification method provided in the embodiment of the present application, combined with Figure 4 The steps shown are explained below:

[0115] Step S401, obtaining input VIS image and IR image.

[0116] Step S402: performing channel decoupling on the VIS image.

[0117] Here, the input VIS image is decoupled into three single-channel images of R, G, and B.

[0118] Step S403 , extracting features from the decoupled single-channel VIS image and IR image respectively, and fusing the extracted features of multiple single-channel VIS images and IR images respectively.

[0119] Step S404: perform similarity measurement based on the fused VIS image features and IR image features to obtain a recognition result.

[0120] Here, the obtained R channel image and IR image are sent to the neural network for feature extraction; at the same time, the G channel image and IR image and the B channel image and IR image are sent to the other two convolutional neural networks for feature extraction, and three one-dimensional vectors are obtained. In the process of training the neural network, according to the pedestrian label (corresponding to the sample label in the above embodiment), the classification loss is calculated for each of the three feature vectors, and the gradient of the classification loss is fed back to the input end of the neural network to be trained to optimize the network parameters of the neural network, thereby obtaining a trained neural network.

[0121] In the embodiment of the present application, the cross-modal target re-identification algorithm is improved, mainly by decoupling the VIS image at the channel level and training the network with the IR combination to reduce the modality difference between the multi-channel VIS image and the single-channel IR image. Since the matching training between single-channel images is more conducive to network clustering, the network using the three-branch fusion feature has a high accuracy.

[0122] Figure 5 A schematic diagram of the implementation framework of the target re-identification method provided in the embodiment of the present application, combined with Figure 5 The following instructions are given:

[0123] The VIS image input module 501 is used to obtain an input VIS image.

[0124] The channel decoupling module 502 decouples the VIS image into three single-channel images of R, G, and B, and obtains an R channel image 521 , a G channel image 522 , and a B channel image 523 .

[0125] Here, after decoupling the VIS image, the R channel image and the IR image 524 are sent to the first branch 503 of the neural network, the G channel image and the IR image are sent to the second branch 504 of the neural network, and the B channel image and the IR image are sent to the third branch 505 of the neural network.

[0126] The first branch 503 is used to extract features from the R channel image and the IR image to obtain R channel image features 531 and IR image features 532 .

[0127] The second branch 504 is used to extract features from the G channel image and the IR image to obtain G channel image features 541 and IR image features 542 .

[0128] The third branch 505 is used to extract features from the B channel image and the IR image to obtain B channel image features 551 and IR image features 552 .

[0129] The first feature fusion module 506 is used to perform feature fusion on the R channel image feature 531 , the G channel image feature 541 and the B channel image feature 551 to obtain a first fused feature.

[0130] Here, the R channel image feature 531, the G channel image feature 541 and the B channel image feature 551 are connected in length to obtain a one-dimensional vector, namely, the first fusion feature.

[0131] The second feature fusion module 507 is used to perform feature fusion on the IR image feature 532 , the IR image feature 542 , and the IR image feature 552 to obtain a second fused feature.

[0132] Here, the IR image feature 532, the IR image feature 542, and the IR image feature 552 are connected in length to obtain another one-dimensional vector, namely, the second fusion feature.

[0133] The similarity measurement module 508 is used to determine the similarity between the first fusion feature and the second fusion feature using a neural network, and sort the IR images to be retrieved based on the similarity to obtain the re-identification result. In the process of training the neural network, according to the pedestrian label and the label of the pedestrian to be identified in the IR image as the training set, the triple loss function is calculated for the obtained fusion feature, and the gradient of the triple loss function is returned to the input end of the neural network to be trained to optimize the network parameters of the neural network to be trained, thereby obtaining a trained neural network.

[0134] The embodiment of the present application significantly reduces the difference between modalities and improves the accuracy of cross-modal target re-identification by decoupling the R, G, and B channels of the VIS image and using a three-branch convolutional neural network to match them with the IR image respectively; on this basis, the accuracy of cross-modal target re-identification is further improved by fusing the features extracted from the R, G, and B channels through the convolutional neural network and then matching them with the IR image features.

[0135] The present application provides a target re-identification device. Figure 6 Schematic diagram of the structure of the target re-identification device according to the embodiment of the present application. Figure 6 As shown, the device 600 includes:

[0136] A first determination module 601 is used to determine the number of color channels of the acquired image to be identified;

[0137] A first decomposition module 602 is used to decompose a target image including a target object into a plurality of channel images having the same number of color channels as the image to be identified, to obtain a channel image set;

[0138] A first extraction module 603 is used to extract features from the channel images in the channel image set and the image to be identified, and obtain a channel image feature set and features of the image to be identified respectively;

[0139] The second determination module 604 is used to determine the similarity between the object to be identified in the image to be identified and the target object based on the channel image feature set and the image to be identified features, so as to obtain a recognition result.

[0140] In the above device, the second determining module 604 includes:

[0141] A first fusion submodule, used for fusing the channel image features in the channel image feature set to obtain a first fusion feature;

[0142] The first determination submodule is used to determine the similarity between the object to be identified and the target object based on the first fusion feature and the feature of the image to be identified, so as to obtain a recognition result.

[0143] In the above device, if the image to be identified is a color single-channel image, the first decomposition module 602 includes:

[0144] A second determination submodule, used to determine the number of color channels of the target image;

[0145] The first decomposition submodule is used to decompose the target image into channel images with a single color channel based on the number of color channels of the target image to obtain the channel image set.

[0146] In the above device, the first extraction module 603 includes:

[0147] A third determination submodule is used to determine the number N of channel images in the channel image set; wherein N is an integer greater than 0;

[0148] A first acquisition submodule is used to acquire a neural network having N branches;

[0149] A first input submodule, used for inputting the i-th channel image and the image to be identified into the i-th branch of the neural network for feature extraction, to obtain the i-th channel image feature and the i-th image to be identified feature; wherein i is an integer greater than 0 and less than or equal to N;

[0150] The second input submodule is used to input the N channel images and the N to-be-recognized images into the N branches of the neural network in a one-to-one correspondence to perform feature extraction, so as to obtain N channel image features and N to-be-recognized image features.

[0151] In the above device, the first fusion submodule is further used to: splice the N channel image features in the channel image feature set in length to obtain the first fusion feature;

[0152] Correspondingly, the first determining submodule includes:

[0153] A first splicing unit, used for splicing the N image features to be identified in length to obtain the second fusion feature;

[0154] The first determining unit is used to determine the similarity between the object to be identified and the target object based on the first fusion feature and the second fusion feature to obtain a recognition result.

[0155] In the above device, the first determining unit includes:

[0156] a first input subunit, configured to determine a feature distance based on the first fused feature and the second fused feature, so as to obtain a similarity between the first fused feature and the second fused feature;

[0157] A first determining subunit, configured to determine a sorting position of the image to be identified in a library of images to be identified based on a similarity between the first fusion feature and the second fusion feature;

[0158] A second determining subunit, used to determine a matching relationship between the sorting position and a preset condition;

[0159] The third determining subunit is used to determine the recognition result based on the matching relationship.

[0160] In the above device, the device further includes a first training module, which is used to train the neural network to be trained; the first training module includes:

[0161] A second acquisition submodule is used to acquire a sample label of the target object in the target image and an object label of the object to be identified in the image to be identified;

[0162] A first extraction submodule is used to input the channel images in the channel image set into the neural network to be trained for feature extraction to obtain a candidate channel image feature set;

[0163] A first loss submodule, used to determine the classification loss of the candidate channel image feature based on the sample label;

[0164] A third input submodule, configured to determine a candidate similarity between the target image and the image to be identified based on the first fusion feature and the second fusion feature;

[0165] A second loss submodule, configured to determine a triplet loss of the candidate similarity based on the sample label and the object label;

[0166] The first adjustment submodule is used to use the classification loss and the triplet loss to adjust the network parameters of the neural network to be trained, so that the classification loss of the candidate channel image features and the triplet loss of the candidate similarity output by the adjusted neural network to be trained meet the convergence conditions to obtain the neural network.

[0167] It should be noted that the description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0168] It should be noted that in the embodiments of the present application, if the above-mentioned target re-identification method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including a number of instructions to enable a target re-identification device (which can be a terminal, server, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a sports hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0169] Correspondingly, an embodiment of the present application further provides a computer program product, which includes computer executable instructions. After the computer executable instructions are executed, the steps in the target re-identification method provided in the embodiment of the present application can be implemented.

[0170] An embodiment of the present application further provides a computer-readable storage medium, on which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the steps of the target re-identification method provided in the above embodiment are implemented.

[0171] The present application embodiment provides a target re-identification device, Figure 7 Schematic diagram of the structure of the target re-identification device according to the embodiment of the present application. Figure 7 As shown, the target re-identification device 700 includes: a processor 701, at least one communication bus, a communication interface 702, at least one external communication interface and a memory 703. The communication interface 702 is configured to realize the connection and communication between these components. The communication interface 702 may include a display screen, and the external communication interface may include a standard wired interface and a wireless interface. The processor 701 is configured to execute the image processing program in the memory to implement the steps of the target re-identification method provided in the above embodiment.

[0172] The description of the above target re-identification device, target re-identification equipment and storage medium embodiments is similar to the description of the above method embodiments, and has similar technical descriptions and beneficial effects as the corresponding method embodiments. Due to space limitations, the above method embodiments can be used as examples, so they will not be repeated here. For technical details not disclosed in the target re-identification device, target re-identification device and storage medium embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0173] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0174] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0175] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0176] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0177] In addition, all functional units in the embodiments of the present application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the above integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional units. A person of ordinary skill in the art can understand that all or part of the steps of the above method embodiments can be completed by hardware related to program instructions, and the above program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above method embodiments are executed; and the above storage medium includes: mobile storage devices, read-only memory (ROM), disks or optical disks, etc. Various media that can store program codes.

[0178] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application can be embodied in the form of a software product in essence or in other words, the part that contributes to the prior art can be stored in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks. The above is only a specific implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the present application, and they should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A target re-identification method, characterized in that: The method comprises: Determine the number of color channels of the acquired image to be identified; Decomposing a target image including a target object into a plurality of channel images having the same number of color channels as the image to be identified, to obtain a channel image set, including: Determining the number of color channels of the target image; Based on the number of color channels of the target image, decomposing the target image into channel images having a single color channel to obtain the channel image set; Extracting features of the channel images in the channel image set and the image to be identified, and obtaining a channel image feature set and features of the image to be identified respectively; Based on the channel image feature set and the features of the image to be identified, determining the similarity between the object to be identified in the image to be identified and the target object to obtain a recognition result, including: The channel image features in the channel image feature set are fused to obtain a first fused feature, including: splicing N channel image features in the channel image feature set in length to obtain the first fusion feature; Correspondingly, determining the similarity between the object to be identified and the target object based on the first fusion feature and the feature of the image to be identified to obtain a recognition result includes: The N image features to be identified are concatenated in length to obtain a second fusion feature; Determining the similarity between the object to be identified and the target object based on the first fusion feature and the second fusion feature to obtain a recognition result includes: Determining a feature distance based on the first fused feature and the second fused feature to obtain a similarity between the first fused feature and the second fused feature; Determining a sorting position of the image to be identified in a library of images to be identified based on the similarity between the first fusion feature and the second fusion feature; Determine a matching relationship between the sorting position and a preset condition; and determine the recognition result based on the matching relationship; Based on the first fusion feature and the feature of the image to be identified, determining the similarity between the object to be identified and the target object to obtain a recognition result; The process of extracting features of the channel images in the channel image set and the image to be identified to obtain a channel image feature set and features of the image to be identified respectively includes: Determine the number N of channel images in the channel image set; wherein N is an integer greater than 0; obtain a neural network having N branches; Inputting the i-th channel image and the image to be identified into the i-th branch of the neural network for feature extraction to obtain the i-th channel image feature and the i-th image to be identified feature; wherein i is an integer greater than 0 and less than or equal to N; The N channel images and the N to-be-recognized images are input into the N branches of the neural network in a one-to-one correspondence to perform feature extraction, thereby obtaining N channel image features and N to-be-recognized image features.

2. The method according to claim 1, characterized in that The training process of the neural network includes: Acquire a sample label of the target object in the target image and an object label of the object to be identified in the image to be identified; Inputting the channel images in the channel image set into the neural network to be trained for feature extraction to obtain a candidate channel image feature set; Based on the sample labels, determining the classification loss of the candidate channel image features; Determining a candidate similarity between the target image and the image to be identified based on the first fusion feature and the second fusion feature; Determining a triplet loss of the candidate similarity based on the sample label and the object label; The classification loss and the triplet loss are used to adjust the network parameters of the neural network to be trained, so that the classification loss of the candidate channel image features and the triplet loss of the candidate similarities output by the adjusted neural network to be trained meet the convergence conditions, thereby obtaining the neural network.

3. A target re-identification device, using the target re-identification method according to claim 1, characterized in that: The device comprises: A first determination module is used to determine the number of color channels of the acquired image to be identified; A first decomposition module is used to decompose a target image including a target object into a plurality of channel images having the same number of color channels as the number of color channels of the image to be identified, to obtain a channel image set; A first extraction module is used to extract features from the channel images in the channel image set and the image to be identified, and obtain a channel image feature set and features of the image to be identified respectively; The second determination module is used to determine the similarity between the object to be identified in the image to be identified and the target object based on the channel image feature set and the image feature to be identified, so as to obtain a recognition result.

4. A target re-identification device, characterized in that: The device comprises: A memory and a processor, wherein the memory stores computer executable instructions, and when the processor runs the computer executable instructions on the memory, the steps of the method described in any one of claims 1 to 2 can be implemented.

5. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which, after being executed, can implement the steps of the method described in any one of claims 1 to 2.