Visible light-infrared ray person re-identification method based on patch mixed cross-mode

Abstract

The invention relates to a patch mixed cross-modal-based visible light-infrared ray person re-identification method. The method comprises the following steps: acquiring a visible light image and an infrared image of a pedestrian; processing the visible light image and the infrared image based on a pre-established patch mixing cross-modal framework, and generating an image patch mixing image based on a preset patch mixing strategy; obtaining an output distribution result according to the visible light image, the infrared image and the image patch mixed image, and calculating a first alignment degree of the image patch mixed image and the visible light image and a second alignment degree of the image patch mixed image and the infrared image based on the output distribution result, and obtaining a personnel re-identification result according to the first alignment degree and the second alignment degree. Therefore, the problems that the modes are unbalanced and the pedestrian re-identification effect is poor due to the fact that the heterogeneous modes are difficult to align and the semantic corresponding relation between the two modes cannot be well learned in the prior art are solved, and the robustness and the identification accuracy of the pedestrian re-identification model are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a visible light-infrared person re-identification method based on patch mixing cross-modality. Background Art

[0002] Visible-infrared person re-identification (VI-ReID) aims to match target persons between RGB visible light cameras and low-light infrared (IR) cameras, retrieving images of the same person from different modalities. This task has gained increasing research interest due to its significant value in practical 24-hour surveillance systems. The challenge lies in the significant modality differences, where different wavelengths can lead to significantly different visual appearances (e.g., color, texture).

[0003] Related technologies generally use GANs, grayscale scaling or hybrid strategies to generate intermediate images to alleviate modality differences. There are three main types of methods: (1) representation learning-based methods, in which the network is designed to learn discriminative features in the modality shared space; (2) metric learning-based methods, in which the loss function is designed to bridge the modality gap; (3) modality conversion-based methods, in which modalities are converted to each other to achieve style consistency.

[0004] However, related technologies do not consider the correspondence between infrared images and RGB images, making it difficult to align heterogeneous modalities. They may also introduce additional noise and fail to learn the semantic correspondence between the two modes. This results in poor person re-identification performance and urgently needs to be addressed. Summary of the Invention

[0005] The present invention provides a visible light-infrared person re-identification method based on patch hybrid cross-modality to solve the problems that related technologies have difficulty in aligning heterogeneous modalities and cannot well learn the semantic correspondence between the two modalities, resulting in modal imbalance and poor pedestrian re-identification effect, thereby improving the robustness and recognition accuracy of the pedestrian re-identification model.

[0006] The first aspect of the present invention provides a visible light-infrared person re-identification method based on patch mixing cross-modality, comprising the following steps: obtaining a visible light image and an infrared image of a pedestrian; processing the visible light image and the infrared image based on a pre-established patch mixing cross-modality framework, and generating an image patch mixed image according to the processing results based on a preset patch mixing strategy; obtaining an output distribution result according to the visible light image, the infrared image and the image patch mixed image, and calculating a first alignment degree between the image patch mixed image and the visible light image and a second alignment degree between the image patch mixed image and the infrared image based on the output distribution result, and obtaining a person re-identification result according to the first alignment degree and the second alignment degree.

[0007] Further, in some embodiments, processing the visible light image and the infrared image based on the pre-established patch mixing cross-modal framework, and generating an image patch mixing image according to the processing result based on a preset patch mixing strategy, includes: respectively extracting features from the visible light image and the infrared image to obtain the global features of the visible light image, the global features of the infrared image, the local features of the visible light image, and the local features of the infrared image; based on a preset alignment strategy, obtaining the aligned visible light image features and the aligned infrared image features according to the global features of the visible light image, the global features of the infrared image, the local features of the visible light image, and the local features of the infrared image; based on the pre-established patch mixing cross-modal framework, splicing the aligned visible light image features and the aligned infrared image features to obtain the image patch mixing image.

[0008] Further, in some embodiments, the respectively extracting features from the visible light image and the infrared image to obtain the global features of the visible light image, the global features of the infrared image, the local features of the visible light image, and the local features of the infrared image includes: based on a preset two-stream network, extracting features from the visible light image and the infrared image to obtain the global features of the visible light image and the global features of the infrared image; extracting the visible light image and the infrared image through horizontal stripes to obtain the local features of the visible light image of the pedestrian and the local features of the infrared image, wherein the identity of each local feature is classified through a cross-entropy loss function, and the similarity of the same-class local features is constrained through a sample-to-sample loss function.

[0009] Further, in some embodiments, the obtaining the aligned visible light image features and the aligned infrared image features according to the global features of the visible light image, the global features of the infrared image, the local features of the visible light image, and the local features of the infrared image based on a preset alignment strategy includes: constructing a memory bank based on the visible light image and the infrared image, wherein the memory bank is used to maintain the feature center data of each pedestrian identity; calculating the loss between the center of the visible light image features and the center of the infrared image features according to the feature center data of each pedestrian identity; based on the loss, aligning the global features of the visible light image and the global features of the infrared image, and at the same time aligning the local features of the visible light image and the local features of the infrared image to obtain the aligned visible light image features and the aligned infrared image features.

[0010] Further, in some embodiments, obtaining the output distribution result according to the visible light image, the infrared image, and the image patch mixed image includes: determining the visible light image, the infrared image, and the image patch mixed image of each pedestrian; and inputting the visible light image, the infrared image, and the image patch mixed image of each pedestrian into a preset patch mixing cross-modal algorithm to obtain the output distribution result.

[0011] According to the visible light-infrared person re-identification method based on patch mixing cross-modal provided by the embodiments of the present invention, the visible light image and the infrared image of the same user are segmented into patches, and a patch mixed image is generated according to the patch mixing strategy for learning, which can reflect the semantic correspondence relationship, and by minimizing the loss between each modality, a person re-identification result is obtained. It solves the problems in the related art that it is difficult to align heterogeneous modalities and it is impossible to learn the semantic correspondence relationship between the two modalities well, resulting in modality imbalance and poor person re-identification effect, and improves the robustness and recognition accuracy of the person re-identification model.

[0012] An embodiment of the second aspect of the present invention provides a visible light-infrared person re-identification device based on patch mixing cross-modal. The device includes: an acquisition module, configured to acquire a visible light image and an infrared image; a data processing module, configured to process the visible light and infrared images based on a patch mixing cross-modal framework, generate an image patch mixed image through a preset patch mixing strategy, and calculate the output distribution of the visible light image and the image patch mixed image; and an output module, configured to calculate the alignment degree between the image patch mixed image and the infrared image based on the output distribution result and output a person re-identification result.

[0013] Further, in some embodiments, the data processing module is specifically configured to: respectively perform feature extraction on the visible light image and the infrared image to obtain the global feature of the visible light image, the global feature of the infrared image, the local feature of the visible light image, and the local feature of the infrared image; based on a preset alignment strategy, obtain the aligned visible light image feature and the aligned infrared image feature according to the global feature of the visible light image, the global feature of the infrared image, the local feature of the visible light image, and the local feature of the infrared image; and based on a pre-established patch mixing cross-modal framework, splice the aligned visible light image feature and the aligned infrared image feature to obtain the image patch mixed image.

[0014] According to the visible light-infrared person re-identification device based on patch mixing cross-modal provided by the embodiments of the present invention, the visible light image and the infrared image of the same user are segmented into patches, and the patch mixing image is obtained according to the patch mixing strategy for learning, which can reflect the semantic correspondence relationship, and by minimizing the loss between modalities, the pedestrian re-identification result is obtained. It solves the problems in the related art that it is difficult to align heterogeneous modalities and it is impossible to learn the semantic correspondence relationship between the two modalities well, resulting in modality imbalance and poor pedestrian re-identification effect, and improves the robustness and recognition accuracy of the pedestrian re-identification model.

[0015] The third aspect of the embodiments of the present invention provides an electronic device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor executes the program to implement the above-mentioned visible light-infrared person re-identification method based on patch mixing cross-modal.

[0016] The fourth aspect of the embodiments of the present invention provides a computer-readable storage medium, on which a computer program is stored, and the program is executed by a processor to be used to implement the visible light-infrared person re-identification method based on patch mixing cross-modal as described in the above embodiments.

[0017] The fifth aspect of the embodiments of the present invention provides a computer program product, including a computer program, and the computer program is executed to be used to implement the visible light-infrared person re-identification method based on patch mixing cross-modal as described in any one of the above.

[0018] Therefore, the present invention has the following beneficial effects:

[0019] (1) The present invention proposes a novel patch mixing cross-modal learning framework for the visible light-infrared person re-identification (VI-ReID) task, which effectively encourages the model to process RGB and IR images in the same way and alleviates the problem of modality imbalance.

[0020] (2) The present invention proposes a center-to-center loss to keep the identity centers of each modality consistent. At the same time, a partial alignment loss is proposed to constrain the consistency of the partial and global prediction distributions, so as to obtain more discriminability. Considering the variations between and within modalities further enhances the learned model. Description of the Drawings

[0021] The above and / or additional aspects and advantages of the present invention will become obvious and easy to understand from the following description of the embodiments in conjunction with the drawings, where:

[0022] Figure 1 It is a flowchart of the visible light-infrared person re-identification method based on patch mixing cross-modal provided by the embodiments of the present invention;

[0023] Figure 2 Flowchart of a visible-light to infrared person re-identification method based on patch mixing cross-modal according to a specific embodiment of the present invention;

[0024] Figure 3 Block diagram of a visible-light to infrared person re-identification device based on patch mixing cross-modal according to an embodiment of the present invention;

[0025] Figure 4 Schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed implementation manners

[0026] The embodiments of the present invention will be described in detail below. Examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to explain the present invention and should not be construed as limiting the present invention.

[0027] The visible-light to infrared person re-identification based on patch mixing cross-modal according to an embodiment of the present invention will be described below with reference to the accompanying drawings. In view of the problems in the related technologies mentioned in the above background art that it is difficult to align heterogeneous modalities and it is impossible to learn the semantic correspondence relationship between the two modalities well, resulting in modality imbalance and poor pedestrian re-identification effect, the present invention provides a visible-light to infrared person re-identification method based on patch mixing cross-modal. The problems in the related technologies that it is difficult to align heterogeneous modalities and it is impossible to learn the semantic correspondence relationship between the two modalities well, resulting in modality imbalance and poor pedestrian re-identification effect, are solved, and the robustness and recognition accuracy of the pedestrian re-identification model are improved.

[0028] Specifically, Figure 1 Flowchart of a visible-light to infrared person re-identification method based on patch mixing cross-modal according to an embodiment of the present invention.

[0029] As Figure 1 shown, the visible-light to infrared person re-identification method based on patch mixing cross-modal includes the following steps:

[0030] In step S101, a visible-light image and an infrared image of a pedestrian are acquired;

[0031] Among them, the visible-light image is an image captured by an ordinary RGB camera, covering all colors visible to the human eye; the infrared image is an image captured by an infrared camera, mainly reflecting the thermal radiation information of an object.

[0032] In step S102, based on the pre-established patch mixing cross-modal framework, the visible light image and the infrared image are processed, and based on the preset patch mixing strategy, an image patch mixing image is generated according to the processing result.

[0033] Among them, in some embodiments, based on the pre-established patch mixing cross-modal framework, the visible light image and the infrared image are processed, and based on the preset patch mixing strategy, an image patch mixing image is generated according to the processing result, including: respectively extracting features from the visible light image and the infrared image to obtain the global features of the visible light image, the global features of the infrared image, the local features of the visible light image, and the local features of the infrared image; based on the preset alignment strategy, the aligned visible light image features and the aligned infrared image features are obtained according to the global features of the visible light image, the global features of the infrared image, the local features of the visible light image, and the local features of the infrared image; based on the pre-established patch mixing cross-modal framework, the aligned visible light image features and the aligned infrared image features are spliced to obtain an image patch mixing image.

[0034] Further, in some embodiments, respectively extracting features from the visible light image and the infrared image to obtain the global features of the visible light image, the global features of the infrared image, the local features of the visible light image, and the local features of the infrared image, including: based on the preset two-stream network, extracting features from the visible light image and the infrared image to obtain the global features of the visible light image and the global features of the infrared image; the visible light image and the infrared image are extracted through horizontal stripes to obtain the local features of the visible light image of the pedestrian and the local features of the infrared image, wherein the identity of each local feature is classified by a cross-entropy loss function, and the similarity of the same-class local features is constrained by a sample-to-sample loss function.

[0035] Specifically, a two-stream network is used as the baseline. In a mini-batch, the same number of visible light images and infrared images are sampled. The two-stream network uses Resnet50 as the backbone network. Among them, the first convolutional block of Resnet50 is modality-unique to learn modality-invariant low-level features, while the others are weighted and shared to capture discriminative features; in addition, the channel-wise attention in the Mixture of Attention Mechanisms (MAMs) is changed to part-wise attention, and it is inserted into the backbone network to extract modality-independent feature maps; global average pooling is performed on the obtained feature maps to obtain global features, and a batch normalization layer is used to obtain query representations.

[0036] Furthermore, the embodiments of the present invention use horizontal stripes to obtain partial features, and use cross-entropy loss and sample-to-sample loss to obtain a loss function based on the partial features: Then, the KL divergence of the two output distributions is calculated as a regularization term to learn more general partial features.

[0037] Specifically, the part alignment loss can be expressed as: To accelerate the convergence speed of the model, the weight of the loss involving partial features is reduced in the early stage of training, which may cause relatively large interference. Therefore, a weight μ is set for the loss of partial features, and this weight increases linearly with the passage of time. Then, the total loss can be expressed as: Where, is the loss corresponding to the identity of each local feature, is the loss for constraining the similarity of local features of the same class, P is the number of parts, represents the k-th part feature of the i-th identity, C g (·) and are the classifiers for the global feature and the k-th part feature.

[0038] Furthermore, in some embodiments, based on a preset alignment strategy, the aligned visible light image features and the aligned infrared image features are obtained according to the global features of the visible light image, the global features of the infrared image, the local features of the visible light image, and the local features of the infrared image, including: based on the visible light image and the infrared image, a memory bank is constructed, where the memory bank is used to maintain the feature center data of each pedestrian identity; according to the feature center data of each pedestrian identity, the loss between the center of the visible light image features and the center of the infrared image features is calculated; based on the loss, the global features of the visible light image and the global features of the infrared image are aligned, and at the same time, the local features of the visible light image and the local features of the infrared image are aligned to obtain the aligned visible light image features and the aligned infrared image features.

[0039] Specifically, the embodiments of the present invention use the memory bank M V ∈R Y×d to represent each identity of the RGB image, and the memory bank is updated during the training process by: using the current training sample f i to update the center of the corresponding class y i Considering the global features, the loss from identity center to identity center is defined as Taking into account the partial features, a weight parameter μ is introduced to balance the global loss and the partial loss. Then, the total loss from identity center to identity center is Where, Y is the number of identities, d is the feature dimension, represents the feature of the i-th class, μm is a momentum factor, is the center of the i-th identity in the visible light image, is the center of the i-th identity in the infrared image, and λ2, λ3, and μ are weight parameters.

[0040] Furthermore, based on the pre-established patch hybrid cross-modal framework, the aligned visible light image features and the aligned infrared image features are spliced to obtain an image patch hybrid image.

[0041] Optionally, an embodiment of the present invention may select a visible light image x V and an infrared image x I to generate an image patch hybrid image x M . Set the probability of selecting a visible light image patch to be p, then the probability of selecting an infrared image patch is 1 - p. Among them, each patch x M (i,j) is formed by x V (i,j) or x I (i,j), and i and j respectively represent the patch indices of the image length and width.

[0042] In step S103, an output distribution result is obtained according to the visible light image, the infrared image, and the image patch hybrid image, and based on the output distribution result, the first alignment degree between the image patch hybrid image and the visible light image and the second alignment degree between the image patch hybrid image and the infrared image are calculated, and a person re-identification result is obtained according to the first alignment degree and the second alignment degree.

[0043] Furthermore, in some embodiments, obtaining an output distribution result according to the visible light image, the infrared image, and the image patch hybrid image includes: determining the visible light image, the infrared image, and the image patch hybrid image of each pedestrian; inputting the visible light image, the infrared image, and the image patch hybrid image of each pedestrian into a preset patch hybrid cross-modal algorithm to obtain an output distribution result.

[0044] Specifically, the visible light image, the infrared image, and the image patch hybrid image training samples are input into the network for training, and the KL divergence of the output distribution is calculated to constrain the learning of the hybrid modality, thereby obtaining an accurate output distribution.

[0045] Furthermore, the output distribution result logits are recombined by modality, and alignment is performed between the image hybrid patch image and the visible light image and the infrared image. The alignment loss function is:

[0046] Among them, the global feature alignment result is:

[0047]

[0048] The result of local features is as follows:

[0049]

[0050] Among them, represents the k-th partial feature of the i-th identity, and C g (·) and are the classifiers of the global feature and the k-th partial feature respectively.

[0051] Furthermore, calculate the first alignment degree between the image patch mixed image and the visible light image, and calculate the second alignment degree between the image patch mixed image and the infrared image to obtain the total alignment degree loss function:

[0052]

[0053] Among them, is the loss function of the first alignment degree, is the loss function of the second alignment degree, and p is used to balance the weights of the two losses. The role of the ratio p is that the more image information of a certain modality contained in the mixed modality, the more similar it is to the source modality. By minimizing the total loss, the person re-identification result is obtained.

[0054] To enable those skilled in the relevant art to better understand the person re-identification effect of the embodiments of the present invention, the following will be explained with specific embodiments.

[0055] Figure 2 FIG. is a flowchart of a visible light-infrared person re-identification method based on patch mixing cross-modal according to a specific embodiment of the present invention; Table 1 is an ablation experiment result table of SYSU-MM01 in all-search and single-scan modes according to the embodiments of the present invention.

[0056] Among them, as Figure 2 shown, using the visible light RGB image and the infrared IR image as inputs, feature extraction is performed through the backbone network, and the patch mixing technology is used to mix the visible light RGB image and the infrared IR image. The baseline BASE loss, the center-to-center C2C loss, and the patch mixing cross-modal learning PMML loss are calculated respectively, and the person re-identification output result is obtained by minimizing the total loss.

[0057] Furthermore, as shown in Table 1, the effectiveness of some baselines (Part): When exploring partial features on the global-based baseline, the performance on Rank-1 and mAP is improved by 2.66% and 2.49% respectively, indicating its enhancement effect on feature representation learning.

[0058] Table 1

[0059] Method Rank-1 Rank-10 Rank-20 mAP baseline 64.58 93.58 97.67 62.41 +Part 67.24 94.70 98.38 64.90 +PartAlign 69.52 95.98 98.82 65.37 +C2C 70.66 96.22 98.97 66.06 +PatchMix 73.76 97.34 99.22 69.88 +PMML 75.54 97.49 99.30 71.16

[0060] Furthermore, the effectiveness of the Part Align loss regularizes the global and part feature predictions. The improvements in Rank-1 accuracy and mAP in Table 1 are 2.32% and 0.47% respectively, demonstrating that the part alignment loss better exploits discriminative part features.

[0061] Furthermore, the effectiveness of the Center-to-Center (C2C) loss is verified. Compared with the baseline, C2C improves the Rank-1 accuracy and mAP by 1.14% and 0.69% respectively, which benefits from C2C pulling closer the centers of different modalities of the same identity to extract modality-invariant features.

[0062] The effectiveness of Patch Mix for training with mixed images, as shown in the row "+PatchMix" in the table, after adding mixed images for training, the Rank-1 accuracy and mAP increase by 3.10% and 3.82% respectively, indicating its effectiveness.

[0063] Furthermore, the effectiveness of Patch Mix-based Multimodal Learning (PMML) is verified. The row '+PMML' in Table 1 provides sufficient evidence that PMML can further align between mixed modalities and two modalities, with its Rank-1 accuracy and mAP increasing by 1.78% and 1.28% respectively.

[0064] Therefore, the patch mix-based cross-modal framework for VI-ReID proposed in the present invention introduces a patch mix mode to learn the correspondence between visible light and infrared images. The model learns to recognize a person through different-style patches, and the modal semantic correspondence is directly reflected. Through a flexible image generation strategy, the patch mix image can freely adjust the proportion of different-modal image patches and alleviate the problem of mode imbalance. At the same time, to further reduce the differences between and within modalities, a center-to-center loss is proposed to align the centers of the same identity from different modalities, and a part alignment loss is proposed to constrain the consistency of part and global features, improving the accuracy of person re-identification.

[0065] According to the visible light-infrared person re-identification method based on patch mix cross-modal provided by the embodiment of the present invention, the visible light image and the infrared image of the same user are segmented into patches, and a patch mix image is obtained according to the patch mix strategy for learning, which can reflect the semantic correspondence, and by minimizing the loss between modalities, the person re-identification result is obtained. It solves the problems in the related art that it is difficult to align heterogeneous modalities and it is impossible to learn the semantic correspondence between two modalities well, resulting in mode imbalance and poor person re-identification effect, and improves the robustness and recognition accuracy of the person re-identification model.

[0066] Next, a visible light-infrared person re-identification device based on patch mixing cross-modal according to an embodiment of the present invention will be described with reference to the accompanying drawings.

[0067] Figure 3 It is a block diagram of a visible light-infrared person re-identification device based on patch mixing cross-modal according to an embodiment of the present invention.

[0068] As Figure 3 shown, the visible light-infrared person re-identification device 10 based on patch mixing cross-modal includes: an acquisition module 100, a data processing module 200, and an output module 300.

[0069] Among them, the acquisition module 100 is used to acquire visible light images and infrared images; the data processing module 200 is used to process the visible light and infrared images based on the patch mixing cross-modal framework, generate an image patch mixed image through a preset patch mixing strategy, and calculate the output distributions of the visible light image and the image patch mixed image. The output module 300 is used to calculate the alignment degree between the image patch mixed image and the infrared image based on the output distribution result, and output the person re-identification result.

[0070] Further, in some embodiments, the data processing module 200 is specifically used for: respectively extracting features from the visible light image and the infrared image to obtain the global feature of the visible light image, the global feature of the infrared image, the local feature of the visible light image, and the local feature of the infrared image; based on a preset alignment strategy, obtaining the aligned visible light image feature and the aligned infrared image feature according to the global feature of the visible light image, the global feature of the infrared image, the local feature of the visible light image, and the local feature of the infrared image; based on a pre-established patch mixing cross-modal framework, splicing the aligned visible light image feature and the aligned infrared image feature to obtain an image patch mixed image.

[0071] Further, in some embodiments, the data processing module 200 is further used for: extracting features from the visible light image and the infrared image based on a preset two-stream network to obtain the global feature of the visible light image and the global feature of the infrared image; extracting the visible light image and the infrared image through horizontal stripes to obtain the local feature of the visible light image of the pedestrian and the local feature of the infrared image of the pedestrian, wherein, the identity of each local feature is classified through a cross-entropy loss function, and the similarity of the same-class local features is constrained through a sample-to-sample loss function.

[0072] Further, in some embodiments, the data processing module 200 is further configured to: construct a memory bank based on the visible light image and the infrared image, where the memory bank is used to maintain the feature center data of each pedestrian identity; calculate the loss between the center of the visible light image features and the center of the infrared image features according to the feature center data of each pedestrian identity; and align the global features of the visible light image and the global features of the infrared image, and at the same time align the local features of the visible light image and the local features of the infrared image based on the loss, so as to obtain the aligned visible light image features and the aligned infrared image features.

[0073] Further, in some embodiments, the output module 300 is specifically configured to: determine the visible light image, the infrared image, and the image patch mixed image of each pedestrian; and input the visible light image, the infrared image, and the image patch mixed image of each pedestrian into a preset patch mixed cross-modal algorithm to obtain an output distribution result.

[0074] It should be noted that the above explanation of the embodiments of the visible light-infrared person re-identification method based on patch mixing cross-modal also applies to the visible light-infrared person re-identification device based on patch mixing cross-modal in this embodiment, and will not be elaborated here.

[0075] According to the visible light-infrared person re-identification device based on patch mixing cross-modal provided by the embodiments of the present invention, the visible light image and the infrared image of the same user are segmented into patches, and the patch mixed image is obtained according to the patch mixing strategy for learning, which can reflect the semantic correspondence relationship, and by minimizing the loss between each modality, the person re-identification result is obtained. It solves the problems in the related art that it is difficult to align heterogeneous modalities and it is impossible to learn the semantic correspondence relationship between the two modalities well, resulting in modality imbalance and poor person re-identification effect, and improves the robustness and recognition accuracy of the person re-identification model.

[0076] In addition, as Figure 4 shown, the embodiments of the present invention further provide an electronic device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor executes the program to implement the visible light-infrared person re-identification method based on patch mixing cross-modal as described above.

[0077] Figure 4 The structural schematic diagram of the electronic device provided by the embodiments of the present invention is shown. The electronic device may include:

[0078] a memory 401, a processor 402, and a computer program stored on the memory 401 and executable on the processor 402.

[0079] When the processor 402 executes the program, it implements the visible light-infrared person re-identification method based on patch mixing cross-modal provided in the above embodiments.

[0080] Further, the vehicle further includes: a communication interface 403 for communication between the memory 401 and the processor 402. The memory 401 is used to store computer programs that can run on the processor 402. The memory 401 may include a high-speed RAM memory, and may also include non-volatile memory, such as at least one disk memory.

[0081] If the memory 401, the processor 402, and the communication interface 403 are implemented independently, the communication interface 403, the memory 401, and the processor 402 can be interconnected through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of simplicity of representation, Figure 4 only a thick line is used to represent it in the figure, but it does not mean that there is only one bus or one type of bus.

[0082] Optionally, in a specific implementation, if the memory 401, the processor 402, and the communication interface 403 are integrated on a single chip, the memory 401, the processor 402, and the communication interface 403 can communicate with each other through an internal interface.

[0083] The processor 402 may be a Central Processing Unit (CPU), or an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention.

[0084] According to the electronic device proposed by the embodiments of the present invention, through the above-mentioned visible light-infrared person re-identification device based on patch mixing cross-modal, the problems in the related art that it is difficult to align heterogeneous modalities and it is impossible to learn the semantic correspondence relationship between the two modalities well, resulting in modality imbalance and poor pedestrian re-identification effect, are solved, and the robustness and recognition accuracy of the pedestrian re-identification model are improved.

[0085] In addition, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and characterized in that when the program is executed by a processor, the above-mentioned visible light-infrared person re-identification method based on patch mixing cross-modal is implemented.

[0086] In addition, an embodiment of the present invention further provides a computer program product, including a computer program, and the computer program is executed to implement the visible light-infrared person re-identification method based on patch mixing cross-modal as described in any one of the above.

[0087] In the description of this specification, the description with reference to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0088] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "N" is at least two, such as two, three, etc., unless otherwise specifically defined.

[0089] Any process or method description shown in the flowchart or described in other ways herein can be understood as representing a module, segment, or part of code including one or more executable instructions for implementing a customized logic function or process, and the scope of the preferred embodiments of the present invention includes additional implementations, where the functions can be executed in a substantially simultaneous manner or in a reverse order according to the involved functions, rather than in the order shown or discussed, which should be understood by those skilled in the art of the embodiments of the present invention.

[0090] It should be understood that each part of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0091] Those of ordinary skill in the art can understand that all or part of the steps carried out in implementing the methods of the above embodiments can be completed by instructing relevant hardware through a program. The program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.

Claims

1. A visible light-infrared person re-identification method based on patch mixing cross-modalities, characterized in that It includes the following steps: Obtain the visible light image and the infrared image of the pedestrian; Based on the pre-established patch mixing cross-modal framework, process the visible light image and the infrared image, and generate an image patch mixing image according to the processing result based on the preset patch mixing strategy; Obtain the output distribution result according to the visible light image, the infrared image and the image patch mixing image, and based on the output distribution result, calculate the first alignment degree between the image patch mixing image and the visible light image and the second alignment degree between the image patch mixing image and the infrared image, and obtain the person re-identification result according to the first alignment degree and the second alignment degree.

2. The visible light-infrared person re-identification method based on patch mixing cross-modal according to claim 1, characterized in that The step of processing the visible light image and the infrared image based on the pre-established patch mixing cross-modal framework and generating an image patch mixing image according to the processing result based on the preset patch mixing strategy includes: Extract features from the visible light image and the infrared image respectively to obtain the global feature of the visible light image, the global feature of the infrared image, the local feature of the visible light image and the local feature of the infrared image; Based on the preset alignment strategy, obtain the aligned visible light image feature and the aligned infrared image feature according to the global feature of the visible light image, the global feature of the infrared image, the local feature of the visible light image and the local feature of the infrared image; Based on the pre-established patch mixing cross-modal framework, splice the aligned visible light image feature and the aligned infrared image feature to obtain the image patch mixing image.

3. The method according to claim 2, wherein The step of extracting features from the visible light image and the infrared image respectively to obtain the global feature of the visible light image, the global feature of the infrared image, the local feature of the visible light image and the local feature of the infrared image includes: Based on the preset two-stream network, extract features from the visible light image and the infrared image to obtain the global feature of the visible light image and the global feature of the infrared image; Extract the visible light image and the infrared image through horizontal stripes to obtain the local feature of the visible light image of the pedestrian and the local feature of the infrared image, wherein the identity of each local feature is classified through the cross-entropy loss function, and the similarity of the same-class local features is constrained through the sample-to-sample loss function.

4. The method according to claim 2, wherein The step of obtaining the aligned visible light image feature and the aligned infrared image feature according to the global feature of the visible light image, the global feature of the infrared image, the local feature of the visible light image and the local feature of the infrared image based on the preset alignment strategy includes: Based on the visible light image and the infrared image, construct a memory bank, wherein the memory bank is used to maintain the feature center data of each pedestrian identity; According to the feature center data of each pedestrian identity, calculate the loss between the center of the visible light image feature and the center of the infrared image feature. Based on the loss, align the global features of the visible light image and the global features of the infrared image, and at the same time align the local features of the visible light image and the local features of the infrared image, to obtain the aligned visible light image features and the aligned infrared image features.

5. The visible light-infrared person re-identification method based on patch hybrid cross-modal according to claim 1, characterized in that, The obtaining of the output distribution result according to the visible light image, the infrared image and the image patch mixed image includes: Determine the visible light image, the infrared image and the image patch mixed image of each pedestrian; Input the visible light image, the infrared image and the image patch mixed image of each pedestrian into a preset patch mixing cross-modal algorithm to obtain the output distribution result.

6. A visible light-infrared person re-identification device based on patch mixing cross-modal, characterized in that, The device includes: An acquisition module, configured to acquire a visible light image and an infrared image; A data processing module, configured to process the visible light and infrared images based on a patch mixing cross-modal framework, generate an image patch mixed image through a preset patch mixing strategy, and calculate the output distribution of the visible light image and the image patch mixed image; An output module, configured to calculate the alignment degree between the image patch mixed image and the infrared image based on the output distribution result, and output a person re-identification result.

7. The visible light-infrared person re-identification device based on patch mixing cross-modal according to claim 6, wherein The data acquisition module is specifically configured to: Extract features from the visible light image and the infrared image respectively to obtain the global features of the visible light image, the global features of the infrared image, the local features of the visible light image and the local features of the infrared image; Based on a preset alignment strategy, obtain the aligned visible light image features and the aligned infrared image features according to the global features of the visible light image, the global features of the infrared image, the local features of the visible light image and the local features of the infrared image; Based on a pre-established patch mixing cross-modal framework, splice the aligned visible light image features and the aligned infrared image features to obtain the image patch mixed image.

8. An electronic device, characterized in that, Includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor executes the program to implement the patch mixing cross-modal based visible light-infrared person re-identification method according to any one of claims 1-5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to be used to implement the patch mixing cross-modal based visible light-infrared person re-identification method according to any one of claims 1-5.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the patch mixing cross-modal based visible light-infrared person re-identification method according to any one of claims 1-5.

Images