Cross-modal pedestrian re-identification method, system and device based on modal supplement and feature cooperation, and medium
Through modal supplementation and feature collaborative learning, the modal differences between visible light and infrared images are all alleviated, and the accuracy and efficiency of cross-modal pedestrian re-identification are improved, and the time-consuming and low-light conditions of manual screening in traditional methods is solved. Identify difficult problems and are suitable for the field of intelligent security.
Patent Information
- Application Number
- CN202411127811.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-16
- Publication Date
- 2024-12-27
AI Technical Summary
The traditional manual screening method is time-consuming and labor-intensive and susceptible to subjective factors. The infrared images obtained by the surveillance camera under low light conditions are very different from those of the visible light during the day, resulting in low cross-modal pedestrian recognition accuracy and cannot meet the complex environment. actual needs.
Using a cross-modal pedestrian re-identification method based on modal supplementation and feature coordination, modal supplementation and feature extraction are performed by obtaining visible light and infrared modal images, and global correlation features are obtained using the Global-Context Block and shuffle attention mechanism, combined with KL Divergence and central loss functions perform feature collaborative learning to generate intermediate modal images to alleviate modal differences.
It improves the accuracy and efficiency of pedestrian re-identification under different lighting conditions and different image modes, reduces modal differences, and realizes accurate positioning and tracking of key personnel, which is suitable for the field of intelligent security.
Smart Images
Figure CN119206771A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of intelligent security technology, and specifically relates to a cross-modal pedestrian re-identification method, system, device and medium based on modal supplementation and feature collaboration. Background Art
[0002] In recent years, with the accelerating pace of urbanization and rapid population growth, public safety and security management have faced increasingly severe challenges. In places where people gather and move, such as airports, train stations, and shopping malls, security personnel need to effectively monitor and manage personnel to promptly detect and respond to potential security risks. Furthermore, in crowded places, people often get lost or separated. Images captured by surveillance cameras can help track and locate missing individuals, such as elderly people or children, or track specific individuals. With the development of intelligent surveillance systems, identifying key individuals has become a key function and development trend.
[0003] Traditional manual screening methods have numerous problems, such as being time-consuming and labor-intensive, having limited effectiveness, and being susceptible to subjective factors. Furthermore, in low-light conditions, most surveillance cameras can switch to nighttime infrared mode. However, the images captured by infrared cameras differ significantly from those captured during daytime visible light, making it challenging to identify the same person in both visible light and infrared images.
[0004] Disadvantages of existing technology:
[0005] (1) Manual screening: Traditional manual screening methods not only consume a lot of time and manpower when handling personnel retrieval tasks, but also often produce unsatisfactory results. The main reason is that this method is easily affected by human factors, resulting in low recognition accuracy.
[0006] (2) Monitoring performance under low-light conditions: In environments with poor lighting conditions, although most surveillance cameras can switch to nighttime infrared camera mode, the infrared images obtained are significantly different from the daytime visible light images. This makes it extremely difficult to identify the same pedestrian in both visible light and infrared images.
[0007] (3) Person Re-ID in a Single Modality: Existing person re-ID technologies mainly focus on single-modality, that is, searching through visible light images from cameras with different viewpoints. However, this method is greatly limited in its application when faced with images in different modalities (such as visible light and infrared) at the same time, and cannot meet the actual needs in complex environments.
[0008] (4) Challenges of cross-modal pedestrian re-identification: In cross-modal pedestrian re-identification tasks, existing methods find it difficult to fully reduce the modal differences between visible light images and infrared images. This results in low accuracy when the model retrieves pedestrians between these two different modal images, making it impossible to accurately locate and track key personnel. Some existing methods use adversarial generative networks to generate intermediate modal information to enhance data. However, since infrared images do not contain information such as color, even infrared images with similar appearances may have large differences in the actual visible light images corresponding to them. This method is prone to introducing more random noise. Therefore, existing methods are not ideal in cross-modal pedestrian recognition tasks.
[0009] Therefore, to accurately locate and track key individuals in nighttime and complex environments, a highly accurate, low-cost, and easily implemented cross-modal person re-identification method is needed. This paper addresses these shortcomings of existing technologies and proposes a novel solution to improve the accuracy and efficiency of person re-identification under varying lighting conditions and image modalities. Summary of the Invention
[0010] In order to overcome the above-mentioned technical defects, the present invention provides a cross-modal pedestrian re-identification method, system, equipment and medium based on modal supplementation and feature collaboration, which can perform cross-modal pedestrian re-identification in a variety of monitoring environments, especially at night or in insufficient light conditions, effectively reduce cross-modal differences, improve recognition accuracy, and provide strong technical support for the field of intelligent security.
[0011] To solve the above problems, in a first aspect, the present invention provides a cross-modal person re-identification method based on modality complementation and feature collaboration, comprising:
[0012] Obtain a visible light modality image and an infrared modality image under the same pedestrian label, perform modality supplementation on the visible light modality image and the infrared modality image respectively, and obtain a first intermediate modality image corresponding to the visible light modality image and a second intermediate modality image corresponding to the infrared modality image;
[0013] Build a cross-modal person re-identification model;
[0014] Using visible light modal images, infrared modal images, first intermediate modal images, and second intermediate modal images, a cross-modal person re-identification model is subjected to feature extraction and feature collaborative training to obtain a trained model.
[0015] The visible light modality image and / or infrared modality image to be re-identified is input into the trained cross-modal pedestrian re-identification model for cross-modal re-identification.
[0016] Furthermore, the performing modality supplementation on the visible light modality image and the infrared modality image respectively to obtain a first intermediate modality image corresponding to the visible light modality image and a second intermediate modality image corresponding to the infrared modality image includes:
[0017] A data enhancement method based on channel exchange is used to convert the visible light modality image into a grayscale image, and the grayscale image is the first intermediate modality image;
[0018] The visible light modality image and the infrared modality image are horizontally segmented, and the segmented visible light modality image and the infrared modality image are randomly spliced to generate a spliced image, which is the second intermediate modality image.
[0019] Furthermore, the use of the visible light modality image, the infrared modality image, the first intermediate modality image, and the second intermediate modality image to perform feature extraction and feature collaborative training on the cross-modal person re-identification model includes:
[0020] Using a feature extraction network, image features of the visible light modality image, the infrared modality image, the first intermediate modality image, and the second intermediate modality image are extracted respectively;
[0021] The loss function is used to perform feature collaborative learning on the extracted image features.
[0022] Furthermore, the feature extraction network is used to extract the image features of the visible light modality image, the infrared modality image, the first intermediate modality image, and the second intermediate modality image, respectively, including:
[0023] Use the Global-Context Block mechanism to obtain global correlation features:
[0024]
[0025] Among them, the feature input is Np represents the number of locations, x and z represent the input and output of Global-ContextBlock respectively, i represents the index of the location, j is used to enumerate all possible related locations, W v and W k is the linear transformation matrix;
[0026] The shuffle attention mechanism is used to obtain channel feature information.
[0027] Furthermore, the method of using the loss function to perform collaborative feature learning on the extracted image features includes:
[0028] The KL divergence loss function is used to constrain the logical distribution of the first intermediate modality image features, the logical distribution of the visible light modality image features, the logical distribution of the second intermediate modality image features, and the logical distribution of the infrared modality image features:
[0029]
[0030] Where N represents the number of samples in a training batch, θ inter and θ base are the feature extractors of the intermediate mode and the corresponding mode respectively, and f represents the extracted features;
[0031] The central loss function is used to move the features of the visible light modality image and the features of the infrared modality image closer to the first intermediate modality image and the second intermediate modality image in the feature space, respectively, and feature clustering is performed on the first intermediate modality image and the second intermediate modality image.
[0032] In a second aspect, the present invention provides a cross-modal person re-identification system based on modality complementation and feature collaboration, comprising:
[0033] The front-end module is used to obtain visible light images, infrared images, user instructions, and display pedestrian re-identification results;
[0034] The backend module is used to implement the above-mentioned cross-modal pedestrian re-identification method based on modality supplementation and feature collaboration.
[0035] Furthermore, the backend module includes:
[0036] A modality supplementation submodule is used to perform modality supplementation on the visible light modality image and the infrared modality image under the same pedestrian label to obtain a first intermediate modality image corresponding to the visible light modality image and a second intermediate modality image corresponding to the infrared modality image;
[0037] a global perception submodule, configured to extract image features of the visible light modality image, the first intermediate modality image, the infrared modality image, and the second intermediate modality image;
[0038] The feature collaboration submodule is used to move the features of the visible light modality image and the features of the infrared modality image closer to the first intermediate modality image and the second intermediate modality image in the feature space, and to perform feature clustering on the first intermediate modality image and the second intermediate modality image.
[0039] In a third aspect, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.
[0040] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.
[0041] Compared with the prior art, the present invention has the following beneficial effects:
[0042] The present invention discloses a cross-modal pedestrian re-identification method, system, device and medium based on modal supplementation and feature collaboration, wherein the cross-modal pedestrian re-identification method comprises: obtaining a visible light modal image and an infrared modal image under the same pedestrian label, performing modal supplementation on the visible light modal image and the infrared modal image respectively, obtaining a first intermediate modal image corresponding to the visible light modal image and a second intermediate modal image corresponding to the infrared modal image; constructing a cross-modal pedestrian re-identification model; using the visible light modal image, the infrared modal image, the first intermediate modal image and the second intermediate modal image to perform feature extraction and feature collaborative training on the cross-modal pedestrian re-identification model to obtain a trained model; and The identified visible light modal image or infrared modal image is input into the trained cross-modal pedestrian re-identification model to identify the target; compared with the intermediate modality supplementation method using adversarial generative networks in the prior art, the present invention uses visible light modal images and infrared modal images to generate two corresponding intermediate modalities without introducing additional noise as much as possible. Through the combined effect of the intermediate modalities, the cross-modal differences between visible light modal images and infrared modal images are alleviated, so that the model can more accurately identify and match pedestrians, and improve the accuracy and efficiency of pedestrian re-identification under different lighting conditions and different image modalities; at the same time, it can enrich the input of training data and further enhance the recognition accuracy of the algorithm network. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings, wherein:
[0044] Figure 1 This is a flowchart of the cross-modal person re-identification method described in Example 1;
[0045] Figure 2 Schematic diagram of modal supplementation of the cross-modal person re-identification method described in Example 1;
[0046] Figure 3 This is a flow chart of the cross-modal person re-identification method using a feature extraction network described in Example 1;
[0047] Figure 4 The key area heat maps are obtained by using the feature extraction network of the conventional method and the cross-modal pedestrian re-identification method described in Example 1 respectively;
[0048] Figure 5A comparison diagram of the distribution of features extracted in the feature space when only the KL divergence loss function is used and when both the KL divergence loss function and the center loss function of the cross-modal person re-identification method described in Example 1 are used;
[0049] Figure 6 Schematic diagram of the operation of the cross-modal person re-identification system described in Example 2;
[0050] Figure 7 Schematic diagram of the device described in Example 3. DETAILED DESCRIPTION
[0051] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.
[0052] In addition, in this application, descriptions such as "first" and "second" are only used for descriptive purposes and do not specifically refer to the order or ranking, nor are they used to limit this application. They are only used to distinguish components or operations described with the same technical terms, and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one such feature. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in this field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by this application.
[0053] Example 1
[0054] This embodiment discloses a cross-modal person re-identification method based on modality supplementation and feature collaboration, such as Figure 1 ,include:
[0055] S1. Obtain a visible light modality image and an infrared modality image with the same pedestrian label, perform modal complementation on each of the visible light modality image and the infrared modality image to obtain a first intermediate modality image corresponding to the visible light modality image and a second intermediate modality image corresponding to the infrared modality image. The visible light modality image, the first intermediate modality image, the infrared modality image, and the second intermediate modality image are used to train a cross-modal person re-identification model.
[0056] Specifically, if Figure 2 , step S1 includes:
[0057] A data augmentation method based on channel swapping is used to convert the visible light modality image into a grayscale image. The grayscale image is the first intermediate modality image. The visible light modality image can be divided into three color channels: R, G, and B. A randomly selected channel is used to replace the information of the other channels to generate a grayscale image, the first intermediate modality image. The spectral structure of the first intermediate modality image is similar to that of the infrared image.
[0058] The visible light modality image and the infrared modality image are horizontally segmented and randomly spliced together to generate a stitched image, which serves as the second intermediate modality image. The visible light modality image and the infrared modality image are horizontally segmented and stitched together using random starting points and a predetermined mixing ratio. Specifically, a random starting point is selected on the infrared modality image, a stitching interval is calculated based on a preset mixing ratio, and the image within the stitching interval is replaced with the image at the corresponding position in the visible light modality image. Using random starting points increases the complexity of the image data; different preset mixing ratios result in different proportions of the visible light modality image and the infrared modality image in the generated stitched image.
[0059] S2. Build a cross-modal person re-identification model.
[0060] S3. Perform feature extraction and feature collaborative training on a cross-modal person re-identification model using the visible light modal image, the infrared modal image, the first intermediate modal image, and the second intermediate modal image to obtain a trained model. The first intermediate modal image and the second intermediate modal image can enrich the training data, effectively alleviate the differences between the visible light modal image and the infrared modal image, and improve the cross-modal person re-identification model's adaptability to cross-modal changes.
[0061] Specifically, if Figure 3 , step S3 includes:
[0062] S31. Using a feature extraction network, extract image features of the visible light modality image, the infrared modality image, the first intermediate modality image, and the second intermediate modality image respectively:
[0063] In the model backbone, the Global-Context Block mechanism is used to obtain global correlation features:
[0064]
[0065] Among them, the feature input is Np represents the number of locations, x and z represent the input and output of Global-ContextBlock respectively, i represents the index of the location, j is used to enumerate all possible related locations, W vand W k is the linear transformation matrix. Figure 4 , (a) is a heat map of the key area obtained by conventional methods, and (b) is a heat map of the key area obtained by using the feature extraction network described in this embodiment.
[0066] S32. Use the shuffle attention mechanism to obtain channel feature information. While using the Global-Context Block mechanism to obtain global correlation features, considering the better utilization of feature channel information in the cross-modal pedestrian re-identification model, the shuffle attention mechanism is used to strengthen the cross-modal pedestrian re-identification model's attention to channel feature information. Step S3 uses the Global-Context Block mechanism and the shuffle attention mechanism to enhance the cross-modal pedestrian re-identification model's ability to perceive pedestrian global correlation features and channel feature information.
[0067] S33, using the loss function to perform feature collaborative learning on each extracted image feature, including:
[0068] To ensure that the features extracted from the two intermediate modal images correspond to the features of their corresponding optical modality images and infrared modality images, the KL divergence loss function is used to constrain the logical distribution of the features of the first intermediate modality image, the logical distribution of the features of the visible light modality image, the logical distribution of the features of the second intermediate modality image, and the logical distribution of the features of the infrared modality image. The KL divergence loss can be expressed as:
[0069]
[0070] Where N represents the number of samples in a training batch, θ inter and θ base are feature extractors for the intermediate modality image and its corresponding modality image, respectively. Represents the features extracted from the intermediate modal image, Represents the features extracted from the visible light or infrared modality image corresponding to the intermediate modality image. Using the KL divergence loss, the logical distribution of the intermediate modality image features is gradually brought closer to the logical distribution of the features of the corresponding modality image, ensuring that the features extracted from the intermediate modality image are valid features that can match the corresponding visible light or infrared modality image.
[0071] A central loss function is used to align the features of the visible modality image and the infrared modality image in feature space toward the first intermediate modality image and the second intermediate modality image, respectively. Feature clustering is also performed on the first and second intermediate modality images. Specifically, the central loss function is used to control the feature extractor to reduce the distance between features in the different modal images, thereby reducing the differences between the different modal images. By using the intermediate modality images to reduce the differences between the different modal images, the distance between the features generated by the feature extractor in the visible and infrared modal images in feature space is gradually reduced through training.
[0072] In one embodiment, the center loss function can be expressed as:
[0073]
[0074] Among them, c yi Indicates that all labels in the current batch are y i The average value of the features, f i is the feature extracted by the feature extractor; using the center loss function, different inputs belonging to the same pedestrian label can be brought closer in the feature space; the modal relaxation loss implemented based on the center loss function is:
[0075]
[0076] Among them, f v is the feature of the visible light modality image obtained by the model, f t is the feature of the infrared modality image obtained by the model, is the feature of the first intermediate modal image obtained by the model, is the feature of the second intermediate modality image obtained by the model, λ a ,λ b ,λ c The hyperparameters used to train the model specify the degree of mitigation of differences between pairs of images in different modalities. Appropriate hyperparameter values can be selected through experimental comparison to achieve optimal model performance. Aggregating the first and second intermediate modal images allows for a more moderate and balanced integration of the two input modalities (visible light and infrared), thereby encouraging the model to learn common features between these modalities.
[0077] In summary, feature collaborative learning can be expressed as:
[0078]
[0079] Among them, λ k ,λ sFor the hyperparameters used to train the model, specify The emphasis of the two loss functions can be compared experimentally to select appropriate hyperparameter values to achieve the best performance of the model.
[0080] In order to constrain the feature representation of a pedestrian, most cross-modal person re-identification methods use cross entropy to implement an identity loss function: the images of the same person in different modalities are regarded as the same category, and y represents the corresponding true identity label, f(x i ) represents the features extracted by the network model from the input modal image. In a batch N, the identity loss is expressed as:
[0081]
[0082] At the same time, for the intra-class and inter-class differences between pedestrian identity labels, triplet loss is used to reduce the intra-class differences and increase the inter-class differences. Usually, a triplet loss contains an anchor sample x i , a positive sample x with the same identity as the anchor sample j , and a negative sample x from a different identity k , ρ represents the interval parameter, and the Euclidean distance is used to represent the distance difference between them. The triplet loss can be expressed as:
[0083]
[0084] The present invention combines the identity loss function and the triple loss function as the baseline loss function, and the baseline loss function is:
[0085]
[0086] The baseline loss function is collaboratively learned with the above features to construct a loss function for training the cross-modal person re-identification model described in the present invention.
[0087] Guided by the loss function, the cross-modal pedestrian re-identification model learns to extract the common features between the two modal images, reduces the differences between different modal images, and improves the accuracy of cross-modal pedestrian re-identification; uses the intermediate modal image to alleviate color differences and detail information, and guides the cross-modal pedestrian re-identification model to tend to learn other high-level semantic features of pedestrian representation between visible light modal images and infrared modal images, such as human body contours, clothing details, etc. Through feature collaborative learning, it is possible to optimize different modal features in the feature space, making the feature representations of the same pedestrian in different modalities closer, thereby improving the discrimination ability and recognition robustness of the cross-modal pedestrian re-identification model. Figure 5, (c) is the distribution of features in the feature space extracted by the common cross-entropy loss function and the triplet loss function commonly used in ordinary cross-modal pedestrian re-identification methods, and (d) is the distribution of features in the feature space extracted by this method when the KL divergence loss function and the center loss function are used on the basis of the above-mentioned existing technologies.
[0088] S4. Input the visible light modality image and / or infrared modality image to be re-identified into the trained cross-modal person re-identification model for cross-modal re-identification.
[0089] This invention is based on the idea of using an intermediate modality to mitigate the differences between the two modalities. It uses two modal images to generate an intermediate modality image, guiding the model to learn the common features between the two modalities to achieve the task requirements of cross-modal person re-identification. By inputting visible light and infrared modal images corresponding to the same pedestrian label, as well as two intermediate modality images generated based on these two, into a feature extraction network for feature extraction, the features extracted from the intermediate modal images are used to guide the cross-modal person re-identification model to perform collaborative learning between the two modalities, which are significantly different, enabling the cross-modal person re-identification model to learn the invariant features between the modalities. During model training, an intermediate modality image is used as a bridge between the two modal images. Through feature collaborative learning, the features extracted from the visible light modality image and the infrared modality image are moved closer to their respective corresponding intermediate modality images in the feature space, that is, the visible light modality image is closer to the first intermediate modality image, and the infrared modality image is closer to the second intermediate modality image. At the same time, feature clustering is also performed on the two intermediate modality images, thereby guiding the learning of the cross-modal pedestrian re-identification model in a relatively gentle manner and minimizing the differences between the modalities. By constraining the feature distribution between the two modal images and performing collaborative learning through KL divergence loss and center loss function, the accuracy of cross-modal pedestrian recognition is significantly improved. After training, the cross-modal pedestrian re-identification network model is obtained, which can realize the cross-modal re-identification task of pedestrians in visible light-infrared images. A high-efficiency and high-accuracy method suitable for cross-modal pedestrian re-identification of visible light-infrared images is obtained. It can maintain high recognition accuracy and robustness in various monitoring environments, especially at night or under low light conditions, and is suitable for various application scenarios such as smart security.
[0090] Example 2
[0091] This embodiment discloses a cross-modal pedestrian re-identification system based on modal supplementation and feature collaboration, including a front-end module and a back-end module. The front-end module is used to acquire visible light images, infrared images, interact with users, and display pedestrian re-identification results; the back-end module is used to implement the cross-modal pedestrian re-identification method based on modal supplementation and feature collaboration described in Example 1.
[0092] In one embodiment, the back-end module includes a modality supplementation submodule, a global perception submodule and a feature collaboration submodule. The modality supplementation submodule is used to perform modality supplementation on the visible light modality image and the infrared modality image under the same pedestrian label to obtain a first intermediate modality image corresponding to the visible light modality image and a second intermediate modality image corresponding to the infrared modality image.
[0093] The global perception submodule is used to extract image features of the visible light modality image, the first intermediate modality image, the infrared modality image, and the second intermediate modality image. The feature collaboration submodule is used to move the features of the visible light modality image and the infrared modality image closer to the first intermediate modality image and the second intermediate modality image in the feature space, respectively, and to perform feature clustering on the first intermediate modality image and the second intermediate modality image.
[0094] For the specific implementation details of the modality supplement submodule, global perception submodule and feature collaboration submodule, please refer to Example 1 and will not be repeated here.
[0095] The following is an explanation of the specific implementation process of this cross-modal person re-identification system:
[0096] Step 1: Detect pedestrians:
[0097] For the input monitoring images, the pedestrian detection network is used to quickly detect the positions of pedestrians in visible light modal images and infrared modal images.
[0098] Step 2: Establish a pedestrian retrieval database:
[0099] After pedestrians are detected, their images are aggregated into a pedestrian retrieval database (Gallery).
[0100] Step 3: Match pedestrian identity:
[0101] When it is necessary to retrieve a specific pedestrian or match pedestrian identity information, a specific pedestrian image is input as a query (Query). Through the cross-modal pedestrian re-identification method based on modality supplementation and feature collaboration described in Example 1, the deep semantic features of the query image are extracted through the cross-modal pedestrian re-identification model, and the similarity measurement method is used to match the query image in the above-mentioned pedestrian retrieval database, and the matching degree is sorted from high to low. Figure 6 , which is a schematic diagram of the operation of the cross-modal pedestrian re-identification system described in this embodiment.
[0102] This invention enables cross-modal person re-identification, maintaining high accuracy, particularly in complex environments and low-light conditions. By generating intermediate modal images and introducing a global perception submodule and a feature collaboration submodule, the model achieves excellent performance in cross-modal person re-identification tasks, demonstrating high reliability and adaptability.
[0103] Example 3
[0104] Each module in the aforementioned cross-modal person re-ID system based on modal complementation and feature collaboration can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor within a computer device in hardware form, or stored in a computer device memory in software form, allowing the processor to call and execute the corresponding operations of each module.
[0105] In one embodiment, a computer device is provided. The computer device may be a server or terminal of an integrated scheduler, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as the mapping relationship between the shared video memory area and the virtual machine, the total load of the virtual machine, etc. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a cross-modal pedestrian re-identification method based on modal supplementation and feature collaboration is implemented.
[0106] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.
[0107] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:
[0108] Obtain a visible light modality image and an infrared modality image under the same pedestrian label, perform modality supplementation on the visible light modality image and the infrared modality image respectively, and obtain a first intermediate modality image corresponding to the visible light modality image and a second intermediate modality image corresponding to the infrared modality image;
[0109] Build a cross-modal person re-identification model;
[0110] Using visible light modal images, infrared modal images, first intermediate modal images, and second intermediate modal images, a cross-modal person re-identification model is subjected to feature extraction and feature collaborative training to obtain a trained model.
[0111] The visible light modality image and / or infrared modality image to be re-identified is input into the trained cross-modal pedestrian re-identification model for cross-modal re-identification.
[0112] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0113] A data enhancement method based on channel exchange is used to convert the visible light modality image into a grayscale image, and the grayscale image is the first intermediate modality image;
[0114] The visible light modality image and the infrared modality image are horizontally segmented, and the segmented visible light modality image and the infrared modality image are randomly spliced to generate a spliced image, which is the second intermediate modality image.
[0115] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0116] Using a feature extraction network, image features of the visible light modality image, the infrared modality image, the first intermediate modality image, and the second intermediate modality image are extracted respectively;
[0117] The loss function is used to perform feature collaborative learning on the extracted image features.
[0118] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0119] Use the Global-Context Block mechanism to obtain global correlation features:
[0120]
[0121] Among them, the feature input is Np represents the number of locations, x and z represent the input and output of Global-ContextBlock respectively, i represents the index of the location, j is used to enumerate all possible related locations, W v and W k is the linear transformation matrix;
[0122] The shuffle attention mechanism is used to obtain channel feature information.
[0123] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0124] The KL divergence loss function is used to constrain the logical distribution of the first intermediate modality image features, the logical distribution of the visible light modality image features, the logical distribution of the second intermediate modality image features, and the logical distribution of the infrared modality image features:
[0125]
[0126] Where N represents the number of samples in a training batch, θ inter and θ base are the feature extractors of the intermediate mode and the corresponding mode respectively, and f represents the extracted features;
[0127] The central loss function is used to move the features of the visible light modality image and the features of the infrared modality image closer to the first intermediate modality image and the second intermediate modality image in the feature space, respectively, and feature clustering is performed on the first intermediate modality image and the second intermediate modality image.
[0128] Example 4
[0129] This embodiment discloses a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the following steps are implemented:
[0130] Obtain a visible light modality image and an infrared modality image under the same pedestrian label, perform modality supplementation on the visible light modality image and the infrared modality image respectively, and obtain a first intermediate modality image corresponding to the visible light modality image and a second intermediate modality image corresponding to the infrared modality image;
[0131] Build a cross-modal person re-identification model;
[0132] Using visible light modal images, infrared modal images, first intermediate modal images, and second intermediate modal images, a cross-modal person re-identification model is subjected to feature extraction and feature collaborative training to obtain a trained model.
[0133] The visible light modality image and / or infrared modality image to be re-identified is input into the trained cross-modal pedestrian re-identification model for cross-modal re-identification.
[0134] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:
[0135] A data enhancement method based on channel exchange is used to convert the visible light modality image into a grayscale image, and the grayscale image is the first intermediate modality image;
[0136] The visible light modality image and the infrared modality image are horizontally segmented, and the segmented visible light modality image and the infrared modality image are randomly spliced to generate a spliced image, which is the second intermediate modality image.
[0137] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:
[0138] Using a feature extraction network, image features of the visible light modality image, the infrared modality image, the first intermediate modality image, and the second intermediate modality image are extracted respectively;
[0139] The loss function is used to perform feature collaborative learning on the extracted image features.
[0140] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:
[0141] Use the Global-Context Block mechanism to obtain global correlation features:
[0142]
[0143] Among them, the feature input is Np represents the number of locations, x and z represent the input and output of Global-ContextBlock respectively, i represents the index of the location, j is used to enumerate all possible related locations, W v and W k is the linear transformation matrix;
[0144] The shuffle attention mechanism is used to obtain channel feature information.
[0145] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:
[0146] The KL divergence loss function is used to constrain the logical distribution of the first intermediate modality image features, the logical distribution of the visible light modality image features, the logical distribution of the second intermediate modality image features, and the logical distribution of the infrared modality image features:
[0147]
[0148] Where N represents the number of samples in a training batch, θ inter and θ base are the feature extractors of the intermediate mode and the corresponding mode respectively, and f represents the extracted features;
[0149] The central loss function is used to move the features of the visible light modality image and the features of the infrared modality image closer to the first intermediate modality image and the second intermediate modality image in the feature space, respectively, and feature clustering is performed on the first intermediate modality image and the second intermediate modality image.
[0150] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.
[0151] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0152] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Therefore, any modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall still fall within the scope of the technical solution of the present invention.
Claims
1. A cross-modal person re-identification method based on modality complementation and feature collaboration, characterized in that: include: Obtain a visible light modality image and an infrared modality image under the same pedestrian label, perform modality complementation on the visible light modality image and the infrared modality image respectively, and obtain a first intermediate modality image corresponding to the visible light modality image and a second intermediate modality image corresponding to the infrared modality image; Build a cross-modal person re-identification model; Using visible light modality images, infrared modality images, first intermediate modality images, and second intermediate modality images to perform feature extraction and feature collaborative training on a cross-modal person re-identification model to obtain a trained model; The visible light modality image and / or infrared modality image to be re-identified is input into the trained cross-modal pedestrian re-identification model for cross-modal re-identification.
2. The cross-modal person re-identification method based on modality complementation and feature collaboration according to claim 1 is characterized in that: The method of performing modality supplementation on the visible light modality image and the infrared modality image respectively to obtain a first intermediate modality image corresponding to the visible light modality image and a second intermediate modality image corresponding to the infrared modality image includes: A data enhancement method based on channel exchange is adopted to convert the visible light modality image into a grayscale image, and the grayscale image is the first intermediate modality image; The visible light modality image and the infrared modality image are horizontally segmented, and the segmented visible light modality image and the infrared modality image are randomly spliced to generate a spliced image, which is the second intermediate modality image.
3. The cross-modal person re-identification method based on modality complementation and feature collaboration according to claim 2 is characterized in that: The method of extracting features and co-training features of a cross-modal person re-identification model using visible light modality images, infrared modality images, first intermediate modality images, and second intermediate modality images includes: Using a feature extraction network, respectively extract image features of the visible light modality image, the infrared modality image, the first intermediate modality image, and the second intermediate modality image; The loss function is used to perform feature collaborative learning on the extracted image features.
4. The cross-modal person re-identification method based on modality complementation and feature collaboration according to claim 3 is characterized in that: The feature extraction network is used to extract the image features of the visible light modality image, the infrared modality image, the first intermediate modality image, and the second intermediate modality image, respectively, including: Use the Global-Context Block mechanism to obtain global correlation features: Among them, the feature input is Np represents the number of positions, x and z represent the input and output of Global-ContextBlock respectively, i represents the index of the position, j is used to enumerate all possible related positions, and W v and W k is the linear transformation matrix; The shuffle attention mechanism is used to obtain channel feature information.
5. The cross-modal person re-identification method based on modality complementation and feature collaboration according to claim 4 is characterized in that: The method of using the loss function to perform feature collaborative learning on each extracted image feature includes: The KL divergence loss function is used to constrain the logical distribution of the first intermediate modality image features, the logical distribution of the visible light modality image features, the logical distribution of the second intermediate modality image features, and the logical distribution of the infrared modality image features: Where N represents the number of samples in a training batch, θ inter and θ base They are the feature extractors of the intermediate mode and the corresponding mode respectively, and f represents the extracted features; The central loss function is used to move the features of the visible light modality image and the features of the infrared modality image closer to the first intermediate modality image and the second intermediate modality image in the feature space, respectively, and feature clustering is performed on the first intermediate modality image and the second intermediate modality image.
6. A cross-modal person re-identification system based on modality complementation and feature collaboration, characterized in that: include: The front-end module is used to obtain visible light images, infrared images, user instructions, and display pedestrian re-identification results; A back-end module, used to implement the cross-modal pedestrian re-identification method based on modality supplementation and feature collaboration as described in any one of claims 1-5.
7. The cross-modal person re-identification system based on modality complementation and feature collaboration according to claim 6, characterized in that: The backend modules include: A modality supplement submodule is used to perform modality supplement on the visible light modality image and the infrared modality image under the same pedestrian label to obtain a first intermediate modality image corresponding to the visible light modality image and a second intermediate modality image corresponding to the infrared modality image; A global perception submodule, used for extracting image features of a visible light modality image, a first intermediate modality image, an infrared modality image, and a second intermediate modality image; The feature coordination submodule is used to bring the features of the visible light modality image and the features of the infrared modality image closer to the first intermediate modality image and the second intermediate modality image in the feature space, respectively, and to perform feature clustering on the first intermediate modality image and the second intermediate modality image.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.