A pedestrian re-identification method and system based on end-cloud collaboration multi-modal fusion

Abstract

The application discloses a kind of pedestrian re-identification method and system based on end cloud cooperation multi-modal fusion, the method first utilizes uncertainty guide sampling module to filter out the pedestrian image needing to be uploaded to cloud side.Then, adopt multi-modal pedestrian re-identification network, the weighted average of face and gait feature is calculated and adaptive fusion feature is generated, and feature classification is carried out through fully connected layer and softmax function.In addition, the method of the application optimizes visual prompt in cloud, realizes the knowledge transfer of cloud side based on prompt learning.The performance of end side student model is improved by jointly optimizing teacher model and student model.Finally, the visual prompt learned by cloud side is transmitted to end side and applied to face image and gait image to improve the accuracy of pedestrian re-identification.The application effectively improves the retrieval performance of cross-appearance pedestrian re-identification model, and solves the generalization problem of network in different environments.

Description

Technical Field

[0001] This invention belongs to the field of multimodal pedestrian re-identification, and particularly relates to a pedestrian re-identification method and system based on edge-cloud collaborative multimodal fusion. Background Technology

[0002] Pedestrian re-identification is an important technology widely used in fields such as intelligent security and intelligent monitoring. It aims to match images or videos of the same pedestrian from different cameras, thereby making up for the visual limitations of fixed cameras.

[0003] Pedestrian re-identification based on multimodal features has been proposed by researchers, such as combining gait modality with anthropometry features (e.g., height), combining contour and skeletal features, and combining gait and facial features. However, these methods perform poorly in real-world scenarios. A significant limitation is the significant inconsistency in the distribution of images or videos captured by surveillance cameras in real-world scenes. Faced with continuous changes in real-world environments, such as daytime, nighttime, foggy weather, and sunny weather, the performance of existing methods deteriorates significantly. Furthermore, due to computational limitations of edge devices, it is impossible to deploy models with stronger generalization capabilities. Existing methods cannot adaptively update edge model parameters in a timely manner, making them unable to adapt to continuously changing scenarios. Summary of the Invention

[0004] The purpose of this invention is to solve the problems in the prior art, such as the inconsistent distribution of images or videos captured by surveillance cameras in real-world scenarios and the inability to deploy models with strong generalization capabilities due to the computational limitations of edge devices, and to provide a pedestrian re-identification method and system based on multimodal fusion of edge-cloud collaboration.

[0005] To achieve the above-mentioned objectives, the present invention specifically adopts the following technical solution:

[0006] In a first aspect, the present invention provides a pedestrian re-identification method based on edge-cloud collaborative multimodal fusion, comprising the following steps:

[0007] S1. The pedestrian image is input to the uncertainty-guided sampling module on the edge side, and the output is the judgment result of whether the pedestrian image needs to be uploaded to the cloud side for processing. After the edge side extracts the face image and gait image of the pedestrian image, if the judgment result is that the pedestrian image needs to be uploaded to the cloud side, then the face image and gait image are uploaded to the cloud side for processing. If the judgment result is that the pedestrian image does not need to be uploaded to the cloud side, then the multimodal pedestrian re-identification network trained on the edge side is used to process the face image and gait image.

[0008] S2. On the edge, the face image and gait image are input into the edge multimodal pedestrian re-identification network. In the edge multimodal pedestrian re-identification network, the feature extraction network first extracts the face features of the face image and the gait features of the gait image. The face features and gait features are input into the pre-trained context-adaptive multimodal fusion module to obtain context-aware adaptive fusion features. The adaptive fusion features are then passed through a fully connected layer and a softmax function to classify the adaptive fusion features, and the classification result of the edge pedestrian image is obtained.

[0009] S3. The cloud side acquires the face image and gait image output from the terminal side, and jointly trains and optimizes the pre-trained student model and the pre-trained teacher model on the cloud side. In each iteration of the joint training and optimization process, the face cue matrix and face image updated in the previous iteration are added together to obtain a new face image. The gait cue matrix and gait image updated in the previous iteration are added together to obtain a new gait image. The new face image and the new gait image are used as input to the teacher model updated in the previous iteration. After updating the parameters of the teacher model, the face cue matrix, and the gait cue matrix, the pseudo-label output by the updated teacher model is used as the real label of the student model. The new face image, the new gait image, and the real label of the student model are input into the student model for training. The parameters of the teacher model, the face cue matrix, and the gait cue matrix are fixed, and the parameters of the student model are updated separately. The teacher model and the student model each use a loss to update their parameters.

[0010] S4. After the teacher model and student model are trained, the cloud side sends the trained student model parameters, as well as the face cue matrix parameters and gait cue matrix parameters, to the edge side. The edge side uses the trained student model to process the face image and gait image obtained in S1. The edge side adds the face image obtained in S1 to the trained face cue matrix to obtain the face image to be classified, and adds the gait image obtained in S1 to the trained gait cue matrix to obtain the gait image to be classified. The face image and gait image to be classified are input into the trained student model to obtain the classification result of the pedestrian image on the edge side.

[0011] Based on the above scheme, each step can be implemented in the following preferred manner.

[0012] As a preferred embodiment of the first aspect mentioned above, in step S1, after the uncertainty-guided sampling module calculates the uncertainty of the pedestrian image, if the uncertainty of the pedestrian image is greater than a preset uncertainty threshold, then the pedestrian image needs to be uploaded to the cloud; wherein, the uncertainty V of the pedestrian image... unc The calculation method is as follows:

[0013]

[0014] In the formula, μ represents the average output probability of the uncertainty-guided sampling module in n predictions; n represents the preset number of predictions; (p i (y|x) represents the output probability of the uncertainty-guided sampling module during the i-th prediction.

[0015] As a preferred embodiment of the first aspect mentioned above, the specific process for obtaining the context-aware adaptive fusion features in step S2 is as follows:

[0016] S21. Utilize the context-adaptive multimodal fusion module to output the global individual attention weights of the face. Global individual attention weights for gait Calculate the weighted global attention weight α for facial features and the weighted global attention weight β for gait features:

[0017]

[0018] S22. Multiply the face feature by its weighted global attention weight α to obtain the weighted face feature. Multiply the gait feature by its weighted global attention weight β to obtain the weighted gait feature. Add the weighted face feature and the weighted gait feature to generate the context-aware adaptive fusion feature Z. The specific calculation method is as follows:

[0019] Z = αF + βG

[0020] Where F represents facial features, αF represents weighted facial features; G represents gait features, and βG represents weighted gait features.

[0021] As a preferred embodiment of the first aspect mentioned above, in step S3, the loss L of the teacher model... tea for:

[0022]

[0023] Where λ represents the weight of the adversarial alignment loss; L d G represents the adversarial alignment loss; d represents the domain classifier of the teacher model; F represents the feature extractor of the teacher model; This represents a face image in the source domain data; Represents the gait image in the source domain data; θ tea Indicates the teacher model parameters; This represents a new facial image; Represents a new gait image; y d For domain labels; L represents supervised loss; X represents domain label ... tea This represents the output of the teacher model; y sThis indicates the source domain data label.

[0024] As a preferred embodiment of the first aspect mentioned above, in step S3, the loss L of the student model... stu for:

[0025]

[0026] Among them, L f Represents the supervised classification loss; θ stu X represents the student model parameters; stu This represents the output of the student model; y p This represents the pseudo-labels generated by the teacher model.

[0027] As a preferred embodiment of the first aspect mentioned above, in step S3, the structure of the student model is the same as the structure of the end-side multimodal pedestrian re-identification network.

[0028] Secondly, the present invention provides a pedestrian re-identification system based on end-to-cloud collaborative multimodal fusion, comprising: an end-side device and a cloud-side device, wherein the end-side device and the cloud-side device are connected for communication, for implementing a pedestrian re-identification method based on end-to-cloud collaborative multimodal fusion as described in any of the solutions in the first aspect above.

[0029] As a preferred embodiment of the second aspect described above, the end-side device includes a computer.

[0030] As a preferred embodiment of the second aspect above, the cloud-side device includes a server.

[0031] As a preferred embodiment of the second aspect above, the end-side device further includes a camera deployed in a real-world scene for acquiring pedestrian images that require pedestrian re-identification.

[0032] Compared with the prior art, the present invention has the following advantages:

[0033] This invention employs a multimodal fusion-based pedestrian re-identification method with edge-cloud collaboration to achieve high accuracy in real-world changing scenarios. For example, when using cameras for pedestrian tracking, the effectiveness of pedestrian re-identification in different environments such as rainy, foggy, and sunny days needs to be considered. This invention can effectively improve the accuracy of pedestrian re-identification in different scenarios.

[0034] The method of this invention first uses an uncertainty-guided sampling module to determine the uncertainty of the pedestrian image detected by the camera. It then determines whether the image requires cloud-based computing power for pedestrian re-identification. If the scene changes little, facial and gait features are directly extracted on the device side and inferred through a multimodal pedestrian re-identification network to obtain the recognition result. If, due to continuous scene changes, the uncertainty-guided sampling module determines that the device side cannot perform recognition, the extracted facial and gait images are transmitted to the cloud side.

[0035] To ensure accuracy across different scenarios, the cloud-based teacher model, with strong generalization ability, transfers knowledge to the student model, which has weaker generalization ability. The student model learns visual cues and then transmits these learned cues to the edge model for inference. This invention focuses on addressing the challenge of pedestrian re-identification in the security field, introducing for the first time a multimodal edge-cloud collaborative mode based on cue fine-tuning to achieve pedestrian re-identification technology. The method of this invention combines edge-side processing with cloud-based collaboration, utilizing cue fine-tuning to optimize the student model, thereby improving pedestrian re-identification performance in various scenarios. This invention is the first to propose a multimodal fusion-based pedestrian re-identification method based on edge-cloud collaboration, applied to pedestrian re-identification in the security field. By utilizing multimodal features of face and gait, the accuracy of recognition is improved. The method of this invention effectively improves the generalization ability of lightweight edge models, further enhancing the accuracy of pedestrian re-identification in diverse scenarios. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the overall solution flow of the present invention;

[0037] Figure 2 This is a schematic diagram of the edge-cloud collaborative design scheme of the present invention;

[0038] Figure 3 This is a schematic diagram of the multimodal pedestrian re-identification network of the present invention;

[0039] Figure 4 This is a system block diagram of the present invention. Detailed Implementation

[0040] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in the various embodiments of the present invention can be combined accordingly without mutual conflict.

[0041] like Figure 1As shown, in a preferred embodiment of the present invention, the above-mentioned pedestrian re-identification method based on edge-cloud collaborative multimodal fusion includes the following steps S1 to S5. The specific implementation process of each step will be described in detail below.

[0042] S1: The pedestrian image is input to the uncertainty-guided sampling module on the edge side, and the output is a judgment result on whether the pedestrian image needs to be uploaded to the cloud side for processing. After the edge side extracts the face image and gait image of the pedestrian image, if the judgment result is that the pedestrian image needs to be uploaded to the cloud side, then the face image and gait image are uploaded to the cloud side for processing. If the judgment result is that the pedestrian image does not need to be uploaded to the cloud side, then the multimodal pedestrian re-identification network trained on the edge side is used to process the face image and gait image.

[0043] It should be noted that in step S1 of this invention, after the uncertainty-guided sampling module calculates the uncertainty of the pedestrian image, if the uncertainty of the pedestrian image is greater than a preset uncertainty threshold, the pedestrian image needs to be uploaded to the cloud; wherein, the uncertainty V of the pedestrian image... ync The calculation method is as follows:

[0044]

[0045] In the formula, μ represents the average output probability of the uncertainty-guided sampling module in n predictions; n represents the preset number of predictions; (p i (y|x) represents the output probability of the uncertainty-guided sampling module during the i-th prediction.

[0046] In this embodiment, due to the limited communication bandwidth between the cloud and the edge, it is impractical for the edge to transmit all extracted target images containing pedestrians to the cloud. Therefore, this invention designs an uncertainty-guided sampling module to filter out pedestrian images that only require cloud-side computing power for processing.

[0047] Specifically, this invention selects pedestrian images to be transmitted through uncertainty estimation. Given a pedestrian image, the Dropout method is used to predict the output probability p of the uncertainty-guided sampling module for n iterations. i (y|x), calculate the variance of these output probabilities as the uncertainty V of the pedestrian image. unc The calculation method is as described above and will not be repeated here. If the uncertainty V of the pedestrian image... unc If the uncertainty exceeds a certain threshold, the face and gait images extracted from the pedestrian images will be uploaded to the cloud. Through this uncertainty-guided sampling module, the edge device can filter out pedestrian images that require cloud-based computing power.

[0048] S2: On the edge, the face image and gait image are input into the edge multimodal pedestrian re-identification network. In the edge multimodal pedestrian re-identification network, the feature extraction network first extracts the face features of the face image and the gait features of the gait image. The face features and gait features are input into the pre-trained context-adaptive multimodal fusion module to obtain context-aware adaptive fusion features. The adaptive fusion features are then passed through a fully connected layer and a softmax function to classify the adaptive fusion features, and the classification result of the edge pedestrian image is obtained.

[0049] It should be noted that, in step S2 of this invention, the specific process of obtaining the context-aware adaptive fusion feature is as follows:

[0050] S21. Utilize the context-adaptive multimodal fusion module to output the global individual attention weights of the face. Global individual attention weights for gait Calculate the weighted global attention weight α for facial features and the weighted global attention weight β for gait features:

[0051]

[0052] S22. Multiply the face feature by its weighted global attention weight α to obtain the weighted face feature. Multiply the gait feature by its weighted global attention weight β to obtain the weighted gait feature. Add the weighted face feature and the weighted gait feature to generate the context-aware adaptive fusion feature Z. The specific calculation method is as follows:

[0053] Z = αF + βG

[0054] Where G represents facial features, αF represents weighted facial features; G represents gait features, and βG represents weighted gait features.

[0055] In this embodiment, as Figure 3 The diagram shows the architecture of the multimodal person re-identification network on the edge. In this network, a feature extraction network first extracts facial features from the face image and gait features from the gait image. In this invention, the feature extraction network can be selected according to actual needs, as long as it can extract facial features and gait features separately. In this embodiment, the feature extraction network uses ConvLSTM (…). Figure 3 The implementation of the convlstm module is existing technology and will not be elaborated further. After feature extraction, the pre-trained context-aware adaptive multimodal fusion module outputs the global individual attention weights of the face. Global individual attention weights for gait Next, we utilize the global individual attention weights of the face. Global individual attention weights for gait The weighted global attention weights α for facial features and β for gait features are calculated as described previously and will not be repeated here. These weighted global attention weights are then applied to their respective features—that is, facial features and gait features are multiplied by their respective weighted global attention weights and adaptively fused—to generate context-aware adaptive fused features. The adaptive fused feature Z is then passed to a fully connected layer and classified using a softmax function. Based on the provided K pedestrian ID categories, the column vector R, normalized by the softmax function, is then used to determine the pedestrian category identifier (pedestrian ID) via ID(Z) = argmax(R).

[0056] S3: The cloud side acquires the face and gait images output from the edge side and jointly trains and optimizes the pre-trained student and teacher models on the cloud side. In each iteration of the joint training and optimization process, the face cue matrix and face image updated in the previous iteration are added together to obtain a new face image. The gait cue matrix and gait image updated in the previous iteration are added together to obtain a new gait image. The new face image and new gait image are used together as input to the teacher model updated in the previous iteration. After updating the parameters of the teacher model, face cue matrix, and gait cue matrix, the pseudo-labels output by the updated teacher model are used as the real labels of the student model. The new face image, new gait image, and the real labels of the student model are input into the student model for training. The parameters of the teacher model, face cue matrix, and gait cue matrix are fixed, and the parameters of the student model are updated separately. The teacher model and student model each use a loss function for parameter updates.

[0057] In step S3 of this invention, based on the uncertainty estimation in step S1, when the uncertainty of the pedestrian image exceeds a certain uncertainty threshold, the face image and gait image extracted from the pedestrian image are uploaded to the cloud. Then, the cloud acquires the face image and gait image output from the endpoint and uses a multimodal cue-based learning model distillation method to jointly train and optimize the teacher model and student model. During the training of the teacher and student models, the face image and gait image extracted in step S1 are also used to update the model parameters.

[0058] It should be noted that in step S3 of this invention, a knowledge transfer process based on cue learning is introduced on the cloud side. Specifically, the cloud-side cue learning method is knowledge transfer. A visual cue learning module (VPL) is deployed on the cloud side for jointly training and optimizing the student model and the teacher model. The cloud-side teacher model has a large number of parameters, is trained on data from multiple domains, and has strong generalization ability, while the cloud-side student model has a small number of parameters, the same as the number of parameters in the edge-side multimodal person re-identification network, and therefore has limited generalization ability. Therefore, this invention optimizes visual cues (gait cue matrix and face cue matrix), aligns the features of the teacher model and the student model, and uses the teacher model to generate pseudo-labels for the student model. These pseudo-labels are affected by the student model loss L. stu The penalty is to ensure that the representations of the teacher model and the student model remain consistent.

[0059] Specifically, the teacher model generates predictions based on the reconstructed face and gait images. Due to its stronger generalization ability, the generated pseudo-labels are of high quality. To further improve the accuracy of the pseudo-labels, this invention further filters out predictions with confidence levels exceeding a threshold and uses them as input to the student model. Then, by inputting the reconstructed face and gait images and their corresponding pseudo-labels into the student model for training, knowledge transfer from teacher to student is achieved.

[0060] It should also be noted that this invention incorporates the concept of cue learning when reconstructing the face and gait images. Cue learning is a machine learning method commonly used in transfer learning or knowledge transfer contexts. It helps the model learn and generalize better to new tasks or domains by providing additional information or guidance. Cue learning originates from the field of natural language processing. Due to the lightweight nature of the current edge model and its limited computational power, this invention employs cue learning to improve the generalization ability of the student model.

[0061] Specifically, for the face image and gait image extracted in step S1, a pixel-level face cue matrix is ​​introduced. and gait cue matrix Face prompt matrix and face image x f After adding them together, a new face image is obtained. Use the gait cue matrix and gait image x g After addition, a new gait image is obtained.

[0062]

[0063] Among them, the face prompt matrix and gait cue matrix All of them are learnable.

[0064] It should be noted that in step S3 of this invention, the teacher model and student model are jointly optimized on the cloud side. For the teacher model, it has already been used in the source domain data. Pre-training was performed on it. This represents a face image in the source domain data; Represents the gait image in the source domain data; y s This indicates the source domain data label.

[0065] After the teacher model is pre-trained, new face images will be added with two types of cue matrices. and new gait images As target data, this invention uses adversarial alignment loss and supervised loss to construct the loss of the teacher model, thereby optimizing the parameters of the teacher model and the face cue matrix and gait cue matrix. The teacher model is first trained on a large-scale source domain dataset, and optimizing the adversarial alignment loss makes it difficult for the teacher model's domain classifier to distinguish between the two cue matrices (face cue matrix and gait cue matrix). and gait cue matrix The information comes from either the target domain or the source domain, thereby facilitating the teacher model's learning of the face cue matrix. and gait cue matrix Simultaneously, the generalization ability of the teacher model was optimized. Specifically, the loss L of the teacher model... tea as follows:

[0066]

[0067] Where λ represents the weight of the adversarial alignment loss; the first term L d G represents the adversarial alignment loss; d It represents the domain classifier of the teacher model; F represents the feature extractor of the teacher model; θ tea Indicates the teacher model parameters; y d For the domain label, the second term L represents the supervised loss; X tea This represents the output of the teacher model, i.e., the classification result predicted by the teacher model.

[0068] The pseudo-label y output by the student model through the teacher model p To optimize, the loss L of the student model stu As shown in the following formula, through this loss function L stu The student model can effectively learn information from the corresponding scene and improve pedestrian re-identification performance in that scene. By minimizing the loss of the teacher model and the student model, the teacher model and the student model are jointly trained and optimized, ultimately yielding a face cue matrix, gait cue matrix, and student model with better performance.

[0069]

[0070] Among them, L f Represents the supervised classification loss; θ stu X represents the student model parameters; stu This represents the output of the student model, i.e., the classification result predicted by the student model.

[0071] It should also be noted that in step S3 of this invention, the teacher model and student model structures can be any form of neural network model, as long as they have a feature extraction structure and can perform image classification tasks. In this embodiment, the structure of the student model is the same as the structure of the multimodal person re-identification network deployed on the edge, but their parameters are different.

[0072] S4: After the teacher model and student model are trained, the cloud side sends the trained student model parameters, as well as the face cue matrix parameters and gait cue matrix parameters, to the edge side. The edge side uses the trained student model to process the face image and gait image obtained in S1. The edge side adds the face image obtained in S1 to the trained face cue matrix to obtain the face image to be classified, and adds the gait image obtained in S1 to the trained gait cue matrix to obtain the gait image to be classified. The face image and gait image to be classified are input into the trained student model to obtain the classification result of the pedestrian image on the edge side.

[0073] It should be noted that in step S4 of this invention, after the pedestrian image captured by the camera is calculated and judged by the uncertainty-guided sampling module, if it is necessary to upload the pedestrian image to the cloud for processing, the visual cues (face cues matrix and gait cues matrix) learned by the cloud side are sent back to the edge side. At the same time, the student model learned by the cloud side is also sent to the edge side. The learned face cues matrix and gait cues matrix are added to the input of the corresponding student model on the edge side to obtain the output result.

[0074] Step S4 of this invention realizes the transmission of cues from the cloud side to the edge side, which is a cloud-to-edge knowledge transfer process. The visual cues learned on the cloud side are transmitted to the edge side and applied to the face image and gait image obtained in step S1. The lightweight student model parameters are also transmitted to the edge side, and inference is performed through the student model deployed on the edge side. At this time, the visual cues learned on the cloud side can effectively improve the performance degradation caused by environmental changes, enabling the lightweight student model on the edge side to have good generalization performance and high accuracy in pedestrian re-identification in different scenarios.

[0075] In summary, such as Figure 2As shown, this invention first utilizes the uncertainty-guided sampling module on the edge to filter pedestrian images that need to be uploaded to the cloud for processing. The processing flow in step S2 is implemented by the edge inference module. A multimodal pedestrian re-identification network based on face and gait is used to calculate the weighted average of facial and gait features. After adaptive fusion, context-aware adaptive fusion features are generated. Feature classification is performed through fully connected layers and a softmax function to obtain the classification results of the edge pedestrian images. Then, knowledge transfer learning prompts are implemented on the cloud-side teacher-student models (teacher model and student model), i.e., knowledge transfer based on prompt learning, enhancing the generalization ability of the edge student model. Visual prompts are optimized in the cloud, and knowledge transfer between the teacher and student models is achieved. The performance of the edge student model is improved by jointly optimizing the teacher and student models. The visual prompts learned on the cloud are transmitted to the edge and applied to face and gait images to improve the accuracy of pedestrian re-identification.

[0076] It should also be noted that the pedestrian re-identification method based on edge-cloud collaborative multimodal fusion in the above embodiments can essentially be executed by a computer program or module. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a pedestrian re-identification system based on edge-cloud collaborative multimodal fusion, corresponding to the pedestrian re-identification method based on edge-cloud collaborative multimodal fusion provided in the above embodiments, such as... Figure 4 As shown, it includes:

[0077] The device on the edge and the device on the cloud, and the communication connection between the edge and the cloud, are used to implement a pedestrian re-identification method based on edge-cloud collaborative multimodal fusion as described in the above embodiments. The edge device includes a small computer, and the cloud device includes a large server. Additionally, the edge device may include a camera deployed in a real-world scene to acquire pedestrian images that require re-identification.

[0078] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A pedestrian re-identification method based on edge-cloud collaborative multimodal fusion, characterized in that, Includes the following steps: S1. The pedestrian image is input to the uncertainty-guided sampling module on the edge side, and the output is the judgment result of whether the pedestrian image needs to be uploaded to the cloud side for processing. After the edge side extracts the face image and gait image of the pedestrian image, if the judgment result is that the pedestrian image needs to be uploaded to the cloud side, then the face image and gait image are uploaded to the cloud side for processing. If the judgment result is that the pedestrian image does not need to be uploaded to the cloud side, then the multimodal pedestrian re-identification network trained on the edge side is used to process the face image and gait image. S2. On the edge, the face image and gait image are input into the edge multimodal pedestrian re-identification network. In the edge multimodal pedestrian re-identification network, the feature extraction network first extracts the face features of the face image and the gait features of the gait image. The face features and gait features are input into the pre-trained context-adaptive multimodal fusion module to obtain context-aware adaptive fusion features. The adaptive fusion features are then passed through a fully connected layer and a softmax function to classify the adaptive fusion features, and the classification result of the edge pedestrian image is obtained. S3. The cloud side acquires the face image and gait image output from the terminal side, and jointly trains and optimizes the pre-trained student model and the pre-trained teacher model on the cloud side. In each iteration of the joint training and optimization process, the face cue matrix and face image updated in the previous iteration are added together to obtain a new face image. The gait cue matrix and gait image updated in the previous iteration are added together to obtain a new gait image. The new face image and the new gait image are used as input to the teacher model updated in the previous iteration. After updating the parameters of the teacher model, the face cue matrix, and the gait cue matrix, the pseudo-label output by the updated teacher model is used as the real label of the student model. The new face image, the new gait image, and the real label of the student model are input into the student model for training. The parameters of the teacher model, the face cue matrix, and the gait cue matrix are fixed, and the parameters of the student model are updated separately. The teacher model and the student model each use a loss to update their parameters. S4. After the teacher model and student model are trained, the cloud side sends the trained student model parameters, as well as the face cue matrix parameters and gait cue matrix parameters, to the edge side. The edge side uses the trained student model to process the face image and gait image obtained in S1. The edge side adds the face image obtained in S1 to the trained face cue matrix to obtain the face image to be classified, and adds the gait image obtained in S1 to the trained gait cue matrix to obtain the gait image to be classified. The face image and gait image to be classified are input into the trained student model to obtain the classification result of the pedestrian image on the edge side.

2. The pedestrian re-identification method based on edge-cloud collaborative multimodal fusion according to claim 1, characterized in that, In step S1, after the uncertainty-guided sampling module calculates the uncertainty of the pedestrian image, if the uncertainty of the pedestrian image is greater than a preset uncertainty threshold, the pedestrian image needs to be uploaded to the cloud; wherein, the uncertainty V of the pedestrian image... unc The calculation method is as follows: In the formula, μ represents the average output probability of the uncertainty-guided sampling module in n predictions; n represents the preset number of predictions; (p i (y|x) represents the output probability of the uncertainty-guided sampling module during the i-th prediction.

3. The pedestrian re-identification method based on edge-cloud collaborative multimodal fusion according to claim 1, characterized in that, In step S2, the specific process of obtaining the context-aware adaptive fusion features is as follows: S21. Utilize the context-adaptive multimodal fusion module to output the global individual attention weights of the face. Global individual attention weights for gait Calculate the weighted global attention weight α for facial features and the weighted global attention weight β for gait features: S22. Multiply the face feature by its weighted global attention weight α to obtain the weighted face feature. Multiply the gait feature by its weighted global attention weight β to obtain the weighted gait feature. Add the weighted face feature and the weighted gait feature to generate the context-aware adaptive fusion feature Z. The specific calculation method is as follows: Z = αF + βG Where F represents facial features, αF represents weighted facial features; G represents gait features, and βG represents weighted gait features.

4. The pedestrian re-identification method based on edge-cloud collaborative multimodal fusion according to claim 1, characterized in that, In step S3, the loss L of the teacher model tea for: Where λ represents the weight of the adversarial alignment loss; L d G represents the adversarial alignment loss; d represents the domain classifier of the teacher model; F represents the feature extractor of the teacher model; This represents a face image in the source domain data; Represents the gait image in the source domain data; θ tea Indicates the teacher model parameters; This represents a new facial image; Represents a new gait image; y d For domain labels; L represents supervised loss; X represents domain label ... tea This represents the output of the teacher model; y s This indicates the source domain data label.

5. The pedestrian re-identification method based on edge-cloud collaborative multimodal fusion according to claim 4, characterized in that, In step S3, the loss L of the student model stu for: Among them, L f Represents the supervised classification loss; θ stu X represents the student model parameters; stu This represents the output of the student model; y p This represents the pseudo-labels generated by the teacher model.

6. The pedestrian re-identification method based on edge-cloud collaborative multimodal fusion according to claim 1, characterized in that, In step S3, the structure of the student model is the same as that of the end-side multimodal pedestrian re-identification network.

7. A pedestrian re-identification system based on edge-cloud collaborative multimodal fusion, characterized in that, include: The device on the edge and the device on the cloud, and the communication connection between the device on the edge and the device on the cloud, are used to implement the pedestrian re-identification method based on edge-cloud collaborative multimodal fusion as described in any one of claims 1 to 6.

8. The pedestrian re-identification system based on edge-cloud collaborative multimodal fusion as described in claim 7, characterized in that, The end-side device includes a computer.

9. The pedestrian re-identification system based on edge-cloud collaborative multimodal fusion as described in claim 7, characterized in that, The cloud-side equipment includes servers.

10. The pedestrian re-identification system based on edge-cloud collaborative multimodal fusion as described in claim 8, characterized in that, The edge device also includes a camera deployed in a real-world scenario to capture pedestrian images that require pedestrian re-identification.

Images