A cross-modal gait recognition method and system from event cameras to visual silhouettes

Abstract

The application relates to a cross-modal gait recognition method and system from an event camera to a visual profile, which comprises the following steps: acquiring asynchronous event stream data collected by an event camera and performing voxel grid coding, obtaining the transverse center point position of an estimated human body through boundary detection and head center positioning, cutting a human body region subgraph based on the center point position to form voxel data; taking the voxel data and acquired RGB image data as inputs of a shared backbone network in a gait recognition model to extract modal features consistent in a feature space; based on the modal features, sequentially performing time pooling, horizontal pyramid pooling, full connection, standardization and classification in the gait recognition model to obtain a finally predicted gait identity category, wherein the gait recognition model adopts a saliency guided training strategy and is trained based on a classification loss, a triple loss, a modal consistency loss and an adversarial domain discrimination loss for cross-domain adaptation.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a cross-modal gait recognition method and system that uses an event camera to visualize the gait profile. Background Technology

[0002] Neuromorphic vision sensors, also known as event cameras, are a new type of bio-inspired vision sensor that mimics the human retina's response to dynamic scenes. These sensors trigger events by detecting changes in light intensity at each pixel location that exceed a certain threshold, thus generating an asynchronous, sparse stream of events. Unlike traditional RGB cameras that rely on a fixed frame rate to acquire a complete image, event cameras generate irregular, asynchronous sets of events. This characteristic gives them advantages such as high temporal resolution (microseconds), high dynamic range (140dB), low latency, and low power consumption, making them particularly suitable for challenging scenarios such as high-speed motion, low-light conditions, or strong light interference.

[0003] In recent years, event cameras have been widely used in optical flow estimation, object detection, visual inertial navigation, object tracking, and pose recognition, achieving a series of cutting-edge results. However, in the specific field of gait recognition, event cameras face problems such as data sparsity and lack of texture information, making it difficult to directly adapt to existing recognition algorithms based on RGB images. Furthermore, event modalities and RGB modalities have inherent structural differences in data representation, spatial distribution, and semantic features. Achieving feature alignment and recognition fusion across modalities has become a critical technical challenge that urgently needs to be addressed.

[0004] Chinese patent application publication number CN114973422A discloses a gait recognition method based on point cloud feature encoding of 3D human body modeling. By employing the SMPL model and point cloud feature encoding technology in gait recognition, and combining point feature, region feature, and temporal feature encoding, it solves the problem of high degree of freedom and low information dimensionality in skeleton output space, achieving more accurate gait and emotion recognition and breaking through the limitations of existing technologies. However, its gait recognition relies on a parametric human body model, resulting in high implementation costs.

[0005] Against this backdrop, how to achieve alignment with the structural information of RGB images while taking into account the advantages of high spatiotemporal resolution of event cameras and improving the accuracy of cross-modal gait recognition has become a technical bottleneck and research hotspot in the current field. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art by providing a cross-modal gait recognition method and system based on event camera to visual contours, so as to solve or partially solve problems such as inconsistency between the modalities of event cameras and RGB cameras, large differences in feature distribution, and low recognition accuracy.

[0007] The objective of this invention can be achieved through the following technical solutions:

[0008] One aspect of the present invention provides a cross-modal gait recognition method from an event camera to a visual contour, comprising the following steps:

[0009] Asynchronous event stream data collected by the event camera is acquired and voxel mesh encoding is performed. The estimated lateral center point position of the human body is obtained through boundary detection and head center localization. Based on the center point position, human body region sub-images are cropped to form voxel data.

[0010] The voxel data and the acquired RGB image data are used as inputs to the shared backbone network in the gait recognition model to extract modal features consistent with the feature space.

[0011] Based on the modal features, the gait recognition model sequentially performs temporal pooling, horizontal pyramid pooling, fully connected layers, normalization, and classification to obtain the final predicted gait identity category.

[0012] The gait recognition model is trained on RGB image data using a saliency-guided training strategy, based on classification loss, triplet loss, modality consistency loss, and cross-domain adaptive adversarial discriminant loss.

[0013] As a preferred technical solution, the process of acquiring asynchronous event stream data collected by the event camera and performing voxel mesh encoding includes the following steps:

[0014] Asynchronous event stream data acquired by the event camera is acquired and voxel-encoded. Events generated within a continuous time period are divided into multiple small segments in chronological order. Based on the number of events at each pixel position in each time segment, a three-dimensional data structure V∈R containing temporal, spatial, and polarity information is constructed. C×T×H×W As tensor data, C represents the number of polarity channels of the event, T represents the number of time channels, and H and W represent the spatial height and width of the event image.

[0015] As a preferred technical solution, the process of obtaining the estimated lateral center point position of the human body through boundary detection and head center localization includes the following steps:

[0016] Based on the voxel data aggregated from the asynchronous event stream data, the image is scanned line by line from the top. When a line contains a or more consecutive non-zero event pixels, it is determined to be an entry into the human activity area, and this line is taken as the upper boundary.

[0017] Based on the voxel data aggregated from the asynchronous event stream data, the image is scanned line by line from the bottom upwards. In response to a line containing b or more non-zero event pixels, that line is taken as the lower boundary.

[0018] Based on the upper boundary, the position of the horizontal center point of the human body is estimated by calculating the centroid of the event pixels in the adjacent rows of the boundary.

[0019] As a preferred technical solution, a = W × 2%, b = W × 2%, where W is the image width.

[0020] As a preferred technical solution, the process of cropping the human body region sub-image based on the center point position to form voxel data includes the following steps:

[0021] Based on the horizontal center point position, the upper and lower boundaries of the image, a human body region sub-image of a fixed size is obtained by cropping. The size of the cropping window is configured according to the experimental settings.

[0022] As a preferred technical solution, the saliency-guided training strategy includes the following steps:

[0023] By calculating the pixel differences between the current RGB image frame and another random RGB image frame, a mask map of the changed areas in the current RGB image frame compared to the random RGB image frame is obtained;

[0024] The mask image is smoothed to obtain a saliency image;

[0025] Based on the saliency of the current RGB image frame and multiple random RGB image frames, the original current RGB image frame is weighted, with higher weights assigned to regions of significant change.

[0026] The weights are controlled by a control factor, which gradually decreases during the training process. In the early stages of training, the model focuses strongly on salient regions, while in the later stages of training, the guidance is gradually relaxed to avoid the model overfitting to local regions.

[0027] As a preferred technical solution, the modal consistency loss calculates the difference in category distribution of modal features between the event modality and the RGB modality. By minimizing this difference, the model learns the shared features between the two modalities, achieving soft alignment. The adversarial domain discriminant loss inverts the gradient signal and transmits it back to the backbone network, making the backbone network more inclined to learn modality-independent features when updating parameters.

[0028] As a preferred technical solution, the training objective function composed of the classification loss, triplet loss, modality consistency loss, and cross-domain adaptive adversarial discriminant loss is as follows:

[0029] L total=L ce +λ1L triplet +λ2L mutual +λ3L adversarial

[0030]

[0031]

[0032] Among them, L total To train the objective function, L ce L triplet L mutual L adversarial These are the classification loss, triplet loss, modality consistency loss, and adversarial domain discriminant loss, respectively, where λ1, λ2, and λ3 are coefficients, and y i , Let represent the true class of the i-th sample and the probability output by the gait recognition model, respectively. Let α represent the feature vector of the anchor sample in the i-th sample, the feature vector of the positive sample belonging to the same class as the anchor sample, and the feature vector of the negative sample belonging to a different class from the anchor sample (i.e., the current sample, the positive sample, and the negative sample), respectively. α also represents the minimum margin between the positive and negative samples. These represent the feature mean values ​​of RGB and event modalities within a batch, respectively, where N is the total number of samples in the batch, and d... i , They respectively represent, [] + denoted as ReLU function, ensuring that the loss value is not negative, λ is the inversion coefficient of gradient inversion in adversarial domain discrimination, p is the current training progress, and γ is a hyperparameter that controls the rate of change of the inversion coefficient.

[0033] Another aspect of the present invention provides a cross-modal gait recognition system from an event camera to a visual profile, for implementing the aforementioned cross-modal gait recognition method from an event camera to a visual profile, the system comprising:

[0034] The data preprocessing and characterization module is used to preprocess and characterize the input asynchronous event stream data and RGB images. For the event stream data, it is converted into structured three-dimensional tensor event voxel data through voxel grid encoding. For the RGB images, it is normalized and cropped.

[0035] The modal feature extraction module is used to extract features from the event voxel data and RGB images through a shared backbone network.

[0036] The feature alignment module achieves cross-modal feature alignment through a domain discriminator and a mutual learning mechanism.

[0037] The gait modeling module first compresses the key frame information in the gait sequence through the temporal pooling module, removes redundant frames and retains discriminative temporal features. Then, the horizontal pyramid pooling module extracts the local spatial features of the gait through multi-scale pooling operations and enhances the ability to perceive fine-grained regional differences.

[0038] The discrimination and classification module maps the gait modeling features through a fully connected layer to form a unified feature space. The features are then normalized using a BNNeck structure. The standardized features are then input into a Softmax classifier to complete the gait identity discrimination.

[0039] In another aspect, an electronic device is provided, including one or more processors, a memory, and one or more programs stored in the memory, the one or more programs including instructions for performing the aforementioned cross-modal gait recognition method from event camera to visual profile.

[0040] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0041] (1) The structural information of the event camera and the RGB image are aligned, resulting in high cross-modal gait recognition accuracy: This invention models asynchronous event stream data as voxel data and enters the shared backbone network with the RGB image for feature extraction. It performs feature representation of input data from different sources at a unified scale to ensure that the feature spaces of the two modalities remain consistent when extracting features. In addition, this invention is trained based on classification loss, triplet loss, modality consistency loss and cross-domain adaptive adversarial domain discrimination loss. The gait recognition model combines a domain discriminator and a mutual learning mechanism. The two work together to ensure that the features from the event camera and the RGB image can be effectively aligned in the shared feature space, thereby improving the cross-domain recognition performance of the model. Furthermore, the saliency-guided training strategy is adopted during the training process of the gait recognition model to automatically generate attention regions and guide the model to focus on the key parts of the human body that change in the image sequence.

[0042] (2) The voxel data obtained from the event stream data fully characterizes the human body region: This invention introduces an image cropping strategy based on human body boundary and head positioning after the event image is encoded, which ensures that subsequent processing focuses on key areas, can eliminate invalid background areas at the top and bottom of the image, and accurately extract the vertical position range of the human body.

[0043] (3) While increasing the attention to salient regions, avoid overfitting: In the salient guidance training strategy of the present invention, the fusion weight is controlled by the control factor. The control factor gradually decays with the training process. In the early stage of training, the model pays strong attention to salient regions. In the later stage of training, the guidance is gradually relaxed to avoid the model overfitting to local regions. This enables the model to automatically discover and emphasize the regions in which the human body plays a major role in the image sequence, and significantly improves the model's perception accuracy and cross-modal recognition ability in complex backgrounds.

[0044] (4) The required data modalities are simple and easy to obtain: the gait recognition of the present invention only requires two types of data, event stream data and RGB image data, to complete the recognition. Attached Figure Description

[0045] Figure 1 This is a flowchart of the cross-modal gait recognition method from event camera to visual contour in the embodiment;

[0046] Figure 2 This is a schematic diagram illustrating the calculation of the loss function in the gait recognition model in the embodiment;

[0047] Figure 3 This is a schematic diagram of a cross-modal gait recognition system from an event camera to a visual profile, as described in the embodiment.

[0048] Figure 4 This is a schematic diagram of the electronic device in the embodiment. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0050] Example 1

[0051] To address the aforementioned problems in the existing technology, this embodiment provides an event camera and RGB camera gait recognition method and system capable of supporting cross-modal recognition tasks. It solves problems such as inconsistency in modalities between event cameras and RGB cameras, large differences in feature distribution, and low recognition accuracy in the prior art. By combining complementary information from event cameras and RGB cameras and employing a unified feature modeling and modal alignment mechanism, it can improve recognition accuracy and generalization ability in cross-modal gait recognition tasks. Furthermore, the method is modular, has low latency, and is highly deployable, making it suitable for identity recognition and intelligent perception tasks in various real-world scenarios.

[0052] The problems solved by this embodiment include:

[0053] (1) To address the problem of modeling the spatiotemporal characteristics of event stream data, a method for representing event data based on voxel mesh representation is provided. By transforming asynchronous event streams into a three-dimensional voxel mesh structure, the temporal and spatial distribution information of the event data is preserved and used as the input of event modalities for subsequent unified modeling of the network.

[0054] (2) To address the problem of information region imbalance in the RGB modality, a saliency-guided training strategy is proposed. This strategy guides the network to focus on key regions in the image, such as the movement parts of the human limbs, thereby improving the model's ability to discriminate gait information in the RGB modality.

[0055] (3) A modality-sharing backbone network structure is constructed, using a lightweight ResNet9 network as the backbone network to process both event modality and RGB modality inputs. The backend of the network sequentially integrates temporal pooling (TP), horizontal pyramid pooling (HPP), fully connected layers, and BNNecks modules to enhance the multi-scale spatiotemporal modeling capability of gait features.

[0056] (4) A modality alignment module is designed, and a domain classifier is introduced. Adversarial training is used to reduce the differences in feature distribution between different modalities. At the same time, a mutual learning mechanism is designed so that the event modality and the RGB modality can guide and supplement each other's information during network training, thereby enhancing the model's cross-modal generalization ability.

[0057] (5) Finally, the extracted unified features are output as recognition results through a fully connected layer and a gait identity classifier. During the testing phase, it supports individual or joint input of event modality or RGB modality to realize a variety of cross-modal recognition applications.

[0058] See Figure 1 This method mainly includes the following steps:

[0059] Step S1, Event Stream Encoding and Salience Guidance Strategy. Specifically, Step S1 includes steps S101-S105.

[0060] Step S101, event stream encoding.

[0061] Voxel grid encoding is employed to process asynchronous event stream data acquired by an event camera. Since event stream data is temporally sequential and sparse, and unlike the fixed frames of conventional images, a three-dimensional data structure containing temporal, spatial, and polarity information is constructed by dividing events generated within a continuous time period into multiple smaller segments in chronological order, and counting the number of events at each pixel location within each segment. This structure can be represented as V∈R C×T×H×W Where C represents the number of polarity channels of the event (e.g., positive to negative change), T represents the number of time channels, i.e., the number of time periods into which the event is divided, and H and W represent the spatial height and width of the event image, respectively. This encoding method can transform sparse, asynchronous event streams into tensor data with an image appearance, facilitating unified processing by subsequent feature extraction modules.

[0062] Step S102: Boundary detection and head center localization.

[0063] To further improve the accuracy and robustness of recognition, an image cropping strategy based on human body boundaries and head localization was introduced after event image encoding to ensure that subsequent processing focuses on key areas. This strategy includes the following steps:

[0064] 1. Upper boundary detection: Starting from the top of the image, scan downwards line by line. When a line contains a consecutive non-zero event pixels, it is considered to have entered the human activity area, and the first line is taken as the upper boundary position.

[0065] 2. Lower Boundary Detection: Starting from the bottom of the image, scan upwards line by line. Once a line contains b or more non-zero event pixels, it is considered the lower boundary. This standard can effectively identify the lower part of the human body (such as leg movements) where events are relatively concentrated and determine the lower boundary position.

[0066] Where a = W × 2%, b = W × 2%, and W is the image width.

[0067] In this way, invalid background areas at the top and bottom of the image can be eliminated, and the vertical position range of the human body can be accurately extracted.

[0068] 3. Head Center Location: Near the defined upper boundary, further analyze the horizontal distribution of event pixels. Since the human head usually generates event responses first and presents a compact and concentrated shape in the contour map, the horizontal center point of the human body is estimated by calculating the centroid of event pixels in several rows near the boundary. This position is used as the center reference point for subsequent cropping.

[0069] Step S103, window cropping.

[0070] Based on the above localization steps, a fixed-size human body region sub-image is cropped according to the horizontal center point and vertical boundary. The size of this window is uniformly configured according to the experimental settings to ensure consistency in human image size and alignment among all samples, thereby improving the consistency and stability of subsequent feature extraction.

[0071] Step S104: To improve the recognition effect of human body contours in RGB image modalities, a saliency-guided training strategy is designed to automatically generate attention regions and guide the model to focus on key parts of the human body that change in the image sequence. The specific strategy is as follows:

[0072] 1. Inter-frame difference calculation: First, random RGB image frames are compared pairwise to calculate the pixel difference between the current frame and another random frame, thereby generating a mask map of the changed areas in the current frame compared to other frames. These changed areas usually correspond to key parts of human movement changes, such as arm swings and leg movements, and have high discriminative value in gait recognition.

[0073] 2. Convolution Smoothing: In order to reduce noise and enhance the continuity of salient regions, convolution operations are used to smooth the mask image of each frame, generating a smoother salient image, thereby more accurately emphasizing these key regions.

[0074] 3. Weighting Operation: The generated saliency map is used to weight the original image, assigning higher weights to regions with significant changes. During neural network training, the model focuses more attention on these representative areas, thereby improving the accuracy of gait recognition.

[0075] Step S105, Dynamic Control Factor

[0076] To adapt to the dynamic adjustments during training, the saliency guidance strategy also introduces a gradually decaying control factor. This factor ensures that the model strongly focuses on salient regions in the early stages of training, while gradually relaxing the guidance in the later stages to prevent the model from overfitting to local regions. This dynamic adjustment mechanism effectively balances the model's focusing ability with its overall generalization ability.

[0077] This strategy can automatically identify and highlight the regions in which the human body plays a major role in an image sequence, significantly improving the model's perception accuracy and cross-modal recognition capabilities in complex backgrounds.

[0078] Step S2, extracting the backbone features.

[0079] To achieve cross-modal gait recognition, voxel data acquired by event cameras and traditional RGB image data are processed through the same shared backbone network. Specifically, data from both modalities (i.e., event voxels and RGB images) are fed into a lightweight convolutional neural network, ResNet9, where feature extraction is performed. This approach enables uniform-scale feature representation of input data from different sources, ensuring consistency in the feature spaces of the two modalities during feature extraction. This consistency is crucial for subsequent cross-modal alignment tasks, as effective alignment between different modalities can only be achieved within the same feature space, thereby improving the accuracy and robustness of cross-modal recognition.

[0080] By sharing networks, potential differences in feature representations between different modalities can be eliminated, allowing features from both modalities to be processed and optimized in the same space. This approach not only improves the training efficiency of the network but also helps the model learn the intrinsic connections between modalities, providing more accurate feature representations for the final gait recognition task.

[0081] ResNet9 is a simplification of the traditional ResNet model, employing a smaller depth and fewer computational layers.

[0082] The ResNet9 structure in this embodiment mainly consists of the following parts:

[0083] 1. Initial convolutional layer and BatchNorm layer:

[0084] conv1: The input image first passes through a 3x3 convolutional layer, which maps the number of input channels from 1 to 64 output channels, preparing for subsequent feature extraction.

[0085] bn1: After convolution, the feature maps are standardized by the BatchNorm layer to ensure that different features maintain a consistent distribution in different training processes, thus promoting the stability of the training process.

[0086] 2. Residual Blocks:

[0087] The core of the network consists of four residual blocks, each containing multiple convolutional layers and skip connections. The output of each residual block is determined not only by the computation result of the current block but also by the direct transmission of the input signal. This avoids the layer-by-layer attenuation of information in deep networks, ensuring that deep networks can learn effectively.

[0088] The number of output channels in each residual block gradually increases, forming a deep feature extraction process. The number of channels are 64, 128, 256, and 512, respectively.

[0089] 3. Pooling and downsampling:

[0090] MaxPooling: To further reduce the spatial size of the image and enhance the extraction of high-level semantic features, ResNet9 applies max-pooling operations after certain layers. Pooling layers help improve the expressive power of features and reduce computational cost.

[0091] 4. Output layer:

[0092] Finally, after feature extraction from four residual blocks, the network generates feature maps with high semantic representation. These feature maps are then used as input for subsequent tasks such as cross-modal alignment and gait modeling.

[0093] Step S3, cross-modal alignment.

[0094] To achieve cross-modal feature alignment, this embodiment designs two key modules: a domain classifier and a mutual learning mechanism. These two modules work together to ensure that features from the event camera and the RGB image can be effectively aligned in a shared feature space, thereby improving the model's cross-domain recognition performance.

[0095] 1. Domain discriminator.

[0096] In cross-modal tasks, event modality and RGB modality often exhibit significant feature differences. To overcome this issue, a domain classifier module is introduced. Figure 2 The DANN module in the dataset primarily determines which modality the input features originate from—either the event modality or the RGB modality. By classifying the features, the domain discriminator can identify and distinguish input data from different modalities, thus providing guidance for subsequent cross-modal alignment.

[0097] To further enhance the model's generalization ability, a Gradient Reversal Layer (GRL) is introduced into the domain discriminator. The introduction of GRL reverses the gradient signal of the domain discriminator during training and propagates it back to the feature extraction network. This allows the feature extraction network to intentionally avoid learning features that are overly dependent on modal information when updating its parameters, instead favoring modality-independent features. This gradient reversal mechanism enables the feature extraction network to extract task-useful features that can be shared across modalities, thus improving the model's generalization ability in cross-domain scenarios.

[0098] Through training the domain discriminator, the network can gradually eliminate the feature differences between the event modality and the RGB modality, thereby effectively processing data of both modalities in the same feature space.

[0099] 2. Mutual learning mechanism.

[0100] To further enhance feature alignment between the two modalities, a mutual learning mechanism was introduced. This mechanism forces the network to learn the similarity between the event modality and the RGB modality in class distribution by imposing consistency constraints on the outputs of the two modalities during the training phase, thereby achieving soft alignment.

[0101] Specifically, the mutual learning mechanism compares the network outputs of the event modality and the RGB modality, calculating the difference in their class distributions. By minimizing this difference, the network can learn shared features between the two modalities, making the features of the two modalities more similar in the feature space.

[0102] This paper proposes an effective alignment strategy to address the differences in features between different modalities in cross-modal gait recognition tasks, utilizing two modules: a domain discriminator and a mutual learning mechanism. The domain discriminator employs a gradient inversion layer (GRL) to extract modality-independent features, thereby improving the model's generalization ability across different modalities. The mutual learning mechanism, through consistency constraints, promotes the alignment of the outputs of the two modalities in the class distribution, enabling the model to achieve soft alignment in the feature space, thus enhancing the accuracy and robustness of cross-modal recognition. The combination of these two modules allows for more precise processing and fusion of features from event cameras and RGB images, providing a solid foundation for cross-modal gait recognition.

[0103] Step S4: Gait modeling and classification.

[0104] To further improve recognition performance and reduce redundant information, multiple modules were designed to further process and optimize these temporal features. The specific process includes steps S401-S404.

[0105] Step S401, Temporal Pooling (TP).

[0106] The temporal feature sequence output from the backbone network is first fed into the Temporal Pooling (TP) module. The main purpose of this module is to compress keyframe information in the gait sequence. In gait recognition, typically only a few frames contain useful discriminative information, while many frames may be redundant or even interfere with the recognition process. The Temporal Pooling module reduces the impact of redundant frames on model training by performing pooling operations on the input sequence, preserving discriminative information in the temporal dimension, thereby extracting more compact and effective temporal features. This process helps enhance the model's focus on key moments in the gait sequence and improves its sensitivity to dynamic changes.

[0107] Step S402, Horizontal Pyramid Pooling (HPP).

[0108] The compressed feature sequence then enters the Horizontal Pyramid Pooling (HPP) module, which is designed to further enhance the model's ability to perceive local regions from a spatial perspective. The HPP module extracts local spatial gait features by dividing the feature map into multiple regions of different scales in the vertical direction and performing average pooling within each scale region. This process helps the model capture gait details at different scales, such as local motion features of different parts of the body or subtle posture changes. By pooling at multiple scales, the HPP module can better identify fine-grained regional differences in gait, improving the model's robustness in complex scenes.

[0109] Step S403, Feature splicing and Fully Connected Layers (FCs).

[0110] Local features extracted at all scales are processed by the HPP module, concatenated into a long vector, and then fed into fully connected layers (FCs) for dimension mapping. This process unifies local features from various scales into a shared feature space, facilitating subsequent processing. In FCs, the dimensions of features are mapped and unified, which not only ensures that features from different modalities have the same representation but also provides a stable foundation for subsequent standardization and classification operations.

[0111] Step S404: Feature normalization and classification.

[0112] The feature vectors mapped by the fully connected layer are then fed into the BNNeck module, which employs Batch Normalization (BN) to normalize the features. By normalizing the features, the BNNeck module ensures a more stable feature distribution across different samples, reducing training fluctuations caused by differences in features between samples. This process helps improve the convergence speed and stability of the training process.

[0113] Finally, the standardized features are input into a Softmax classifier, which outputs a probability distribution for each identity category based on the normalized feature vectors. By maximizing these probability values, the model can ultimately output the identity category corresponding to the gait sample, thus achieving accurate identification of gait identity.

[0114] See Figure 2In this method, a multi-task joint training strategy is adopted to improve cross-modal recognition capabilities. Multiple loss terms are integrated during training to constrain the model.

[0115] Specifically, the losses mainly include the following four categories:

[0116] (1) Classification loss (Cross Entropy Loss).

[0117] The supervised model correctly distinguishes different identities, denoted as: Among them, y i Indicates the true category, This represents the probability output by the model.

[0118] (2) Triplet Loss.

[0119] A triplet consisting of anchor samples, positive samples, and negative samples is constructed. A triplet loss is designed to enhance the properties of intra-class aggregation and inter-class separation, thereby narrowing the distance between samples with the same identity and widening the distance between samples with different identities. The definition is as follows:

[0120] (3) Modal consistency loss (Mutual Learning Loss).

[0121] To enhance the alignment between the RGB modality and the event modality at the feature representation level, Mean Squared Error (MSE) is used to measure the difference between the mean features of the two modalities. The loss is defined as follows:

[0122]

[0123] in, These represent the feature mean values ​​of RGB and event modality within a batch, respectively.

[0124] (4) Domain Adversarial Loss.

[0125] The domain discriminant is used to distinguish modes and inversely optimize shared features to make the feature distribution mode-independent. The loss is defined as follows:

[0126] Based on the above losses, the training objective function is defined as:

[0127] L total =L ce +λ1L triplet +λ2L mutual +λ3L adversarial

[0128] Furthermore, to further enhance cross-domain adaptability, the inversion coefficient λ of the gradient inversion layer (GRL) used in the adversarial module is gradually adjusted as the training progresses, and its change pattern is as follows:

[0129]

[0130] Where p represents the current training progress (usually the number of training iterations or epochs), and γ is a hyperparameter that controls the rate of change of the inversion coefficient. This formula ensures that the inversion coefficient of GRL is high in the early stages of training, thus strongly promoting feature modality independence; as training progresses, the inversion coefficient gradually decreases, allowing the model to focus more on specific tasks (such as gait recognition) rather than adversarial training.

[0131] Example 2

[0132] Based on Example 1, see Figure 3 This embodiment provides a cross-modal gait recognition system from an event camera to a visual contour, used to implement the cross-modal gait recognition method from an event camera to a visual contour as described in Embodiment 1, aiming to achieve the task of cross-modal gait recognition between an event camera and an RGB visual contour. This system combines the high temporal resolution and low power consumption of the event camera with the rich details of the image information in the RGB camera. By introducing a multi-module fusion strategy, it achieves feature alignment between the two modalities, thereby improving the accuracy of gait recognition under cross-domain conditions.

[0133] See Figure 3 The system includes:

[0134] (1) Data Preprocessing and Representation Module: This module is responsible for preprocessing and representing the input event voxel data and RGB images. For event stream data, voxel grid encoding is used to convert it into a structured three-dimensional tensor for easier subsequent processing; while RGB images are processed by standardization and cropping to ensure the quality and uniformity of the input data and prepare for feature extraction.

[0135] (2) Modal Feature Extraction Module: In this module, event voxel data and RGB images are used for feature extraction through a shared backbone network (such as ResNet9). The shared network ensures the consistency of the feature spaces of the two modalities, making subsequent cross-modal alignment more efficient and reliable. Through a lightweight convolutional network, the extracted features have a uniform scale and strong discriminative ability.

[0136] (3) Feature Alignment Module: This module achieves cross-modal feature alignment through a domain classifier and a mutual learning mechanism. The domain classifier guides the model to extract modality-independent features through a gradient inversion layer (GRL), enhancing cross-domain generalization ability; the mutual learning mechanism further optimizes feature alignment by constraining the consistency of outputs between the event modality and the RGB modality, ensuring that features from different modalities can share useful information.

[0137] (4) Gait Modeling Module: In the gait modeling module, the keyframe information in the gait sequence is first compressed using the temporal pooling module (TP) to remove redundant frames and retain discriminative temporal features. Then, the horizontal pyramid pooling module (HPP) extracts the local spatial features of the gait through multi-scale pooling operations and enhances the ability to perceive fine-grained regional differences. In this way, the model can capture the dynamic features of the gait in both temporal and spatial dimensions.

[0138] (5) Discriminant Classification Module: In the discriminant classification module, the features modeled by gait are mapped through fully connected layers (FCs) to form a unified feature space. Subsequently, the features are normalized through a BNNeck structure to ensure that the feature distribution among different samples is more consistent, thereby improving the stability of the model. Finally, the normalized features are input into the Softmax classifier to complete the accurate discrimination of gait identity.

[0139] During system operation, event cameras and RGB cameras collect data and simultaneously input them into the system. After their respective preprocessing modules, the data is uniformly fed into the backbone network for feature extraction. Following modal alignment, the data enters the gait modeling module to generate the final gait embedding. The system uses a pre-trained classifier for identity determination, making it suitable for real-world scenarios such as security monitoring and identity recognition.

[0140] This invention combines complementary information from event cameras and RGB cameras, and employs a unified feature modeling and modal alignment mechanism to improve recognition accuracy and generalization ability in cross-modal gait recognition tasks. Furthermore, the system architecture proposed in this invention features modularity, low latency, and strong deployability, making it suitable for identity recognition and intelligent sensing tasks in various real-world scenarios.

[0141] Example 3

[0142] This embodiment provides an electronic device, including: one or more processors and a memory, wherein the memory stores one or more programs, the one or more programs including instructions for executing the cross-modal gait recognition method from event camera to visual contour as described in Embodiment 1.

[0143] like Figure 4At the hardware level, the electronic device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for the business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to achieve the above-mentioned functions. Figure 1 The method described herein. Of course, in addition to software implementation, this invention does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0144] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0145] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0146] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A cross-modal gait recognition method based on event camera and visual contours, characterized in that, Includes the following steps: Asynchronous event stream data collected by the event camera is acquired and voxel mesh encoding is performed. The estimated lateral center point position of the human body is obtained through boundary detection and head center localization. Based on the center point position, human body region sub-images are cropped to form voxel data. The voxel data and the acquired RGB image data are used as inputs to the shared backbone network in the gait recognition model to extract modal features consistent with the feature space. Based on the modal features, the gait recognition model sequentially performs temporal pooling, horizontal pyramid pooling, fully connected layers, normalization, and classification to obtain the final predicted gait identity category. The gait recognition model is trained in RGB image data using a saliency-guided training strategy, based on classification loss, triplet loss, modality consistency loss, and cross-domain adaptive adversarial discriminant loss. The process of obtaining the estimated lateral center point position of the human body through boundary detection and head center localization includes the following steps: Based on the voxel data aggregated from the asynchronous event stream data, the image is scanned line by line from the top down, responding to a line containing continuous... If there are 1 or more non-zero event pixels, it is determined that the human activity area has been entered, and this row is taken as the upper boundary. Based on the voxel data aggregated from the asynchronous event stream data, the image is scanned line by line from the bottom upwards, responding to a line containing... If there are 1 or more non-zero event pixels, the row will be used as the lower boundary. Based on the upper boundary, the position of the horizontal center point of the human body is estimated by calculating the centroid of the event pixels in the adjacent rows of the boundary. The saliency-guided training strategy includes the following steps: By calculating the pixel differences between the current RGB image frame and another random RGB image frame, a mask map of the changed areas in the current RGB image frame compared to the random RGB image frame is obtained; The mask image is smoothed to obtain a saliency image; Based on the saliency of the current RGB image frame and multiple random RGB image frames, the original current RGB image frame is weighted, with higher weights assigned to regions of significant change. The weights are controlled by a control factor, which gradually decreases during the training process. In the early stages of training, the model focuses strongly on salient regions, while in the later stages of training, the guidance is gradually relaxed to avoid the model overfitting to local regions.

2. The cross-modal gait recognition method based on event camera to visual contours according to claim 1, characterized in that, The process of acquiring asynchronous event stream data collected by the event camera and performing voxel mesh encoding includes the following steps: Asynchronous event stream data acquired by the event camera is acquired and voxel mesh encoded. Events generated within a continuous time period are divided into multiple small segments in chronological order. Based on the number of events at each pixel position in each time segment, a three-dimensional data structure containing temporal, spatial, and polarity information is constructed. As tensor data, in C The number of polarity channels representing an event. T Indicates the number of time channels. H , W This indicates the spatial height and width of the event image.

3. The cross-modal gait recognition method from event camera to visual contour as described in claim 1, characterized in that, , , This represents the image width.

4. The cross-modal gait recognition method from event camera to visual contour as described in claim 1, characterized in that, The process of cropping the human body region sub-image based on the center point position to form voxel data includes the following steps: Based on the horizontal center point position, the upper and lower boundaries of the image, a human body region sub-image of a fixed size is obtained by cropping. The size of the cropping window is configured according to the experimental settings.

5. The cross-modal gait recognition method from event camera to visual contour as described in claim 1, characterized in that, The modality consistency loss calculates the difference in class distribution of modal features between the event modality and the RGB modality. By minimizing this difference, the model learns the shared features between the two modalities, achieving soft alignment. The adversarial domain discriminant loss inverts the gradient signal and transmits it back to the backbone network, making the backbone network more inclined to learn modality-independent features when updating parameters.

6. The cross-modal gait recognition method from event camera to visual contour as described in claim 1, characterized in that, The training objective function, comprised of the classification loss, triplet loss, modality consistency loss, and cross-domain adaptive adversarial discriminant loss, is as follows: in, To train the objective function, , , , These are classification loss, triplet loss, modality consistency loss, and adversarial domain discrimination loss, respectively. , , For coefficients, , They represent the first The true class of each sample and the probability output by the gait recognition model. , , They represent the first i The feature vectors of the anchor sample, the feature vectors of the positive samples belonging to the same class as the anchor sample, and the feature vectors of the negative samples belonging to a different class than the anchor sample are all represented in the sample set. This represents the minimum margin between positive and negative samples. , These represent the feature mean values ​​of RGB and event modality within a batch, respectively. This represents the total number of samples within a batch. , These represent the true label and the discriminator's predicted probability, respectively. Represents the ReLU function. To address the inversion coefficients of gradient inversion in adversarial domain discrimination, For the current training progress, This is a hyperparameter used to control the rate of change of the inversion coefficient.

7. A cross-modal gait recognition system based on an event camera and visual contours, characterized in that, For implementing the cross-modal gait recognition method from event camera to visual contour as described in any one of claims 1-6, the system comprises: The data preprocessing and characterization module is used to preprocess and characterize the input asynchronous event stream data and RGB images. For the event stream data, it is converted into structured three-dimensional tensor event voxel data through voxel grid encoding. For the RGB images, it is normalized and cropped. The modal feature extraction module is used to extract features from the event voxel data and RGB images through a shared backbone network. The feature alignment module achieves cross-modal feature alignment through a domain discriminator and a mutual learning mechanism. The gait modeling module first compresses the key frame information in the gait sequence through the temporal pooling module, removes redundant frames and retains discriminative temporal features. Then, the horizontal pyramid pooling module extracts the local spatial features of the gait through multi-scale pooling operations and enhances the ability to perceive fine-grained regional differences. The discrimination and classification module maps the gait modeling features through a fully connected layer to form a unified feature space. The features are then normalized using a BNNeck structure. The standardized features are then input into a Softmax classifier to complete the gait identity discrimination.

8. An electronic device, characterized in that, It includes one or more processors, memory, and one or more programs stored in the memory, said one or more programs including instructions for performing the cross-modal gait recognition method from event camera to visual profile as described in any one of claims 1-6.

Images