Gait recognition method based on bimodal fusion

Abstract

The application discloses a gait recognition method and system based on dual-mode fusion, and belongs to the technical field of gait recognition.The application takes a contour map as a fixed acquisition mode, combines environmental light intensity, motion blur degree and target distance, adaptively selects RGB or an optical flow as a complementary mode, and constructs a dynamically switchable dual-mode acquisition combination; after processing dual-mode data, mode features are extracted, real-time quality of each mode is respectively evaluated, and fusion weights are dynamically distributed according to the real-time quality, so that adaptive weighted fusion of dual-mode features is completed; meanwhile, aiming at a multi-target overlapping and shielding working condition, the mode fusion weight is dynamically adjusted according to a shielding grade.The application solves the problems of fixed gait recognition mode, non-adaptive allocation of fusion weights and poor adaptability to complex environments, and improves rationality of multi-mode fusion and scene adaptation capability.

Description

Technical Field

[0001] This invention relates to the field of gait recognition technology, and more specifically to a gait recognition method based on dual-modal fusion. Background Technology

[0002] Most existing gait recognition methods adopt single-modal recognition, with fixed modal types that cannot adapt to changes in the environment, making it difficult to adapt to complex scenarios such as lighting fluctuations, motion blur, and changes in target distance. Traditional multimodal solutions often use fixed combination and fixed weight fusion modes, which cannot dynamically adjust the fusion weights based on real-time modal imaging quality, resulting in insufficient utilization of modal redundancy and effective features. Furthermore, in scenarios with multiple overlapping and occlusion, there is a lack of dynamic suppression and adaptive adjustment mechanisms for modal weights, leading to poor modal fusion adaptability, weak anti-interference ability, and difficulty in ensuring recognition stability. Summary of the Invention

[0003] One object of the present invention is to solve at least the above-mentioned problems by means of a gait recognition method based on bimodal fusion.

[0004] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: a gait recognition method based on dual-modal fusion, comprising the following steps: Step S1: Multi-target gait data acquisition and tracking. Humans in the video stream are detected by a target detector. Each target is assigned a unique ID and continuously tracked. The contour map sequence is a fixed acquisition mode. The RGB image sequence and the optical flow map sequence are adaptively selected as the other mode based on the ambient light intensity, motion blur degree and target distance, forming a contour map plus RGB dual-modal combination or a contour map plus optical flow map dual-modal combination for synchronous acquisition. Step S2: Modal alignment and preprocessing. Spatial alignment, temporal alignment, environmental adaptation and cross-view alignment are performed on the acquired dual-modal data. Spatial alignment adopts normalized coordinate mapping of human skeleton key points to align the optical flow trajectory to the contour map coordinate system. Step S3: Modality-specific feature encoding. Construct RGB encoding network, contour map encoding network and optical flow encoding network respectively, and enable the corresponding two networks according to the selected modality combination to extract modality feature vectors with the same dimension. Step S4: Bimodal adaptive fusion, calculate the quality score of each modality, dynamically allocate fusion weights according to the quality score, and perform element-wise weighted summation of the feature vectors of the two modalities. That is, the i-th dimension of the fused feature is equal to the i-th dimension of the first modality feature multiplied by the first modality weight plus the i-th dimension of the second modality feature multiplied by the second modality weight. Output the fused gait embedding feature. Step S5: Gait representation and recognition. The gait embedding features are matched with the templates in the feature library using cosine similarity. The recognition result or candidate list is then output. Step S6: Output and feedback. Output the recognition results and modal quality information, and link with external devices or upload logs.

[0005] Preferably, the specific rule for adaptively selecting mode combinations in step S1 is as follows: First, map the ambient light intensity L, motion blur level B, and target distance D to quality prediction scores between 0 and 1: Illuminance fraction sL = min(1, L / 100); The ambiguity score sB = 1 - B, where B is calculated using the Laplace variance method; Distance score sD = max(0, 1 - D / 20); Calculate the overall preference score P = 0.4•sL + 0.3•sB + 0.3•sD; Apply an exponential moving average filter to the overall preference score: Psmooth[t] = 0.3•P[t] + 0.7•Psmooth[t-1]; Hysteresis threshold decision-making is adopted: if Psmooth > 0.6 and remains so for 3 consecutive frames, and the time interval between the last mode switch is not less than 1 second, then the contour map plus RGB dual-modal combination is selected; if Psmooth < 0.4 and remains so for 3 consecutive frames, and the time interval between the last mode switch is not less than 1 second, then the contour map plus optical flow map dual-modal combination is selected; when it is in the closed interval of 0.4 to 0.6, the current mode remains unchanged.

[0006] Preferably, MediaPipe is used to detect 17 skeletal key points of the human body, calculate the minimum bounding box, and normalize the key point coordinates to the interval between 0 and 1 to achieve distance normalization; for optical flow vectors at different distances, they are scaled by the same ratio, i.e., unorm=u / Wbox, vnorm=v / Hbox, thereby generating scale-independent motion trajectory features.

[0007] Preferably, the optical flow coding network in step S3 takes grayscale images of adjacent frames and coordinates of 17 human core joints as input. First, the Lucas-Kanade pyramid sparse optical flow algorithm is used to extract the optical flow vectors and confidence scores of these 17 joints, forming a 34-dimensional vector. Then, two steps are performed sequentially: First, temporal median filtering with a filter window length of 3; Second, Kalman smoothing, where the state vector of the Kalman filter is 68 dimensions in total (34-dimensional position and 34-dimensional velocity). The prediction step outputs 68 dimensions, and after updating, only the 34-dimensional position part is taken as the output. The filtered 34-dimensional vector is fed into a three-layer MLP network for temporal coding, and finally outputs a 512-dimensional optical flow motion feature vector.

[0008] Preferably, the calculation method of the modality quality scores and fusion weights in step S4 is as follows: for the RGB modality, the image entropy is used to measure the definition, and the quality score qRGB is obtained by calculation; for the contour map modality, the contour integrity score is adopted to calculate the quality score qsil; for the optical flow modality, the quality score qOF is calculated based on the key point missing rate and trajectory deviation; for the currently enabled dual-modality combination, the fusion weights are calculated according to the quality scores of the two modalities, and the 512-dimensional feature vectors of the two modalities are subjected to element-wise weighted summation to obtain the fused gait embedding feature, which is directly used for the matching in step S5.

[0009] Preferably, for the gait representation and recognition in step S5, a joint loss function is used to train the encoder and the fuser, and the joint loss function is a combination of triplet loss and additive angular margin loss, wherein the margin parameter of the triplet loss is set to 0.3, and the margin parameter of the additive angular margin loss is set to 0.5 and the scale parameter is set to 64; in the inference stage, cosine similarity comparison is adopted, and the reference threshold is set to 0.85.

[0010] Preferably, step S1 further includes multi-target overlapping processing: defining the overlapping rate IoU = intersection area / union area; when IoU > 70%, it is determined as severe occlusion, in this case, if the current modality combination includes optical flow, the quality score of the optical flow modality is forcibly set to 0, so that the optical flow fusion weight is reduced to 0, and the motion trajectory is predicted only by the contour map through Kalman filtering; when 30% < IoU ≤ 70%, it is determined as partial occlusion, the current modality combination is maintained, but the fusion weight of the optical flow modality is linearly reduced according to IoU: the original fusion weight of the optical flow modality is multiplied by the coefficient (1-IoU), and then renormalized with the contour modality weight, that is, wOF'=wOF•(1-IoU) / (wOF•(1-IoU)+wsil), wsil'=wsil / (wOF•(1-IoU)+wsil); when IoU ≤ 30%, it is determined as overlapping separation or no overlapping, and conventional processing is resumed; after the overlapping is separated, that is, when IoU drops below 30%, the contour map feature at the moment of separation is used to perform identity recovery matching with the historical features cached before separation, and the position predicted by Kalman filtering is combined: the overlap between the current detection frame position and the predicted position of each cached trajectory is calculated, which is weighted with the cosine similarity of contour features with each weight being 0.5, and if the highest score is greater than 0.7, the original ID is restored, otherwise a new ID is assigned.

[0011] A gait recognition system based on dual-modality fusion, configured to execute the above method, the system comprising: a dual-modality data acquisition module, integrated with an RGB camera and a sparse optical flow acquisition unit, and comprising a modality switching unit; a multi-target tracking module, integrated with an object detector and a tracker; The modal preprocessing and alignment module includes spatial alignment, temporal alignment, environment adaptation, and cross-viewpoint alignment units. The modality-specific encoding module includes an RGB encoding network, a contour map encoding network, an optical flow encoding network, and an encoding switching unit; The dual-modal fusion module includes a modal quality assessment unit and a weighted summation fusion unit; The gait recognition module includes a feature matching unit, a loss function training unit, and a multi-target recognition queue management unit. The output and feedback module includes a result output interface, a device linkage interface, and a log management unit. The lightweight deployment module includes a channel attention pruning unit, a parallel inference scheduling unit, and a dynamic frame rate adaptive unit.

[0012] Preferably, the dynamic frame rate adaptive unit in the lightweight deployment module adjusts the sampling frame rate according to the target walking speed: reducing the frame rate at low speeds and increasing the frame rate at high speeds to reduce computational load.

[0013] Preferably, the sparse optical flow acquisition unit in the dual-modal data acquisition module is implemented based on a hardware acceleration unit of the optical flow algorithm. This unit is integrated with the RGB camera in the same image signal processor. During mode switching, the hardware unit is dynamically powered off or on, and the first frame after the switch is discarded to ensure data synchronization.

[0014] As described above, the gait recognition method based on dual-modal fusion provided by this invention has the following beneficial effects: This invention uses the contour map as a fixed modality and can adaptively switch between RGB or optical flow to form a dual-modal combination based on ambient lighting, motion blur, and target distance, achieving dynamic adaptation of modal configuration to the scene; by independently evaluating the quality of each modality, dynamically calculating and allocating fusion weights based on the quality score, it achieves adaptive weighted fusion of dual-modal features; simultaneously, in multi-target overlapping and occlusion scenarios, it dynamically adjusts the fusion weights of corresponding modalities according to the degree of occlusion, avoiding interference from low-quality modalities in the recognition effect, and significantly improving the suitability of multi-modal combinations, the rationality of feature fusion, and the stability and reliability of gait recognition under complex conditions. Detailed Implementation

[0015] The present invention will be further described below through specific embodiments.

[0016] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0017] This invention relates to a gait recognition method based on dual-modal fusion, comprising the following steps: Step S1: Multi-target gait data acquisition and tracking. Humans in the video stream are detected by a target detector. Each target is assigned a unique ID and continuously tracked. The contour map sequence is a fixed acquisition mode. The RGB image sequence and the optical flow map sequence are adaptively selected as the other mode based on the ambient light intensity, motion blur degree and target distance, forming a contour map plus RGB dual-modal combination or a contour map plus optical flow map dual-modal combination for synchronous acquisition. Step S2: Modal alignment and preprocessing. Spatial alignment, temporal alignment, environmental adaptation and cross-view alignment are performed on the acquired dual-modal data. Spatial alignment adopts normalized coordinate mapping of human skeleton key points to align the optical flow trajectory to the contour map coordinate system. Step S3: Modality-specific feature encoding. Construct RGB encoding network, contour map encoding network and optical flow encoding network respectively, and enable the corresponding two networks according to the selected modality combination to extract modality feature vectors with the same dimension. Step S4: Bimodal adaptive fusion, calculate the quality score of each modality, dynamically allocate fusion weights according to the quality score, and perform element-wise weighted summation of the feature vectors of the two modalities. That is, the i-th dimension of the fused feature is equal to the i-th dimension of the first modality feature multiplied by the first modality weight plus the i-th dimension of the second modality feature multiplied by the second modality weight. Output the fused gait embedding feature. Step S5: Gait representation and recognition. The gait embedding features are matched with the templates in the feature library using cosine similarity. The recognition result or candidate list is then output. Step S6: Output and Feedback. Output the recognition results and modal quality information, and link with external devices or upload logs. This method uses the contour map sequence as the fixed acquisition modality, and adaptively selects RGB or optical flow as the complementary modality according to environmental conditions to form a dual-modal combination. The quality score is used for fusion weight allocation.

[0018] The specific rules for adaptively selecting mode combinations in step S1 are as follows: First, map the ambient light intensity L, motion blur level B, and target distance D to quality prediction scores between 0 and 1: Illuminance fraction sL = min(1, L / 100); The ambiguity score sB = 1 - B, where B is calculated using the Laplace variance method; Distance score sD = max(0, 1 - D / 20); Calculate the overall preference score P = 0.4•sL + 0.3•sB + 0.3•sD; Apply an exponential moving average filter to the overall preference score: Psmooth[t] = 0.3•P[t] + 0.7•Psmooth[t-1]; Hysteresis threshold decision-making is adopted: if Psmooth > 0.6 and remains so for 3 consecutive frames, and the time interval between the last mode switch is not less than 1 second, then the contour map plus RGB dual-modal combination is selected; if Psmooth < 0.4 and remains so for 3 consecutive frames, and the time interval between the last mode switch is not less than 1 second, then the contour map plus optical flow map dual-modal combination is selected; when it is in the closed interval of 0.4 to 0.6, the current mode remains unchanged.

[0019] L represents the illumination intensity in lux; B represents the motion blur level, ranging from 0 to 1, calculated from the Laplacian variance: B = max(0, min(1, 1 - Vcurrent / Vclear)), where Vclear is the pre-calibrated baseline variance of the fully clear image; D represents the target distance in meters; P represents the overall preference score, calculated as Psmooth using an exponential moving average. The hysteresis thresholds for Psmooth are 0.4 and 0.6, used for anti-shake purposes. For example, if the current illumination L = 80 lux, then sL = min(1, 80 / 100) = 0.8; the Laplacian variance Vcurrent = 450, Vclear = 600, B = 1 - 450 / 600 = 0.25, then sB = 0.75; the target distance D = 8 meters, then sD = max(0, 1 - 8 / 20) = 0.6. Therefore, P = 0.4 × 0.8 + 0.3 × 0.75 + 0.3 × 0.6 = 0.32 + 0.225 + 0.18 = 0.725. In the previous frame, Psmooth[t-1] = 0.7, so the current Psmooth = 0.3 × 0.725 + 0.7 × 0.7 = 0.2175 + 0.49 = 0.7075. Since 0.7075 > 0.6 and has been maintained for 3 consecutive frames, and 1.5 seconds have passed since the last switch, we should maintain or switch to the contour map plus RGB dual-modal combination. If Psmooth drops to 0.35, we should maintain or switch to the optical flow plus contour map combination.

[0020] MediaPipe was used to detect 17 skeletal key points of the human body, and the minimum bounding box was calculated. The coordinates of the key points were normalized to the interval between 0 and 1 to achieve distance normalization. For optical flow vectors at different distances, they were scaled by the same ratio, i.e., unorm=u / Wbox and vnorm=v / Hbox, thereby generating scale-independent motion trajectory features.

[0021] u and v are the original optical flow vectors; Wbox and Hbox are the width and height of the minimum bounding box of the human skeleton keypoints; unorm and vnorm are the scale-normalized optical flow vectors. This method only eliminates the effect of distance scaling, not the viewpoint tilt, hence it is called scale-independent. For example, if the bounding box of a target has a width Wbox = 200 pixels and a height Hbox = 400 pixels, and the optical flow vector of a keypoint is (15, 10) pixels per frame, then after normalization, unorm = 15 / 200 = 0.075, and vnorm = 10 / 400 = 0.025. If the target is closer in another frame, the bounding box becomes 300×600, and the optical flow vector generated by the same motion is (22.5, 15), which remains (0.075, 0.025) after normalization, achieving scale independence.

[0022] The optical flow coding network in step S3 takes grayscale images of adjacent frames and coordinates of 17 human core joints as input. First, the Lucas-Kanade pyramid sparse optical flow algorithm is used to extract the optical flow vectors and confidence scores of these 17 joints, forming a 34-dimensional vector. Then, two steps are performed sequentially: First, temporal median filtering with a filter window length of 3; Second, Kalman smoothing, where the state vector of the Kalman filter is 68 dimensions in total (34-dimensional position and 34-dimensional velocity). The prediction step outputs 68 dimensions, and after updating, only the 34-dimensional position part is taken as the output. The filtered 34-dimensional vector is fed into a three-layer MLP network for temporal coding, and finally outputs a 512-dimensional optical flow motion feature vector.

[0023] The modal quality score and fusion weight calculation methods in step S4 are as follows: For the RGB modality, image entropy is used to measure sharpness, and the quality score qRGB is calculated; for the contour map modality, contour integrity score is used, and the quality score qsil is calculated; for the optical flow modality, the quality score qOF is calculated based on the keypoint missing rate and trajectory deviation; for the currently enabled dual-modal combination, the fusion weight is calculated based on the quality scores of the two modalities, and the 512-dimensional feature vectors of the two modalities are summed element-wise to obtain the fused gait embedding features, which are directly used for matching in step S5.

[0024] The RGB modality quality score uses the image entropy value, where the entropy value H=-∑p_ilog_2p_i, p_i is the probability of gray level i, and qRGB=H / 8; the silhouette modality quality score qsil=min(1,S / Sstd), where S is the area of the foreground pixel connected domain, Sstd is the area of the standard template, and Sstd is pre-obtained by statistics of the median of the connected domain areas of all valid silhouette maps in the training data; for the optical flow modality quality score, the missing rate rmiss=1-number of valid key points / 17 is calculated first, then the average trajectory deviation D of adjacent frames is calculated, and the score is obtained according to the rules: if rmiss≤10%, then Smiss=0.8; if 10%<rmiss<50%, then Smiss=0.8-(rmiss-0.1)×2.0; if rmiss≥50%, then Smiss=0; if D≤5 pixels, then Sdev=0.9; if 5<D<20 pixels, then Sdev=0.9-(D-5)×0.06; if D≥20 pixels, then Sdev=0; the optical flow modality score qOF=0.5×Smiss+0.5×Sdev. For the currently enabled bimodal combination, let the quality scores of the two modalities be q1 and q2 respectively. To prevent the denominator from being zero, let the denominator denom=q1+q2+ε (ε=10^{-6}), then the fusion weight w1=q1 / denom, w2=q2 / denom; the fused gait embedding feature is ffused=w1•f1+w2•f2, the addition is element-wise vector addition, that is, the j-th element of ffused=w1×f1[j]+w2×f2[j], both f1 and f2 are 512-dimensional vectors, and the fused feature is directly used for the cosine similarity matching in step S5.

[0025] For gait representation and recognition in step S5, a joint loss function is used to train the encoder and the fuser. The joint loss function is a combination of triplet loss and additive angular margin loss, wherein the margin parameter of the triplet loss is set to 0.3, and the margin parameter of the additive angular margin loss is set to 0.5 and the scale parameter is set to 64; the inference stage adopts cosine similarity comparison, and the baseline threshold is set to 0.85.

[0026] The step S1 further includes multi-target overlapping processing: define the intersection-over-union rate IoU = intersection area / union area; when IoU > 70%, it is determined as severe occlusion, and if the current modality combination contains optical flow, the quality score of the optical flow modality is forcibly set to 0, so that the optical flow fusion weight is reduced to 0, and the motion trajectory is predicted only by relying on the contour map through Kalman filtering; when 30% < IoU ≤ 70%, it is determined as partial occlusion, the current modality combination is maintained, but the fusion weight of the optical flow modality is linearly reduced according to IoU: the original fusion weight of the optical flow modality is multiplied by the coefficient (1-IoU), and then renormalized with the contour modality weight, that is, wOF'=wOF•(1-IoU) / (wOF•(1-IoU)+wsil), wsil'=wsil / (wOF•(1-IoU)+wsil); when IoU ≤ 30%, it is determined as overlapping separation or no overlap, and conventional processing is resumed; after the overlapping is separated, that is, when IoU drops below 30%, the contour map features at the moment of separation are used to perform identity recovery matching with the cached historical features before separation, and the position predicted by Kalman filtering is combined at the same time: the overlap between the current detection frame position and the predicted position of each cached trajectory is calculated, weighted with the cosine similarity of contour features with a weight of 0.5 for each, if the highest score is greater than 0.7, the original ID is restored, otherwise a new ID is assigned.

[0027] A gait recognition system based on dual-modality fusion, configured to execute the above method, the system comprising: A dual-modality data acquisition module, integrated with an RGB camera and a sparse optical flow acquisition unit, and comprising a modality switching unit; A multi-target tracking module, integrated with an object detector and a tracker; A modality preprocessing and alignment module, comprising units for spatial alignment, temporal alignment, environment adaptation and cross-view alignment; A modality-specific encoding module, comprising an RGB encoding network, a contour map encoding network, an optical flow encoding network and an encoding switching unit; A dual-modality fusion module, comprising a modality quality assessment unit and a weighted sum fusion unit; A gait recognition module, comprising a feature matching unit, a loss function training unit and a multi-target recognition queue management unit; An output and feedback module, comprising a result output interface, a device linkage interface and a log management unit; A lightweight deployment module, comprising a channel attention pruning unit, a parallel inference scheduling unit and a dynamic frame rate adaptation unit.

[0028] The dynamic frame rate adaptation unit in the lightweight deployment module adjusts the sampling frame rate according to the walking speed of the target: reducing the frame rate at low speed and increasing the frame rate at high speed, so as to reduce the computational load.

[0029] The sparse optical flow acquisition unit in the dual-modal data acquisition module is implemented based on the hardware acceleration unit of the optical flow algorithm. This unit is integrated with the RGB camera in the same image signal processor. When switching modes, the hardware unit is dynamically powered off or on, and the first frame after the switch is discarded to ensure data synchronization.

[0030] The above are merely some specific embodiments of the present invention, but the design concept of the present invention is not limited thereto. Any non-substantial modifications made to the present invention using this concept shall be considered as infringing upon the protection scope of the present invention.

Claims

1. A gait recognition method based on dual-modal fusion, characterized in that: Includes the following steps: Step S1: Multi-target gait data acquisition and tracking. Humans in the video stream are detected by a target detector. Each target is assigned a unique ID and continuously tracked. The contour map sequence is a fixed acquisition mode. The RGB image sequence and the optical flow map sequence are adaptively selected as the other mode based on the ambient light intensity, motion blur degree and target distance, forming a contour map plus RGB dual-modal combination or a contour map plus optical flow map dual-modal combination for synchronous acquisition. Step S2: Modal alignment and preprocessing. Spatial alignment, temporal alignment, environmental adaptation and cross-view alignment are performed on the acquired dual-modal data. Spatial alignment adopts normalized coordinate mapping of human skeleton key points to align the optical flow trajectory to the contour map coordinate system. Step S3: Modality-specific feature encoding. Construct RGB encoding network, contour map encoding network and optical flow encoding network respectively, and enable the corresponding two networks according to the selected modality combination to extract modality feature vectors with the same dimension. Step S4: Bimodal adaptive fusion, calculate the quality score of each modality, dynamically allocate fusion weights according to the quality score, and perform element-wise weighted summation of the feature vectors of the two modalities. That is, the i-th dimension of the fused feature is equal to the i-th dimension of the first modality feature multiplied by the first modality weight plus the i-th dimension of the second modality feature multiplied by the second modality weight. Output the fused gait embedding feature. Step S5: Gait representation and recognition. The gait embedding features are matched with the templates in the feature library using cosine similarity. The recognition result or candidate list is then output. Step S6: Output and feedback. Output the recognition results and modal quality information, and link with external devices or upload logs.

2. The gait recognition method based on dual-modal fusion according to claim 1, characterized in that: The specific rules for adaptively selecting mode combinations in step S1 are as follows: First, map the ambient light intensity L, motion blur level B, and target distance D to quality prediction scores between 0 and 1: Illuminance fraction sL = min(1, L / 100); The ambiguity score sB = 1 - B, where B is calculated using the Laplace variance method; Distance score sD = max(0, 1 - D / 20); Calculate the overall preference score P = 0.4·sL + 0.3·sB + 0.3·sD; Apply an exponential moving average filter to the overall preference score: Psmooth[t] = 0.3·P[t] + 0.7·Psmooth[t-1]; Hysteresis threshold decision-making is adopted: if Psmooth > 0.6 and remains so for 3 consecutive frames, and the time interval between the last mode switch is not less than 1 second, then the contour map plus RGB dual-modal combination is selected; if Psmooth < 0.4 and remains so for 3 consecutive frames, and the time interval between the last mode switch is not less than 1 second, then the contour map plus optical flow map dual-modal combination is selected; when it is in the closed interval of 0.4 to 0.6, the current mode remains unchanged.

3. The gait recognition method based on dual-modal fusion according to claim 1, characterized in that: MediaPipe was used to detect 17 skeletal key points of the human body, and the minimum bounding box was calculated. The coordinates of the key points were normalized to the interval between 0 and 1 to achieve distance normalization. For optical flow vectors at different distances, they were scaled by the same ratio, i.e., unorm=u / Wbox and vnorm=v / Hbox, thereby generating scale-independent motion trajectory features.

4. The gait recognition method based on dual-modal fusion according to claim 3, characterized in that: The optical flow coding network in step S3 takes grayscale images of adjacent frames and coordinates of 17 human core joints as input. First, the Lucas-Kanade pyramid sparse optical flow algorithm is used to extract the optical flow vectors and confidence scores of these 17 joints, forming a 34-dimensional vector. Then, two steps are performed sequentially: First, temporal median filtering with a filter window length of 3; Second, Kalman smoothing, where the state vector of the Kalman filter is 68 dimensions in total (34-dimensional position and 34-dimensional velocity). The prediction step outputs 68 dimensions, and after updating, only the 34-dimensional position part is taken as the output. The filtered 34-dimensional vector is fed into a three-layer MLP network for temporal coding, and finally outputs a 512-dimensional optical flow motion feature vector.

5. The gait recognition method based on dual-modal fusion according to claim 1, characterized in that: The modal quality score and fusion weight calculation methods in step S4 are as follows: For the RGB modality, image entropy is used to measure sharpness, and the quality score qRGB is calculated; for the contour map modality, contour integrity score is used, and the quality score qsil is calculated; for the optical flow modality, the quality score qOF is calculated based on the keypoint missing rate and trajectory deviation; for the currently enabled dual-modal combination, the fusion weight is calculated based on the quality scores of the two modalities, and the 512-dimensional feature vectors of the two modalities are summed element-wise to obtain the fused gait embedding features, which are directly used for matching in step S5.

6. The gait recognition method based on dual-modal fusion according to claim 1, characterized in that: In step S5, gait representation and recognition employ a joint loss function to train the encoder and fusion unit. The joint loss function is a combination of triplet loss and additive angular edge loss, where the edge parameter of triplet loss is set to 0.3, the edge parameter of additive angular edge loss is set to 0.5, and the scale parameter is set to 64. In the inference stage, cosine similarity comparison is used, with a baseline threshold set to 0.

85.

7. The gait recognition method based on dual-modal fusion according to claim 1, characterized in that: The step S1 further includes multi-target overlapping processing: defining the overlapping rate IoU = intersection area / union area; when IoU>70%, it is determined as severe occlusion, in this case, if the current modality combination contains optical flow, the quality score of the optical flow modality is forcibly set to 0, so that the optical flow fusion weight is reduced to 0, and the motion trajectory is predicted only by the contour map through Kalman filtering; when 30%<IoU≤70%, it is determined as partial occlusion, the current modality combination is maintained, but the fusion weight of the optical flow modality is linearly reduced according to IoU: the original fusion weight of the optical flow modality is multiplied by the coefficient (1-IoU), and then renormalized with the contour modality weight, that is, wOF'=wOF·(1-IoU) / (wOF·(1-IoU)+wsil), wsil'=wsil / (wOF·(1-IoU)+wsil); when IoU≤30%, it is determined as overlapping separation or non-overlapping, and conventional processing is resumed; after the overlapping is separated, that is, when IoU drops below 30%, the contour map feature at the moment of separation and the historical feature cached before separation are used for identity recovery matching, combined with the position predicted by Kalman filtering: the overlap between the current detection frame position and the predicted position of each cached trajectory is calculated, weighted with the cosine similarity of the contour features with a weight of 0.5 each, and the original ID is restored if the highest score is >0.7, otherwise a new ID is assigned.

8. A gait recognition system based on dual-modal fusion, used to execute the method according to any one of claims 1 to 7, characterized in that: The system includes: A dual-modality data acquisition module, integrating an RGB camera and a sparse optical flow acquisition unit, and comprising a modality switching unit; A multi-target tracking module, integrating an object detector and a tracker; A modality preprocessing and alignment module, comprising spatial alignment, temporal alignment, environment adaptation and cross-view alignment units; A modality-specific encoding module, comprising an RGB encoding network, a contour map encoding network, an optical flow encoding network and an encoding switching unit; A dual-modality fusion module, comprising a modality quality assessment unit and a weighted sum fusion unit; A gait recognition module, comprising a feature matching unit, a loss function training unit and a multi-target recognition queue management unit; An output and feedback module, comprising a result output interface, a device linkage interface and a log management unit; A lightweight deployment module, comprising a channel attention pruning unit, a parallel inference scheduling unit and a dynamic frame rate adaptation unit.

9. The gait recognition method based on dual-modal fusion according to claim 1, characterized in that: The dynamic frame rate adaptation unit in the lightweight deployment module adjusts the sampling frame rate according to the walking speed of the target: reducing the frame rate at low speed and increasing the frame rate at high speed, so as to reduce the computational load.

10. The gait recognition method based on dual-modal fusion according to claim 1, characterized in that: The sparse optical flow acquisition unit in the dual-modality data acquisition module is implemented based on a hardware acceleration unit of an optical flow algorithm, and this unit is integrated in the same image signal processor as the RGB camera; when modality switching is performed, the hardware unit is dynamically powered off or powered on, and the first frame after switching is discarded to ensure data synchronization.