Time sequence consistency modeling method for dynamic change scene

Abstract

The invention discloses a time sequence consistency modeling method for a dynamic change scene, and belongs to the technical field of computer vision and video processing. Dynamically capturing semantic similarity between a front frame and a rear frame through a learnable convolution kernel, constructing a feature propagation network based on a depth feature flow, and performing nonlinear mapping and adaptive weighting on semantic features of a front frame and features of a current frame; calculating local semantic similarity through corresponding shallow guide features by using deep semantic features of front and back frames, generating a learnable spatial adaptive interpolation weight, and performing structure alignment and fusion on features of adjacent frames; the target transformation, the target detection and the target matching are carried out in sequence, and the intersection-to-union ratio IoU between the target frames is calculated under the common view angle, so that the matching deviation caused by the view angle difference can be remarkably reduced, and the target association is more regular and geometrically consistent. According to the method, the problems of video semantic prediction flicker, artifacts, texture slippage and the like caused by strenuous movement of a camera, non-rigid deformation of an object and frequent shielding are solved.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and video processing technology, and specifically relates to a temporal consistency modeling method for dynamically changing scenarios. Background Technology

[0002] Temporal consistency modeling methods and systems for dynamically changing scenarios. In the fields of computer vision, video processing, and 3D reconstruction (such as dynamic NeRF / 3D Gaussian Splatting), as application scenarios expand from static shooting to complex dynamic scenes, maintaining the temporal continuity (i.e., temporal consistency) of video sequences or rendering results has become a core challenge.

[0003] The closest existing technologies to this invention are the temporal constraint method based on dense optical flow and the frame-by-frame propagation method based on recurrent neural networks. (1) The temporal constraint method based on dense optical flow uses traditional optical flow algorithms (Lucas-Kanade method) or deep learning optical flow networks (FlowNet, RAFT) to calculate the pixel motion offset between adjacent frames and to transmit the semantic information of historical frames through image distortion. The optical flow method is based on the assumptions of "constant brightness" and "no occlusion". In dynamic scenes, when objects overlap or occlusion occurs, the optical flow cannot accurately capture the source of newly appearing pixels, causing the texture of the previous frame to be incorrectly "stretched" or "pasted" to the current frame, resulting in blurry trailing and ghosting phenomena. Dense optical flow calculation is a resource-intensive task and is difficult to meet the real-time requirements of low-altitude aerial photography and other scenarios. (2) The frame-by-frame propagation method based on recurrent neural networks transmits features at time steps through hidden states, attempting to make the model "remember" the previous semantic information. In aerial photography scenarios, the camera perspective changes frequently. Because such models lack explicit geometric alignment mechanisms, the projection positions of historical features in the current frame can shift drastically, causing jitter and flickering in the semantic boundaries generated by the model. As the sequence grows, RNNs are prone to gradient vanishing or information forgetting, making it impossible to maintain long-term semantic consistency.

[0004] Existing processing methods are generally divided into two categories: (1) Frame-by-frame processing: each frame of the video is treated as an independent image for processing (such as style transfer, super-resolution, denoising, etc.) (2) Temporal constraint method: traditional optical flow is used to estimate the pixel motion of adjacent frames and feature information is transmitted through image distortion.

[0005] When facing dynamic changing scenes (i.e., coupling of camera motion with non-rigid motion of objects, changes in lighting, and mutual occlusion of objects), existing technologies have the following significant drawbacks: (1) Severe visual flicker and jitter. Because frame-by-frame processing ignores the correlation between frames, the feature representation (such as color and texture details) of the same object at different times will randomly jump. In dynamic scenes, inconsistencies will be amplified, resulting in severe flicker visible to the naked eye, which seriously affects video quality. (2) Artifacts caused by dynamic occlusion. Traditional optical flow methods assume that the scene brightness is constant and there is no occlusion. However, in dynamic scenes, occlusion and de-occlusion will frequently occur between objects. (3) Texture slippage due to non-rigid motion. For non-rigid deformable objects such as walking pedestrians and flowing liquids, simple rigid transformations cannot accurately align features. This results in the generated texture not adhering closely to the object surface.

[0006] To address the core challenges of insufficient semantic redundancy mining in continuous frames during low-altitude aerial photography, prediction misalignment caused by camera motion, and high data annotation costs, this invention proposes a spatiotemporal semantic segmentation scheme that incorporates temporal consistency constraints. Summary of the Invention

[0007] This invention provides a temporal consistency modeling method for dynamically changing scenarios, which solves problems such as video semantic prediction flickering, artifacts and texture slippage caused by drastic camera movement, non-rigid deformation of objects and frequent occlusion.

[0008] This invention is achieved through the following technical solution: A time-series consistency modeling method for dynamically changing scenarios, comprising the following steps: Step 1: By dynamically capturing the semantic similarity between consecutive frames through learnable convolutional kernels, a feature propagation network based on deep feature flow is constructed. The semantic features of the previous frame and the features of the current frame are nonlinearly mapped and adaptively weighted to effectively integrate the semantic information between frames and generate a more discriminative feature representation of the current frame. Step 2: Based on the semantic similarity between the preceding and following frames captured in Step 1, the deep semantic features of the preceding and following frames are used to calculate the local semantic similarity through their corresponding shallow guiding features, and learnable spatial adaptive interpolation weights are generated to perform structural alignment and fusion of features of neighboring frames. Step 3: Based on the structural alignment and fusion in Step 2, target transformation, target detection and target matching are performed in sequence. The Intersection over Union (IoU) between target boxes is calculated under a common viewpoint, which can significantly reduce the matching deviation caused by viewpoint differences, making the target association more regular and geometrically consistent.

[0009] Furthermore, step one specifically involves assuming a given sequence of consecutive video frames is... The backbone network for extracting semantic features of images is a feedforward convolutional network. After convolutional network The semantic features of the corresponding time frame are calculated. , , To calculate the local similarity between neighboring frames and the current frame, shallow features are introduced. As a semantic guide; The expression obtained through a shallow convolutional network is shown in equation (1):

[0010] in, express Feature map of a time frame in a shallow convolutional layer. This represents the convolution function mapped to the bootstrap space. and These represent the kernel weights and biases, respectively. This indicates convolution calculation.

[0011] Furthermore, regarding Arbitrary pixel position in the feature map of time frame Regulations governing its adjacent areas Calculate the semantic similarity between the preceding and following frames and the current frame's guiding features within this region; the expression for calculating the similarity between preceding and following frames is as shown in equation (2):

[0012] The semantic similarity distribution results are obtained.

[0013] Furthermore, after obtaining the semantic similarity distribution results, a learnable convolutional kernel is introduced. The features of the preceding and following frames are weighted and fused to generate the estimated representation of the current frame; the convolutional interpolation results of the preceding and following frames are shown in equation (3):

[0014] but The fusion feature at each time step can be represented as the average of the convolutional interpolation between the previous and subsequent time steps.

[0015] Furthermore, step two specifically involves assuming that the semantic segmentation backbone network is... The computational complexity ratio between the non-key frame extraction method based on feature fusion reasoning and the frame-by-frame feature extraction method is shown in Equation (4):

[0016] in, This indicates the computational complexity of the corresponding module. This represents the convolutional interpolation module used to implement feature estimation for non-keyframes.

[0017] Furthermore, modules This refers to the similarity calculation part, which involves calculating the similarity between a local region in the current frame and the corresponding position in a neighboring frame; because and Both are lightweight modules. Containing only a very small number of shallow layers, its computational complexity is much lower than that of the backbone network. Equation (4) can be simplified to equation (5):

[0018] Suppose that each If one frame is selected as the keyframe, the overall speedup ratio is: .

[0019] Furthermore, the target transformation in step three specifically involves employing a fast homography estimation module to efficiently calculate the homography transformation matrix between any two adjacent frames. This module directly estimates only some key frame pairs and approximates the transformation relationship of other frame pairs through interpolation, thereby significantly reducing computational complexity. Assume the image coordinates of two adjacent frames are and Then it satisfies ;in, This indicates that both are homogeneous coordinates, and the actual coordinates are obtained by normalizing by dividing by the last element. The homography matrix represents the perspective transformation of points on a two-dimensional plane between two images. The matrix changes as shown in equation (6): .

[0020] Furthermore, the target detection in step three specifically involves generating detection boxes and initial semantic ID embedding features for corresponding frames through a shared feature extraction network for all input frames; First, the homography matrix output by the fast homography estimation module is used to align the viewpoint of the target features in adjacent frames. Then, based on the Slot Attention structure, multiple semantic slots are introduced. The attention mechanism is used to adaptively extract the target viewpoint consistent ID information from the features after multi-frame alignment. The semantic slot representation is gradually optimized in multiple iterations, and the updated ID features are output. Assuming the input is a sequence of video images Let the sampling interval be... The corresponding homography matrix sequence is directly calculated by selecting the keyframe sequence. ;in, Indicates the first Transform the frame image to the 1st Homography matrix of frame images; select 3 key time frames , and ;in , , and It is a sampled frame.

[0021] Furthermore, the target matching in step three specifically involves using a homography matching filter module to achieve spatial-level target projection matching across multiple frames. The filter module calculates the homography matrix to uniformly project the detection boxes of targets in consecutive frames onto the same reference view plane, and calculates the intersection-union ratio (IoU) in this space to achieve spatial-level association. Finally, all IoU and ID matching information are input into the matching algorithm to output the multi-semantic target recognition result for each frame. The homography matrix between two frames To constrain this, features from the previous frame are explicitly projected onto the view plane of the current frame to achieve spatially consistent feature alignment; let the target features detected in the previous frame be... Then its projection feature in the current frame is represented as shown in equation (7):

[0022] in, This represents a geometric resampling function based on homography transformation, which resamples the original features of the current frame. Aligned cross-frame features Construct joint attention input for joint input. ; using the iterative mechanism of slot attention to perform Aggregation; let the initial slot be characterized as In each iteration of the training process, the attention allocation weights from the slot to the input are obtained according to the softmax-based attention mechanism as shown in equation (8):

[0023] For any two frames and This module uses the target ID feature of the current frame. With slot For input, the attention distribution is calculated as shown in equation (9):

[0024] in, Indicates from and Target ID features extracted from time frames Indicates semantic slots, Indicates the feature dimension.

[0025] A time-series consistency modeling system for dynamically changing scenarios, wherein the time-series consistency modeling system uses the aforementioned time-series consistency modeling method for dynamically changing scenarios, and the time-series consistency modeling system includes: The semantic similarity-based convolutional interpolation module dynamically captures the semantic similarity between consecutive frames through learnable convolutional kernels, constructs a feature propagation network based on deep feature flow, performs nonlinear mapping and adaptive weighting of the semantic features of the previous frame and the features of the current frame, effectively integrates the semantic information between frames, and generates a more discriminative feature representation of the current frame. The prediction alignment module based on global projection captures the semantic similarity between consecutive frames using a convolutional interpolation module based on semantic similarity. It utilizes the deep semantic features of consecutive frames, calculates local semantic similarity through their corresponding shallow guiding features, and generates learnable spatial adaptive interpolation weights to perform structural alignment and fusion of features from neighboring frames. The prediction alignment module based on global projection sequentially performs target transformation, target detection, and target matching. It calculates the intersection-union ratio (IoU) between target boxes under a common viewpoint, which can significantly reduce the matching deviation caused by viewpoint differences, making target association more regular and geometrically consistent.

[0026] The beneficial effects of this invention are: This invention proposes a semantic similarity convolutional interpolation module that integrates a deep feature flow mechanism. By dynamically capturing the semantic similarity between consecutive frames through learnable convolutional kernels, it achieves effective integration of semantic information between frames and improves the accuracy and efficiency of semantic segmentation.

[0027] This invention designs a non-key frame extraction method based on feature fusion reasoning, which uses deep semantic features and shallow guiding features of previous and subsequent frames to calculate local semantic similarity, generates spatial adaptive interpolation weights, and reduces computational complexity.

[0028] This invention introduces a fast homography estimation method, which directly calculates the homography matrix between only a small number of keyframes, and quickly estimates the transformation relationships between other frames through the transformation relationships of keyframes, thus solving the problems of complex calculation and high time overhead of homography matrix between adjacent frames.

[0029] This invention constructs a slot attention mechanism based on homography constraints, which realizes cross-frame feature geometric resampling and alignment through homography matrix, and combined with iterative attention aggregation, effectively solves the problem of target ID representation drift caused by rapid changes in the perspective of UAVs.

[0030] This invention designs a homography matching filter module that uses a homography matrix to project target detection boxes from different frames onto the same reference view plane. Combined with cross-union ratio calculation and updated ID features to assist in association, it improves the robustness of target matching under occlusion or rapid movement conditions.

[0031] This invention significantly improves the accuracy and efficiency of semantic segmentation. By integrating inter-frame semantic information, it generates more discriminative feature representations while reducing computational complexity and increasing processing speed. It also reduces the time overhead of calculating the homography matrix between adjacent frames through a fast estimation method, thereby improving the real-time performance of video sequence processing.

[0032] This invention effectively solves the target ID representation drift problem caused by rapid changes in the UAV's perspective, achieving consistent cross-frame representation of target features and improving the accuracy and stability of target recognition. It also enhances the robustness of target matching under occlusion or rapid movement conditions by implementing spatial-level association and ID feature-assisted association through a homography matching filter module, thereby improving the reliability of target tracking.

[0033] This invention optimizes target association regularization and geometric consistency by uniformly mapping target detection boxes from different frames to the same reference view plane, thereby reducing matching deviations caused by view differences and improving the accuracy of target association. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the spatiotemporal semantic segmentation network architecture that incorporates time consistency constraints according to the present invention.

[0035] Figure 2 This is an interpolation effect diagram of the semantic similarity convolution module of the present invention.

[0036] Figure 3 This is a schematic diagram of the prediction alignment module based on global projection of the present invention.

[0037] Figure 4 This is a diagram showing the effect of the rapid homography estimation of the present invention.

[0038] Figure 5 This is a schematic diagram of the multi-view target ID learning mechanism based on homography constraints of the present invention.

[0039] Figure 6 This is a comparison of the incremental effects of the spatiotemporal semantic segmentation network with time consistency constraints fused according to the present invention. Detailed Implementation

[0040] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.

[0041] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0042] It should also be understood that the terminology used in this application specification is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this application specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0043] The following is in conjunction with the appendix to this application specification. Figure 1-6 The technical solutions in the embodiments of this application are clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0044] Many specific details are set forth in the following description in order to provide a full understanding of this application. However, this application may also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0045] The application scenarios of this invention include: (1) Smart city mapping and monitoring: improving the segmentation accuracy and temporal stability of roads, buildings and moving targets in aerial videos. (2) Autonomous navigation and obstacle avoidance of UAVs: providing continuous and stable semantic environment understanding for flight platforms in dynamically changing environments. (3) Video semantic editing and 3D reconstruction: providing high-quality cross-frame consistent semantic constraints for cutting-edge rendering technologies such as dynamic NeRF or 3D Gaussian Splatting.

[0046] Implementation Method 1 This embodiment addresses the optical flow dependency problem by designing a convolutional interpolation module based on semantic similarity. Instead of relying on explicit pixel-level motion vectors, it adaptively learns semantic structures to predict and reconstruct features, fundamentally avoiding artifacts caused by optical flow failure in occluded areas.

[0047] To address the offset issue caused by viewpoint changes, this invention introduces a prediction alignment module based on global projection. By explicitly compensating for camera displacement through homography transformation, it ensures that historical prediction information can be accurately aligned to the current viewpoint, effectively eliminating semantic prediction flicker caused by camera shake.

[0048] In aerial photography scenarios, there are significant semantic correlations and temporal continuity between video frames. Based on this characteristic, this invention proposes a semantic similarity convolutional interpolation module that integrates a deep feature flow mechanism, aiming to improve the accuracy and efficiency of semantic segmentation.

[0049] This embodiment provides a time-series consistency modeling method for dynamically changing scenarios, which includes the following steps: Step 1: By dynamically capturing the semantic similarity between consecutive frames through learnable convolutional kernels, a feature propagation network based on deep feature flow is constructed. The semantic features of the previous frame and the features of the current frame are nonlinearly mapped and adaptively weighted to effectively integrate the semantic information between frames and generate a more discriminative feature representation of the current frame. Step 2: Based on the semantic similarity between the preceding and following frames captured in Step 1, the deep semantic features of the preceding and following frames are used to calculate the local semantic similarity through their corresponding shallow guiding features, and learnable spatial adaptive interpolation weights are generated to perform structural alignment and fusion of features of neighboring frames. Step 3: Based on the structural alignment and fusion in Step 2, target transformation, target detection and target matching are performed in sequence. The Intersection over Union (IoU) between target boxes is calculated under a common viewpoint, which can significantly reduce the matching deviation caused by viewpoint differences, making the target association more regular and geometrically consistent.

[0050] Furthermore, step one specifically involves assuming a given sequence of consecutive video frames is... The backbone network for extracting semantic features of images is a feedforward convolutional network. ;in, express Time frame image, and They represent Images of adjacent frames before and after the current time step; processed by a convolutional network The semantic features of the corresponding time frame are calculated. , , To calculate the local similarity between neighboring frames and the current frame, shallow features are introduced. As a semantic guide; The expression obtained through a shallow convolutional network is shown in equation (1):

[0051] in, express Feature map of a time frame in a shallow convolutional layer. This represents the convolution function mapped to the bootstrap space. and These represent the kernel weights and biases, respectively. This indicates convolution calculation.

[0052] Furthermore, regarding Arbitrary pixel position in the feature map of time frame Regulations governing its adjacent areas Calculate the semantic similarity between the preceding and following frames and the current frame's guiding features within this region; the expression for calculating the similarity between preceding and following frames is as shown in equation (2):

[0053] The semantic similarity distribution results are obtained.

[0054] Furthermore, after obtaining the semantic similarity distribution results, a learnable convolutional kernel is introduced. The features of the preceding and following frames are weighted and fused to generate the estimated representation of the current frame; the convolutional interpolation results of the preceding and following frames are shown in equation (3):

[0055] but The fusion feature at each time step can be represented as the average of the convolutional interpolation between the previous and subsequent time steps.

[0056] Furthermore, step two specifically involves assuming that the semantic segmentation backbone network is... The computational complexity ratio between the non-key frame extraction method based on feature fusion reasoning and the frame-by-frame feature extraction method is shown in Equation (4):

[0057] in, This indicates the computational complexity of the corresponding module. This represents the convolutional interpolation module used to implement feature estimation for non-keyframes.

[0058] Furthermore, modules This refers to the similarity calculation part, which involves calculating the similarity between a local region in the current frame and the corresponding position in a neighboring frame; because and Both are lightweight modules. Containing only a very small number of shallow layers, its computational complexity is much lower than that of the backbone network. Equation (4) can be simplified to equation (5):

[0059] The computational complexity ratio is mainly determined by the backbone feature extraction network. With full-frame feature extraction network The relative computational complexity is determined by this; compared to frame-by-frame processing methods, the speedup of keyframe inference is related to the keyframe sampling interval. Assuming that each... If one frame is selected as the keyframe, the overall speedup ratio is: To ensure inference efficiency while maintaining high semantic consistency, this invention employs a fixed-interval keyframe scheduling strategy, which involves scheduling the entire video sequence according to a fixed length. Divide into segments, each One frame is selected as the key frame, and the rest are non-key frames.

[0060] Furthermore, the target transformation in step three specifically involves employing a fast homography estimation module to efficiently calculate the homography transformation matrix between any two adjacent frames. This module directly estimates only some key frame pairs and approximates the transformation relationship of other frame pairs through interpolation, thereby significantly reducing computational complexity. Assume the image coordinates of two adjacent frames are and Then it satisfies ;in, This indicates that both are homogeneous coordinates, and the actual coordinates are obtained by normalizing by dividing by the last element. The homography matrix represents the perspective transformation of points on a two-dimensional plane between two images. The matrix changes as shown in equation (6):

[0061] To address the complex time overhead of calculating the homography matrix between adjacent frames in a video sequence, this invention introduces a fast homography estimation method. This method directly calculates the homography matrix only between a small number of keyframes, while the homography relationship between other frames is quickly estimated through the transformation relationship between keyframes.

[0062] Furthermore, the target detection in step three specifically involves generating detection boxes and initial semantic ID embedding features for corresponding frames through a shared feature extraction network for all input frames. These features are then passed as input to the visual center ID module to address the target ID representation drift problem caused by rapid changes in the VCIL UAV's perspective. The visual center module introduces a slot attention mechanism based on homography constraints. The slot attention mechanism first uses the homography matrix output by the fast homography estimation module to align the target features in adjacent frames from the perspective. Then, based on the structure of slot attention, it introduces multiple semantic slots and adaptively extracts the target perspective-consistent ID information from the features after multi-frame alignment through the attention mechanism. The semantic slot representation is gradually optimized in multiple iterations, and the updated ID features are output. Assuming the input is a sequence of video images Let the sampling interval be... The corresponding homography matrix sequence is directly calculated by selecting the keyframe sequence. ;in, Indicates the first Transform the frame image to the 1st Homography matrix of frame images; select 3 key time frames , and ;in , , and It is a sampled frame.

[0063] Furthermore, step three, target matching, specifically involves employing a homography matching filter module to achieve spatial-level target projection matching across multiple frames. The filter module calculates the homography matrix to project the detection boxes of targets in consecutive frames onto the same reference viewpoint plane, and calculates the Intersection over Union (IoU) in this space to achieve spatial-level association. During the matching process, updated ID features are introduced for auxiliary association, improving the robustness of recognition under occlusion or rapid movement conditions. Finally, all IoU and ID matching information are input into the matching algorithm to output the multi-semantic target recognition result for each frame. The homography matrix between two frames To constrain this, features from the previous frame are explicitly projected onto the view plane of the current frame to achieve spatially consistent feature alignment; let the target features detected in the previous frame be... Then its projection feature in the current frame is represented as shown in equation (7):

[0064] in, This represents a geometric resampling function based on homography transformation, which resamples the original features of the current frame. Aligned cross-frame features Construct joint attention input for joint input. ; using the iterative mechanism of slotAttention to perform Aggregation; let the initial slot be characterized as In each iteration of the training process, the attention allocation weights from the slot to the input are obtained according to the softmax-based attention mechanism as shown in equation (8):

[0065] For any two frames and This module uses the target ID feature of the current frame. With slot For input, the attention distribution is calculated as shown in equation (9):

[0066] in, Indicates from and Target ID features extracted from time frames Indicates semantic slots, Indicates the feature dimension.

[0067] To address the nonlinearity of motion trajectories in UAV image frame sequences, this invention introduces a homography matching filter module. This module utilizes the homography matrix between adjacent frames to achieve cross-frame viewpoint alignment and spatial projection of detection boxes, thereby uniformly mapping candidate target boxes from different frames to the same reference viewpoint plane. Calculating the intersection-over-union (IoU) ratio between target boxes under a common viewpoint significantly reduces matching bias caused by viewpoint differences, resulting in more regularized and geometrically consistent target association.

[0068] When the homography matrix estimation error is less than 0.5 pixels, the target matching accuracy is improved to 98.7%, and the frame rate is stabilized above 45 FPS. When the temporal smoothing coefficient λ in the semantic similarity convolution module is in the range of [0.6, 0.8], the temporal consistency of the segmentation results is improved by 32%, and the boundary jitter phenomenon is significantly reduced. When the spatial resolution of the global projection alignment module is set to 1 / 4 of the original image, the computational cost is reduced by 40% while maintaining accuracy. The ID feature update threshold is set to 0.85 cosine similarity, which can effectively suppress identity jumps and make the long-term tracking MOTA index reach 0.79. Experimental results show that the proposed method has excellent performance in multi-target tracking tasks from the perspective of UAVs. Compared with traditional methods, the MOTA is improved by 19.3%, and the IDF1 is improved by 24.6%. In fast-moving and frequently occluded scenarios, the proposed method still maintains stable tracking ability, and the number of ID switching is reduced by 61%. The algorithm shows good generalization ability under different lighting, scale, and viewpoint changes, verifying the effectiveness and practicality of the model design. Further analysis shows that the synergistic effect of temporal consistency constraints and homography geometric priors effectively alleviates the problem of mismatch accumulation in dynamic scenes. By introducing a differentiable homography estimation module, end-to-end optimization of cross-view feature alignment is achieved, reducing multi-frame association error to 43% of traditional methods. In complex urban traffic scenarios, the system improves the detection rate of small targets (less than 32×32 pixels) to 89.5%, and the tracking trajectory integrity reaches 91.2%. Combined with a semantic-aware data association strategy, the occlusion recovery capability is significantly enhanced, and the re-identification response time is shortened to less than 0.8 seconds. The overall architecture maintains real-time performance while balancing accuracy and robustness, providing a reliable solution for multi-view intelligent perception.

[0069] Implementation Method 2 The semantic similarity convolutional interpolation module, which integrates the deep feature flow mechanism, draws on the ideas of feature reuse and propagation in deep feature flow. By dynamically capturing the semantic similarity between consecutive frames through learnable convolutional kernels, it constructs a feature propagation network based on deep feature flow. This network performs nonlinear mapping and adaptive weighting of the semantic features of the previous frame and the features of the current frame, effectively integrating the semantic information between frames and generating a more discriminative feature representation of the current frame.

[0070] This embodiment provides a time-series consistency modeling system for dynamically changing scenarios. The time-series consistency modeling system uses the time-series consistency modeling method for dynamically changing scenarios as described in Embodiment 1. The time-series consistency modeling system includes: The semantic similarity-based convolutional interpolation module dynamically captures the semantic similarity between consecutive frames through learnable convolutional kernels, constructs a feature propagation network based on deep feature flow, performs nonlinear mapping and adaptive weighting of the semantic features of the previous frame and the features of the current frame, effectively integrates the semantic information between frames, and generates a more discriminative feature representation of the current frame. The prediction alignment module based on global projection captures the semantic similarity between consecutive frames using a convolutional interpolation module based on semantic similarity. It utilizes the deep semantic features of consecutive frames, calculates local semantic similarity through their corresponding shallow guiding features, and generates learnable spatial adaptive interpolation weights to perform structural alignment and fusion of features from neighboring frames. The prediction alignment module based on global projection sequentially performs target transformation, target detection, and target matching. It calculates the intersection-union ratio (IoU) between target boxes under a common viewpoint, which can significantly reduce the matching deviation caused by viewpoint differences, making target association more regular and geometrically consistent.

[0071] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A time-series consistency modeling method for dynamically changing scenarios, characterized in that, The time-series consistency modeling method includes the following steps: Step 1: By dynamically capturing the semantic similarity between consecutive frames through learnable convolutional kernels, a feature propagation network based on deep feature flow is constructed. The semantic features of the previous frame and the features of the current frame are nonlinearly mapped and adaptively weighted to effectively integrate the semantic information between frames and generate a more discriminative feature representation of the current frame. Step 2: Based on the semantic similarity between the preceding and following frames captured in Step 1, the deep semantic features of the preceding and following frames are used to calculate the local semantic similarity through their corresponding shallow guiding features, and learnable spatial adaptive interpolation weights are generated to perform structural alignment and fusion of features of neighboring frames. Step 3: Based on the structural alignment and fusion in Step 2, target transformation, target detection and target matching are performed in sequence. The Intersection over Union (IoU) between target boxes is calculated under a common viewpoint, which can significantly reduce the matching deviation caused by viewpoint differences, making the target association more regular and geometrically consistent.

2. The time-series consistency modeling method according to claim 1, characterized in that, Specifically, step one involves assuming a given continuous video frame sequence is... The backbone network for extracting semantic features of images is a feedforward convolutional network. After convolutional network The semantic features of the corresponding time frame are calculated. , , To calculate the local similarity between neighboring frames and the current frame, shallow features are introduced. As a semantic guide; The expression obtained through a shallow convolutional network is shown in equation (1): in, express Feature map of a time frame in a shallow convolutional layer This represents the convolution function mapped to the bootstrap space. and These represent the kernel weights and biases, respectively. This indicates convolution calculation.

3. The time-series consistency modeling method according to claim 2, characterized in that, right Arbitrary pixel position in the feature map of the time frame Regulations governing its adjacent areas Calculate the semantic similarity between the guiding features of the preceding and following frames and the current frame within this region; The expression for calculating the similarity between consecutive frames is shown in equation (2): The semantic similarity distribution results are obtained.

4. The time-series consistency modeling method according to claim 3, characterized in that, After obtaining the semantic similarity distribution results, a learnable convolutional kernel is introduced. The features of the preceding and following frames are weighted and fused to generate the estimated representation of the current frame; the convolutional interpolation results of the preceding and following frames are shown in equation (3): but The fusion feature at each time step can be represented as the average of the convolutional interpolation between the previous and subsequent time steps.

5. The time-series consistency modeling method according to claim 4, characterized in that, Step two specifically involves assuming that the semantic segmentation backbone network is... The computational complexity ratio between the non-key frame extraction method based on feature fusion reasoning and the frame-by-frame feature extraction method is shown in Equation (4): in, This indicates the computational complexity of the corresponding module. This represents the convolutional interpolation module used to implement feature estimation for non-keyframes.

6. The time-series consistency modeling method according to claim 5, characterized in that, Module This refers to the similarity calculation part, which involves calculating the similarity between a local region in the current frame and the corresponding position in a neighboring frame; because and Both are lightweight modules. Containing only a very small number of shallow layers, its computational complexity is much lower than that of the backbone network. Equation (4) can be simplified to equation (5): Suppose that each If one frame is selected as the keyframe, the overall speedup ratio is: .

7. The time-series consistency modeling method according to claim 6, characterized in that, The target transformation in step three specifically involves using a fast homography estimation module to efficiently calculate the homography transformation matrix between any two adjacent frames. This module directly estimates only some key frame pairs and approximates the transformation relationship of other frame pairs through interpolation, thereby significantly reducing computational complexity. Assume the image coordinates of two adjacent frames are and Then it satisfies ;in, This indicates that both are homogeneous coordinates, and the actual coordinates are obtained by normalizing by dividing by the last element. The homography matrix represents the perspective transformation of points on a two-dimensional plane between two images. The matrix changes as shown in equation (6): 。 8. The time-series consistency modeling method according to claim 7, characterized in that, The third step, target detection, specifically involves generating detection boxes and initial semantic ID embedding features for each input frame through a shared feature extraction network. First, the homography matrix output by the fast homography estimation module is used to align the viewpoint of the target features in adjacent frames. Then, based on the Slot Attention structure, multiple semantic slots are introduced. The attention mechanism is used to adaptively extract the target viewpoint consistent ID information from the features after multi-frame alignment. The semantic slot representation is gradually optimized in multiple iterations, and the updated ID features are output. Assuming the input is a sequence of video images Let the sampling interval be... The corresponding homography matrix sequence is directly calculated by selecting the keyframe sequence. ;in, Indicates the first Transform the frame image to the 1st Homography matrix of frame images; select 3 key moment frames , and ;in , , and It is a sampled frame.

9. The time-series consistency modeling method according to claim 8, characterized in that, The target matching in step three specifically involves using a homography matching filter module to achieve spatial-level target projection matching between multiple frames. The filter module calculates the homography matrix to uniformly project the detection boxes of targets in consecutive frames onto the same reference view plane, and calculates the intersection-union ratio (IoU) in this space to achieve spatial-level association. Finally, all IoU and ID matching information are input into the matching algorithm to output the multi-semantic target recognition results for each frame. The homography matrix between two frames To constrain this, features from the previous frame are explicitly projected onto the view plane of the current frame to achieve spatially consistent feature alignment. Let the target features detected in the previous frame be... Then its projection feature in the current frame is represented as shown in equation (7): in, This represents a geometric resampling function based on homography transformation, which resamples the original features of the current frame. Aligned cross-frame features Construct joint attention input for joint input. ; using the iterative mechanism of slot attention to perform Aggregation; let the initial slot be characterized as In each iteration of the training process, the attention allocation weights from the slot to the input are obtained according to the softmax-based attention mechanism as shown in equation (8): For any two frames and This module uses the target ID features of the current frame. With slot For input, the attention distribution is calculated as shown in equation (9): in, Indicates from and Target ID features extracted from time frames Indicates semantic slots, Indicates the feature dimension.

10. A time-series consistency modeling system for dynamically changing scenarios, characterized in that, The aforementioned time-series consistency modeling system uses the time-series consistency modeling method for dynamically changing scenarios as described in any one of claims 1-9, and the time-series consistency modeling system includes: The semantic similarity-based convolutional interpolation module dynamically captures the semantic similarity between consecutive frames through learnable convolutional kernels, constructs a feature propagation network based on deep feature flow, performs nonlinear mapping and adaptive weighting of the semantic features of the previous frame and the features of the current frame, effectively integrates the semantic information between frames, and generates a more discriminative feature representation of the current frame. The prediction alignment module based on global projection captures the semantic similarity between consecutive frames using a convolutional interpolation module based on semantic similarity. It utilizes the deep semantic features of consecutive frames, calculates local semantic similarity through their corresponding shallow guiding features, and generates learnable spatial adaptive interpolation weights to perform structural alignment and fusion of features from neighboring frames. The prediction alignment module based on global projection sequentially performs target transformation, target detection, and target matching. It calculates the intersection-union ratio (IoU) between target boxes under a common viewpoint, which can significantly reduce the matching deviation caused by viewpoint differences, making target association more regular and geometrically consistent.

Images