A query passing based motion-aware multi-target tracking method

Abstract

The present application relates to the technical field of multi-target tracking, in particular to a motion-aware multi-target tracking method based on query transfer, comprising: receiving visual target query and multi-source data stream; processing the visual query by using a combined expert decoder to generate visual behavior representation adaptable to complex motion of the target; performing time sequence coding on the multi-source data stream to generate data sequence; generating time sequence consistency reward value by calculating the overlap degree of visual residence and data active time sequence, and using the value to perform reinforcement learning training on the association strategy model; inputting the fused state space into the model to generate association decision and construct global identity trajectory; when the target reappears, performing re-binding through a multi-dimensional matching mechanism that fuses appearance, motion and spatial priori. By fusing visual trajectory and multi-source Internet of Things data and introducing reinforcement learning to optimize long-term association strategy, long-term cross-modal target tracking with precision is realized in a complex farm interaction scene.

Description

Technical Field

[0001] This invention relates to the field of multi-target tracking technology, specifically a motion-aware multi-target tracking method based on query propagation. Background Technology

[0002] Multi-target tracking technology plays a crucial role in fields such as smart agriculture and security monitoring. In particular, in dynamic scenarios where individuals have similar appearances and frequent interactions, higher demands are placed on the accuracy, long-term tracking, and behavior recognition of targets.

[0003] In specific applications such as smart agriculture, in addition to video sequences, there is often multi-source IoT data reflecting the individual's state, such as data streams from electronic ear tags or feeding stations. Existing technologies rarely effectively integrate these two heterogeneous data sources, resulting in an incomplete understanding of the target state.

[0004] Furthermore, some tracking methods extract instantaneous motion information of the target by analyzing video frames. However, this information is usually only used for association decisions in the current frame. There is a lack of a mechanism to effectively encode the instantaneous state and pass it to subsequent frames for long-term analysis, which limits the ability to analyze the target's continuous behavior patterns.

[0005] Meanwhile, identity re-identification is a core challenge when a target is occluded for a long time or enters or exits the field of view. Existing methods treat identity association as a series of independent frame-by-frame matching tasks. The greedy matching strategy they employ lacks consideration for long-term temporal consistency. This decision-making mechanism is prone to irreversible identity switching due to momentary misjudgment after the target is occluded for a long time. How to model this process as a sequential decision problem that can learn the long-term optimal association strategy and introduce frameworks such as reinforcement learning to solve it is a challenge currently facing the technology.

[0006] Therefore, this invention proposes a motion-aware multi-target tracking method based on query propagation. Summary of the Invention

[0007] The purpose of this invention is to provide a motion-aware multi-target tracking method based on query propagation. By fusing visual trajectories with multi-source IoT data and introducing reinforcement learning to optimize long-term association strategies, it achieves accurate and long-term cross-modal target tracking in complex farm interaction scenarios.

[0008] To achieve the above objectives, the present invention provides the following technical solution: A motion-aware multi-target tracking method based on query propagation includes: Receive detected visual target queries and multi-source data streams containing feeding station data from farm surveillance video; The visual target query input is combined with an expert decoder, and the computation results of the expert branches are fused through a gating network to generate appearance query embedding and instantaneous motion gating weights, which are then temporally bound to generate a visual behavior representation. Temporal feature encoding is performed on multi-source data streams to generate data sequences containing feeding station data, and visual behavior representations and data sequences are combined into a cross-modal associated state space; The temporal correlation between the dwell time of visual behavior representation and the activity time of data sequence is analyzed and calculated to generate a temporal consistency reward value, and the association policy model is trained by reinforcement learning; the cross-modal association state space is input into the trained association policy model to generate association decision actions. Based on associated decision actions, a global identity trajectory is generated; when a visual target moves out of the monitoring field of view, the global identity trajectory is stored in the dormancy pool; when a new target enters, a new visual behavior representation is generated, and the new visual behavior representation is matched with the global identity trajectory in the dormancy pool in multiple dimensions, and rebinding is performed.

[0009] Preferably, the combined expert decoder adopts a multi-branch parallel structure: the visual target query is structured data, including the target's position information in the current video frame and feature vectors that can characterize appearance; the multi-branch parallel structure receives the target position information from the previous video frame; wherein, each expert branch extrapolates the target position information from the previous video frame to generate a hypothetical future target region, and extracts candidate visual features from the hypothetical future target region as the calculation result; the motion prior on which the extrapolation operation is based is defined by assigning different hypothetical motion models to each expert branch, the hypothetical motion models being used to predict future displacement based on the motion prior and define the hypothetical future target region; the gating network parses instantaneous motion gating weights based on the matching degree between the visual target query and the candidate visual features, and uses the instantaneous motion gating weights to weight and fuse the candidate visual features output by each expert branch to generate motion-aware contextual features; the visual target query interacts with the motion-aware contextual features to update the appearance query embedding of the current frame.

[0010] Preferably, the process of generating visual behavior representation includes: embedding the updated appearance query of the current video frame into a fixed-length appearance feature queue, and storing the corresponding instantaneous motion gating weights into a fixed-length motion feature queue, thereby generating a temporal appearance state sequence and a temporal motion state sequence respectively; using a temporal multimodal fusion network, performing internal temporal dependency modeling on the temporal appearance state sequence and the temporal motion state sequence respectively, and performing cross-modal temporal information interaction on the temporal appearance state sequence and the temporal motion state sequence, and aggregating the output of the fusion network into a fixed-dimensional vector, which is the visual behavior representation.

[0011] Preferably, the process of combining the cross-modal associated state space includes: the multi-source data stream further includes electronic ear tag data; at each corresponding time step, weight data and physiological data are extracted from the feeding station data and electronic ear tag data respectively, and after normalization processing, the weight data and physiological data are mapped and combined into a unified high-dimensional feature vector; the unified high-dimensional feature vectors generated at multiple time steps constitute a temporal feature sequence; the temporal feature sequence is input into a temporal coding network, which models the temporal dependencies within the sequence through an attention mechanism to generate the data sequence; the visual behavior representation is combined with the data sequence to generate the cross-modal associated state space.

[0012] Preferably, the process of generating the temporal consistency reward value includes: determining a dwell time interval defined by a set of consecutive video frames, wherein in each frame of the consecutive video frames constituting the visual behavior representation, the original target spatial coordinates of the visual target query in this video frame all fall within a preset feeding station geofence; extracting the original timestamp information contained therein from the original temporal feature sequence corresponding to the input end of the data sequence at the temporal coding network, and aggregating consecutive timestamps with a time interval less than a preset threshold into a weight data active time interval; calculating the total overlap duration between the dwell time interval and the weight data active time interval, and normalizing the duration as the temporal consistency reward value.

[0013] Preferably, the associated decision action is generated by an association strategy model, which includes: a Transformer encoder containing a self-attention layer and a cross-attention layer, which performs multimodal fusion encoding on the visual behavior representation and data sequence in the cross-modal association state space; wherein, the self-attention layer is responsible for modeling the internal temporal dependencies of each sequence, and the cross-attention layer is responsible for cross-modal information interaction between the two sequences to jointly generate the fused state encoding vector; and an action decoding network that receives the state encoding vector and parses out the associated decision action.

[0014] Preferably, the process of generating a global identity trajectory includes: receiving corresponding visual target queries in parallel from multiple different surveillance video streams covering the farm; the global identity trajectory is constructed into a unified data structure, which binds the persistent global identity with multiple appearance query embedding sequences corresponding to the visual target in different surveillance video streams, as well as the associated data sequences, for cross-camera tracking.

[0015] Preferably, the process of generating a new visual behavior representation includes: when a new target enters, caching the appearance query embedding sequence and instantaneous motion gating weight sequence of the new target within a preset initial time window; at each time step of the initial time window, constructing an appearance state vector from the corresponding appearance query embedding and a motion state vector from the instantaneous motion gating weights, thus forming an initial appearance state sequence and an initial motion state sequence, respectively; applying a temporal multimodal fusion network to the initial appearance state sequence and the initial motion state sequence, wherein the temporal multimodal fusion network performs internal temporal dependency modeling on the initial appearance state sequence and the initial motion state sequence, and performs cross-modal temporal information interaction on the initial appearance state sequence and the initial motion state sequence, and aggregating the output of the fusion network into a fixed-dimensional vector, wherein the fixed-dimensional vector is the new visual behavior representation.

[0016] Preferably, the rebinding process includes: calculating the highest similarity between the appearance query embedding sequence within the initial time window and the appearance query embedding sequence contained in each global identity trajectory in the dormant pool to obtain a first similarity score; aggregating the instantaneous motion gating weight sequence within the initial time window into an initial motion pattern vector through a temporal coding network, aggregating the historical motion weight sequence contained in each global identity trajectory in the dormant pool into a historical motion pattern vector through the same temporal coding network, calculating the cosine similarity between the initial motion pattern vector and the historical motion pattern vector to obtain a second similarity score; weighting the first similarity score, the second similarity score, and the cross-camera re-entry probability provided by the camera topology map, performing candidate sorting on the global identity trajectories in the dormant pool, and selecting the Top-K candidate data sequences; selecting the candidate identity with the highest comprehensive score from the Top-K candidates according to a preset decision threshold; if selected, activating the global identity trajectory of the candidate identity from the dormant pool and rebinding it with the new visual behavior representation of the current new target.

[0017] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. By combining expert decoders to analyze instantaneous motion with a temporally bound multimodal fusion network, a deep representation of target dynamic behavior is achieved. This invention utilizes a combined expert decoder to extract the instantaneous motion gating weights of the target, embedding them together with appearance queries into a temporal queue, and then using a fusion network for dependency modeling and information interaction. This design effectively encodes and transmits the instantaneous motion state of the target to subsequent frames, aggregating it into a visual behavior representation that combines appearance and motion temporal information. This solves the problem of the lack of long-term motion analysis mechanisms in existing technologies, providing a richer and more robust feature foundation for subsequent accurate association and behavior recognition.

[0018] 2. By constructing a cross-modal associated state space and applying a reinforcement learning association method based on temporal consistency rewards, long-term optimal identity association under heterogeneous data fusion is achieved. This invention effectively integrates visual behavioral representations with IoT data sequences from feeding stations and electronic ear tags, overcoming the limitation of existing technologies that cannot fully understand the target state. More importantly, it models identity association as a sequential decision problem, using temporal consistency rewards to guide the reinforcement learning model to escape local optima and learn an association strategy with long-term temporal consistency. This overcomes the fundamental defect of traditional greedy matching, which leads to irreversible identity switching due to instantaneous misjudgment after long-term target occlusion.

[0019] 3. By establishing a global identity trajectory data structure and designing a dormant rebinding collaborative mechanism based on multi-dimensional similarity matching, seamless identity tracking across cameras and over long time periods is achieved. This invention constructs a persistent global identity for each target, binding visual features from multiple cameras and IoT data. When a target re-enters the field of view, this mechanism does not rely solely on a single appearance feature, but rather makes a decision through a multi-dimensional matching strategy that integrates appearance sequence similarity, motion pattern vector similarity, and camera topology probability. This significantly improves the accuracy of identity re-identification in scenarios where the target disappears for a long time or reappears across cameras, ensuring the continuity and global uniqueness of tracking in large-scale monitoring scenarios such as smart farms. Attached Figure Description

[0020] Figure 1 This is a flowchart of a motion-aware multi-target tracking method based on query propagation according to the present invention; Figure 2 This is a flowchart of a method for generating cross-modal related decision actions according to an embodiment of the present invention; Figure 3 This is a flowchart of a method for rebinding newly emerging targets according to an embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Other embodiments obtained by those skilled in the art based on the ideas in this specification without creative effort all fall within the protection scope of this invention.

[0022] Reference Figures 1 to 3 This invention provides a motion-aware multi-target tracking method based on query propagation, the technical solution of which is as follows.

[0023] A motion-aware multi-target tracking method based on query propagation includes: Receive detected visual target queries and multi-source data streams containing feeding station data from farm surveillance video; The visual target query input is combined with an expert decoder, and the computation results of the expert branches are fused through a gating network to generate appearance query embedding and instantaneous motion gating weights, which are then temporally bound to generate a visual behavior representation. Temporal feature encoding is performed on multi-source data streams to generate data sequences containing feeding station data, and visual behavior representations and data sequences are combined into a cross-modal associated state space; The temporal correlation between the dwell time of visual behavior representation and the activity time of data sequence is analyzed and calculated to generate a temporal consistency reward value, and the association policy model is trained by reinforcement learning; the cross-modal association state space is input into the trained association policy model to generate association decision actions. Based on associated decision actions, a global identity trajectory is generated; when a visual target moves out of the monitoring field of view, the global identity trajectory is stored in the dormancy pool; when a new target enters, a new visual behavior representation is generated, and the new visual behavior representation is matched with the global identity trajectory in the dormancy pool in multiple dimensions, and rebinding is performed.

[0024] Example 1: This embodiment provides a motion-aware multi-target tracking method based on query propagation, referring to... Figure 1 This method was deployed in a modern dairy farm to continuously track the identity of each cow in the barn and automatically record its behavior.

[0025] Specifically, multiple fixed-position cameras are deployed in a monitoring network covering the main activity areas of the cattle shed (including rest areas and feeding areas). These cameras continuously capture video streams at a frame rate of 25 frames per second and a resolution of 1920×1080 pixels. The video stream is fed into a pre-defined object detection model. This model processes the video frame by frame, and when a cow is identified, it generates a visual object query for that target. This visual object query is a structured data packet containing: the bounding box position of the cow in the current frame's image coordinate system (e.g., a rectangular area defined by the coordinates of the top-left and bottom-right corners), and a 256-dimensional appearance feature vector extracted by the intermediate layer network of the object detection model, characterizing the cow's local appearance texture. Simultaneously, the method receives a separate multi-source data stream from the farm's central management system via a data interface. This data stream contains data records from all automated feeding stations. For example, when cow number "N073" enters feeding station 3, the RFID reader inside the station identifies its electronic ear tag, the weighing module records its weight, and the feed dispensing system records its feed intake. A corresponding feeding station data record is generated, containing an event timestamp, the feeding station ID "FS03", the cow's global identity ID "N073", and the weight data "715.6kg". Both of these data streams (visual target query stream and feeding station data stream) have precise timestamps and are continuously received by the method as the basis for subsequent time-series correlation.

[0026] Furthermore, the combined expert decoder adopts a multi-branch parallel structure: the visual target query is structured data, containing the target's position information in the current video frame and feature vectors that characterize its appearance; the multi-branch parallel structure receives the target position information from the previous video frame; wherein, each expert branch extrapolates the target position information from the previous video frame to generate a hypothetical future target region, and extracts candidate visual features from the hypothetical future target region as the calculation result; the motion prior on which the extrapolation operation is based is defined by assigning different hypothetical motion models to each expert branch, and the hypothetical motion models are used to predict future displacements based on the motion prior to define the hypothetical future target region; the gating network parses instantaneous motion gating weights based on the matching degree between the visual target query and the candidate visual features, and uses the instantaneous motion gating weights to weight and fuse the candidate visual features output by each expert branch to generate motion-aware contextual features; the visual target query interacts with the motion-aware contextual features to update the appearance query embedding of the current frame.

[0027] Specifically, the visual target query is a structured data packet, including the bounding box position of the cow in the current frame image coordinate system (e.g., a rectangular region defined by the coordinates of the top left and bottom right corners), and a 256-dimensional appearance feature vector extracted by the intermediate layer network of the target detection model, which characterizes the local appearance texture of the cow. When processing the t-th frame of the video stream, the combined expert decoder receives the target position information of a tracked cow from the previous frame (t-1 frame), for example, its bounding box coordinates are (210, 350, 290, 480). The decoder internally has four parallel expert branches, each pre-assigned an independent hypothetical motion model. For example, the first branch's motion model is "stationary," predicting a displacement of zero; the second branch is "slowly moving straight," predicting a displacement of 5 pixels along the target's current orientation; the third branch is "turning left," predicting a horizontal displacement of 4 pixels to the left; and the fourth branch is "turning right," predicting a horizontal displacement of 4 pixels to the right. Each expert branch extrapolates the position information (i.e. bounding box coordinates) of the (t-1)th frame based on its own motion model. The extrapolation operation is as follows: the target bounding box coordinates of the previous video frame are translated according to the hypothetical motion model (such as the 5-pixel displacement defined by the "slow straight" model) to generate new bounding box coordinates. These new coordinates define the hypothetical future target area.

[0028] Subsequently, this method performs region-of-interest alignment on the feature map regions corresponding to the four hypothetical future target regions from a shared feature map extracted from the image at frame t by a backbone network (e.g., a backbone network shared with the object detection model). The aligned features are then input into a small head network (e.g., two fully connected layers) to extract four independent 256-dimensional candidate visual features. The gating network receives the original visual target query from the current frame t and calculates the cosine similarity between its appearance feature vector and the four candidate visual features. This similarity is considered the matching degree. If the cow actually turned its head to the left between frames t-1 and t, the candidate visual features extracted by the third branch have the highest matching degree with the visual target query. The gating network normalizes this set of matching degree values ​​using the Softmax function to ensure that the sum of the weights of all expert branch outputs is 1, generating a set of instantaneous motion gating weights, such as [0.1, 0.2, 0.6, 0.1]. This set of weights is used to sum the four candidate visual features in a weighted manner, and then fuse them to generate a single, motion-aware contextual feature.

[0029] Finally, the motion-aware contextual features interact with the feature vector of the original visual target query at the feature level. Specifically, the two are concatenated and then input into a fully connected layer for dimensionality reduction to update and generate the final, more accurate appearance query embedding for the current frame.

[0030] In a preferred embodiment, the method further includes: periodically analyzing the historical instantaneous motion gating weight sequence contained in the global identity trajectory to identify high-frequency specific motion patterns; enabling a preset adaptive expert branch and updating the hypothetical motion model parameters to the identified specific motion patterns.

[0031] Specifically, after continuously tracking cow "N073" for over an hour, the method analyzes the accumulated historical instantaneous motion gating weight sequence in its global identity trajectory. The analysis reveals that the weight values ​​of the third expert branch (representing "turning head left") and the fourth expert branch (representing "turning head right") frequently alternate at high values ​​within consecutive frames; this is identified as "N073's" unique "frequent head shaking" motion pattern. Based on this, the method activates the fifth (adaptive) expert branch reserved in the original combined expert decoder and defines its hypothetical motion model as a small sinusoidal back-and-forth displacement of the target in the horizontal direction. When "N073" performs the head shaking action again, this adaptively generated new expert branch can provide the most accurate position extrapolation, thus obtaining the highest gating weight.

[0032] This approach enables the motion model to evolve from a fixed set of prior knowledge into a dynamic model library capable of self-learning and adapting to individual behaviors, significantly improving the tracking robustness for targets with specific motion habits.

[0033] As a preferred implementation of the adaptive expert branch in this embodiment, the identification of the specific motion pattern can also be achieved by analyzing the actual historical trajectory coordinates of the target.

[0034] Specifically, in the global identity trajectory data structure, in addition to storing the appearance sequence and motion weight sequence, the historical bounding box center point coordinate sequence of the target should also be stored (e.g., a time series table of (x, y) coordinates).

[0035] When periodically analyzing the historical trajectory coordinate sequence, for example, by using short-time Fourier transform or motion pattern clustering algorithms, high-frequency, small-amplitude oscillations in the coordinates along a specific dimension (e.g., the x-axis) can be identified. This analysis based on real coordinates is identified as a specific motion pattern of "frequent head shaking" and used to update the motion model of the adaptive expert branch. This analysis method based on actual coordinates avoids the causal inversion problem that may result from analyzing only model weights, leading to more accurate identification.

[0036] This multi-branch parallel structure can concurrently evaluate multiple potential motion patterns. By using a gating network to dynamically weight different motion hypotheses based on visual evidence, the generated feature representation can adapt to sudden changes in the target's motion state. This mechanism enhances the tracking stability of targets performing nonlinear complex motions (such as sudden stops or turns) and improves the robustness of feature representation.

[0037] Furthermore, the process of generating visual behavior representations includes: embedding the updated appearance query of the current video frame into a fixed-length appearance feature queue, and storing the corresponding instantaneous motion gating weights into a fixed-length motion feature queue, thereby generating a temporal appearance state sequence and a temporal motion state sequence respectively; using a temporal multimodal fusion network, performing internal temporal dependency modeling on the temporal appearance state sequence and the temporal motion state sequence respectively, and performing cross-modal temporal information interaction on the temporal appearance state sequence and the temporal motion state sequence; and aggregating the output of the fusion network into a fixed-dimensional vector, which is the visual behavior representation.

[0038] Specifically, for a continuously tracked cow, two fixed-length queues in a first-in, first-out (FIFO) pattern are maintained, with a queue length of 32, corresponding to 32 consecutive video frames covering approximately 1.28 seconds. When processing each frame, the updated 256-dimensional appearance query is embedded and pushed to the end of the appearance feature queue; simultaneously, the generated 4-dimensional instantaneous motion gating weights are pushed to the end of the motion feature queue. The contents of these two queues constitute the temporal appearance state sequence and the temporal motion state sequence. These two sequences are input together into a temporal multimodal fusion network employing a two-stream attention structure. First, the temporal appearance state sequence and the temporal motion state sequence pass through independent gated recurrent unit (GRU) layers to model the internal temporal dependencies of their respective sequences.

[0039] Subsequently, the output sequences of the two GRU layers are fed into a cross-attention module to perform cross-modal temporal information interaction between the two sequences, establishing a correspondence between appearance changes and motion patterns. For example, the appearance sequence of a head-down position is associated with the motion weight sequence representing "stillness". The output of this fusion network is aggregated into a single 512-dimensional vector after passing through a global average pooling layer. This vector is the visual behavior representation, which condenses the comprehensive dynamic information of the cow's appearance and motion over the past 32 frames.

[0040] Specifically, the complete technical process of storing the appearance query embedding and instantaneous motion gating weights of the current frame into fixed-length queues, and applying a temporal multimodal fusion network for modeling, interaction and aggregation, and finally generating a fixed-dimensional vector, is a specific technical implementation of the temporal binding step.

[0041] This invention extends instantaneous visual and motion features into temporal sequences and performs deep modeling through a multimodal fusion network. The resulting visual behavior representation is no longer a static single-frame snapshot, but a compact description of the target's short-term dynamic behavior. The introduction of this temporal information significantly enhances the discriminative power of the features, providing high-quality input for subsequent more accurate identity association and behavior analysis.

[0042] Furthermore, referring to Figure 2 The process of combining the cross-modal associated state space includes: the multi-source data stream further includes electronic ear tag data; at each corresponding time step, weight data and physiological data are extracted from the feeding station data and electronic ear tag data respectively, and after normalization, the weight data and physiological data are mapped and combined into a unified high-dimensional feature vector; the unified high-dimensional feature vectors generated at multiple time steps constitute a temporal feature sequence; the temporal feature sequence is input into a temporal coding network, which models the temporal dependencies within the sequence through an attention mechanism to generate the data sequence; the visual behavior representation is combined with the data sequence to generate the cross-modal associated state space.

[0043] Specifically, in addition to feeding station data, the multi-source data stream also includes electronic ear tag data bound to the cow's global identity ID. This method processes non-visual data in a uniform time step of 60 seconds. Within each time step, the latest feeding station weight data (e.g., 715.6 kg) and physiological data uploaded by the electronic ear tag, such as body temperature (e.g., 38.5 degrees Celsius) and activity level calculated based on accelerometer (e.g., a value in the range of 0 to 10, currently 3), are extracted. These raw data are normalized by a preset minimum-maximum value and mapped to the interval between 0 and 1 respectively.

[0044] The normalized values ​​are concatenated into a vector, and then mapped to a unified 128-dimensional high-dimensional feature vector through a fully connected layer. A total of 32 high-dimensional feature vectors generated in the past 32 time steps (i.e., 32 minutes) are cached to form a temporal feature sequence. This temporal feature sequence is input into a temporal coding network, which consists of a multi-head self-attention layer and subsequent feedforward network layers. This network models the temporal dependencies within the sequence, such as identifying a pattern where a weight record appears after a continuous decrease in activity level. The network finally outputs a 256-dimensional vector as the data sequence.

[0045] Finally, the aforementioned combination step is performed. The 256-dimensional data sequence and the 512-dimensional visual behavior representation (and its underlying 32-frame temporal sequence) generated in the previous steps together constitute the cross-modal associated state space. The specific combination method is as follows: the data sequence vector and the visual behavior representation vector are concatenated to form a 768-dimensional fused state vector.

[0046] This invention integrates discrete multi-source sensor data points into a time-dependent data sequence. By fusing this non-visual data sequence with visual behavioral representations, a comprehensive state space containing internal physiological states and external behavioral patterns is constructed, providing richer and more discriminative judgment criteria for subsequent association decisions and reducing the ambiguity that may arise from relying solely on visual information.

[0047] Further, the process of generating the temporal consistency reward value includes: determining a dwell time interval defined by a set of consecutive video frames, wherein in each frame of the consecutive video frames constituting the visual behavior representation, the original target spatial coordinates of the visual target query in this video frame all fall within a preset feeding station geofence; extracting the original timestamp information contained therein from the original temporal feature sequence corresponding to the input end of the data sequence at the temporal coding network, and aggregating consecutive timestamps with a time interval less than a preset threshold into a weight data active time interval; calculating the total overlap duration between the dwell time interval and the weight data active time interval, and normalizing the duration as the temporal consistency reward value.

[0048] Specifically, in the camera's monitoring footage, a polygonal region is pre-defined as its geofence within the image coordinate system based on the physical boundaries of feeding station No. 3. When processing a visual behavior representation generated from visual information from frame 1000 to frame 1031 (a total of 32 frames), the coordinates of the center point of the bounding box of the corresponding cow in each of these 32 frames are checked frame by frame. If the center point coordinates of each frame fall within this geofence, a dwell time interval is determined. For example, if the timestamp of frame 1000 is "14:30:15.000" and the timestamp of frame 1031 is "14:30:16.240", then the dwell time interval is [14:30:15.000, 14:30:16.240].

[0049] Meanwhile, the original timestamps of all weight data are extracted from the original time-series feature sequence used to generate the data sequence. Assuming the extracted relevant timestamps are [..., 14:30:15.500, 14:30:15.900, 14:35:00.000, ...], a time interval threshold is set to 2 seconds. This threshold is set based on the maximum time interval of consecutive data packets that may be generated for the same weighing event in the data reporting protocol of the automated feeding station hardware. Since the time difference (0.4 seconds) between "14:30:15.500" and "14:30:15.900" is less than 2 seconds, these two time points are aggregated to form a weight data active time interval [14:30:15.500, 14:30:15.900].

[0050] Subsequently, the total overlap between the active time interval and the aforementioned dwell time interval was calculated, yielding a result of 0.4 seconds. Finally, this overlap duration was normalized by dividing it by the total dwell time interval (1.24 seconds), resulting in a temporal consistency reward value of approximately 0.32.

[0051] As a preferred implementation, the process of generating the temporal consistency reward value further includes: defining multiple concentrically expanding neighboring buffer zones around the preset feeding station geofence; assigning a spatial decay weight coefficient to each neighboring buffer zone, wherein the weight coefficient decreases as the distance between the buffer zone and the geofence increases; and multiplying the calculated total overlap duration by the spatial decay weight coefficient corresponding to the area where the original target spatial coordinates of the visual target query fall, to generate a weighted temporal consistency reward value.

[0052] Specifically, in addition to the feeding station's geofence itself (weight coefficient of 1.0), a "neighborhood zone" with a width of 1 meter (weight coefficient of 0.5) and a "potential zone" with a width of 2 meters (weight coefficient of 0.2) are defined outside it. When a cow wanders in the "neighborhood zone" and stays for 10 seconds, and this time period overlaps with an active time of weight data by 2 seconds, the calculated reward value is not 0, but (overlap time of 2 seconds / stay time of 10 seconds) * 0.5 = 0.1. If it completely enters the fence, the reward value is (overlap time of 2 seconds / stay time of 10 seconds) * 1.0 = 0.2.

[0053] This mechanism transforms the original binary, sparse reward signal into a continuous, dense reward function, enabling the model to learn not only entry behavior but also more predictive approach or loitering behavior.

[0054] As another preferred implementation of this embodiment, in order to solve the problem of sparse reward signals that may be caused by the above method (which requires each frame to fall within the fence), an event-triggered reward calculation mechanism can be adopted: Determining visual dwell events: Examine the continuous video frames (e.g., 32 frames) that constitute the visual behavior representation. If the coordinates of the visual target query fall into the preset feeding station geofence in any one or more frames of the sequence, it is considered that a visual dwell event has been triggered.

[0055] Definition of the dwell time interval: The dwell time interval is defined as the time stamp from the first entry into the geofence in the sequence to the last exit from the geofence.

[0056] Determining active data events: Each weight data timestamp extracted from the original time-series feature sequence is regarded as a discrete active data time point (rather than being aggregated into an interval).

[0057] Reward value generation: The correlation between the data activity time point and the dwell time interval is calculated. For example, a high positive reward value (e.g., +1.0) is generated if and only if at least one data activity time point falls strictly within the dwell time interval. If a visual dwell event is triggered, but no data activity time point appears in the interval or its adjacent time windows, a negative reward value (e.g., -0.1) is generated.

[0058] This alternative precisely correlates rewards with key “entry” behaviors and discrete “data reporting” events, providing a more effective and denser training signal.

[0059] This process provides a quantitative, real-world temporal synchronization-based supervisory signal for training the association strategy model. By calculating the temporal overlap between visual dwell behavior and sensor data activity events, it establishes an objective evaluation standard. This standard can guide the model to learn the inherent causal relationships between different data modalities, thereby generating more accurate and reliable association decisions.

[0060] Furthermore, the associated decision action is generated by an association strategy model, which includes: a Transformer encoder containing a self-attention layer and a cross-attention layer, which performs multimodal fusion encoding on the visual behavior representation and data sequence in the cross-modal association state space; wherein, the self-attention layer is responsible for modeling the internal temporal dependencies of each sequence, and the cross-attention layer is responsible for cross-modal information interaction between the two sequences to jointly generate the fused state encoding vector; and an action decoding network, which receives the state encoding vector and parses out the associated decision action.

[0061] In this embodiment, the association decision and model training is an iterative process. Specifically, during the tracking process, the cross-modal association state space is used as the current state and input into the association policy model. The model outputs an association decision action based on the current state. After the system executes the action, it calculates the temporal consistency reward value as described below, and uses the reward value together with the state, action and other information to perform reinforcement learning training on the association policy model.

[0062] It should be noted that the association decision-making and model training of this invention is an iterative closed-loop process. In actual operation, the association policy model (Q-value network) receives the current state (cross-modal association state space) at each step and generates an association decision action (inference). After executing this action, the temporal consistency reward value is immediately calculated. This reward value, together with the current state, action, and new state, constitutes an empirical tuple, which is used for real-time or batch reinforcement learning training of the association policy model.

[0063] Specifically, the association strategy model in this embodiment consists of a Transformer encoder for multimodal fusion and an action decoding network for parsing decision-making. Visual behavioral representations (based on a 32-frame visual information sequence at the bottom layer) and data sequences (based on a 32-time-step sensor information sequence at the bottom layer) extracted from the cross-modal association state space are input into the Transformer encoder in the model for processing. Inside the encoder, the self-attention layer first acts independently on the two sequences. For example, it establishes a connection within the visual information sequence, identifying the keyframes of the cow's continuous action from "standing" to "lowering its head to eat"; at the same time, it also models within the sensor information sequence, capturing the temporal pattern of "activity data stabilization" followed immediately by "weight data recording".

[0064] Subsequently, the cross-attention layer establishes a correlation between the two sequences. It performs a high-weighted matching of the keyframe of "head down eating" identified in the visual sequence with the time step of "weight data recording" in the sensor sequence. Through this multi-level internal and cross-modal information interaction, the encoder finally outputs a 768-dimensional fused state encoding vector. This vector is input into an action decoding network consisting of three fully connected layers. The network finally parses and outputs a discrete association decision action, such as "execute association" or "do not execute association".

[0065] To achieve the aforementioned reinforcement learning training, this embodiment employs a Deep Q-Network (DQN) algorithm to train the association policy model, specifically: State and Action Definitions: The cross-modal association state space (e.g., the 768-dimensional fused state vector) is defined as a "state" during the training process; the association policy model (e.g., the Transformer encoder and action decoding network) is used as a Q-value network with network parameters θ. This network outputs the Q-values ​​corresponding to the two discrete actions of "performing association" and "not performing association" for the current state.

[0066] Experience replay: During the tracking process, the experience tuple generated at each step (including the current state, the action performed, the reward obtained, and the next state) is stored in a fixed-capacity experience replay pool. The current state is the fused state vector, the action performed is the action selected by the model based on the ε-greedy policy, the reward obtained is the temporal consistency reward value calculated based on the method, and the next state is the new state after the transition.

[0067] Training iteration: During training, a mini-batch of experiences is randomly sampled from the experience replay pool, using an independent target network (whose network parameters θ) - The target Q-value is estimated by periodically copying from the main network θ. This target Q-value is equal to the immediate reward of the current step, plus a preset discount factor γ multiplied by the maximum Q-value of the next state estimated by the target network.

[0068] Loss Calculation and Update: The difference between the "target Q-value" and the "predicted Q-value" output by the current Q-value network is measured by calculating the mean squared error loss function; then, the parameters θ of the main network are updated using the Adam optimizer to minimize this difference (loss). This method utilizes self-attention and cross-attention mechanisms to deeply fuse two temporal data streams, one visual and one non-visual. This process not only models the dynamic changes within each data stream, but more importantly, it captures the fine correspondence between the two data streams in the time dimension. The resulting decision foundation is more robust, improving the accuracy and reliability of associated decisions in multi-objective, high-density scenarios.

[0069] Furthermore, referring to Figure 3 The process of generating a global identity trajectory includes: receiving corresponding visual target queries in parallel from multiple different surveillance video streams covering the farm; the global identity trajectory is constructed into a unified data structure, which binds the persistent global identity with multiple appearance query embedding sequences corresponding to the visual target in different surveillance video streams, as well as the associated data sequences, for cross-camera tracking.

[0070] Specifically, two cameras, numbered C01 and C02, are deployed in the rest area and feeding area, respectively, within the farm. When a cow with a global identity ID "N073" moves from the rest area (C01's field of view) to the feeding area (C02's field of view), the method first continuously receives visual target queries about the target from the video stream of C01. After confirming through association decision that the visual target corresponds to the data sequence of ID "N073", a global identity trajectory is created. This trajectory is a unified data structure whose contents are organized as follows: a "global identity" field, assigned the value "N073"; an "appearance sequence set" field, whose initial content is {"C01": [a list containing the appearance query embeddings of the past 32 frames]}; a "motion sequence set" field, whose initial content is {'C01': [a list containing the instantaneous motion gating weights corresponding to 32 frames]}; and a "association data" field, which stores the data sequence bound to "N073". When the cow leaves the field of view of C01 and enters the field of view of C02, the method receives a new visual target query from the C02 video stream. After confirming that the new target is still "N073" in the subsequent rebinding process, the aforementioned unified data structure is updated.

[0071] Specifically, the "Appearance Sequence Set" field was updated to {"C01": [...], "C02": [a list containing the latest 32 frames of appearance query embeddings extracted from the C02 video stream]}. Through continuous maintenance and updates of this data structure, the identity of the cow "N073" was maintained and tracked across different cameras.

[0072] To support adaptive motion model updates and multi-dimensional rebinding in this embodiment, as a preferred implementation, the global identity trajectory unified data structure further includes the following during updates: A "Motion Sequence Set" field is used to store a sequence of historical instantaneous motion gating weights corresponding to the "Appearance Sequence Set" (e.g., {"C01": [a list containing instantaneous motion gating weights for the past 32 frames]}), which is used to calculate the second similarity score when rebinding.

[0073] A "trajectory coordinate set" field is used to store the historical bounding box center point coordinate sequence of the target under each camera. This sequence is used for the identification and adaptive updating of specific motion patterns.

[0074] When the global identity trajectory is stored in the hibernation pool, the data structure should contain its historical appearance sequence and historical motion weight sequence within the C01 field of view.

[0075] This method constructs a persistent, unified identity profile across cameras. This data structure binds the visual performance of the same target in different times and spaces with continuous non-visual data streams, overcoming the problems of tracking interruption and identity fragmentation caused by limited field of view. It provides a structured data foundation for achieving long-term, uninterrupted individual tracking covering the entire activity space.

[0076] Furthermore, the process of generating a new visual behavior representation includes: when a new target enters, caching the appearance query embedding sequence and instantaneous motion gating weight sequence of the new target within a preset initial time window; at each time step of the initial time window, constructing an appearance state vector from the corresponding appearance query embedding and a motion state vector from the instantaneous motion gating weights, thus forming an initial appearance state sequence and an initial motion state sequence, respectively; applying a temporal multimodal fusion network to the initial appearance state sequence and the initial motion state sequence, wherein the temporal multimodal fusion network performs internal temporal dependency modeling on the initial appearance state sequence and the initial motion state sequence, and performs cross-modal temporal information interaction on the initial appearance state sequence and the initial motion state sequence, and aggregating the output of the fusion network into a fixed-dimensional vector, wherein the fixed-dimensional vector is the new visual behavior representation.

[0077] Specifically, when a previously untracked cow enters the field of view of camera C02, the method activates the feature cache for new targets, with a preset initial time window set to 32 frames. Within the first 32 frames of the cow's appearance, a 256-dimensional appearance query embedding and a 4-dimensional instantaneous motion gating weight are generated for each frame. These two values ​​are recorded in each frame, forming an appearance query embedding sequence of length 32 and an instantaneous motion gating weight sequence of length 32, respectively. These two sequences constitute the initial appearance state sequence and the initial motion state sequence.

[0078] Subsequently, the two sequences were applied together to a temporal multimodal fusion network containing a gated recurrent unit (GRU) layer and a cross-attention mechanism, which is the same as the aforementioned process for generating visual behavioral representations. Specifically, the network uses parallel GRU layers to extract temporal features from the two sequences to achieve internal temporal dependency modeling. Then, the cross-attention layer captures the correspondence between the two sequences at time steps to achieve cross-modal temporal information interaction.

[0079] The network processes the initial appearance state sequence to model the appearance posture changes of the new target cow within 1.28 seconds; at the same time, the network also processes the initial motion state sequence to analyze its initial motion pattern. Through cross-modal information interaction within the network, the appearance changes and motion patterns are correlated.

[0080] Finally, the output of the fusion network is aggregated into a single 512-dimensional vector by a global average pooling layer, which is the new visual behavior representation of the new target.

[0081] This process generates a feature representation of the newly emerging target that contains its short-term dynamic behavior, rather than a static single-frame appearance snapshot. By capturing the evolution of its appearance and motion at the initial contact stage, the generated representation has higher discriminative power. This provides a much more reliable and robust basis for judgment than single-frame features for the subsequent rebinding step of matching with historical trajectories.

[0082] Further, the rebinding process includes: calculating the highest similarity between the appearance query embedding sequence within the initial time window and the appearance query embedding sequence contained in each global identity trajectory in the dormant pool to obtain a first similarity score; aggregating the instantaneous motion gating weight sequence within the initial time window into an initial motion pattern vector through a temporal coding network, aggregating the historical motion weight sequence contained in each global identity trajectory in the dormant pool into a historical motion pattern vector through the same temporal coding network, calculating the cosine similarity between the initial motion pattern vector and the historical motion pattern vector to obtain a second similarity score; weighting and combining the first similarity score, the second similarity score, and the cross-camera re-entry probability provided by the camera topology map, performing candidate sorting on the global identity trajectories in the dormant pool, and selecting the Top-K candidate data sequences; selecting the candidate identity with the highest comprehensive score from the Top-K candidates according to a preset decision threshold; if selected, activating the global identity trajectory of the candidate identity from the dormant pool and rebinding it with the new visual behavior representation of the current new target.

[0083] Specifically, when a new target appears in the field of view of camera C02, its initial appearance query embedding sequence is extracted. This sequence is compared with each global identity trajectory in the dormancy pool (e.g., including the appearance sequence of cow "N073" in the field of view of C01 last time). A first similarity score, assumed to be 0.85, is obtained by calculating the cosine similarity between each appearance query embedding in the new sequence and all appearance query embeddings in the historical sequence, and taking the maximum value of all calculated results. Simultaneously, the initial instantaneous motion gating weight sequence of the new target is input into a motion temporal coding network (which consists of a gated recurrent unit (GRU) layer and a subsequent global average pooling layer) to generate a 128-dimensional initial motion pattern vector. The historical motion weight sequence of cow "N073" is input into the same motion temporal coding network to generate a 128-dimensional historical motion pattern vector as well. The cosine similarity between these two motion pattern vectors is calculated to obtain a second similarity score, assumed to be 0.78. Based on this, the method also introduces a cross-camera re-entry probability based on the camera topology map.

[0084] A camera topology map is a mathematical model of the physical layout of surveillance cameras on a farm and the possible movement paths of targets. In the scenario of this embodiment, the topology map explicitly defines that camera C01 is deployed in the rest area of ​​the cattle shed, while camera CO2 covers the main passageway leading to the feeding station. Based on the analysis of historical activity data of the cattle herd, the model determines that moving from the rest area (C01's field of view) to the feeding passageway (CO2's field of view) is a high-frequency daily behavior pattern of the cattle. Therefore, the topology map assigns a high cross-camera re-entry probability of 0.9 to this path from C01 to CO2, while the probability of moving to other non-primary areas is assigned a lower value. This probability value (0.9), together with the aforementioned two similarity scores, is used to calculate a comprehensive score of 0.839 using a pre-defined weighted summation formula. Specifically, the weighting coefficients in the weighting formula (0.5, 0.3, and 0.2 respectively) are hyperparameters that can be empirically adjusted based on the characteristics of the actual deployment scenario (such as the impact of lighting changes on appearance, the uniqueness of the target's motion pattern, etc.). This process is repeated for all trajectories in the dormant pool, and the top-5 candidates are selected by sorting them from high to low based on their comprehensive scores. A preset decision threshold is 0.8. Since "N073" has the highest score and is greater than this threshold, it is selected as the matching identity.

[0085] Subsequently, the global identity trajectory of "N073" is activated from the dormant pool and re-bound to the visual behavioral representation of the new target.

[0086] In a preferred embodiment, after performing candidate sorting on the global identity trajectories in the dormant pool, the method further includes: extracting the associated data sequences of at least two candidate identities with the highest comprehensive scores to determine their respective recent behavioral states; predicting a most likely future intention based on the new visual behavioral representation of the new target; calculating a behavioral logic consistency score between the future intention and the recent behavioral states of each candidate identity; and making a final decision selection on the candidate identities based on the behavioral logic consistency score to perform the rebinding.

[0087] Specifically, when a new target appears, the two highest-ranking candidate identities are "N073" (overall score 0.84) and "N081" (overall score 0.83). The method retrieves their associated data sequences and finds that the recent behavioral state of "N073" is "fed 5 minutes ago", while the state of "N081" is "no feeding record in 3 hours". At the same time, based on the visual behavioral representation of the new target (its orientation and movement speed), its future intention is predicted to be "enter the feeding station". The intention to "enter the feeding station" has low logical consistency with the state of "fed 5 minutes ago" (score 0.2), but high logical consistency with the state of "no feeding record" (score 0.9). Therefore, the method ultimately selects "N081" as the matching identity, even though its initial overall score is slightly lower.

[0088] As another preferred implementation of the above-mentioned behavioral logic arbitration scheme, to avoid the unreliability of predicting future intentions based solely on short-time series data, and to mitigate the risk that the final decision might lead to the rejection of high-confidence visual / motor matching, the arbitration logic is set as an auxiliary weighting term: Estimate movement trends: Based on new visual behavioral representations (especially their movement information), estimate their current movement trends (e.g., by analyzing their velocity vector and orientation, determine whether they are "moving toward the feeding station" or "moving away from the feeding station").

[0089] Calculate the logical consistency score: Calculate the logical consistency score between the "movement trend" and the recent behavioral status of the candidate identity (such as "N073" or "N081") (such as "fed 5 minutes ago" or "no feeding record in 3 hours").

[0090] Combined weighting: Instead of using this score for the "final decision selection", it is used as a fourth similarity dimension (e.g., given a low weight coefficient, such as 0.1 or 0.2) and combined with the first similarity score (appearance), the second similarity score (motion) and the cross-camera re-entry probability (topology) in a weighted combination.

[0091] Final decision: Sort the candidates based on the final composite score, which includes four scores, and select the candidate with the highest score.

[0092] This alternative uses behavioral logic as supplementary information rather than rejection information, ensuring the dominance of visual and motion features in re-identification and improving the robustness of the system.

[0093] This scheme adds an arbitration layer of behavioral logic on the basis of multi-dimensional similarity matching, and uses the contextual information provided by non-visual data to solve the problem of highly ambiguous re-identification, which significantly reduces the error rate of identity swapping.

[0094] This rebinding process integrates information from three dimensions: appearance, motion pattern, and spatiotemporal priors for decision-making. This multi-dimensional matching mechanism greatly improves the accuracy of target re-identification compared to methods that rely solely on appearance features. It can effectively distinguish individuals with similar appearances and uses motion habits and scene logic to assist in judgment, significantly reducing the identity error switching rate when tracking across cameras.

[0095] This invention provides a motion-aware multi-target tracking method based on query propagation. By deeply fusing multi-source data streams containing precise identities, such as those from feeding stations, with visual information from farm monitoring videos, this method solves the problems of identity errors and tracking loss that easily occur in traditional pure visual tracking methods in scenarios with similar target appearances, dense occlusion, and complex motion. This method employs a combined expert decoder to adapt to the complex instantaneous motion of the target and uses temporal consistency rewards to perform reinforcement learning training on the association strategy model, enabling it to autonomously learn the intrinsic causal relationship between visual behavior and sensor data. By constructing a global identity trajectory and combining it with a rebinding mechanism based on multi-dimensional similarity matching, stable tracking across cameras and over long time periods is achieved. This method significantly improves the accuracy, robustness, and continuity of multi-target tracking, providing reliable technical support for individualized and refined management in smart agriculture.

[0096] Example 2: This application provides a complete application process of a query-based motion-sensing multi-target tracking method in a modern dairy farm. The scenario is set in a large modern dairy farm, which is functionally divided into area A (rest area) and area B (feeding area). Camera CA completely covers area A, and camera CB completely covers area B, with no overlapping areas in the monitoring fields of the two cameras. In area B, an automated feeding station FS-B is deployed. This feeding station integrates an RFID reader and a weighing module, which can automatically identify the electronic ear tags of the cattle entering the farm and accurately record their weight and feeding timestamp. The camera topology of the farm is pre-modeled, wherein the cross-camera re-entry probability of moving from area A to area B is set to a high value.

[0097] At 9:00 AM, a cow with a global identity ID "BN-316" was first detected by camera CA in area A. The method immediately began processing visual target queries about the target received from the CA video stream. Within the initial 1.28-second time window (corresponding to 32 video frames) of the target's appearance, the combined expert decoder analyzed its motion frame by frame, generating an initial appearance state sequence consisting of 32 appearance query embeddings and an initial motion state sequence consisting of 32 instantaneous motion gating weights. These two temporal sequences were input into a temporal multimodal fusion network, and after internal temporal modeling and cross-modal information interaction, they were aggregated into a visual behavioral representation that could characterize the dynamic process of the cow "from lying down to walking". This representation was bound to the global identity "BN-316" to form an active global identity trajectory.

[0098] At 9:05 AM, cow "BN-316" slowly walked out of area A and disappeared from the field of view of camera CA. The target loss detection mechanism of the method was triggered, and the global identity trajectory of "BN-316" which was active in the field of view of CA was then transferred to the dormancy pool. The trajectory data structure stored in the dormancy pool completely preserves the cow's historical appearance query embedding sequence and historical movement weight sequence before it disappeared, which serves as the key basis for subsequent identity re-identification.

[0099] At 9:06 AM, a cow entered area B and was captured by camera CB. The method identified it as a new target and immediately activated the feature cache, generating a new visual behavioral representation for it within an initial time window of 32 frames. Subsequently, a rebinding process was executed, which calculated the initial appearance query embedding sequence of the new target with the historical appearance sequence of the trajectory of "BN-316" in the dormant pool, obtaining a first similarity score of 0.88. Simultaneously, the initial motion weight sequence of the new target and the historical motion weight sequence of "BN-316" were aggregated into motion pattern vectors through a temporal coding network, and the cosine similarity between the two was calculated, obtaining a second similarity score of 0.81. Combined with the high re-entry probability provided by the camera topology map, the comprehensive score calculated by weighted combination significantly exceeded the preset decision threshold of 0.8. Therefore, the method determined that the new target was "BN-316", activated its global identity trajectory from the dormant pool, and rebinded it with the new visual behavioral representation currently in the CB's field of view, successfully achieving cross-camera identity tracking.

[0100] To further confirm this binding relationship, at 9:07 AM, when the re-identified cow "BN-316" entered the automated feeding station FS-B, the method performed cross-modal association. On one hand, it detected that the target's visual bounding box continuously fell within the preset feeding station geofence, thus defining a visual dwell time interval. On the other hand, a feeding station record with a timestamp of 9:07:15 was received from the multi-source data stream, containing the identity ID "BN-316" and weight 721.3 kg. The timestamp of this record constituted a weight data active time interval. After calculation, the visual dwell interval and the data active interval highly overlapped on the time axis. After combining the visual behavior representation of "head down to feed" with the data sequence containing the weight record into a cross-modal association state space, it was input into the trained association strategy model. The model output a high-confidence execution association decision action. This decision action was ultimately based on non-visual data to confirm the result of cross-camera rebinding.

[0101] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the scope of protection defined in the claims.

Claims

1. A motion-aware multi-target tracking method based on query propagation, characterized in that, include: Receive detected visual target queries and multi-source data streams containing feeding station data from farm surveillance video; The visual target query is input into a combined expert decoder. The computational results of the expert branches are fused through a gating network to generate appearance query embeddings and instantaneous motion gating weights, which are then temporally bound to generate a visual behavior representation. The combined expert decoder employs a multi-branch parallel structure: the visual target query is structured data containing the target's position information in the current video frame and feature vectors that characterize its appearance; the multi-branch parallel structure receives target position information from the previous video frame; each expert branch extrapolates the target position information from the previous video frame to generate a hypothetical future target region and extracts candidate visual features from this hypothetical future target region. The feature is used as the calculation result; the motion prior on which the extrapolation operation is based is defined by assigning different hypothetical motion models to each expert branch. The hypothetical motion models are used to predict future displacement based on the motion prior and define the hypothetical future target region; the gating network parses the instantaneous motion gating weights based on the matching degree between the visual target query and the candidate visual features, and uses the instantaneous motion gating weights to perform weighted fusion of the candidate visual features output by each expert branch to generate motion-aware contextual features; the visual target query interacts with the motion-aware contextual features to update the appearance query embedding of the current frame; Temporal feature encoding is performed on multi-source data streams to generate data sequences containing feeding station data, and visual behavior representations and data sequences are combined into a cross-modal associated state space; The temporal correlation between the dwell time of visual behavioral representations and the activity time of data sequences is analyzed and calculated to generate a temporal consistency reward value. The association policy model is then trained using reinforcement learning. The cross-modal association state space is input into the trained association policy model to generate association decision actions. These association decision actions are generated by the association policy model, which includes: a Transformer encoder containing a self-attention layer and a cross-attention layer to perform multimodal fusion encoding of the visual behavioral representations and data sequences in the cross-modal association state space; wherein the self-attention layer is responsible for modeling the internal temporal dependencies of each sequence, and the cross-attention layer is responsible for cross-modal information interaction between the two sequences to jointly generate a fused state encoding vector; and an action decoding network receives the state encoding vector and parses out the association decision actions. Based on associated decision actions, a global identity trajectory is generated; when a visual target moves out of the monitoring field of view, the global identity trajectory is stored in the dormancy pool; when a new target enters, a new visual behavior representation is generated, and the new visual behavior representation is matched with the global identity trajectory in the dormancy pool in multiple dimensions, and rebinding is performed.

2. The motion-aware multi-target tracking method based on query propagation according to claim 1, characterized in that, The process of generating visual behavior representations includes: embedding the updated appearance query of the current video frame into a fixed-length appearance feature queue, and storing the corresponding instantaneous motion gating weights into a fixed-length motion feature queue, thereby generating a temporal appearance state sequence and a temporal motion state sequence; using a temporal multimodal fusion network, performing internal temporal dependency modeling on the temporal appearance state sequence and the temporal motion state sequence, and performing cross-modal temporal information interaction between the temporal appearance state sequence and the temporal motion state sequence; and aggregating the output of the fusion network into a fixed-dimensional vector, which is the visual behavior representation.

3. The motion-aware multi-target tracking method based on query propagation according to claim 1, characterized in that, The process of combining the cross-modal associated state space includes: the multi-source data stream further includes electronic ear tag data; at each corresponding time step, weight data and physiological data are extracted from the feeding station data and electronic ear tag data respectively, and after normalization, the weight data and physiological data are mapped and combined into a unified high-dimensional feature vector; the unified high-dimensional feature vectors generated at multiple time steps constitute a temporal feature sequence; the temporal feature sequence is input into a temporal coding network, which models the temporal dependencies within the sequence through an attention mechanism to generate the data sequence; the visual behavior representation is combined with the data sequence to generate the cross-modal associated state space.

4. The motion-aware multi-target tracking method based on query propagation according to claim 1, characterized in that, The process of generating the temporal consistency reward value includes: determining a dwell time interval defined by a set of consecutive video frames, wherein in each frame of the consecutive video frames constituting the visual behavior representation, the original target spatial coordinates of the visual target query in this video frame all fall within a preset feeding station geofence; extracting the original timestamp information contained therein from the original temporal feature sequence corresponding to the input end of the data sequence at the temporal coding network, and aggregating consecutive timestamps with a time interval less than a preset threshold into a weight data active time interval; calculating the total overlap duration between the dwell time interval and the weight data active time interval, and normalizing the duration as the temporal consistency reward value.

5. The motion-aware multi-target tracking method based on query propagation according to claim 1, characterized in that, The process of generating a global identity trajectory includes: receiving corresponding visual target queries in parallel from multiple different surveillance video streams covering the farm; the global identity trajectory is constructed into a unified data structure, which binds the persistent global identity with multiple appearance query embedding sequences corresponding to the visual target in different surveillance video streams, as well as the associated data sequences, for cross-camera tracking.

6. The motion-aware multi-target tracking method based on query propagation according to claim 1, characterized in that, The process of generating a new visual behavior representation includes: when a new target enters, caching the appearance query embedding sequence and instantaneous motion gating weight sequence of the new target within a preset initial time window; at each time step of the initial time window, constructing an appearance state vector from the corresponding appearance query embedding and a motion state vector from the instantaneous motion gating weights, thus forming an initial appearance state sequence and an initial motion state sequence, respectively; applying a temporal multimodal fusion network to the initial appearance state sequence and the initial motion state sequence, wherein the temporal multimodal fusion network performs internal temporal dependency modeling on the initial appearance state sequence and the initial motion state sequence, and performs cross-modal temporal information interaction on the initial appearance state sequence and the initial motion state sequence, and aggregating the output of the fusion network into a fixed-dimensional vector, which is the new visual behavior representation.

7. The motion-aware multi-target tracking method based on query propagation according to claim 6, characterized in that, The rebinding process includes: calculating the highest similarity between the appearance query embedding sequence within the initial time window and the appearance query embedding sequence contained in each global identity trajectory in the dormant pool to obtain a first similarity score; aggregating the instantaneous motion gating weight sequence within the initial time window into an initial motion pattern vector through a temporal coding network, aggregating the historical motion weight sequence contained in each global identity trajectory in the dormant pool into a historical motion pattern vector through the same temporal coding network, calculating the cosine similarity between the initial motion pattern vector and the historical motion pattern vector to obtain a second similarity score; weighting the first similarity score, the second similarity score, and the cross-camera re-entry probability provided by the camera topology map, performing candidate sorting on the global identity trajectories in the dormant pool, and selecting the Top-K candidate data sequences; selecting the candidate identity with the highest comprehensive score from the Top-K candidates according to a preset decision threshold; if selected, activating the global identity trajectory of the candidate identity from the dormant pool and rebinding it with the new visual behavior representation of the current new target.

Images