Motion sensing multi-target tracking method based on query transfer

Abstract

The invention relates to the technical field of multi-target tracking, in particular to a motion perception multi-target tracking method based on query transfer, which comprises the following steps: receiving visual target query and multi-source data stream; a combined expert decoder is adopted to process the visual query, and visual behavior characterization capable of adapting to complex motion of the target is generated; performing time sequence coding on the multi-source data stream to generate a data sequence; a time sequence consistency reward value is generated by calculating the overlapping degree of the visual residence time sequence and the data active time sequence, and reinforcement learning training is carried out on the association strategy model according to the time sequence consistency reward value; inputting the fused state space into the model to generate an association decision, and constructing a global identity trajectory; and when the target is reproduced, rebinding is executed through a multi-dimensional matching mechanism fusing appearance, motion and space prior. Through fusion of a visual trajectory and multi-source Internet of Things data and introduction of a reinforcement learning optimization long-term association strategy, accurate and long-term cross-modal target tracking in a complex farm interaction scene is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-target tracking, in particular to a motion-aware multi-target tracking method based on query transmission. BACKGROUND

[0002] Multi-target tracking technology plays a key role in smart agriculture, security monitoring and other fields, especially in dynamic scenes where individuals have similar appearances and frequent interactions, requiring more accurate and long-term tracking and behavior recognition of targets.

[0003] In specific applications such as smart agriculture, in addition to video sequences, there are usually multi-source Internet of Things data reflecting the state of individuals, such as data streams from electronic ear tags or feeding stations. Few existing technologies effectively fuse these two heterogeneous data sources, resulting in an incomplete understanding of target state.

[0004] In addition, some tracking methods extract instantaneous motion information of targets by analyzing video frames, however, this information is usually only used for association decision-making in the current frame, lacking a mechanism to effectively encode and transmit instantaneous state to subsequent frames for long-term analysis, limiting the ability to analyze the continuous behavior patterns of targets.

[0005] At the same time, when a target encounters long-term occlusion or enters and exits the field of view, identity re-identification is a core problem. Existing methods consider identity association as a series of independent frame-by-frame matching tasks, and the greedy matching strategy they use lacks consideration of long-term temporal consistency. This decision-making mechanism is prone to irreversible identity switching due to instantaneous misjudgment after a target is occluded for a long time. How to model this process as a sequential decision problem that can learn a long-term optimal association strategy and introduce reinforcement learning and other frameworks to solve it is a challenge faced by current technology.

[0006] Therefore, the present application provides a motion-aware multi-target tracking method based on query transmission. SUMMARY

[0007] The purpose of the present application is to provide a motion-aware multi-target tracking method based on query transmission, which fuses visual trajectories and multi-source Internet of Things data and introduces reinforcement learning to optimize long-term association strategies, achieving accurate and long-term cross-modal target tracking in complex farm interaction scenarios.

[0008] To achieve the above purpose, the present application provides the following technical solutions: A motion-aware multi-target tracking method based on query transmission, comprising: receiving detected visual target queries and multi-source data streams containing feeding station data from farm monitoring videos; The visual target query is input into a combined expert decoder, the calculation results of the expert branches are fused through a gating network to generate an appearance query embedding and a transient motion gating weight, and time binding is performed to generate a visual behavior representation; The multi-source data stream is encoded for time sequence characteristics to generate a data sequence containing feeding station data, and the visual behavior representation and the data sequence are combined into a cross-modal associated state space; The time sequence correlation of the resident time sequence of the visual behavior representation and the active time sequence of the data sequence is analyzed and calculated to generate a time sequence consistency reward value, and the associated strategy model is trained through reinforcement learning; the cross-modal associated state space is input into the trained associated strategy model to generate an associated decision action; Based on the associated decision action, a global identity trajectory is generated; when the visual target moves out of the monitoring field of view, the global identity trajectory is stored in a dormant pool; when a new target enters, a new visual behavior representation is generated, and multi-dimensional similarity matching is performed between the new visual behavior representation and the global identity trajectory in the dormant pool to perform re-binding.

[0009] Preferably, the combined expert decoder adopts a multi-branch parallel structure: the visual target query is structured data containing target position information in the current video frame and a feature vector capable of representing appearance; the multi-branch parallel structure receives the target position information of the previous video frame; each expert branch extrapolates the target position information of the previous video frame to generate a hypothetical future target area and extracts candidate visual features from the hypothetical future target area as calculation results; the motion prior for the extrapolation operation is defined by assigning different hypothetical motion models to each expert branch, which are used to predict future displacement based on the motion prior and define the hypothetical future target area; the gating network analyzes the transient motion gating weight based on the matching degree of the visual target query and the candidate visual features, and uses the transient motion gating weight to weight and fuse the candidate visual features output by each expert branch to generate motion-aware context features; the visual target query and the motion-aware context features interact to update the appearance query embedding of the current frame.

[0010] Preferably, the process of generating a visual behavior representation includes: storing the updated appearance query embedding of the current video frame in a fixed-length appearance feature queue and storing the corresponding transient motion gating weight in a fixed-length motion feature queue to generate a time sequence appearance state sequence and a time sequence motion state sequence, respectively; through a time sequence multi-modal fusion network, the time sequence appearance state sequence and the time sequence motion state sequence are modeled for internal time sequence dependency and cross-modal time sequence information interaction, and the output of the fusion network is aggregated into a fixed-dimension vector, which is the visual behavior representation.

[0011] Preferably, the process of combining the cross-modal associated state space comprises: the multi-source data stream further comprises electronic ear tag data; at each corresponding time step, weight data and physiological data are extracted from the feeding station data and electronic ear tag data respectively, and after normalization processing, the weight data and physiological data are mapped and combined into a unified high-dimensional feature vector; the unified high-dimensional feature vectors generated at multiple time steps are combined to form a time sequence feature sequence; the time sequence feature sequence is input into a time sequence encoding network, the time sequence encoding network models the time sequence dependency within the sequence through an attention mechanism, and generates the data sequence; the visual behavior representation and the data sequence are combined to generate the cross-modal associated state space.

[0012] Preferably, the process of generating the time sequence consistency reward value comprises: determining a residence time interval defined by a group of consecutive video frames, wherein in each frame of the consecutive video frames corresponding to the visual behavior representation, the visual target query falls within the preset feeding station geographic fence in the original target space coordinates of the video frame; extracting the original timestamp information contained in the original time sequence feature sequence corresponding to the input end of the time sequence encoding network from the data sequence, and aggregating the consecutive timestamps with a time interval less than a preset threshold as a weight data active time interval; calculating the total overlap length of the residence time interval and the weight data active time interval, and performing normalization processing on the length as the time sequence consistency reward value.

[0013] Preferably, the associated decision action is generated by an associated policy model, and the associated policy model comprises: a Transformer encoder comprising a self-attention layer and a cross-attention layer, which performs multi-modal fusion encoding on the visual behavior representation and the data sequence in the cross-modal associated state space; wherein the self-attention layer is responsible for modeling the internal time sequence dependency of each sequence, and the cross-attention layer is responsible for cross-modal information interaction between the two sequences, and together generates a fused state encoding vector; an action decoding network that receives the state encoding vector and parses out the associated decision action.

[0014] Preferably, the process of generating the global identity trajectory comprises: receiving the corresponding visual target query from multiple different monitoring video streams covering the farm in parallel; the global identity trajectory is constructed as a unified data structure, which binds the persistent global identity, the multiple appearance query embedding sequences of the visual target in different monitoring video streams, and the associated data sequence, and performs cross-camera tracking.

[0015] Preferably, the process of generating the new visual behavior representation comprises: when a new target enters, caching the appearance query embedding sequence and the transient motion gating weight sequence of the new target within a preset initial time window; at each time step of the initial time window, forming an appearance state vector with the corresponding appearance query embedding and forming a motion state vector with the transient motion gating weight, to form an initial appearance state sequence and an initial motion state sequence respectively; applying a time series multi-modal fusion network to the initial appearance state sequence and the initial motion state sequence, the time series multi-modal fusion network respectively models the internal time series dependency of the initial appearance state sequence and the initial motion state sequence, and interacts the initial appearance state sequence and the initial motion state sequence with cross-modal time series information, and aggregates the output of the fusion network into a fixed-dimensional vector, which is the new visual behavior representation.

[0016] Preferably, the re-binding process comprises: calculating the highest similarity between the appearance query embedding sequence within the initial time window and the appearance query embedding sequence contained in each global identity track in the dormant pool, to obtain a first similarity score; aggregating the transient motion gating weight sequence within the initial time window into an initial motion pattern vector through a time series encoding network, and aggregating the historical motion weight sequence contained in each global identity track in the dormant pool into a historical motion pattern vector through the same time series encoding network, and calculating the cosine similarity between the initial motion pattern vector and the historical motion pattern vector, to obtain a second similarity score; weighting and combining the first similarity score, the second similarity score and the cross-camera re-entry probability provided by the camera topology graph, performing candidate sorting on the global identity tracks in the dormant pool, and screening out Top-K candidate data sequences; selecting the candidate identity with the highest comprehensive score from the Top-K candidates according to a preset decision threshold; if selected, activating the global identity track of the candidate identity from the dormant pool and re-binding it with the new visual behavior representation of the current new target.

[0017] Compared with the prior art, the present application has the following beneficial effects: 1. By combining the expert decoder to analyze the transient motion and the time series multi-modal fusion network for time series binding, the dynamic behavior of the target is deeply represented. The present application uses the combined expert decoder to extract the transient motion gating weight of the target, and stores it together with the appearance query embedding into the time series queue, and then performs dependency modeling and information interaction through the fusion network. This design effectively encodes and transmits the transient motion state of the target to the subsequent frame, and aggregates it into a visual behavior representation with appearance and motion time series information, solving the problem of lack of long-term motion analysis mechanism in the prior art, and providing a more rich and robust feature basis for subsequent precise association and behavior recognition.

[0018] 2. By constructing a cross-modal association state space and applying a reinforcement learning association method based on timing consistency rewards, long-term optimal identity association under heterogeneous data fusion is achieved. The present application effectively fuses visual behavior representation with Internet of Things data sequences from feeding stations and electronic ear tags, solving the limitations of existing technologies that cannot comprehensively understand target states. More importantly, it models identity association as a sequential decision problem, using timing consistency rewards to guide the reinforcement learning model to jump out of local optimization and learn an association strategy with long-term timing consistency, thereby overcoming the fundamental defect of irreversible identity switching caused by instantaneous misjudgment after long-term target occlusion in traditional greedy matching.

[0019] 3. By establishing a global identity trajectory data structure and designing a dormant rebinding coordination mechanism based on multi-dimensional similarity matching, seamless identity tracking across cameras and long-term tracking is achieved. The present application constructs a persistent global identity for each target that binds multiple camera visual features and Internet of Things data. When the target re-enters the field of view, this mechanism does not rely solely on appearance features, but rather makes decisions through a multi-dimensional matching strategy that integrates appearance sequence similarity, motion pattern vector similarity, and camera topology probability, greatly improving the accuracy of identity re-identification in long-term disappearance or cross-camera scene, ensuring the continuity and global uniqueness of tracking in large-scale monitoring scenarios such as smart farms. BRIEF DESCRIPTION OF DRAWINGS

[0020] Fig. 1 A flowchart of a motion-aware multi-target tracking method based on query propagation according to the present application; Fig. 2 A flowchart of a method for generating cross-modal association decision actions according to an embodiment of the present application; Fig. 3 A flowchart of a method for performing re-binding on newly appearing targets according to an embodiment of the present application. DETAILED DESCRIPTION

[0021] To make the objectives, technical solutions, and advantages of the present application clearer, specific embodiments will be described below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, and not to limit the present application. Other embodiments obtained by those skilled in the art based on the ideas in the present specification, without creative labor, fall within the scope of protection of the present application.

[0022] Reference Figs. 1 to 3 The present application provides a motion-aware multi-target tracking method based on query propagation, and the technical solution is as follows.

[0023] A motion-aware multi-target tracking method based on query propagation, comprising: receiving a detected visual target query and a multi-source data stream containing feeding station data from a farm monitoring video; inputting the visual target query into a combined expert decoder, fusing the calculation results of the expert branches through a gating network to generate an appearance query embedding and an instantaneous motion gating weight, and performing temporal binding to generate a visual behavior representation; temporally encoding the multi-source data stream to generate a data sequence containing feeding station data, and combining the visual behavior representation and the data sequence into a cross-modal associated state space; analyzing and calculating the temporal correlation between the resident temporal sequence of the visual behavior representation and the active temporal sequence of the data sequence to generate a temporal consistency reward value, and performing reinforcement learning training on the associated policy model; inputting the cross-modal associated state space into the trained associated policy model to generate an associated decision action; based on the associated decision action, generating a global identity trajectory; when the visual target moves out of the monitoring field of view, storing the global identity trajectory in a dormant pool; when a new target enters, generating a new visual behavior representation, and performing multi-dimensional similarity matching between the new visual behavior representation and the global identity trajectory in the dormant pool to perform re-binding.

[0024] Embodiment 1: The embodiment provides a motion-aware multi-target tracking method based on query transmission, referring to Fig. 1 The method is deployed in a modern dairy farm to continuously track the identity of each cow in the cowshed and automatically record the behavior of each cow.

[0025] Specifically, in a surveillance network covering the main activity area of a cowshed (including resting area and feeding area), multiple fixed-position cameras are deployed. These cameras continuously capture video streams at a frame rate of 25 frames per second and a resolution of 1920x1080 pixels. The video streams are fed into a pre-trained object detection model. The model processes the video frame by frame, and when a cow is recognized, a visual target query is generated for the target. The visual target query is a structured data packet containing the bounding box position of the cow in the current frame coordinate system (for example, a rectangular region defined by the coordinates of the top-left corner and the bottom-right corner) and a 256-dimensional appearance feature vector extracted by the intermediate layer network of the object detection model, which can represent the local appearance texture of the cow. At the same time, the method receives an independent multi-source data stream from the central management system of the farm through a data interface. This data stream contains data records from all automated feeding stations. For example, when cow N073 enters feeding station 3, the RFID reader in the station recognizes its electronic ear tag, the weighing module records its weight, and the feed distribution system records its feed intake. A corresponding feeding station data record is generated, containing the event timestamp, feeding station ID "FS03", cow global ID "N073", and weight data "715.6 kg". Both data streams (visual target query stream and feeding station data stream) have accurate timestamps, which are continuously received by the method as the basis for subsequent temporal correlation.

[0026] Further, the combined expert decoder adopts a multi-branch parallel structure: the visual target query is a structured data containing the position information of the target in the current video frame and the feature vector representing the appearance; the multi-branch parallel structure receives the target position information of the previous video frame; each expert branch extrapolates the target position information of the previous video frame to generate a hypothetical future target area and extracts candidate visual features from the hypothetical future target area as the calculation result; the motion prior for the extrapolation operation is defined by assigning different hypothetical motion models to each expert branch, which are used to predict the future displacement based on the motion prior and define the hypothetical future target area; the gating network analyzes the instantaneous motion gating weight based on the matching degree between the visual target query and the candidate visual features, and uses the instantaneous motion gating weight to weight and fuse the candidate visual features output by each expert branch to generate motion-aware context features; the visual target query and the motion-aware context features interact to update the appearance query embedding for the current frame.

[0027] Specifically, the visual target query is a structured data packet, which specifically includes the bounding box position of the cow in the current frame image coordinate system (for example, a rectangular region defined by the coordinates of the upper left corner and the lower right corner), and a 256-dimensional appearance feature vector extracted by the intermediate layer network of the target detection model, which can represent the local appearance texture of the cow. When processing the t-th frame of the video stream, the combined expert decoder receives the target position information of a certain tracked cow in the previous frame (t-1-th frame), for example, its bounding box coordinates are (210, 350, 290, 480). The decoder is internally provided with four parallel expert branches, each of which is pre-assigned an independent hypothetical motion model. For example, the motion model of the first branch is "stationary", and the predicted displacement is zero; the second branch is "slow straight", and the predicted displacement is 5 pixel points along the current direction of the target; the third branch is "turning left", and the predicted horizontal displacement to the left is 4 pixel points; the fourth branch is "turning right", and the predicted horizontal displacement to the right is 4 pixel points. Each expert branch extrapolates the position information (i.e. the bounding box coordinates) of the t-1-th frame according to its own motion model. The extrapolation operation is specifically: the target bounding box coordinates of the previous video frame are translated according to the hypothetical motion model (such as the 5-pixel displacement defined by the "slow straight" model) to generate new bounding box coordinates, which define the hypothetical future target region.

[0028] Subsequently, the method performs a region of interest alignment operation on the feature map regions corresponding to the above four hypothetical future target regions from a shared feature map extracted by a backbone network (for example, a backbone network shared with the target detection model) on the t-th frame image, and inputs the aligned features into a small head network (for example, two fully connected layers) to extract four independent 256-dimensional candidate visual features, respectively. The gating network receives the original visual target query of the current t-th frame and calculates the cosine similarity between the appearance feature vector and the above four candidate visual features, respectively. The similarity is regarded as the matching degree. If the cow actually performs a left turn action between the t-1-th frame and the t-th frame, the candidate visual feature extracted by the third branch has the highest matching degree with the visual target query. The gating network applies the Softmax function to the set of matching degree values for normalization processing, ensuring that the sum of the weights output by all expert branches is 1, generating a set of instantaneous motion gating weights, for example, [0.1, 0.2, 0.6, 0.1]. This set of weights is used to weight and sum the four candidate visual features to generate a single, motion-aware context feature.

[0029] Finally, the motion-aware context feature interacts with the feature vector of the original visual target query at the feature level, specifically: after splicing the two, input a fully connected layer for dimension reduction, to update and generate the final, more accurate appearance query embedding of the current frame.

[0030] As a preferred implementation, the method further comprises: periodically analyzing the historical instantaneous motion gating weight sequence contained in the global identity trajectory, identifying a specific motion pattern that appears frequently; enabling a preset adaptive expert branch, and updating the hypothetical motion model parameters to the identified specific motion pattern.

[0031] Specifically, after more than an hour of continuous tracking of cow "N073", the method analyzes the historical instantaneous motion gating weight sequence accumulated in its global identity trajectory, and finds that the weight values of the third expert branch representing "turning head left" and the fourth expert branch representing "turning head right" frequently appear high values in consecutive frames; this is identified as the specific motion pattern of "frequent head shaking" unique to "N073". Based on this, the method activates the fifth (i.e. adaptive) expert branch reserved in the original combined expert decoder, and defines its hypothetical motion model as a small sinusoidal reciprocating displacement of the target in the horizontal direction. When "N073" performs the head shaking action again, this newly generated adaptive expert branch can give the most accurate position extrapolation, thereby obtaining the highest gating weight.

[0032] This scheme enables the motion model to evolve from a set of fixed prior knowledge to a dynamic model library that can self-learn and adapt to individual behavior, significantly improving the tracking robustness of targets with specific motion habits.

[0033] As a preferred implementation of the adaptive expert branch in this embodiment, the identification of the specific motion pattern can also be achieved by analyzing the actual historical trajectory coordinates of the target.

[0034] Specifically, in the global identity trajectory data structure, in addition to storing the appearance sequence and motion weight sequence, the historical bounding box center point coordinate sequence (for example, a time sequence list of (x, y) coordinates) of the target should also be stored.

[0035] When periodically analyzing the historical trajectory coordinate sequence, for example, by short-time Fourier transform or motion pattern clustering algorithm, it is identified that the coordinates have high-frequency, small-amplitude reciprocating oscillations in a certain dimension (for example, the x-axis). This analysis result based on real coordinates is identified as the specific motion pattern of "frequent head shaking", and is used to update the motion model of the adaptive expert branch. This analysis method based on actual coordinates avoids the causality inversion problem that may be caused by analyzing only the model weights, and is more accurate.

[0036] The multi-branch parallel structure can concurrently evaluate multiple potential motion patterns, dynamically weight different motion hypotheses according to visual evidence through a gating network, and generate a feature representation that can adapt to sudden changes in the motion state of a target; this mechanism enhances the tracking stability of a target performing nonlinear complex motion (such as sudden stopping or turning around) and improves the robustness of feature expression.

[0037] Further, the process of generating the visual behavior representation comprises: storing the updated appearance query embedding of the current video frame in a fixed-length appearance feature queue and storing the corresponding instantaneous motion gating weight in a fixed-length motion feature queue, respectively generating a time-series appearance state sequence and a time-series motion state sequence; modeling the internal time-series dependency of the time-series appearance state sequence and the time-series motion state sequence through a time-series multi-modal fusion network, and performing cross-modal time-series information interaction on the time-series appearance state sequence and the time-series motion state sequence, and aggregating the output of the fusion network into a fixed-dimension vector, which is the visual behavior representation.

[0038] Specifically, for a certain cow that is continuously tracked, two fixed-length queues in a first-in-first-out (FIFO) mode are maintained, and the queue length is set to 32, corresponding to 32 consecutive video frames covering a duration of about 1.28 seconds. When processing each frame, the 256-dimensional appearance query embedding of the updated frame is pushed to the end of the appearance feature queue; at the same time, the 4-dimensional instantaneous motion gating weight generated by the frame is pushed to the end of the motion feature queue. The contents of these two queues constitute a time-series appearance state sequence and a time-series motion state sequence. These two sequences are input into a time-series multi-modal fusion network together. The network uses a double-flow attention structure. First, the time-series appearance state sequence and the time-series motion state sequence are respectively processed through independent gated recurrent unit (GRU) layers to model the internal time-series dependency of each sequence.

[0039] Subsequently, the output sequences of the two GRU layers are sent to a cross-attention module to perform cross-modal time-series information interaction between the two sequences and establish a correspondence between appearance changes and motion patterns, for example, associating the appearance sequence of a lowered head with the motion weight sequence representing "stillness". The output of the fusion network is aggregated into a single 512-dimensional vector through a global average pooling layer, and this vector is the visual behavior representation, which condenses the comprehensive dynamic information of the appearance and motion of the cow in the past 32 frames.

[0040] Specifically, the complete technical process of storing the appearance query embedding and the instantaneous motion gating weight of the current frame in fixed-length queues, modeling, interacting, and aggregating through a time-series multi-modal fusion network, and finally generating a fixed-dimension vector, is a specific technical implementation of the time-series binding step.

[0041] The application extends the instantaneous visual and motion features into a time sequence and performs deep modeling through a multi-modal fusion network, so that the visual behavior representation generated is no longer a static single-frame snapshot, but a compact description containing the target short-time dynamic behavior. The introduction of this time sequence significantly enhances the distinguishability of the features, providing high-quality input for subsequent more accurate identity association and behavior analysis.

[0042] Further, with reference to Fig. 2 , the process of combining the cross-modal association state space includes: the multi-source data stream further contains electronic ear tag data; at each corresponding time step, weight data and physiological data are extracted from the feeding station data and electronic ear tag data respectively, and after normalization processing of the weight data and physiological data, they are mapped and combined into a unified high-dimensional feature vector; the unified high-dimensional feature vectors generated at multiple time steps form a time sequence feature sequence; the time sequence feature sequence is input into a time sequence encoding network, which models the time sequence dependency within the sequence through an attention mechanism to generate the data sequence; the visual behavior representation and the data sequence are combined to generate the cross-modal association state space.

[0043] Specifically, in addition to the feeding station data, the multi-source data stream also contains electronic ear tag data bound to the global identity ID of the cow. This method processes non-visual data with 60 seconds as a unified time step. Within each time step, the latest feeding station weight data (e.g. 715.6 kg) and physiological data uploaded by the electronic ear tag, such as body temperature (e.g. 38.5 degrees Celsius) and activity amount calculated based on the accelerometer (e.g. a value ranging from 0 to 10, currently 3), are extracted. These raw data are normalized by a pre-set minimum-maximum value and mapped to the interval of 0 to 1 respectively.

[0044] The normalized multiple values are spliced into a vector, which is then mapped to a unified 128-dimensional high-dimensional feature vector through a fully connected layer. A time sequence feature sequence is formed by buffering the 32 high-dimensional feature vectors generated in the past 32 time steps (i.e. 32 minutes). The time sequence feature sequence is input into a time sequence encoding network, which is composed of a multi-head self-attention layer and a subsequent feedforward network layer. The network models the time sequence dependency within the sequence, such as identifying a pattern of a continuous decrease in activity amount followed by a weight record. The network finally outputs a 256-dimensional vector as the data sequence.

[0045] Finally, the combination step is performed, and the 256-dimensional data sequence and the 512-dimensional visual behavior representation generated in the foregoing steps jointly constitute the cross-modal association state space, and the combination is performed in the following manner: the data sequence vector and the visual behavior representation vector are spliced to form a 768-dimensional fusion state vector.

[0046] The present application integrates discrete multi-source sensor data points into data sequences with time sequence dependence, and constructs a comprehensive state space including internal physiological state and external behavior mode by fusing the non-visual data sequence and the visual behavior representation, thereby providing richer and more distinguishable judgment basis for subsequent association decision, and reducing the ambiguity that may be caused by visual information alone.

[0047] Further, the process of generating the time sequence consistency reward value comprises the following steps: determining a residence time interval defined by a group of continuous video frames, wherein in each frame of the continuous video frames corresponding to the visual behavior representation, the visual target query falls within the preset feeding station geographic fence in the original target space coordinates of the video frame; extracting the original timestamp information contained in the original time sequence feature sequence corresponding to the input end of the time sequence coding network from the data sequence, and aggregating the continuous timestamps with a time interval less than a preset threshold as the weight data active time interval; calculating the total overlap length of the residence time interval and the weight data active time interval, and performing normalization processing on the length as the time sequence consistency reward value.

[0048] Specifically, in the monitoring screen of the camera, a polygon region is set as the geographic fence in the image coordinate system according to the physical boundary of the No. 3 feeding station in advance, and when a visual behavior representation generated from visual information of 1000th frame to 1031th frame (a total of 32 frames) is processed, the center point coordinates of the bounding box corresponding to the cow in the 32 frames are checked frame by frame, and if the center point coordinates of each frame fall within the geographic fence, a residence time interval is determined, for example, if the timestamp of the 1000th frame is “14:30:15.000” and the timestamp of the 1031th frame is “14:30:16.240”, the residence time interval is [14:30:15.000, 14:30:16.240].

[0049] Meanwhile, from the original time series feature sequence used to generate the data sequence, the original timestamp of all weight data is extracted, assuming that the extracted relevant timestamp is [..., 14:30:15.500, 14:30:15.900, 14:35:00.000,...], a time interval threshold is set to 2 seconds, and the setting of the threshold is based on the maximum time interval of the continuous data packet that may be generated for the same weighing event in the data reporting protocol of the automated feeding station hardware. Since the time difference (0.4 seconds) between "14:30:15.500" and "14:30:15.900" is less than 2 seconds, the two time points are aggregated to form a weight data active time interval [14:30:15.500, 14:30:15.900].

[0050] Subsequently, the total overlap duration of the active time interval and the above-mentioned residence time interval is calculated, resulting in 0.4 seconds. Finally, the overlap duration is normalized by dividing the total duration of the residence time interval (1.24 seconds) to obtain a time series consistency reward value of about 0.32.

[0051] As a preferred embodiment, the process of generating the time series consistency reward value further comprises: defining a plurality of concentrically expanded adjacent buffer areas around the preset feeding station geographic fence; assigning a spatial decay weight coefficient to each adjacent buffer area, wherein the weight coefficient decreases as the distance between the buffer area and the geographic fence increases; and multiplying the calculated total overlap duration by the spatial decay weight coefficient corresponding to the area in which the original target spatial coordinates of the visual target query fall, to generate a weighted time series consistency reward value.

[0052] Specifically, in addition to the feeding station geographic fence itself (weight coefficient 1.0), a "neighborhood" area (weight coefficient 0.5) with a width of 1 meter and a "potential" area (weight coefficient 0.2) with a width of 2 meters are defined outside it. When a cow wanders in the "neighborhood" area and generates a residence time of 10 seconds, and the time period overlaps with an active time of weight data for 2 seconds, the calculated reward value is not 0, but (overlap duration 2 seconds / residence duration 10 seconds)*0.5=0.1, and if it enters the fence completely, the reward value is (overlap duration 2 seconds / residence duration 10 seconds)*1.0=0.2.

[0053] This mechanism converts the originally binary and sparse reward signal into a continuous and dense reward function, enabling the model to not only learn the entry behavior, but also learn more predictive approach or wandering behavior.

[0054] As another preferred implementation of the present embodiment, in order to solve the reward signal sparsity problem caused by the above method (requiring each frame to fall within the fence), an event-triggered reward calculation mechanism can be used: Determination of visual residence event: check the consecutive video frames (e.g. 32 frames) that constitute the visual behavior representation, as long as the coordinates of the visual target query fall within the preset feeding station geographic fence in any one or more frames in the sequence, it is considered that the visual residence event is triggered.

[0055] Definition of residence time interval: the residence time interval is defined as the timestamp of the first time entering the geographic fence to the timestamp of the last time leaving the geographic fence in the sequence.

[0056] Determination of data active event: each individual weight data timestamp extracted from the original time series feature sequence is considered as a discrete data active time point (rather than aggregated as an interval).

[0057] Generation of reward value: calculate the relevance of the data active time point and the residence time interval. For example, a high positive reward value (e.g. +1.0) is generated only when at least one data active time point strictly falls within the residence time interval. If the visual residence event is triggered, but there is no data active time point in the interval and its adjacent time window, a negative reward value (e.g. -0.1) is generated.

[0058] This alternative solution accurately associates rewards with key "entry" behaviors and discrete "data reporting" events, providing more effective and denser training signals.

[0059] This process provides a quantitative, real-world time series synchronization-based supervision signal for the training of the association policy model. By calculating the degree of overlap in time between visual residence behavior and sensor data active events, it establishes an objective evaluation standard that can guide the model to learn the inherent causal relationship between different data modalities, thereby generating more accurate and reliable association decisions.

[0060] Further, the association decision action is generated by an association policy model, which includes: a Transformer encoder containing a self-attention layer and a cross-attention layer, which performs multi-modal fusion encoding on the visual behavior representation and the data sequence in the cross-modal association state space; wherein the self-attention layer is responsible for modeling the internal time series dependency of each sequence, and the cross-attention layer is responsible for cross-modal information interaction between the two sequences, together generating a fused state encoding vector; an action decoding network that receives the state encoding vector and parses out the association decision action.

[0061] In this embodiment, the association decision and model training is an iterative process, specifically: in the tracking process, the cross-modal association state space is input to the association strategy model as the current state; the model outputs an association decision action based on the current state; after the system executes the action, the timing consistency reward value is calculated as described below, and the reward value is used together with the state, action and other information to train the association strategy model by reinforcement learning.

[0062] It should be noted that the association decision and model training of the present application is an iterative closed-loop process. In actual operation, the association strategy model (Q value network) receives the current state (cross-modal association state space) at each step and generates an association decision action (inference). After executing the action, the timing consistency reward value is immediately calculated. The reward value, together with the current state, action, and new state, forms an experience tuple, which is used to train the association strategy model by reinforcement learning in real time or in batches.

[0063] Specifically, the association strategy model in this embodiment is composed of a Transformer encoder for multi-modal fusion and an action decoding network for decision analysis. The visual behavior representation (its underlying layer is a 32-frame visual information sequence) and the data sequence (its underlying layer is a 32-time step sensor information sequence) extracted from the cross-modal association state space are input into the Transformer encoder in the model for processing. Within the encoder, the self-attention layer first acts independently on the two sequences, for example, it establishes a connection within the visual information sequence and identifies the key frames of the continuous action of the cow from "standing" to "bending to feed"; at the same time, it also models within the sensor information sequence and captures the timing pattern of "stable activity data" followed by "body weight data recording".

[0064] Subsequently, the cross-attention layer establishes an association between the two sequences, which highly matches the key frame of "bending to feed" in the visual sequence with the time step of "body weight data recording" in the sensor sequence. Through this multi-level internal and cross-modal information interaction, the encoder finally outputs a 768-dimensional, fused state encoding vector, which is input into an action decoding network composed of three fully connected layers. The network finally analyzes and outputs a discrete association decision action, such as "execute association" or "do not execute association".

[0065] To implement the reinforcement learning training, the deep Q network (DQN) algorithm is used to train the association strategy model, specifically: State and Action Definition: The cross-modal association state space (e.g., the 768-dimensional fusion state vector) is defined as the "state" in the training process; the association policy model (e.g., the Transformer encoder and action decoding network) is used as a Q-value network, whose network parameters are θ, which outputs the Q-value corresponding to the "perform association" and "do not perform association" discrete actions respectively for the current state.

[0066] Experience Replay: During the tracking process, the experience tuples generated at each step (containing the current state, the executed action, the obtained reward, and the next state) are stored in a fixed-capacity experience replay pool, where the current state is the fusion state vector, the executed action is the action selected by the model based on the ε-greedy strategy, the obtained reward is the temporal consistency reward value calculated based on the method, and the next state is the new state after transition.

[0067] Training Iteration: During training, a small batch of experiences is randomly sampled from the experience replay pool, and a separate target network (whose network parameters θ - are periodically copied from the main network θ) is used to estimate the target Q-value, which is equal to the immediate reward of the current step plus a preset discount factor γ multiplied by the maximum Q-value of the next state estimated by the target network.

[0068] Loss Calculation and Update: The difference between the "target Q-value" and the "predicted Q-value" output by the current Q-value network is measured by calculating the mean squared error loss function; then, the Adam optimizer is used to update the parameters θ of the main network to minimize this gap (loss) This method uses self-attention and cross-attention mechanisms to deeply fuse visual and non-visual two-time series data streams, which not only models the dynamic changes within each data stream, but more importantly captures the fine correspondence between the two data streams in the time dimension; the decision basis generated by this process is more solid, which improves the accuracy and reliability of the association decision in multi-target and high-density scenarios.

[0069] Further, with reference to Fig. 3 , the process of generating a global identity trajectory includes: receiving corresponding visual target queries in parallel from multiple different monitoring video streams covering a farm; the global identity trajectory is constructed as a unified data structure that binds the persistent global identity, the multiple appearance query embedding sequences of the visual target in different monitoring video streams, and the associated data sequences for cross-camera tracking.

[0070] Specifically, two cameras C01 and C02 are deployed in the farm respectively in the resting area and the feeding area. When a cow with global identity ID “N073” walks from the resting area (C01 view) to the feeding area (C02 view), the method first continuously receives visual target queries about the target from the video stream of C01. After confirming that the visual target corresponds to the data sequence of ID “N073” through association decision, a global identity track is created. The track is a unified data structure, the content of which is organized as: a “global identity” field, assigned with “N073”; a “appearance sequence set” field, whose initial content is {“C01”: [a list containing 32 past frame appearance query embeddings]}; a “motion sequence set” field, whose initial content is {‘C01’: [a list containing 32 frame transient motion gating weights corresponding]}; and an “association data” field, storing the data sequence bound to “N073”. When the cow leaves the C01 view and enters the C02 view, the method receives new visual target queries from the C02 video stream. After confirming that the new target is still “N073” in the subsequent rebinding process, the aforementioned unified data structure is updated.

[0071] Specifically, the “appearance sequence set” field is updated to {“C01”: [...], “C02”: [a list containing the latest 32 frame appearance query embeddings extracted from the C02 video stream]}. Through continuous maintenance and update of this data structure, the identity of the cow “N073” is maintained and tracked across different cameras.

[0072] To support adaptive motion model updating and multi-dimensional rebinding in this embodiment, as a preferred implementation, the global identity track unified data structure also includes, when updated: a “motion sequence set” field, for storing the historical transient motion gating weight sequence corresponding to the “appearance sequence set” (for example, {“C01”: [a list containing 32 past frame transient motion gating weights]}), which is used to calculate the second similarity score in rebinding.

[0073] a “track coordinate set” field, for storing the historical bounding box center point coordinate sequence of the target under each camera, which is used for specific motion pattern recognition and adaptive updating.

[0074] When the global identity track is stored in the dormant pool, the data structure should contain its historical appearance sequence and historical motion weight sequence in the C01 view.

[0075] The method constructs a persistent and cross-camera unified identity profile, and the data structure binds the visual representation of the same target at different times and spaces with continuous non-visual data streams, and overcomes the tracking interruption and identity fragmentation caused by the limited field of view; and it provides a structured data basis for realizing long-term and uninterrupted individual tracking covering the entire activity space.

[0076] Further, the process of generating the new visual behavior representation comprises: when a new target enters, caching the appearance query embedding sequence and the transient motion gating weight sequence of the new target within a preset initial time window; at each time step of the initial time window, forming an appearance state vector from the corresponding appearance query embedding and forming a motion state vector from the transient motion gating weight, to respectively form an initial appearance state sequence and an initial motion state sequence; applying a time series multi-modal fusion network to the initial appearance state sequence and the initial motion state sequence, the time series multi-modal fusion network respectively models the internal time series dependency of the initial appearance state sequence and the initial motion state sequence, and performs cross-modal time series information interaction on the initial appearance state sequence and the initial motion state sequence, and aggregates the output of the fusion network into a fixed-dimensional vector, which is the new visual behavior representation.

[0077] Specifically, when a previously untracked cow enters the field of view of camera C02, the method starts the feature caching of the new target, and a preset initial time window is set to 32 frames. Within the first 32 frames of the appearance of the cow, a 256-dimensional appearance query embedding and a 4-dimensional transient motion gating weight are generated for each frame, and these two values are recorded for each frame, thereby forming an appearance query embedding sequence with a length of 32 and a transient motion gating weight sequence with a length of 32, which constitute the initial appearance state sequence and the initial motion state sequence.

[0078] Subsequently, the two sequences are applied together to a time series multi-modal fusion network containing a gated recurrent unit (GRU) layer and a cross-attention mechanism, which is the same as the aforementioned process of generating a visual behavior representation. Specifically, the network uses parallel GRU layers to extract time series features from the two sequences to model the internal time series dependency, and then captures the correspondence between the two sequences at the time step through the cross-attention layer to realize cross-modal time series information interaction.

[0079] The network processes the initial appearance state sequence to model the appearance posture changes of the new target cow within 1.28 seconds; at the same time, the network also processes the initial motion state sequence to analyze its initial motion pattern, and through the cross-modal information interaction within the network, the appearance changes and the motion pattern are associated.

[0080] Finally, the output of the fusion network is aggregated by a global average pooling layer into a single 512-dimensional vector, which is the new visual behavior representation of the new target.

[0081] The process generates a feature representation for the newly appearing target that contains its short-term dynamic behavior, rather than a static single-frame appearance snapshot. By capturing the evolution of its appearance and motion at the initial contact stage, the generated representation has higher discriminability. This provides a much more reliable and robust basis for the subsequent re-binding step of matching with historical trajectories.

[0082] Further, the re-binding process includes: calculating the highest similarity between the appearance query embedding sequence in the initial time window and the appearance query embedding sequence contained in each global identity trajectory in the dormant pool, obtaining a first similarity score; aggregating the instantaneous motion gating weight sequence in the initial time window into an initial motion pattern vector through a temporal encoding network, and aggregating the historical motion weight sequence contained in each global identity trajectory in the dormant pool into a historical motion pattern vector through the same temporal encoding network, calculating the cosine similarity between the initial motion pattern vector and the historical motion pattern vector, obtaining a second similarity score; weighting and combining the first similarity score, the second similarity score, and the cross-camera re-entry probability provided by the camera topology map, performing candidate sorting on the global identity trajectories in the dormant pool, and screening out Top-K candidate data sequences; selecting the candidate identity with the highest comprehensive score from the Top-K candidates according to a preset decision threshold; if selected, activating the global identity trajectory of the candidate identity from the dormant pool and re-binding it with the new visual behavior representation of the current new target.

[0083] Specifically, when a new target appears in the field of view of camera C02, its initial appearance query embedding sequence is extracted. This sequence is compared with each global identity trajectory in the dormant pool (for example, containing the appearance sequence of cow "N073" last in the field of view of C01). By calculating the cosine similarity between each appearance query embedding in the new sequence and all appearance query embeddings in the historical sequence, and taking the maximum value of all calculation results, a first similarity score is obtained, which is assumed to be 0.85. At the same time, the initial instantaneous motion gating weight sequence of the new target is input into a motion temporal encoding network (which is composed of a gated recurrent unit GRU layer and a subsequent global average pooling layer), generating an initial motion pattern vector of 128 dimensions. The historical motion weight sequence of cow "N073" is input into the same motion temporal encoding network, also generating a historical motion pattern vector of 128 dimensions. The cosine similarity between the two motion pattern vectors is calculated to obtain a second similarity score, which is assumed to be 0.78. On this basis, the method also introduces the cross-camera re-entry probability based on the camera topology map.

[0084] The camera topology map is a mathematical modeling of the physical layout of the surveillance cameras and the possible movement paths of the target within the farm. In the scenario of the present embodiment, the topology map defines unambiguously that camera COI is deployed at the resting area of the cowshed, while camera CO2 covers the main passage leading to the feeding station. Based on the analysis of the historical activity data of the cow herd, the model determines that moving from the resting area (COI field of view) to the feeding passage (CO2 field of view) is a daily high-frequency behavior pattern of the cows. Therefore, the topology map assigns a cross-camera reentry probability of up to 0.9 to this path from COI to CO2, while the probabilities to other non-main areas are assigned lower values. The probability value (0.9) together with the two aforementioned similarity scores are used to calculate a comprehensive score by a pre-set weighted summation formula, resulting in 0.839. Specifically, the weight coefficients (0.5, 0.3, 0.2 respectively) in the weighted formula are hyperparameters that can be empirically adjusted according to the characteristics of the actual deployment scenario (such as the influence of light changes on appearance, the uniqueness of target motion patterns, etc.); the process is repeated for all trajectories in the hibernation pool, and the Top-5 candidates are selected by ranking the comprehensive scores from high to low, with a pre-set decision threshold of 0.8. Since the score of “N073” is the highest and greater than the threshold, it is selected as the matching identity.

[0085] Subsequently, the global identity trajectory of “N073” is activated from the hibernation pool and re-bound with the visual behavior representation of the current new target.

[0086] As a preferred implementation, after performing candidate ranking on the global identity trajectories in the hibernation pool, the method further comprises: for at least two candidate identities with the highest comprehensive scores, extracting their associated data sequences respectively to determine their respective recent behavior states; predicting a most likely future intention of the new target based on the new visual behavior representation of the new target; calculating a behavior logic consistency score between the future intention and the recent behavior state of each candidate identity; and according to the behavior logic consistency score, performing a final decision selection on the candidate identities to execute the re-binding.

[0087] Specifically, after a new target appears, the two highest-ranked candidate identities are "N073" (overall score 0.84) and "N081" (overall score 0.83). The method retrieves their associated data sequences and finds that "N073" has a recent behavior state of "completed feeding 5 minutes ago", while "N081" has a state of "no feeding record in the last 3 hours". At the same time, based on the visual behavior representation of the new target (its orientation and movement speed), its future intention is predicted to be "enter the feeding station". The logical consistency between the intention of "enter the feeding station" and the state of "completed feeding" is low (score 0.2), while the logical consistency between the intention of "enter the feeding station" and the state of "no feeding record" is high (score 0.9). Therefore, the method ultimately selects "N081" as the matching identity, even though its initial overall score is slightly lower.

[0088] As another preferred implementation of the above behavior logic arbitration scheme, to avoid the unreliability of predicting future intentions based solely on short sequences, and to avoid the risk that the final decision selection may lead to the rejection of high-confidence visual / movement matching, the arbitration logic is set as an auxiliary weighting item: Estimate movement trend: based on the new visual behavior representation (especially its movement information), estimate its current movement trend (for example, by analyzing its speed vector and orientation, determine whether it is "moving towards the feeding station" or "moving away from the feeding station").

[0089] Calculate logical consistency score: calculate the logical consistency score between the "movement trend" and the recent behavior state of the candidate identity (such as "N073" or "N081") (such as "fed 5 minutes ago" or "no feeding record in the last 3 hours").

[0090] Combine weighting: do not use this score for "final decision selection", but use it as a fourth similarity dimension (for example, assign a lower weight coefficient, such as 0.1 or 0.2), and combine it with the first similarity score (appearance), the second similarity score (movement), and the cross-camera reentry probability (topology) together.

[0091] Final decision: sort according to the final overall score containing four scores, and select the candidate identity with the highest score.

[0092] This alternative scheme uses behavior logic as auxiliary information, not as a rejection information, ensuring the dominance of visual and movement features in re-identification, and improving the robustness of the system.

[0093] This scheme adds an arbitration layer of behavior logic based on multi-dimensional similarity matching, uses the context information provided by non-visual data to solve the high-ambiguity re-identification problem, and significantly reduces the error rate of identity exchange.

[0094] The rebinding process integrates the appearance, motion pattern and spatiotemporal prior three-dimensional information to make decisions, and the multi-dimensional matching mechanism greatly improves the accuracy of target re-identification compared with the method of single dependence on appearance features; it can effectively distinguish individuals with similar appearance, and use the habit of movement and scene logic to assist in judgment, which significantly reduces the identity error switching rate during cross-camera tracking.

[0095] The application provides a motion-aware multi-target tracking method based on query transmission, which solves the problems of identity errors and tracking loss of traditional pure visual tracking methods in similar target appearance, dense occlusion and complex motion scenes by deeply fusing multi-source data streams containing accurate identities such as feeding stations with visual information in farm monitoring videos, the method adopts a combined expert decoder to adapt to the complex instantaneous motion of the target, and uses a timing consistency reward to enhance the learning training of an association strategy model, so that the model can learn the internal causal relationship between visual behavior and sensor data; by constructing a global identity trajectory and combining a rebinding mechanism of multi-dimensional similarity matching, cross-camera and long-term stable tracking is realized; the method significantly improves the accuracy, robustness and continuity of multi-target tracking, and provides reliable technical support for individual fine management in smart agriculture.

[0096] Embodiment 2: The application embodiment provides a complete application process of a motion-aware multi-target tracking method based on query transmission in a modern dairy farm, and the scene of the embodiment is set in a large modern dairy farm, the farm is functionally divided into an A area (rest area) and a B area (feeding area), a camera C-A completely covers the A area, a camera C-B completely covers the B area, and the monitoring fields of view of the two cameras do not overlap; in the B area, an automatic feeding station FS-B is deployed, the feeding station integrates an RFID reader and a weighing module, can automatically identify the electronic ear tags of the entering cattle, and accurately record the body weight and feeding time stamp; the camera topology relationship of the farm is pre-modeled, wherein the cross-mirror reentry probability from the A area to the B area is set to a high value.

[0097] At 9:00 am, a cow with global identity ID "BN-316" is first detected by camera C-A in area A. The method immediately starts processing the visual target query received from the C-A video stream about the target, and within the initial 1.28-second time window (corresponding to 32 video frames) of the target's appearance, the group expert decoder analyzes its motion frame by frame, generating an initial appearance state sequence consisting of 32 appearance query embeddings and an initial motion state sequence consisting of 32 instantaneous motion gating weights; these two time series are input into the time series multi-modal fusion network, and after internal time series modeling and cross-modal information interaction, they are aggregated into a visual behavior representation that can represent the "from lying to walking" dynamic process. The representation is bound with the global identity "BN-316" to form an active global identity track.

[0098] By 9:05 am, the cow "BN-316" walks out of area A, and its body disappears from the field of view of camera C-A. The target loss detection mechanism of the method is triggered, and the global identity track of "BN-316" in the active state in the C-A field of view is immediately transferred to the dormant pool. The track data structure stored in the dormant pool retains a complete history of appearance query embedding sequences and history motion weight sequences of the cow before it disappears, serving as a key basis for subsequent identity re-identification.

[0099] At 9:06 am, a cow enters area B and is captured by camera C-B. The method identifies it as a new target and immediately starts feature caching, generating a new visual behavior representation within the initial 32-frame time window. Then, the re-binding process is performed, which calculates the first similarity score of 0.88 between the initial appearance query embedding sequence of the new target and the history appearance sequence of the "BN-316" track in the dormant pool. At the same time, the initial motion weight sequence of the new target and the history motion weight sequence of "BN-316" are aggregated into motion pattern vectors through the time series encoding network, and the cosine similarity between them is calculated, obtaining a second similarity score of 0.81. Combined with the high re-entry probability provided by the camera topology graph, the comprehensive score calculated by weighted combination significantly exceeds the preset decision threshold of 0.8. Therefore, the method determines that the new target is "BN-316", activates its global identity track from the dormant pool, and rebinds it with the new visual behavior representation in the C-B field of view, successfully achieving cross-camera identity tracking.

[0100] In order to further confirm the binding relationship, at 9:07 am, when the re-identified cow "BN-316" enters the automatic feeding station FS-B, the method performs cross-modal association, on the one hand, it is detected that the visual bounding box of the target continuously falls within the preset feeding station geographic fence, thereby defining a visual residence time interval; on the other hand, a feeding station record with a timestamp of 9:07:15 is received from the multi-source data stream, containing the identity ID "BN-316" and the body weight 721.3 kg, the timestamp of the record constitutes a body weight data active time interval; after calculation, the visual residence interval and the data active interval are highly overlapped on the time axis, after combining the visual behavior representation representing "head-down feeding" with the data sequence containing the body weight record into a cross-modal association state space, inputting into the trained association strategy model, the model outputs a high-confidence execution association decision action, which finally confirms the cross-camera re-binding result based on non-visual data.

[0101] The above merely illustrates the specific embodiments of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of the changes or replacements within the technical range disclosed by the present application, which shall be covered within the protection scope of the present application, therefore, the protection scope of the present application shall be limited by the protection scope defined in the claims.

Claims

1. A query-by-demonstration motion-aware multi-target tracking method, characterized in that, The method comprises the following steps: receiving a detected visual target query from a farm monitoring video and a multi-source data stream containing feeding station data; inputting the visual target query into a combined expert decoder, fusing the calculation results of expert branches through a gating network to generate an appearance query embedding and an instantaneous motion gating weight, and performing temporal binding to generate a visual behavior representation; temporally encoding the multi-source data stream to generate a data sequence containing feeding station data, and combining the visual behavior representation and the data sequence into a cross-modal associated state space; analyzing and calculating the temporal correlation between the resident time sequence of the visual behavior representation and the active time sequence of the data sequence to generate a temporal consistency reward value, and performing reinforcement learning training on the associated policy model; inputting the cross-modal associated state space into the trained associated policy model to generate an associated decision action; generating a global identity trajectory based on the associated decision action; storing the global identity trajectory in a dormant pool when the visual target moves out of the monitoring field of view; when a new target enters, generating a new visual behavior representation, and performing multi-dimensional similarity matching between the new visual behavior representation and the global identity trajectory in the dormant pool to perform rebinding.

2. The motion-aware multi-target tracking method based on query passing according to claim 1, wherein, The combined expert decoder adopts a multi-branch parallel structure: the visual target query is structured data containing target position information in the current video frame and a feature vector capable of representing appearance; the multi-branch parallel structure receives the target position information of the previous video frame; each expert branch extrapolates the target position information of the previous video frame to generate a hypothetical future target area and extracts candidate visual features from the hypothetical future target area as calculation results; the motion prior for the extrapolation operation is defined by assigning different hypothetical motion models to each expert branch, which are used to predict future displacement based on the motion prior and define the hypothetical future target area; the gating network analyzes the instantaneous motion gating weight based on the matching degree between the visual target query and the candidate visual features, and uses the instantaneous motion gating weight to weight and fuse the candidate visual features output by each expert branch to generate motion-aware context features; the visual target query and the motion-aware context features interact to update the appearance query embedding of the current frame.

3. The motion-aware multi-target tracking method based on query passing according to claim 1, wherein, The process of generating a visual behavior representation includes: storing the updated appearance query embedding of the current video frame in a fixed-length appearance feature queue and storing the corresponding instantaneous motion gating weight in a fixed-length motion feature queue to generate a time sequence appearance state sequence and a time sequence motion state sequence, respectively; modeling the internal temporal dependency of the time sequence appearance state sequence and the time sequence motion state sequence through a temporal multi-modal fusion network, and interacting the time sequence appearance state sequence and the time sequence motion state sequence across modal temporal information, and aggregating the output of the fusion network into a fixed-dimensional vector, which is the visual behavior representation.

4. The motion-aware multi-target tracking method based on query passing according to claim 1, wherein, The process of combining the cross-modal association state space includes: the multi-source data stream further includes electronic ear tag data; at each corresponding time step, weight data and physiological data are extracted from the feeding station data and electronic ear tag data respectively, and after normalization processing of the weight data and physiological data, the weight data and physiological data are mapped and combined into a unified high-dimensional feature vector; the unified high-dimensional feature vectors generated at multiple time steps form a time sequence feature sequence; the time sequence feature sequence is input into a time sequence encoding network, the time sequence encoding network models the time sequence dependency within the sequence through an attention mechanism, and generates the data sequence; the visual behavior representation and the data sequence are combined to generate the cross-modal association state space.

5. The motion-aware multi-target tracking method based on query passing according to claim 1, wherein, The process of generating the time sequence consistency reward value includes: determining a residence time interval defined by a group of consecutive video frames, wherein in each frame of the consecutive video frames corresponding to the visual behavior representation, the visual target query falls within the preset feeding station geographic fence in the original target space coordinates of the video frame; from the original time sequence feature sequence corresponding to the input end of the time sequence encoding network of the data sequence, the original timestamp information contained is extracted, and consecutive timestamps with a time interval less than a preset threshold are aggregated into a weight data active time interval; the total overlap length of the residence time interval and the weight data active time interval is calculated, and the length is normalized as the time sequence consistency reward value.

6. The motion-aware multi-target tracking method based on query passing according to claim 1, wherein, The association decision action is generated by an association strategy model, and the association strategy model includes: a Transformer encoder including a self-attention layer and a cross-attention layer, which performs multi-modal fusion coding on the visual behavior representation and the data sequence in the cross-modal association state space; wherein the self-attention layer is responsible for modeling the internal time sequence dependency of each sequence, and the cross-attention layer is responsible for cross-modal information interaction between the two sequences, and jointly generates a fused state encoding vector; an action decoding network receives the state encoding vector and parses out the association decision action.

7. The motion-aware multi-target tracking method based on query passing according to claim 1, wherein, The process of generating the global identity trajectory includes: receiving corresponding visual target queries from multiple different monitoring video streams covering a farm in parallel; the global identity trajectory is constructed as a unified data structure, which binds the persistent global identity, the multiple appearance query embedding sequences of the visual target in different monitoring video streams, and the associated data sequence, and performs cross-camera tracking.

8. The motion-aware multi-target tracking method based on query passing according to claim 1, wherein, The process of generating the new visual behavior representation comprises: when a new target enters, caching the appearance query embedding sequence and the transient motion gating weight sequence of the new target within a preset initial time window; at each time step of the initial time window, constructing an appearance state vector from the corresponding appearance query embedding and constructing a motion state vector from the transient motion gating weight, to form an initial appearance state sequence and an initial motion state sequence, respectively; applying a temporal multi-modal fusion network to the initial appearance state sequence and the initial motion state sequence, the temporal multi-modal fusion network modeling the internal temporal dependency of the initial appearance state sequence and the initial motion state sequence, respectively, and performing cross-modal temporal information interaction between the initial appearance state sequence and the initial motion state sequence, and aggregating the output of the fusion network into a fixed-dimensional vector, which is the new visual behavior representation.

9. The motion-aware multi-target tracking method based on query passing according to claim 8, wherein, The process of re-binding comprises: calculating the highest similarity between the appearance query embedding sequence within the initial time window and the appearance query embedding sequence contained in each global identity track in the dormant pool, to obtain a first similarity score; aggregating the transient motion gating weight sequence within the initial time window into an initial motion mode vector through a temporal encoding network, and aggregating the historical motion weight sequence contained in each global identity track in the dormant pool into a historical motion mode vector through the same temporal encoding network, calculating the cosine similarity between the initial motion mode vector and the historical motion mode vector, to obtain a second similarity score; weighting and combining the first similarity score, the second similarity score, and the cross-mirror re-entry probability provided by the camera topology graph, performing candidate sorting on the global identity tracks in the dormant pool, and screening out Top-K candidate data sequences; selecting the candidate identity with the highest comprehensive score from the Top-K candidates according to a preset decision threshold; if selected, activating the global identity track of the candidate identity from the dormant pool and re-binding it with the new visual behavior representation of the current new target.

Images