A method and device for tracking spatio-temporal relationships of objects

Abstract

The application provides an object space-time relation tracking method and device. The method comprises: reasoning a visual abstract result of a video frame based on a qualitative space relation reasoner model, obtaining a plurality of candidate space relation graphs corresponding to the video frame, analyzing the plurality of candidate space relation graphs corresponding to the video frame based on a preset quantitative space relation analyzer model, predicting a motion trajectory of an invisible object in the video frame, and obtaining a complete space relation graph corresponding to each object in the video frame; determining a probability value corresponding to the complete space relation graph respectively, determining a target space relation graph corresponding to the video frame according to the probability value, and obtaining a corresponding target tracking result. The object space-time relation tracking method provided by the application can quickly predict the motion trajectory of an invisible object in a video, effectively improves the accuracy and efficiency of the motion trajectory prediction of the invisible object in the video frame over time, and thus improves the object space-time relation tracking effect.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a method and apparatus for tracking the spatiotemporal relationships of objects. It also relates to an electronic device and a processor-readable storage medium. Background Technology

[0002] Object persistence is a fundamental concept in developmental psychology, describing an agent's ability to understand an object's existence even when it is no longer observable. Research indicates that early human infants can only understand occluded objects. As for contained (i.e., enclosed) objects, infants don't grasp object persistence until much later. Infants typically develop a concept of object persistence after age two; before that, they are unaware that the world exists outside their concepts and experiences. This evidence suggests that understanding object persistence is a challenging task requiring thorough world modeling based on perceptual interactions with objects. Higher-level reasoning is only possible after establishing the concept of object persistence, as objects in natural scenes often dynamically occlude and contain each other. Therefore, object persistence is crucial for guiding tracking tasks and even helping agents understand and model the objective world. However, most current object spatiotemporal relationship reasoning and tracking methods are only at the stage of understanding occluded objects, while the few methods capable of tracking contained objects are still immature. For example, PermaTrack uses a spatial recurrent network to accumulate a representation of a scene and uses this representation to locate occluded instances, but lacks an understanding of the contained objects. OPNet uses two consecutive LSTMs (Long Short-Term Memory) to predict "who to track" and "where it is," thus enabling the location of completely invisible instances. However, both of these existing implementations have significant limitations, severely restricting their application in real-world scenarios. Therefore, developing a more efficient object spatiotemporal relationship reasoning and tracking scheme is a pressing issue. Summary of the Invention

[0003] To address this, the present invention provides a method and apparatus for tracking object spatiotemporal relationships, thereby overcoming the shortcomings of existing object spatiotemporal relationship tracking schemes, which have high limitations and thus lead to significant limitations in practical applications.

[0004] In a first aspect, the present invention provides a method for tracking the spatiotemporal relationship of objects, comprising:

[0005] Obtain the visual abstraction result of the input video frame;

[0006] The visual abstraction result is inferred based on a preset qualitative spatial relationship inference model to obtain multiple candidate spatial relationship graphs corresponding to the video frame; wherein, the candidate spatial relationship graph is a directed graph structure composed of the spatial relationships between visible objects in the video frame, and each visible object corresponds to a node in the directed graph structure.

[0007] Based on a preset quantitative spatial relationship analyzer model, multiple candidate spatial relationship graphs corresponding to the video frame are analyzed to predict the motion trajectory of invisible objects in the video frame, so as to obtain a complete spatial relationship graph corresponding to each object in the video frame; the probability value corresponding to the complete spatial relationship graph is determined, and the target spatial relationship graph corresponding to the video frame is determined according to the probability value to obtain the corresponding target tracking result; wherein, the invisible objects include objects in the video frame that are occluded or contained by visible objects.

[0008] Furthermore, the step of reasoning about the visual abstraction result based on a preset qualitative spatial relationship inference model to obtain multiple candidate spatial relationship graphs corresponding to the video frame specifically includes:

[0009] The spatial relationship predictor model trained in the qualitative spatial relationship inference model is used to perform inference analysis on the visual abstraction results to obtain event prediction results; wherein, the spatial relationship predictor model is a multi-head attention model obtained by self-supervised training or supervised training based on sample video frames and sample events corresponding to the sample video frames.

[0010] Based on the event prediction results and the historical spatial relationship graph of historical video frames, a candidate spatial relationship graph corresponding to the current video frame is generated; wherein, the historical spatial relationship graph of historical video frames is the spatial relationship graph corresponding to several video frames before the current video frame; the event prediction results are events that cause changes in spatial relationships, and the events are occlusion relationship events, inclusion relationship events, or events with no direct relationship.

[0011] Furthermore, the analysis of multiple candidate spatial relationship graphs corresponding to the video frame based on a preset quantitative spatial relationship analyzer model, and the prediction of the motion trajectory of invisible objects in the video frame, to obtain a complete spatial relationship graph corresponding to each object in the video frame, specifically includes:

[0012] The trajectory predictor model in the quantitative spatial relationship analyzer model is used to analyze multiple candidate spatial relationship graphs corresponding to the video frame in order to predict the motion trajectory of invisible objects in the video frame.

[0013] The trajectory predictor model is a diffusion model trained using historical video frames and the motion trajectories of various objects in the historical video frames as training samples.

[0014] Based on multiple candidate spatial relationship graphs corresponding to the video frame and the motion trajectories of invisible objects in the video frame, a complete spatial relationship graph corresponding to each object in the video frame is obtained.

[0015] Furthermore, determining the probability values ​​corresponding to the complete spatial relationship graphs, and determining the target spatial relationship graph corresponding to the video frame based on the probability values ​​to obtain the corresponding target tracking results, specifically includes:

[0016] Based on the preset evaluation algorithm model in the quantitative spatial relationship analyzer model, the probability value of the complete spatial relationship graph corresponding to the video frame is calculated. The complete spatial relationship graph corresponding to the largest probability value is determined as the target spatial relationship graph corresponding to the video frame. The target spatial relationship graph is selected to interpret the motion trajectory of the invisible object, so as to obtain the final target tracking result including the motion trajectory of the invisible object.

[0017] Furthermore, the acquisition of the visual abstraction result of the input video frame specifically includes:

[0018] Obtain the original video to be analyzed;

[0019] The positions of visible objects in video frames within the original video are detected to obtain position change information of the visible objects in the video frames; based on the change information of the visible objects in the video frames, corresponding visual feature information is extracted, and the corresponding visual abstraction result is obtained based on the visual feature information; wherein, the visual abstraction result includes the motion trajectory of the visible objects in the video frames.

[0020] Furthermore, before analyzing the multiple candidate spatial relationship graphs corresponding to the video frame based on the trajectory predictor model in the quantitative spatial relationship analyzer model, the method further includes:

[0021] The visual abstraction results corresponding to the video frame are optimized using the preset correction model in the quantitative spatial relationship analyzer model to obtain multiple candidate spatial relationship graphs corresponding to the video frame.

[0022] Furthermore, the step of performing reasoning analysis on the visual abstraction result based on the spatial relationship predictor model trained in the qualitative spatial relationship inference model to obtain event prediction results specifically includes: performing reasoning analysis on the visual abstraction result based on the spatial relationship predictor model trained in the qualitative spatial relationship inference model to obtain the probability vectors of each event that causes invisible objects to be generated in the video frame, and obtaining the corresponding event prediction results based on the probability vectors of each event.

[0023] Secondly, the present invention also provides an object spatiotemporal relationship tracking device, comprising:

[0024] The visual perception unit is used to acquire the visual abstraction result of the input video frame;

[0025] A qualitative spatial relationship reasoning unit is used to reason about the visual abstraction result based on a preset qualitative spatial relationship reasoning model to obtain multiple candidate spatial relationship graphs corresponding to the video frame; wherein, the candidate spatial relationship graph is a directed graph structure composed of the spatial relationships between visible objects in the video frame, and each visible object corresponds to a node in the directed graph structure.

[0026] The quantitative spatial relationship analysis unit is used to analyze multiple candidate spatial relationship graphs corresponding to the video frame based on a preset quantitative spatial relationship analyzer model, predict the motion trajectory of invisible objects in the video frame, and obtain a complete spatial relationship graph corresponding to each object in the video frame; determine the probability value corresponding to each of the complete spatial relationship graphs, and determine the target spatial relationship graph corresponding to the video frame based on the probability value, so as to obtain the corresponding target tracking result; wherein, the invisible objects include objects in the video frame that are occluded or contained by visible objects.

[0027] Furthermore, the qualitative spatial relationship reasoning unit is specifically used for:

[0028] The spatial relationship predictor model trained in the qualitative spatial relationship inference model is used to perform inference analysis on the visual abstraction results to obtain event prediction results; wherein, the spatial relationship predictor model is a multi-head attention model obtained by self-supervised training or supervised training based on sample video frames and sample events corresponding to the sample video frames.

[0029] Based on the event prediction results and the historical spatial relationship graph of historical video frames, a candidate spatial relationship graph corresponding to the current video frame is generated; wherein, the historical spatial relationship graph of historical video frames is the spatial relationship graph corresponding to several video frames before the current video frame; the event prediction results are events that cause changes in spatial relationships, and the events are occlusion relationship events, inclusion relationship events, or events with no direct relationship.

[0030] Furthermore, the basic quantitative spatial relationship analysis unit is specifically used for:

[0031] The trajectory predictor model in the quantitative spatial relationship analyzer model is used to analyze multiple candidate spatial relationship graphs corresponding to the video frame in order to predict the motion trajectory of invisible objects in the video frame.

[0032] The trajectory predictor model is a diffusion model trained using historical video frames and the motion trajectories of various objects in the historical video frames as training samples.

[0033] Based on multiple candidate spatial relationship graphs corresponding to the video frame and the motion trajectories of invisible objects in the video frame, a complete spatial relationship graph corresponding to each object in the video frame is obtained.

[0034] Furthermore, the basic quantitative spatial relationship analysis unit is specifically used for:

[0035] Based on the preset evaluation algorithm model in the quantitative spatial relationship analyzer model, the probability value of the complete spatial relationship graph corresponding to the video frame is calculated. The complete spatial relationship graph corresponding to the largest probability value is determined as the target spatial relationship graph corresponding to the video frame. The target spatial relationship graph is selected to interpret the motion trajectory of the invisible object, so as to obtain the final target tracking result including the motion trajectory of the invisible object.

[0036] Furthermore, the visual perception unit is specifically used for: acquiring the input original video to be analyzed; detecting the position of visible objects in video frames within the original video, and obtaining position change information of visible objects in the video frames; extracting corresponding visual feature information based on the change information of visible objects in the video frames, and obtaining the corresponding visual abstraction result based on the visual feature information; wherein, the visual abstraction result includes the motion trajectory of visible objects in the video frames.

[0037] Furthermore, before analyzing the multiple candidate spatial relationship graphs corresponding to the video frame based on the trajectory predictor model in the quantitative spatial relationship analyzer model, the method further includes:

[0038] The correction processing unit is used to optimize the visual abstraction result corresponding to the video frame using the preset correction model in the quantitative spatial relationship analyzer model, so as to obtain multiple candidate spatial relationship graphs corresponding to the video frame.

[0039] Furthermore, the step of performing reasoning analysis on the visual abstraction result based on the spatial relationship predictor model trained in the qualitative spatial relationship inference model to obtain event prediction results specifically includes: performing reasoning analysis on the visual abstraction result based on the spatial relationship predictor model trained in the qualitative spatial relationship inference model to obtain the probability vectors of each event that causes invisible objects to be generated in the video frame, and obtaining the corresponding event prediction results based on the probability vectors of each event.

[0040] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the object spatiotemporal relationship tracking method as described in any of the above.

[0041] Fourthly, the present invention also provides a processor-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the object spatiotemporal relationship tracking method as described in any of the preceding claims.

[0042] The object spatiotemporal relationship tracking method provided by this invention infers the visual abstraction results of video frames based on a qualitative spatial relationship inference model to obtain multiple candidate spatial relationship graphs corresponding to the video frames. Based on a preset quantitative spatial relationship analyzer model, the multiple candidate spatial relationship graphs corresponding to the video frames are analyzed to predict the motion trajectory of invisible objects in the video frames, thereby obtaining complete spatial relationship graphs corresponding to each object in the video frames. The probability values ​​corresponding to the complete spatial relationship graphs are determined, and the target spatial relationship graph corresponding to the video frames is determined based on the probability values ​​to obtain the corresponding target tracking results. It can quickly predict the motion trajectory of invisible objects in videos, effectively improving the accuracy and efficiency of predicting the motion trajectory of invisible objects as video frames change over time, thereby improving the object spatiotemporal relationship tracking effect. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a flowchart illustrating the object spatiotemporal relationship tracking method provided in an embodiment of the present invention;

[0045] Figure 2 This is one of the flowcharts illustrating the qualitative and quantitative spatial relationship reasoning and prediction framework provided in this embodiment of the invention;

[0046] Figure 3 This is the second flowchart illustrating the qualitative and quantitative spatial relationship reasoning and prediction framework provided in this embodiment of the invention.

[0047] Figure 4 This is the third flowchart illustrating the qualitative and quantitative spatial relationship reasoning and prediction framework provided in this embodiment of the invention.

[0048] Figure 5 This is a schematic diagram of the spatial relationship provided in an embodiment of the present invention;

[0049] Figure 6 This is a schematic diagram of the qualitative spatial relationship reasoning process provided in an embodiment of the present invention;

[0050] Figure 7 This is a schematic diagram of the object spatiotemporal relationship tracking device provided in an embodiment of the present invention;

[0051] Figure 8 This is a schematic diagram of the physical structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] It should be noted that the terms "first," "second," etc., in the specification and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0054] Current technologies suffer from two main drawbacks: Rule-based methods typically define spatiotemporal relationships between objects manually and guide visual reasoning tasks through rules or assumptions. These methods often require more information or assumptions to guide the reasoning and have significant limitations in practical applications. Data-based methods, on the other hand, learn feature representations that express spatiotemporal relationships between objects from large amounts of training data and use these features to guide visual reasoning tasks. However, the features learned by these data-based methods are often implicit and abstract, lacking interpretability and difficult to connect with the reasoning structures of human cognition.

[0055] This invention proposes a method that combines the ideas of rule-based and data-based approaches, leveraging the advantages of both. It defines a reasonable and interpretable representation of spatial relationships and can learn object spatial relationships in a self-supervised manner using only video annotations. The object tracking task is viewed as studying how its spatial relationships change over time, whether visible, occluded, or contained. Figure 2 As shown, this invention decomposes spatial relationship reasoning into three levels of analysis: image level, relationship level, and event level. At the image level, visual feature information of the input video frames can be extracted using a visual module (VM). Then, at the relationship level, a spatial-relation reasoner (SRR) is used to qualitatively hypothesize possible spatial relationship graphs based on a qualitative spatial relationship reasoner model. At the event level, a spatial-relation analyzer (SRA) is used based on a quantitative spatial relationship analyzer model to infer events leading to changes in spatial relationships according to a diffusion model, and outputs bounding boxes in the image space to obtain target tracking results.

[0056] The following section first describes in detail the embodiments of the object spatiotemporal relationship tracking method described in this invention. For example... Figure 1 The diagram shown is a flowchart of the object spatiotemporal relationship tracking method provided in an embodiment of the present invention. The specific implementation process includes the following steps:

[0057] Step 101: Obtain the visual abstraction result of the input video frame.

[0058] Specifically, the input original video to be analyzed can be obtained through a preset visual perception model. The positions of visible objects in the video frames within the original video are detected to obtain positional change information of the visible objects in the video frames. Based on this change information, corresponding visual feature information is extracted, and the corresponding visual abstraction result is obtained based on the visual feature information. The visual abstraction result includes the motion trajectory of the visible objects in the video frames. It should be noted that the visual perception model includes a target detection module for detecting the positions of visible objects in the video frames within the original video and a human motion module for detecting human posture. The purpose is to detect visible objects and human postures in each video frame as visual abstraction results for further inference. The target detection module can be implemented using the same network architecture as OPNet to detect targets in synthetic scenes. This target detection module uses the Faster R-CNN network dataset pre-trained on COCO (Common Objects in COntext) as the backbone network, and the classifier head is fine-tuned on the LA-CATER dataset. In real-world object detection, the AutoMatch model can be used as the target detection module. For the human motion module, the human skeleton can be extracted using the PoseC3D model; no specific limitations are specified here. The "visible object" refers to any object that is visible.

[0059] Step 102: Based on a preset qualitative spatial relationship inference model, reason about the visual abstraction result to obtain multiple candidate spatial relationship graphs corresponding to the video frame; wherein, the candidate spatial relationship graph is a directed graph structure composed of the spatial relationships between visible objects in the video frame, and each visible object corresponds to a node in the directed graph structure; the qualitative spatial relationship inference model is trained based on sample video frames and the sample spatial relationship graphs corresponding to the sample video frames.

[0060] Specifically, the visual abstraction result can be inferred and analyzed based on the spatial relationship predictor model trained in the qualitative spatial relationship inference model to obtain event prediction results. The spatial relationship predictor model is a multi-head attention model obtained through self-supervised or supervised training based on sample video frames and their corresponding sample events. Further, based on the event prediction results and the historical spatial relationship graphs of historical video frames, a candidate spatial relationship graph corresponding to the current video frame is generated. The historical spatial relationship graphs of historical video frames are the spatial relationship graphs corresponding to several video frames preceding the current video frame; the event prediction results are events that cause changes in spatial relationships, and these events can be occlusion events, inclusion events, or events with no direct relationship. The process of using a spatial relationship predictor model trained in the qualitative spatial relationship inference model to perform reasoning analysis on the visual abstraction result and obtain event prediction results includes: performing reasoning analysis on the visual abstraction result using the spatial relationship predictor model trained in the qualitative spatial relationship inference model to obtain probability vectors of various events that lead to the generation of invisible objects in the video frame, and obtaining corresponding event prediction results based on the probability vectors of each event. The invisible objects are those that are not visible.

[0061] It should be noted that during the reasoning process of the visual abstraction result based on the preset qualitative spatial relationship inference model, the spatial relationships between objects (i.e., objects) at each moment can be summarized into three types: occlusion relationship, containment relationship (i.e., containment relationship), and no direct relationship. Each object can be regarded as a node, and the spatial relationships between each pair of objects form a directed graph structure. The directed graph constructed in this way is defined as the spatial relationship graph of the current video frame. In practice, very little information can be obtained about contained and occluded objects, making it difficult to determine the unique and correct spatial relationship graph corresponding to each video frame. Therefore, this invention generates at most K spatial relationship graphs as candidate spatial relationship graphs for the current input video frame. Figure 5 The example of the candidate spatial relationship diagram corresponding to the scene on the left is shown, omitting those without direct relationship. The circular object in the rightmost triangle can be the target to be detected (i.e. the invisible object occluded by the rightmost triangle in front). The arrows indicate spatial relationships, which are spatiotemporal relationships.

[0062] Additionally, it should be noted that during the reasoning process of the visual abstraction result based on the preset qualitative spatial relationship inference model, the spatial-relation predictor model in the qualitative spatial relationship inference model is used to predict events that cause changes in object visibility (such as occlusion events, i.e., contain events; inclusion events, i.e., occlude events). Considering that the perception of changes in the spatial relationship of an object largely depends on people's prior knowledge of previous motion, this invention predicts the probability of each event by using the trajectories in the past L video frames (i.e., historical video frames). The spatial relationship predictor model uses a multi-head attention model to learn the motion features of the target x relative to other objects that may lead to occlusion or inclusion events, and the final probability vector is obtained through a perceptron. After the spatial relationship predictor model predicts the possible events, it obtains the event prediction result. The qualitative spatial relationship inference model (SRR) then uses the event prediction result and the spatial relationship graph of the previous video frame to generate a candidate spatial relationship graph for the current input video frame, such as... Figure 6 As shown, the Predictor is the trained spatial relationship predictor model. In this embodiment, two methods are provided to train the spatial relationship predictor model. Supervised training generates training trajectories and obtains true probability vectors based on the motion labels of the training data; that is, the contain or occlude relationship corresponding to the motion label is 1, and the rest are 0. Self-supervised training uses only the original video as input and generates corresponding probability vectors based on the intersection-union (IoU) ratio of the bounding boxes of the inferred subsequent reappearance locations of objects and the bounding boxes of the predicted locations.

[0063] In other words, based on a pre-defined qualitative spatial relationship inference model, the visual abstraction results of the visual perception model are inferred, and hypotheses about spatial relationships are generated, including relationships between people and relationships between objects. At this level, based on the visual abstraction results obtained from the visual perception model and the spatial relationships inferred from the previous moment in historical video frames, it is determined whether the state of the target (i.e., invisible objects) has changed, and the corresponding spatial relationship diagram is updated qualitatively.

[0064] Step 103: Analyze multiple candidate spatial relationship graphs corresponding to the video frame based on a preset quantitative spatial relationship analyzer model, predict the motion trajectory of invisible objects in the video frame, and obtain a complete spatial relationship graph corresponding to each object in the video frame; determine the probability value corresponding to each complete spatial relationship graph, and determine the target spatial relationship graph corresponding to the video frame based on the probability value, so as to obtain the corresponding target tracking result. The invisible objects include objects in the video frame that are occluded or contained by visible objects; the quantitative spatial relationship analyzer model is trained based on sample spatial relationship graphs and the sample motion trajectories corresponding to the sample spatial relationship graphs.

[0065] The process involves analyzing multiple candidate spatial relationship graphs corresponding to the video frame based on a preset quantitative spatial relationship analyzer model to predict the motion trajectories of invisible objects in the video frame, thereby obtaining a complete spatial relationship graph corresponding to each object in the video frame. This process may include: analyzing multiple candidate spatial relationship graphs corresponding to the video frame based on a trajectory predictor model within the quantitative spatial relationship analyzer model to predict the motion trajectories of invisible objects in the video frame. The trajectory predictor model is a diffusion model trained using historical video frames and the motion trajectories of objects within those historical video frames as training samples. Further, based on the multiple candidate spatial relationship graphs corresponding to the video frame and the motion trajectories of invisible objects in the video frame, a complete spatial relationship graph corresponding to each object in the video frame is obtained.

[0066] It should be noted that before analyzing the multiple candidate spatial relationship maps corresponding to the video frame based on the trajectory predictor model in the quantitative spatial relationship analyzer model, it is necessary to optimize the visual abstraction results corresponding to the video frame using the preset correction model in the quantitative spatial relationship analyzer model to obtain multiple candidate spatial relationship maps corresponding to the video frame. Furthermore, the process of determining the probability values ​​corresponding to the complete spatial relationship maps and determining the target spatial relationship map corresponding to the video frame based on the probability values ​​to obtain the corresponding target tracking results includes: calculating the probability values ​​of the complete spatial relationship maps corresponding to the video frame based on the preset evaluation algorithm model in the quantitative spatial relationship analyzer model; determining the complete spatial relationship map corresponding to the largest probability value as the target spatial relationship map corresponding to the video frame; and selecting the target spatial relationship map to interpret the motion trajectory of the invisible object to obtain the final target tracking result including the motion trajectory of the invisible object.

[0067] It should be noted that after obtaining a qualitative model of the spatial relationships of objects, hypotheses can be used to infer related events and track invisible objects. The Quantitative Spatial Relationship Analyzer (SRA) model is used to analyze the possible spatial relationship graphs (i.e., candidate spatial relationship graphs) corresponding to each video frame and predict the motion trajectory of invisible objects under the given assumptions. The SRA model can be divided into three models: a correction model to correct some errors in the visual perception model, a trajectory predictor model to predict the motion trajectory of invisible objects, and an evaluation model to calculate the probability value of each candidate spatial relationship graph. Specifically, the correction model can be used to check whether the visual abstraction result of the current input video frame is reliable, for example, assuming that the coordinates of the object do not change in the case of jitter or switching. The trajectory predictor model treats the trajectory prediction task as a conditional generation task and attempts to apply a diffusion model to solve it. Under high uncertainty, the possible motion trajectory of the invisible object can be regarded as a noisy Gaussian distribution, representing the fuzzy region controlled by the container's motion trajectory. As the uncertainty decreases, the distribution gradually approaches the true distribution that generates the correct trajectory. The diffusion model is a Markov noise process. Given the motion trajectories derived from the data distribution, this process aims to progressively add noise sequences. The uncertainty continues until the actual ground motion trajectory is mixed with noise.

[0068]

[0069]

[0070] in, and It is a constant hyperparameter. q(τ) t |τ t-1 ) is the probability distribution function used by the diffusion model to generate the noise sequence, i.e., the average value is The variance is (1-α) t The Gaussian distribution of I. When T is sufficiently large... When it is small enough, τ can be approximated. t ~N(0,I).

[0071] Next, the generation of the motion trajectory is described as a backdiffusion process. The backdiffusion process for predicting each invisible object is defined as follows.

[0072] p θ (τ t-1 |τ t ,f motion )=N(τ t-1 μ θ (τ t ,t,f motion ),Σ θ (τt ,t))

[0073] p θ (τ T )=G(f motion )

[0074] in, G(f) represents motion features in past historical video frames; motion ) represents the motion trajectory f through historical video frames. motion A naive rule-based method for predicting the motion trajectory of the current input video frame; θ represents the parameters of the diffusion model; μ θ , Σ θ p represents the mean function and variance function corresponding to the diffusion model. θ This is the probability distribution function used for denoising in the diffusion model. The model directly predicts the trajectory itself, μ. θ (τ t ,t,f motion The perceptron is used as the diffusion model and trained on the motion trajectory with a simple loss function objective, namely, using the L2 loss of the actual trajectory and the trajectory predicted by the diffusion model.

[0075]

[0076] In addition, the evaluation model is used to calculate the probability value of the spatial relationship graph for each video frame, so as to retain the K most likely candidate spatial relationship graphs and select the target spatial relationship graph to explain the motion trajectory of the target (i.e., the invisible object) as the final target tracking result. That is, based on the preset quantitative spatial relationship analyzer model, the inference results based on the qualitative spatial relationship inferencer model are analyzed to infer the events that cause changes in spatial relationships and evaluate the probability of the hypothesis proposed by the qualitative spatial relationship inferencer model. Based on the motion trajectory of the object in historical video frames, the most likely set of spatial relationships (i.e., the target spatial relationship graph) in the candidate spatial relationship graph set is inferred, and the motion trajectory of the invisible object is predicted quantitatively. Specifically, the algorithm corresponding to the evaluation model is as follows: Let the spatial relationship graph... It is obtained through the prediction of the Predictor of the m-th Spatial Relationship Inference Model (SRR), where the probability value of the i-th prediction is p. i Then the evaluation probability of the candidate spatial relationship graph is

[0077]

[0078] like Figure 2 and 3As shown, this invention presents a qualitative and quantitative reasoning and analysis framework. It achieves the tracking of invisible objects from the input raw video by performing causal reasoning on spatial relationships. First, a visual perception model (VM) is used to locate visible objects and human poses in each video frame for visual abstraction, obtaining the corresponding visual abstraction results. Then, a qualitative spatial relationship inferencer model (SRR) is used to infer the spatial relationships between people and objects based on the visual abstraction results. Finally, a quantitative spatial relationship analyzer model (SRA) is used to estimate the probability of spatial relationship candidates based on a diffusion model, obtaining a target spatial relationship graph and predicting the motion trajectory of invisible objects.

[0079] Specifically, in Figure 2 In this model, l1-l4 represent video frames that change over time. First, the video frames are input into a Visual Perception Model (VM) to obtain corresponding visual abstraction results. Then, these visual abstraction results are input into a pre-defined Qualitative Spatial Relationship Inference (SRR) model for inference analysis, resulting in several candidate spatial relationship graphs. In the Relation Level, dark arrows represent containment relationships, where an object at the tail of the arrow is contained by the object pointed to by the arrow, making it invisible. Dashed lines in the Relation Level represent occlusion relationships, where an object below the dashed line is occluded by an object above the dashed line, making it invisible. Finally, multiple candidate spatial relationship graphs are input into a Quantitative Spatial Relationship Analyzer (SRA) model for probability analysis and motion trajectory prediction to obtain target tracking results. In the Event Level, the arrows marking "contain" represent event prediction results.

[0080] like Figure 3 As shown, in the quantitative analysis (Qualitative Reasoner) process, l1-l n For video frames that change over time, the video frames are first input into a visual perception model (VM) to obtain the corresponding visual abstraction result, id. i The rectangle in the video frame is numbered to indicate which object corresponds to the rectangle, in order to identify different objects; The quadruple represents the position of the top, bottom, left, and right points of the rectangle, and k represents the number of all objects in the video frame. The coordinates of a target body part (such as the coordinates of a human hand) are represented by "person", and "person" represents the number of target body parts. This represents a possible spatial relationship graph, i.e., multiple candidate spatial relationship graphs, where K represents the number of candidate spatial relationship graphs. In the qualitative analysis process, `results` represents the event prediction results; `n-1loops` represents the complete spatial relationship graph of historical video frames. In this diagram, x represents the target object (i.e., an invisible object), t represents the current time, and 1 represents the complete spatial relationship graph with the highest probability. See also the example below for more details. Figure 4 The process shown will not be described in detail here.

[0081] The object spatiotemporal relationship tracking method described in this invention infers the visual abstraction results of video frames based on a qualitative spatial relationship inference model to obtain multiple candidate spatial relationship graphs corresponding to the video frames. It then analyzes these candidate spatial relationship graphs based on a preset quantitative spatial relationship analyzer model to predict the motion trajectories of invisible objects in the video frames, thereby obtaining complete spatial relationship graphs corresponding to each object in the video frames. The method also determines the probability values ​​corresponding to each complete spatial relationship graph and determines the target spatial relationship graph corresponding to the video frame based on the probability values, thus obtaining the corresponding target tracking results. This method can quickly predict the motion trajectories of invisible objects in videos, effectively improving the accuracy and efficiency of predicting the motion trajectories of invisible objects as video frames change over time, thereby enhancing the object spatiotemporal relationship tracking effect.

[0082] Corresponding to the object spatiotemporal relationship tracking method provided above, this invention also provides an object spatiotemporal relationship tracking device. Since the embodiment of this device is similar to the method embodiment described above, the description is relatively simple. For relevant details, please refer to the description in the method embodiment section above. The embodiment of the object spatiotemporal relationship tracking device described below is merely illustrative. Please refer to... Figure 7 The diagram shown is a structural schematic of an object spatiotemporal relationship tracking device provided in an embodiment of the present invention. The object spatiotemporal relationship tracking device of the present invention specifically includes the following parts:

[0083] The visual perception unit 701 is used to acquire the visual abstraction result of the input video frame;

[0084] The qualitative spatial relationship reasoning unit 702 is used to reason about the visual abstraction result based on a preset qualitative spatial relationship reasoning model to obtain multiple candidate spatial relationship graphs corresponding to the video frame; wherein, the candidate spatial relationship graph is a directed graph structure composed of the spatial relationships between visible objects in the video frame, and each visible object corresponds to a node in the directed graph structure; the qualitative spatial relationship reasoning model is trained based on sample video frames and the sample spatial relationship graphs corresponding to the sample video frames.

[0085] The quantitative spatial relationship analysis unit 703 is used to analyze multiple candidate spatial relationship maps corresponding to the video frame based on a preset quantitative spatial relationship analyzer model, predict the motion trajectory of invisible objects in the video frame, so as to obtain a complete spatial relationship map corresponding to each object in the video frame; determine the probability value corresponding to the complete spatial relationship map respectively, and determine the target spatial relationship map corresponding to the video frame according to the probability value, so as to obtain the corresponding target tracking result.

[0086] The invisible objects include objects in the video frame that are occluded or contained by visible objects; the quantitative spatial relationship analyzer model is trained based on the sample spatial relationship graph and the sample motion trajectory corresponding to the sample spatial relationship graph.

[0087] Furthermore, the qualitative spatial relationship reasoning unit is specifically used for:

[0088] The spatial relationship predictor model trained in the qualitative spatial relationship inference model is used to perform inference analysis on the visual abstraction results to obtain event prediction results; wherein, the spatial relationship predictor model is a multi-head attention model obtained by self-supervised training or supervised training based on sample video frames and sample events corresponding to the sample video frames.

[0089] Based on the event prediction results and the historical spatial relationship graph of historical video frames, a candidate spatial relationship graph corresponding to the current video frame is generated; wherein, the historical spatial relationship graph of historical video frames is the spatial relationship graph corresponding to several video frames before the current video frame; the event prediction results are events that cause changes in spatial relationships, and the events are occlusion relationship events, inclusion relationship events, or events with no direct relationship.

[0090] Furthermore, the basic quantitative spatial relationship analysis unit is specifically used for:

[0091] The trajectory predictor model in the quantitative spatial relationship analyzer model is used to analyze multiple candidate spatial relationship graphs corresponding to the video frame in order to predict the motion trajectory of invisible objects in the video frame.

[0092] The trajectory predictor model is a diffusion model trained using historical video frames and the motion trajectories of various objects in the historical video frames as training samples.

[0093] Based on multiple candidate spatial relationship graphs corresponding to the video frame and the motion trajectories of invisible objects in the video frame, a complete spatial relationship graph corresponding to each object in the video frame is obtained.

[0094] Furthermore, the basic quantitative spatial relationship analysis unit is specifically used for:

[0095] Based on the preset evaluation algorithm model in the quantitative spatial relationship analyzer model, the probability value of the complete spatial relationship graph corresponding to the video frame is calculated. The complete spatial relationship graph corresponding to the largest probability value is determined as the target spatial relationship graph corresponding to the video frame. The target spatial relationship graph is selected to interpret the motion trajectory of the invisible object, so as to obtain the final target tracking result including the motion trajectory of the invisible object.

[0096] Furthermore, the visual perception unit is specifically used for: acquiring the input original video to be analyzed; detecting the position of visible objects in video frames within the original video, and obtaining position change information of visible objects in the video frames; extracting corresponding visual feature information based on the change information of visible objects in the video frames, and obtaining the corresponding visual abstraction result based on the visual feature information; wherein, the visual abstraction result includes the motion trajectory of visible objects in the video frames.

[0097] Furthermore, before analyzing the multiple candidate spatial relationship graphs corresponding to the video frame based on the trajectory predictor model in the quantitative spatial relationship analyzer model, the method further includes:

[0098] The correction processing unit is used to optimize the visual abstraction result corresponding to the video frame using the preset correction model in the quantitative spatial relationship analyzer model, so as to obtain multiple candidate spatial relationship graphs corresponding to the video frame.

[0099] Furthermore, the step of performing reasoning analysis on the visual abstraction result based on the spatial relationship predictor model trained in the qualitative spatial relationship inference model to obtain event prediction results specifically includes: performing reasoning analysis on the visual abstraction result based on the spatial relationship predictor model trained in the qualitative spatial relationship inference model to obtain the probability vectors of each event that causes invisible objects to be generated in the video frame, and obtaining the corresponding event prediction results based on the probability vectors of each event.

[0100] The object spatiotemporal relationship tracking device described in this embodiment of the invention infers the visual abstraction results of video frames based on a qualitative spatial relationship inference model to obtain multiple candidate spatial relationship maps corresponding to the video frames. It then analyzes these candidate spatial relationship maps based on a preset quantitative spatial relationship analyzer model to predict the motion trajectories of invisible objects in the video frames, thereby obtaining complete spatial relationship maps corresponding to each object in the video frames. The device also determines the probability values ​​corresponding to each complete spatial relationship map and determines the target spatial relationship map corresponding to the video frames based on the probability values, thus obtaining the corresponding target tracking results. This device can quickly predict the motion trajectories of invisible objects in videos, effectively improving the accuracy and efficiency of predicting the motion trajectories of invisible objects as video frames change over time, thereby enhancing the object spatiotemporal relationship tracking effect.

[0101] Corresponding to the object spatiotemporal relationship tracking method provided above, this invention also provides an electronic device. Since the embodiment of this electronic device is similar to the method embodiment described above, it is described simply. For relevant details, please refer to the description in the method embodiment section above. The electronic device described below is merely illustrative. Figure 8 The diagram shown is a schematic representation of the physical structure of an electronic device disclosed in an embodiment of the present invention. The electronic device may include a processor 801, a memory 802, and a communication bus 803. The processor 801 and the memory 802 communicate with each other via the communication bus 803 and communicate with the outside world via a communication interface 804. The processor 801 can call logical instructions in the memory 802 to execute an object spatiotemporal relationship tracking method. This method includes: acquiring a visual abstraction result of an input video frame; reasoning about the visual abstraction result based on a preset qualitative spatial relationship inference model to obtain multiple candidate spatial relationship graphs corresponding to the video frame; wherein the candidate spatial relationship graph is a directed graph structure composed of spatial relationships between visible objects in the video frame, and each visible object corresponds to a node in the directed graph structure; the qualitative spatial relationship inference model is obtained by training based on sample video frames and the sample spatial relationship graphs corresponding to the sample video frames. The quantitative spatial relationship analyzer model is used to analyze multiple candidate spatial relationship graphs corresponding to the video frame, predict the motion trajectory of invisible objects in the video frame, and obtain the complete spatial relationship graph corresponding to each object in the video frame. The probability value corresponding to each complete spatial relationship graph is determined, and the target spatial relationship graph corresponding to the video frame is determined according to the probability value to obtain the corresponding target tracking result. The invisible objects include objects in the video frame that are occluded or contained by visible objects. The quantitative spatial relationship analyzer model is trained based on the sample spatial relationship graph and the sample motion trajectory corresponding to the sample spatial relationship graph.

[0102] Furthermore, the logical instructions in the aforementioned memory 802 can be implemented as software functional modules and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as memory chips, USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0103] On the other hand, embodiments of the present invention also provide a computer program product, the computer program product including a computer program stored on a processor-readable storage medium, the computer program including program instructions, and when the program instructions are executed by a computer, the computer is able to execute the object spatiotemporal relationship tracking method provided in the above method embodiments. The method includes: acquiring a visual abstraction of an input video frame; reasoning about the visual abstraction based on a preset qualitative spatial relationship inference model to obtain multiple candidate spatial relationship graphs corresponding to the video frame; wherein, the candidate spatial relationship graph is a directed graph structure composed of spatial relationships between visible objects in the video frame, and each visible object corresponds to a node in the directed graph structure; the qualitative spatial relationship inference model is trained based on sample video frames and sample spatial relationship graphs corresponding to the sample video frames; analyzing the multiple candidate spatial relationship graphs corresponding to the video frame based on a preset quantitative spatial relationship analyzer model to predict the motion trajectory of invisible objects in the video frame, so as to obtain a complete spatial relationship graph corresponding to each object in the video frame; determining the probability value corresponding to each of the complete spatial relationship graphs, and determining the target spatial relationship graph corresponding to the video frame based on the probability value, so as to obtain the corresponding target tracking result; wherein, the invisible objects include objects in the video frame that are occluded or contained by visible objects; the quantitative spatial relationship analyzer model is trained based on sample spatial relationship graphs and sample motion trajectories corresponding to the sample spatial relationship graphs.

[0104] In another aspect, embodiments of the present invention also provide a processor-readable storage medium storing a computer program that, when executed by a processor, implements the object spatiotemporal relationship tracking method provided in the above embodiments. The method includes: acquiring a visual abstraction of an input video frame; reasoning about the visual abstraction based on a preset qualitative spatial relationship inference model to obtain multiple candidate spatial relationship graphs corresponding to the video frame; wherein, the candidate spatial relationship graph is a directed graph structure composed of spatial relationships between visible objects in the video frame, and each visible object corresponds to a node in the directed graph structure; the qualitative spatial relationship inference model is trained based on sample video frames and sample spatial relationship graphs corresponding to the sample video frames; analyzing the multiple candidate spatial relationship graphs corresponding to the video frame based on a preset quantitative spatial relationship analyzer model to predict the motion trajectory of invisible objects in the video frame, so as to obtain a complete spatial relationship graph corresponding to each object in the video frame; determining the probability value corresponding to each of the complete spatial relationship graphs, and determining the target spatial relationship graph corresponding to the video frame based on the probability value, so as to obtain the corresponding target tracking result; wherein, the invisible objects include objects in the video frame that are occluded or contained by visible objects; the quantitative spatial relationship analyzer model is trained based on sample spatial relationship graphs and sample motion trajectories corresponding to the sample spatial relationship graphs.

[0105] The processor-readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic memory (e.g., floppy disk, hard disk, magnetic tape, magneto-optical disk (MO)), optical memory (e.g., CD, DVD, BD, HVD), and semiconductor memory (e.g., ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)).

[0106] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0107] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for tracking the spatiotemporal relationship of objects, characterized in that, include: Obtain the visual abstraction result of the input video frame; The spatial relationship predictor model trained in the qualitative spatial relationship inference model is used to perform inference analysis on the visual abstraction results to obtain event prediction results; wherein, the spatial relationship predictor model is a multi-head attention model obtained by self-supervised training or supervised training based on sample video frames and sample events corresponding to the sample video frames. Based on the event prediction results and the historical spatial relationship graphs of historical video frames, a candidate spatial relationship graph corresponding to the current video frame is generated; wherein, the historical spatial relationship graphs of historical video frames are the spatial relationship graphs corresponding to several video frames preceding the current video frame; the event prediction results are events that cause changes in spatial relationships, and the events are occlusion relationship events, inclusion relationship events, or events with no direct relationship; wherein, the candidate spatial relationship graph is a directed graph structure composed of the spatial relationships between visible objects in the video frame, and each visible object corresponds to a node in the directed graph structure; The trajectory predictor model in the quantitative spatial relationship analyzer model is used to analyze multiple candidate spatial relationship graphs corresponding to the video frame in order to predict the motion trajectory of invisible objects in the video frame; wherein, the trajectory predictor model is a diffusion model trained with historical video frames and the motion trajectories of each object in the historical video frames as training samples. Based on multiple candidate spatial relationship graphs corresponding to the video frame and the motion trajectory of invisible objects in the video frame, a complete spatial relationship graph corresponding to each object in the video frame is obtained. Determine the probability values ​​corresponding to the complete spatial relationship graphs, and determine the target spatial relationship graphs corresponding to the video frames based on the probability values ​​to obtain the corresponding target tracking results; wherein, the invisible objects include objects in the video frames that are occluded or contained by visible objects.

2. The object spatiotemporal relationship tracking method according to claim 1, characterized in that, The step of determining the probability values ​​corresponding to the complete spatial relationship graphs, and determining the target spatial relationship graph corresponding to the video frame based on the probability values ​​to obtain the corresponding target tracking results, specifically includes: Based on the preset evaluation algorithm model in the quantitative spatial relationship analyzer model, the probability value of the complete spatial relationship graph corresponding to the video frame is calculated. The complete spatial relationship graph corresponding to the largest probability value is determined as the target spatial relationship graph corresponding to the video frame. The target spatial relationship graph is selected to interpret the motion trajectory of the invisible object, so as to obtain the final target tracking result including the motion trajectory of the invisible object.

3. The object spatiotemporal relationship tracking method according to claim 1, characterized in that, The acquisition of the visual abstraction result of the input video frame specifically includes: Obtain the original video to be analyzed; The positions of visible objects in video frames within the original video are detected to obtain position change information of the visible objects in the video frames; based on the change information of the visible objects in the video frames, corresponding visual feature information is extracted, and the corresponding visual abstraction result is obtained based on the visual feature information; wherein, the visual abstraction result includes the motion trajectory of the visible objects in the video frames.

4. The object spatiotemporal relationship tracking method according to claim 1, characterized in that, Before analyzing the multiple candidate spatial relationship graphs corresponding to the video frame based on the trajectory predictor model in the quantitative spatial relationship analyzer model, the method further includes: The visual abstraction results corresponding to the video frame are optimized using the preset correction model in the quantitative spatial relationship analyzer model to obtain multiple candidate spatial relationship graphs corresponding to the video frame.

5. The object spatiotemporal relationship tracking method according to claim 1, characterized in that, The spatial relationship predictor model trained in the qualitative spatial relationship inference model performs inference analysis on the visual abstraction result to obtain event prediction results. Specifically, this includes: performing inference analysis on the visual abstraction result based on the spatial relationship predictor model trained in the qualitative spatial relationship inference model to obtain the probability vectors of each event that causes invisible objects to be generated in the video frame, and obtaining the corresponding event prediction results based on the probability vectors of each event.

6. A device for tracking the spatiotemporal relationship of an object, characterized in that, include: The visual perception unit is used to acquire the visual abstraction result of the input video frame; A qualitative spatial relationship reasoning unit is used to perform reasoning analysis on the visual abstraction result based on the spatial relationship predictor model trained in the qualitative spatial relationship inference model, and to obtain the event prediction result; wherein, the spatial relationship predictor model is a multi-head attention model obtained by self-supervised training or supervised training based on sample video frames and sample events corresponding to the sample video frames. Based on the event prediction results and the historical spatial relationship graphs of historical video frames, a candidate spatial relationship graph corresponding to the current video frame is generated; wherein, the historical spatial relationship graphs of historical video frames are the spatial relationship graphs corresponding to several video frames preceding the current video frame; the event prediction results are events that cause changes in spatial relationships, and the events are occlusion relationship events, inclusion relationship events, or events with no direct relationship; wherein, the candidate spatial relationship graph is a directed graph structure composed of the spatial relationships between visible objects in the video frame, and each visible object corresponds to a node in the directed graph structure; The quantitative spatial relationship analysis unit is used to analyze multiple candidate spatial relationship graphs corresponding to the video frame based on the trajectory predictor model in the quantitative spatial relationship analyzer model, so as to predict the motion trajectory of invisible objects in the video frame; wherein, the trajectory predictor model is a diffusion model trained with historical video frames and the motion trajectories of each object in the historical video frames as training samples. Based on multiple candidate spatial relationship graphs corresponding to the video frame and the motion trajectory of invisible objects in the video frame, a complete spatial relationship graph corresponding to each object in the video frame is obtained; the probability value corresponding to each of the complete spatial relationship graphs is determined, and the target spatial relationship graph corresponding to the video frame is determined according to the probability value to obtain the corresponding target tracking result; wherein, the invisible objects include objects in the video frame that are occluded or contained by visible objects.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the object spatiotemporal relationship tracking method as described in any one of claims 1 to 5.

8. A processor-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the object spatiotemporal relationship tracking method as described in any one of claims 1 to 5.

Images