Hierarchical time-space semantic extraction method oriented to video understanding

Abstract

The invention relates to the field of digital marketing, and discloses a video understanding-oriented hierarchical spatio-temporal semantic extraction method, which analyzes video content through a three-stage progressive analysis framework: firstly, detecting an object instance in a key frame sequence and constructing a spatio-temporal trajectory to ensure object identity continuity; modeling an object state evolution law by using a recurrent neural network, and quantifying an interaction relationship between objects; and finally, constructing an event causal graph based on the state mutation points, and generating a directed causal chain through a statistical causal test algorithm. Video content is abstracted into event causal logic from pixel features, a semantic path of'track-state-causal 'is formed, structured semantic description of decisions can be directly driven, video semantics and user behavior data are coupled, user preference portraits are dynamically constructed, and personalized material generation is guided.

Description

Technical Field

[0001] This invention relates to the field of digital marketing, and in particular to a hierarchical spatiotemporal semantic extraction method for video understanding. Background Technology

[0002] In the field of digital marketing, especially in advertising delivery scenarios, companies need to quickly analyze massive amounts of advertising video creatives to optimize their delivery strategies. However, existing technologies have significant bottlenecks: traditional video analysis methods rely on frame-by-frame processing or manual sampling, which takes hours to days when dealing with A / B testing creative libraries of thousands of items, failing to meet the real-time requirements of minute-level delivery decisions; at the same time, mainstream solutions focus on object recognition and behavior classification, but cannot analyze the dynamic logic chain of user decisions—for example, they cannot quantify the causal relationship of "the appearance of a discount pop-up leading to a sharp drop in the click-through rate of the product details page," resulting in a lack of targeted optimization strategies.

[0003] While current advertising platforms provide user behavior tracking data, these metrics are disconnected from the video content. Operators can only observe a "30% spike in user bounce rate at 13-15 seconds," but cannot pinpoint specific design flaws in the video. This disconnect between data and semantics causes ad optimization to become blind trial and error, with a large portion of the budget wasted on ineffective iterations. Summary of the Invention

[0004] The purpose of this invention is to provide a hierarchical spatiotemporal semantic extraction method for video understanding. Through keyframe-driven adaptive sampling, object-state-causal three-level semantic extraction, and event causal graph construction, it effectively solves the technical problems existing in the prior art, such as low real-time analysis efficiency, shallow semantic understanding, and the separation of video content and behavioral data, and realizes minute-level deep attribution and precise delivery decision support for advertising videos.

[0005] To achieve the aforementioned technical objectives and effects, this invention discloses a hierarchical spatiotemporal semantic extraction method for video understanding, comprising the following steps:

[0006] S1. Keyframe-driven preprocessing: Adaptive keyframe sampling is performed on the input video stream to generate a frame sequence containing core semantic changes;

[0007] S2. Hierarchical semantic extraction:

[0008] S21. Object-level extraction: Detect object instances in keyframe sequences and construct spatiotemporal trajectories;

[0009] S22. State-level extraction: Analyze the state evolution and interaction relationships in the trajectory of an object;

[0010] S23. Causal-level extraction: Constructing an event causal graph based on state evolution;

[0011] S3. Semantic Reasoning: Analyzes the video logic chain through causal graphs and outputs a structured semantic description.

[0012] Specifically, the adaptive keyframe sampling in step S1 includes:

[0013] Calculate the inter-frame difference between consecutive video frames. When the inter-frame difference exceeds a preset adaptive threshold δ, mark the current frame as a key frame.

[0014] The adaptive threshold δ > 0 and is dynamically adjusted according to the video content.

[0015] The inter-frame difference is calculated using the mean square error, as shown in the following formula:

[0016] ;

[0017] in:

[0018] D(F t ,F t+1 ) represents the t-th frame F t With frame t+1 F t+1 The degree of difference between them;

[0019] W represents the pixel width of the frame, W > 0;

[0020] H represents the pixel height of the frame, H > 0;

[0021] F t (i,j) represents the pixel value or feature value at position (i,j) in frame t;

[0022] F t+1 (i,j) represents the pixel value or feature value at position (i,j) in the (t+1)th frame.

[0023] Preferably, the object detection instance in step S21 is implemented using a deep learning-based object detection model, wherein the model is selected from the YOLO series, Faster R-CNN or SSD model;

[0024] The spatiotemporal trajectory is constructed by associating the detection boxes of the same object in adjacent keyframes. The association condition is that the intersection-union ratio of the object detection boxes in adjacent frames is greater than the threshold ε or the feature similarity is greater than the threshold ζ, where 0 < ε and ζ ≤ 1.

[0025] Preferably, the formula for calculating the crossover-union ratio is as follows:

[0026] ;

[0027] in:

[0028] Bt k This represents the detection box of the k-th object in frame t;

[0029] B t+1 m This represents the detection box of the m-th object in the (t+1)-th frame.

[0030] Preferably, the analysis of state evolution laws in step S22 specifically includes:

[0031] Extract the sequence of appearance, motion, and position features of an object on its spatiotemporal trajectory, input it into a recurrent neural network or its variants for modeling, and learn the temporal change pattern of the object's state.

[0032] The analysis of interaction relationships specifically includes: calculating the spatiotemporal proximity, motion correlation, or interaction characteristics of different object trajectories, and identifying contact, chasing, following, cooperation, or adversarial relationships between objects.

[0033] Preferably, the recurrent neural network is an LSTM, and its state update formula at time step t is as follows:

[0034] ;

[0035] ;

[0036] ;

[0037] ;

[0038] ;

[0039] ;

[0040] in:

[0041] x t This represents the feature vector of the object at time t, with dimension d. x ;

[0042] h t-1 This represents the hidden state at time t-1, with dimension d. h ;

[0043] C t-1 This represents the state of memory cells at time t-1, with dimension d. h ;

[0044] f t This indicates the output of the forget gate, controlling the degree to which historical memories are retained;

[0045] i tThis indicates the input gate output, controlling the degree of updating of the new memory;

[0046] C t 'Indicates the state of candidate memory cells;

[0047] C t This indicates the updated state of memory cells;

[0048] o t Indicates the output of the output gate;

[0049] h t Output the current hidden state;

[0050] W f W i W C W o This is the weight matrix, with dimension d. h ×(d h +d x );

[0051] b f ,b i ,b C ,b o The bias vector has dimension d. h ;

[0052] σ represents the sigmoid activation function; tanh represents the hyperbolic tangent activation function.

[0053] Preferably, the construction of the causal graph in step S23 specifically includes:

[0054] The object state change events and interaction events identified in step S22 are used as nodes;

[0055] The causal probability P between events can be calculated based on the temporal sequence of events, the statistical correlation of state changes, or by using causal discovery algorithms. cause (e i →e j );

[0056] When P cause (e i →e j When ) > η, establish a connection from event node e i Pointing to event node e j The directed edge,

[0057] Where η is the preset causal probability threshold, 0.5 < η ≤ 1.

[0058] Preferably, the causal probability P cause (e i →e jGranger causality analysis was used for calculation, and the formula is as follows:

[0059] ;

[0060] in:

[0061] F test Based on the comparison of events e using a vector autoregressive model i Historical information and predicted events e without this information j The significance test statistic for future states;

[0062] p value This is the p-value corresponding to the F-test;

[0063] P cause (e i →e j The range of ) is [0,1].

[0064] The structured semantic description in step S3 includes at least the following:

[0065] The main objects in the video, the key state change sequences of the objects, the main interaction events between the objects, the causal logic chain between the events, and decision suggestions based on causal relationships.

[0066] The present invention has the following beneficial effects:

[0067] 1. This invention compresses 90% of invalid frames through an adaptive keyframe sampling mechanism and combines a three-level semantic extraction architecture of object → state → causality to achieve deep video analysis at the second level, breaking through the bottleneck of minute-level latency in traditional frame-by-frame analysis; at the same time, the constructed event causal graph penetrates the surface of behavior, locates the root cause nodes of problems, and solves the problem of shallow semantic understanding.

[0068] 2. This application not only outputs structured semantic descriptions that directly drive ad delivery decisions, but also constructs a spatiotemporally aligned multimodal fusion analysis framework by coupling video semantics with user behavior tracking data, achieving powerful dynamic profile generation and iteration capabilities. Based on LSTM modeling of user response sequences to video elements and combined with event triggering relationships in a causal graph, the system automatically classifies user decision types. By matching historical causal patterns in real time, the system dynamically updates profile tag confidence, forming a continuously evolving user cognitive model, and finally outputs spatiotemporally customized suggestions based on profile features. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments.

[0070] Example 1

[0071] This embodiment provides a hierarchical spatiotemporal semantic extraction method for video understanding, including the following steps:

[0072] S1. Keyframe-driven preprocessing: Adaptive keyframe sampling is performed on the input video stream to generate a frame sequence containing core semantic changes;

[0073] Specifically, the adaptive keyframe sampling in step S1 includes:

[0074] Calculate the inter-frame difference between consecutive video frames. When the inter-frame difference exceeds a preset adaptive threshold δ, mark the current frame as a key frame.

[0075] The adaptive threshold δ > 0 and is dynamically adjusted according to the video content.

[0076] The inter-frame difference is calculated using the mean square error, as shown in the following formula:

[0077] ;

[0078] D(F t ,F t+1 ) represents frame t, F t With frame t+1 F t+1 The difference between them; W represents the pixel width of the frame, W > 0; H represents the pixel height of the frame, H > 0; F t (i,j) represents the pixel value or feature value at position (i,j) in frame t; F t+1 (i,j) represents the pixel value or feature value at position (i,j) in the (t+1)th frame.

[0079] The core design of this step lies in dynamically perceiving semantic transitions in video and capturing nodes of sudden content changes through an adaptive threshold mechanism. Specifically, it employs inter-frame difference calculation and sets a dynamic threshold δ to filter static redundant frames. By linking the threshold with content complexity, when a sudden increase in the motion entropy of an object in the frame is detected, δ is automatically reduced to 0.1 to increase the sampling density; conversely, for static display scenes, δ is increased to 0.2 to reduce computation, achieving 90% compression of invalid frames while retaining core semantic transition frames.

[0080] S2. Hierarchical semantic extraction includes the following steps:

[0081] S21. Object-level extraction: Detect object instances in the keyframe sequence and construct spatiotemporal trajectories; wherein, the object instance detection in step S21 is implemented using a deep learning-based object detection model, and the model is selected from YOLO series, Faster R-CNN or SSD model;

[0082] The spatiotemporal trajectory is constructed by associating the detection boxes of the same object in adjacent keyframes. The association condition is that the intersection-union ratio of the object detection boxes in adjacent frames is greater than the threshold ε or the feature similarity is greater than the threshold ζ, where 0 < ε and ζ ≤ 1.

[0083] The formula for calculating the intersection-union ratio is as follows:

[0084] ;

[0085] in:

[0086] B t k This represents the detection box of the k-th object in frame t;

[0087] B t+1 m This represents the detection box of the m-th object in the (t+1)-th frame.

[0088] S22. State-level extraction: Analyzing the state evolution patterns and interaction relationships in the object's trajectory; the analysis of state evolution patterns in step S22 specifically includes:

[0089] Extract the sequence of appearance, motion, and position features of an object on its spatiotemporal trajectory, input it into a recurrent neural network or its variants for modeling, and learn the temporal change pattern of the object's state.

[0090] The analysis of interaction relationships specifically includes: calculating the spatiotemporal proximity, motion correlation, or interaction characteristics of different object trajectories, and identifying contact, chasing, following, cooperation, or adversarial relationships between objects.

[0091] The recurrent neural network is an LSTM, and its state update formula at time step t is as follows:

[0092] ;

[0093] ;

[0094] ;

[0095] ;

[0096] ;

[0097] ;

[0098] in:

[0099] x t This represents the feature vector of the object at time t, with dimension d. x h t-1This represents the hidden state at time t-1, with dimension d. h C t-1 This represents the state of memory cells at time t-1, with dimension d. h ;ft represents the forget gate output, controlling the degree to which historical memories are retained; i t This represents the input gate output, controlling the degree of update of the new memory; C t 'Indicates the state of candidate memory cells; C t Indicates the updated state of memory cells; o t Indicates the output gate output; h t Indicates the current hidden state output; W f W i W C W o This is the weight matrix, with dimension d. h ×(d h +d x );b f ,b i ,b C ,b o The bias vector has dimension d. h σ represents the sigmoid activation function; tanh represents the hyperbolic tangent activation function.

[0100] S23. Causal level extraction: Constructing an event causal graph based on state evolution; wherein, the construction of the causal graph in step S23 specifically includes:

[0101] The object state change events and interaction events identified in step S22 are used as nodes;

[0102] The causal probability P between events can be calculated based on the temporal sequence of events, the statistical correlation of state changes, or by using causal discovery algorithms. cause (e i →e j );

[0103] When P cause (e i →e j When ) > η, establish a connection from event node e i Pointing to event node e j The directed edge,

[0104] Where η is the preset causal probability threshold, 0.5 < η ≤ 1.

[0105] Preferably, the causal probability P cause (e i →e j Granger causality analysis was used for calculation, and the formula is as follows:

[0106] ;

[0107] in:

[0108] F test Based on the comparison of events e using a vector autoregressive model i Historical information and predicted events e without this information j The significance test statistic for future states;

[0109] p value This is the p-value corresponding to the F-test;

[0110] P cause (e i →e j The range of ) is [0,1].

[0111] This step achieves a semantic leap from pixel features to event causality through a three-level progressive abstraction. First, object instances are detected and spatiotemporal trajectories are constructed based on keyframe sequences, using a dual correlation condition of intersection-union (IU) threshold and feature similarity threshold to ensure the continuity of object identity. Next, a recurrent neural network is used to model the temporal evolution of the object's dynamic attributes, capturing continuous patterns of state changes; simultaneously, interaction relationships are defined by quantifying spatial proximity and motion correlation, transforming discrete trajectories into behavioral semantic units. Finally, a causal graph is constructed based on state-abrupt events, and a statistical causal testing algorithm is used to calculate the causal probability between events. Directed causal edges are generated according to preset probability thresholds, completing the connected understanding from microscopic motion to macroscopic logic.

[0112] S3. Semantic Reasoning: Analyze the video logic chain using a causal graph to output a structured semantic description. The structured semantic description in step S3 includes at least:

[0113] The main objects in the video, the key state change sequences of the objects, the main interaction events between the objects, the causal logic chain between the events, and decision suggestions based on causal relationships.

[0114] This step transforms causal graphs into decision knowledge, converting complex event chains into actionable suggestions. It generates natural language descriptions by extracting key nodes, state transition points, and edge weights from the causal graph and combining them with a domain knowledge base.

[0115] Example 2

[0116] Based on the technology of Example 1, this embodiment uses a specific video as an example to illustrate the application. This embodiment addresses a typical problem in e-commerce advertising scenarios: In short video ads where celebrities / live streamers hold and display products, the position, size, and angle of the product in the frame change dynamically with the demonstration actions, such as a celebrity holding and rotating a lipstick. If the discount pop-up appears in a fixed position and size, it is very easy to obscure the core area of ​​the product, causing users to lose access to key information.

[0117] The input data is a 15-second beauty advertisement video with a resolution of 1080P and a frame rate of 30 frames per second. The video content contains four time segments: the initial 0-3 seconds feature a close-up of a celebrity holding and rotating a lipstick; 4-8 seconds demonstrate the lipstick application effect; 9-12 seconds show a semi-transparent discount pop-up window in the lower right corner, with the transparency gradually increasing from 0% to 80%; 13-15 seconds show a countdown bar at the bottom.

[0118] Step S1: Keyframe-driven preprocessing performs adaptive sampling. Calculate the mean square error difference between consecutive frames:

[0119] ;

[0120] Among them, F t (i,j) represents the RGB pixel value at position (i,j) in frame t, with video resolution W=1920, H=1080. An adaptive threshold δ=0.15 is set; keyframes are marked when the difference exceeds this threshold. The output consists of four key content segments, whose temporal division is completely consistent with the input description.

[0121] Step S2, hierarchical semantic extraction, is divided into three progressive processing stages. First, in the object-level extraction stage, YOLOv is used to detect celebrity faces, lipstick objects, discount pop-ups, and countdown bar instances in keyframes. The spatial overlap of object detection boxes in adjacent frames is calculated using the intersection-over-union (IoU) formula.

[0122] ;

[0123] When the crossover ratio is greater than 0.6, the output shows the rotation trajectory of the lipstick from 0 seconds to 12 seconds and the positioning trajectory of the discount pop-up in the lower right corner from 9 to 12 seconds.

[0124] Then, state-level extraction is performed, extracting the opacity and center coordinates of the discounted pop-up as temporal features input to the LSTM network: the forget gate controls the degree of retention of historical memories through the sigmoid function, the input gate adjusts the intensity of new memory updates, candidate memories generate temporary states through hyperbolic tangent transformation, memory cells fuse forgetting and input gate outputs, and the output gate filters the final hidden states. This network models the linear growth process of the pop-up opacity from 0% at 9 seconds to 80% at 12 seconds. Simultaneously, the spatial overlap rate between the lipstick and the pop-up is calculated; when this value exceeds 60%, an occlusion event is triggered.

[0125] Finally, causal extraction is performed to construct a causal graph centered on event nodes. Event e1 is defined as the pop-up's transparency increasing to 50%, event e2 as the lipstick being obscured, and event e3 as the user bounce rate spikeing. The Granger causality test formula is used to calculate the causal probability.

[0126] ;

[0127] Step S3 involves semantic reasoning, parsing the causal graph to generate a structured description. Core objects include lipstick, discount pop-up, and countdown bar; the key event chain covers three nodes: lipstick rotation display, pop-up gradually obscuring the product, and a surge in user bounce rate; attribution analysis indicates that a pop-up design conflict caused the product display to be interrupted; design recommendations specify that the pop-up size should be reduced to 5% of the screen size, the maximum transparency should be adjusted to 50%, and dynamic position tracking should be implemented.

[0128] In the causal extraction stage, a key event is defined based on the state evolution output: when the spatial overlap between the lipstick and the pop-up exceeds 60%, a surge in user bounce rate is triggered. The Granger causality test formula is used to directly verify the causality of e2→e3: a vector autoregression (VAR) model is established to analyze the predictive ability of historical information from e2 on the future value of e3. Based on this, a simplified causal chain is generated: "pop-up obscuring product → user bounces". The pop-up transparency increasing to 50% is used as a state warning condition for the e2 event; since it is not a direct cause of the outcome event, it is not included in the causal graph nodes.

[0129] This embodiment constructs a simplified causal chain of "state event → behavioral response," forming a rapidly transferable single-cause-single-effect analysis paradigm. Its core lies in abstracting hierarchical semantic extraction into a general causal attribution engine—when a specific state event is detected, the system automatically associates it with sudden changes in user behavior, outputting high-confidence attribution conclusions and operational suggestions. This allows publishers to deploy targeted optimization solutions within minutes, achieving significant improvements in campaign performance without complex parameter tuning. The framework can be seamlessly extended to scenarios such as industrial quality inspection and intelligent transportation, providing plug-and-play causal decision support across various domains.

[0130] Example 3

[0131] This embodiment, building upon the technology of Embodiment 1, uses a specific video as an example to illustrate its application. This embodiment addresses the issue of differentiated user responses to price information in live-streaming e-commerce scenarios. In live-streaming video advertising, user decision-making patterns exhibit significant differences: some users click immediately upon seeing a price tag, while others require a deeper understanding of product features before converting. Traditional analysis methods relying on post-event questionnaires cannot link video content and user behavior in real time. This solution achieves dynamic preference modeling through hierarchical semantic extraction.

[0132] The input data is a 30-second live-stream video of a home appliance, with a resolution of 1080P and a frame rate of 30 frames per second. The video content contains four time segments: 0-8 seconds: the host demonstrates the functions of an air fryer without any price information; 9-15 seconds: a "¥399" price tag is continuously displayed in the upper left corner; 16-22 seconds: a "Limited-time discount of ¥100" discount bar with 50% transparency pops up at the bottom; 23-30 seconds: a comparative experiment showing this product with competing products.

[0133] Step S1: Keyframe-driven preprocessing performs color feature-based sampling. The mean squared error difference between consecutive frames is calculated.

[0134] ;

[0135] Among them, F t (i,j) represents the RGB pixel value at position (i,j) in frame t, with video resolution W=1920, H=1080. An adaptive threshold δ=0.12 is set to adapt to product close-up switching scenes; keyframes are marked when the difference exceeds the threshold. The output consists of four key content segments, whose temporal division is completely consistent with the input description.

[0136] Step S2 involves hierarchical semantic extraction, achieving a three-level abstraction leap. First, in the object-level extraction stage, Faster R-CNN is used to detect the air fryer itself, price tags, discount strips, and competitor packaging instances. The spatial overlap of objects in adjacent frames is calculated using the intersection-over-union (IoU) formula.

[0137] ;

[0138] When the crossover ratio is greater than 0.8, the associated trajectory is output, showing the price tag fixed in the upper left corner for 9-15 seconds and the discount bar sliding in from the bottom for 16-22 seconds.

[0139] Subsequently, state-level extraction is performed, extracting the price tag display state as a temporal feature input to the LSTM network. A state update mechanism is implemented: a forgetting gate controls the retention of historical memories, the input gate adjusts the intensity of new memory updates, candidate memories are transformed using a hyperbolic tangent transform to generate temporary states, memory cells are fused with the outputs of the forgetting and input gates, and finally, the output gate filters out hidden states. This network models an association pattern where the click-through rate instantly increases by 15% when the price tag appears. Two types of interaction behaviors are defined: clicking the details button immediately when the price tag is displayed is marked as an immediate click event, and clicking after watching a comparison experiment for more than 5 seconds after the discount bar appears is marked as a hesitant price comparison event.

[0140] Finally, causal extraction is performed to construct a causal graph centered on price events. The display of the price tag is defined as event e1, immediate user click on details as event e2, discount bar transparency exceeding 50% as event e3, and delayed user click as event e4. The Granger causality test formula is used to calculate the causal probability:

[0141] ;

[0142] In step S3, semantic reasoning stage, structured user segmentation tags are generated. The core characteristic of high price-sensitive users is that they click on the details page within 3 seconds after the price tag appears. This group accounts for 38% and the attribution confidence level is 99.97%. The corresponding operation strategy should highlight the discount amount on the first screen to strengthen the conversion incentive. Discount-induced users rely on the dual drive of high transparency discount prompts and comparison with competitors. They account for 27% and the confidence level is 93%.

[0143] In the causal extraction stage, event e3 serves as the pivotal point in the user's decision-making path, capturing the visual transition of promotional information from latent to explicit. When the price tag directly triggers an immediate click, a short causal chain e1→e2 is formed for highly sensitive users. Conversely, when the transparency of the discount bar exceeds 50%, the probability of shifting user attention from product features to discount conditions increases by 47%. This shift, by extending the decision-making lag by an average of 8.2 seconds, guides users into the comparative experiment viewing stage and ultimately triggers a delayed click, forming e1→e3→e4. Granger quantification shows that the causal strength from e3 to e4 reaches 93% confidence, while the weak causality of e1→e4 is only 62%, further demonstrating e3's pivotal role in reshaping the decision-making path. This mechanism ultimately supports a two-tiered operational system that directly converts price-sensitive users and facilitates in-depth decision-making for discount-induced users, achieving a technical closed loop from causal attribution to business strategy.

[0144] This embodiment constructs a composite causal graph of "multi-state events → behavioral response network" to form a decision support paradigm capable of resolving concurrent factors. Its core lies in extending hierarchical semantic extraction into a multi-path attribution engine—when multiple events occur concurrently, such as price tag display (e1), discount bar transparency exceeding a threshold (e3), and competitor comparison demonstration initiation, the system quantifies the independent contribution of each event to user behavior through Granger causality testing and identifies synergistic effects between events. It outputs different user type decisions: for price-sensitive users, it emphasizes the discount amount on the first screen; for discount-inducing users, it extends the competitor comparison time to 8 seconds; and for feature-oriented users, it compresses the lead time for price information. This framework can also be extended to areas such as online course focus analysis and athlete training videos.

[0145] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A hierarchical spatiotemporal semantic extraction method for video understanding, characterized in that, Includes the following steps: S1. Keyframe-driven preprocessing: Adaptive keyframe sampling is performed on the input video stream to generate a frame sequence containing core semantic changes; S2. Hierarchical semantic extraction: S21. Object-level extraction: Detect object instances in keyframe sequences and construct spatiotemporal trajectories; S22. State-level extraction: Analyze the state evolution and interaction relationships in the trajectory of an object; S23. Causal-level extraction: Constructing an event causal graph based on state evolution; S3. Semantic Reasoning: Analyzes the video logic chain through causal graphs and outputs a structured semantic description.

2. The hierarchical spatiotemporal semantic extraction method for video understanding according to claim 1, characterized in that, The adaptive keyframe sampling in step S1 specifically includes: Calculate the inter-frame difference between consecutive video frames. When the inter-frame difference exceeds a preset adaptive threshold δ, mark the current frame as a key frame. The adaptive threshold δ > 0 and is dynamically adjusted according to the video content.

3. The hierarchical spatiotemporal semantic extraction method for video understanding according to claim 1, characterized in that, The inter-frame difference is calculated using the mean squared error, as shown in the following formula: ； in: D(F t ,F t+1 ) represents frame t, F t With frame t+1 F t+1 The degree of difference between them; W represents the pixel width of the frame, W > 0; H represents the pixel height of the frame, H > 0; F t (i,j) represents the pixel value or feature value at position (i,j) in frame t; F t+1 (i,j) represents the pixel value or feature value at position (i,j) in the (t+1)th frame.

4. The hierarchical spatiotemporal semantic extraction method for video understanding according to claim 1, characterized in that, The detected object instances in step S21 are implemented using a deep learning-based object detection model, which is selected from the YOLO series, Faster R-CNN or SSD model. The spatiotemporal trajectory is constructed by associating the detection boxes of the same object in adjacent keyframes. The association condition is that the intersection-union ratio of the object detection boxes in adjacent frames is greater than the threshold ε or the feature similarity is greater than the threshold ζ, where 0 < ε and ζ ≤ 1.

5. A hierarchical spatiotemporal semantic extraction method for video understanding according to claim 4, characterized in that, The formula for calculating the crossover-union ratio is as follows: ； in: B t k This represents the detection box of the k-th object in frame t; B t+1 m This represents the detection box of the m-th object in the (t+1)-th frame.

6. The hierarchical spatiotemporal semantic extraction method for video understanding according to claim 1, characterized in that, The analysis of state evolution laws in step S22 specifically includes: Extract the sequence of appearance, motion, and position features of an object on its spatiotemporal trajectory, input it into a recurrent neural network or its variants for modeling, and learn the temporal change pattern of the object's state. The analysis of interaction relationships specifically includes: calculating the spatiotemporal proximity, motion correlation, or interaction characteristics of different object trajectories, and identifying contact, chasing, following, cooperation, or adversarial relationships between objects.

7. A hierarchical spatiotemporal semantic extraction method for video understanding according to claim 6, characterized in that, The recurrent neural network is an LSTM, and its state update formula at time step t is as follows: ； ； ； ； ； ； in: x t This represents the feature vector of the object at time t, with dimension d. x ; h t-1 This represents the hidden state at time t-1, with dimension d. h ; C t-1 This represents the state of memory cells at time t-1, with dimension d. h ; f t This indicates the output of the forget gate, controlling the degree to which historical memories are retained; i t This indicates the input gate output, controlling the degree of updating of the new memory; C t 'Indicates the state of candidate memory cells; C t This indicates the updated state of memory cells; o t Indicates the output of the output gate; h t Output the current hidden state; W f W i W C W o This is the weight matrix, with dimension d. h ×(d h +d x ); b f ,b i ,b C ,b o The bias vector has dimension d. h ; σ represents the sigmoid activation function; tanh represents the hyperbolic tangent activation function.

8. The hierarchical spatiotemporal semantic extraction method for video understanding according to claim 1, characterized in that, The construction of the cause-effect graph in step S23 specifically includes: The object state change events and interaction events identified in step S22 are used as nodes; The causal probability P between events can be calculated based on the temporal sequence of events, the statistical correlation of state changes, or by using causal discovery algorithms. cause (e i →e j ); When P cause (e i →e j When ) > η, establish a connection from event node e. i Pointing to event node e j The directed edge, Where η is the preset causal probability threshold, 0.5 < η ≤ 1.

9. A hierarchical spatiotemporal semantic extraction method for video understanding according to claim 8, characterized in that, The causal probability P cause (e i →e j Granger causality analysis was used for calculation, and the formula is as follows: ； in: F test Based on the comparison of events e using a vector autoregressive model i Historical information and predicted events e without this information j The significance test statistic for future states; p value This is the p-value corresponding to the F-test; P cause (e i →e j The range of ) is [0,1].

10. A hierarchical spatiotemporal semantic extraction method for video understanding according to claim 1, characterized in that, The structured semantic description in step S3 includes at least: The main objects in the video, the key state change sequences of the objects, the main interaction events between the objects, the causal logic chain between the events, and decision suggestions based on causal relationships.

Images