Multi-modal video knowledge query system combined with agent task intent

Abstract

This invention provides a multimodal video knowledge query system that integrates the task intent of an intelligent agent, relating to the field of multimodal fusion technology. It includes: a feature extraction module for receiving video clips and text questions; processing the video frame sequence using a lightweight 3D convolutional network to generate video temporal feature vectors; performing semantic analysis on the text questions to generate text semantic feature vectors representing the semantics of the questions; and a mapping module for resolving the correspondence between the spatial coordinates of video keyframes and the text semantics based on the correlation between the video temporal feature vectors and the text semantic feature vectors, constructing a dynamic spatial semantic mapping representation, and determining an initial spatial reference position based on the mapping representation. This invention, through multimodal feature fusion, dynamic spatiotemporal constraints, and external knowledge integration, captures the task intent of an intelligent agent, generates comprehensive and scenario-appropriate answers, and outputs them via voice, improving the accuracy, efficiency, and interactivity of multimodal video knowledge queries.

Description

Technical Field

[0001] This invention relates to the field of multimodal fusion technology, and in particular to a multimodal video knowledge query system that combines the task intent of an intelligent agent. Background Technology

[0002] Traditional video question answering technology mainly revolves around cross-modal alignment and understanding of video visual information and user natural language questions. Its typical process includes: extracting visual features of key frames or segments of the video through deep convolutional networks, extracting semantic features of the question text with the help of pre-trained language models, and then fusing multimodal features to predict the answer.

[0003] However, there may be room for improvement in the practical application of these technologies, with one of the most prominent areas being spatiotemporal information modeling. For example, when capturing complex spatiotemporal dynamic changes of objects in videos (such as motion trajectories and interaction sequences), existing temporal pooling or attention mechanisms sometimes fall short of being sufficiently precise, which may affect the understanding of issues involving motion, action sequences, or spatiotemporal relationships. Summary of the Invention

[0004] The technical problem to be solved by this invention is to provide a multimodal video knowledge query system that combines the task intent of an intelligent agent, thereby improving the overall accuracy of multimodal video knowledge query.

[0005] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows: Firstly, a multimodal video knowledge query system that integrates the task intent of intelligent agents includes: The feature extraction module receives video clips and text questions, processes the video frame sequence using a lightweight 3D convolutional network to generate video temporal feature vectors, and performs semantic analysis on the text questions to generate text semantic feature vectors representing the semantics of the questions. The mapping module is used to analyze the correspondence between the spatial coordinates of video keyframes and the semantics of text based on the correlation between the video temporal feature vector and the text semantic feature vector, construct a dynamic spatial semantic mapping representation, and determine an initial spatial reference position based on the mapping representation. The region construction module is used to construct a directional dynamic spatiotemporal detection region along the main direction of motion of the video object, starting from the initial spatial reference position. The spatiotemporal constraint generation module is used to select internal and external temporal position points within and outside the dynamic spatiotemporal detection area; based on the time series of the internal and external temporal position points, a temporal constraint path is formed to obtain the spatiotemporal constraint factor; The knowledge fusion module is used to fuse text semantic feature vectors with spatiotemporal constraint factors to output task intent vectors; it constructs structured query statements based on task intent vectors, retrieves relevant entity information from external knowledge bases and embeds it as knowledge feature vectors; The global fusion module is used to fuse video temporal feature vectors, spatiotemporal constraint factors, and knowledge feature vectors to generate a global fusion feature vector and predict the answer, and then convert the predicted answer into speech output.

[0006] In a second aspect, a computer-readable storage medium storing a program that, when executed by a processor, implements the system.

[0007] The above-described solution of the present invention has at least the following beneficial effects: The construction of dynamic spatial semantic mapping representation, combined with the motion displacement of video objects, makes the correspondence between text semantics and the spatial coordinates of video keyframes more closely match the dynamic changes of the actual scene. The determination of the initial spatial reference position is also more targeted, improving the accuracy of spatial positioning. The dynamic spatiotemporal detection region constructed along the main direction of motion, starting from the initial reference, can effectively focus on the core spatiotemporal range related to the task in the video, reducing interference from irrelevant information. The temporal constraint path integrates temporal position points inside and outside the region, motion continuity features, and time decay features, enabling the spatiotemporal constraint factors to more reasonably reflect the temporal logic and time weight of the video content, providing strong support for the parsing of task intent.

[0008] The entity information from the external knowledge base, combined with textual semantics and spatiotemporal constraints to generate a task intent vector, guides knowledge retrieval. This allows the system to not only rely on the information in the video itself but also integrate external knowledge, making the answers more comprehensive and in-depth. By integrating video temporal features, spatiotemporal constraint factors, and knowledge features, the fusion of multi-dimensional information makes the globally fused feature vector more representative. Compared with the problems of one-sided information fusion or insufficient correlation of modal features often found in existing technologies, this further enhances the completeness of feature expression and the correlation between multimodal information, thereby improving the accuracy of answer prediction. Attached Figure Description

[0009] Figure 1 This is a schematic diagram of a multimodal video knowledge query system that combines the task intent of an intelligent agent, provided by an embodiment of the present invention.

[0010] Figure 2 This is a schematic diagram of the process provided by an embodiment of the present invention, which integrates video temporal feature vectors, spatiotemporal constraint factors, and knowledge feature vectors to generate a global fusion feature vector and predict the answer, and then converts the predicted answer into speech output. Detailed Implementation

[0011] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0012] like Figure 1 As shown, embodiments of the present invention propose a multimodal video knowledge query system that combines the task intent of an intelligent agent, including: The feature extraction module receives video clips and text questions, processes the video frame sequence using a lightweight 3D convolutional network to generate video temporal feature vectors, and performs semantic analysis on the text questions to generate text semantic feature vectors representing the semantics of the questions. The mapping module is used to analyze the correspondence between the spatial coordinates of video keyframes and the semantics of text based on the correlation between the video temporal feature vector and the text semantic feature vector, construct a dynamic spatial semantic mapping representation, and determine an initial spatial reference position based on the mapping representation. The region construction module is used to construct a directional dynamic spatiotemporal detection region along the main direction of motion of the video object, starting from the initial spatial reference position. The spatiotemporal constraint generation module is used to select internal and external temporal position points within and outside the dynamic spatiotemporal detection area; based on the time series of the internal and external temporal position points, a temporal constraint path is formed to obtain the spatiotemporal constraint factor; The knowledge fusion module is used to fuse text semantic feature vectors with spatiotemporal constraint factors to output task intent vectors; it constructs structured query statements based on task intent vectors, retrieves relevant entity information from external knowledge bases and embeds it as knowledge feature vectors; The global fusion module is used to fuse video temporal feature vectors, spatiotemporal constraint factors, and knowledge feature vectors to generate a global fusion feature vector and predict the answer, and then convert the predicted answer into speech output.

[0013] In this embodiment of the invention, the construction of the dynamic spatial semantic mapping representation incorporates the motion displacement of video objects, making the correspondence between text semantics and the spatial coordinates of video keyframes more closely match the dynamic changes of the actual scene. The determination of the initial spatial reference position is also more targeted, improving the accuracy of spatial positioning. The dynamic spatiotemporal detection region constructed along the main direction of motion, starting from the initial reference, can effectively focus on the core spatiotemporal range related to the task in the video, reducing interference from irrelevant information. The temporal constraint path integrates temporal position points inside and outside the region, motion continuity features, and time decay features, enabling the spatiotemporal constraint factors to more reasonably reflect the temporal logic and time weight of the video content, providing strong support for the parsing of task intent.

[0014] The entity information from the external knowledge base, combined with textual semantics and spatiotemporal constraints to generate a task intent vector, guides knowledge retrieval. This allows the system to not only rely on the information in the video itself but also integrate external knowledge, making the answers more comprehensive and in-depth. By integrating video temporal features, spatiotemporal constraint factors, and knowledge features, the fusion of multi-dimensional information makes the globally fused feature vector more representative. Compared with the problems of one-sided information fusion or insufficient correlation of modal features often found in existing technologies, this further enhances the completeness of feature expression and the correlation between multimodal information, thereby improving the accuracy of answer prediction.

[0015] In a preferred embodiment of the present invention, a video segment and a text question are received; the video frame sequence is processed by a lightweight 3D convolutional network to generate a video temporal feature vector; semantic analysis is performed on the text question to generate a text semantic feature vector representing the semantics of the question, which may include: Video features are extracted hierarchically using a lightweight 3D convolutional network. The first convolutional layer captures local spatiotemporal correlation features, and the second convolutional layer extracts global temporal correlation features. After compression in the time dimension, the video temporal feature vector is output. The text problem is lexicalized and mapped into a sequence of word vectors through an embedding layer; A bidirectional gated cyclic unit is used to process word vector sequences, where forward encoding generates sequential semantic vectors and backward encoding generates reverse semantic vectors. Concatenate sequential semantic vectors and reverse semantic vectors to generate text semantic feature vectors.

[0016] In this embodiment of the invention, the first convolutional layer captures local spatiotemporal correlation features. The input is a video frame sequence of “30×224×224×3” (30 frames, 224×224 pixels, RGB three channels). The core extracts local spatiotemporal features using a 3×3×3 3D convolutional kernel. The details of the weight parameters involved are as follows: Kernel weights and biases: Each 3D convolutional kernel contains 27 weight parameters (corresponding to 3 time × 3 height × 3 width) and 1 bias parameter.

[0017] The initial values ​​of the weight parameters are strictly limited to the range of [-0.01, 0.01]. For example, the weight corresponding to the red channel in the first row and first column of the first frame might be 0.002, the weight corresponding to the green channel in the second row and second column of the second frame might be -0.003, and the weight corresponding to the blue channel in the third row and third column of the third frame might be 0.001. This range ensures that the initial feature extraction will not ignore details or cause gradient explosion due to excessively large weights.

[0018] The initial value of the bias parameter is fixed in [-0.001, 0.001]. For example, the bias of a certain convolution kernel is 0.0005 or -0.0002. This is used to fine-tune the overall offset of the feature values ​​and avoid the zero mean from deviating too much.

[0019] Details of local feature calculation: When the convolution kernel covers the "frames 1-3, image (1-3, 1-3) pixel region", it multiplies the 81 pixel values ​​(3 frames × 3 × 3 pixels × 3 channels) within that region with the corresponding weights. For example: The red channel value of pixel (1,1) in frame 1 (assumed to be 255) × weight 0.002 → 0.51; The green channel value of pixel (2,2) in frame 2 (assuming it is 100) × weight -0.003 → -0.3; The summation of all 81 product results is -5.2. Adding a bias of 0.0005, the final local spatiotemporal feature value is -5.1995. The 32 convolutional kernels generate 32 feature maps of 28×222×222. The feature value range of each map varies depending on the weights and input, and is within [-10, 10] (determined by multiplying the pixel value 0-255 with the small weights).

[0020] Second convolutional layer: Extracts global temporal correlation features. The input consists of 32 local feature maps of 28×222×222 each. Global temporal features are integrated using a 5×5×5 convolutional kernel. Weight details are as follows: Kernel weights and biases: Each convolutional kernel contains 4000 weight parameters (32 input images × 5×5×5 windows) and 1 bias parameter. The initial values ​​of the weight parameters range from [-0.02, 0.02], slightly larger than the first layer (to integrate more local features). For example, the weight corresponding to the 3rd row and 4th column of the 2nd frame of the 1st input image is 0.015, and the weight corresponding to the 1st row and 2nd column of the 4th frame of the 10th input image is -0.012; the bias parameter is still within the range of [-0.001, 0.001], for example, the bias of a certain convolutional kernel is -0.0003.

[0021] Details of global feature integration: When the convolution kernel covers the "frames 3-7, image (5-9, 5-9) pixel region", it is necessary to process the feature values ​​of the corresponding region in 32 input images (assuming the feature value of this region is 2.5 in the first image and -1.8 in the tenth image): The eigenvalue of the first image is 2.5 × weight 0.015 → 0.0375; The eigenvalue of the 10th image is reduced by 1.8 × weight and reduced by 0.012 to 0.0216. The sum of all 4000 products is 8.3. With a bias of -0.0003, the final global feature value is 8.2997. The 16 convolutional kernels generate 16 feature maps of 12×109×109, and the feature value range is expanded to [-50, 50] (due to the integration and accumulation of multiple maps).

[0022] Time dimension compression and vector generation: Temporal average pooling: For each 12×109×109 feature map, the 12 temporal feature values ​​at each spatial location (109×109) are averaged. For example, if the 12 values ​​at a certain location are [3.2, 4.1, ..., 2.8], the average value is 3.5, and the value at that location is 3.5 after compression.

[0023] Spatial dimension max pooling: takes the maximum value in a 109×109 spatial dimension. For example, the maximum value of a feature map is 8.9, and another is -7.2.

[0024] The final 16-dimensional video temporal feature vector has a value range of [-50, 50] for each dimension (obtained by global feature pooling), for example, the vector is [8.9, -7.2, 5.3, ..., 2.1].

[0025] The process of generating text semantic feature vectors: The word vector matrix is ​​10000×128 (10000 words, 128 dimensions), and the weights (word vector values) are detailed below: The dimension values ​​of each word vector are limited to the range [-1, 1]. For example: The word vector for "video" is [0.12, -0.35, 0.08, ..., 0.29] (128 dimensions, each value is between -1 and 1). The word vector for "red" could be [0.67, -0.02, -0.41, ..., 0.81]; This range ensures that the cosine distance between semantically similar word vectors (such as "car" and "sedan") is in the range of [0.5, 0.8] (the smaller the distance, the closer the semantics).

[0026] Bidirectional gated recurrent unit (Bi-GRU) processes word vector sequences: Weight matrices (Wr, Ur, Wz, Uz, W, U): The elements of each matrix range from [-0.1, 0.1] (gradient stability is ensured through Xavier initialization). For example: The element in row 5, column 10 of Wr (reset gate input weights) is 0.03, and the element in row 12, column 20 may be -0.05; The element in the 3rd row and 8th column of Ur (reset door hidden state weight) is 0.07, and the element in the 9th row and 15th column may be -0.02; All matrices are 128×128 in dimension (because word vectors and hidden states are both 128-dimensional).

[0027] Bias parameters (br, bz, b): The initial value range for each bias parameter is [-0.01, 0.01], for example: Reset the gate bias br to 0.005, update the gate bias bz to -0.003, and set the candidate state bias b to 0.001.

[0028] Gating output value range: The reset gate (r) and update gate (z) are calculated using the sigmoid function, and the output value is in the range [0, 1]. For example: when processing "video", r1=0.2 (meaning 20% ​​of the historical hidden state is retained), z1=0.8 (meaning 80% of the new information is used for updating); when processing "red", r2=0.3 (30% of h1 is retained), z2=0.7 (h2 is updated with 70% of the new information).

[0029] Hidden state value range: The candidate hidden state h' is calculated using the tanh function, with a value range of [-1, 1]; the final hidden state h (h1, h2, etc.) is calculated by z×h'+(1-z)×h_prev (the hidden state at the previous time step), and its value range is also [-1, 1]. For example: h1 (the hidden state of "video") is [0.2, -0.1, 0.3, ..., 0.5]; h5 (the forward final state) may be [0.1, 0.3, -0.2, ..., 0.5] (128 dimensions, each value between -1 and 1).

[0030] Concatenate to generate text semantic feature vectors: The forward hidden state h5 (128 dimensions, value range [-1, 1]) and the backward hidden state h5' (128 dimensions, value range [-1, 1]) are concatenated to form a 256-dimensional vector. For example: the first 128 bits: [0.1, 0.3, -0.2, ..., 0.5]; the last 128 bits: [0.2, -0.4, 0.6, ..., 0.3]; the value range of each dimension of the entire vector is [-1, 1], fully representing bidirectional semantics.

[0031] Lightweight networks and temporal compression reduce redundant computation, improving processing efficiency while ensuring feature quality. Lexicalization enables structured text decomposition, and the embedding layer converts text into semantic vectors, solving the adaptation problem between text symbols and numerical calculations. Bidirectional GRU captures bidirectional context, avoiding semantic loss caused by unidirectional processing (e.g., the negation semantics of "no" in "I don't like" needs to be combined with the context), making text semantic features more accurate and laying a reliable foundation for cross-modal association. The generation process of both video and text features is optimized for "multimodal association". Video temporal features retain dynamic spatiotemporal information, and text semantic features cover the complete context. The accurate representation of both provides high-quality input for the subsequent mapping module's "text-video spatial association", improving the system's ability to understand multimodal information from the source.

[0032] In a preferred embodiment of the present invention, based on the correlation between video temporal feature vectors and text semantic feature vectors, the correspondence between the spatial coordinates of video keyframes and text semantics is analyzed to construct a dynamic spatial semantic mapping representation, and an initial spatial reference position is determined based on the mapping representation. This may include: Identify key noun lexical units in text semantic feature vectors and locate the corresponding visual activation regions in video keyframes; The center position of each activated region is extracted as the reference positioning point, and the inter-frame motion displacement vector is calculated by combining the video temporal feature vector; Perform time-sequential sampling along the direction of the motion displacement vector to generate a dynamic position sequence; Merge all reference positioning points with dynamic position sequences to form a spatial semantic mapping representation; Spatial distribution statistics are performed on the spatial semantic mapping representation, and the geometric center of the location set is calculated as the initial spatial reference position.

[0033] In this embodiment of the invention, key noun lexical identification is performed as follows: The text semantic feature vector is 256-dimensional (concatenated by the sequential and reverse vectors output by the bidirectional GRU), with each dimension corresponding to the activation intensity of a word in a specific semantic dimension. The system pre-sets a "noun word feature library" containing standard semantic feature vectors (256-dimensional, obtained through training with a large amount of text) of common nouns (such as "car", "pedestrian", "table").

[0034] The cosine similarity (range [0, 1]) between the semantic feature vector of the text and the feature vector of each noun in the database is calculated, and a similarity threshold of 0.65 is set—nouns exceeding this threshold are identified as key nouns. For example, the semantic vector of the text question "What brand is the red car in the video?" has a similarity of 0.72 with the standard vector of "car", which exceeds 0.65, so "car" is identified as a key noun. Video keyframe visual activation region localization: Keyframes are extracted from the video temporal feature vector (16-dimensional) (usually the middle frame of the video and frames with significant motion changes, such as the 10th and 20th frames). The spatial feature map of each keyframe is 109×109 (after compression by the second convolution layer).

[0035] The cosine similarity between the feature value of each spatial location (pixel) in the feature map and the feature vector of the key noun lexical unit is calculated, and an activation threshold of 0.55 is set—continuous pixel regions with similarity exceeding this value are marked as visual activation regions. For example, the activation region corresponding to "car" might be the pixel range (30-50, 40-60) in frame 10 (x-axis 30-50, y-axis 40-60), forming a rectangular bounding box. Reference point extraction: For each visually activated region's rectangular bounding box (e.g., x1=30, x2=50, y1=40, y2=60), calculate the center coordinates: The center x-coordinate = (x1 + x2) / 2 = (30 + 50) / 2 = 40; The center y-coordinate = (y1 + y2) / 2 = (40 + 60) / 2 = 50; The (40, 50) is the reference point for the active area. If the video contains 3 keyframes, 3 reference points are obtained: (40, 50) (frame 10), (42, 51) (frame 15), and (45, 53) (frame 20).

[0036] Calculation of inter-frame motion displacement vector: Based on the temporal correlation (reflecting inter-frame changes) in the video temporal feature vector, the displacement of the reference positioning point between consecutive keyframes is calculated. For example, from frame 10 to frame 15: displacement Δx1 = 42 - 40 = 2, Δy1 = 51 - 50 = 1, displacement vector is (2, 1); from frame 15 to frame 20: displacement Δx2 = 45 - 42 = 3, Δy2 = 53 - 51 = 2, displacement vector is (3, 2); the average displacement vector of multiple frames is taken to obtain the average motion displacement vector: ((2+3) / 2, (1+2) / 2) = (2.5, 1.5), which reflects the main direction and average velocity of the object's motion. Determine the sampling parameters: Using the average motion displacement vector (2.5, 1.5) as the direction reference, the sampling time interval is set to 500 milliseconds (i.e., sampling once every 0.5 seconds), and the number of samples is 5 points (covering the 2.5-second time range). First, the direction coefficients of the displacement vector are calculated: the x-direction displacement every 0.5 seconds is 2.5 / 2 = 1.25 (since the original vector is the displacement of 2 frame intervals, it is assumed that 1 frame = 0.25 seconds), and the y-direction displacement every 0.5 seconds is 1.5 / 2 = 0.75. Generate dynamic position sequences: Starting from the last reference positioning point ((45, 53) in frame 20), sample along the direction coefficient: First sampling point (+0.5 seconds): x=45+1.25=46.25, y=53+0.75=53.75→(46.25, 53.75); Second sampling point (+1.0 sec): x = 46.25 + 1.25 = 47.5, y = 53.75 + 0.75 = 54.5 → (47.5, 54.5); Similarly, the third sampling point (+1.5 seconds), the fourth sampling point (+2.0 seconds), and the fifth sampling point (+2.5 seconds) yield a dynamic position sequence, with each point accompanied by a corresponding timestamp. By merging all reference points and dynamic position sequences, a spatial semantic mapping representation is formed: Collect all reference positioning points (3) and dynamic position sequences (5) to form a set of 8 points, each containing "spatial coordinates (x, y) + timestamp t", for example, [(40, 50, t=10 frames), (42, 51, t=15 frames), (45, 53, t=20 frames), (46.25, 53.75, t=22 frames), (47.5, 54.5, t=24 frames), (48.75, 55.25, t=26 frames), (50, 56, t=28 frames), (51.25, 56.75, t=30 frames)].

[0037] This set is a dynamic spatial semantic mapping representation that intuitively presents the spatial location distribution of "textual key nouns (such as cars)" in the video as time changes.

[0038] For the eight points in the mapping representation, the average values ​​of the x and y coordinates are calculated to obtain the geometric center. This point serves as the initial spatial reference position, taking into account the spatial distribution of historical reference points and future dynamic sampling points, and is representative.

[0039] Key terms are selected using a cosine similarity threshold, and activation regions are located using video features to ensure accurate correspondence between text semantics and video visual content, avoiding interference from irrelevant areas. Inter-frame displacement vector calculation reflects the actual motion state of objects, and temporal equidistant sampling extends the temporal dimension of the position sequence, enabling spatial semantic mapping to not only include static positions but also reflect dynamic changes, better aligning with the temporal characteristics of the video. The geometric center integrates the spatial distribution of historical reference points and future sampling points, avoiding the randomness of a single frame's position.

[0040] In a preferred embodiment of the present invention, constructing a directional dynamic spatiotemporal detection region along the main direction of motion of the video object, starting from an initial spatial reference position, may include: Extracting the dominant motion direction vector based on the motion displacement vector field; Using the initial spatial reference position as the origin, a spatial extension direction is constructed along the dominant motion direction vector, and the dynamic range parameters are determined based on the spatial distribution range of the motion displacement vector. Based on the origin, spatial extension direction, and dynamic range parameters, a cone-shaped dynamic spatiotemporal detection region with directional constraints is constructed.

[0041] In this embodiment of the invention, the motion displacement vector field refers to the set of motion displacement vectors between all consecutive frames in a video (not limited to keyframes, but also including displacement vectors of non-keyframes). For example, a 10-second video contains 40 frames (4 frames per second), which can be divided into 39 inter-frame displacement vectors (frames 1-2, 2-3, ..., 39-40). Each vector is in the format of (Δx, Δy, timestamp), such as (2.1, 1.3, t1), (2.3, 1.5, t2), (1.9, 1.2, t3), (3.0, 1.8, t4) ... (a total of 39).

[0042] The specific process for extracting the dominant motion direction vector is as follows: Vector filtering: First, outliers (such as noise vectors with excessive displacement) are removed. A displacement threshold is set: vectors with an absolute value of Δx exceeding 10 or an absolute value of Δy exceeding 10 are considered outliers. Assuming that 3 out of 39 vectors are outliers, 36 valid vectors are retained.

[0043] Time-weighted processing: Vectors are assigned time weights—vectors closer to the current time have higher weights (recent movements better reflect current trends). The weighting rules are: the 10 most recent vectors have a weight of 1.0, the previous 10-20 have a weight of 0.8, and the previous 20-36 have a weight of 0.6. For example, the vector (3.2, 1.7) at time t39 (the most recent) has a weight of 1.0, the vector (2.8, 1.6) at time t30 has a weight of 1.0, the vector (2.5, 1.4) at time t20 has a weight of 0.8, and the vector (2.0, 1.1) at time t10 has a weight of 0.6.

[0044] The dominant direction of weighted average calculation: Calculate the weighted sum of all effective vectors Δx: (3.2×1.0+2.8×1.0+...+2.5×0.8+...+2.0×0.6) = total Δx = 85.6; Calculate the weighted sum of all effective vectors Δy: (1.7×1.0+1.6×1.0+...+1.4×0.8+...+1.1×0.6) = total Δy = 51.3; The dominant motion direction vector is (total Δx / 36, total Δy / 36), which reflects the overall motion direction of the video object (e.g., upward to the right).

[0045] Determine the spatial extension direction and dynamic range parameters: Spatial extension direction: The direction reference is directly taken as the dominant motion direction vector (2.38, 1.42), that is, the spatial extension direction is "the direction pointed along the vector (2.38, 1.42)" (which can be understood as the angle with the x-axis is about 31°, since tanθ=1.42 / 2.38≈0.6, corresponding to θ≈31°).

[0046] Dynamic range parameter calculation (including cone angle and length along the direction): Conical angle (reflecting directional dispersion): Calculate the angle between all effective displacement vectors and the dominant direction vector, and take the 90th quantile of the angle as half of the conical angle. For example, the angles between 36 vectors and (2.38, 1.42) are 5°, 8°, 12°... with a maximum of 15°. The 90th quantile is 12°, so the conical angle is 2 × 12° = 24° (ensuring that 90% of the motion direction is within the conical range).

[0047] Length along the direction (reflecting the distance of motion): Calculate the magnitude of all effective displacement vectors, take the average of the magnitudes, and multiply by the time coefficient (1 / 3 of the total video duration). For example, if the average magnitude is 2.76, the total video duration is 10 seconds, and the time coefficient is 10 / 3≈3.33, then the length = 2.76 × 3.33≈9.2 (extending 9.2 spatial units along the dominant direction).

[0048] Maximum radial width (radius of the cone's base): Calculated based on the angle and length, base radius = length × tan(angle / 2) = 9.2 × tan(12°) (to ensure that the cone's coverage width at the end is reasonable).

[0049] Using the initial spatial reference position (46.34, 53.78) as the origin, and combining the above parameters, a cone-shaped spatiotemporal region is constructed, specifically including both spatial and temporal extents: Spatial range (conical region): Vertex: Initial reference position (46.34, 53.78); Axis direction: along the dominant motion direction vector (2.38, 1.42), that is, extending from the origin to (46.34+9.2×(2.38 / 2.76), 53.78+9.2×(1.42 / 2.76)) (since 2.38 / 2.76 and 1.42 / 2.76 are the unit components of the direction vector), the endpoint coordinates are approximately (46.34+7.9, 53.78+4.7) = (54.24, 58.48); Cross-section: Each cross-section perpendicular to the axis is circular, with the radius increasing linearly from 0 at the apex to 1.96 at the base (radius 0 at the origin and 1.96 at the endpoint), and the center of all cross-sections lies on the axis. Directional constraints: Only include spatial regions within a 24° angle range; regions outside this angle (such as beyond 12° to the left or right) are excluded.

[0050] Time range: Based on the video temporal characteristics, the time range is set to "the time corresponding to the initial reference position ± 2 seconds" (assuming the time corresponding to the initial reference position is 5 seconds, then the time range is 3-7 seconds) to ensure that the detection area covers the key time period of the object's movement.

[0051] The final dynamic spatiotemporal detection area is a "conical space with a vertex at (46.34, 53.78) and extending 9.2 units along the direction of (2.38, 1.42) within a time range of 3-7 seconds, with an angle of 24°". This area accurately focuses on the main spatiotemporal range of the object's movement.

[0052] By using time weighting and outlier removal, the dominant motion direction vector can accurately reflect the main motion trend of the object, avoiding interference from accidental frame displacements and providing a reliable directional reference for region construction. The angle is calculated based on the dispersion of motion direction, and the length is combined with the motion distance and time coefficient, so that the region range can adapt to the motion intensity of different videos (e.g., the region of fast-moving objects is longer, and the angle of dispersed objects is larger), avoiding missed detections or redundancy caused by fixed ranges. The detection region that integrates spatial and temporal ranges considers both the spatial motion trajectory of the object and limits the key time window to ensure that the region contains the most relevant spatiotemporal information.

[0053] In a preferred embodiment of the present invention, internal and external temporal position points are selected within and outside the dynamic spatiotemporal detection region; based on the time series of the internal and external temporal position points, a temporal constraint path is formed to obtain the spatiotemporal constraint factor, which may include: Within the dynamic spatiotemporal detection area, three internal temporal location points are selected at fixed time intervals based on the timestamp sequence of the video temporal feature vector. Based on the start timestamp of the video temporal feature vector, a first external temporal position point is selected outside the dynamic spatiotemporal detection region; based on the end timestamp of the video temporal feature vector, a second external temporal position point is selected outside the dynamic spatiotemporal detection region. Connect all internal time-series location points in time-stamp order to form the main motion path, and connect the starting end of the main motion path to the first external time-series location point; connect the ending end of the main motion path to the second external time-series location point to form a complete spatiotemporal constraint path; Motion coherence characteristics are calculated based on the coordinate sequence of the complete spatiotemporal constrained path, and time decay characteristics are calculated based on the timestamp sequence of each location point. The spatiotemporal constraint factor is generated by integrating motion continuity characteristics and time decay characteristics.

[0054] In this embodiment of the invention, the time range of the dynamic spatiotemporal detection region has been determined (e.g., 3-7 seconds, corresponding to the 12th-28th frames of the video, 4 frames per second), and the timestamp sequence of the video temporal feature vector is the continuous time points within this region (t=3.0s, 3.5s, 4.0s, ..., 7.0s).

[0055] The selection process is as follows: Fixed time interval setting: Select a time interval of 1 second (i.e., select 1 point every 1 second) to ensure the key time sequence nodes within the coverage area.

[0056] Location time points: Within the range of 3-7 seconds, select 3 timestamps at intervals: t1=4.0s (earlier in the central part of the region), t2=5.0s (center of the region), t3=6.0s (later in the central part of the region).

[0057] Determine spatial coordinates: Based on the motion trajectory within the dynamic spatiotemporal detection area (determined by the previous dynamic position sequence and dominant direction vector), find the spatial location corresponding to each time point: t1=4.0s: corresponds to (47.5, 54.5) in the dynamic position sequence (position along the dominant direction vector (2.38, 1.42) at 4 seconds). t2=5.0s: Located at the center of the region, with coordinates (48.75, 55.25) (calculated based on motion trend interpolation); t3=6.0s: Continue to extend along the dominant direction, with coordinates (50.0, 56.0). The three internal timing positions were finally obtained: P_internal1 (47.5, 54.5, 4.0s), P_internal2 (48.75, 55.25, 5.0s), and P_internal3 (50.0, 56.0, 6.0s).

[0058] Two external temporal location points are selected outside the dynamic spatiotemporal detection region: The time range of the video temporal feature vector covers the entire video (e.g., 0-10 seconds), while the time range outside the dynamic spatiotemporal detection area is 0-3 seconds (front side) and 7-10 seconds (back side).

[0059] The selection process is as follows: First external temporal location point (front of the region): Based on the starting timestamp (t_start=0.0s) of the video temporal feature vector, this time point is outside the detection area (0-3 seconds).

[0060] The spatial coordinates are obtained by reverse deducing the motion trajectory: from the inner point P_inner1 (47.5, 54.5, 4.0s), backtracking for 4 seconds (from 4.0s to 0.0s) in the opposite direction of the dominant motion direction (-2.38, -1.42), backtracking by 1 unit vector distance per second, the coordinates are: x = 47.5 - 4 × 2.38 ≈ 38.0; y = 54.5 - 4 × 1.42 ≈ 48.8; thus, the first outer point is obtained: P_outer1 (38.0, 48.8, 0.0s).

[0061] Second external temporal location point (behind the region): The termination timestamp (t_end=10.0s) based on the video temporal feature vector indicates that this time point is outside the detection area (7-10 seconds).

[0062] The spatial coordinates are obtained by forward deduction of the motion trajectory: from the inner point P_inner3 (50.0, 56.0, 6.0s) along the dominant motion direction (2.38, 1.42) for 4 seconds (from 6.0s to 10.0s), extending by 1 unit vector distance per second, so the coordinates are: x = 50.0 + 4 × 2.38 ≈ 59.5; y = 56.0 + 4 × 1.42 ≈ 61.7; thus, the second outer point is obtained: P_outer2 (59.5, 61.7, 10.0s).

[0063] Connect all time-series locations in chronological order from earliest to latest to form a continuous path: Main motion path: connecting three internal points, namely P_inner1 (4.0s) → P_inner2 (5.0s) → P_inner3 (6.0s), reflecting the core motion trajectory within the region.

[0064] Additional path: The main path starts at the first external point: P_outer1 (0.0s) → P_inner1 (4.0s), the starting point of the movement before the coverage area; The main path terminates at the second external point: P_inner3 (6.0s) → P_outer2 (10.0s), the endpoint of the movement after the coverage area; Complete path sequence: sorted by timestamp as follows: P_outer1 (0.0s) → P_inner1 (4.0s) → P_inner2 (5.0s) → P_inner3 (6.0s) → P_outer2 (10.0s). This path includes both the key movements within the region and defines the overall temporal range of the movements through external points.

[0065] Calculate the displacement distance between adjacent points: The displacement distance from P_outer1 to P_inner1 is assumed to be 1.08; the displacement distance from P_inner1 to P_inner2 is assumed to be 1.46; the displacement distance from P_inner2 to P_inner3 is assumed to be 1.46; and the displacement distance from P_inner3 to P_outer2 is assumed to be 1.08. Calculate the standard deviation of displacement (to measure fluctuations): The distance sequence is [11.08, 1.46, 1.46, 11.08], with a mean of 6.27 and a standard deviation of 4.81. The motion coherence characteristic is 1 - standard deviation / mean (the larger the value, the better the coherence).

[0066] Calculation of time decay characteristics: Based on the distance between each point's timestamp and the "current query time" (assumed to be 5.0s, i.e., the regional central time), time weights are assigned (the closer the distance, the higher the weight): Time decay coefficient formula: weight = 1 / (1 + |t - 5.0s|) (ensure the weight is between 0 and 1); Weight calculation for each point: Assume that the weight of P_outer1 (0.0s) is 0.17; the weight of P_inner1 (4.0s) is 0.5; the weight of P_inner2 (5.0s) is 1.0; the weight of P_inner3 (6.0s) is 0.5; and the weight of P_outer2 (10.0s) is 0.17. The time decay feature is the normalized result of the weight sequence: summing [0.17, 0.5, 1.0, 0.5, 0.17] yields 2.34, and the sequence obtained by dividing each weight by 2.34 is used as the quantization feature of time decay.

[0067] The process of integrating motion continuity characteristics (0.23) and time decay characteristics is as follows: Unified feature dimension: The motion coherence feature (scalar) is expanded into a vector of the same length as the time decay feature (repeated 5 times). Weighted fusion: Multiply two vectors element-wise and then sum them to obtain the fused value; Normalization: The fused values ​​are mapped to the range [0, 1] (which is already within the range here), and the spatiotemporal constraint factor is finally generated as 0.23 (scalar form, or extended to a combination feature containing the time decay vector, such as [0.23, 0.07, 0.21, 0.43, 0.21, 0.07]), which quantifies the spatiotemporal correlation of the path.

[0068] Three points within a region are selected at fixed intervals, covering the morning, noon, and evening periods to ensure the capture of core changes in the motion trajectory and avoid the loss of temporal information due to single-point deviations. External points are selected based on the video start and end timestamps, extending the path beyond the region and fully covering the "cause and effect" of the object's motion, preventing temporal constraints from being limited to a local area and enhancing the global temporal logic of the path. The orderly connection of internal and external points forms a continuous path, including both core motion (internal) and boundary reference (external), making spatiotemporal relationships easier to resolve. Motion coherence features reflect the smoothness of the path (distinguishing between stable motion and abrupt changes), while time decay features highlight the importance of recent information. The spatiotemporal constraint factor generated by the fusion of these two features can comprehensively quantify spatiotemporal correlations, providing precise constraints for task intent resolution.

[0069] In a preferred embodiment of the present invention, the text semantic feature vector is fused with the spatiotemporal constraint factor to output a task intent vector; a structured query statement is constructed based on the task intent vector, and relevant entity information is retrieved from an external knowledge base and embedded as a knowledge feature vector, which may include: The text semantic feature vector is concatenated with the spatiotemporal constraint factor by feature dimension to generate a joint feature representation; Perform a cascaded nonlinear transformation on the joint feature representation to output semantic-spatiotemporal fusion features, specifically including: Input the joint feature representation into the first transform layer, and output the primary fused feature; Core feature elements are extracted from the primary fusion features, and the weight distribution of the core feature elements is dynamically adjusted according to the motion trajectory parameters of the spatiotemporal constraint factors to obtain the adjusted feature elements. The weighted feature elements are input into the feature reorganization layer, and the semantic-spatiotemporal fusion features are output. Calculate the matching degree between semantic-spatiotemporal fusion features and a preset intent category library, and select intent categories with matching degrees exceeding a threshold to encode as task intent vectors; Based on the semantic structure parsing results of the task intent vector, a structured query statement is generated; Execute structured query statements to retrieve target entities from external knowledge bases and transform entity information into knowledge feature vectors.

[0070] In this embodiment of the invention, the text semantic feature vector is 256-dimensional (including bidirectional semantics of the text), and the spatiotemporal constraint factor is a 6-dimensional combined feature (such as [0.23, 0.07, 0.21, 0.43, 0.21, 0.07], where 0.23 is motion coherence and the last 5 bits are time decay weights).

[0071] The specific process of feature dimension concatenation: The features are concatenated in the order of "textual semantic feature vector first, spatiotemporal constraint factor second" to form a joint feature representation.

[0072] Dimensionality calculation: 256 (text) + 6 (spatiotemporal) = 262 dimensions, meaning the joint feature is represented as a 262-dimensional vector, for example: The first 256 bits: [0.1, 0.3, -0.2, ..., 0.5, 0.2, -0.4, ...] (textual semantic features); The last 6 digits are: [0.23, 0.07, 0.21, 0.43, 0.21, 0.07] (spatiotemporal constraint factor).

[0073] The first transform layer outputs primary fused features: The first transformation layer is a fully connected layer (a linear transformation without an activation function, but a non-linear activation is added here). Specific parameters and process are as follows: Input and output dimensions: The input is a 262-dimensional joint feature representation, and the output is a 128-dimensional primary fusion feature (dimensional compression to extract core information).

[0074] Nonlinear transformation process: The layer contains 128 neurons, each connected to the 262-dimensional input features via weights (initial weight values ​​range from -0.01 to 0.01). For example, the first neuron performs a weighted summation of the 262 input feature values ​​(e.g., 1st feature × 0.002 + 2nd feature × (-0.003) + ... + 262nd feature × 0.001), and adds a bias (range from -0.001 to 0.001) to obtain the original output value.

[0075] Apply the ReLU activation function to the original output value (keeping positive values ​​and setting negative values ​​to 0). For example, if the original output of a neuron is -0.05, it becomes 0, and if the output is 0.12, it is kept as 0.12.

[0076] The outputs of 128 neurons are combined to form a 128-dimensional primary fusion feature, for example: [0.12, 0, 0.08, ..., 0.23, 0, ..., 0.15].

[0077] Core feature element extraction: Calculate the absolute value of each element in the primary fusion feature (128 dimensions), sort them from largest to smallest, and select the top 64 elements as core feature elements (accounting for 50%, retaining the most significant features). For example, the elements with the top 64 absolute values ​​are the 3rd, 5th, 8th...127th elements, with values ​​of 0.12, 0.08, 0.23, ..., 0.15 respectively.

[0078] Weight adjustment based on motion trajectory parameters: The motion trajectory parameters of the spatiotemporal constraint factor include motion coherence (0.23) and time decay weight ([0.07, 0.21, 0.43, 0.21, 0.07]). The maximum value of the time decay weight, 0.43, corresponds to the center time point of the region (5.0s), reflecting the high correlation at this time point.

[0079] Adjustment rules: If the core feature elements are related to "time" and "movement" (determined by feature dimension labels, such as the dimensions corresponding to "when" and "move" in the text), then the weight is multiplied by 1.5 (to enhance motion-related features). If it is related to "static attributes" (such as "color" or "shape"), the weight is multiplied by 1.2 (moderately enhanced). If the semantics are irrelevant (such as "weather", and there is no relevant information in the video), the weight is multiplied by 0.5 (to weaken irrelevant features).

[0080] The final 64-dimensional adjusted feature elements are: [0.345, 0.144, 0.04, ..., 0.225].

[0081] The feature recombination layer recombines the 64-dimensional adjusted feature elements into 96-dimensional semantic-spatiotemporal fusion features, as follows: The 64-dimensional elements are expanded using a "copy-interpolation" method: each element is copied 1-2 times according to its weight (high weight is copied twice, low weight is copied once). For example, 0.345 (high weight) is copied to 0.345, 0.345, 0.144 (medium weight) is copied to 0.144, and 0.04 (low weight) is copied to 0.04, eventually expanding to 96 dimensions.

[0082] The expanded vector is locally normalized: the sum of every three adjacent elements is normalized to 1, and each element is divided by 0.834 to ensure that the feature values ​​are within a reasonable range and the local correlation is enhanced; finally, a 96-dimensional semantic-spatiotemporal fusion feature is output.

[0083] Task intent vector generation: Preset intent category library: contains 100 common intent categories (such as "querying object brand", "asking about movement speed", "identifying color", etc.), each category corresponds to a 96-dimensional standard feature vector (obtained through training with a large number of samples).

[0084] Matching degree calculation: Calculate the cosine similarity (range [0, 1]) between the semantic-temporal fusion feature (96 dimensions) and each standard vector in the library. For example: The similarity to the standard vector of "querying object brand" is 0.82; The similarity to "color recognition" is 0.65; The similarity with "inquiry speed" is 0.32.

[0085] Threshold Screening and Encoding: Set the matching threshold to 0.7, and select "Query Object Brand" (0.82 > 0.7) as the valid intent. Encode this intent into a 20-dimensional task intent vector (using one-hot encoding: the 5th bit is 1, and the rest are 0, because "Query Brand" ranks 5th in the library), that is, the vector is [0, 0, 0, 0, 1, 0,..., 0] (20-dimensional).

[0086] Construct a Structured Query Statement: Semantic Structure Parsing: The task intent vector "Query Object Brand" corresponds to the semantic structure "[Object]'s [Attribute = Brand]". Combining with the key noun "car" in the text semantic feature vector, determine the query object as "the red car in the video", and generate a SPARQL statement, where "car", "color", and "brand" are entity and attribute labels in an external knowledge base (such as a car knowledge graph).

[0087] Knowledge Feature Vector Generation: Execute the SPARQL statement to match entities in the knowledge base, and get the entity corresponding to "red car" as "Te ×× M ×× l3", and the associated information includes "Brand = Te ××", "Model = M ×× l3", "Production Year = 2023", "Color = Red", "Manufacturer = Te ×× Company", etc.

[0088] Entity Information Embedding: Information Structured Splitting: Split the retrieved entity information into independent attribute-value pairs, that is, "Brand: Te ××", "Model: M ×× l3", "Production Year: 2023", "Color: Red", "Manufacturer: Te ×× Company", a total of 5 groups of key information.

[0089] Attribute Dimension Allocation: Preset a 128-dimensional feature dimension allocation rule, for example: The 1st - 20th dimensions correspond to the "Brand" attribute (20 dimensions); the 21st - 40th dimensions correspond to the "Model" attribute (20 dimensions); the 41st - 60th dimensions correspond to the "Production Year" attribute (20 dimensions); the 61st - 80th dimensions correspond to the "Color" attribute (20 dimensions); the 81st - 100th dimensions correspond to the "Manufacturer" attribute (20 dimensions); the remaining 28 dimensions are reserved dimensions (filled with 0).

[0090] Attribute Value Quantification Mapping: For "Brand = Te ××": In the 1st - 20th dimensions, through the pre-defined brand vocabulary mapping (such as "Te ××" corresponding to the activation of the 5th - 8th dimensions), set the value to 0.8 - 0.9 (high activation indicates significant brand features), and the rest of the dimensions are below 0.1; For “Model = M××l3”: In dimensions 21-40, “M××l3” corresponds to dimensions 25-28, and the value is set to 0.7-0.8; For “Production Year = 2023”: In dimensions 41-60, 2023 corresponds to dimensions 45-47, and the value is set to 0.6-0.7; For “Color = Red”: In dimensions 61-80, “Red” corresponds to dimensions 63-65, and the value is set to 0.8-0.9 (echoing the semantic meaning of “red” in the text); For “Manufacturer = Special ×× Company”: In dimensions 81-100, corresponding to dimensions 85-87, the value is set to 0.7-0.8 (related to brand attributes).

[0091] Generate knowledge feature vector: Integrate the above quantification results to form a 128-dimensional vector. The value range of each dimension of the overall vector is [0, 1], which fully represents the retrieved entity attribute information.

[0092] The dimensional concatenation of text semantics and spatiotemporal constraints preserves the integrity of the original features. The cascaded nonlinear transformation strengthens the correlation between semantics and spatiotemporal constraints through dimensional compression, weight adjustment and reorganization, avoiding the problem of feature silos. The dynamic weight adjustment is based on motion trajectory parameters, prioritizing the enhancement of core semantics related to spatiotemporal constraints (such as "car movement") and weakening irrelevant information, making the fused features more focused on the task intent.

[0093] By using pre-defined category libraries for matching and threshold filtering, task intent vectors can accurately pinpoint user needs (such as "search for brands" rather than "identify colors"), reducing ambiguity and providing a clear direction for query generation. Structured query statements, generated based on intent vectors and semantic parsing, can directly match entities and attributes from external knowledge bases, avoiding redundant results caused by fuzzy queries. Entity information is transformed into knowledge feature vectors of a unified dimension, adapted to video and text feature dimensions, providing standardized input for subsequent global fusion and ensuring the effective integration of multimodal knowledge.

[0094] like Figure 2 As shown, in another preferred embodiment of the present invention, fusing video temporal feature vectors, spatiotemporal constraint factors, and knowledge feature vectors to generate a global fused feature vector and predict the answer, and converting the predicted answer into speech output, may include: The initial fusion feature is formed by connecting the video temporal feature vector, the spatiotemporal constraint factor, and the knowledge feature vector. Based on the trajectory change parameters of the spatiotemporal constraint factors, the adaptive weighting coefficients of the three vectors are calculated. The three vectors are weighted separately using weighting coefficients and then concatenated to generate optimized fusion features; By analyzing the spatial distribution of calibration and optimization fusion features through cross-modal feature collaborative analysis, the calibration features are dimensionality reduced to generate a global fusion feature vector; The answer text is predicted based on the globally fused feature vector, and then converted into a digital audio signal for output. Specifically, this includes: Map the global fusion feature vector to the predicted value vector of the candidate answer; The predicted value vector is normalized to generate the probability distribution vector of the candidate answers; The candidate answer with the highest probability value is determined from the probability distribution vector and output as the answer text. The output answer text is converted into a digital audio signal using a text-to-speech converter.

[0095] In this embodiment of the invention, the video temporal feature vector is 16-dimensional, reflecting the spatiotemporal dynamics of the video; the spatiotemporal constraint factor is 6-dimensional, quantifying the spatiotemporal correlation; and the knowledge feature vector is 128-dimensional, representing external entity information.

[0096] The initial fused features are formed by concatenating dimensions in the order of "video temporal feature vector → spatiotemporal constraint factor → knowledge feature vector".

[0097] Dimension calculation: 16 + 6 + 128 = 150 dimensions, for example: First 16 bits: [8.9, -7.2, 5.3, ..., 2.1] (video features); The middle 6 bits: [0.23, 0.07, 0.21, 0.43, 0.21, 0.07] (spatial-temporal constraints); The last 128 bits: [0.8, 0.7, ..., 0.5, 0.3, ...] (knowledge characteristics); The initial fused features are 150-dimensional vectors, which fully preserve the original information of the three-modal features.

[0098] Based on the trajectory change parameters of the spatiotemporal constraint factor (including motion coherence 0.23 and time decay weights [0.07, 0.21, 0.43, 0.21, 0.07]), calculate the weighting coefficients of the video, spatiotemporal, and knowledge vectors (the sum of the coefficients is 1 to ensure reasonable weight allocation): The weighting coefficients of the video temporal feature vectors are positively correlated with motion coherence (the more coherent the motion, the more important the video features). The calculation formula is "motion coherence × 0.4 + 0.2".

[0099] The weighting coefficient of the spatiotemporal constraint factor is positively correlated with trajectory stability (the more stable the trajectory, the more reliable the spatiotemporal constraint). It is set to 0.3 times the motion coherence factor.

[0100] The weighting coefficient of the knowledge feature vector is positively correlated with the maximum value of the time decay weight (the more important the recent information, the more critical the knowledge association). The calculation formula is "maximum time decay value × 0.5 + 0.1", and the maximum time decay value is 0.43.

[0101] Calibration coefficients: Ensure that the sum of the three coefficients is 1, and finally determine the weighting coefficients as follows: video 0.3, spatiotemporal 0.1, knowledge 0.6.

[0102] The three vectors are weighted separately using weighting coefficients and then concatenated, as follows: Video temporal feature vector weighting: Each dimension value of the 16-dimensional vector is multiplied by 0.3 to obtain 16-dimensional weighted video features.

[0103] Spatiotemporal constraint factor weighting: Each dimension value of the 6-dimensional vector is multiplied by 0.1 to obtain the 6-dimensional weighted spatiotemporal features.

[0104] Knowledge feature vector weighting: Each dimension value of the 128-dimensional vector is multiplied by 0.6 to obtain 128-dimensional weighted knowledge features.

[0105] Optimized and fused features: The three weighted vectors above are spliced ​​together in their original order to form 16+6+128=150-dimensional optimized and fused features, highlighting the contributions of knowledge features (weight 0.6) and video features (0.3).

[0106] Cross-modal feature co-analysis calibration: Compare the correlation dimensions of features across different modalities (such as the "red" feature dimension in the video, the "red" attribute dimension in the knowledge, and the "color-related time points" dimension in the spatiotemporal constraints), and adjust inconsistent feature values. For example, the dimension value corresponding to "red" in the video is 1.59, while the dimension value corresponding to "red" in the knowledge is 0.48. The two should be positively correlated, so the video dimension value is adjusted to 1.59 × 0.8 (closer to the knowledge value) = 1.27, and the knowledge dimension value is adjusted to 0.48 × 1.2 (closer to the video value) = 0.58 to ensure cross-modal consistency.

[0107] Dimensionality reduction: Principal component analysis is used to retain the top 90% of the feature dimensions with the highest contribution (removing redundant information); among the 150 features, the cumulative contribution of the top 64 dimensions reaches 92%, so the top 64 dimensions are retained to generate a 64-dimensional global fusion feature vector.

[0108] Mapped to a vector of predicted candidate answers: A preset candidate answer database (such as "Special ××", "Red", "2023", "M××l3", etc., a total of 10 candidate answers) is used. Each answer corresponds to a 64-dimensional standard vector. The similarity between the global fusion feature vector and each standard vector (range [0, 10]) is calculated to obtain the predicted value vector: [8.5, 3.2, 2.1, 7.8, ...] (10-dimensional, each value corresponds to the matching score of the candidate answer).

[0109] Generate probability distribution vector: Normalize the predicted value vector (divide each value by the sum), which is the probability distribution vector.

[0110] Determine the output answer text: Select the candidate answer with the highest probability. For example, the probability of "Special ××" is 0.28, which is the highest. Therefore, the output answer text is "Special ××".

[0111] Convert "Special ××" into a digital audio signal through a text-to-speech converter. The process is as follows: Decompose the text into phonemes: Split "Special ××" into the pinyin "tè ××", and then convert it into international phonemes (with tones marked).

[0112] Generate the fundamental frequency and duration: Assign a fundamental frequency to each phoneme (the fundamental frequency of "tè" is about 150 Hz, "×" is about 160 Hz, and "×" is about 155 Hz) and a duration (each phoneme is about 0.3 seconds, and the total duration is 0.9 seconds).

[0113] Synthesize the audio waveform: Generate a continuous audio waveform (amplitude range [-1, 1]) based on the phonemes, fundamental frequency, and duration, and then convert it into a digital signal (sampling rate 44.1 kHz, and each sampling point is represented by 16-bit binary). Finally, output a playable digital audio signal.

[0114] Dynamically calculate the weights based on spatio-temporal constraints to make the importance of knowledge features (0.6) and video features (0.3) adapt to the scenario (such as knowledge is more critical when querying a brand), and enhance the pertinence of feature contributions. Adjust the association dimensions through collaborative analysis to reduce the conflicts between video, spatio-temporal, and knowledge features (such as the cross-modal unification of the "red" feature), making the global fusion features more reliable. Generate a probability distribution through normalization, select the answer with the highest probability, reduce the influence of random errors, and enhance the reliability of the answer (such as the probability of "Special ××" is the highest, avoiding misselecting other answers); text-to-speech converts the answer into natural audio to adapt to the auditory interaction scenario (such as no visual attention is required when driving), and enhances the convenience of the user experience.

[0115] An embodiment of the present invention also provides a computer-readable storage medium storing instructions. When the instructions run on a computer, the computer is made to execute the system as described above. All implementation manners in the above system embodiment are applicable to this embodiment and can also achieve the same technical effects.

[0116] The above is the preferred embodiment of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle described in the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A multimodal video knowledge query system that integrates agent task intent, characterized in that, include: The feature extraction module receives video clips and text questions, and processes the video frame sequence using a lightweight 3D convolutional network to generate video temporal feature vectors. Perform semantic analysis on the text question to generate a text semantic feature vector that represents the semantics of the question; The mapping module is used to analyze the correspondence between the spatial coordinates of video keyframes and the semantics of text based on the correlation between the video temporal feature vector and the text semantic feature vector, construct a dynamic spatial semantic mapping representation, and determine an initial spatial reference position based on the mapping representation. The region construction module is used to construct a directional dynamic spatiotemporal detection region along the main direction of motion of the video object, starting from the initial spatial reference position. The spatiotemporal constraint generation module is used to select internal and external temporal position points within and outside the dynamic spatiotemporal detection area; based on the time series of the internal and external temporal position points, a temporal constraint path is formed to obtain the spatiotemporal constraint factor; The knowledge fusion module is used to fuse text semantic feature vectors with spatiotemporal constraint factors to output task intent vectors; it constructs structured query statements based on task intent vectors, retrieves relevant entity information from external knowledge bases and embeds it as knowledge feature vectors; The global fusion module is used to fuse video temporal feature vectors, spatiotemporal constraint factors, and knowledge feature vectors to generate a global fusion feature vector and predict the answer, and then convert the predicted answer into speech output.

2. The multimodal video knowledge query system combining intelligent agent task intent according to claim 1, characterized in that, The problem involves receiving video clips and text, and then processing the video frame sequence using a lightweight 3D convolutional network to generate video temporal feature vectors. Semantic analysis is performed on the text question to generate a text semantic feature vector representing the semantics of the question, including: Video features are extracted hierarchically using a lightweight 3D convolutional network. The first convolutional layer captures local spatiotemporal correlation features, and the second convolutional layer extracts global temporal correlation features. After compression in the time dimension, the video temporal feature vector is output. The text problem is lexicalized and mapped into a sequence of word vectors through an embedding layer; A bidirectional gated cyclic unit is used to process word vector sequences, where forward encoding generates sequential semantic vectors and backward encoding generates reverse semantic vectors. Concatenate sequential semantic vectors and reverse semantic vectors to generate text semantic feature vectors.

3. The multimodal video knowledge query system combining intelligent agent task intent according to claim 2, characterized in that, Based on the correlation between video temporal feature vectors and text semantic feature vectors, the correspondence between the spatial coordinates of video keyframes and text semantics is analyzed, a dynamic spatial semantic mapping representation is constructed, and an initial spatial reference position is determined based on the mapping representation, including: Identify key noun lexical units in text semantic feature vectors and locate the corresponding visual activation regions in video keyframes; The center position of each activated region is extracted as the reference positioning point, and the inter-frame motion displacement vector is calculated by combining the video temporal feature vector; Perform time-sequential sampling along the direction of the motion displacement vector to generate a dynamic position sequence; Merge all reference positioning points with dynamic position sequences to form a spatial semantic mapping representation; Spatial distribution statistics are performed on the spatial semantic mapping representation, and the geometric center of the location set is calculated as the initial spatial reference position.

4. The multimodal video knowledge query system combining agent task intent according to claim 3, characterized in that, Starting from the initial spatial reference position, a directional dynamic spatiotemporal detection region is constructed along the main direction of motion of the video object, including: Extract the dominant motion direction vector based on the motion displacement vector field; Using the initial spatial reference position as the origin, a spatial extension direction is constructed along the dominant motion direction vector, and the dynamic range parameters are determined based on the spatial distribution range of the motion displacement vector. Based on the origin, spatial extension direction, and dynamic range parameters, a cone-shaped dynamic spatiotemporal detection region with directional constraints is constructed.

5. The multimodal video knowledge query system combining agent task intent according to claim 4, characterized in that, Select internal and external temporal location points within and outside the dynamic spatiotemporal detection area; Based on the time series of internal and external temporal location points, a temporal constraint path is formed to obtain the spatiotemporal constraint factor, including: Within the dynamic spatiotemporal detection area, three internal temporal location points are selected at fixed time intervals based on the timestamp sequence of the video temporal feature vector. Based on the start timestamp of the video temporal feature vector, a first external temporal position point is selected outside the dynamic spatiotemporal detection region; based on the end timestamp of the video temporal feature vector, a second external temporal position point is selected outside the dynamic spatiotemporal detection region. Connect all internal time-series location points in time-stamp order to form the main motion path, and connect the starting end of the main motion path to the first external time-series location point; connect the ending end of the main motion path to the second external time-series location point to form a complete spatiotemporal constraint path; Motion coherence characteristics are calculated based on the coordinate sequence of the complete spatiotemporal constrained path, and time decay characteristics are calculated based on the timestamp sequence of each location point. The spatiotemporal constraint factor is generated by integrating motion continuity characteristics and time decay characteristics.

6. The multimodal video knowledge query system combining agent task intent according to claim 5, characterized in that, By fusing textual semantic feature vectors with spatiotemporal constraint factors, a task intent vector is output. Structured query statements are constructed based on task intent vectors, and relevant entity information is retrieved from external knowledge bases and embedded as knowledge feature vectors, including: The text semantic feature vector is concatenated with the spatiotemporal constraint factor by feature dimension to generate a joint feature representation; Perform a cascaded nonlinear transformation on the joint feature representation to output semantic-spatiotemporal fusion features; Calculate the matching degree between semantic-spatiotemporal fusion features and a preset intent category library, and select intent categories with matching degrees exceeding a threshold to encode as task intent vectors; Based on the semantic structure parsing results of the task intent vector, a structured query statement is generated; Execute structured query statements to retrieve target entities from external knowledge bases and transform entity information into knowledge feature vectors.

7. The multimodal video knowledge query system combining agent task intent according to claim 6, characterized in that, Perform a cascaded nonlinear transformation on the joint feature representation to output semantic-spatiotemporal fusion features, including: Input the joint feature representation into the first transform layer, and output the primary fused feature; Core feature elements are extracted from the primary fusion features, and the weight distribution of the core feature elements is dynamically adjusted according to the motion trajectory parameters of the spatiotemporal constraint factors to obtain the adjusted feature elements. The weighted feature elements are input into the feature reorganization layer, and the semantic-spatiotemporal fusion features are output.

8. The multimodal video knowledge query system combining agent task intent according to claim 7, characterized in that, By fusing video temporal feature vectors, spatiotemporal constraint factors, and knowledge feature vectors, a global fused feature vector is generated to predict the answer. The predicted answer is then converted into speech output, including: The initial fusion feature is formed by connecting the video temporal feature vector, the spatiotemporal constraint factor, and the knowledge feature vector. Based on the trajectory change parameters of the spatiotemporal constraint factors, the adaptive weighting coefficients of the three vectors are calculated. The three vectors are weighted separately using weighting coefficients and then concatenated to generate optimized fusion features; By analyzing the spatial distribution of calibration and optimization fusion features through cross-modal feature collaborative analysis, the calibration features are dimensionality reduced to generate a global fusion feature vector; The answer text is predicted based on the global fusion feature vector, and then converted into a digital audio signal for output.

9. The multimodal video knowledge query system combining agent task intent according to claim 8, characterized in that, The answer text is predicted based on the globally fused feature vector, and then converted into a digital audio signal for output, including: Map the global fusion feature vector to the predicted value vector of the candidate answer; The predicted value vector is normalized to generate the probability distribution vector of the candidate answers; The candidate answer with the highest probability value is determined from the probability distribution vector and output as the answer text. The output answer text is converted into a digital audio signal using a text-to-speech converter.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program that, when executed by a processor, implements the system as described in any one of claims 1 to 9.

Images