A space intelligent dynamic video frame sampling method and system based on question sentence semantics

Abstract

The application relates to the technical field of video understanding, and provides a space intelligent dynamic video frame sampling method and system based on question sentence semantics, which can realize paradigm transition from traditional 'passive video sampling' to'semantic active perception'. By analyzing the question sentence semantics in advance and dynamically guiding frame selection by using the cross-modal correlation analysis mechanism of semantic vectors and visual features, the video can be watched with the question, and key information such as the spatial position, action stage, object change and the like focused on by the question can be actively focused. In addition, the application proposes a joint optimization strategy of semantic score, inter-frame difference, diversity and spatiotemporal consistency, so that the final key frame can meet the requirements of high semantic correlation, time sequence coherence and complete spatial coverage. Moreover, the task adaptive mechanism of the application can automatically adjust the sampling strategy according to the question type, so that the sampling process has task sensitivity and intelligent adjustment capability, and has excellent generalization and robustness among different tasks.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of video understanding, and particularly relates to a spatial intelligent dynamic video frame sampling method and system based on question semantics. BACKGROUND

[0002] Current video question answering systems and multi-modal large language models usually adopt uniform frame sampling or fixed interval sampling to select frame sequences from input videos for model analysis when processing video information. However, such static sampling strategies have obvious deficiencies in semantic relevance and computational efficiency.

[0003] Specifically, existing methods only rely on the temporal uniformity of video frames and cannot select key frames that are truly relevant to the question based on the semantic focus of the question. For questions involving action stages, spatial positions or time sequences, the model often ignores critical moments, leading to understanding bias. Moreover, equal-interval sampling introduces a large number of irrelevant frames, causing information redundancy and possibly missing important frames, which makes the model output distorted answers. When processing spatial understanding tasks, such as questions involving object positions, spatial relationships or path changes, fixed sampling cannot capture key spatial structure changes, thus limiting the spatial intelligence of the system. In addition, due to the high proportion of irrelevant frames, the input size of the multi-modal large model expands, significantly increasing the memory usage and inference time, resulting in low computational efficiency.

[0004] Therefore, there is an urgent need for an intelligent video frame dynamic sampling method that can improve the overall performance and efficiency in spatial understanding, time sequence perception and question answering inference. SUMMARY

[0005] The present application provides a spatial intelligent dynamic video frame sampling method and system based on question semantics to solve the problem that the prior art cannot understand the semantics of the question, lacks semantic guidance and task adaptive ability in the sampling process, resulting in inaccurate key frame selection and computational redundancy.

[0006] To solve the above technical problems, the embodiments of the present application disclose the following technical solutions:

[0007] One aspect of the present application provides a spatial intelligent dynamic video frame sampling method based on question semantics, comprising:

[0008] performing semantic analysis on the input language question to generate a semantic vector and determine a semantic category, the semantic category including a spatial category, an action category, a time category, a quantity category and an object category;

[0009] performing preliminary frame sampling on the input video and extracting visual features of each sampled frame;

[0010] map the semantic vector and the visual features of each sampling frame into a unified feature space, calculate a relevance score of the semantic vector and the visual features of each sampling frame, and obtain a semantic relevance distribution;

[0011] determine a key frame set of the video according to the semantic relevance distribution and a semantic category of the question sentence;

[0012] perform spatio-temporal constraint optimization on the key frame set to obtain a final key frame sequence.

[0013] Optionally, the semantic analysis on the input language question sentence, the generation of the semantic vector, and the determination of the semantic category include:

[0014] converting the question sentence into a high-dimensional semantic vector by using a preset natural language processing model;

[0015] recognizing semantic elements of the question sentence by using a syntax tree and a dependency analysis method;

[0016] determining the semantic category of the question sentence based on a preset classifier;

[0017] generating a keyword mask of the question sentence in combination with the semantic vector and the semantic elements.

[0018] Optionally, the preliminary frame sampling on the input video and the extraction of visual features of each sampling frame include:

[0019] sampling frames of the input video at a preset time interval to obtain a preliminary frame set;

[0020] inputting the preliminary frame set into a visual encoder to extract multi-layer visual features, the visual features including at least semantic features, spatial structure features, and temporal change features;

[0021] integrating the visual features of the preliminary frame set by using a temporal feature fusion mechanism to obtain a visual feature sequence with temporal continuity.

[0022] Optionally, the mapping of the semantic vector and the visual features of each sampling frame into a unified feature space, the calculation of a relevance score of the semantic vector and the visual features of each sampling frame, and the obtaining of a semantic relevance distribution include:

[0023] inputting the semantic vector of the question sentence and the visual features of each sampling frame into a preset cross-modal mapping network respectively, projecting the two into a unified feature space, and aligning the semantic vector and the visual features;

[0024] calculating a similarity score between the semantic vector and the visual features of each sampling frame in the feature space;

[0025] combining the similarity scores of all the sampling frames in a time sequence to form a semantic relevance distribution.

[0026] Optionally, the combining the similarity scores of all the sampling frames in time sequence to form a semantic correlation distribution comprises:

[0027] The similarity scores of the sampling frames are smoothed between frames by using a semantic smoothing mechanism.

[0028] The semantic correlation distribution is generated based on the smoothed similarity scores of the sampling frames.

[0029] Optionally, the determining the key frame set of the video according to the semantic correlation distribution and the semantic category of the question comprises:

[0030] For each sampling frame, an information amount score is calculated based on the similarity score, the visual difference degree between the front and rear frames, and the picture information entropy in the frame.

[0031] According to the information amount score of each sampling frame and the preset threshold corresponding to the semantic category of the question, a key frame set whose score exceeds the preset threshold is screened out.

[0032] Optionally, the method further comprises:

[0033] The similar frames in the key frame set are deleted by using a clustering analysis and sparsity optimization method.

[0034] Optionally, the spatio-temporal constraint optimization of the key frame set to obtain a final key frame sequence comprises:

[0035] The time continuity analysis and the space coverage analysis are performed on the key frame set after the sparsity processing to detect whether there is a time transition loss or a space coverage loss between the key frames.

[0036] If yes, the missing part is supplemented by using the sampling frames adjacent to the missing part in time.

[0037] The final key frame sequence is formed based on the key frame set after the time and space loss supplement, and each frame in the sequence has a corresponding similarity score.

[0038] Optionally, the information amount score is calculated for each sampling frame based on the similarity score, the visual difference degree between the front and rear frames, and the picture information entropy in the frame.

[0039] When the semantic category of the question is a space category, the weight of the picture information entropy in the sampling frame in the information amount score is increased.

[0040] When the semantic category of the question is an action category, the weight of the visual difference degree between the sampling frame and the front and rear frames in the information amount score is increased.

[0041] When the semantic category of the question sentence is the time category, the weight of the similarity score of the sampling frame in the information amount score is increased;

[0042] When the semantic category of the question sentence is the object category or the quantity category, the weight of the similarity score of the sampling frame and the picture information entropy in the sampling frame in the information amount score is increased.

[0043] Another aspect of the present application provides a spatial intelligent dynamic video frame sampling system based on the semantics of a question sentence, comprising:

[0044] a language semantic coding module configured to perform semantic analysis on an input language question sentence, generate a semantic vector and determine a semantic category, wherein the semantic category comprises a space category, an action category, a time category, a quantity category and an object category;

[0045] a visual feature extraction module configured to perform preliminary frame sampling on an input video and extract visual features of each sampling frame;

[0046] a semantic and visual matching module configured to map the semantic vector and the visual features of each sampling frame into a unified feature space, calculate a correlation score of the semantic vector and the visual features of each sampling frame, and obtain a semantic correlation distribution;

[0047] a key frame selection module configured to determine a key frame set of the video according to the semantic correlation distribution and the semantic category of the question sentence;

[0048] a space and time constraint optimization module configured to perform space-time constraint optimization on the key frame set to obtain a final key frame sequence.

[0049] The present application discloses a spatial intelligent dynamic video frame sampling method and system based on the semantics of a question sentence, which can realize the paradigm transition from traditional "passive video sampling" to "semantic active perception". By analyzing the semantics of the question sentence before sampling and using the cross-modal correlation analysis mechanism of the semantic vector and the visual features to dynamically guide frame selection, the video can be watched "with the question", and the key information such as the spatial position, the action stage and the object change concerned by the question sentence can be actively focused, so that the accuracy and the calculation efficiency of the video question answering and the multi-modal reasoning are greatly improved. On this basis, the present application proposes a joint optimization strategy of semantic score, frame difference, diversity and space-time consistency, so that the final key frame meets the requirements of high semantic correlation, time sequence continuity and space coverage integrity at the same time, and the information utilization rate and the scene integrity are significantly improved.

[0050] In addition, the task adaptive mechanism of the present application can automatically adjust the sampling strategy according to the question type: the space category strengthens the spatial diversity, the action category focuses on the stage change, the time category maintains the equidistant distribution, and the object category improves the detection weight, so that the sampling process has task sensitivity and intelligent adjustment ability, and has excellent generalization and robustness between different tasks.

[0051] The above mentioned and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings, in which: BRIEF DESCRIPTION OF DRAWINGS

[0052] The above mentioned and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings, in which:

[0053] Figure 1 A flowchart of a space intelligent dynamic video frame sampling method based on question sentence semantics provided for an embodiment of the present application;

[0054] Figure 2 A flowchart of a process of implementing step S100 in the method of the present application; Figure 1

[0055] A flowchart of a process of implementing step S200 in the method of the present application; Figure 3 Figure 1 A flowchart of a process of implementing step S300 in the method of the present application;

[0056] Figure 4 A flowchart of a process of implementing step S303 in the method of the present application; Figure 1

[0057] A flowchart of a process of implementing step S400 in the method of the present application; Figure 5 Figure 4 A flowchart of a process of implementing step S500 in the method of the present application;

[0058] Figure 6 Figure 1 A flowchart of a process of implementing step S500 in the method of the present application;

[0059] Figure 7 A flowchart of a process of implementing step S500 in the method of the present application; Figure 1

[0060] A flowchart of a process of implementing step S500 in the method of the present application; Figure 8 DETAILED DESCRIPTION

[0061] ​​​​Embodiments of the present disclosure will be described below in greater detail with reference to the accompanying drawings. While embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be more thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0062] The term "comprising" and variations thereof as used herein are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements is not necessarily limited to those elements, but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Unless otherwise specified, the term "or" as used herein is intended to mean "and / or". The term "based on" means "based, at least in part, on". The terms "one example embodiment" and "an embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", and the like can refer to different or the same objects. Other explicit or implicit definitions can also be included below.

[0063] Figure 1 A flowchart of a space intelligent dynamic video frame sampling method based on question sentence semantics provided for an embodiment of the present application is shown in FIG. 1, which includes the following steps: Figure 1

[0064] Step 100: Perform semantic analysis on the input language question sentence to generate a semantic vector and determine a semantic category.

[0065] This step performs full-link semantic analysis on the input natural language question sentence to extract semantic structure information that can be used for subsequent visual sampling and cross-modal matching. In combination with a pre-trained language model, a syntactic structure analysis method, and a semantic classifier, the conversion from natural language to machine-understandable semantic representation is realized. Through this step, the high-dimensional semantic vector Q of the question sentence, the semantic category label, and the keyword mask can be obtained, thereby providing a unified and quantifiable semantic driving signal for subsequent video frame sampling decision-making.

[0066] In one embodiment of the present disclosure, as shown in FIG. 2, the following method can be used to implement this step: Figure 2

[0067] Step 101: Convert the question sentence into a high-dimensional semantic vector using a pre-set natural language processing model.

[0068] The input question sentence q is encoded for language features, and a pre-trained large-scale natural language processing model (such as Qwen2, BERT, LLaMA3, or CLIP-Text Encoder, etc.) is used to vectorize the question sentence. Such models have strong language understanding capabilities and can capture entities, action intentions, temporal relationships, spatial relationships, and semantic directionality in the question sentence. After inputting the question sentence into the model, a high-dimensional semantic vector of dimension d is obtained ​​As the global semantic representation of the question, it captures the entities, actions, relationships and time directions in the question. This representation not only contains the explicit lexical information in the sentence, but also reflects the linguistic contextual relationship through the semantic modeling capability of the model, so that the subsequent module can realize the cross-modal semantic alignment of language and visual features based on the semantic vector Q.

[0069] Step 102: Identify the semantic elements of the question using syntax tree and dependency analysis method.

[0070] In order to further refine the internal structure of the question, this step performs syntax tree analysis and dependency relationship analysis on the question. By constructing a syntax tree, the subject, object, predicate phrase and modifier phrase in the sentence can be identified, and the hierarchical structure of each component can be determined. Dependency analysis is used to determine the semantic dependency relationship between words, such as: (1) subject and object, such as "person", "table", "cup"; (2) predicate and action type, such as "sit down", "move", "open"; (3) spatial relationship, such as "next to", "above", "left"; (4) time attribute, such as "when", "after", "before". These semantic elements provide a basis for subsequent semantic category determination and key word mask generation.

[0071] Step 103: Determine the semantic category of the question based on the pre-set classifier.

[0072] In order to enable the subsequent frame sampling module to adopt differentiated strategies according to different types of questions, this step uses a pre-trained semantic classifier to determine the semantic category of the question. The classifier can use a Softmax classification head, and its input is the semantic vector Q generated in step 101 and the key semantic elements extracted in step 102.

[0073] The classifier divides the question into five pre-set semantic types according to the semantic distribution characteristics: spatial class (such as "above whom?", "where is the position?"), action class (such as "when do you sit down?", "who picks up the cup?"), time class (such as "when does it happen?"), quantity class (such as "how many?"), object class (such as "what is this object?").

[0074] Step 104: Generate the key word mask of the question based on the semantic vector and the semantic elements.

[0075] In order to improve the accuracy of semantic alignment in the subsequent cross-modal matching process, this step generates the key word mask of the question based on the semantic vector Q and the parsed semantic elements. The key word mask is used to mark the key semantic fragments in the question that are most contributive to visual sampling, such as action words, object names, spatial relationship words or time modifier words. The importance of each word in the complete semantics is calculated through attention mechanism, weight selection strategy or word importance scoring model.

[0076] Step 200: Preliminary frame sampling of the input video and extraction of visual features of each sampled frame.

[0077] This step performs preliminary frame sampling on the input video stream to obtain representative frames from the continuous video sequence and construct the basic visual features. First, a number of frames are extracted from the video stream at fixed time intervals to form a preliminary frame set. Then, these frames are input into the visual encoder for feature extraction, and a multi-level visual feature representation containing semantic information, spatial geometric information, and temporal change attributes is output. To further enhance the semantic continuity between frames, the system integrates the preliminary features through a temporal context fusion mechanism, enabling the expression of action trends, object position changes, and scene transitions between adjacent frames in a unified feature space.

[0078] In one embodiment of the present disclosure, as shown in Figure 3 This step can be implemented as follows:

[0079] Step 201: Frame sampling of the input video at a preset time interval to obtain a preliminary frame set.

[0080] According to the preset time interval Δt, the input video stream is sampled, usually 3 to 5 frames per second, to obtain a preliminary frame set that can cover the main scene changes of the video. The frame sequence obtained by sampling is denoted as .

[0081] Step 202: Input the preliminary frame set into the visual encoder to extract multi-level visual features.

[0082] Each frame of image will be input into the visual encoder (such as Swin Transformer) and generate visual feature tensors of semantic layer, spatial layer, and temporal layer at multiple scale levels.

[0083] The visual encoder outputs the feature representation , where is the semantic layer feature, used to represent object classes, action types, and scene semantics in the frame, is the spatial geometric layer feature, used to represent object positions, structure layouts, depth changes, and other geometric information, is the temporal layer feature, which captures motion amplitude, action rate, and scene switching by comparing the differences between consecutive frames.

[0084] Step 203: Integrate the visual features of the preliminary frame set through a temporal feature fusion mechanism to obtain a visual feature sequence with temporal continuity.

[0085] The step integrates the visual features of each frame by using a temporal context fusion mechanism, so that the feature sequence can reflect the continuous change relationship of the video in the time dimension. The temporal attention is used to propagate semantic, spatial and dynamic information between different frames, so that the motion association, object position change and scene transition between frames can form a continuous and differentiable dynamic semantic field in the feature space.

[0086] Step 300: Map the semantic vector and the visual features of each sampling frame into a unified feature space, calculate the relevance score of the semantic vector and the visual features of each sampling frame, and obtain the semantic relevance distribution.

[0087] This step can align the cross-modal between the question semantic information and the visual content of the video frame, and realize the semantic matching of the two in the unified feature space. First, the semantic representation vector Q of the question is mapped into the feature space by the visual encoder, and the feature vector of each frame is obtained. The input is projected into a unified feature space by a preset cross-modal mapping network, so that the semantic information and the visual information are comparable. Then, the similarity score between the semantic vector and the visual features of each frame is calculated in the unified feature space, so as to obtain the relevance index of the question semantic corresponding to each frame. In order to improve the stability and continuity of the matching, the similarity score is further processed by frame semantic smoothing to weaken the noise influence caused by light change, camera jitter, etc., so that the finally obtained semantic relevance distribution can truly reflect the continuous change trend of the semantic elements in the video over time.

[0088] In one embodiment of the present disclosure, as shown in Figure 4 The step can be implemented in the following way:

[0089] Step 301: The semantic vector of the question and the visual features of each sampling frame are input into a preset cross-modal mapping network, and the two are projected into a unified feature space, so that the semantic vector and the visual features are aligned.

[0090] The cross-modal mapping network can include a multi-layer feedforward network, a feature alignment layer and a normalization module, which are used to encode the semantic vector and the visual features into embedded vectors of unified dimension and unified scale, so that the two can be projected and matched in the same vector space. In addition, during the cross-modal mapping process, the semantic vector Q of the question will be used as the query signal of the cross-modal attention mechanism, and the local region features in the visual features will be regarded as the retrieval content (Key, Value). By calculating the attention weight of Query-Key, the scene or target in the video that best matches the question semantic can be automatically focused, realizing double-layer semantic alignment at frame level and region level.

[0091] Step 302: Calculate the similarity score between the semantic vector and the visual features of each sampling frame in the feature space.

[0092] After projection into the unified feature space, the semantic similarity score S between the semantic vector Q and each frame visual feature vector F is calculated based on their representations , indicating the importance of the frame in answering the question. The final semantic relevance distribution curve , intuitively reflects the response strength of each frame in the video to the question semantics, providing a decision basis for subsequent dynamic frame sampling.

[0093] The similarity calculation can adopt dot product similarity, cosine similarity, etc., to measure the matching degree of specific objects, actions or scenes in the frame and the semantic elements of the question. Cross-modal attention further associates question semantics with visual features in different regions within the frame, focusing on the most valuable visual clues in the spatial dimension. For example, when the question contains the action semantics "sit down", the model will identify the body posture change area in the visual features and give it a higher weight, making the similarity score of such frames higher. Thus, the frame-level semantic relevance score sequence is obtained.

[0094] Step 303: Combine the similarity scores of all sampled frames in chronological order to form a semantic relevance distribution.

[0095] The similarity scores of all sampled frames are arranged in chronological order to construct the original semantic relevance curve. This curve describes the response strength variation of the question semantics along the entire video timeline, and can intuitively present the positions of semantic key actions or scenes in the video. In addition, to improve the stability of the distribution and reduce the impact of sudden noise, this step further sets up a semantic smoothing mechanism to perform inter-frame smoothing on the original similarity sequence, so that the similarity curve presents temporal continuity.

[0096] In one embodiment of the present disclosure, as shown in Figure 5 , the following sub-steps can be used to implement step 303:

[0097] Step 3031: Use a semantic smoothing mechanism to perform inter-frame smoothing on the similarity scores of each sampled frame.

[0098] Perform semantic smoothing operation on the similarity score sequence obtained in step 302 to prevent abnormal high or low scores caused by instantaneous light changes, image blur or camera jitter. The semantic smoothing mechanism can be based on sliding window averaging, temporal convolution filtering or lightweight temporal attention network, which propagates semantic relevance between adjacent frames to make the similarity score present smooth transition along the time axis.

[0099] Step 3032: Generate a semantic relevance distribution based on the smoothed similarity scores of the sampled frames.

[0100] ​After the smoothing process, the smoothed similarity score sequence is recombined into the final semantic relevance distribution. The distribution presents a time-continuous relevance curve, and the peak position indicates the key moment that best matches the semantic of the question, while the width of the curve reflects the time span of the semantic relevant event. According to the distribution, several high-relevance frames can be selected, and their context frames before and after them are automatically expanded to ensure time continuity and semantic integrity.

[0101] Step 400: According to the semantic relevance distribution and the semantic category of the question, determine the key frame set of the video.

[0102] After obtaining the semantic relevance distribution generated in step 300, enter the dynamic frame selection stage to realize the self-adaptive video key frame screening for the semantic of the question. This step comprehensively considers the semantic category of the question, the semantic relevance of the frame, the visual change amplitude, and the picture complexity in the frame, and automatically determines the optimal key frame set through a dynamically adjusted semantic-driven sampling mechanism. The system first calculates the information amount score of each sampling frame , and dynamically adjusts the weight of each element in the score according to the question type, so that the sampling strategy can adapt to different question semantics such as action type, time type, space type, and object type. Then, according to the information amount score curve and the variance change, an adaptive threshold is set to select the candidate key frames whose scores exceed the threshold. To avoid redundancy caused by repeated continuous frames, clustering analysis, sparse optimization, and spatial diversity constraints are further used to remove similar frames from the candidate key frames, so that the final key frame set has high semantic representativeness, visual diversity, and time sequence integrity.

[0103] In one embodiment of the present disclosure, as shown in Figure 6 , the following method can be used to implement this step:

[0104] Step 401: For each sampling frame, calculate the information amount score based on the similarity score, the visual difference between the frame and the adjacent frames, and the picture information entropy in the frame.

[0105] An information amount score index that comprehensively reflects the semantic and visual information contribution of each frame is constructed. The information amount score is composed of three parts: the first part is the relevance of the frame to the semantic of the question; the second part is the visual difference between the frame and the adjacent frames, which is used to reflect the scene change and action mutation; and the third part is the visual information entropy in the frame, which represents the complexity and visual information density of the picture. According to different question semantic categories, the weights of the three parts in the information amount score are adaptively adjusted through coefficients, thereby realizing a semantic-driven scoring mechanism.

[0106] In one embodiment of the present disclosure, the following method can be used to implement step S401:

[0107] (1) When the semantic category of the question is space, the weight of the picture information entropy in the information score of the sampling frame is increased; frames with rich structural features and reflecting the spatial layout are preferentially selected into the key frame set.

[0108] (2) When the semantic category of the question is action, the weight of the visual difference between the sampling frame and the previous and next frames in the information score is increased; the detection of action mutation frames is strengthened, so that the moments of action occurrence, change or end are more easily captured.

[0109] (3) When the semantic category of the question is time, the weight of the similarity score of the sampling frame in the information score is increased; the system can more accurately locate the time point or time period corresponding to the semantic category of the question.

[0110] (4) When the semantic category of the question is object or quantity, the weight of the similarity score of the sampling frame and the picture information entropy in the information score is increased. Frames containing target objects and having sufficient visual details are more easily selected.

[0111] Step 402: According to the information score of each sampling frame and the preset threshold corresponding to the semantic category of the question, a key frame set with a score exceeding the preset threshold is screened out.

[0112] According to the information score obtained in step 401, the adaptive threshold is dynamically determined according to the question category , and the key frame with a score exceeding the threshold is selected, i.e. the condition for the frame to be selected is: The threshold is not a fixed value, but is adjusted according to the fluctuation of the semantic correlation score, for example:

[0113]

[0114] Wherein, represents the variance of semantic correlation with time, which is used to measure the degree of semantic change.

[0115] When the semantic correlation fluctuates greatly, the system automatically reduces the threshold to capture more turning frames and semantic change frames; when the semantic distribution is stable, the threshold is increased to reduce redundant frames. Through this adaptive gating mechanism, the system can dynamically and automatically adjust the number and density of key frames according to the question category and semantic category, so that the sampling result is more accurate.

[0116] In one embodiment of the present disclosure, after obtaining the set of key frames, a clustering analysis and sparsity optimization method is further introduced to eliminate redundant frames with similar content or too close in time, ensuring that the key frames are representative in visual information and temporal structure. Clustering division is performed on the visual embedding vectors of the key frames, and only one frame with high semantic relevance is retained from the frames belonging to the same visual cluster and having highly similar content. To prevent the key frames from being concentrated in a certain view or a certain static segment, a diversity regularization term is further added:

[0117]

[0118] The optimization objective is to maximize spatial coverage :

[0119]

[0120] wherein, is the spatial occupancy area of the target object in the i-th frame, so that the finally selected key frames maintain sufficient differences in visual space.

[0121] In addition, for spatial questions involving scene structure understanding, spatial coverage is calculated in the frame selection process to measure the proportion of the scene area covered by the selected frames, and the key frame set is optimized according to the coverage range of the target object under different views, so that it can provide more complete spatial context.

[0122] Step 500: Temporal and spatial constraint optimization is performed on the set of key frames to obtain the final key frame sequence.

[0123] After obtaining the set of key frames generated in step 400, the temporal and spatial constraint optimization phase is entered to further improve the semantic integrity and visual continuity of the final key frame sequence. This step mainly performs global optimization on the set of key frames through temporal continuity constraint, spatial coverage optimization and task adaptive weight adjustment, to ensure that the output frame sequence meets the semantic requirements of the question while considering the temporal smoothness and spatial information integrity of the video.

[0124] In one embodiment of the present disclosure, as Figure 7 shown, this step can be implemented in the following way:

[0125] Step 501: Time continuity analysis and spatial coverage analysis are performed on the set of key frames after sparsity processing to detect whether there is a lack of temporal transition or spatial coverage between the key frames.

[0126] The key frame set after the sparsification processing is analyzed to identify potential missing in time and space. In the time dimension, the semantic relevance score curve between frames is smoothed to detect whether there is a significant score mutation or time jump between adjacent frames, such as "frame skipping" or time fault. If it is detected that the key action or event paragraph is not fully covered, mark the time gap. In the spatial dimension, the coverage is calculated according to the spatial distribution of the target object or scene area to identify whether some main spatial area in the key frame set is not effectively covered, especially for video scenes involving multiple objects or complex layout.

[0127] If there is a time transition missing or spatial coverage missing between key frames, step 502 is performed.

[0128] Step 502: Interpolation is performed using the sampling frames adjacent to the missing part.

[0129] For the detected time or space missing paragraph, select the frames adjacent to the missing area from the original frame sampling set for interpolation. The interpolation process is based on semantic relevance, visual difference and frame information amount for screening to ensure that the supplemented frames have continuity in semantic and visual features with the surrounding key frames. For time missing, prefer to select frames with smooth semantic scores and coherent actions or events with the previous and next frames to eliminate fault; for spatial missing, select frames that cover unsampled spatial areas so that key objects are included under different perspectives.

[0130] If there is no time transition missing or spatial coverage missing between key frames, the final key frame sequence is formed based on the current key frame set.

[0131] Step 503: Form the final key frame sequence based on the key frame set after completing the time and space missing interpolation, each frame in the sequence has a corresponding similarity score.

[0132] After completing the interpolation, the preliminary key frame set is merged with the supplemented frames, and the final key frame sequence is formed according to the time sequence, each frame in the sequence is continuous in time and fully covered in space. The semantic relevance score is retained for each frame in the sequence to allow downstream tasks (such as Video-LLM question answering or event detection) to perform weight weighting or further analysis.

[0133] In addition, the sequence strategy can be adjusted adaptively according to the task type, such as enhancing the perspective coverage for spatial problems, the action continuity for action problems, the sampling equidistance for time problems, or the target detection weight for object problems, so as to realize efficient key frame output for different question sentences.

[0134] For ease of understanding, the processing process of two different types of question sentences is given as follows:

[0135] Embodiment one provides a dynamic spatio-temporal keyframe sampling method based on question semantics, mainly for spatio-temporal reasoning type video question answering tasks, i.e. the question contains obvious time, action or sequence information, such as "when did this person leave the room", "when was the door opened", "where did the cup pick up and put back" and the like. This type of task requires the system to have the ability of semantic tracking across time periods and the ability to understand spatial changes.

[0136] The specific implementation process is as follows:

[0137] 1. The system is deployed on an intelligent video question answering platform with video reading and natural language processing functions, and can run on a GPU or an edge computing terminal. The input includes: a video stream; a natural language question; and optional task parameters (such as a sampling upper limit, a minimum time interval, etc.).

[0138] 2. When the question is "when did this person sit down?", the system needs to focus on the key frame area related to the "sitting down" action according to the question semantics, without scanning the entire video.

[0139] 3. The first stage of the system is language semantic encoding. The input question is processed by a semantic parsing model and divided into different semantic units: action (such as "sit down"), subject (such as "this person"), and time modifier (such as "when"). The semantic encoding module converts the question into a semantic embedding vector and labels the question type as "action-time type", i.e. focusing on the start, turning point and end time of the action. Further, the system internally defines a semantic attention template for this type of problem, so that the subsequent video sampling module preferentially detects frame features related to action change rate and character state transition. At the same time, the system establishes a semantic keyword table (such as "open", "walk", "drop", "pick up", etc.) for semantic mapping of corresponding events in visual features.

[0140] 4. The system extracts a preliminary frame set at a fixed interval in the video stream, generally extracting 3 to 5 frames per second to form a basic video segment sequence. Each frame is extracted by a visual encoder to extract features, including scene structure, object detection results and motion estimation information.

[0141] 5. To improve semantic perception ability, the system records the spatial position, motion direction, background change and other auxiliary attributes of the object, so that the frame features not only contain pixel layer information, but also have high-level semantic representation. After completing feature extraction, the system performs adjacent frame comparison on each frame to estimate the action intensity, object motion amplitude and scene switching degree. These dynamic information will be combined with language semantics to identify potential "event occurrence paragraphs".

[0142] 6、The system matches the semantic representation of the question with the video frame features to compute the semantic relevance of each frame to the question. When the question is "when did he sit down", the system looks for frames where the body posture changes significantly in the visual region corresponding to the action semantic "sit down", such as from standing to bending to sitting. The matching results form a "semantic relevance curve", whose peak position often corresponds to the semantically most critical moment in the video. Unlike traditional uniform sampling, the system dynamically selects a number of high-relevance frames according to the curve, and automatically extends a few context frames before and after to ensure temporal coherence. A preliminary set of key frames is thus obtained.

[0143] 7、After obtaining the preliminary key frames, the system enters the dynamic sampling phase to ensure that the key frame set covers both the semantic focus of the question and the temporal integrity. The system automatically sets the sampling number and density according to the semantic type: for time-sensitive questions (e.g. "when"), the sampling interval is small to maintain temporal continuity; for spatial path questions (e.g. "from where to where"), the sampling distribution is more uniform to cover the start and end points; for action phase questions (e.g. "when did he finish"), the system automatically inserts key frames when a significant action change is detected. The system defines an information amount indicator for each frame :

[0144]

[0145] wherein, is the question-frame semantic relevance; ; is the frame visual entropy, reflecting the complexity of the frame; a, b, g are adaptive coefficients. Through the above indicators, the system can capture key frames at moments of high semantic relevance and obvious scene dynamics. The condition for a frame to be selected is:

[0146]

[0147] wherein, is the question category adaptive threshold:

[0148]

[0149] wherein, is the semantic score variance.

[0150]

[0151] When the semantic change is dramatic (with large relevance fluctuations), the system lowers the threshold to capture turning frames.

[0152] 8、Further temporal smoothing is performed on these frames to smoothly transition the semantic changes between adjacent frames, avoiding semantic misjudgments due to camera jitter or lighting changes.

[0153] 9、System output frame sequence will be input downstream multimodal video question and answer model. In the task of the question "when does this person sit down?", the model only needs to process dozens of frames, rather than hundreds of frames, to accurately locate the "sitting down" time period and give the answer "at about 14 seconds of the video". While maintaining the accuracy of the question and answer, the input frame amount is reduced, the inference delay is shortened by half, and the overall efficiency is significantly improved.

[0154] The semantic of the question can effectively guide the video sampling, so that the model focuses on the "problem related time period". The system has good adaptability, time sequence integrity and efficiency, and provides more accurate semantic input for the video question and answer model.

[0155] Embodiment two provides a key frame sampling method based on spatial semantics and object relationship, which is aimed at spatial relationship understanding type problems, such as "where is the table in the room?", "who is next to the window?", "is the painting hung above or below the wall?" and the like. This kind of problem pays more attention to the spatial layout and the relative position relationship between objects, so the sampling strategy should focus on the frames containing complete spatial information, rather than all time-continuous segments.

[0156] The specific implementation is as follows:

[0157] 1、This embodiment is suitable for video question and answer tasks in static or slowly changing scenes, such as indoor monitoring video, scene roaming video, etc. The input content includes video stream and natural language question. The video usually contains multiple objects and complex background, and the spatial relationship will be different at different angles. The goal of the system is to select the key frame that best reflects the spatial relationship, so that the subsequent Video-LLM can accurately infer the relative position of the object.

[0158] 2、After the system receives the question, the semantic module will identify the core spatial relationship words (such as "above", "next to", "in front of", "opposite") and automatically determine that the question type is "spatial relationship type". Then extract the entity names involved (such as "table", "window", "person"), and establish the object pair relationship, for example, <table, person>, the relationship is "next to". These semantic elements will provide retrieval targets for the visual module, so that subsequent sampling can focus on frames containing these two objects.

[0159] The system defines the weight template for spatial type problems:

[0160]

[0161] Among them, semantic matching weight; spatial distribution weight; time continuity weight.

[0162] 3、Video feature extraction module extracts frame sequences at fixed intervals in the preliminary sampling stage, and uses object detection network (such as DETR) to identify all semantic entities and their spatial coordinates in the frame. The system matches the positions of the objects detected in each frame with the target entities in the question, and calculates spatial distance, relative orientation and other spatial feature indicators. In this process, the system retains the spatial coordinate distribution of the objects and the scene structure information, so as to compare the relative position changes of the objects between different frames. For example, if the "table" and "person" are closest in a certain frame and the direction satisfies the "next to" relationship, then this frame is marked as a high correlation frame.

[0163] 4、The system generates a spatial correlation distribution curve according to the spatial relationship matching degree of each frame, indicating the degree of semantic matching of each frame with the question. The system automatically selects frames with high correlation as a candidate frame set, and performs spatial distribution optimization to ensure that the selected frames can cover the spatial layout from different angles. When the question involves room structure (such as "is the window on the left or right side"), the system will retain representative frames from different angles to provide complete spatial context.

[0164] 5、To prevent adjacent frames from repeating spatial information, the system introduces a spatial diversity strategy: if the relative position difference between two frames is small, only one of them is retained, thereby reducing redundancy. To prevent the frame selection from focusing on the same angle, the system introduces a diversity regularization term:

[0165]

[0166] The optimization goal is to maximize spatial coverage :

[0167]

[0168] where, is the spatial occupancy area of the target object (object) in the i-th frame.

[0169] 6、Spatial class problems are mainly based on spatial relationships, but if there is lens movement or scene change in the video, the system still needs to ensure temporal continuity. Therefore, the system retains some adjacent frames as "transition frames" during the sampling process, so that the spatial semantic change process remains stable. This module uses a lightweight temporal smoothing algorithm to suppress the fluctuations in spatial matching degree, preventing false sampling caused by lens jitter. The final generated key frame set usually contains 6 to 10 frames, which can fully cover the relative orientation information between room layout and target objects.

[0170] 7、In the spatial question answering task, the key frame generated by the embodiment can significantly improve the spatial understanding ability of the model. For example, in the question "who is the window next to", the key frame selected by the system clearly shows the relative position of the person and the window, so that the Video-LLM can accurately answer "the window is on the right side close to the person position". Compared with traditional uniform sampling, the number of input frames is reduced, but the question answering accuracy is improved. At the same time, the spatial positioning error is significantly reduced, especially in scenes with light changes or partial occlusions.

[0171] 8、The method occupies low computing resources when running, and can be executed in real time on edge devices. The scheme can be directly extended to video summary, spatial navigation, visual retrieval and other tasks. When the type of the question changes, the system automatically switches the sampling strategy without human intervention, and has strong universality and migration.

[0172] Figure 8 The structure diagram of a spatial intelligent dynamic video frame sampling system based on a question sentence semantic provided by the embodiment of the application is shown in Figure 8 As shown in the drawing, the system comprises the following modules:

[0173] The language semantic coding module 1 is configured to perform semantic analysis on the input language question, generate a semantic vector and determine a semantic category, wherein the semantic category includes a spatial category, an action category, a time category, a quantity category and an object category.

[0174] The visual feature extraction module 2 is configured to perform preliminary frame sampling on the input video and extract visual features of each sampling frame.

[0175] The semantic and visual matching module 3 is configured to map the semantic vector and the visual features of each sampling frame into a unified feature space, calculate the correlation score of the semantic vector and the visual features of each sampling frame, and obtain a semantic correlation distribution.

[0176] The key frame selection module 4 is configured to determine a key frame set of the video according to the semantic correlation distribution and the semantic category of the question.

[0177] The space and time constraint optimization module 5 is configured to perform space-time constraint optimization on the key frame set to obtain a final key frame sequence.

[0178] In one embodiment of the present application, the system further comprises an output and interface module for structuring and arranging the results of the aforementioned frame sampling process, and is responsible for connecting with downstream video understanding or multi-modal large models.

[0179] The module generates a semantic correlation heat map to show the response intensity of the question sentence semantics on different video frames, and outputs the frame weight matrix itself so that the downstream model can further utilize the sampling basis. In addition, the module also constructs an interpretable visualization result, including a semantic attention map that superimposes semantic attention information on the original video frame, thereby intuitively showing the reason for frame selection.

[0180] At the interface level, the system provides a unified Video-LLM Adapter API to deliver the optimal frame set and related features to the downstream inference model in a unified format, realizing seamless connection of the question-driven video analysis, retrieval or inference task, thereby improving the scalability and interaction efficiency of the overall system.

[0181] The above has described various embodiments of the present disclosure, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes are obvious to those skilled in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, practical application or technical improvement to the technology in the market, or to enable other ordinary skilled persons in the art to understand the embodiments disclosed herein.

Claims

1. A spatial intelligent dynamic video frame sampling method based on question semantics, characterized in that, include: The input language question is semantically parsed to generate a semantic vector and determine the semantic category, which includes spatial, action, temporal, quantity, and object categories. The input video is subjected to preliminary frame sampling, and the visual features of each sampled frame are extracted; The semantic vectors and visual features of each sampled frame are mapped to a unified feature space, and the correlation scores between the semantic vectors and the visual features of each sampled frame are calculated to obtain the semantic correlation distribution. Based on the semantic relevance distribution and the semantic category of the question, the set of keyframes for the video is determined, including: For each sampled frame, an information content score is calculated based on the similarity score, the visual difference between the frame and the preceding and following frames, and the intra-frame image information entropy, including: When the semantic category of the question is spatial, increase the weight of the image information entropy within the sampled frame in the information content score; When the semantic category of the question is action-related, increase the weight of the visual difference between the sampled frame and the preceding and following frames in the information content score; When the semantic category of the question is time-related, increase the weight of the similarity score of the sampled frames in the information content score; When the semantic category of the question is object or quantity, increase the similarity score of the sampled frames and the weight of the image information entropy within the sampled frames in the information content score. Based on the information content score of each sampled frame and the preset threshold corresponding to the semantic category of the question, a set of key frames with scores exceeding the preset threshold is selected. Spatiotemporal constraint optimization is performed on the keyframe set to obtain the final keyframe sequence.

2. The method according to claim 1, characterized in that, The process of semantically parsing the input language question, generating a semantic vector, and determining the semantic category includes: The question is converted into a high-dimensional semantic vector using a pre-defined natural language processing model; Using syntax trees and dependency parsing methods to identify semantic elements of questions; The semantic category of the question is determined based on a pre-defined classifier; Generate keyword masks for questions by combining semantic vectors and semantic elements.

3. The method according to claim 1, characterized in that, The initial frame sampling of the input video and the extraction of visual features from each sampled frame include: The input video is sampled at preset time intervals to obtain a preliminary frame set; The initial set of frames is input into the visual encoder to extract multi-layer visual features, which include at least semantic representation, spatial structure and temporal variation. The visual features of the initial frame set are integrated through a temporal feature fusion mechanism to obtain a visual feature sequence with temporal continuity.

4. The method according to claim 1, characterized in that, The step of mapping the semantic vector and the visual features of each sampled frame into a unified feature space, calculating the correlation score between the semantic vector and the visual features of each sampled frame, and obtaining the semantic correlation distribution includes: The semantic vector of the question and the visual features of each sampled frame are input into a pre-defined cross-modal mapping network, and the two are projected into a unified feature space to align the semantic vector with the visual features. Calculate the similarity score between the semantic vector and the visual features of each sampled frame in the feature space; The similarity scores of all sampled frames are combined in chronological order to form a semantic relevance distribution.

5. The method according to claim 4, characterized in that, The step of combining the similarity scores of all sampled frames in chronological order to form a semantic relevance distribution includes: A semantic smoothing mechanism is used to smooth the similarity scores of each sampled frame. A semantic relevance distribution is generated based on the smoothed sampled frame similarity scores.

6. The method according to claim 1, characterized in that, The method further includes: Clustering analysis and sparsity optimization methods are used to remove similar frames from the keyframe set.

7. The method according to claim 6, characterized in that, The process of performing spatiotemporal constraint optimization on the keyframe set to obtain the final keyframe sequence includes: Temporal continuity analysis and spatial coverage analysis are performed on the sparse keyframe set to detect whether there are missing temporal transitions or missing spatial coverage between keyframes. If so, use sampling frames from the time period adjacent to the missing part for back-insertion to supplement; The final keyframe sequence is formed by supplementing the keyframe set after completing the time and space missing information. Each frame in the sequence has a corresponding similarity score.

8. A spatial intelligent dynamic video frame sampling system based on question semantics, characterized in that, include: The language semantic encoding module is configured to perform semantic parsing on the input language question, generate a semantic vector, and determine the semantic category, which includes spatial, action, temporal, quantity, and object categories. The visual feature extraction module is configured to perform preliminary frame sampling on the input video and extract the visual features of each sampled frame; The semantic and visual matching module is configured to map the semantic vector and the visual features of each sampled frame into a unified feature space, calculate the correlation score between the semantic vector and the visual features of each sampled frame, and obtain the semantic correlation distribution. The keyframe selection module is configured to determine the set of keyframes for the video based on the semantic relevance distribution and the semantic category of the question, including: For each sampled frame, an information content score is calculated based on the similarity score, the visual difference between the frame and the preceding and following frames, and the intra-frame image information entropy, including: When the semantic category of the question is spatial, increase the weight of the image information entropy within the sampled frame in the information content score; When the semantic category of the question is action-related, increase the weight of the visual difference between the sampled frame and the preceding and following frames in the information content score; When the semantic category of the question is time-related, increase the weight of the similarity score of the sampled frames in the information content score; When the semantic category of the question is object or quantity, increase the similarity score of the sampled frames and the weight of the image information entropy within the sampled frames in the information content score. Based on the information content score of each sampled frame and the preset threshold corresponding to the semantic category of the question, a set of key frames with scores exceeding the preset threshold is selected. The spatial and temporal constraint optimization module is configured to perform spatiotemporal constraint optimization on the keyframe set to obtain the final keyframe sequence.

Images