Method and system based on dual-relationship time alignment and adaptive region optimization

Abstract

The present invention proposes a method and system based on dual-relationship temporal alignment and adaptive region optimization. The method comprises: obtaining a video and a question prompt text, extracting features from the video using a visual encoder to obtain visual features, extracting features from the question prompt text using a text encoder to obtain text features, performing global average pooling on the visual features, and calculating cosine similarity with the text features to obtain a set of correlation scores between the visual feature representation and the text features of each frame, and screening based on the correlation scores between the visual feature representation and the text features of each frame to obtain a set of highly correlated frames and a set of moderately correlated frames. The present invention introduces a dynamic spatial sampling module to extract the maximum connected region from the candidate heat map region associated with the prompt to eliminate scattered interference and enhance the compactness of the spatial feature expression, thereby greatly reducing the interference of spatial noise.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and video processing technology, and in particular to a method and system based on dual-relationship time alignment and adaptive region optimization. Background Art

[0002] Video Question Answering (VQA) aims to empower machines to understand and reason about the semantics of multimodal video content, enabling them to respond to natural language queries. This task has become a core criterion for measuring the cognitive performance of multimodal AI systems.

[0003] In recent years, video understanding models that do not require training have become an emerging research direction due to their plug-and-play nature. By freezing the pre-trained visual-language model and designing a lightweight adaptation strategy, the original video frame sequence is converted into an input representation compatible with the model, thus avoiding the high resource cost of training. Models such as IG-VLM and FreeVA demonstrate the potential of converting video frames into a format compatible with high-performance visual-language models. These methods successfully bypass the computational burden of traditional video models by reconstructing frame representations and combining lightweight temporal reasoning, while maintaining comparable semantic understanding capabilities. However, existing VQA methods that do not require training still face two key challenges:

[0004] Challenge 1: Temporal Misalignment. The redundancy of high-frame-rate videos makes temporal sampling strategies susceptible to noise. Traditional temporal sampling methods, such as uniform sampling or two-stream feature extraction (e.g., SF-LLaVA), cannot effectively handle the uneven semantic distribution within video segments. Recent approaches such as Free Video-LLM, while introducing hint-guided frame selection, fail to establish a collaborative reasoning mechanism between textual semantics and inter-frame relationships. Due to the lack of joint modeling of the question context and inter-frame semantic consistency, these methods may select non-key frames or omit key action segments.

[0005] Challenge 2: Spatial Noise Interference. In complex video scenes, irrelevant regions such as background clutter and localized body parts can contaminate visual feature representations. Existing methods typically employ global pooling, resulting in severe information loss, or use fixed-size cue-guided cropping (such as Free Video-LLM), making it difficult to dynamically focus on semantic regions relevant to the task. This problem is particularly prominent in precise spatial localization tasks, such as object counting and relational reasoning, where scattered interfering regions can mislead the model into generating incorrect answers.

[0006] As mentioned above, there is still a lack of reliable methods based on non-training VQA methods, which directly affects the accuracy of video question answering. Based on this, it is necessary to propose a non-training VQA method that reduces the computational burden while maintaining comparable semantic understanding capabilities to address the above technical issues. Summary of the Invention

[0007] In view of the above situation, the main purpose of the present invention is to propose a method and system based on dual-relationship time alignment and adaptive region optimization to solve the above technical problems.

[0008] The present invention proposes a method based on dual-relation time alignment and adaptive region optimization, which includes the following steps:

[0009] Step 1: Obtain the video and question prompt text, use the visual encoder to extract features from the video to obtain visual features, and use the text encoder to extract features from the question prompt text to obtain text features;

[0010] Step 2: Perform global average pooling on the visual features and calculate cosine similarity with the text features to obtain the correlation score between the visual feature representation and the text features of each frame. Filter the frames based on the correlation score between the visual feature representation and the text features of each frame to obtain a set of highly correlated frames and a set of moderately correlated frames.

[0011] Step 3: Based on the highly correlated frame set, calculate the sum of similarities between each frame and the remaining frames, and filter them to obtain the remaining frame set;

[0012] Step 4: Calculate the sum of similarities between each frame in the moderately relevant frame set and all frames in the remaining frame set, and screen each frame in the moderately relevant frame set to obtain a candidate frame set; add all frames in the candidate frame set to the remaining frame set to obtain a key frame set;

[0013] Step 5: Reshape the visual feature representation of each frame in the key frame set into a video feature map, and calculate the correlation score with the text features to determine the largest connected area; confirm the center coordinates and area size of the region of interest based on the largest connected area, and crop the video feature map to determine the region of interest; flatten the region of interest into a set of embedding vectors, embed them with the text features, and then input them into the large language model for processing to obtain the final answer.

[0014] The present invention further proposes a system based on dual-relationship time alignment and adaptive region optimization, wherein the system applies the dual-relationship time alignment and adaptive region optimization method described above, and the system includes:

[0015] Feature extraction module, used to:

[0016] Obtain a video and a question prompt text, use a visual encoder to extract features from the video to obtain visual features, and use a text encoder to extract features from the question prompt text to obtain text features;

[0017] Dual-correlated time sampling module for:

[0018] The visual features are globally averaged pooled and cosine similarity is calculated with the text features to obtain the correlation score between the visual feature representation and the text features of each frame. The frames are then filtered based on the correlation score between the visual feature representation and the text features of each frame to obtain a set of highly correlated frames and a set of moderately correlated frames.

[0019] Based on the highly correlated frame set, the sum of similarities between each frame and the remaining frames is calculated and filtered to obtain the remaining frame set;

[0020] Calculating the sum of similarities between each frame in the moderately correlated frame set and all frames in the remaining frame set, and screening each frame in the moderately correlated frame set to obtain a candidate frame set; adding all frames in the candidate frame set to the remaining frame set to obtain a key frame set;

[0021] Dynamic spatial sampling module for:

[0022] The visual feature representation of each frame in the key frame set is reshaped into a video feature map, and the correlation score is calculated with the text features to determine the largest connected area; the center coordinates and area size of the region of interest are confirmed based on the largest connected area, and the video feature map is cropped to determine the region of interest; the region of interest is flattened into a set of embedding vectors, which are embedded and combined with the text features and then input into the large language model for processing to obtain the final answer.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] 1. This invention uses text and video as conditions, combines a dual-correlation temporal sampling module to dynamically adjust frames with mismatched text-image relevance, and uses a dynamic spatial sampling strategy to analyze regions highly relevant to the prompt, extract the largest connected region, and dynamically optimize the spatial feature representation of the video frame. This not only reduces the computational burden but also maintains comparable semantic understanding capabilities, enabling accurate localization of key spatiotemporal clues in complex scenes and generating answers to video questions and answers.

[0025] 2. This paper proposes a dual-linkage temporal sampling method based on text-visual alignment and inter-frame semantic consistency. It selects key frame sequences through bidirectional interactive reasoning and simultaneously removes redundant frames that conflict with the text context, thereby alleviating the problem of temporal alignment imbalance.

[0026] 3. The present invention introduces a dynamic spatial sampling module to extract the largest connected area from the candidate areas of the heat map related to the prompt to eliminate scattered interference and enhance the compactness of spatial feature expression, thereby greatly reducing the interference of spatial noise.

[0027] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a flowchart of the steps of the method based on dual-relationship time alignment and adaptive region optimization proposed by the present invention;

[0029] Figure 2 This is the overall framework diagram of the dual-relationship time alignment and adaptive region optimization method proposed in the present invention;

[0030] Figure 3 This is a system structure diagram of the dual-relationship time alignment and adaptive regional optimization system proposed in the present invention. DETAILED DESCRIPTION

[0031] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0032] These and other aspects of the embodiments of the present invention will become clear with reference to the following description and accompanying drawings. In these descriptions and accompanying drawings, some specific implementations of the embodiments of the present invention are specifically disclosed to illustrate some ways of implementing the principles of the embodiments of the present invention, but it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0033] See also Figure 1 This embodiment provides a method for dual-relationship time alignment and adaptive region optimization, the method comprising the following steps:

[0034] Step 1: Obtain the video and question prompt text, use the visual encoder to extract features from the video to obtain visual features, and use the text encoder to extract features from the question prompt text to obtain text features.

[0035] See also Figure 2 In step 1, a video and a question prompt text are obtained, and a visual encoder is used to extract features from the video to obtain visual features, and a text encoder is used to extract features from the question prompt text to obtain text features. Specifically, the following sub-steps are included:

[0036] Get the video and question prompt text, use the frame sampler to sample the video, and obtain an image sequence;

[0037] The visual encoder is used to extract features from the image sequence to obtain visual features. The following relationship exists in the corresponding process:

[0038] ;

[0039] in, Represents visual features, Indicates video, Indicates that after the visual encoder feature extraction, represents an image sequence, Both represent each frame of sampled video, Indicates the total number of frames sampled;

[0040] The text encoder is used to extract features from the question prompt text to obtain text features. The following relationship exists in the corresponding process:

[0041] ;

[0042] in, Represents text features, Indicates feature extraction after text encoder, Indicates the question prompt text.

[0043] It should be noted that in Figure 2 middle, represents the region of interest, Represents a large language model.

[0044] Step 2: Perform global average pooling on the visual features and calculate cosine similarity with the text features to obtain the correlation score between the visual feature representation and the text features of each frame. Screen according to the correlation score between the visual feature representation and the text features of each frame to obtain a set of highly correlated frames and a set of moderately correlated frames.

[0045] In step 2, the visual features are globally average pooled and cosine similarity is calculated with the text features to obtain the correlation score between the visual feature representation and the text features of each frame. The correlation scores between the visual feature representation and the text features of each frame are then filtered to obtain a set of highly correlated frames and a set of moderately correlated frames. The specific steps include the following:

[0046] Perform global average pooling on the visual features to obtain a set of visual feature representations for each frame;

[0047] Based on the set of visual feature representations of each frame, the cosine similarity calculation is performed on the visual feature representation of each frame and the text feature to obtain the correlation score between the visual feature representation of each frame and the text feature. The following relationship exists in the corresponding process:

[0048] ;

[0049] in, Indicates the The correlation score between the visual feature representation of the frame and the textual features, Indicates the Visual feature representation of the frame, Indicates taking the two norm, The time index of the frame;

[0050] The frames in the set of visual feature representations of each frame are sorted from large to small according to the correlation score between the visual feature representation of each frame and the text feature, and the top frames are selected. frames as high correlation frames to obtain a high correlation frame set; filter out the to individual frames as medium-correlation frames to obtain a medium-correlation frame set, and discard all remaining frames;

[0051] in, represents the total number of frames in the set of highly correlated frames, Indicates the total number of frames in the medium-correlated frame set.

[0052] Step 3: Based on the highly correlated frame set, calculate the sum of similarities between each frame and the remaining frames, and perform screening to obtain the remaining frame set.

[0053] In step 3, based on the highly correlated frame set, the sum of similarities between each frame and the remaining frames is calculated and filtered to obtain the remaining frame set, which specifically includes the following sub-steps:

[0054] Based on the set of highly correlated frames, the sum of the similarities between each frame and all other frames is calculated to obtain the similarity score of each frame in the set of highly correlated frames. The following relationship exists in the corresponding process:

[0055] ;

[0056] in, Indicates the first The similarity score of each frame, Indicates the first The visual feature representation of each frame, Indicates the first Visual feature representation of frames;

[0057] Based on the similarity score of each frame in the highly correlated frame set, the frames in the highly correlated frame set are sorted from small to large, and the top frames in the highly correlated frame set are discarded. frames to obtain the remaining frame set;

[0058] in, Indicates the total number of frames in the remaining frame set.

[0059] Step 4: Calculate the sum of similarities between each frame in the moderately relevant frame set and all frames in the remaining frame set, and screen each frame in the moderately relevant frame set to obtain a candidate frame set; add all frames in the candidate frame set to the remaining frame set to obtain a key frame set.

[0060] In step 4, the sum of similarities between each frame in the moderately relevant frame set and all frames in the remaining frame set is calculated, and each frame in the moderately relevant frame set is screened to obtain a candidate frame set; all frames in the candidate frame set are added to the remaining frame set to obtain a key frame set, which specifically includes the following sub-steps:

[0061] The sum of similarities between each frame in the moderately correlated frame set and all frames in the remaining frame set is calculated to obtain a similarity score for each frame in the moderately correlated frame set. The following relationship exists in the corresponding process:

[0062] ;

[0063] in, Indicates the number of frames in the medium correlation frame set The similarity score of each frame, Indicates the number of frames in the medium correlation frame set The visual feature representation of each frame, Indicates the remaining frame set Visual feature representation of frames;

[0064] Based on the similarity score of each frame in the medium-correlated frame set, the frames in the medium-correlated frame set are sorted from small to large, and the top frames in the medium-correlated frame set are discarded. frames to obtain a set of candidate frames;

[0065] All frames in the candidate frame set are added to the remaining frame set to obtain a key frame set.

[0066] Step 5: Reshape the visual feature representation of each frame in the key frame set into a video feature map, and calculate the correlation score with the text features to determine the largest connected area; confirm the center coordinates and area size of the region of interest based on the largest connected area, and crop the video feature map to determine the region of interest; flatten the region of interest into a set of embedding vectors, embed them with the text features, and then input them into the large language model for processing to obtain the final answer.

[0067] In step 5, the visual feature representation of each frame in the key frame set is reshaped into a video feature map, and the correlation score is calculated with the text features to determine the largest connected area; the center coordinates and area size of the region of interest are determined based on the largest connected area, and the video feature map is cropped to determine the region of interest; the region of interest is flattened into a set of embedding vectors, which are embedded and combined with the text features and then input into the large language model for processing to obtain the final answer. The specific sub-steps include the following:

[0068] Based on the key frame set, the visual feature representation of each frame is spatially reshaped to obtain a video feature map;

[0069] Based on the video feature map, the cosine similarity between the feature vector at each spatial position and the text feature is calculated to obtain the correlation score between the video feature map and the text feature. The following relationship exists in the corresponding process:

[0070] ;

[0071] in, Indicates the spatial position of the video feature map The correlation score between the feature vector and the text feature, Indicates the spatial position of the video feature map The video feature vector on represents the height of the visual feature map, Indicates the width of the visual feature map;

[0072] Based on the correlation score between the video feature map and the text feature, the feature vectors of each position in the video feature map are sorted from large to small, and the top The spatial heat map is generated by the features, and the connected domain analysis algorithm is used to analyze the spatial heat map to determine the maximum connected area, where ;

[0073] in, The number of feature vectors representing the positions within the selected video feature map, Indicates the weight value;

[0074] Calculate the average position of the features in the largest connected region to obtain the center coordinates of the region of interest. The following relationship exists in the corresponding process:

[0075] ;

[0076] in, represents the center coordinates of the region of interest, represents the number of features in the largest connected region, Indicates the first The height of the feature, Indicates the maximum connected area The width of the feature;

[0077] The maximum vertical span of the largest connected area is used as the height of the region of interest, and the maximum horizontal span of the largest connected area is used as the width of the region of interest to obtain the region size of the region of interest. The following relationship exists in the corresponding process:

[0078] ;

[0079] in, represents the height of the region of interest, represents the width of the region of interest, represents the maximum vertical span of the largest connected area, Indicates the maximum horizontal span of the largest connected area;

[0080] The video feature map is cropped based on the center coordinates and area size of the region of interest to determine the region of interest;

[0081] The region of interest is flattened into a set of embedding vectors, which are then embedded and combined with text features and input into a large language model for processing to obtain the final answer.

[0082] See also Figure 3 This embodiment further provides a system for dual-relationship time alignment and adaptive region optimization, wherein the system applies the dual-relationship time alignment and adaptive region optimization method described above, and the system includes:

[0083] Feature extraction module, used to:

[0084] Obtain a video and a question prompt text, use a visual encoder to extract features from the video to obtain visual features, and use a text encoder to extract features from the question prompt text to obtain text features;

[0085] Dual-correlated time sampling module for:

[0086] The visual features are globally averaged pooled and cosine similarity is calculated with the text features to obtain the correlation score between the visual feature representation and the text features of each frame. The frames are then filtered based on the correlation score between the visual feature representation and the text features of each frame to obtain a set of highly correlated frames and a set of moderately correlated frames.

[0087] Based on the highly correlated frame set, the sum of similarities between each frame and the remaining frames is calculated and filtered to obtain the remaining frame set;

[0088] Calculating the sum of similarities between each frame in the moderately correlated frame set and all frames in the remaining frame set, and screening each frame in the moderately correlated frame set to obtain a candidate frame set; adding all frames in the candidate frame set to the remaining frame set to obtain a key frame set;

[0089] Dynamic spatial sampling module for:

[0090] The visual feature representation of each frame in the key frame set is reshaped into a video feature map, and the correlation score is calculated with the text features to determine the largest connected area; the center coordinates and area size of the region of interest are confirmed based on the largest connected area, and the video feature map is cropped to determine the region of interest; the region of interest is flattened into a set of embedding vectors, which are embedded and combined with the text features and then input into the large language model for processing to obtain the final answer.

[0091] It should be understood that, although the various steps in the flow chart of each embodiment of the present invention are shown in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless otherwise specified herein, the execution of these steps is not strictly limited in order, and these steps can be performed in other orders. Moreover, at least a portion of the steps in each embodiment may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0092] It should be understood that various components of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having logic gate circuits for implementing logic functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array (PGA), a field-programmable gate array (FPGA), etc.

[0093] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0094] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A method based on dual-relation time alignment and adaptive region optimization, characterized in that: The method comprises the following steps: Step 1: Obtain the video and question prompt text, use the visual encoder to extract features from the video to obtain visual features, and use the text encoder to extract features from the question prompt text to obtain text features; Step 2: Perform global average pooling on the visual features and calculate cosine similarity with the text features to obtain the correlation score between the visual feature representation and the text features of each frame. Filter the frames based on the correlation score between the visual feature representation and the text features of each frame to obtain a set of highly correlated frames and a set of moderately correlated frames. Step 3: Based on the highly correlated frame set, calculate the sum of similarities between each frame and the remaining frames, and filter them to obtain the remaining frame set; Step 4: Calculate the sum of similarities between each frame in the moderately relevant frame set and all frames in the remaining frame set, and screen each frame in the moderately relevant frame set to obtain a candidate frame set; add all frames in the candidate frame set to the remaining frame set to obtain a key frame set; Step 5: Reshape the visual feature representation of each frame in the key frame set into a video feature map, and calculate the correlation score with the text features to determine the largest connected area; based on the largest connected area, determine the center coordinates and area size of the region of interest, and crop the video feature map to determine the region of interest; The region of interest is flattened into a set of embedding vectors, which are then embedded and combined with text features and input into a large language model for processing to obtain the final answer.

2. The method of dual-relationship time alignment and adaptive region optimization according to claim 1, characterized in that: In the step 1, a video and a question prompt text are obtained, and a visual encoder is used to extract features from the video to obtain visual features, and a text encoder is used to extract features from the question prompt text to obtain text features. Specifically, the steps include the following: Get the video and question prompt text, use the frame sampler to sample the video, and obtain an image sequence; Use the visual encoder to extract features from the image sequence to obtain visual features; The text encoder is used to extract features from the question prompt text to obtain text features.

3. The method of dual-relationship time alignment and adaptive region optimization according to claim 2, characterized in that: In the step of extracting features from image sequences using a visual encoder to obtain visual features, the following relationship exists: ； in, Represents visual features, Indicates video, Indicates that after the visual encoder feature extraction, represents an image sequence, Both represent each frame of sampled video, Indicates the total number of frames sampled; In the step of extracting features from the question prompt text using the text encoder to obtain text features, the following relationship exists: ； in, Represents text features, Indicates that the feature extraction is done by the text encoder. Indicates the question prompt text.

4. The method of dual-relationship time alignment and adaptive region optimization according to claim 3, characterized in that: In step 2, the visual features are subjected to global average pooling processing and cosine similarity calculation is performed with the text features to obtain a correlation score between the visual feature representation and the text features of each frame. Screening is performed based on the correlation score between the visual feature representation and the text features of each frame to obtain a highly correlated frame set and a medium correlated frame set, which specifically includes the following sub-steps: Perform global average pooling on the visual features to obtain a set of visual feature representations for each frame; Based on the set of visual feature representations of each frame, the cosine similarity between the visual feature representation of each frame and the text feature is calculated to obtain the correlation score between the visual feature representation of each frame and the text feature; The frames in the set of visual feature representations of each frame are sorted from large to small according to the correlation score between the visual feature representation of each frame and the text feature, and the top frames are selected. frames as high correlation frames to obtain a high correlation frame set; filter out the to frames as medium-correlation frames to obtain a medium-correlation frame set, and discard all remaining frames; in, represents the total number of frames in the set of highly correlated frames, Indicates the total number of frames in the medium-correlated frame set.

5. The method of dual-relationship time alignment and adaptive region optimization according to claim 4, characterized in that: In the step of calculating the cosine similarity between the visual feature representation of each frame and the text feature based on the set of visual feature representations of each frame to obtain a set of correlation scores between the visual feature representation of each frame and the text feature, the following relationship exists: ； in, Indicates the The correlation score between the visual feature representation of the frame and the textual features, Indicates the Visual feature representation of the frame, Indicates taking the two norm, Indicates the time index of the frame.

6. The method of dual-relationship time alignment and adaptive region optimization according to claim 5, characterized in that: In step 3, based on the highly correlated frame set, the sum of similarities between each frame and the remaining frames is calculated, and screening is performed to obtain the remaining frame set, which specifically includes the following sub-steps: Based on the set of highly correlated frames, the sum of the similarities between each frame and all other frames is calculated to obtain the similarity score of each frame in the set of highly correlated frames. The following relationship exists in the corresponding process: ； in, Indicates the first The similarity score of each frame, Indicates the first The visual feature representation of each frame, Indicates the first Visual feature representation of frames; Based on the similarity score of each frame in the highly correlated frame set, the frames in the highly correlated frame set are sorted from small to large, and the top frames in the highly correlated frame set are discarded. frames to obtain the remaining frame set; in, Indicates the total number of frames in the remaining frame set.

7. The method of dual-relationship time alignment and adaptive region optimization according to claim 6, characterized in that: In step 4, the sum of similarities between each frame in the moderately relevant frame set and all frames in the remaining frame set is calculated, and each frame in the moderately relevant frame set is screened to obtain a candidate frame set; all frames in the candidate frame set are added to the remaining frame set to obtain a key frame set, which specifically includes the following sub-steps: The sum of similarities between each frame in the moderately correlated frame set and all frames in the remaining frame set is calculated to obtain a similarity score for each frame in the moderately correlated frame set. The following relationship exists in the corresponding process: ； in, Indicates the number of frames in the medium correlation frame set The similarity score of each frame, Indicates the number of frames in the medium correlation frame set The visual feature representation of each frame, Indicates the remaining frame set Visual feature representation of frames; Based on the similarity score of each frame in the medium-correlated frame set, the frames in the medium-correlated frame set are sorted from small to large, and the top frames in the medium-correlated frame set are discarded. frames to obtain a set of candidate frames; All frames in the candidate frame set are added to the remaining frame set to obtain a key frame set.

8. The method of dual-relationship time alignment and adaptive region optimization according to claim 7, characterized in that: In step 5, the visual feature representation of each frame in the key frame set is reshaped into a video feature map, and a correlation score is calculated with the text features to determine the maximum connected area; the center coordinates and area size of the region of interest are determined based on the maximum connected area, and the video feature map is cropped to determine the region of interest; the region of interest is flattened into a set of embedding vectors, which are embedded and combined with the text features and then input into the large language model for processing to obtain the final answer. The specific sub-steps include the following: Based on the key frame set, the visual feature representation of each frame is spatially reshaped to obtain a video feature map; Based on the video feature map, the cosine similarity between the feature vector at each spatial position and the text feature is calculated to obtain the correlation score between the video feature map and the text feature; Based on the correlation score between the video feature map and the text feature, the feature vectors of each position in the video feature map are sorted from large to small, and the top The spatial heat map is generated by the features, and the connected domain analysis algorithm is used to analyze the spatial heat map to determine the largest connected area; Calculate the average position of the features in the largest connected region and obtain the center coordinates of the region of interest; The maximum vertical span of the largest connected area is used as the height of the region of interest, and the maximum horizontal span of the largest connected area is used as the width of the region of interest to obtain the area size of the region of interest; The video feature map is cropped based on the center coordinates and area size of the region of interest to determine the region of interest; The region of interest is flattened into a set of embedding vectors, which are then embedded and combined with text features and input into a large language model for processing to obtain the final answer.

9. The method of dual-relationship time alignment and adaptive region optimization according to claim 8, characterized in that: In the step of calculating the cosine similarity between the feature vector at each spatial position and the text feature based on the video feature map to obtain the correlation score between the video feature map and the text feature, the following relationship exists: ； in, Indicates the spatial position of the video feature map The correlation score between the feature vector and the text feature, Indicates the spatial position of the video feature map The video feature vector on represents the height of the visual feature map, Indicates the width of the visual feature map; Based on the correlation score between the video feature map and the text feature, the feature vectors of each position in the video feature map are sorted from large to small, and the top Features are used to generate a spatial heat map, and the connected domain analysis algorithm is used to analyze the spatial heat map to determine the maximum connected area. ; in, The number of feature vectors representing the positions within the selected video feature map, Indicates the weight value; In the step of calculating the average position of features within the largest connected region and obtaining the center coordinates of the region of interest, the following relationship exists: ； in, represents the center coordinates of the region of interest, represents the number of features in the largest connected region, Indicates the first The height of the feature, Indicates the maximum connected area The width of the feature; In the step of using the maximum vertical span of the largest connected area as the height of the region of interest and the maximum horizontal span of the largest connected area as the width of the region of interest to obtain the region size of the region of interest, the following relationship exists: ； in, represents the height of the region of interest, represents the width of the region of interest, represents the maximum vertical span of the largest connected area, Indicates the maximum horizontal span of the largest connected region.

10. A dual-relationship time alignment and adaptive region optimization system, characterized in that: The system applies the dual-relationship time alignment and adaptive region optimization method according to any one of claims 1 to 9, and the system includes: Feature extraction module, used to: Obtain a video and a question prompt text, use a visual encoder to extract features from the video to obtain visual features, and use a text encoder to extract features from the question prompt text to obtain text features; Dual-correlated time sampling module for: The visual features are globally averaged pooled and cosine similarity is calculated with the text features to obtain the correlation score between the visual feature representation and the text features of each frame. The frames are then filtered based on the correlation score between the visual feature representation and the text features of each frame to obtain a set of highly correlated frames and a set of moderately correlated frames. Based on the highly correlated frame set, the sum of similarities between each frame and the remaining frames is calculated and filtered to obtain the remaining frame set; Calculating the sum of similarities between each frame in the moderately correlated frame set and all frames in the remaining frame set, and screening each frame in the moderately correlated frame set to obtain a candidate frame set; adding all frames in the candidate frame set to the remaining frame set to obtain a key frame set; Dynamic spatial sampling module for: The visual feature representation of each frame in the key frame set is reshaped into a video feature map, and the correlation score is calculated with the text features to determine the largest connected area; the center coordinates and area size of the region of interest are confirmed based on the largest connected area, and the video feature map is cropped to determine the region of interest; the region of interest is flattened into a set of embedding vectors, which are embedded and combined with the text features and then input into the large language model for processing to obtain the final answer.

Images