A method and device for retrieving an enhanced cross-frame semantic cache video recovery agent

Abstract

The application discloses a retrieval enhanced cross-frame semantic cache video restoration intelligent agent method and device, and belongs to the technical field of computer vision, video enhancement and video restoration. An input degraded video is acquired, quality alignment features and degradation representation information are extracted, and the quality alignment features are used as query vectors to perform similarity retrieval in a retrieval enhancement memory. If the similarity is not lower than a threshold, a historical optimal restoration tool track is directly reused as a restoration plan, otherwise a tool scheduling sequence is generated. According to the plan or the sequence, multiple restoration tools are called to process each frame step by step, and current frame semantic features and degradation parameters are extracted in the process and matched with cross-frame semantic cache. If the matching is successful, a historical frame tool sequence is reused to process the current frame and subsequent frames, otherwise a new sequence is generated and the cache is updated, and finally a restored video is output. The application is used for reducing the additional calculation overhead generated by cross-frame redundant calculation of the degraded video, controlling the calculation resource consumption in the video restoration process, and obtaining significant effects.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, video enhancement, and video restoration, and in particular to a method and apparatus for retrieving enhanced cross-frame semantic cache video restoration intelligent agents. Background Technology

[0002] Video restoration and enhancement aims to recover high-quality visual content from degraded videos, improving performance in areas such as sharpness, noise suppression, detail fidelity, and temporal consistency. It is widely used in tasks such as super-resolution, deblurring, denoising, and low-light enhancement. Compared to static image restoration, video restoration also requires handling inter-frame information propagation, motion compensation, and cross-frame consistency constraints, making the task more complex. Existing methods are mainly based on deep learning and fall into two categories: one type designs dedicated networks for single degradations, whose performance depends on preset degradation types and is difficult to adapt to scenarios with multiple degradations overlapping or varying distributions; the other type is an integrated model for multiple degradations, which, while offering improved generalization ability, has a fixed network structure and inference process, making it impossible to dynamically adjust according to the degradation composition and intensity of the video. In real-world applications, multiple sources of degradation, such as low light, rain, blur, noise, and compression artifacts, often coexist and are time-varying and coupled. Fixed models or processes can easily lead to insufficient restoration, over-enhancement, and mutual interference between degradations.

[0003] Furthermore, existing methods lack explicit modeling and dynamic decision-making capabilities for degraded states, making it difficult to adaptively select the optimal restoration tools and order. For videos with similar degradation, there is a lack of historical experience reuse mechanisms, leading to redundant computations. In video stream processing, adjacent frames have highly similar semantics and degradation states; processing each frame independently cannot reuse decision results, resulting in additional consumption of computational resources and affecting the temporal consistency of the output video. In recent years, the development of multimodal large models and intelligent agent technologies has provided a technical foundation for constructing a unified framework for video restoration with perception, decision-making, and execution capabilities.

[0004] Therefore, a technical solution is needed that can dynamically adjust the processing path based on degradation characteristics and reuse historical processing experience. Summary of the Invention

[0005] The present invention aims to at least partially solve one of the technical problems in the related art.

[0006] To address this, this invention proposes a retrieval-enhanced cross-frame semantic cache video restoration agent method. This method acquires the input degraded video and extracts quality alignment features and degradation characterization information. The quality alignment features are used as query vectors for similarity retrieval in the retrieval enhancement memory. If the similarity is not lower than a threshold, the historical best restoration tool trajectory is directly reused as the restoration plan; otherwise, a tool scheduling sequence is generated. Multiple restoration tools are sequentially invoked according to the restoration plan or scheduling sequence to restore each frame step by step. During processing, the semantic features and degradation parameters of the current frame are extracted and matched with the cross-frame semantic cache. If a match is successful, the tool invocation sequence of the historical frame is reused; otherwise, a new sequence is generated and the cache is updated, ultimately outputting the restored video. This invention enables the reuse of historical processing experience and the utilization of cross-frame redundant information to control computational resource consumption and maintain temporal consistency during video restoration.

[0007] Another objective of this invention is to propose a smart agent device for retrieving enhanced cross-frame semantic cached video.

[0008] To achieve the above objectives, this invention proposes a method for retrieving enhanced cross-frame semantic cached video restoration intelligent agents, comprising:

[0009] The input degraded video is obtained, and the corresponding video-level quality alignment features and degradation characterization information are extracted. The quality alignment features are then used as query vectors to be input into the retrieval enhancement memory. Perform a similarity search in the enhanced memory database to obtain the most similar historical degradation video and the corresponding optimal restoration tool trajectory; If the search similarity is not lower than the preset threshold, the optimal restoration tool trajectory is directly reused as the restoration plan; otherwise, the degradation representation information is input into the video restoration agent to generate a restoration tool scheduling sequence. According to the restoration plan or scheduling sequence, multiple restoration tools are called sequentially to perform step-by-step restoration processing on each frame of the input degraded video; during the step-by-step restoration process, semantic features and degradation parameters are extracted for the current frame and matched with the constructed cross-frame semantic cache; If a match is successful, the tool call sequence corresponding to the historical frame in the cache is reused to process the current frame and subsequent frames; if a match fails, a new tool call sequence is generated for the current frame and subsequent frames and the cross-frame semantic cache is updated; finally, the restored video is output.

[0010] The intelligent agent method for retrieving and enhancing cross-frame semantic cached video in an embodiment of the present invention may also have the following additional technical features: In one embodiment of the present invention, extracting video-level quality alignment features and degradation characterization information includes: The input degraded video is sampled from multiple frames to obtain several video frames; Each video frame is input into a feature extraction network for encoding to obtain frame-level feature vectors for each frame; The feature vectors of each frame are aggregated to generate video-level quality alignment features that can simultaneously characterize the degree of degradation and the semantic meaning of the content. The video-level quality alignment features are input into the degradation mapping function, and the corresponding degradation characterization information is output. The degradation characterization information includes the degradation type and its intensity parameters.

[0011] In one embodiment of the present invention, the retrieval enhancement memory is pre-built through the following steps: Multiple historical degraded video samples are obtained, and quality alignment features are extracted from each sample to obtain historical quality alignment features. For each historical degraded video sample, the optimal restoration tool trajectory is determined by progressively comparing the restoration results or historical annotation results; The historical quality alignment features of each sample are associated and mapped with the corresponding optimal restoration tool trajectory, and stored as a searchable memory structure.

[0012] In one embodiment of the present invention, performing a similarity search in a retrieval enhancement memory includes: The quality alignment features of the input degraded video are used as the query vector. Cosine similarity is calculated for each historical quality alignment feature in the memory. The historical degraded video with the largest cosine similarity value is selected as the most similar sample. The optimal restoration tool trajectory corresponding to the most similar sample is output as the retrieval result.

[0013] In one embodiment of the present invention, inputting degradation characterization information into a video restoration agent to generate a restoration tool scheduling sequence includes: The degradation characterization information of the input degraded video and the preset restoration task requirements are input into the video restoration agent. Then, based on the multimodal large model, reasoning and decision-making are performed on the degradation type, intensity and task requirements to generate an ordered restoration tool scheduling sequence. The restoration tool scheduling sequence includes multiple restoration tools and their corresponding execution order.

[0014] In one embodiment of the present invention, multiple restoration tools are sequentially invoked according to a restoration plan or scheduling sequence to perform step-by-step restoration processing on each frame of the input degraded video, including: The input degraded video is input to the first restoration tool specified by the restoration plan or scheduling sequence, and the first restoration tool performs the corresponding degradation enhancement processing and outputs the first stage intermediate result; The intermediate results of the first stage are used as input for the next restoration tool, and the second to the Kth restoration tools are called in sequence; each restoration tool receives the output of the previous stage, performs enhancement processing for a specific degradation type, and outputs the intermediate results of this stage; The result of the Kth restoration tool is used as the final restored video output.

[0015] In one embodiment of the present invention, the cross-frame semantic cache is constructed and updated through the following steps: During the step-by-step restoration process, for the currently processed video frame, the semantic features and degradation parameters in the video frame are extracted, and the actual tool call sequence used in the video frame is recorded; The semantic features, degradation parameters, and tool call sequences are associated and stored as a cache unit; wherein, the cache unit is organized in chronological order or by feature index. When subsequent frame matching fails and a new tool call sequence is generated, the new cache unit is added to the cross-frame semantic cache.

[0016] In one embodiment of the present invention, matching with an existing cross-frame semantic cache includes: Extract the semantic features and degradation parameters of the current frame and combine them into a matching query vector; The semantic similarity of the semantic features in the matching query vector is calculated one by one with the semantic features of each historical frame in the cache; the degradation consistency of the degradation parameters in the matching query vector is calculated one by one with the degradation parameters of each historical frame in the cache; and then the calculated semantic similarity and degradation consistency are weighted and fused to obtain the comprehensive matching score. Determine whether the overall matching score has reached the preset threshold. If it has, the matching is considered successful, and the tool call sequence corresponding to the historical matching frame is output; otherwise, the matching is considered unsuccessful.

[0017] In one embodiment of the present invention, semantic similarity is calculated using cosine similarity, and degradation consistency is calculated using cosine similarity or Euclidean distance between degradation parameter vectors; the weight coefficients α and β of the weighted fusion satisfy α+β=1, and both α and β are positive numbers.

[0018] To achieve the above objectives, another aspect of the present invention proposes a retrieval-enhanced cross-frame semantic cache video restoration intelligent agent device, comprising: The extraction module is used to acquire the input degraded video, extract the corresponding video-level quality alignment features and degradation characterization information, and input the quality alignment features as query vectors into the retrieval enhancement memory. The retrieval module is used to perform similarity retrieval in the retrieval enhancement memory to obtain the most similar historical degradation video and the corresponding optimal restoration tool trajectory; The decision module is used to directly reuse the optimal restoration tool trajectory as the restoration plan if the retrieval similarity is not lower than a preset threshold; otherwise, it inputs the degradation representation information into the video restoration agent to generate a restoration tool scheduling sequence. The restoration matching module is used to sequentially call multiple restoration tools to perform step-by-step restoration processing on each frame of the input degraded video according to the restoration plan or scheduling sequence. During the step-by-step restoration process, semantic features and degradation parameters are extracted for the current frame and matched with the constructed cross-frame semantic cache. The cached output module is used to process the current frame and subsequent frames by reusing the tool call sequence corresponding to the historical frames in the cache if the match is successful; if the match fails, it generates a new tool call sequence for the current frame and subsequent frames and updates the cross-frame semantic cache; and finally outputs the restored video.

[0019] This invention discloses a method and apparatus for video restoration using a retrieval-enhanced cross-frame semantic cache. By extracting the quality alignment features and degradation characterization information of the input degraded video, it performs similarity retrieval in a retrieval-enhanced memory to achieve intelligent reuse of historically optimal restoration tool trajectories. For cases of missed trajectories, the video restoration agent generates an adaptive tool scheduling sequence and constructs a cross-frame semantic cache during the gradual restoration process, enabling adjacent frames to match and reuse the tool call sequences of historical frames. This invention reduces redundant calculations caused by a lack of historical experience and unused cross-frame redundancy, and mitigates the risk of temporal inconsistencies. It provides a processing flow from degradation perception to video output, improving the computational resource utilization efficiency and output quality of the video restoration process, and enhancing the adaptability and engineering practical value of the method in scenarios with multiple degradation layers.

[0020] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0021] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart of a method for retrieving and enhancing cross-frame semantic cached video restoration intelligent agents according to an embodiment of the present invention; Figure 2 This is a system architecture diagram according to an embodiment of the present invention; Figure 3 This is a schematic diagram of a retrieval-enhanced cross-frame semantic cache video restoration intelligent agent device according to an embodiment of the present invention. Detailed Implementation

[0022] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0023] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0024] The following describes, with reference to the accompanying drawings, a method and apparatus for retrieving enhanced cross-frame semantic cached video restoration intelligent agents according to embodiments of the present invention.

[0025] The core idea of ​​this invention is to address the insufficient processing capabilities of fixed models caused by multi-source superposition and time-varying characteristics in degraded videos. It provides a technical solution by constructing an intelligent agent method covering the entire process of "degradation perception—experience retrieval—dynamic decision-making—cross-frame caching," utilizing historical processing experience and cross-frame redundant information. This solution dynamically adjusts the processing path according to the degradation state and records inter-frame processing strategies. It is also used to control the computational resource consumption during video restoration and maintain the temporal consistency of the output video.

[0026] The following describes, with reference to the accompanying drawings, a method and apparatus for retrieving enhanced cross-frame semantic cached video restoration intelligent agents according to embodiments of the present invention.

[0027] The present invention provides a method for retrieving enhanced cross-frame semantic cached video recovery intelligent agents, comprising the following steps: S1, obtain the input degraded video, extract the corresponding video-level quality alignment features and degradation characterization information, and input the quality alignment features as query vectors into the retrieval enhancement memory; S2, perform a similarity search in the enhanced memory database to obtain the most similar historical degraded video and the corresponding optimal restoration tool trajectory; S3. If the search similarity is not lower than the preset threshold, the optimal restoration tool trajectory is directly reused as the restoration plan; otherwise, the degradation representation information is input into the video restoration agent to generate a restoration tool scheduling sequence. S4, according to the restoration plan or scheduling sequence, calls multiple restoration tools in sequence to perform step-by-step restoration processing on each frame of the input degraded video; during the step-by-step restoration process, semantic features and degradation parameters are extracted for the current frame and matched with the constructed cross-frame semantic cache; S5. If the match is successful, the tool call sequence corresponding to the historical frame in the cache is reused to process the current frame and subsequent frames; if the match fails, a new tool call sequence is generated for the current frame and subsequent frames and the cross-frame semantic cache is updated; finally, the restored video is output.

[0028] In summary, the present invention provides a method for retrieving and enhancing cross-frame semantic caching for video restoration. This method constructs a decision module based on a multimodal large model and generates a restoration tool invocation sequence according to video degradation features. By constructing a mapping relationship between degraded video features and the optimal restoration tool trajectory, and introducing a cross-frame semantic caching mechanism, the video restoration process is transformed from a fixed model inference method to an agent scheduling process based on degradation perception and dynamic decision-making, achieving adaptive restoration of complex, multi-degraded videos.

[0029] Specifically, to achieve the above objectives, the present invention adopts the following technical solution: First, an offline retrieval enhancement memory is constructed to perform feature extraction and restoration analysis on historical degraded videos, establish a mapping relationship between the quality alignment features of degraded videos and the trajectory of the optimal restoration tool, and store them in association.

[0030] Secondly, multi-frame sampling and feature extraction are performed on the input degraded video to obtain video-level quality alignment features and degradation characterization information. Based on the quality alignment features, a similarity search is performed in the retrieval enhancement memory to determine the historical degraded video sample most similar to the input video.

[0031] Then, based on the similarity retrieval results, it is determined whether to trigger the single-step retrieval and restoration mechanism. When the matching degree between the input video and the historical degraded video meets the preset conditions, the optimal restoration tool trajectory of the corresponding historical sample is directly reused to restore the input video.

[0032] When the matching degree does not meet the preset conditions, the degradation characterization information of the input video and the restoration task requirements are input into the video restoration agent. The video restoration agent generates a restoration tool scheduling sequence and performs multi-stage restoration processing according to the scheduling sequence.

[0033] Furthermore, a cross-frame semantic cache is constructed during video stream processing to associate and store the semantic features, degradation parameters, and corresponding tool call sequences of historical frames, forming a mapping relationship between historical frames and restoration strategies.

[0034] For the current video frame, calculate its semantic consistency and degradation consistency with historical frames, and determine whether the cross-frame semantic cache is hit. If the cache is hit, reuse the tool call sequence corresponding to the historical frame; otherwise, generate a new tool call sequence and update the cross-frame semantic cache.

[0035] Finally, the restoration operators are executed sequentially on the input video or video frames according to the restoration tool trajectory or the tool call sequence to output the final restored video.

[0036] Furthermore, the process of constructing a retrieval enhancement memory library involves extracting quality alignment features from historical degraded videos and gradually restoring them by comparing or determining the corresponding optimal restoration tool trajectory based on historical annotation results. The quality alignment features are then associated with and stored with the optimal restoration tool trajectory to form a retrieval-enabled memory structure.

[0037] Furthermore, the process involves extracting features from the input degraded video, sampling frames from the input video, encoding features from the sampled frames, and aggregating them to obtain video-level quality alignment features that can simultaneously characterize the degree of video degradation and content information.

[0038] Furthermore, the similarity retrieval involves matching the quality alignment features of the input video with the features of historical videos in the memory bank, determining the most similar historical degraded video sample based on the matching results, and obtaining its corresponding optimal restoration tool trajectory.

[0039] Furthermore, the single-step retrieval and restoration mechanism, when the similarity between the input video and the historical video reaches a preset threshold, directly calls the optimal restoration tool trajectory of the corresponding historical video, and executes the restoration operator in the order of the tool trajectory to complete the video enhancement.

[0040] Furthermore, the cross-frame semantic caching mechanism extracts the semantic features and degradation parameters of historical frames, records the corresponding tool call sequence, and establishes a mapping relationship between historical frames and restoration strategies.

[0041] Furthermore, the cache reuse involves calculating the semantic similarity and degradation consistency between the current frame and historical frames. When the matching degree meets a preset condition, the historical tool call sequence is reused; otherwise, the tool call sequence is regenerated and the cache is updated.

[0042] Furthermore, the restoration process executes multiple restoration operators sequentially according to the tool call sequence, processes different degradations in stages, and outputs the final restored video.

[0043] This invention constructs a retrieval enhancement memory to explicitly model and reuse historical restoration experience, avoiding repeated searches for the optimal restoration path and reducing reasoning complexity. Through a single-step retrieval restoration mechanism, it directly reuses historically optimal tool trajectories in scenarios with similar degradation patterns, reducing multi-round reasoning and trial-and-error processes and improving restoration efficiency. Through a cross-frame semantic caching mechanism, it reuses restoration strategies between consecutive frames in video stream processing, reducing redundant computation and enhancing processing consistency. By modeling the video restoration process as a tool scheduling process driven by an intelligent agent, it achieves a shift from fixed-model restoration to adaptive intelligent agent restoration, thereby simultaneously improving restoration quality, computational efficiency, and temporal stability in complex multi-degradation scenarios.

[0044] Furthermore, such as Figure 1As shown, the implementation of the method in this embodiment of the invention specifically includes: S101. Extract quality alignment features and perceive degradation status of input degraded video to obtain video degradation representation, semantic information and restoration requirements; S102. Construct a retrieval enhancement memory for degraded video features and the trajectory of the optimal restoration tool, and perform similarity retrieval based on input video features to match historical degradation patterns; S103. Determine whether to trigger the single-step retrieval restoration mechanism based on the retrieval results. If the conditions are met, directly reuse the historical best restoration tool trajectory to generate a restoration plan. S104. When the search conditions are not met, input the video, degradation information and task requirements into the restoration agent to generate the restoration tool scheduling sequence for the current video. S105. Activate the corresponding restoration tools in sequence according to the restoration plan or tool scheduling sequence, perform step-by-step restoration processing on the input video and obtain intermediate results; S106. During the restoration process, record the semantic features, degradation parameters and tool call sequence of the current frame, construct a cross-frame semantic cache and establish a mapping relationship between historical frames and restoration strategies; S107. Perform semantic cache matching for subsequent video frames, and determine whether the cache is hit based on semantic consistency and degradation consistency, and reuse the historical tool sequence. S108. After completing the video restoration, output the final restored video, as well as the search matching results, tool call trajectory, and semantic cache reuse information.

[0045] In step S101, multi-frame sampling and feature extraction are performed on the input degraded video to obtain a unified feature representation that can simultaneously characterize image quality degradation and content semantics. Let the input video be:

[0046] in, This represents the m-th frame of the image. Each frame is encoded using a feature extraction network: .in This represents the feature extraction function, used to extract low-level statistical information and structural semantic information related to degradation. Further global aggregation of frame-level features yields video-level features.

[0047] To achieve structured degradation perception, the video degradation state is further mapped into a parameter vector: To represent the degradation type and corresponding intensity information of the video, the function... This is used to achieve degradation classification and parameter regression. It maps the input video from pixel space to degradation semantic space, providing a unified basis for subsequent retrieval matching and agent decision-making.

[0048] In step S102, the retrieval enhancement memory is constructed and matched to build a retrieval enhancement memory containing historical degraded videos and their optimal restoration strategies.

[0049] in, The quality characteristics of the historical video are represented by Ti, which represents the trajectory of the corresponding optimal restoration tool.

[0050] During the retrieval phase, historical degradation pattern matching is performed, matching the input video features with features from the memory database using cosine similarity. Further select the optimal match This enables the system to quickly find the most similar degradation pattern from historical data, thereby avoiding repeated exploration of the optimal recovery path.

[0051] In step S103, based on the above optimal matching result S Determine whether the single-step retrieval mechanism has been triggered:

[0052] When the conditions are met, the historical tool sequence can be reused directly: By using historical best strategies to replace complex reasoning processes, the complex multi-step reasoning process is transformed into a single retrieval operation, thereby significantly reducing computational overhead and improving processing efficiency.

[0053] In step S104, when a search fails, dynamic scheduling is performed using an intelligent agent:

[0054] Among them, the function Ψ( Taking into account video content, degradation characteristics, and task objectives, a sequence of restoration tools is generated. Adaptive modeling is performed for unknown and complex degradation scenarios to achieve flexible tool selection and sequence planning.

[0055] In step S105, the restoration operation is executed step by step according to the restoration tool scheduling sequence. For the k-th restoration stage, the output of the previous stage is used as input. In each stage, the restoration tool performs targeted processing for the corresponding degradation type, including but not limited to operations such as rain removal, low-light enhancement, deblurring, noise reduction, decompression, and super-resolution reconstruction, so that different types of degradation are gradually eliminated at different stages.

[0056] Through the aforementioned stepwise restoration mechanism, the complex multi-degeneration problem is decomposed into multiple sub-problems for processing, effectively decoupling the interference between different degradations and thus improving the overall restoration effect. Simultaneously, since the restoration path is generated by retrieval or the agent, the tool's execution order is targeted, avoiding over-processing or under-processing issues caused by fixed processing flows.

[0057] In step S106, a cross-frame semantic cache is constructed during the video stream restoration process to enable the reuse of restoration strategies between consecutive frames. Semantic cache construction is performed for the current processing frame; to reduce redundant computation, the semantic cache is constructed during video stream processing. ,in Represents semantic features, It is a degradation parameter. This step represents the restoration execution strategy. By recording historical decision information, it enables cross-frame experience reuse. The cache unit Ct is stored in a cross-frame semantic cache set and organized according to time order or feature indexing to support subsequent fast matching and retrieval. This step not only records the visual semantic information of historical frames but also their corresponding degradation states and restoration decision results, thus forming a complete perception-decision joint memory structure. Constructing the above-mentioned cross-frame semantic cache enables the system to directly reuse existing restoration strategies in subsequent frame processing, avoiding repeated execution of agent reasoning or multi-round tool search processes.

[0058] In step S107, for the current frame The system matches the current frame against historical frames in the cross-frame semantic cache to determine whether to reuse an existing recovery strategy. Specifically, this is done for the current frame. First, its semantic features and degradation parameters are extracted and represented as follows: and Compare the current frame with historical frames in the cache. Matching is performed by calculating the semantic features of the current frame. semantic features of historical frames Similarity is used to measure the degree of consistency between two frames in terms of scene structure and content; degradation consistency matching compares the degradation parameters of the current frame. Degradation parameters of historical frames This is used to measure the consistency between two frames in terms of degradation type and degradation intensity. Furthermore, semantic consistency and degradation consistency are weighted and fused to obtain a comprehensive matching score:

[0059] Where α and β are weighting coefficients, α and β satisfy α+β=1, and both α and β are positive numbers. When the matching score is not lower than a preset threshold, it is determined that the current frame hits the cross-frame semantic cache, and the tool call sequence of the corresponding historical frame is directly reused: T(t+1) = Tt. Otherwise, it is determined that the cache is not hit, and the restoration agent regenerates a new tool call sequence and writes the new sequence into the cache. This significantly reduces repeated inference and repeated tool search processes in consecutive frames, while ensuring that adjacent frames use consistent or similar restoration paths, thereby improving the temporal stability of the video and reducing flickering.

[0060] In step S108, the final restored video is output. and restoration path information This is used to build a retrieval memory, which describes the retrieval strategies and cache reuse during the restoration process. The restoration path information is associated with the corresponding video features and can be used to update or expand the retrieval enhancement memory, thereby achieving continuous optimization of the system.

[0061] The method of this invention effectively solves the problems of repeated calculations and timing inconsistencies caused by the lack of historical experience reuse and cross-frame redundancy utilization in existing methods. It realizes integrated processing of the entire process from degradation perception, experience retrieval, dynamic decision-making to cross-frame caching, significantly improving video restoration efficiency and output quality, and enhancing the adaptability and engineering practical value of the method in multiple degradation superposition scenarios.

[0062] like Figure 2 As shown, the system architecture of this embodiment includes: after the input module acquires the degraded video, the feature extraction module extracts video-level quality alignment features and degradation characterization information. The retrieval enhancement module performs a similarity search in the memory bank based on these features. If the conditions are met, the single-step scheduling module is triggered to directly reuse the historical best tool trajectory as the restoration plan; if the conditions are not met, the agent inference module is called to generate a tool scheduling sequence. The restoration execution module calls the restoration tools step by step according to the plan or sequence. During this process, the cache matching module matches the semantic features and degradation parameters of the current frame with the historical frames in the semantic cache module: if a match is found, the tool call sequence in the cache is reused; if a match is not found, a new sequence is generated and the semantic cache is updated. Finally, the result output module outputs the restored video. This architecture achieves closed-loop collaboration between experience reuse and cross-frame redundancy utilization.

[0063] To further illustrate the practical application of the technical solution of this invention in real physical scenarios, the following description, using typical mobile shooting scenarios, explains the implementation process, data settings, comparison with existing technologies, and technical effects of this invention. It should be understood that the following embodiments are for illustrative purposes only and do not constitute a limitation on the scope of protection of this invention.

[0064] In this embodiment, the invention is applied to video restoration captured by a mobile phone. The physical degradation in the video mainly originates from non-uniform motion blur caused by hand shake, brightness instability caused by automatic exposure changes, noise from high sensor gain, and detail loss due to platform compression. Since the hand motion trajectory changes continuously over time, if different restoration strategies are used frame by frame, inconsistent edge sharpening intensity and background texture flickering are likely to occur. This embodiment uses the publicly available RealBlur dataset and GoPro dataset to jointly construct test samples, including 50 video clips with two resolutions: 1280×720 and 1920×1080. After degradation perception of the input video, the system identifies it as a composite degradation of non-uniform motion blur + high ISO noise + slight compression, and matches similar historical trajectories in the retrieval enhancement memory. When the similarity is lower than the single-step retrieval threshold, the restoration agent generates a tool scheduling sequence for video stabilization or motion compensation, deblurring, temporal denoising, and detail enhancement based on the degradation parameters; when the semantic consistency and degradation consistency of subsequent frames meet the threshold conditions, the above sequence is directly reused in the cross-frame semantic cache.

[0065] Compared to the single-video deblurring model DeblurNet, this invention does not simply perform end-to-end mapping of blur. Instead, it first determines the primary and secondary relationships between blur, noise, and compression based on their physical causes, and then selects the corresponding tools and execution order. When the noise intensity is higher than the blur intensity, the system prioritizes temporal denoising to avoid amplifying noise in subsequent deblurring. When the motion blur intensity is higher, the system prioritizes motion compensation and deblurring to restore structural edges. Test results show that on a subset of videos with clear references, the single deblurring model achieves an average PSNR of 28.41 dB and an SSIM of 0.872, while the method of this invention achieves an average PSNR of 29.73 dB and an SSIM of 0.901. On videos without reference samples, the temporal distortion error of this invention is reduced by approximately 24.7% compared to frame-by-frame independent processing. These results demonstrate that this invention can adaptively select the restoration path based on the physical causes of degradation, avoiding the problem of insufficient handling of complex degradation by a single model.

[0066] To further verify the technical effectiveness of this invention compared to existing methods, in addition to the single video deblurring model DeblurNet, a cascaded scheme of the video denoising model RVDT and the deblurring model DeblurNet was selected as a comparison method. This cascaded scheme performs denoising and deblurring in a fixed order, achieving an average PSNR of 28.56 dB and an SSIM of 0.879. The comparison shows that while the fixed cascaded method can alleviate the composite degradation problem to some extent, its overall restoration effect is still limited due to the lack of dynamic modeling of the degradation state and the lack of utilization of historical restoration experience and cross-frame redundancy information. This invention achieves superior restoration results by combining retrieval enhancement with cross-frame semantic caching to achieve adaptive selection and reuse of the restoration path.

[0067] In the application scenarios of the embodiments of this invention, on publicly available datasets, compared to the single deblurring model DeblurNet, the average PSNR of this invention is improved from 28.41dB to 29.73dB, and the SSIM is improved from 0.872 to 0.901; compared to fixed cascaded denoising and deblurring schemes, PSNR is improved by 1.17dB, and SSIM is improved by 0.022. On unreferenced real-world videos, the temporal distortion error is reduced by approximately 24.7% compared to frame-by-frame independent processing. Therefore, this invention, through degradation-aware dynamic scheduling of tool sequences and utilizing a retrieval-enhanced memory and cross-frame semantic cache reuse restoration strategy, can effectively solve complex degradation problems such as non-uniform motion blur, high ISO noise, and compression, avoiding processing mismatch and temporal flicker caused by single models or fixed cascaded sequences. This allows for the control of computational resource consumption while achieving high accuracy and stability in the output video.

[0068] To achieve the above invention, such as Figure 3 As shown, this embodiment also provides a retrieval-enhanced cross-frame semantic cache video restoration intelligent agent device 10, which includes: The extraction module 100 is used to acquire the input degraded video, extract the corresponding video-level quality alignment features and degradation characterization information, and input the quality alignment features as a query vector into the retrieval enhancement memory.

[0069] The retrieval module 200 is used to perform similarity retrieval in the retrieval enhancement memory to obtain the most similar historical degradation video and the corresponding optimal restoration tool trajectory.

[0070] The decision module 300 is used to directly reuse the optimal restoration tool trajectory as the restoration plan if the retrieval similarity is not lower than a preset threshold; otherwise, it inputs the degradation representation information into the video restoration agent to generate a restoration tool scheduling sequence.

[0071] The restoration matching module 400 is used to sequentially call multiple restoration tools to perform step-by-step restoration processing on each frame of the input degraded video according to the restoration plan or scheduling sequence. During the step-by-step restoration process, semantic features and degradation parameters are extracted for the current frame and matched with the constructed cross-frame semantic cache.

[0072] The cache output module 500 is used to process the current frame and subsequent frames by reusing the tool call sequence corresponding to the historical frame in the cache if the match is successful; if the match fails, it generates a new tool call sequence for the current frame and subsequent frames and updates the cross-frame semantic cache; and finally outputs the restored video.

[0073] The apparatus of this invention effectively solves the problems of redundant calculation and timing inconsistency caused by the lack of historical experience reuse and cross-frame redundancy utilization in existing video restoration methods. It realizes integrated processing of the entire process from degradation perception, experience retrieval, dynamic decision-making to cross-frame caching, and significantly improves video restoration efficiency and output quality.

[0074] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0075] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A method for retrieving enhanced cross-frame semantic cached video recovery intelligent agents, characterized in that, include: The input degraded video is obtained, and the corresponding video-level quality alignment features and degradation characterization information are extracted. The quality alignment features are then used as query vectors to be input into the retrieval enhancement memory. Perform a similarity search in the enhanced memory database to obtain the most similar historical degradation video and the corresponding optimal restoration tool trajectory; If the search similarity is not lower than the preset threshold, the optimal restoration tool trajectory is directly reused as the restoration plan; otherwise, the degradation representation information is input into the video restoration agent to generate a restoration tool scheduling sequence. According to the restoration plan or scheduling sequence, multiple restoration tools are called sequentially to perform step-by-step restoration processing on each frame of the input degraded video; during the step-by-step restoration process, semantic features and degradation parameters are extracted for the current frame and matched with the constructed cross-frame semantic cache; If a match is successful, the tool call sequence corresponding to the historical frame in the cache is reused to process the current frame and subsequent frames; if a match fails, a new tool call sequence is generated for the current frame and subsequent frames and the cross-frame semantic cache is updated; finally, the restored video is output.

2. The method as described in claim 1, characterized in that, Extract video-level quality alignment features and degradation characterization information, including: The input degraded video is sampled from multiple frames to obtain several video frames; Each video frame is input into a feature extraction network for encoding to obtain frame-level feature vectors for each frame; The feature vectors of each frame are aggregated to generate video-level quality alignment features that can simultaneously characterize the degree of degradation and the semantic meaning of the content. The video-level quality alignment features are input into the degradation mapping function, and the corresponding degradation characterization information is output. The degradation characterization information includes the degradation type and its intensity parameters.

3. The method as described in claim 1, characterized in that, The retrieval enhancement memory is pre-built through the following steps: Multiple historical degraded video samples are obtained, and quality alignment features are extracted from each sample to obtain historical quality alignment features. For each historical degraded video sample, the optimal restoration tool trajectory is determined by progressively comparing the restoration results or historical annotation results; The historical quality alignment features of each sample are associated and mapped with the corresponding optimal restoration tool trajectory, and stored as a searchable memory structure.

4. The method as described in claim 1, characterized in that, Perform similarity retrieval within the enhanced retrieval memory, including: The quality alignment features of the input degraded video are used as the query vector. Cosine similarity is calculated for each historical quality alignment feature in the memory. The historical degraded video with the largest cosine similarity value is selected as the most similar sample. The optimal restoration tool trajectory corresponding to the most similar sample is output as the retrieval result.

5. The method as described in claim 1, characterized in that, The degradation characterization information is input into the video restoration agent to generate a restoration tool scheduling sequence, including: The degradation characterization information of the input degraded video and the preset restoration task requirements are input into the video restoration agent. Then, based on the multimodal large model, reasoning and decision-making are performed on the degradation type, intensity and task requirements to generate an ordered restoration tool scheduling sequence. The restoration tool scheduling sequence includes multiple restoration tools and their corresponding execution order.

6. The method as described in claim 1, characterized in that, According to the restoration plan or scheduling sequence, multiple restoration tools are sequentially invoked to perform step-by-step restoration processing on each frame of the input degraded video, including: The input degraded video is input to the first restoration tool specified by the restoration plan or scheduling sequence, and the first restoration tool performs the corresponding degradation enhancement processing and outputs the first stage intermediate result; The intermediate results of the first stage are used as input for the next restoration tool, and the second to the Kth restoration tools are called in sequence; each restoration tool receives the output of the previous stage, performs enhancement processing for a specific degradation type, and outputs the intermediate results of this stage; The result of the Kth restoration tool is used as the final restored video output.

7. The method according to claim 1, characterized in that, Cross-frame semantic caching is built and updated through the following steps: During the step-by-step restoration process, for the currently processed video frame, the semantic features and degradation parameters in the video frame are extracted, and the actual tool call sequence used in the video frame is recorded; The semantic features, degradation parameters, and tool call sequences are associated and stored as a cache unit; wherein, the cache unit is organized in chronological order or by feature index. When subsequent frame matching fails and a new tool call sequence is generated, the new cache unit is added to the cross-frame semantic cache.

8. The method according to claim 1, characterized in that, Matching with the established cross-frame semantic cache, including: Extract the semantic features and degradation parameters of the current frame and combine them into a matching query vector; The semantic similarity of the semantic features in the matching query vector is calculated one by one with the semantic features of each historical frame in the cache; the degradation consistency of the degradation parameters in the matching query vector is calculated one by one with the degradation parameters of each historical frame in the cache; and then the calculated semantic similarity and degradation consistency are weighted and fused to obtain the comprehensive matching score. Determine whether the overall matching score has reached the preset threshold. If it has, the matching is considered successful, and the tool call sequence corresponding to the historical matching frame is output; otherwise, the matching is considered unsuccessful.

9. The method as described in claim 8, characterized in that, Semantic similarity is calculated using cosine similarity, while degradation consistency is calculated using cosine similarity or Euclidean distance between degradation parameter vectors. The weight coefficients α and β of the weighted fusion satisfy α+β=1, and both α and β are positive numbers.

10. A retrieval-enhanced cross-frame semantic cache video restoration intelligent agent device, characterized in that, include: The extraction module is used to acquire the input degraded video, extract the corresponding video-level quality alignment features and degradation characterization information, and input the quality alignment features as query vectors into the retrieval enhancement memory. The retrieval module is used to perform similarity retrieval in the retrieval enhancement memory to obtain the most similar historical degradation video and the corresponding optimal restoration tool trajectory; The decision module is used to directly reuse the optimal restoration tool trajectory as the restoration plan if the retrieval similarity is not lower than a preset threshold; otherwise, it inputs the degradation representation information into the video restoration agent to generate a restoration tool scheduling sequence. The restoration matching module is used to sequentially call multiple restoration tools to perform step-by-step restoration processing on each frame of the input degraded video according to the restoration plan or scheduling sequence. During the gradual restoration process, semantic features and degradation parameters are extracted for the current frame and matched with the constructed cross-frame semantic cache; The cached output module is used to process the current frame and subsequent frames by reusing the tool call sequence corresponding to the historical frames in the cache if the match is successful; if the match fails, it generates a new tool call sequence for the current frame and subsequent frames and updates the cross-frame semantic cache; and finally outputs the restored video.

Images