Method and device for real-time video stream image stabilization and multi-magnification clear imaging

Abstract

This invention discloses a method and apparatus for real-time video stream image stabilization and multi-magnification clear imaging. It utilizes an intelligent video stream stabilization linkage module to perform real-time stabilization processing on the video stream, dynamically controlling the triggering of the recognition algorithm and achieving smooth front-end display. A video frame imaging clarity processing middleware selects the highest-resolution frame within the same field of view and outputs a clear image in response to shooting requests. A multi-magnification field-of-view switching module enables real-time seamless switching between different magnification fields of view. The real-time video stream image stabilization and multi-magnification clear imaging method and apparatus provided by this invention can automatically and intelligently overcome practical problems such as video jitter and image blur, consistently providing users with clear and accurate recognition results, thereby greatly improving the system's practicality and user experience satisfaction.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and intelligent image processing, and specifically relates to a method and apparatus for real-time video stream image stabilization and multi-magnification clear imaging. Background Technology

[0002] In scenarios involving real-time processing and analysis of video streams, such as intelligent video surveillance systems, real-time industrial vision inspection systems, and remote video analysis systems, current video stream-based image processing methods still face several technical challenges in practical applications, primarily including:

[0003] 1. Stability and resource consumption issues in real-time video stream recognition: During real-time video stream processing and recognition, imaging equipment is susceptible to interference from external environmental factors such as vibration, causing continuous video image jitter. This severely reduces the stability of the input image, thereby affecting the accuracy of subsequent AI recognition algorithms, with a typical recognition error rate of around 5%. Simultaneously, the real-time inference process consumes significant amounts of video memory and computing resources, limiting system performance under high load and long-term continuous operation, impacting user experience and overall operational smoothness.

[0004] 2. Quality degradation of real-time captured images: During dynamic shooting, vibration can also cause problems such as image blurring, geometric distortion, or offset, resulting in loss of image details and difficulty in feature extraction. This significantly increases the time required for subsequent manual review and analysis, and the accuracy of manual judgment decreases due to image quality issues, thus affecting the consistency of analysis results and overall processing efficiency.

[0005] 3. Lack of real-time multi-magnification field-of-view switching capability: During shooting, the system does not support real-time and dynamic switching of the field-of-view magnification during video stream processing, and cannot flexibly adjust the view range according to task requirements for fine observation. This deficiency limits the system's application capabilities in professional scenarios requiring multi-scale visual support, such as high-precision detection, microstructure recognition, and local feature analysis.

[0006] Therefore, there is an urgent need for a real-time video stream image stabilization and multi-magnification clear imaging method and device to achieve video stream stabilization, image clarity, and multi-magnification field of view switching. Summary of the Invention

[0007] Therefore, embodiments of the present invention provide a method and apparatus for real-time video stream image stabilization and multi-magnification clear imaging to solve at least one technical problem existing in the prior art.

[0008] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions: This invention provides a method for real-time video stream image stabilization and multi-magnification sharp imaging, the method comprising: The intelligent video stream stabilization linkage module performs real-time stabilization processing on the video stream, dynamically controls the triggering of the recognition algorithm, and achieves smooth front-end display. The video frame imaging clarity processing middleware filters the highest resolution frame under the same field of view and outputs a clear image in response to shooting requests. The multi-magnification field-of-view switching module enables real-time seamless switching between different magnifications.

[0009] In some embodiments, the video stream is stabilized in real time through an intelligent video stream stabilization linkage module, specifically including: For each frame of the input video stream, a field of view similarity analysis is performed. The ORB feature extraction and matching algorithm is used to dynamically determine whether consecutive frames belong to the same field of view. If the data is determined to be from the same field of view and the recognition results are consistent, then the data should not be sent to the recognition algorithm repeatedly. If the field of view is inconsistent, it is re-entered into the algorithm for recognition, and the stored field of view map ID and recognition result are updated. The sharpness score of each frame is calculated using the Laplace variance method, and the sharpness value and corresponding image of the current frame are recorded. When a video frame is determined to have inconsistent field of view, a ResNet-based deep feature recognition model is activated to analyze the current frame and return the recognition results to the backend system. The backend pushes the processed, stabilized field of view and recognition results to the frontend interface in real time to enable video display.

[0010] In some embodiments, the ORB feature extraction and matching algorithm is used to dynamically determine whether consecutive frames belong to the same field of view, specifically including: Detect salient key points in the image and calculate a binary descriptor for each key point; Calculate the Hamming distance between two binary descriptors as a similarity measure; The number of matching points with a Hamming distance below a threshold is counted to dynamically determine whether consecutive video frames belong to the same field of view.

[0011] In some embodiments, the Laplace variance method is used to calculate the sharpness score for each frame, specifically including: Convert an RGB image to a grayscale image; Applying Laplacian convolution to extract image edge texture features; The variance of pixel values ​​in the feature map is calculated and mapped to a score range of 0-100 to obtain a sharpness score.

[0012] In some embodiments, the ResNet-based deep feature recognition model employs the Mask R-CNN detection algorithm, specifically including: The image is input into the ResNet-FPN backbone network to generate a multi-scale feature pyramid. After the region proposal network generates candidate boxes, it simultaneously performs three prediction tasks for each candidate box: classification, bounding box regression, and mask segmentation. Output structured target information, which includes at least category, coordinates, and mask.

[0013] In some embodiments, the highest resolution frame within the same field of view is selected using video frame imaging clarity processing middleware, specifically including: The backend manages each frame of the video stream, extracts and maintains metadata, which includes at least the same field of view ID, field of view sharpness, sharp field of view, and sharp field of view recognition result. In continuous multi-frame analysis, the frame with the highest clarity in the same field of view is automatically selected, and its recognition result is used as the final output of the user's shot, and the corresponding metadata in the cache is updated. After the user clicks to take a picture on the front-end interface, the system quickly matches and extracts the clearest frame and its recognition result in the same field of view based on the stored metadata, and generates a clear captured image.

[0014] In some embodiments, a multi-magnification field-of-view switching module enables real-time seamless switching between different magnifications, specifically including: Receive a real-time 100X video stream and perform the following operations through parallel processing channels: Channel 1 buffers the original 100X video frames; Channel 2 performs high-ratio detail-preserving downsampling in real time to generate 40X video frames. Dual video streams are simultaneously output to the display system, while maintaining timestamp and content alignment; When a user requests to switch to the 40X view, the scheduling module immediately extracts the corresponding 40X video frame from the preprocessing cache to achieve seamless switching of the display stream, while continuously maintaining the 40X processing channel in the background.

[0015] The present invention also provides a real-time video stream image stabilization and multi-magnification clear imaging device, the device comprising: The intelligent video stream stabilization linkage module is used to stabilize the video stream in real time, dynamically control the triggering of the recognition algorithm, and achieve smooth front-end display. The video frame imaging clarity processing middleware is used to filter the highest resolution frame under the same field of view and output a clear image in response to shooting requests. The multi-magnification field-of-view switching module is used to achieve real-time seamless switching between different magnification fields of view.

[0016] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method described above.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0018] The real-time video stream image stabilization and multi-magnification clear imaging method and apparatus provided by the present invention perform real-time stabilization processing on the video stream through an intelligent video stream stabilization linkage module, dynamically control the triggering of the recognition algorithm and realize smooth front-end display; through video frame imaging clear processing middleware, the highest resolution frame under the same field of view is selected, and a clear image is output in response to shooting requests; and through a multi-magnification field of view switching module, real-time seamless switching between different magnification fields of view is realized.

[0019] This invention significantly improves video stability through the coordinated operation of the three modules mentioned above. It effectively overcomes image jitter caused by environmental vibrations through an intelligent filtering mechanism that uses both frame similarity and sharpness as criteria. It optimizes image sharpness and recognition accuracy by employing a sharpness-optimized shooting mechanism and a depth recognition model to improve imaging quality and algorithm accuracy. It supports real-time multi-magnification observation by using multi-resolution parallel processing and seamless switching technology to meet the field-of-view scaling needs in different scenarios. It also improves system resource efficiency by reducing redundant computation and video memory usage through dynamic frame management and caching mechanisms. Thus, the real-time video stream image stabilization and multi-magnification sharpness imaging method and device provided by this invention can automatically and intelligently overcome real-world problems such as video jitter and image blur, consistently providing users with clear and accurate recognition results, thereby greatly improving the system's practicality and user experience satisfaction. Attached Figure Description

[0020] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0021] The structures, proportions, sizes, etc. illustrated in this specification are only for the purpose of assisting those skilled in the art in understanding and reading the content disclosed herein, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0022] Figure 1 A flowchart of the real-time video stream image stabilization and multi-magnification clear imaging method provided by the present invention; Figure 2 This is a structural block diagram of the real-time video stream image stabilization and multi-magnification clear imaging device provided by the present invention. Figure 3 This is a structural block diagram of a computer device provided by the present invention. Detailed Implementation

[0023] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] In one specific implementation, such as Figure 1 As shown, the real-time video stream image stabilization and multi-magnification clear imaging method provided by the present invention includes the following steps: S110: The intelligent video stream stabilization linkage module performs real-time stabilization processing on the video stream, dynamically controls the triggering of the recognition algorithm, and achieves smooth front-end display. S120: The highest resolution frame in the same field of view is selected through video frame imaging clarity processing middleware, and a clear image is output in response to shooting requests; S130: Real-time seamless switching between different magnifications is achieved through a multi-magnification field-of-view switching module.

[0025] The method provided by this invention constructs a full-link solution for stabilization processing, sharpening output, and multi-magnification switching. It systematically solves industry pain points in real-time video stream processing, such as jitter affecting recognition accuracy, image blur reducing analysis efficiency, and fixed field of view limiting the observation range. While improving video stability and imaging clarity, it reduces redundant calculations by dynamically controlling the recognition algorithm trigger, optimizes the utilization of system memory and computing power resources, and achieves a dual satisfaction of smooth operation experience and professional observation needs. It is suitable for real-time video processing needs in multiple scenarios such as intelligent monitoring and industrial inspection.

[0026] In step S110, the video stream is stabilized in real time through the intelligent video stream stabilization linkage module, specifically including the following steps: S210: Perform field-of-view similarity analysis on each frame of the input video stream, and use ORB feature extraction and matching algorithm to dynamically determine whether consecutive frames belong to the same field of view; S220: If the same field of view is determined and the recognition results are consistent, avoid sending the data into the recognition algorithm repeatedly. S230: If the field of view is inconsistent, the algorithm will be re-entered for recognition, and the stored field of view map ID and recognition result will be updated. S240: The sharpness score of each frame is calculated using the Laplace variance method, and the sharpness value and corresponding image of the current frame are recorded. S250: When a video frame is determined to have inconsistent field of view, a ResNet-based deep feature recognition model is started to analyze the current frame and the recognition result is returned to the backend system. S260: The backend pushes the processed and stabilized field of view map and recognition results to the frontend interface in real time to enable video display.

[0027] In this way, the ORB feature matching algorithm accurately determines the same field of view, avoiding repeated recognition calculations under the same field of view, so as to reduce the memory usage; the Laplacian variance method quickly quantifies the image sharpness, providing an objective basis for frame selection; combined with the ResNet deep feature recognition model, it ensures the recognition accuracy when the field of view changes, and the recognition error rate is reduced in typical scenarios; the front-end and back-end collaboratively push stable images, effectively suppressing the screen jitter caused by environmental vibration, improving the smoothness of video display and the user viewing experience.

[0028] In step S210 above, the ORB feature extraction and matching algorithm is used to dynamically determine whether consecutive frames belong to the same field of view, specifically including: Detect salient key points in the image and calculate a binary descriptor for each key point; Calculate the Hamming distance between two binary descriptors as a similarity measure; The number of matching points with a Hamming distance below a threshold is counted to dynamically determine whether consecutive video frames belong to the same field of view.

[0029] The ORB feature extraction algorithm combines high efficiency and robustness. Its binary descriptor is small in size and has a fast matching speed, which significantly improves the processing efficiency compared to traditional feature matching algorithms. Hamming distance, as a similarity metric, can accurately reflect the degree of feature point matching and improve the judgment accuracy under the same field of view. By counting the number of matching points, continuous frame field of view judgment is realized, which provides a reliable basis for avoiding repeated recognition and reducing resource consumption, and ensures the real-time performance and accuracy of stable processing.

[0030] In step S240 above, the sharpness score for each frame is calculated using the Laplace variance method, specifically including: Convert an RGB image to a grayscale image; Applying Laplacian convolution to extract image edge texture features; The variance of pixel values ​​in the feature map is calculated and mapped to a score range of 0-100 to obtain a sharpness score.

[0031] In this way, converting RGB images to grayscale simplifies computational complexity, and the Laplacian operator can efficiently extract image edge texture features, accurately reflecting the image sharpness. Through variance calculation and score mapping, a quantitative assessment of sharpness is achieved, and the scoring results are intuitive and easy to understand. This method has low computational cost, with single-frame processing time controlled in milliseconds, and does not affect the real-time processing efficiency of the video stream. It provides a fast and reliable basis for subsequent selection of the highest-resolution frame and improvement of image quality.

[0032] In step S250, the ResNet-based deep feature recognition model employs the Mask R-CNN detection algorithm, specifically including: The image is input into the ResNet-FPN backbone network to generate a multi-scale feature pyramid. After the region proposal network generates candidate boxes, it simultaneously performs three prediction tasks for each candidate box: classification, bounding box regression, and mask segmentation. Output structured target information, which includes at least category, coordinates, and mask.

[0033] Because the ResNet-FPN backbone network can generate multi-scale feature pyramids, it can adapt to the recognition needs of targets of different sizes and improve the recognition ability of small targets and detailed features. It can simultaneously perform classification, localization and segmentation tasks, which can significantly improve processing efficiency compared with step-by-step processing algorithms. Moreover, the output structured target information is complete, providing high-precision data support for subsequent stable display and clear processing. The MaskR-CNN algorithm has powerful feature extraction and recognition capabilities, and can still maintain high recognition accuracy in complex scenes, ensuring recognition reliability when the field of view changes.

[0034] Specifically, the intelligent video stream stabilization and linkage module operates during the real-time detection and recognition phase of the video stream, employing a collaborative workflow of front-end display and back-end processing, which includes the following steps: S1: Backend intelligent stabilization processing; a) Perform field-of-view similarity analysis on each frame of the input video stream, using ORB feature extraction and matching algorithms to dynamically determine whether consecutive frames belong to the same field of view. If they are determined to be the same field of view and the recognition results are consistent, the data is avoided from being repeatedly sent to the recognition algorithm, reducing video memory usage; if the fields of view are inconsistent, the data is re-sent to the algorithm for recognition, and the stored field-of-view map ID and recognition results are updated. b) Video stream frame sharpness determination: The sharpness score of each frame is calculated using the Laplacian variance method (value range 0-100), and the sharpness value of the current frame and the corresponding image are recorded.

[0035] S2: Dynamic detection and recognition algorithm; when a video frame is determined to have inconsistent field of view, a ResNet-based deep feature recognition model is activated to analyze the current frame and return the recognition result to the backend system.

[0036] S3: The front end displays the video in real time; the back end pushes the processed and stabilized view map and recognition results to the front end interface in real time to achieve smooth and stable video display.

[0037] This intelligent video stream stabilization module effectively suppresses image jitter caused by environmental vibrations, improves the stability of video stream input and the smoothness of user viewing, and reduces video memory usage through intelligent frame filtering, thereby improving system resource utilization.

[0038] In step S210 above, the highest resolution frame within the same field of view is selected using video frame imaging clarity processing middleware, specifically including: The backend manages each frame of the video stream, extracts and maintains metadata, which includes at least the same field of view ID, field of view sharpness, sharp field of view, and sharp field of view recognition result. In continuous multi-frame analysis, the frame with the highest clarity in the same field of view is automatically selected, and its recognition result is used as the final output of the user's shot, and the corresponding metadata in the cache is updated. After the user clicks to take a picture on the front-end interface, the system quickly matches and extracts the clearest frame and its recognition result in the same field of view based on the stored metadata, and generates a clear captured image.

[0039] In this way, by maintaining standardized metadata, a data foundation is provided for frame selection and shooting response, realizing a collaborative architecture of front-end quick preview and back-end fine-tuning output; the highest resolution frame under the same field of view is automatically selected to avoid image blurring caused by vibration, the clarity of the captured image is significantly improved, and the error rate of manual review is reduced; the front-end responds quickly to shooting requests based on metadata, and the response time is shortened to the level of hundreds of milliseconds, taking into account both the smoothness of operation and the quality of output images, improving user experience and system analysis efficiency.

[0040] Specifically, the video frame image sharpening middleware is activated when the user triggers the shooting, and achieves image sharpening through front-end and back-end collaboration. The specific steps are as follows: S1: Backend processing and analysis; The system continuously manages each frame of the video stream, extracting and maintaining the following metadata: same field of view ID, field of view sharpness, sharp field of view, and sharp field of view recognition result. In continuous multi-frame analysis, the system automatically selects the frame with the highest sharpness under the same field of view, uses its recognition result as the final output of the user's shooting, and updates the corresponding metadata in the cache, providing a stable and high-quality image source for the shooting function.

[0041] S2 front-end triggers shooting; after the user clicks to shoot on the front-end interface, the system quickly matches and extracts the clearest frame and its recognition result under the same field of view based on the stored same field of view ID and field of view clarity data, and instantly generates a clear captured image.

[0042] In this way, the video frame imaging clarity processing middleware can significantly improve the clarity and recognition accuracy of captured images, avoid image blurring caused by vibration, shorten manual review time, and improve the accuracy of user perception and the overall efficiency of the system.

[0043] In step S130 above, the real-time seamless switching of different magnification fields of view is achieved through the multi-magnification field-of-view switching module, specifically including: Receive a real-time 100X video stream and perform the following operations through parallel processing channels: Channel 1 buffers the original 100X video frames; Channel 2 performs high-ratio detail-preserving downsampling in real time to generate 40X video frames. Dual video streams are simultaneously output to the display system, while maintaining timestamp and content alignment; When a user requests to switch to the 40X view, the scheduling module immediately extracts the corresponding 40X video frame from the preprocessing cache to achieve seamless switching of the display stream, while continuously maintaining the 40X processing channel in the background.

[0044] In this way, a dual-channel parallel processing mechanism is adopted to maintain both high- and low-resolution video streams simultaneously, ensuring real-time availability of different magnification views; high-ratio detail preservation downsampling reduces resolution while preserving image details to the maximum extent, and key features of the target can still be clearly presented in 40X view; the timestamps and content of the dual video streams are aligned, and the switching response time is short, achieving seamless switching without lag or delay; it meets the multi-level visual needs from global monitoring (40X) to local detail analysis (100X), expanding the system's application capabilities in professional scenarios such as high-precision detection and microstructure recognition.

[0045] Specifically, the multi-magnification field-of-view switching module allows users to dynamically switch between different magnifications in the real-time video stream, enabling seamless observation of both global and local details. The specific process is as follows: S1: Initialization and Preprocessing. The system receives a real-time 100X video stream (1920×1080 resolution) and performs the following operations through parallel processing channels: Channel 1: Buffers raw 100X video frames; Channel 2: Real-time high-ratio detail-preserving downsampling is performed to generate 40X video frames (768×432 resolution). The dual video streams are output to the display system simultaneously, while maintaining timestamp and content alignment.

[0046] S2: Real-time view switching response: When a user requests to switch to the 40X view, the scheduling module immediately extracts the corresponding 40X video frame from the preprocessing cache to achieve seamless switching of the display stream. At the same time, it continuously maintains the 40X processing channel in the background to ensure the continuity of video processing.

[0047] In this way, the multi-magnification field of view switching module can achieve smooth switching of multi-magnification fields of view, meeting the multi-level visual needs from global monitoring to local detail analysis.

[0048] To facilitate understanding, the following uses a specific application scenario as an example to briefly describe the overall implementation process of the method provided by this invention.

[0049] This embodiment aims to achieve image stabilization, clear imaging, and multi-magnification switching in digital images of blood cell slides, which is accomplished through the following steps: S1: Intelligent video stream stabilization linkage.

[0050] 1. Backend intelligent stabilization processing.

[0051] a) Perform field-of-view similarity analysis on each frame of the input video stream, and use the ORB feature extraction and matching algorithm to dynamically determine whether consecutive frames belong to the same field of view (default_image). If they are determined to belong to the same field of view and the recognition results (default_image_box_info) are consistent, then the frames are avoided from being repeatedly sent to the recognition algorithm to reduce memory usage; if the fields of view are inconsistent, they are re-sent to the algorithm for recognition, and the stored field-of-view map ID and recognition results are updated.

[0052] The ORB feature extraction and matching algorithm achieves efficient similarity analysis by generating and comparing binary descriptors of images. The algorithm first detects salient key points in the image (such as corners, edges, and other locations with obvious features), and then calculates a compact binary descriptor for each key point. This descriptor is generated by systematically sampling the grayscale values ​​of pixel pairs around the key point and comparing the brightness of each pair of pixels. The comparison results (greater than 1, less than 0) are concatenated into a fixed-length binary sequence (e.g., a 256-bit "10110011..." pattern). In the matching stage, the algorithm directly calculates the Hamming distance between two binary descriptors (i.e., the number of different corresponding bits in the two binary strings) as a similarity measure: the smaller the Hamming distance, the more similar the two feature points are. By counting the number of matching points with a Hamming distance below a threshold, the system can dynamically determine whether consecutive video frames belong to the same field of view, thereby intelligently controlling the triggering timing of subsequent recognition algorithms. In this process, the binary descriptor serves as the carrier of feature representation, while the Hamming distance is a direct measure of the similarity between carriers; together, they form the foundation for the algorithm's efficient operation.

[0053] b. Video Stream Frame Sharpness Determination: First, convert the view image to grayscale. Second, extract sharpness features to calculate the sharpness of the image's edges and textures. Then, calculate the variance of the pixel values ​​after processing with the Laplacian operator. The larger the variance value, the sharper the image, resulting in a clarity score (range 0-100). (Compare the scores of multiple images of the same scene; the one with the highest score is the sharpest.) The Laplacian variance-based clarity determination process uses a single frame of video as input and outputs a normalized clarity score through three core steps: first, the RGB image is converted to grayscale; second, the Laplacian operator is applied for convolution to extract image edge texture features; and finally, the pixel value variance of the feature map is calculated and mapped to a score range of 0-100. The clarity score output by this algorithm directly reflects the sharpness of the image—the larger the variance and the higher the score, the clearer the image. In practical applications, the system automatically selects the highest-scoring frame as the optimal processing target by comparing the clarity scores of consecutive frames in the same scene, ensuring recognition accuracy while avoiding redundant calculations on blurry frames.

[0054] 2. Dynamic Algorithm Detection and Recognition. When a video frame is determined to have inconsistent field of view, the input image for re-detection is first processed using the Mask R-CNN detection algorithm. A deep network (ResNet) extracts multi-scale features of the image (e.g., objects of sizes 320×180, 160×90, 80×45, 40×23, and 20×12), outputting feature maps of different levels of abstraction. Next, various sizes of bounding boxes are preset on the feature maps to quickly determine which boxes may contain objects and output them. Then, each candidate box is simultaneously classified, located, and segmented. Finally, the results are merged and output, and the recognition results are returned to the backend system.

[0055] In the implementation of the Mask R-CNN detection algorithm, when a video frame is determined to have inconsistent field of view, the system initiates the Mask R-CNN detection process: First, the image is input into the ResNet-FPN backbone network, which outputs feature maps at five scales through a multi-level residual structure and feature pyramid (e.g., when inputting a 1920×1200 image, multi-resolution feature pyramids are generated from 320×18, 160×90, 80×45, 40×23, and 20×12). Based on these features, after the region proposal network generates candidate boxes, the system simultaneously performs three prediction tasks for each candidate box: 1) classification task to determine the object category; 2) bounding box regression task to refine the coordinate position; 3) mask segmentation task to generate pixel-level contours. Finally, structured target information (category / coordinates / mask) is output, and this result is returned to the backend system as a new benchmark, completing the closed loop from scene perception to data update.

[0056] 3. Real-time dynamic display on the front end: The back end pushes the processed and stabilized field of view map and recognition results to the front end interface in real time, achieving smooth and stable video display.

[0057] S2: Video frame imaging clarity processing middleware.

[0058] The analyzer performs the following operations during single image capture: 1. Backend Algorithm Intelligent Analysis: First, manage each frame of the video stream and obtain metadata of the field of view (LAD): LAD ID, LAD sharpness, sharp LAD, and sharp LAD recognition result; when continuously analyzing multiple LADs in the video stream, compare the sharpness values ​​of multiple LADs. If the value is the highest, the LAD algorithm recognition result of the sharpest LAD in the same field of view is determined to be the result of the user's photo; update the cache and return to the frontend to load video frames, managing each frame of the video stream to provide stable and clean raw materials for subsequent shooting analysis; 2. Front-end shooting: After clicking to capture the field of view on the page, based on the analyzer's default data ("same field of view ID", "field of view sharpness"), the "clicked field of view" is compared with the analyzer's default image data. The "field of view sharpness" value of the same field of view ID is the highest. If the value is the highest, the field of view image with the clearest image between the current shooting field of view and the default metadata is quickly generated to produce the user's photo result. Among these: a) Check if the cache already contains ("same view map ID", "view map clarity"); if not, return the page response directly. b) If there is a cache, start the post-processing logic: retrieve the cached data, store the high-resolution image information locally, and respond to the page.

[0059] Among them, "same field of view ID" refers to the field of view ID (value range 0-2^63-1), and "field of view sharpness" refers to the image sharpness (value range 0-100, used to identify the calibration benchmark for the clear result of the recognition).

[0060] S3: Freely switch between multiple magnification views This module allows users to dynamically switch between different magnifications in a real-time video stream, enabling seamless observation of both global and local details. The specific process is as follows: 1. Initialization and Preprocessing: The system receives a real-time 100X video stream (1920×1080 resolution) and performs the following operations through parallel processing channels: a) Channel 1: Buffers the original 100X video frames; b) Channel 2: Real-time high-ratio detail-preserving downsampling is performed to generate 40X video frames (resolution 768×432). c) The dual video streams are output to the display system simultaneously, while keeping the timestamps aligned with the content.

[0061] Real-time view switching response: When a user requests to switch to the 40X view, the scheduling module immediately extracts the corresponding 40X video frame from the preprocessing cache to achieve seamless switching of the display stream. At the same time, the 40X processing channel is continuously maintained in the background to ensure the continuity of video processing.

[0062] In general, the real-time video stream image stabilization and multi-magnification clear imaging system provided by this invention constitutes a complete intelligent video processing system architecture, data flow and working method through intelligent video stream stabilization linkage, video frame imaging clear processing middleware, and free switching of multi-magnification field of view.

[0063] The intelligent video stream stabilization linkage dynamically controls the triggering conditions of the recognition algorithm by performing real-time field-of-view similarity analysis (ORB feature matching) and sharpness determination (Laplacian variance method) on the video stream, reducing redundant calculations, synchronously generating and maintaining structured metadata, avoiding the loss of source information, and providing calibration benchmarks. The video frame imaging sharpness processing middleware quickly generates an initial preview based on metadata at the front end and triggers the capture and saving of video stream frames. The back end accurately calculates and saves image information through algorithm post-processing. The front end focuses on response speed and user experience, while the back end ensures image quality and processing accuracy, achieving a balance between "smooth preview" and "sharp output." The multi-magnification field-of-view switching adopts a dual-channel processing mechanism, simultaneously maintaining the original high-resolution stream and the real-time downsampling stream, enabling smooth multi-magnification field-of-view switching to meet multi-level visual needs from global monitoring to local detail analysis.

[0064] In this way, the method uses intelligent triggering and metadata management to dynamically control the triggering timing of the recognition algorithm through real-time image analysis (similarity + clarity), avoiding invalid calculations. Simultaneously, standardized video metadata is generated to provide a calibration benchmark for all subsequent processing. The front-end and back-end collaborative image enhancement processing adopts a division of labor architecture of "front-end quick preview, back-end fine-tuning and output." The front-end ensures smooth operation, while the back-end ensures output quality, achieving a "what you see is what you get" high-definition result through collaboration. Dual-channel multi-magnification field-of-view switching, by simultaneously maintaining high- and low-resolution dual video streams, achieves seamless, smooth, and rapid field-of-view scaling from global overview to local details, meeting multi-level observation needs.

[0065] Based on the above embodiments, the method provided by the present invention has the following technical effects: Data-driven image stabilization: elevates image stabilization control from the traditional image processing level to the data stream management level; Collaborative processing architecture: Achieving a balance between speed and accuracy through front-end and back-end division of labor; Seamless experience design: Enables real-time switching between multiple processing speeds while ensuring processing quality; Resource efficiency optimization: Reduce system load and improve scalability through intelligent frame management.

[0066] This overall framework systematically solves long-standing industry pain points in real-time video processing, such as "jitter affecting recognition, blurring reducing quality, and fixed field of view limiting observation," and has significant technological advancements and application value.

[0067] Furthermore, through the coordinated operation of the above three modules, the present invention achieves the following comprehensive benefits: Significantly improves video stability: The intelligent filtering mechanism, which combines frame similarity and sharpness, effectively overcomes the problem of image jitter caused by environmental vibration. Optimize image sharpness and recognition accuracy: Improve imaging quality and algorithm accuracy by using a sharpness-optimized shooting mechanism and depth recognition model; Supports real-time multi-magnification observation: Through multi-resolution parallel processing and seamless switching technology, it meets the field-of-view scaling requirements in different scenarios; Improve system resource efficiency: Reduce redundant calculations and video memory usage by using dynamic frame management and caching mechanisms.

[0068] The ultimate goal is to enable the system to automatically and intelligently overcome real-world problems such as video shakiness and image blurring, consistently providing users with clear and accurate recognition results, thereby greatly improving the system's usability and user experience satisfaction.

[0069] In the above specific embodiments, the real-time video stream image stabilization and multi-magnification clear imaging method provided by the present invention performs real-time stabilization processing on the video stream through an intelligent video stream stabilization linkage module, dynamically controls the triggering of the recognition algorithm and realizes smooth front-end display; filters the highest resolution frame under the same field of view through video frame imaging clear processing middleware, and outputs a clear image in response to shooting requests; and realizes real-time seamless switching of different magnification fields of view through a multi-magnification field of view switching module.

[0070] This invention significantly improves video stability through the coordinated operation of the three modules mentioned above. It effectively overcomes image jitter caused by environmental vibrations through an intelligent filtering mechanism that uses both frame similarity and sharpness as criteria. It optimizes image sharpness and recognition accuracy by employing a sharpness-optimized shooting mechanism and a depth recognition model to improve imaging quality and algorithm accuracy. It supports real-time multi-magnification observation by using multi-resolution parallel processing and seamless switching technology to meet the field-of-view scaling needs in different scenarios. It also improves system resource efficiency by reducing redundant computation and video memory usage through dynamic frame management and caching mechanisms. Thus, the real-time video stream image stabilization and multi-magnification sharp imaging method and device provided by this invention can automatically and intelligently overcome real-world problems such as video jitter and image blur, consistently providing users with clear and accurate recognition results, thereby greatly improving the system's practicality and user experience satisfaction.

[0071] In addition to the methods described above, the present invention also provides a real-time video stream image stabilization and multi-magnification clear imaging device, the device comprising: The intelligent video stream stabilization linkage module 210 is used to perform real-time stabilization processing on the video stream, dynamically control the triggering of the recognition algorithm, and achieve smooth front-end display. The video frame imaging clarity processing middleware 220 is used to filter the highest resolution frame under the same field of view and output a clear image in response to the shooting request. The multi-magnification field-of-view switching module 230 is used to achieve real-time seamless switching between different magnification fields of view.

[0072] In some embodiments, the video stream is stabilized in real time through an intelligent video stream stabilization linkage module, specifically including: For each frame of the input video stream, a field of view similarity analysis is performed. The ORB feature extraction and matching algorithm is used to dynamically determine whether consecutive frames belong to the same field of view. If the data is determined to be from the same field of view and the recognition results are consistent, then the data should not be sent to the recognition algorithm repeatedly. If the field of view is inconsistent, it is re-entered into the algorithm for recognition, and the stored field of view map ID and recognition result are updated. The sharpness score of each frame is calculated using the Laplace variance method, and the sharpness value and corresponding image of the current frame are recorded. When a video frame is determined to have inconsistent field of view, a ResNet-based deep feature recognition model is activated to analyze the current frame and return the recognition results to the backend system. The backend pushes the processed, stabilized field of view and recognition results to the frontend interface in real time to enable video display.

[0073] In some embodiments, the ORB feature extraction and matching algorithm is used to dynamically determine whether consecutive frames belong to the same field of view, specifically including: Detect salient key points in the image and calculate a binary descriptor for each key point; Calculate the Hamming distance between two binary descriptors as a similarity measure; The number of matching points with a Hamming distance below a threshold is counted to dynamically determine whether consecutive video frames belong to the same field of view.

[0074] In some embodiments, the Laplace variance method is used to calculate the sharpness score for each frame, specifically including: Convert an RGB image to a grayscale image; Applying Laplacian convolution to extract image edge texture features; The variance of pixel values ​​in the feature map is calculated and mapped to a score range of 0-100 to obtain a sharpness score.

[0075] In some embodiments, the ResNet-based deep feature recognition model employs the Mask R-CNN detection algorithm, specifically including: The image is input into the ResNet-FPN backbone network to generate a multi-scale feature pyramid. After the region proposal network generates candidate boxes, it simultaneously performs three prediction tasks for each candidate box: classification, bounding box regression, and mask segmentation. Output structured target information, which includes at least category, coordinates, and mask.

[0076] In some embodiments, the highest resolution frame within the same field of view is selected using video frame imaging clarity processing middleware, specifically including: The backend manages each frame of the video stream, extracts and maintains metadata, which includes at least the same field of view ID, field of view sharpness, sharp field of view, and sharp field of view recognition result. In continuous multi-frame analysis, the frame with the highest clarity in the same field of view is automatically selected, and its recognition result is used as the final output of the user's shot, and the corresponding metadata in the cache is updated. After the user clicks to take a picture on the front-end interface, the system quickly matches and extracts the clearest frame and its recognition result in the same field of view based on the stored metadata, and generates a clear captured image.

[0077] In some embodiments, a multi-magnification field-of-view switching module enables real-time seamless switching between different magnifications, specifically including: Receive a real-time 100X video stream and perform the following operations through parallel processing channels: Channel 1 buffers the original 100X video frames; Channel 2 performs high-ratio detail-preserving downsampling in real time to generate 40X video frames. Dual video streams are simultaneously output to the display system, while maintaining timestamp and content alignment; When a user requests to switch to the 40X view, the scheduling module immediately extracts the corresponding 40X video frame from the preprocessing cache to achieve seamless switching of the display stream, while continuously maintaining the 40X processing channel in the background.

[0078] In the above specific embodiments, the real-time video stream image stabilization and multi-magnification clear imaging device provided by the present invention performs real-time stabilization processing on the video stream through an intelligent video stream stabilization linkage module, dynamically controls the triggering of the recognition algorithm and realizes smooth front-end display; it filters the highest resolution frame under the same field of view through the video frame imaging clear processing middleware, and outputs a clear image in response to the shooting request; and it realizes real-time seamless switching of different magnification fields of view through the multi-magnification field of view switching module.

[0079] This invention significantly improves video stability through the coordinated operation of the three modules mentioned above. It effectively overcomes image jitter caused by environmental vibrations through an intelligent filtering mechanism that uses both frame similarity and sharpness as criteria. It optimizes image sharpness and recognition accuracy by employing a sharpness-optimized shooting mechanism and a depth recognition model to improve imaging quality and algorithm accuracy. It supports real-time multi-magnification observation by using multi-resolution parallel processing and seamless switching technology to meet the field-of-view scaling needs in different scenarios. It also improves system resource efficiency by reducing redundant computation and video memory usage through dynamic frame management and caching mechanisms. Thus, the real-time video stream image stabilization and multi-magnification sharp imaging method and device provided by this invention can automatically and intelligently overcome real-world problems such as video jitter and image blur, consistently providing users with clear and accurate recognition results, thereby greatly improving the system's practicality and user experience satisfaction.

[0080] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 3 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and model predictions. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The model predictions of the computer device store static and dynamic information data. The network interface of the computer device is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps in the above method embodiments.

[0081] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the computer device to which the present invention is applied. A specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0082] Corresponding to the above embodiments, this invention also provides a computer storage medium containing one or more program instructions. These one or more program instructions are used to execute the method described above.

[0083] The present invention also provides a computer program product, the computer program product including a computer program, the computer program being stored on a non-transitory computer-readable storage medium, and the computer being able to perform the above-described method when the computer program is executed by a processor.

[0084] In this embodiment of the invention, the processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0085] The various methods, steps, and logic diagrams disclosed in the embodiments of this invention can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The processor reads information from the storage medium and, in conjunction with its hardware, completes the steps of the above methods.

[0086] The storage medium can be memory, such as volatile memory or non-volatile memory, or may include both volatile and non-volatile memory.

[0087] Among them, non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory.

[0088] Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM).

[0089] The storage media described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0090] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this invention can be implemented using a combination of hardware and software. When applied as software, the corresponding functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transmission of computer programs from one place to another. Storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0091] The above specific embodiments further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for real-time video stream image stabilization and multi-magnification sharp imaging, characterized in that, The method includes: The intelligent video stream stabilization linkage module performs real-time stabilization processing on the video stream, dynamically controls the triggering of the recognition algorithm, and achieves smooth front-end display. The video frame imaging clarity processing middleware filters the highest resolution frame under the same field of view and outputs a clear image in response to shooting requests. The multi-magnification field-of-view switching module enables real-time seamless switching between different magnifications.

2. The real-time video stream image stabilization and multi-magnification clear imaging method according to claim 1, characterized in that, The video stream is stabilized in real time through an intelligent video stream stabilization linkage module, specifically including: For each frame of the input video stream, a field of view similarity analysis is performed. The ORB feature extraction and matching algorithm is used to dynamically determine whether consecutive frames belong to the same field of view. If the data is determined to be from the same field of view and the recognition results are consistent, then the data should not be sent to the recognition algorithm repeatedly. If the field of view is inconsistent, it is re-entered into the algorithm for recognition, and the stored field of view map ID and recognition result are updated. The sharpness score of each frame is calculated using the Laplace variance method, and the sharpness value and corresponding image of the current frame are recorded. When a video frame is determined to have inconsistent field of view, a ResNet-based deep feature recognition model is activated to analyze the current frame and return the recognition results to the backend system. The backend pushes the processed, stabilized field of view and recognition results to the frontend interface in real time to enable video display.

3. The real-time video stream image stabilization and multi-magnification clear imaging method according to claim 2, characterized in that, The ORB feature extraction and matching algorithm is used to dynamically determine whether consecutive frames belong to the same field of view, specifically including: Detect salient key points in the image and calculate a binary descriptor for each key point; Calculate the Hamming distance between two binary descriptors as a similarity measure; The number of matching points with a Hamming distance below a threshold is counted to dynamically determine whether consecutive video frames belong to the same field of view.

4. The real-time video stream image stabilization and multi-magnification clear imaging method according to claim 2, characterized in that, The sharpness score for each frame is calculated using the Laplace variance method, specifically including: Convert an RGB image to a grayscale image; Applying Laplacian convolution to extract image edge texture features; The variance of pixel values ​​in the feature map is calculated and mapped to a score range of 0-100 to obtain a sharpness score.

5. The real-time video stream image stabilization and multi-magnification clear imaging method according to claim 2, characterized in that, The ResNet-based deep feature recognition model employs the Mask R-CNN detection algorithm, specifically including: The image is input into the ResNet-FPN backbone network to generate a multi-scale feature pyramid. After the region proposal network generates candidate boxes, it simultaneously performs three prediction tasks for each candidate box: classification, bounding box regression, and mask segmentation. Output structured target information, which includes at least category, coordinates, and mask.

6. The real-time video stream image stabilization and multi-magnification clear imaging method according to claim 1, characterized in that, The highest-resolution frames within the same field of view are selected using video frame imaging clarity processing middleware, specifically including: The backend manages each frame of the video stream, extracts and maintains metadata, which includes at least the same field of view ID, field of view sharpness, sharp field of view, and sharp field of view recognition result. In continuous multi-frame analysis, the frame with the highest clarity in the same field of view is automatically selected, and its recognition result is used as the final output of the user's shot, and the corresponding metadata in the cache is updated. After the user clicks to take a picture on the front-end interface, the system quickly matches and extracts the clearest frame and its recognition result in the same field of view based on the stored metadata, and generates a clear captured image.

7. The real-time video stream image stabilization and multi-magnification clear imaging method according to claim 1, characterized in that, A multi-magnification field-of-view switching module enables real-time seamless switching between different magnifications, specifically including: Receive a real-time 100X video stream and perform the following operations through parallel processing channels: Channel 1 buffers the original 100X video frames; Channel 2 performs high-ratio detail-preserving downsampling in real time to generate 40X video frames. Dual video streams are simultaneously output to the display system, while maintaining timestamp and content alignment; When a user requests to switch to the 40X view, the scheduling module immediately extracts the corresponding 40X video frame from the preprocessing cache to achieve seamless switching of the display stream, while continuously maintaining the 40X processing channel in the background.

8. A device for real-time video stream image stabilization and multi-magnification clear imaging, characterized in that, The device includes: The intelligent video stream stabilization linkage module is used to stabilize the video stream in real time, dynamically control the triggering of the recognition algorithm, and achieve smooth front-end display. The video frame imaging clarity processing middleware is used to filter the highest resolution frame under the same field of view and output a clear image in response to shooting requests. The multi-magnification field-of-view switching module is used to achieve real-time seamless switching between different magnification fields of view.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method as described in any one of claims 1-7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-7.

Images