Image processing method and device based on AI frame alignment and blur elimination, equipment and medium

Abstract

The application relates to the technical field of image digitization, in particular to an image processing method and device based on AI frame alignment and blur elimination, equipment and a medium, which comprises the following steps: acquiring continuous multiple frames of image data, performing feature extraction on each frame of image data, the feature extraction comprises detecting image feature points by using a detection algorithm, and a binary feature vector of each image feature point is generated; the binary feature vector is input into a pre-trained lightweight neural network model for feature matching, a geometric transformation model between each frame is calculated, and a transformation matrix is obtained; according to the transformation matrix, geometric transformation is applied to the current frame, alignment with a reference frame is realized through inverse mapping and bilinear interpolation; through a content perception filling algorithm, a boundary missing area generated due to geometric transformation in the aligned image data is identified and repaired, and the aligned image data is filled. The application has the effect of improving the quality of a film and television work after scanning.

Description

Technical Field

[0001] This application relates to the technical field of image digitization, and in particular to an image processing method, apparatus, device and medium based on AI frame alignment and de-shaking. Background Technology

[0002] Among existing image digitization technologies, the 8mm film-to-digital video scanner has gained widespread attention in recent years for its efficient and high-quality preservation and restoration of historical images, particularly in cultural heritage protection, film archive restoration, and personal image digitization. Currently, this technology primarily uses a sophisticated opto-mechanical-electrical system to convert analog images on film into digital video. Typically, the scanner's core components include a film transport mechanism, illumination system, optical lens, image sensor, and analog-to-digital converter. Its advantages are: 1) High-precision mechanical and electronic control ensures smooth film transport and accurate positioning, reducing physical damage; 2) Advanced light sources and sensors ensure accurate color reproduction and high image clarity. This technology is mainly used in film archives, museums, film production companies, and for the digitization of film by private collectors.

[0003] In existing 8mm film-to-digital video scanner systems, the following key technologies are typically used to achieve the digitization process. For example, the film transport mechanism relies on a high-precision motor and gripper to smoothly and accurately pull the film, moving one frame at a time, and fixing the film with positioning pins during exposure to eliminate physical shake; the illumination system provides a uniform and stable backlight source, and modern professional scanners often use tri-color LED light sources, which can precisely control color temperature and brightness, reducing thermal damage to the film; the optical lens uses a high-quality cinematic fixed-focus lens to clearly project the image on the film onto the sensor; the image sensor typically uses a CCD or a global shutter CMOS, which converts the light signal passing through the film into an electrical signal, with the global shutter being able to read the entire frame simultaneously, avoiding the rolling shutter effect in high-speed motion images; and the analog-to-digital converter converts the analog electrical signal generated by the sensor into digital RAW data.

[0004] These existing scanner systems have a limitation: they lack image stabilization and alignment algorithms during video synthesis, potentially leading to shaky final digital videos. Specifically, because the original film may have experienced camera misalignment or mechanical shake during shooting, current technologies, which only generate video by cropping and stitching frame sequences, cannot correct for these accumulated minute displacements and vibrations. Therefore, the synthesized video exhibits unstable visual shakiness, severely impacting the viewing experience and image quality, especially with old, fragile, or handheld film. Summary of the Invention

[0005] To improve the quality of scanned film and television works, this application provides an image processing method, apparatus, device, and medium based on AI frame alignment and de-shaking.

[0006] The above-mentioned objective of this application is achieved through the following technical solution: An image processing method based on AI frame alignment and dejitter, the image processing method based on AI frame alignment and dejitter includes: Acquire continuous multi-frame image data, the image data being obtained from digital scanning of film, and perform feature extraction on each frame of the image data. The feature extraction includes detecting image feature points using a detection algorithm and generating a binary feature vector for each image feature point. The binary feature vector is input into a pre-trained lightweight neural network model for feature matching, and the geometric transformation model between each frame is calculated to obtain the transformation matrix. Based on the transformation matrix, a geometric transformation is applied to the current frame, and alignment with the reference frame is achieved through inverse mapping and bilinear interpolation; The content-aware fill algorithm identifies and repairs boundary missing regions caused by geometric transformations in aligned image data, and then fills the aligned image data.

[0007] By adopting the above technical solutions, multi-frame image data from digitally scanned film is acquired and feature points and their binary feature vectors are extracted, providing key clues for subsequent inter-frame matching and alignment. Inputting the binary feature vectors into a lightweight neural network model for feature matching and geometric transformation model calculation can accurately identify inter-frame offsets, avoiding the limitations of traditional algorithms in low-texture areas. Performing geometric transformation and bilinear interpolation on the current frame based on the transformation matrix to achieve alignment with the reference frame can effectively eliminate the impact of shooting shake. By identifying and repairing boundary missing areas through a content-aware filling algorithm, the effective image information of the original film can be preserved to the maximum extent, improving the quality of the processed image.

[0008] In a preferred embodiment, this application can be further configured such that: the feature extraction of each frame of image data specifically includes: Construct an image pyramid, including the original image of the image data and multi-scale downsampled images, for feature detection in a multi-scale space; Corner detection is performed on each of the aforementioned scale images, where the corner detection determination formula is: For a pixel p, if there are N consecutive pixels on its circumference with intensity I satisfying |I - I_p|>T, then the pixel p is determined to be a corner point, where N is a preset number of consecutive points; The threshold T is adaptively adjusted based on the local contrast of the image. The threshold T = k · σ, where σ is the standard deviation of the pixel intensity in the local region and k is the adjustment factor. The detected corner points are calculated to obtain the image feature points, and the binary feature vector is generated by comparing the pixel intensity difference of preset point pairs.

[0009] By adopting the above technical solutions, the image pyramid can be constructed to perform feature detection in multi-scale space, improving the comprehensiveness of feature extraction; the adaptively adjusted corner detection judgment formula and threshold can accurately determine corners based on the local contrast of the image, improving the accuracy of corner detection; the detected corners are calculated and binary feature vectors are generated, providing effective data support for subsequent feature matching and geometric transformation model calculation, which helps to achieve more accurate frame alignment and jitter reduction, and improve the quality and effect of image processing.

[0010] In a preferred embodiment, this application can be further configured as follows: the step of inputting the binary feature vector into a pre-trained lightweight neural network model for feature matching, and calculating the geometric transformation model between each frame to obtain the transformation matrix, specifically includes: The lightweight neural network model is used to extract deep features of the image, including low-level edge features, mid-level component features, and high-level semantic features; Based on the extracted depth features, feature matching is performed to obtain corresponding point pairs between frames, and feature matching similarity is calculated. The transformation hypothesis is calculated for the randomly sampled minimum point set, the reprojection error is calculated, and the model with the highest proportion of interior points is selected as the final transformation. The reprojection error is formulated as E = Σ ‖x_i' - H x_i‖², where H is the transformation matrix, x_i and x_i' are the coordinates of the matching point pair, and ‖·‖ represents the Euclidean norm.

[0011] By employing the above technical solutions, continuous multi-frame digital scanned image data of film is acquired. Binary feature vectors are generated through feature extraction and input into a pre-trained lightweight neural network model. This model extracts deep features, including low-level edge features, mid-level component features, and high-level semantic features, which helps to accurately understand the content of complex scenes in the image. Feature matching based on deep features can obtain corresponding point pairs between frames and calculate similarity, thereby intelligently distinguishing between subject movement and shooting shake in the image. Transformation hypotheses are calculated for the randomly sampled minimum point set, and the reprojection error is calculated. The model with the highest proportion of interior points is selected as the final transformation, which improves the robustness of processing in challenging scenarios such as low texture and large displacement, obtaining an accurate transformation matrix for subsequent image alignment. A content-aware infill algorithm can identify and repair boundary missing areas caused by geometric transformations in the aligned image and fill them in, maximizing the preservation of effective image information from the original film.

[0012] In a preferred embodiment, this application can be further configured such that: applying a geometric transformation to the current frame based on the transformation matrix, and achieving alignment with the reference frame through inverse mapping and bilinear interpolation, specifically includes: Based on the transformation matrix H, the inverse transformation matrix H_inv is calculated for frame alignment; By reverse mapping, for each pixel position (x, y) in the target image, the corresponding position (u, v) in the source image is calculated as H_inv · (x, y, 1)^T, where · represents matrix multiplication and (x, y, 1)^T are homogeneous coordinates. The pixel value at position (u, v) in the source image is calculated using bilinear interpolation. The interpolation formula is I(x, y) = I(u_0, v_0) · (1 - α) · (1 - β) + I(u_1, v_0) · α · (1 - β) + I(u_0, v_1) · (1 - α) · β + I(u_1, v_1) · α · β, where u_0, u_1, v_0, v_1 ​​are the integer parts of u and v, α = u - u_0, and β = v - v_0.

[0013] By adopting the above technical solution, the inverse transformation matrix is ​​first calculated for frame alignment, then the corresponding position of the source image is calculated through reverse mapping, and finally bilinear interpolation is used to calculate the pixel value of the source image position. The neighborhood information of the point can be used to estimate the pixel value of non-integer positions, improve the accuracy and continuity of pixel value calculation, reduce image distortion and jaggedness, achieve accurate alignment between the current frame and the reference frame, and improve the visual effect and quality of the image.

[0014] In a preferred embodiment, this application can be further configured as follows: the step of identifying and repairing boundary missing regions caused by geometric transformations in the aligned image data using a content-aware fill algorithm, and then filling the aligned image data, specifically includes: Detect the effective region mask of the aligned image and identify regions with missing boundaries; The filling strategy is dynamically selected based on the size of the missing area, including using an edge expansion algorithm for small gaps, a fast content-aware filling algorithm for medium gaps, and a multi-frame spatiotemporal blending algorithm for large gaps. The content-aware filling algorithm synthesizes missing content based on the surrounding pixel texture and uses a cost function to minimize the gradient difference between the filled area and the surrounding area.

[0015] By adopting the above technical solution, the effective region mask of the aligned image can be detected, and the boundary missing regions caused by geometric transformation can be identified. The filling strategy is dynamically selected according to the size of the missing region, and the appropriate filling algorithm can be used for missing regions of different sizes to improve the filling effect and efficiency. The missing content is synthesized based on the surrounding pixel texture and the cost function is used to minimize the gradient difference between the filling region and the surrounding region, which can make the filling region more coordinated with the surrounding image texture and improve the overall quality of the aligned image.

[0016] The second objective of this invention is achieved through the following technical solution: An image processing apparatus based on AI frame alignment and de-jitter, the image processing apparatus based on AI frame alignment and de-jitter includes: The image feature extraction module is used to acquire continuous multi-frame image data, which is obtained by digital scanning of film. The module performs feature extraction on each frame of the image data, which includes detecting image feature points using a detection algorithm and generating a binary feature vector for each image feature point. The matrix construction module is used to input the binary feature vector into a pre-trained lightweight neural network model for feature matching, and to calculate the geometric transformation model between each frame to obtain the transformation matrix. The matrix alignment module is used to apply a geometric transformation to the current frame according to the transformation matrix, and achieve alignment with the reference frame through inverse mapping and bilinear interpolation. The data filling module is used to identify and repair boundary missing regions caused by geometric transformations in aligned image data using a content-aware filling algorithm, and to fill in the aligned image data.

[0017] By adopting the above technical solutions, multi-frame image data from digitally scanned film is acquired and feature points and their binary feature vectors are extracted, providing key clues for subsequent inter-frame matching and alignment. Inputting the binary feature vectors into a lightweight neural network model for feature matching and geometric transformation model calculation can accurately identify inter-frame offsets, avoiding the limitations of traditional algorithms in low-texture areas. Performing geometric transformation and bilinear interpolation on the current frame based on the transformation matrix to achieve alignment with the reference frame can effectively eliminate the impact of shooting shake. By identifying and repairing boundary missing areas through a content-aware filling algorithm, the effective image information of the original film can be preserved to the maximum extent, improving the quality of the processed image.

[0018] The above-mentioned objective three of this application is achieved through the following technical solution: A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described AI-based frame alignment and de-jitter image processing method.

[0019] The fourth objective of this application is achieved through the following technical solution: A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described image processing method based on AI frame alignment and de-jitter.

[0020] In summary, this application includes at least one of the following beneficial technical effects: 1. A pre-trained lightweight neural network model is used for feature matching and geometric transformation calculation, which avoids the limitations of traditional feature matching algorithms in low-texture areas. It can accurately identify inter-frame shifts caused by shooting shake, improve the robustness of processing in challenging scenarios such as low texture and large displacement, and solve the problem that existing technologies cannot correct accumulated small displacements and shakes. 2. Frame alignment is achieved through inverse mapping and bilinear interpolation, and content-aware fill algorithm is used to repair missing boundary areas, ensuring sub-pixel level alignment accuracy. It can also intelligently repair missing edges caused by correction, maximizing the preservation of effective image information from the original film, and improving the viewing experience and image quality of the final synthesized digital video. 3. Constructing an image pyramid for feature detection in a multi-scale space, with thresholds adaptively adjusted based on local image contrast, ensures uniform distribution of feature points, providing a more accurate foundation for subsequent feature matching and alignment, and helps improve the accuracy and stability of the overall processing. Attached Figure Description

[0021] Figure 1 This describes the specific implementation process of the intelligent fill algorithm.

[0022] Figure 2 A flowchart illustrating the steps involved in AI-assisted precise alignment.

[0023] Figure 3 Example diagram of connecting lines for feature points. Detailed Implementation

[0024] The present application will be further described in detail below with reference to the accompanying drawings.

[0025] In one embodiment, such as Figure 1-2 As shown, this application discloses an image processing method based on AI frame alignment and deshaking, which specifically includes the following steps: S10: Acquire continuous multi-frame image data. The image data is obtained from the digital scanning of film. Perform feature extraction on each frame of image data. Feature extraction includes detecting image feature points using a detection algorithm and generating a binary feature vector for each image feature point.

[0026] Specifically, the system acquires continuous multi-frame image data from the digital scanning of film. In practice, a high-precision image sensor scans the film frame by frame, converting the analog images on the film into digital image data. This digital scanning process preserves the original image information of the film to the greatest extent possible, providing a high-quality data foundation for subsequent processing.

[0027] For each frame of image data, meticulous feature extraction is performed. Specifically, an image pyramid is first constructed, consisting of the original image and multi-scale downsampled images. The construction of the image pyramid is significant because it enables feature detection in a multi-scale space. Images at different scales can reflect the features of the image at different levels of detail, helping to detect feature points in the image more comprehensively and accurately. For example, the overall structural features of the image can be detected on larger-scale images, while more subtle local features can be captured on smaller-scale images.

[0028] When detecting corners on each scale of the image, a specific determination formula is used. For a pixel p, the pixels around its circumference are examined. If there are N consecutive pixels with intensities I satisfying |I - I_p|>T, then pixel p is determined to be a corner, where N is a preset number of consecutive pixels. These corners are typically points with obvious features in the image, such as object edges or corners, which can well represent the local features of the image. Furthermore, the threshold T is adaptively adjusted based on the local contrast of the image: threshold T = k · σ, where σ is the standard deviation of pixel intensity in the local region, and k is an adjustment factor. This adaptive threshold adjustment method can flexibly change the corner detection standard according to the actual situation of the image, improving the accuracy of corner detection. In areas with high contrast, the threshold is increased accordingly to avoid misclassifying some noise points as corners; while in areas with low contrast, the threshold is decreased to ensure that true corners are detected.

[0029] The detected corner points are further calculated to obtain image feature points. Then, a binary feature vector for each image feature point is generated by comparing the pixel intensity difference of preset point pairs. The binary feature vector concisely and efficiently expresses the feature information of the image feature points, facilitating subsequent feature matching and processing. The generated binary feature vectors can be quickly compared between different images, laying the foundation for accurate feature matching.

[0030] S20: Input the binary feature vector into the pre-trained lightweight neural network model for feature matching, and calculate the geometric transformation model between each frame to obtain the transformation matrix.

[0031] Specifically, with Figure 3For example, the generated binary feature vector is input into a pre-trained lightweight neural network model for feature matching, and the geometric transformation model between each frame is calculated to obtain the transformation matrix. The pre-trained lightweight neural network model can efficiently extract the depth features of the image, including low-level edge features, mid-level part features, and high-level semantic features.

[0032] Low-level edge features help identify basic features in an image, such as lines and contours, like the edges of objects or clear boundaries. Mid-level part features combine low-level features into meaningful parts; for example, in an image of a person, they can identify parts like the head, arms, and legs. High-level semantic features utilize the model's pre-trained knowledge base to understand the semantic content of the image, such as determining whether an image shows a person running or an animal resting. By extracting multi-layered deep features, the feature information of an image can be described more comprehensively and accurately, improving the accuracy of feature matching.

[0033] Feature matching based on extracted deep features can identify corresponding point pairs between frames and calculate feature matching similarity. During the matching process, the model attempts to establish correspondences between features in different frames, finding dozens to hundreds of "corresponding point pairs." These corresponding point pairs reflect the positional relationships of similar feature points between images in different frames. Simultaneously, calculating feature matching similarity can assess the degree of similarity between images in different frames, providing an important reference for subsequently selecting a suitable geometric transformation model.

[0034] To determine the final geometric transformation model, transformation hypotheses are calculated on a randomly sampled minimum point set, reprojection errors are calculated, and the model with the highest proportion of interior points is selected as the final transformation. The reprojection error formula is E = Σ ‖x_i' - H x_i‖², where H is the transformation matrix, x_i and x_i' are the coordinates of the matching point pairs, and ‖·‖ represents the Euclidean norm. By continuously sampling randomly and calculating the reprojection error, the most suitable transformation matrix can be selected, ensuring that feature points between different frames are aligned as accurately as possible, thereby effectively eliminating image jitter.

[0035] S30: Apply geometric transformation to the current frame based on the transformation matrix, and achieve alignment with the reference frame through inverse mapping and bilinear interpolation.

[0036] Specifically, based on the obtained transformation matrix, a geometric transformation is applied to the current frame, and alignment with the reference frame is achieved through inverse mapping and bilinear interpolation. Specifically, the inverse transformation matrix H_inv is first calculated based on the transformation matrix H for frame alignment. The calculation of the inverse transformation matrix is ​​a crucial step in achieving image alignment; it reverses the transformation of the current frame's image, matching its position and angle with the reference frame.

[0037] Through reverse mapping, for each pixel position (x, y) in the target image, the corresponding position (u, v) in the source image is calculated as (u, v) = H_inv · (x, y, 1)^T, where · represents matrix multiplication and (x, y, 1)^T are homogeneous coordinates. The reverse mapping process essentially maps each pixel position in the target image back to its corresponding position in the source image, thereby achieving image position adjustment.

[0038] Bilinear interpolation is used to calculate the pixel value at position (u, v) in the source image. The interpolation formula is I(x, y) = I(u_0, v_0) · (1 - α) · (1 - β) + I(u_1, v_0) · α · (1 - β) + I(u_0, v_1) · (1 - α) · β + I(u_1, v_1) · α · β, where u_0, u_1, v_0, and v_1 are the integer parts of u and v, α = u - u_0, and β = v - v_0. Bilinear interpolation calculates the pixel value of the target pixel by weighted averaging the pixel values ​​of the four neighboring pixels in the source image. This ensures that the image maintains good smoothness and continuity during the transformation process, avoiding jagged edges or pixel distortion, and improving the quality of image alignment.

[0039] S40: Using a content-aware fill algorithm, identify and repair boundary missing regions caused by geometric transformations in the aligned image data, and fill the aligned image data.

[0040] Specifically, a content-aware infill algorithm is used to identify and repair boundary missing regions caused by geometric transformations in aligned image data, and then infill the aligned image data. First, the effective region mask of the aligned image is detected to identify boundary missing regions. The effective region mask can clearly distinguish between the effective regions in the image and the missing regions caused by geometric transformations, providing a clear target for subsequent infilling operations.

[0041] The filling strategy is dynamically selected based on the size of the missing region. For small gaps, an edge expansion algorithm is used. This algorithm fills small missing regions by copying pixel values ​​near the boundary, which is simple, efficient, and maintains the continuity of image edges. For medium gaps, a fast content-aware filling algorithm is used. This algorithm synthesizes missing content based on the texture of surrounding pixels and uses a cost function to minimize the gradient difference between the filled region and the surrounding region, so that the texture and color of the filled region can match the surrounding region, achieving a better visual effect. For large gaps, a multi-frame spatiotemporal blending algorithm is used. This algorithm combines information from multiple frames of images, comprehensively considering the temporal and spatial information of the image, to fill large missing regions more accurately and naturally, restoring the original information of the image to the greatest extent.

[0042] In summary, this AI-based frame alignment and de-shake image processing method, through the synergistic effect of the aforementioned series of steps, can effectively solve the jitter problem in digitally scanned film images. At the same time, it can reasonably repair and fill the boundary missing areas caused by geometric transformations, significantly improving the image quality and visual effect, and providing strong technical support for the digital preservation and display of film images.

[0043] In one embodiment, step S10, which involves feature extraction for each frame of image data, specifically includes: S11: Construct an image pyramid, which includes the original image data and multi-scale downsampled images, for feature detection in multi-scale space.

[0044] Specifically, constructing the image pyramid is the crucial first step. An image pyramid consists of the original image data and multi-scale downsampled images, its purpose being to perform feature detection in a multi-scale space. In practice, the process of constructing the image pyramid is based on the principle of the Gaussian pyramid. The Gaussian pyramid is a classic multi-scale image representation method that generates a series of images at different resolutions by performing Gaussian smoothing and downsampling operations on the original image. Specifically, for the original image, a Gaussian filter is first used to smooth it to reduce noise. The Gaussian filter is a linear smoothing filter that can effectively suppress high-frequency noise in the image while preserving low-frequency information. After Gaussian smoothing, the image will have reduced detail, but the overall contour and structure remain unchanged.

[0045] Next, downsampling is performed on the smoothed image. Downsampling refers to reducing the size of the image, typically by halving both its width and height. When downsampling, a suitable sampling method needs to be chosen to ensure that too much image information is not lost. Common sampling methods include nearest neighbor sampling, bilinear sampling, and bicubic sampling. Nearest neighbor sampling is the simplest method, directly selecting the pixel value closest to the sampling point as the sampling result; bilinear sampling calculates the sampling result by linearly interpolating the four pixels surrounding the sampling point; and bicubic sampling calculates the sampling result by cubically interpolating the sixteen pixels surrounding the sampling point. In practical applications, bilinear sampling and bicubic sampling usually provide better sampling results because they better preserve the image's detailed information.

[0046] By repeatedly performing Gaussian smoothing and downsampling operations, a series of images at different resolutions can be obtained, forming an image pyramid. Each layer of the image pyramid represents the image at a different scale. From the high-resolution image at the bottom to the low-resolution image at the top, the detail information of the image gradually decreases, while the overall contour and structure become more apparent. This multi-scale representation method can effectively address the problem of scale variations in images because image features may exhibit different characteristics at different scales. For example, in low-resolution images, the overall contour and structure of the image are more obvious, while in high-resolution images, the detail information is richer. By performing feature detection at different layers of the image pyramid, the feature information of the image can be captured more comprehensively.

[0047] S12: Perform corner detection on each scale of the image, where the corner detection determination formula is: S121: For a pixel p, if there are N consecutive pixels on its circumference with intensity I satisfying |I - I_p|>T, then pixel p is determined to be a corner point, where N is a preset number of consecutive points.

[0048] S122: The threshold T is adaptively adjusted according to the local contrast of the image. Threshold T = k · σ, where σ is the standard deviation of the pixel intensity in the local region and k is the adjustment factor.

[0049] S123: Calculate the detected corner points to obtain image feature points, and generate binary feature vectors by comparing the pixel intensity difference of preset point pairs.

[0050] Specifically, after constructing the image pyramid, the next step is to perform corner detection on the image at each scale. Corners are points with significant features in an image, typically corresponding to edges, corners, and other similar locations. Corner detection is an important task in computer vision, applicable to image matching, object tracking, 3D reconstruction, and other fields. This technical solution employs a corner detection method based on pixel intensity difference. Specifically, for each pixel p, the pixels surrounding it on a circle are considered. If there exist N consecutive pixels with intensities I satisfying |I - I_p|>T, then pixel p is determined to be a corner, where N is a preset number of consecutive points and T is a threshold.

[0051] In practice, it is necessary to determine appropriate N and T values. The choice of N value affects the sensitivity and accuracy of corner detection. If the N value is too small, too many false corners may be detected; if the N value is too large, some real corners may be missed. Generally, the N value can be adjusted according to the specific application scenario and image characteristics. The choice of T value is even more critical, as it directly affects the accuracy of corner detection. If the T value is too small, too many false corners may be detected; if the T value is too large, some real corners may be missed. To improve the accuracy of corner detection, this technical solution adopts an adaptive threshold adjustment method.

[0052] The threshold T is adaptively adjusted based on the local contrast of the image, specifically using the formula T = k · σ, where σ is the standard deviation of pixel intensity in the local region, and k is an adjustment factor. The standard deviation of pixel intensity in the local region reflects the contrast of that local area; a larger standard deviation indicates higher contrast in that area. By correlating the threshold T with the standard deviation σ of the local pixel intensity, the threshold can be adaptively adjusted according to the local contrast of the image. The adjustment factor k can then be adjusted based on the specific application scenario and image characteristics to achieve optimal corner detection results.

[0053] When performing corner detection, the local region of each pixel must first be determined. The size of the local region can be adjusted according to the specific application scenario and image characteristics. Typically, the size of the local region can be chosen as 3x3, 5x5, or 7x7, etc. For each pixel, the standard deviation σ of its local region pixel intensity is calculated, and the threshold T is calculated according to the formula T = k · σ. Then, the pixels on the circumference surrounding the pixel are checked. If there are N consecutive pixels with intensities I satisfying |I - I_p|>T, then the pixel p is determined to be a corner.

[0054] The detected corner points are calculated to obtain image feature points. After obtaining the corner points, further processing is needed to obtain more representative image feature points. Specifically, a binary feature vector is generated by comparing the pixel intensity differences of preset point pairs. Preset point pairs refer to a set of pixel pairs pre-selected around the corner points. For each corner point, the pixel intensity difference between it and each pixel in the preset point pair is calculated, and a binary feature vector is generated based on these pixel intensity differences. Binary feature vectors are a compact feature representation method that can effectively reduce the storage space and computational cost of features.

[0055] When generating binary feature vectors, the first step is to determine the number and position of preset point pairs. The number and position of these preset point pairs affect the performance of the binary feature vector. Typically, the number of preset point pairs can be chosen as 128, 256, or 512. The position of the preset point pairs can be selected based on the position and orientation of the corner points to ensure that the preset point pairs cover the important areas around the corner points. For each corner point, the pixel intensity difference between it and each pixel in the preset point pair is calculated, and a binary feature vector is generated based on these pixel intensity differences. Specifically, if the pixel intensity difference is greater than zero, the corresponding bit in the binary feature vector is set to 1; otherwise, the corresponding bit is set to 0.

[0056] Through the above steps, feature extraction for each frame of image data can be completed. Constructing an image pyramid enables feature detection in multi-scale space, effectively addressing scale variations in images; corner detection methods based on pixel intensity differences can accurately detect corners in images; adaptive threshold adjustment methods can adaptively adjust the threshold based on the local contrast of the image, improving the accuracy of corner detection; generating binary feature vectors by comparing the pixel intensity differences of preset point pairs yields compact and representative image feature points. These feature points provide crucial information for subsequent image alignment and de-shake processing, thereby improving image processing effects and quality. In practical applications, these feature extraction methods can be widely used in film archives, museums, film production companies, and the digitization of film for private collectors, providing strong support for the protection and transmission of cultural heritage.

[0057] In one embodiment, step S20 involves inputting the binary feature vector into a pre-trained lightweight neural network model for feature matching and calculating the geometric transformation model between each frame to obtain the transformation matrix. Specifically, this includes: S21: Use a lightweight neural network model to extract deep features of the image, including low-level edge features, mid-level component features, and high-level semantic features.

[0058] Specifically, the image depth feature extraction stage is divided into three levels. The first is the extraction of low-level edge features; the model acts like a keen observer, focusing on basic elements in the image such as edges, corners, textures, and color distribution. (Using a cat image as an example...) Figure 3For example, it accurately captures the outline of a kitten, like a line artist sketching its shape, clearly defining its boundaries. For the kitten's eyes, it identifies their rounded edges, as if precisely pinpointing the source of the kitten's lively gaze. The pink spots on the nose are also meticulously detected, like unique embellishments depicting the kitten's adorable face. Even fine, dark lines like the whiskers and the wood grain of the floor are not overlooked by the model's "eyes." These low-level edge features represent the most basic and detailed analysis of the image, providing rich material for the extraction of subsequent, more advanced features.

[0059] The extraction of intermediate-level component features involves integrating low-level features. The model combines previously detected low-level features into meaningful components. In the kitten image, it can identify the "left front paw region," which integrates previously detected edge, texture, and other features related to the left front paw to form a complete component concept; similarly, the "right ear region" combines various features of the right ear, allowing us to see the overall shape of the kitten's ear more clearly; the recognition of the "facial triangle (two eyes and one nose)" further demonstrates the model's comprehensive understanding of image components, combining the features of the eyes and nose into a region with specific meaning; the "torso region" is a summary of the main body of the kitten, integrating various features within this region. The model pays attention to the relative position and shape of these components in the two images, which is crucial for determining the similarity and alignment between images. By extracting intermediate-level component features, we can understand the structure of objects in images from a more macroscopic perspective.

[0060] The extraction of high-level semantic features allows the model to utilize its pre-trained knowledge base to gain a deeper understanding of the image. In the example of the cat image, the model can draw advanced conclusions, such as determining "subject: a cat," which is an accurate identification of the core object in the image; "posture: lying on its back, paws in the air," which is a precise description of the cat's posture, reflecting the model's grasp of the overall dynamics of the image; "action: playing / stretching," which further refines the cat's behavioral state; and "scene: indoor home environment," which sets a specific background environment for the image. The extraction of these high-level semantic features allows the model to go beyond simply analyzing the surface features of the image, delving into the actual meaning the image represents, providing more comprehensive and accurate information for subsequent feature matching.

[0061] S22: Perform feature matching based on the extracted deep features, obtain corresponding point pairs between frames, and calculate feature matching similarity.

[0062] Specifically, when performing feature matching based on extracted deep features, the model strives to find corresponding point pairs between images in different frames. It carefully compares identical or similar features in different images; for example, in the cat image, it matches features such as the cat's eyes, nose, and paws in different frames to find their corresponding positions in different images. This is how corresponding point pairs between frames are obtained. Simultaneously, feature matching similarity is calculated, a crucial indicator of the similarity between images in different frames. Multiple aspects are considered when calculating similarity, such as appearance similarity, comparing color histograms, overall texture, etc. In the cat image, the color distribution (black-and-white ratio) of the two cats is almost identical, resulting in a very high appearance similarity score. Geometric similarity is calculated based on the average reprojection error of "interior points"; the smaller the error, the better the alignment and the higher the similarity. Semantic similarity is also a highly weighted factor because the model judges the two images to be "the same cat in the same pose at almost the same time," so this aspect scores close to full marks. Structural similarity assesses the loss of image structural information (brightness, contrast, structure); due to the almost identical shooting conditions, this aspect also scores highly. By combining similarity calculations across these dimensions, the degree of similarity between different frames of images can be reflected more accurately.

[0063] S23: Calculate the transformation hypothesis for the randomly sampled minimum point set, calculate the reprojection error, and select the model with the highest proportion of interior points as the final transformation. The reprojection error formula is E = Σ ‖x_i' - H x_i‖², where H is the transformation matrix, x_i and x_i' are the coordinates of the matching point pairs, and ‖·‖ represents the Euclidean norm.

[0064] Specifically, in the stage of calculating transformation hypotheses for a randomly sampled minimum set of points, the model randomly selects a minimum number of points from numerous feature point pairs and calculates possible transformation models based on these points. This is analogous to selecting a few pieces from a jigsaw puzzle to try and piece together a possible pattern. Then, the reprojection error is calculated, with the formula E = Σ ‖x_i' - H x_i‖², where H is the transformation matrix, x_i and x_i' are the coordinates of the matched point pair, and ‖·‖ represents the Euclidean norm. This formula measures the error between the actual point and the point after projecting it from one image to another according to the transformation matrix H. By calculating the reprojection error, the accuracy of each transformation hypothesis can be evaluated.

[0065] Finally, the model with the highest proportion of inliers is selected as the final transformation. During the calculation of transformation assumptions and reprojection errors, "inliers" and "outliers" are distinguished. Points that conform to the transformation model are called "inliers," and those that do not are called "outliers." A correct and robust transformation should be supported by a large number of inliers. For example, in a cat image, most feature points related to the cat's body should be inliers, collectively supporting a reasonable small transformation model. Some variable or unreliable matching points in the background may be treated as outliers and discarded. The number and proportion of inliers are important indicators of similarity reliability. Selecting the model with the highest proportion of inliers as the final transformation ensures that the resulting transformation matrix most accurately reflects the geometric relationships between different frames, thereby achieving more precise frame alignment and de-jitter effects.

[0066] Through the above steps, utilizing a pre-trained lightweight neural network model, it is possible to extract image depth features, perform feature matching, calculate the transformation model, and select the final transformation. This provides an effective solution to the shakiness problem caused by the lack of integrated image stabilization and alignment algorithms in existing video synthesis processes. In practical applications, for 8mm film-to-digital video scanner systems, it significantly improves the stability and quality of the final synthesized digital video, avoiding visual shakiness caused by minor displacements and movements during the original film shooting. This provides strong technical support for the digital preservation and transmission of valuable historical data or cultural heritage. Simultaneously, this method also features real-time processing performance. Through zero-copy data transmission and a dedicated AI inference engine, it ensures 24fps real-time processing performance while keeping end-to-end latency within 100ms, meeting the efficiency requirements of practical applications. Furthermore, the system can dynamically select processing strategies based on the type and complexity of inter-frame motion, achieving a balance between accuracy and efficiency, further enhancing the overall system performance and applicability.

[0067] In one embodiment, step S30, which involves applying a geometric transformation to the current frame based on the transformation matrix, and aligning it with the reference frame through inverse mapping and bilinear interpolation, specifically includes: S31: Calculate the inverse transformation matrix H_inv based on the transformation matrix H for frame alignment.

[0068] Specifically, after obtaining the transformation matrix H, its inverse transformation matrix H_inv needs to be calculated for frame alignment. This step is fundamental to the entire alignment process, enabling us to map the pixel positions of the current frame to the corresponding positions in the reference frame.

[0069] In mathematical terms, for an invertible transformation matrix H, its inverse transformation matrix H_inv satisfies H * H_inv = I, where I is the identity matrix. There are several methods for calculating the inverse transformation matrix, commonly including Gaussian-Jordan elimination and the adjoint matrix method.

[0070] Gauss-Jordan elimination is a method that transforms the augmented matrix [H | I] into an identity matrix by performing row operations on H. Simultaneously, the identity matrix I on the right-hand side is transformed into the inverse matrix H_inv of H. The specific steps are as follows: 1. Construct the augmented matrix [H | I], where H is the transformation matrix and I is the identity matrix of the same order as H.

[0071] 2. Perform row operations on the augmented matrix to transform the H part into an identity matrix. Row operations include swapping two rows, multiplying a row by a non-zero constant, and adding a row to another row by a certain multiple.

[0072] 3. When H is transformed into the identity matrix, the matrix on the right is the inverse transformation matrix H_inv.

[0073] The adjoint matrix method obtains the inverse matrix by calculating the adjoint matrix adj(H) and determinant det(H) of matrix H. The specific formula is H_inv = adj(H) / det(H). Here, the adjoint matrix adj(H) is the transpose of the matrix consisting of the algebraic cofactors of matrix H, and the determinant det(H) is a scalar value.

[0074] In practical applications, the choice of which method to use to compute the inverse transformation matrix depends on the size of the matrix and the specific computational environment. For small matrices, the adjoint matrix method may be more intuitive and easier to implement; while for large matrices, Gaussian-Jordan elimination usually has higher computational efficiency.

[0075] After calculating the inverse transformation matrix H_inv, it can be applied to the pixel position of the current frame to achieve alignment with the reference frame. The function of the inverse transformation matrix is ​​to transform the pixel position of the current frame from its own coordinate system to the coordinate system of the reference frame, thereby eliminating inter-frame offset caused by shooting shake or other factors.

[0076] S32: By reverse mapping, for each pixel position (x, y) in the target image, calculate the corresponding position (u, v) in the source image = H_inv · (x, y, 1)^T, where · represents matrix multiplication and (x, y, 1)^T are homogeneous coordinates.

[0077] Specifically, after obtaining the inverse transformation matrix H_inv, the next step is to calculate the corresponding position (u, v) in the source image for each pixel position (x, y) in the target image using inverse mapping: (u, v) = H_inv · (x, y, 1)^T. This inverse mapping is an important image processing technique that maps the pixel positions of the target image back to their corresponding positions in the source image using the inverse transformation matrix.

[0078] First, the pixel position (x, y) of the target image is converted to homogeneous coordinates (x, y, 1)^T. Homogeneous coordinates are a commonly used coordinate representation method in computer graphics and image processing. They simplify matrix operations by adding an extra dimension. In homogeneous coordinates, a point (x, y) on a two-dimensional plane can be represented as (x, y, 1)^T, where T denotes the transpose.

[0079] Then, a matrix multiplication operation is performed between the homogeneous coordinates (x, y, 1)^T and the inverse transformation matrix H_inv to obtain the corresponding position (u, v) in the source image. The rule for matrix multiplication is to multiply each row of the matrix with the corresponding element of each column of the vector and sum the results. Specifically, for the inverse transformation matrix H_inv = [[a, b, c], [d, e, f], [g, h, i]] and the homogeneous coordinates (x, y, 1)^T, the corresponding position (u, v) in the source image is calculated as follows: u = a * x + b * y + c v = d * x + e * y + f It's important to note that because homogeneous coordinates are used, the calculated (u, v) may be a homogeneous coordinate system, which needs to be converted to two-dimensional coordinates. A common approach is to divide the first two components of (u, v) by the third component to obtain the final two-dimensional coordinates.

[0080] The significance of inverse mapping lies in its ability to accurately map the pixel positions of the target image back to their corresponding positions in the source image using the inverse transformation matrix. This allows the corresponding pixel values ​​to be retrieved from the source image during subsequent interpolation, thus achieving alignment between the current frame and the reference frame.

[0081] S33: Use bilinear interpolation to calculate the pixel value at position (u, v) in the source image. The interpolation formula is I(x, y) = I(u_0, v_0) · (1 - α) · (1 - β) + I(u_1, v_0) · α · (1 - β) + I(u_0, v_1) · (1 - α) · β + I(u_1, v_1) · α · β, where u_0, u_1, v_0, v_1 ​​are the integer parts of u and v, α = u - u_0, β = v - v_0.

[0082] Specifically, after calculating the corresponding position (u, v) in the source image, since (u, v) may not be integer coordinates, bilinear interpolation is needed to calculate the pixel value at position (u, v) in the source image. Bilinear interpolation is a commonly used image processing interpolation method that calculates the pixel value at a non-integer coordinate position by taking a weighted average of four adjacent integer coordinate pixels in the source image.

[0083] For a non-integer coordinate position (u, v), bilinear interpolation finds four surrounding integer coordinate pixels: (u_0, v_0), (u_1, v_0), (u_0, v_1), and (u_1, v_1). Here, u_0 and u_1 are the integer parts of u and u plus 1, respectively, and v_0 and v_1 are the integer parts of v and v plus 1, respectively. Then, based on the distances between (u, v) and these four integer coordinate pixels, a weighted average is taken to obtain the pixel value at position (u, v).

[0084] The specific interpolation formula is I(x, y) = I(u_0, v_0) · (1 - α) · (1 - β) + I(u_1,v_0) · α · (1 - β) + I(u_0, v_1) · (1 - α) · β + I(u_1, v_1) · α · β, where α = u - u_0, β = v - v_0.

[0085] The following is the meaning of the formula: - (1 - α) · (1 - β) is the weight of the pixel (u_0, v_0), which represents the degree of influence of the distance between (u, v) and (u_0, v_0) on the interpolation result.

[0086] -α·(1-β) is the weight of the pixel (u_1, v_0), which represents the degree of influence of the distance between (u, v) and (u_1, v_0) on the interpolation result.

[0087] - (1 - α) · β is the weight of the pixel (u_0, v_1), which represents the degree of influence of the distance between (u, v) and (u_0, v_1) on the interpolation result.

[0088] - α · β is the weight of the (u_1, v_1) pixel, which represents the degree of influence of the distance between (u, v) and (u_1, v_1) on the interpolation result.

[0089] By using this weighted averaging method, bilinear interpolation can preserve as much detail as possible in the source image while ensuring a smooth interpolation result. Compared to nearest-neighbor interpolation, bilinear interpolation can avoid jagged edges and improve image quality.

[0090] In actual calculations, the pixel values ​​of (u_0, v_0), (u_1, v_0), (u_0, v_1), and (u_1, v_1) need to be determined first. Then, the weight of each pixel is calculated based on the values ​​of α and β. Finally, each pixel value is multiplied by its corresponding weight and summed to obtain the pixel value at position (u, v).

[0091] By employing the three steps above—calculating the inverse transform matrix, reverse mapping, and bilinear interpolation—precise alignment between the current frame and the reference frame can be achieved. This alignment method has significant application value in AI-based frame alignment and de-jittering image processing.

[0092] When dealing with frame shift caused by camera shake, calculating the inverse transform matrix and reverse mapping can accurately map the pixel position of the current frame to the corresponding position in the reference frame, eliminating the displacement caused by shake. Bilinear interpolation, on the other hand, can preserve image detail as much as possible while maintaining image smoothness, thus improving the quality of the aligned image.

[0093] In the field of cultural heritage preservation, the conversion of 8mm film to digital video often results in shakiness in the final digital video due to potential camera shake during filming. The alignment method described above effectively corrects these accumulated minute shifts and vibrations, improving the viewing experience and image quality.

[0094] This frame alignment method also plays an important role in other fields of computer vision and image processing, such as video surveillance and target tracking. In video surveillance, aligning consecutive frames can improve the accuracy of target detection and tracking; in target tracking, frame alignment can improve the target's positioning accuracy and reduce tracking errors caused by jitter and displacement.

[0095] In one embodiment, step S40, which involves identifying and repairing boundary missing regions caused by geometric transformations in the aligned image data using a content-aware fill algorithm, and then filling the aligned image data, specifically includes: S41: Detect the effective region mask of the aligned image and identify regions with missing boundaries.

[0096] Specifically, the effective region mask of the aligned image is detected, and regions with missing boundaries are identified. After the image is aligned through geometric transformation, it is necessary to determine which regions are valid image content and which regions have missing boundaries due to the transformation. This step is fundamental to subsequent filling operations, and its accuracy directly affects the final filling effect.

[0097] To detect the effective region mask, the system performs a series of analyses and processes on the aligned image. Typically, grayscale values, color information, or other features are used to distinguish between effective and missing regions. For example, a grayscale threshold can be set, and regions with grayscale values ​​below that threshold can be considered potential missing regions. Simultaneously, to improve detection accuracy, morphological operations such as dilation and erosion can be combined to remove noise points and small discontinuities, making the effective region mask clearer and more accurate.

[0098] When identifying regions with missing boundaries, the system further analyzes the detected valid region mask. Edge detection algorithms, such as Canny edge detection, can be used to determine the boundaries of the valid regions. Then, based on this boundary information, those parts located at the image edges that do not belong to the valid regions are identified as regions with missing boundaries. Furthermore, by calculating the connectivity of the valid regions, smaller regions not connected to the main valid regions can also be identified as regions with missing boundaries.

[0099] S42: Dynamically select the filling strategy based on the size of the missing region, including using the edge expansion algorithm for small gaps, the fast content-aware filling algorithm for medium gaps, and the multi-frame spatiotemporal blending algorithm for large gaps.

[0100] Specifically, the filling strategy is dynamically selected based on the size of the missing region. Different sizes of missing regions require different filling strategies to achieve the best filling effect. For small gaps, using an edge expansion algorithm is an effective method.

[0101] The core idea of ​​edge expansion algorithms is to use edge information surrounding the missing region to fill in the missing portion. In practice, the system first identifies the edge pixels of the missing region and analyzes their orientation and texture features. Then, based on these features, the edge pixels are expanded along their direction to fill the missing region. For example, if the edge of the missing region is a horizontal straight line, the system will expand the edge pixels horizontally within the missing region. To make the filling effect more natural, the expanded pixels can also be smoothed to reduce the obvious boundary between the filled region and the surrounding area.

[0102] For medium-sized gaps, a fast content-aware fill algorithm is more suitable. This algorithm synthesizes missing content based on the texture information of surrounding pixels. The system searches for the most similar texture block in the valid region surrounding the missing area. To find the most similar texture block, a similarity metric, such as the Structural Similarity Index (SSIM), is used. By comparing the similarity between different texture blocks and the missing area, the texture block with the highest similarity is selected as the basis for filling.

[0103] After selecting a texture block, the system copies it to the missing area and adjusts it based on the color and brightness information of surrounding pixels to better blend it with the environment. Simultaneously, to ensure accuracy and efficiency in the filling process, a cost function is used to minimize the gradient difference between the filled area and its surroundings. By continuously adjusting the position and parameters of the filling texture block, the cost function is minimized, thus achieving the best filling effect.

[0104] For large gaps, multi-frame spatiotemporal blending algorithms are the preferred choice. Because the missing areas in large gaps are substantial, relying solely on information from a single frame may not yield ideal filling results. Therefore, multi-frame spatiotemporal blending algorithms utilize image information from adjacent frames for filling.

[0105] The algorithm first analyzes adjacent frames to identify the location corresponding to the current missing region. Then, it fuses the image information from corresponding locations in adjacent frames to generate the content needed to fill the missing region. For example, a weighted average method can be used, assigning different weights to the image information of different frames based on the temporal distance and similarity between adjacent frames and the current frame. Frames that are closer in time and have higher similarity receive greater weight for their image information.

[0106] When fusing image information from adjacent frames, it is also necessary to consider image motion and changes. Motion estimation and compensation techniques can be used to correct the images in adjacent frames to ensure that the fused image information is spatially consistent with the image information of the current frame. Furthermore, to improve the quality of the filling effect, image enhancement techniques such as sharpening and contrast adjustment can be combined to process the fused image information.

[0107] S43: The content-aware fill algorithm synthesizes missing content based on the texture of surrounding pixels and uses a cost function to minimize the gradient difference between the filled area and the surrounding area.

[0108] Specifically, content-aware fill algorithms use a cost function to minimize the gradient difference between the filled region and its surrounding region. Gradient difference reflects the degree of variation in a local area of ​​the image; minimizing this gradient difference makes the transition between the filled region and its surrounding region more natural and smooth.

[0109] The cost function is typically a gradient-based function that measures the gradient difference between the filled region and its surrounding regions. During the filling process, the system continuously adjusts the pixel values ​​of the filled region to gradually reduce the cost function. To achieve this, optimization algorithms such as gradient descent can be used. Gradient descent adjusts the pixel values ​​of the filled region step by step according to the gradient direction of the cost function until the cost function reaches its minimum.

[0110] In practical applications, to improve filling efficiency and accuracy, the cost function can be further improved and optimized. For example, a regularization term can be introduced to avoid excessive fluctuations in pixel values ​​in the filled region. Simultaneously, multi-scale analysis methods can be combined to analyze and process the filled region and its surrounding area at different scales, thereby better capturing the image's detailed information.

[0111] Through the above series of steps and methods, the content-aware fill algorithm can effectively identify and repair boundary missing regions caused by geometric transformations in aligned image data. By dynamically selecting the fill strategy and minimizing gradient differences using a cost function, the transition between the filled area and the surrounding area can be made more natural and smooth, thus significantly improving the overall quality and integrity of the image. This algorithm is not only applicable to image de-shaking and alignment processing in 8mm film-to-digital video scanner systems, but can also be applied to image restoration and processing tasks in other fields, showing broad application prospects and significant practical importance. In different application scenarios, the algorithm's parameters and strategies can be adjusted and optimized according to specific needs and image characteristics to achieve the best processing results. Furthermore, with the continuous development of computer technology and image processing algorithms, the content-aware fill algorithm will continue to improve and enhance, bringing more innovation and breakthroughs to the field of image restoration and processing.

[0112] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0113] In one embodiment, an image processing apparatus based on AI frame alignment and dejitter is provided, which corresponds one-to-one with the image processing method based on AI frame alignment and dejitter in the above embodiments. The image processing apparatus based on AI frame alignment and dejitter includes an image feature extraction module, a matrix construction module, a matrix alignment module, and a data filling module. Detailed descriptions of each functional module are as follows: The image feature extraction module is used to acquire continuous multi-frame image data. The image data is obtained by digital scanning of film. Feature extraction is performed on each frame of image data. Feature extraction includes detecting image feature points using a detection algorithm and generating a binary feature vector for each image feature point. The matrix construction module is used to input binary feature vectors into a pre-trained lightweight neural network model for feature matching and to calculate the geometric transformation model between each frame to obtain the transformation matrix. The matrix alignment module is used to apply geometric transformations to the current frame based on the transformation matrix, and achieve alignment with the reference frame through inverse mapping and bilinear interpolation. The data filling module is used to identify and repair boundary missing regions caused by geometric transformations in aligned image data using a content-aware filling algorithm, and to fill in the aligned image data.

[0114] Optionally, the image feature extraction module includes: The image processing submodule is used to construct an image pyramid, which includes the original image data and multi-scale downsampled images, for feature detection in multi-scale space; The corner detection submodule is used to perform corner detection on images at each scale. The corner detection determination formula is as follows: The corner calculation unit is used to determine if pixel p is a corner point if there are N consecutive pixels on the circumference around it, and if the intensity I of these N pixels satisfies |I - I_p|>T, where N is a preset number of consecutive points. The threshold calculation unit is used to adaptively adjust the threshold T according to the local contrast of the image. The threshold T = k · σ, where σ is the standard deviation of the pixel intensity in the local region and k is the adjustment factor. The feature calculation unit is used to calculate the detected corner points to obtain image feature points, and generate binary feature vectors by comparing the pixel intensity difference of preset point pairs.

[0115] Optionally, the matrix building module includes: The deep feature extraction submodule is used to extract deep features of an image using a lightweight neural network model, including low-level edge features, mid-level part features, and high-level semantic features. The similarity calculation submodule is used to perform feature matching based on the extracted deep features, obtain corresponding point pairs between frames, and calculate the feature matching similarity. The point set calculation submodule is used to calculate the transformation hypothesis for the randomly sampled minimum point set, calculate the reprojection error, and select the model with the highest proportion of interior points as the final transformation. The reprojection error formula is E = Σ ‖x_i' - H x_i‖², where H is the transformation matrix, x_i and x_i' are the coordinates of the matching point pairs, and ‖·‖ represents the Euclidean norm.

[0116] Optionally, the matrix alignment module includes: The frame alignment submodule is used to calculate the inverse transformation matrix H_inv based on the transformation matrix H for frame alignment. The reverse mapping submodule is used to calculate the corresponding position (u, v) in the source image for each pixel position (x, y) in the target image by reverse mapping, where · represents matrix multiplication and (x, y, 1)^T are homogeneous coordinates. The pixel value calculation submodule is used to calculate the pixel value at position (u, v) of the source image using bilinear interpolation. The interpolation formula is I(x, y) = I(u_0, v_0) · (1 - α) · (1 - β) + I(u_1, v_0) · α · (1 - β) + I(u_0, v_1) · (1 - α) · β + I(u_1, v_1) · α · β, where u_0, u_1, v_0, v_1 ​​are the integer parts of u and v, α = u - u_0, and β = v - v_0.

[0117] Optionally, the data population module includes: The missing region identification submodule is used to detect the effective region mask of the aligned image and identify the missing boundary regions; The first missing region filling submodule is used to dynamically select a filling strategy based on the size of the missing region, including using an edge expansion algorithm for small gaps, a fast content-aware filling algorithm for medium gaps, and a multi-frame spatiotemporal blending algorithm for large gaps. The second missing region filling submodule is used by the content-aware filling algorithm to synthesize missing content based on the surrounding pixel texture and to minimize the gradient difference between the filled region and the surrounding region using a cost function.

[0118] Specific limitations regarding the AI-based frame alignment and de-jitter image processing device can be found in the limitations of the AI-based frame alignment and de-jitter image processing method described above, and will not be repeated here. Each module in the aforementioned AI-based frame alignment and de-jitter image processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0119] In one embodiment, a computer device is provided, which may be a server. The computer device includes a processor, memory, a network interface, and a database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements an image processing method based on AI frame alignment and dejitter.

[0120] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Acquire continuous multi-frame image data, which is obtained from the digital scanning of film. Perform feature extraction on each frame of image data. Feature extraction includes detecting image feature points using a detection algorithm and generating a binary feature vector for each image feature point. The binary feature vector is input into a pre-trained lightweight neural network model for feature matching, and the geometric transformation model between each frame is calculated to obtain the transformation matrix. Based on the transformation matrix, a geometric transformation is applied to the current frame, and alignment with the reference frame is achieved through inverse mapping and bilinear interpolation. The content-aware fill algorithm identifies and repairs boundary missing regions caused by geometric transformations in aligned image data, and then fills the aligned image data.

[0121] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Acquire continuous multi-frame image data, which is obtained from the digital scanning of film. Perform feature extraction on each frame of image data. Feature extraction includes detecting image feature points using a detection algorithm and generating a binary feature vector for each image feature point. The binary feature vector is input into a pre-trained lightweight neural network model for feature matching, and the geometric transformation model between each frame is calculated to obtain the transformation matrix. Based on the transformation matrix, a geometric transformation is applied to the current frame, and alignment with the reference frame is achieved through inverse mapping and bilinear interpolation. The content-aware fill algorithm identifies and repairs boundary missing regions caused by geometric transformations in aligned image data, and then fills the aligned image data.

[0122] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0123] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0124] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. An image processing method based on AI frame alignment and dejitter, characterized in that, The AI-based frame alignment and dejitter reduction image processing method includes: Acquire continuous multi-frame image data, the image data being obtained from digital scanning of film, and perform feature extraction on each frame of the image data. The feature extraction includes detecting image feature points using a detection algorithm and generating a binary feature vector for each image feature point. The binary feature vector is input into a pre-trained lightweight neural network model for feature matching, and the geometric transformation model between each frame is calculated to obtain the transformation matrix. Based on the transformation matrix, a geometric transformation is applied to the current frame, and alignment with the reference frame is achieved through inverse mapping and bilinear interpolation; The content-aware fill algorithm identifies and repairs boundary missing regions caused by geometric transformations in aligned image data, and then fills the aligned image data.

2. The image processing method based on AI frame alignment and dejittering according to claim 1, characterized in that, The feature extraction for each frame of image data specifically includes: Construct an image pyramid, including the original image of the image data and multi-scale downsampled images, for feature detection in a multi-scale space; Corner detection is performed on each of the aforementioned scale images, where the corner detection determination formula is: For a pixel p, if there are N consecutive pixels on its circumference with intensity I satisfying |I - I_p|> T, then the pixel p is determined to be a corner point, where N is a preset number of consecutive points; The threshold T is adaptively adjusted based on the local contrast of the image. The threshold T = k · σ, where σ is the standard deviation of the pixel intensity in the local region and k is the adjustment factor. The detected corner points are calculated to obtain the image feature points, and the binary feature vector is generated by comparing the pixel intensity difference of preset point pairs.

3. The image processing method based on AI frame alignment and dejittering according to claim 1, characterized in that, The step of inputting the binary feature vector into a pre-trained lightweight neural network model for feature matching and calculating the geometric transformation model between each frame to obtain the transformation matrix specifically includes: The lightweight neural network model is used to extract deep features of the image, including low-level edge features, mid-level component features, and high-level semantic features; Based on the extracted depth features, feature matching is performed to obtain corresponding point pairs between frames, and feature matching similarity is calculated. The transformation hypothesis is calculated for the randomly sampled minimum point set, the reprojection error is calculated, and the model with the highest proportion of interior points is selected as the final transformation. The reprojection error is formulated as E = Σ ‖x_i' - H x_i‖², where H is the transformation matrix, x_i and x_i' are the coordinates of the matching point pair, and ‖·‖ represents the Euclidean norm.

4. The image processing method based on AI frame alignment and dejittering according to claim 1, characterized in that, The step of applying a geometric transformation to the current frame based on the transformation matrix, and aligning it with the reference frame through inverse mapping and bilinear interpolation, specifically includes: Based on the transformation matrix H, the inverse transformation matrix H_inv is calculated for frame alignment; By reverse mapping, for each pixel position (x, y) in the target image, the corresponding position (u, v) in the source image is calculated as H_inv · (x, y, 1)^T, where · represents matrix multiplication and (x, y, 1)^T are homogeneous coordinates. The pixel value at position (u, v) in the source image is calculated using bilinear interpolation. The interpolation formula is I(x, y) = I(u_0, v_0)· (1 - α) · (1 - β) + I(u_1, v_0) · α · (1 - β) + I(u_0, v_1) · (1 - α) ·β + I(u_1, v_1) · α · β, where u_0, u_1, v_0, v_1 ​​are the integer parts of u and v, α = u - u_0, and β = v - v_0.

5. The image processing method based on AI frame alignment and dejittering according to claim 1, characterized in that, The process of identifying and repairing boundary missing regions caused by geometric transformations in aligned image data using a content-aware fill algorithm, and then filling the aligned image data, specifically includes: Detect the effective region mask of the aligned image and identify regions with missing boundaries; The filling strategy is dynamically selected based on the size of the missing area, including using an edge expansion algorithm for small gaps, a fast content-aware filling algorithm for medium gaps, and a multi-frame spatiotemporal blending algorithm for large gaps. The content-aware filling algorithm synthesizes missing content based on the surrounding pixel texture and uses a cost function to minimize the gradient difference between the filled area and the surrounding area.

6. An image processing device based on AI frame alignment and de-jitter, characterized in that, The AI-based frame alignment and de-jitter image processing device includes: The image feature extraction module is used to acquire continuous multi-frame image data, which is obtained by digital scanning of film. The module performs feature extraction on each frame of the image data, which includes detecting image feature points using a detection algorithm and generating a binary feature vector for each image feature point. The matrix construction module is used to input the binary feature vector into a pre-trained lightweight neural network model for feature matching, and to calculate the geometric transformation model between each frame to obtain the transformation matrix. The matrix alignment module is used to apply a geometric transformation to the current frame according to the transformation matrix, and achieve alignment with the reference frame through inverse mapping and bilinear interpolation. The data filling module is used to identify and repair boundary missing regions caused by geometric transformations in aligned image data using a content-aware filling algorithm, and to fill in the aligned image data.

7. The image processing apparatus based on AI frame alignment and de-jitter according to claim 6, characterized in that, The image feature extraction module includes: The image processing submodule is used to construct an image pyramid, which includes the original image of the image data and multi-scale downsampled images, for feature detection in multi-scale space; The corner detection submodule is used to perform corner detection on each scale image, wherein the corner detection determination formula is: The corner point calculation unit is used to determine that pixel point p is a corner point if there are N consecutive pixels on the circumference around it, and if the intensity I of these N pixels satisfies |I - I_p| > T. Here, N is a preset number of consecutive points. A threshold calculation unit is used to adaptively adjust the threshold T according to the local contrast of the image, wherein the threshold T = k · σ, σ is the standard deviation of the pixel intensity of the local region, and k is an adjustment factor; The feature calculation unit is used to calculate the detected corner points to obtain the image feature points, and generate the binary feature vector by comparing the pixel intensity difference of preset point pairs.

8. The image processing apparatus based on AI frame alignment and de-jitter according to claim 6, characterized in that, The matrix construction module includes: The deep feature extraction submodule is used to extract deep features of the image using the lightweight neural network model, including low-level edge features, mid-level component features, and high-level semantic features. The similarity calculation submodule is used to perform feature matching based on the extracted deep features, obtain corresponding point pairs between frames, and calculate feature matching similarity. The point set calculation submodule is used to calculate the transformation hypothesis for the randomly sampled minimum point set, calculate the reprojection error, and select the model with the highest proportion of interior points as the final transformation. The reprojection error formula is E = Σ ‖x_i' - H x_i‖², where H is the transformation matrix, x_i and x_i' are the coordinates of the matching point pair, and ‖·‖ represents the Euclidean norm.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the image processing method based on AI frame alignment and de-jitter as described in any one of claims 1 to 5.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the image processing method based on AI frame alignment and de-jitter as described in any one of claims 1 to 5.

Images