Anti-parallax image non-rigid real-time interactive deformation method

Abstract

The invention relates to the technical field of image processing, and discloses an anti-parallax image non-rigid real-time interactive deformation method, which comprises the following steps: S1, acquiring a first image and a second image; s2, extracting a semantic primitive corresponding to the initial control point pair; s3, receiving an operation instruction of a user for the semantic primitive; s4, updating the position of a control point associated with the operated semantic primitive; s5, calculating a non-rigid deformation field by adopting a moving least square deformation algorithm; and S6, applying the non-rigid deformation field to the second image to generate the second image. According to the method, a structure regular term is introduced, and non-similar components in local affine transformation are quantized and punished, so that a finally calculated non-rigid deformation field keeps the local shape and angle of image content unchanged to the greatest extent while meeting control point constraints, unnatural distortion is avoided, and the accuracy of a deformation result is ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an anti-parallax image non-rigid real-time interactive deformation method. BACKGROUND

[0002] Image stitching is a basic technology in the field of computer vision, which aims to combine multiple images with overlapping fields of view into a single, coherent image with higher resolution or wider field of view. The conventional image stitching process usually includes steps such as feature point detection, feature matching, image alignment and image fusion. For example, by using algorithms such as Scale-Invariant Feature Transform (SIFT) to find stable homonymic points between images, and using Random Sample Consensus (RANSAC) algorithm to estimate a global geometric transformation model (such as homography matrix), the spatial alignment between images is realized. Finally, the overlapping area is processed by multi-band fusion or optimal seam cutting technology to eliminate the visual seam.

[0003] However, when dealing with images with significant parallax, non-coplanar camera motion or complex scene depth changes, the aforementioned traditional method based on a single global geometric transformation model exposes its limitations. A global transformation cannot accurately describe the local, non-rigid geometric correspondence in the scene due to different depths, which directly leads to alignment failure, resulting in visual artifacts such as ghosting, breaking and structural misalignment in the final stitching result, seriously affecting image quality.

[0004] To ensure the robustness of the geometric model estimation, the existing automatic process often uses the Random Sample Consensus (RANSAC) algorithm to remove the mis-matching point pairs. However, the rigid screening criteria of this kind of algorithm may misjudge the effective matching point pairs that deviate from the global model due to local non-rigid deformation or slight parallax as outliers and discard them, which further weakens the accuracy and completeness of the geometric constraints that can be relied on for subsequent deformation and alignment.

[0005] More importantly, existing technical solutions generally lack efficient and intuitive interactive correction mechanisms. When automatic alignment produces local errors, users have difficulty in intervening and correcting the geometric correspondence of specific regions in a simple and direct manner. The operation is usually limited to low-level adjustment of individual feature points, which is tedious and difficult to ensure the integrity and consistency of specific structures (such as building edges) in the image. SUMMARY

[0006] In view of the shortcomings of the prior art, the present application provides an anti-parallax image non-rigid real-time interactive deformation method, which solves the problem that the traditional method is not accurate in local alignment when dealing with parallax and non-rigid scenes due to the reliance on a single global transformation model, and also lacks efficient and intuitive interactive tools for structured correction of local errors.

[0007] To achieve the above object, the present application is implemented by the following technical scheme: a parallax-resistant image non-rigid real-time interactive deformation method, comprising the following steps: First, a first image and a second image are acquired, and a set of initial control point pairs is determined by a feature matching method to provide a basic geometric constraint for subsequent deformation. Then, a set of semantic primitives associated with the initial control point pairs is extracted from the first image and the second image. The semantic primitives are feature units with specific image content meaning formed after the control point pairs are structurally organized. In a specific implementation, the semantic primitives can include linear structure primitives extracted by a line segment detection algorithm and region structure primitives extracted by a superpixel segmentation algorithm. When the automatic extraction result is insufficient, the user is also allowed to supplement the definition by manually drawing.

[0008] Next, a translation, rotation, or scaling operation instruction performed by the user on the semantic primitives is received. According to the received operation instruction, the positions of the control points associated with the operated semantic primitives in the second image are updated.

[0009] The updating process is as follows: first, a local affine transformation is determined according to the user operation; then, all control points attached to the operated semantic primitives are screened out; finally, the local affine transformation is applied to all screened-out control points to calculate their updated target positions. Then, based on the positions of all control points (including the updated control points and the unoperated control points), an improved moving least squares deformation algorithm is used to calculate a global non-rigid deformation field acting on the second image. Finally, the calculated non-rigid deformation field is applied to the second image to generate a deformed image, which is displayed to the user in real time.

[0010] To maintain the local structure of the image content and prevent unnatural distortion during the deformation process, the present application improves the objective function in the aforementioned moving least squares deformation algorithm. A structure regularization term is introduced into the objective function, which acts to penalize the non-similar transformation components (such as shear and non-uniform scaling) in the deformation field. Specifically, when solving the best local affine transformation for any point in the image , the objective function used is defined as: In the formula, the objective function includes two parts. The first part is the data fitting term, which is consistent with the standard MLS algorithm; is the original position of the th control point in the second image, is the updated position; is the local affine transformation applied to the point ; ​the position behind; is a weight function, whose value decreases with the increase of the distance between the point and the point , indicating the influence of different control points on the transformation at the point .

[0011] The second part is the structure regular term introduced by the present application . Wherein is a weight coefficient, used to balance the relationship between data fitting and structure keeping, the mathematical expression of the structure regular term is defined as follows: Let the local affine transformation at the point be represented by a 2x2 matrix , that is: ; In the formula, , , , are the four elements of the 2x2 matrix corresponding to ; then the structure regular term is: ; This expression theoretically punishes the non-similar transformation component, since a pure similar transformation must satisfy the Cauchy-Riemann equation, i.e. and . Therefore, when the transformation contains shear or non-uniform scaling, these equation relationships are broken. The defined by the present application is exactly the measure of the degree of deviation of the elements of the matrix from the similar transformation condition.

[0012] When or , the value of is greater than zero, thus imposing a penalty on the transformation in the process of minimizing the objective function , forcing the finally solved local affine transformation to approach the similar transformation as much as possible while meeting the control point constraints, thus mathematically ensuring the preservation of the local structure of the image.

[0013] To further enhance the intelligence of deformation control, when the received operation instruction is for a region structure primitive, the present scheme also provides a dynamic adjustment mechanism: for all points within the coverage range of the region structure primitive, when calculating the local affine transformation thereof, the weight coefficient of the structure regular term in its objective function is dynamically increased., to enhance the overall rigidity of the region during the deformation process, so that it is more in line with the user's intention to operate a complete object.

[0014] To achieve the real-time of the whole interaction process, the calculation process of the non-rigid deformation field is optimized. Instead of calculating the image pixel by pixel, a control mesh with non-uniform vertex density is first constructed on the second image. The construction of the mesh is based on the content features of the image (such as image gradient and corner response value). In the area with rich details and high feature response value, higher density of mesh vertices is generated, while in the smooth content area, sparse vertices are generated. The moving least squares deformation algorithm containing the structure regularization term is only applied to the vertices of the mesh, and the displacement of other points inside the mesh is quickly obtained by bilinear interpolation, thereby reducing the computational complexity while ensuring the deformation accuracy.

[0015] In addition, the present application also provides a diagnostic visual feedback method. By calculating the Jacobian matrix of the non-rigid deformation field at each point in real time , and performing singular value decomposition ( ) on it, two singular values and can be obtained. These two singular values quantitatively describe the local stretching and compression of the neighborhood of the point. According to these singular values, a distortion metric value is calculated and displayed in the form of a heat map on the deformed image, providing intuitive visual feedback to the user about the quality of the deformation.

[0016] After the user completes the interactive deformation, the present application also includes the step of generating a seamless splicing image. Specifically, in the overlapping area of the first image and the deformed second image, a visually optimal cut seam is calculated by the Graph-Cut algorithm, and the color consistency of the image areas on both sides of the cut seam is corrected to eliminate color differences, finally generating a geometrically accurate and color-unified fusion image.

[0017] To effectively manage complex interactive instructions, the present application also constructs all control points and semantic primitives into a hierarchical graph structure model. Among them, the semantic primitive node is the parent node, and all control point nodes associated with it are connected as child nodes. When receiving operation instructions for different levels of elements and there are conflicts (for example, first translating the parent node as a whole, then translating the child node independently), the nearest coverage principle is used for arbitration, that is, the operation instruction directly applied to the child node has higher priority than the instruction inherited from the parent node, so as to ensure that the user's fine adjustment intention can be accurately executed.

[0018] The present application provides an anti-parallax image non-rigid real-time interactive deformation method. It has the following beneficial effects: 1. This invention introduces a structure regularization term into the objective function of the moving least squares deformation algorithm. By quantifying and penalizing the dissimilar components in the local affine transformation, the final calculated non-rigid deformation field satisfies the control point constraints while maximally maintaining the local shape and angle of the image content, avoiding unnatural distortions and ensuring the accuracy of the deformation results.

[0019] 2. By introducing semantic primitives, this invention elevates the objects of user interaction from unstructured discrete control points to structured features with clear image content meaning. Therefore, users can directly perform overall geometric operations on contour lines or specific areas in the image, improving the intuitiveness and operational efficiency of the interaction.

[0020] 3. The present invention employs a strategy of constructing a control grid with non-uniform vertex density on a second image. This grid is generated based on content features such as image gradient and corner response. Vertices are denser in areas with rich detail and sparser in smooth areas, thereby significantly reducing the overall computational load and ensuring that users can obtain smooth and lag-free visual feedback when performing dragging, rotation and other operations. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the method flow of the present invention; Figure 2 This is a schematic diagram illustrating the user operation semantic primitives and associated control point position updates of the present invention. Figure 3 This is a schematic diagram of the hierarchical diagram structure of the present invention. Detailed Implementation

[0022] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] To better understand the present invention, the above content will be described in detail below with reference to specific embodiments.

[0024] Please see the appendix Figure 1 -Appendix Figure 3 This invention provides a parallax-resistant, non-rigid, real-time interactive image deformation method, which includes: Step S1: Acquire the first image and the second image, and determine an initial set of control point pairs; This step is the data input and initialization phase of the entire process. First, two images to be processed are loaded, named Image 1 and Image 2, respectively. Then, a set of feature points is detected and matched on both images. In one specific implementation, keypoints that are invariant to scale, rotation, and brightness changes are extracted from both images using Scale Invariant Feature Transform (SIFT) or similar feature description algorithms. By performing matching calculations on these descriptors and using the Random Sample Consensus (RANSAC) algorithm to eliminate erroneous matches, a set of feature point pairs with high-confidence correspondence between the two images is obtained. This set of point pairs is determined as the initial control point pairs.

[0025] Step S2: Extract a set of semantic primitives corresponding to the initial control point pairs from the first image and the second image; the semantic primitives are the structured image features of the control point pairs; This step first uses image analysis algorithms, such as Hough Transform and image segmentation, to identify and extract significant linear structures (e.g., building edges) and regional structures (e.g., windows) from two images. When the automatic extraction results are insufficient or fail to cover the user's target of interest, the invention also provides an interactive interface, allowing users to manually draw corresponding line segments or closed regions on the two images as a supplementary semantic primitive. After extracting or defining these geometric structures, all initial control point pairs that fall within the geometric range of a given structure are retrieved and associated with that structure. The final semantic primitive contains not only its own geometric description but also an index list of associated control point pairs.

[0026] Step S3: Receive user operation instructions for semantic primitives; All defined semantic primitives are visualized in a graphical interface. Users select one or more semantic primitives using an input device (such as a mouse) and perform geometric transformations such as translation, rotation, or scaling on them. These graphical operations are then captured, parsed, and quantified into a structured operation instruction. This instruction can record the unique identifier of the semantic primitive being operated on, as well as the specific geometric transformation parameters (e.g., a two-dimensional translation vector or an affine transformation matrix) that describe the user's operation.

[0027] Step S4: Update the control point positions associated with the semantic primitive being operated on, according to the operation instructions; This step is used to pass the user-applied operations on the high-level semantic structure to the associated low-level control points. First, based on the semantic primitive identifier in the operation instruction, all associated control point pairs are read from the primitive's data structure. Then, the geometric transformation parameters contained in the operation instruction are applied to these selected control points. Specifically, the original position coordinates of each associated control point in the second image are transformed using this geometric transformation, and then a new, updated target position coordinate is calculated.

[0028] Step S5: Based on the updated control point positions, use the moving least squares deformation algorithm to calculate the non-rigid deformation field acting on the second image; This step employs a moving least squares deformation algorithm, which introduces a structure regularization term into its objective function. The structure regularization term is used to adjust the dissimilar transformation components (such as shearing and non-uniform scaling) in the deformation field, thereby preserving the local structure of the image content to the greatest extent while satisfying the control point constraints.

[0029] Step S6: Apply a non-rigid deformation field to the second image to generate a deformed second image, and update the display in real time; This step uses the dense, non-rigid deformation field calculated in step S5 as a mapping function, applying it to the original second image. For each pixel in the deformed image, its sampling source location in the original second image is found through this deformation field, and its color value is calculated through texture interpolation. To ensure smooth interaction, the entire deformation application and image rendering process is typically performed on a constructed non-uniform mesh.

[0030] Specifically, in one particular embodiment: Two digital images to be deformed and stitched are loaded from one or more image data sources. The first image can be set as a reference, i.e., the target image; the second image is the source image to be non-rigidly deformed. Image data sources include, but are not limited to, image files on local storage devices, which can be in standard formats such as JPEG, PNG, and TIFF; or image data captured in real time by image acquisition devices connected to a computer system, such as digital cameras, industrial cameras, or scanners, through corresponding drivers.

[0031] After obtaining the original first and second images, a standardization preprocessing operation is performed on the images. Color information, resolution differences, and imaging noise present in the original image data will all affect the accuracy of subsequent control point pair determination.

[0032] The preprocessing operation includes grayscale conversion. Since many subsequent feature detection algorithms operate on single-channel grayscale images, it is necessary to convert the color first and second images into grayscale images. The conversion process is implemented using a weighted average formula, for example: ; in, Indicates pixel coordinates grayscale value at that location , and These represent the pixel points respectively. The intensity values ​​of the red, green, and blue channels.

[0033] Considering the performance requirements of real-time interaction, when the acquired image resolution is too high, the preprocessing operation also includes image size normalization, which is used to uniformly scale the first and second images to a preset processing resolution, such as 1280x720 pixels. Furthermore, image scaling can employ a bicubic interpolation algorithm to reduce the amount of data while preserving the image's detail information to the greatest extent possible.

[0034] Furthermore, to improve the robustness of feature detection, preprocessing operations may include noise suppression. This is achieved by applying a Gaussian blur filter to the grayscale image, which smooths the image by convolving it with a two-dimensional Gaussian kernel. (Two-dimensional Gaussian kernel function) The formula for expressing this can be: ; In the formula, It is the standard deviation of the Gaussian distribution, and its value determines the smoothness of the image; : Indicates the horizontal distance between the current point and the center point of the Gaussian kernel; : Indicates the vertical distance between the current point and the center point of the Gaussian kernel; The base of the natural logarithm, also known as the Euler number; It represents pi (π).

[0035] This step can effectively suppress high-frequency noise introduced by the image sensor; Therefore, the standardized first and second images obtained after the above preprocessing steps can provide an accurate information basis for subsequent steps.

[0036] After preprocessing the first and second images, an initial set of control point pairs can be determined. The purpose is to establish a geometrically corresponding point relationship between the two images, providing geometric constraints for subsequent non-rigid deformation. In a specific implementation, this can be achieved using feature detection, feature description, feature matching, and geometric consistency checks. Specific steps may include: First, the Scale Invariant Feature Transform (SIFT) algorithm is used to detect key points on the preprocessed first and second images. The SIFT algorithm can simulate the multi-scale space of the image by constructing a Gaussian difference pyramid and locate local extrema points in this space as key points. Therefore, the detected points have the characteristic of invariance to changes in the scale, rotation and brightness of the image.

[0037] For each detected keypoint, the SIFT algorithm is used to calculate the gradient direction and magnitude of its neighboring pixels and generate a 128-dimensional feature descriptor vector. This feature descriptor represents the local image region around the point, and then two sets of keypoints and their feature descriptors corresponding to the first and second images can be obtained respectively.

[0038] Then, the two sets of feature descriptors are matched. The matching process can use a nearest neighbor search algorithm. For each feature descriptor in the first image, the two closest Euclidean distances are found among all feature descriptors in the second image. For example, let the Euclidean distances between these two closest feature descriptors and their corresponding features be... and .

[0039] To improve matching accuracy, a ratio test can be used to filter matching results, that is, only matching results with a distance ratio of [missing information] can be selected. A match is considered accurate only when the distance is less than a preset threshold (e.g., 0.75), because a correct match should have a distance significantly smaller than the distance to its second nearest neighbor. Therefore, by filtering in this way, a set of candidate match pairs can be obtained.

[0040] Since the candidate matching pair set still contains false matches caused by repeated textures or occlusions in the image, the Random Sampling Consensus (RANSAC) algorithm can be used for geometric verification to remove these false matches. RANSAC iteratively selects a minimal subset (e.g., 4 matching pairs) from the candidate matching pair set and calculates a global geometric transformation model based on this subset, such as a 3x3 homography matrix. And this homography matrix This describes the projection relationship between two image planes. For each pair of points in the candidate matching pair set... ,in, It is a point in the first image. These are points in the second image, obtained through the calculated homography matrix. Point By projecting the image onto the plane of the second image, the projection point can be obtained. Then, the midpoint of the second image is calculated. projection point Matching points The reprojection error between them.

[0041] If the reprojection error is less than a preset distance threshold, then the matching pair Determined to conform to the current homography matrix An interior point is considered an interior point, while an exterior point is considered an exterior point. This process can be repeated up to a preset number of iterations, recording the number of interior points generated each time. Ultimately, the homography matrix with the maximum number of interior points is obtained. It was selected as the best geometric transformation model.

[0042] Therefore, by utilizing all matching pairs determined as interior points by the optimal geometric transformation model, a geometrically highly consistent set can be constructed. The interior point pairs that pass the consistency check in this set can be identified as the initial control point pairs in this invention. In an extended embodiment, to further ensure the smoothness of subsequent deformations, the set of interior point pairs can be spatially homogenized, for example, by dividing the image into grids and ensuring that only a few control point pairs with the highest confidence are retained within each grid, thereby obtaining a set of uniformly distributed and precisely matched initial control point pairs, providing a basis for extracting semantic primitives in step S2.

[0043] After determining the initial control point pairs, the method of the present invention proceeds to step S2, which aims to elevate the granularity of user interaction from discrete, unstructured single control points to higher-level, structured features with specific image content meaning.

[0044] In this invention, a semantic primitive is defined as a structured image feature of control point pairs. Specifically, a semantic primitive is a data structure that organizes multiple initial control point pairs that are spatially adjacent and together constitute a perceptible image feature with specific geometric or regional attributes into a single unit that can be manipulated as a whole.

[0045] Because a single control point pair can only provide a sparse point-to-point geometric mapping between two images, semantic primitives, on this basis, can aggregate these points to give them collective structural information. This structural information comes directly from the content of the image itself, such as the outline of a building or a region with similar color and texture.

[0046] Therefore, at the data level, a semantic primitive consists of two core parts: One is the geometric or regional descriptor of the primitive itself; for example, for a linear structure, it is the coordinates of its endpoints in the first and second images; for a regional structure, it is the boundary contour of its boundary in the two images. Secondly, there is the index list, which records the identifiers of all initial control point pairs belonging to the semantic primitive.

[0047] Therefore, by introducing semantic primitives, when a user operates on a semantic primitive, the intent of the operation is directly understood as an adjustment to a structural whole, improving the intuitiveness and efficiency of the interaction.

[0048] In a preferred embodiment, the linear structure primitive is implemented by performing several sub-steps on the first and second images: line segment detection, line segment description and matching, and association with initial control point pairs. This is used to identify obvious linear features in the images formed by building edges, road boundaries, etc., that have a corresponding relationship between the two images.

[0049] Specifically, the preprocessed first and second images are processed using a line segment detection algorithm, such as a Line-Segment Detector (LSD). The LSD algorithm can extract sub-pixel precision line segments from an image without parameter adjustment. After execution, the algorithm yields two independent sets of line segments, corresponding to the first and second images respectively. Each line segment is defined by the coordinates of its two endpoints in its respective image coordinate system.

[0050] To establish an accurate correspondence between these two sets of line segments, a feature descriptor with rotation and scale invariance needs to be computed for each line segment. This embodiment can employ the Line-Band-Descriptor (LBD) algorithm. The LBD algorithm constructs a local reference coordinate system for each line segment and calculates the pixel gradient statistics within the neighborhood band of that line segment, thereby generating a binary descriptor to characterize the local appearance around the line segment.

[0051] After all line segments have their LBD descriptors calculated, feature matching algorithms, such as nearest neighbor matching based on Hamming distance, can be used to find the line segment pairs with the most similar descriptors in two sets. The ratio test method can also be used to initially screen the matching results and eliminate fuzzy and ambiguous matches.

[0052] After obtaining the candidate line segment matching pairs, the final geometric consistency needs to be checked. This check utilizes the global homography matrix determined in step S1. This process is performed. For each pair of candidate matching line segments, the endpoints of the line segments in the first image can be processed through a matrix. The projection is onto the second image, and the distance between the projected endpoint and the corresponding line segment endpoint in the second image is calculated. Only when the distance is less than a preset pixel threshold is the line segment pair considered a geometrically consistent valid match.

[0053] Therefore, for each valid line segment matching pair that has been verified, the linear structure primitive is finally constructed by associating it. This operation searches the initial set of control point pairs and associates all control point pairs whose spatial locations are on the corresponding line segment in the first image or within its nearest neighbor with the line segment matching pair, thus forming a complete linear structure primitive.

[0054] In one specific embodiment, the formation of the region structure primitives further includes performing superpixel segmentation, region feature description and matching, and pairing with initial control points on the first and second images.

[0055] Specifically, the first and second images are processed by employing a superpixel segmentation algorithm, such as the Simple Linear Iterative Clustering (SLIC) algorithm. This algorithm performs local iterative clustering of pixels in the CIELAB color space and image space, reorganizing the original pixel-based image grid into perceptually more meaningful, irregular but compact block regions, i.e., superpixels. After the algorithm is executed, two independent sets of superpixels are obtained, where each superpixel is defined by a set of pixel coordinates.

[0056] To establish a correspondence between two sets of superpixels, a feature descriptor needs to be calculated for each superpixel. This descriptor characterizes the overall appearance attributes within the superpixel region. In one implementation, this feature descriptor consists of two parts: First, the average color vector of all pixels within the superpixel in the CIELAB color space; The other is a quantified color histogram; these two parts together describe the color distribution information of the area.

[0057] Furthermore, after calculating feature descriptors for all superpixels, matching is performed between the two sets. The matching process is achieved by calculating the distance between descriptors; for example, Euclidean distance is used to measure the difference between two average color vectors, and chi-squared distance is used to measure the similarity between two color histograms. By combining these two distances, for each superpixel in the first image, a best-matching superpixel in the second image is found, thus forming a set of candidate superpixel matching pairs.

[0058] Similar to the extraction process of linear structure primitives, this set of candidate superpixel matching pairs also needs to be processed through the global homography matrix. Perform a geometric consistency check. The specific implementation of this check is as follows: calculate the geometric center point (centroid) of the superpixels in the first image, and then use the homography matrix... Project it onto the coordinate system of the second image. If the spatial position of the projected point falls within the corresponding matching superpixel region in the second image, then the pair is confirmed as a geometrically consistent valid match.

[0059] For each validated superpixel match pair, a final association operation is performed. This operation searches the initial set of control point pairs, associating all control point pairs whose spatial locations fall within the corresponding superpixel regions of the first image with that superpixel match pair. Finally, a validated superpixel match pair, along with the indices of all the initial control point pairs it contains, constitutes a complete region structure primitive.

[0060] After extracting linear and regional structure primitives through the above-mentioned automated algorithm, the method of the present invention also provides a manual definition mechanism to provide users with a means of direct intervention when the number of automatically extracted semantic primitives is lower than a preset threshold or the automatic extraction results fail to cover the key structures of interest to the user.

[0061] The user selects a line drawing tool in the graphical user interface and draws a straight line segment on the first image displayed side by side by specifying the start and end points. Subsequently, the user draws another corresponding straight line segment on the second image and receives these two user-specified straight line segments as a pair of corresponding geometric descriptions.

[0062] Furthermore, users can select either a polygonal lasso or a free curve drawing tool to draw a closed outline on the first image to delineate a target area, and then draw the corresponding closed area on the second image. These two user-specified closed areas serve as a pair of corresponding geometric descriptions.

[0063] After the user completes the specification of a pair of geometric descriptions (whether line segments or regions), a search is performed in the initial set of control point pairs to associate all control point pairs whose spatial locations fall within the manually drawn geometric description in the first image or its preset neighborhood with the newly created primitive.

[0064] After establishing a set of semantic primitives that includes both automatically extracted and manually defined primitives, the process proceeds to step S3, which involves receiving user operation instructions for the semantic primitives. This is used to achieve human-computer interaction by converting the user's intuitive operation intentions into clear digital signals that the device can process.

[0065] In one specific implementation, all defined semantic primitives are first visualized in the graphical user interface. Specifically, linear structure primitives are displayed as highlighted line segments overlaid on corresponding positions in the first and second images; region structure primitives are displayed as semi-transparent, clearly defined colored polygonal regions. This visualization method provides users with clear and interactive operational targets.

[0066] Users can interact with the visualized semantic primitives using input devices such as a mouse. When a user moves the cursor within the geometric area of ​​a semantic primitive, the hover event can be detected, and the user can be given immediate feedback through visual effects, such as changing the highlight color of the primitive or thickening its borders, to indicate the currently selectable object.

[0067] When a user decides to manipulate a semantic primitive, they confirm their selection by clicking within the primitive's geometric scope. After selecting a semantic primitive, the user can perform one or more geometric transformations on it, including translation, rotation, and scaling.

[0068] Taking a translation operation as an example, the user presses a mouse button on the selected semantic primitive, drags the cursor to a new target position, and then releases the button. During this dragging process, two key screen coordinates are captured and recorded: the coordinates of the initial button press position and the coordinates of the final button release position.

[0069] The vector difference between these two coordinates can be calculated and determined as a two-dimensional translation vector. This translation vector, along with a unique identifier for the semantic primitive being operated on, is encapsulated into a structured operation instruction. This operation instruction serves as the input for the subsequent step S4, which updates the control point position, and quantifies the user's macroscopic interaction intent (e.g., "move this edge here") into a mathematical description that can be processed by the subsequent algorithm.

[0070] After receiving an operation instruction encapsulating the user's intent, step S4 is executed, which involves updating the control point positions associated with the semantic primitive being operated on, based on the operation instruction. In a specific embodiment, this update first determines the local affine transformation corresponding to the operation instruction. The local affine transformation is used to describe a mathematical model of the geometric changes the user applies to the semantic primitive. For example, when the user drags one endpoint of a linear structure primitive from position A to position A′ while keeping the other endpoint B stationary, a unique two-dimensional affine transformation is calculated. This transformation can simultaneously map point A to point A′ while keeping point B unchanged. It includes rotation, scaling, and translation components.

[0071] After determining the local affine transformation Next, all control points spatially attached to the semantic primitive being operated on are selected. This selection directly utilizes the associations established during the construction of the semantic primitive in step S2. Furthermore, each semantic primitive's data structure contains an index list that explicitly records all initial control point pairs belonging to that primitive. Therefore, this selection step specifically involves: based on the unique identifier of the semantic primitive in the operation instruction, directly reading the control point pair index list from its data structure to obtain the set of control points at the positions to be updated.

[0072] Finally, the local affine transformation is applied to all selected control points to determine their updated positions. For each control point in this set, its original position coordinates in the second image are represented as a homogeneous coordinate vector. Let the original homogeneous position coordinates of any such control point be... Local affine transformation It is represented as a 3x3 matrix.

[0073] Then the updated position of the control point, its homogeneous coordinates The following calculation is obtained through matrix-vector multiplication: ; in, These are the homogeneous coordinates of the original positions of the control points in the second image. It is a local affine transformation matrix determined according to user operation commands. These are the homogeneous coordinates of the updated position of the control point; This is the transpose operator. By performing this operation on all control points in the set, the positions of all associated control points are updated. , These are the homogeneous coordinates of the original position.

[0074] After updating the positions of the associated control points, the non-rigid deformation field acting on the second image is calculated based on the updated control point positions.

[0075] In a preferred embodiment, a Moving-Least-Squares (MLS) deformation algorithm is employed. However, standard MLS deformation algorithms are prone to excessive shearing or non-uniform scaling in localized areas when dealing with large parallax or complex deformations, thereby disrupting the intrinsic structure of the image content. To address this issue, this invention introduces a structure regularization term into the objective function of the Moving-Least-Squares deformation algorithm to penalize dissimilar transformation components in non-rigid deformation fields. In two-dimensional geometric transformations, similarity transformations consist only of translation, rotation, and uniform scaling, which preserve the local shape and angles of objects. Dissimilar transformation components, on the other hand, mainly refer to shearing and non-uniform scaling, and these components are the causes of local structural distortion.

[0076] Therefore, in order to mathematically penalize dissimilar transformation components, this invention applies the penalty to any point in the image. Solve for its optimal local affine transformation When using an objective function for calculation, its formula is: ; In the formula, the objective function consists of two parts. The first part is the data fitting term, which is consistent with the standard MLS algorithm; It is the first The original positions of the control points in the second image yes Updated location; It is a local affine transformation Applied to points The position after; It is a weighting function, the value of which varies with the point With point The decrease in distance between control points as the distance increases indicates that the calculation points are at different control points. The magnitude of the impact of the change.

[0077] The second part is the structural regularization term introduced in this invention. .in These are weighting coefficients used to balance the relationship between data fitting and structure preservation; they are also known as structure regularization terms. The mathematical expression for is defined as follows: Setting points Local affine transformation at The linear part consists of a 2x2 matrix. This means, that is: ; In the formula, , , , yes The four elements of the corresponding 2x2 matrix; then the structure regularization term is: ; This expression, in principle, penalizes dissimilar transformation components, because for simple similar transformations, their linear matrix parts must satisfy the Cauchy-Riemann equations, i.e. and Therefore, these equations are broken when the transformation involves shearing or non-uniform scaling. This invention defines... It is precisely the measurement of the matrix The degree to which the elements deviate from the similarity transformation conditions.

[0078] when or hour, The value of is greater than zero, thus minimizing the objective function The process applies a penalty to the transformation, forcing the final solution to obtain a local affine transformation. While satisfying control point constraints, it approaches similarity transformation as closely as possible, thus mathematically guaranteeing the preservation of the local structure of the image.

[0079] In the objective function that includes the structure regularization term, the weight coefficients... The value of directly determines the strength of the structural preservation constraint. In order to achieve differentiated deformation control for different image content regions, this invention further provides a mechanism for dynamically and adaptively adjusting the weight coefficient.

[0080] Specifically, the dynamic adjustment mechanism is activated when the user selects and manipulates an object that is a region structure primitive. Since a region structure primitive is defined as a whole in an image with inherent structural consistency, such as the facade of a building, operations applied to it can move or rotate it as a rigid or semi-rigid whole.

[0081] To achieve this in deformation calculations, the structural integrity within the region can be improved. Specifically, the method involves first identifying the set of all pixels belonging to that region in the second image based on the geometric description of the region's structural primitives.

[0082] Then for any point in this set of pixels Calculate its local affine transformation At that time, the weight coefficients in the objective function it uses It will be set to a preset higher value. For all pixels not located within this region, the weighting coefficients used in the calculation process... Then it will remain at a default baseline value. ,in > .

[0083] In this way, when solving for the optimal local affine transformation of pixels within a region, the objective function... Medium structure regularization term The proportion of this will be increased, thus making the optimization process more inclined to minimize the structure regularization term. This constrains the local transformations of all points within the region to approximate similar transformations, thereby making the manipulated region structure primitives exhibit stronger integrity and rigidity during the deformation process, and more effectively preserving their internal detailed structures, thus enabling users to make overall adjustments to specific regions.

[0084] Furthermore, to ensure the real-time performance of the entire interactive deformation process, optimizing the computational efficiency of the aforementioned moving least squares deformation algorithm is crucial. However, directly solving for the optimal local affine transformation of each pixel in the second image is computationally too costly and cannot meet the requirements of real-time interaction. Therefore, this invention does not perform dense pixel-by-pixel calculations on the image, but instead first constructs a control grid with non-uniform vertex density on the second image. Since the calculation of the non-rigid deformation field is only performed at the vertices of the grid, the displacements of other points within the grid are obtained through rapid interpolation of the vertex displacements of their respective grid cells.

[0085] Therefore, the adaptive mesh is constructed based on the content features of the second image itself, specifically the image gradient and corner response values. Specifically, it involves: first, calculating a feature response map representing the complexity of the image content, which includes two sub-steps: First, by applying the Sobel operator, the gradients of the original second image in the horizontal and vertical directions are calculated, and a gradient magnitude map is synthesized, which can highlight the edge contours in the image.

[0086] Second, by applying the Harris corner detection algorithm, the corner response value of each pixel in the image is calculated, and the response value is high at the corners and in areas with rich texture.

[0087] The gradient magnitude map and corner response map are then weighted and fused to generate the final feature response map. In this map, regions with higher pixel values ​​correspond to regions with richer details in the original image.

[0088] Subsequently, based on this feature response map, a quadtree recursive partitioning method is used to generate a grid with non-uniform vertex density.

[0089] Finally, the final result of this recursive partitioning process is a grid with non-uniform vertex density. In regions with rich image details and high feature response values, the quadtree partitioning is deeper, resulting in a higher density grid vertex distribution; while in regions with smooth image content and low feature response values, the partitioning terminates prematurely, resulting in a sparser grid of vertices. Simply put, regions with rich image details and high feature response values ​​have more and denser grids, while regions with smooth image content and low feature response values ​​have fewer and sparser grids. During deformation calculations, the moving least squares deformation algorithm of this invention, which includes a structure regularization term, is applied only to each vertex of the adaptive mesh to solve for the deformed position of each vertex. For any other pixel within a mesh cell, its deformed position is quickly calculated by bilinear interpolation of the deformed positions of the four vertices of that cell. This reduces the overall computational load while maintaining the deformation accuracy in detailed regions.

[0090] After calculating the deformed positions of the mesh vertices using an adaptive mesh optimization strategy, step S6 is executed, which applies the sparse deformation result to generate a complete second image that has undergone non-rigid deformation and displays it in real time.

[0091] In one specific implementation, this process is achieved through a graphics hardware-accelerated image texture mapping process. In this process, the original second image is loaded as a two-dimensional texture into the video memory of the graphics processing unit (GPU). Simultaneously, the adaptive deformable mesh with updated vertex positions constructed in step S5 is used as a geometry rendering unit.

[0092] The deformed mesh is then decomposed into quadrilateral or triangular units. For each unit, the original positions of its four (or three) vertices are used as texture coordinates sampled from the second image texture; while its updated positions are used as vertex coordinates for drawing on the screen.

[0093] When the graphics rendering pipeline processes these mesh cells, it interpolates the vertex coordinates of each pixel within a cell to determine its final position on the screen. Simultaneously, it interpolates the texture coordinates corresponding to that pixel to determine the coordinates of a sampling point in the original second image texture. Since the coordinates of this sampling point are usually not integers, a bilinear interpolation algorithm is used to calculate the accurate color of the sampling point based on the color values ​​of its four nearest surrounding texture pixels. Finally, this color value is assigned to the corresponding pixel on the screen.

[0094] Furthermore, by performing the aforementioned texture mapping and rendering operations on all cells of the deformed mesh, the method of the present invention can output a complete and smooth second image that has undergone non-rigid deformation and display it on a display device.

[0095] In addition to providing real-time display of deformation results, this invention further provides a diagnostic visualization feedback method to help users make more precise and expected adjustments to the deformation. In one specific embodiment, the process first performs local differential analysis on the current non-rigid deformation field. For any point in the deformed second image, its position is derived from a point in the original second image through a non-rigid deformation function. To quantify the local deformation characteristics of the neighborhood of that point, the Jacobian matrix of the deformation function at that point is calculated. The Jacobian matrix can be a 2x2 matrix, consisting of four first-order partial derivatives of the deformation function with respect to the X and Y coordinates, and is used to describe the best linear approximation of the complex non-rigid deformation at that point.

[0096] Therefore, the Jacobian matrix of this local linear approximation is obtained. Then, singular value decomposition (SVD) is performed on it to deconstruct its intrinsic geometric transformation components. Jacobian matrix The singular value decomposition expression is: ; in, and These are two 2x2 orthogonal matrices, each representing a rotation transformation. yes The transpose of the matrix; It is a 2x2 diagonal matrix, and its diagonal elements are... and These are called singular values; This is the transpose operator.

[0097] These two singular values and These represent the maximum and minimum scaling ratios along two mutually orthogonal principal directions at that point, respectively. Therefore, these two singular values ​​directly and quantitatively reveal the degree of stretching and compression of the local region. If If the singular value is greater than 1, it indicates that the rotation at that point is purely rigid, and no shape change has occurred. If the singular value is greater than 1, it indicates that stretching has occurred in that direction; if it is less than 1, it indicates that compression has occurred.

[0098] To present this quantification result to the user, a scalar distortion metric is calculated, which can be defined as follows: ; This value measures the degree to which local deformation deviates from the rigid transformation. Therefore, this distortion metric can be calculated at each vertex of the adaptive mesh, or at a set of uniformly sampled points.

[0099] Finally, these calculated distortion metrics are converted into a pseudo-color image, or heatmap, using a pre-defined color map. In this heatmap, color variations (e.g., from blue to red) visually represent the magnitude of the distortion metric. For example, low-distortion areas close to zero are displayed in blue, while high-distortion areas with significant stretching or compression are displayed in red. This heatmap can be overlaid as a semi-transparent layer on top of the deformed second image. By observing this heatmap, users can visually identify areas where unnatural distortion occurs during the deformation process and adjust the relevant semantic primitives accordingly to achieve the desired deformation effect.

[0100] Once the user is satisfied with the geometric alignment of the second image through the aforementioned interactive and diagnostic visual feedback, the aligned images can be fused to generate a seamless stitched image, including: Calculate and determine an optimal seam. Within the overlapping region of the first image and the second image after non-rigid deformation, for each pixel location, it is necessary to determine whether its final color originates from the first or second image. A seam is formed by combining the boundaries of these decision points. To make this seam appear visually natural, the graph-cut algorithm can be used to find an optimal seam that minimizes energy. The specific implementation of this algorithm is as follows: (1) First, a graph is constructed, where each pixel in the overlapping region corresponds to a node in the graph. Additionally, a source node (representing the first image) and a sink node (representing the deformed second image) are introduced. Two types of edges are defined in the graph: first, data edges connecting the source and sink nodes to the pixel nodes, whose weights reflect the cost of assigning a pixel to an image; second, neighborhood edges connecting adjacent pixel nodes, whose weights are defined as the cost of setting a cut between these two pixels. This cost is a function of the color difference between the two images at that location: if the color and gradient differences between the two images at adjacent pixels are small, the cost is low; otherwise, it is high. By solving for the minimum cut in the graph, all pixel nodes can be divided into two disjoint sets, one connected to the source node S and the other connected to the sink node T. The optimal cut occurs when the boundary between these two sets corresponds to a path with the minimum total cost in the image space.

[0101] (2) After determining the optimal cut, color consistency correction is performed to eliminate differences in hue, brightness, and contrast caused by shooting the two images under different lighting and exposure conditions. In one embodiment, within the overlapping area, color statistics of the first image and the deformed second image are calculated, such as the mean and standard deviation of each color channel. Subsequently, a color transformation function, such as a simple linear gain model or a more complex gamma correction function, is calculated and applied to the deformed second image to match its color statistics in the overlapping area with those of the first image. After applying this color correction to the entire image, local feathering or Poisson-blending operations can be performed within the narrow transition band around the optimal cut to further smooth any remaining subtle color transitions, ensuring that the final output stitched image achieves a highly consistent and seamless visual effect in both geometry and color.

[0102] In one specific implementation, the present invention can also organize all initial control point pairs and semantic primitives into a graph structure model with a clear parent-child hierarchical relationship, which consists of two types of nodes: The first is the control point node, which is located at the lowest level of this hierarchy. Each control point node uniquely corresponds to a specific pair of initial control points. The second is the semantic primitive node, which is located at a higher level of this hierarchical structure. Each semantic primitive node uniquely corresponds to a linear or regional structure primitive formed through automatic extraction or manual definition.

[0103] The parent-child hierarchy in this graph structure is a direct and formalized representation of the association operation performed in step S2. When a control point is spatially determined to be attached to a semantic primitive, a directed edge is established in this graph structure model from the node corresponding to the semantic primitive (as the parent node) to the node corresponding to the control point (as the child node).

[0104] Therefore, this method can present one or more tree-like structures, in which semantic primitive nodes serve as parent nodes, directly connecting to all associated control point nodes as their child nodes. Each control point node, as a leaf node in this hierarchical structure, has only one parent node, namely the unique semantic primitive to which it belongs. Furthermore, it can organize image content from the low-level, discrete geometric points to the high-level, continuous semantic structure into a unified, multi-layered representation, thus providing the path and structural foundation for subsequent deformation effects. This ensures that operations applied to any parent node (semantic primitive) can be accurately transmitted to all its child nodes (control points), thereby achieving more complex and coordinated deformation control.

[0105] Furthermore, in an extended embodiment, based on this hierarchical graph structure model, user interactions can be applied to nodes at different levels. This necessitates avoiding conflicts between user commands. Therefore, this extended embodiment provides a conflict arbitration mechanism based on the principle of proximity coverage. This arbitration mechanism aims to resolve the problem of determining the final state when a lower-level element (e.g., a control point node) simultaneously receives contradictory operation commands from different levels. For example, a user first performs a global translation operation on a parent node (e.g., a region structure primitive), and then performs an independent translation operation at a different location on a child node within that region (i.e., a specific control point).

[0106] In this scenario, the control point node undergoing the secondary operation logically receives two instructions: first, the initial position update inherited as a parent node member and defined by the overall region translation; and second, a position adjustment directly applied by the user. Since these two instructions typically specify different target locations, a conflict arises.

[0107] To avoid this phenomenon, in a hierarchical graph structure, an operation instruction applied directly to a node always has higher priority than operation instructions inherited from any of its parent nodes. In other words, the instruction closest to the object being operated on will override all broader and more indirect instructions.

[0108] According to this principle, in the aforementioned conflict scenario, when calculating the final updated position for a specific control point, the signal for adjusting the position directly specified by the user in the second instance will be fully adopted, ignoring the position inherited from the first translation operation of the parent node region; thus ensuring that the user's operation can be executed.

Claims

1. A parallax-resistant, non-rigid, real-time interactive image deformation method, characterized in that, Includes the following steps: Step S1: Acquire the first image and the second image, and determine an initial set of control point pairs; Step S2: Extract a set of semantic primitives corresponding to the initial control point pairs from the first image and the second image; the semantic primitives are the structured image features of the control point pairs; Step S3: Receive user operation instructions for the semantic primitive; Step S4: Update the control point position associated with the semantic primitive being operated on according to the operation instruction; Step S5: Based on the updated position of the control points, the non-rigid deformation field acting on the second image is calculated using the moving least squares deformation algorithm; Step S6: Apply the non-rigid deformation field to the second image to generate a deformed second image, and update and display it in real time.

2. The parallax-resistant, non-rigid, real-time interactive image deformation method according to claim 1, characterized in that, In step S2, the semantic primitives include: Linear structure primitives extracted using a line segment detection algorithm; Region structure primitives extracted using a superpixel segmentation algorithm.

3. The parallax-resistant, non-rigid, real-time interactive image deformation method according to claim 1, characterized in that, In step S4, the step of updating the control point position associated with the semantic primitive being operated on specifically includes: Determine the local affine transformation corresponding to the operation command; Filter out all control points that are spatially attached to the semantic primitive being operated on; Apply a local affine transformation to all selected control points to determine their updated positions.

4. The parallax-resistant, non-rigid, real-time interactive image deformation method according to claim 1, characterized in that, In step S5, the objective function of the moving least squares deformation algorithm includes a structure regularization term, which is used to penalize the dissimilar transformation components in the non-rigid deformation field.

5. The parallax-resistant, non-rigid, real-time interactive image deformation method according to claim 1, characterized in that, When the received operation instruction is for a region structure primitive, the method further includes: The weight coefficient of the structure regularization term in the objective function is dynamically increased within the area covered by the regional structure primitives to enhance the structural preservation of the coverage area during deformation.

6. The parallax-resistant, non-rigid, real-time interactive image deformation method according to claim 1, characterized in that, In step S5, the calculation of the non-rigid deformation field is achieved by constructing a mesh on the second image, wherein the mesh has the highest vertex density in the region of the second image with the highest image gradient and corner response value.

7. The parallax-resistant, non-rigid, real-time interactive image deformation method according to claim 1, characterized in that, The method further includes: The Jacobian matrix of the non-rigid deformation field is calculated in real time. The Jacobian matrix is ​​subjected to singular value decomposition, and the degree of stretching and compression of each region of the image is visualized in the form of a heatmap based on the decomposed singular values.

8. The parallax-resistant, non-rigid, real-time interactive image deformation method according to claim 1, characterized in that, In step S2, the step of extracting a set of semantic primitives corresponding to the initial control point pairs includes: When the number of extracted semantic primitives is lower than a preset threshold, the system receives the corresponding linear structure primitives and regional structure primitives that are manually drawn and specified by the user.

9. A non-rigid real-time interactive image deformation method for parallax resistance according to claim 1, characterized in that, Step S6, after the real-time update display, also includes: Within the overlapping area of ​​the first image and the deformed second image, the optimal cut is calculated using a graph cutting algorithm; Color consistency correction is performed on the image regions on both sides of the optimal cut.

10. A non-rigid real-time interactive image deformation method for parallax resistance according to claim 1, characterized in that, The method further includes: The control points and semantic primitives are constructed into a hierarchical graph structure; When multiple conflicting operation instructions are received for elements at different levels, arbitration is carried out using the principle of proximity coverage, that is, operation instructions for child nodes have higher priority than operation instructions for the parent nodes of child nodes.

Images