A stitching camera local mesh adaptive morphing system based on dense optical flow field estimation

Abstract

This invention relates to the field of image processing technology and is a local mesh adaptive deformation system for stitched cameras based on dense optical flow field estimation. It includes: an optical flow field calculation module, which acquires the dense optical flow field and calculates the maximum eigenvalue distribution of the optical flow gradient structure tensor; a motion boundary detection module, which extracts single-pixel wide motion boundary lines; an anisotropic deformation constraint module, which assigns deformation constraint matrices with strong normal constraints and weak tangential constraints to mesh edges crossed by the boundary lines; a two-degree-of-freedom control point construction module, which generates two-degree-of-freedom control points with independent foreground and background offsets but sharing the same initial coordinates at the mesh vertices traversed by the boundary lines; and a two-branch image fusion module, which constructs Sigmoid-type fusion weights to perform weighted fusion of the foreground alignment map and the background alignment map. This invention achieves deformation decoupling between the moving foreground and the static background, effectively eliminating artifacts at motion boundaries.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically a local mesh adaptive deformation system for stitched cameras based on dense optical flow field estimation. Background Technology

[0002] Multi-camera stitching is one of the fundamental technologies in computer vision and image processing, widely used in scenarios such as panoramic imaging, video surveillance, and environmental perception for autonomous driving. Its core task is to generate a seamless wide-field-of-view stitched image by geometrically aligning and fusing images with overlapping areas captured by multiple cameras at the same time.

[0003] Existing stitching methods are mostly based on mesh deformation frameworks. A typical process involves: constructing a control mesh within the overlapping image region; establishing the correspondence between mesh vertices using image matching algorithms (such as feature point matching or dense optical flow estimation); solving for the displacement of each vertex by minimizing an energy function composed of data terms and regularization terms; and finally mapping and fusing the images based on the obtained deformed mesh. Among these methods, APAP (As-Projective-As-Possible) effectively handles alignment errors caused by parallax through local homography transformation and has become a representative solution in this field.

[0004] However, the above methods face fundamental difficulties when moving objects are present in the overlapping region. The projected positions of the moving objects on the imaging planes of the two cameras differ due to the object's own motion, resulting in significant discontinuities in the amplitude and direction of the optical flow vector in this region compared to the static background region. Traditional mesh deformation frameworks assume that the entire scene can be described by a globally smooth displacement field. When encountering moving boundaries, the smoothing constraint of the regularization term forces the displacement of the moving object to propagate to the adjacent static background region, causing non-physical distortion of the background around the moving object. At the same time, obvious artifacts such as streaks, jagged edges, and black borders are also produced at the contours of the moving object itself due to the displacement discontinuity.

[0005] For dynamic scene stitching, existing technologies attempt to introduce a mesh fragmentation strategy, which involves severing the topological relationship of the original control mesh along the motion boundary, allowing the foreground and background to deform independently. This type of method alleviates the problem of deformation error propagation across boundaries to some extent. However, the fragmentation of the mesh topology disrupts the symmetry and positive definiteness of the deformation energy matrix, potentially introducing matrix singularities. Furthermore, the lack of effective constraints on the normal direction at the mesh vertices on both sides of the fragmentation point easily leads to sub-pixel-level misalignment, resulting in black edges or holes in the stitched result. In addition, the transition between the foreground and background at the boundary after mesh fragmentation is also difficult, and hard boundary blending easily produces visible jagged artifacts.

[0006] Therefore, how to effectively isolate deformation interference between the moving foreground and the static background while maintaining the topological integrity of the control mesh, and achieve independent modeling and continuous transition of displacement on both sides of the moving boundary, has become a key technical problem that needs to be solved in this field. Summary of the Invention

[0007] The purpose of this section is to outline some aspects of embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0008] To overcome the problems existing in the prior art as pointed out in the background art, the present invention proposes a local mesh adaptive deformation system for stitched cameras based on dense optical flow field estimation, comprising the following modules: The optical flow field calculation module is used to obtain the dense optical flow field of the left and right views of the overlapping area. After Gaussian smoothing the dense optical flow field, the spatial gradient of each pixel is calculated and the optical flow gradient structure tensor is constructed. The maximum eigenvalue of the structure tensor is solved to obtain the maximum eigenvalue distribution. The motion boundary detection module is used to take the maximum feature value distribution as a grayscale response map, divide the image plane into multiple non-overlapping local blocks, independently calculate the binarization threshold in each local block and combine it with the global threshold for binarization, and perform non-maximum suppression and thinning on the binarization result to obtain a single-pixel wide motion boundary line. The anisotropic deformation constraint module is used to construct a control mesh on the image plane and determine whether each mesh edge intersects with the motion boundary line. For intersecting mesh edges, the local tangent and normal directions of the motion boundary line at the intersection point are obtained, and an anisotropic deformation constraint matrix is ​​constructed for the mesh edge. The constraint response of this matrix in the normal direction is greater than the constraint response in the tangent direction. For non-intersecting mesh edges, isotropic constraints are maintained. The two-degree-of-freedom control point construction module is used to construct two-degree-of-freedom control points at the grid vertices traversed by the motion boundary line. These control points share the same initial coordinates but have independent foreground and background offsets. The foreground and background offsets participate in the deformation solution as two independent degrees of freedom without changing the topological adjacency relationship of the original control grid. The dual-branch image fusion module is used to resample the original image based on the solved foreground offset and background offset to obtain a foreground alignment map and a background alignment map. Based on the signed distance of each pixel to the nearest moving boundary line, the module uses a sigmoid function of signed distance to calculate nonlinear fusion weights and performs weighted fusion on the two alignment maps to obtain the final stitched image.

[0009] Preferably, in the motion boundary detection module, the global threshold is determined by weighted summation of the mean and standard deviation of the entire grayscale response image, and the weighting coefficient ranges from 2.5 to 3.5; the number of local blocks is adaptively set according to the image resolution.

[0010] Preferably, in the anisotropic deformation constraint module, the tangent direction and normal direction of the moving boundary line at the intersection point are obtained by principal component analysis of the boundary line pixel sequence; the anisotropic deformation constraint matrix is ​​jointly determined by the normal constraint weight and the tangential constraint weight, wherein the ratio of the normal constraint weight to the tangential constraint weight is not less than 100.

[0011] Preferably, in the dual-degree-of-freedom control point construction module, the determination condition for the grid vertex through which the motion boundary line passes is: at least one of the grid edges connected to the vertex is determined to intersect with the motion boundary line.

[0012] Preferably, the stitched camera local mesh adaptive deformation system further includes a deformation solving module, used to construct an energy function and solve for the offset of all control points; The deformation solving module uses the motion boundary line to divide the dense optical flow field into a foreground optical flow field and a background optical flow field. The division is based on the signed distance of each pixel to the motion boundary line. The energy function includes a data term and a regularization term. In the data term, for a two-degree-of-freedom control point, the target position of its foreground controller is determined by the value of the foreground optical flow field at the corresponding coordinates, and the target position of its background controller is determined by the value of the background optical flow field at the corresponding coordinates. In the regularization term, for a grid edge that crosses the motion boundary line, the displacement difference between its two control points is decomposed into normal and tangential components along the boundary line normal and tangential directions. The normal component is constrained by a higher normal constraint weight, and the tangential component is constrained by a lower tangential constraint weight. By minimizing this energy function, we can solve for the single offset of all ordinary vertices and the foreground and background offsets of all two-degree-of-freedom control points.

[0013] Preferably, the data item further includes a local confidence weight for each control point, which is determined by the product of the optical flow estimation confidence and the image texture richness; wherein the optical flow estimation confidence is calculated based on the normalized cross-correlation cost, and the image texture richness is calculated based on the gradient magnitude of the neighborhood of the control point; when the gradient magnitude of the neighborhood of the control point is less than half of the average gradient magnitude of the entire image, the local confidence weight of the control point is reduced.

[0014] Preferably, the energy function is formed by multiplying the data term and the regularization term by the weights of the data term and the regularization term, respectively, and then summing the results. The ratio of the weight of the data term to the weight of the regularization term ranges from 0.1 to 10.0.

[0015] Preferably, in the dual-branch image fusion module: For ordinary mesh vertices, both their foreground and background offsets are taken as the single offset of that vertex; for two-degree-of-freedom control points, their independently solved foreground and background offsets are taken respectively. The total foreground offset and total background offset of each pixel are obtained by bilinear interpolation, and then the foreground mapping coordinates and background mapping coordinates are obtained. The foreground alignment map and background alignment map are obtained by bicubic inverse resampling. In the signed distance, the sign is determined as follows: the optical flow vectors of pixels within a preset width range on both sides of the motion boundary line are subjected to binary clustering. The side corresponding to the cluster with the larger average optical flow amplitude is determined as the foreground side and assigned a positive sign; the other side is determined as the background side and assigned a negative sign; the distance value is the Euclidean distance from the pixel to the nearest motion boundary line.

[0016] Preferably, the Sigmoid function has a steepness parameter that controls the steepness of the transition zone and an offset parameter that adjusts the center of the transition zone, wherein the steepness parameter ranges from 0.5 to 3.0.

[0017] Compared with the prior art, the technical effects of the present invention are as follows: First, by introducing a two-degree-of-freedom control point mechanism, the mesh vertices traversed by the moving boundary line simultaneously possess two independent degrees of freedom: foreground offset and background offset. This achieves deformation decoupling between the moving foreground and the static background without altering the original mesh topology adjacency relationships. This fundamentally avoids the matrix singularity problem caused by topology reconstruction in traditional mesh splitting schemes, ensuring the numerical stability of deformation solutions.

[0018] Second, by designing an anisotropic deformation constraint matrix, a differentiated constraint strategy is applied to the mesh edges crossing the motion boundary line, with strong normal constraints and weak tangential constraints. The high constraint weight in the normal direction forces the displacement projections on both sides of the boundary to tend to be consistent, with the difference controlled at the sub-pixel level, effectively preventing black borders or holes in the background caused by misalignment; the low constraint weight in the tangential direction allows relative slippage on both sides of the boundary, completely preserving the displacement discontinuity of the moving object relative to the background.

[0019] Third, in the image fusion stage, a sigmoid-type nonlinear weighting function based on signed distance is used to replace traditional hard boundary fusion or fixed weight fusion. The width and steepness of the fusion transition band are continuously adjustable by parameters, which can adaptively match the displacement difference amplitude on both sides of the moving boundary, achieving a smooth transition while preserving boundary sharpness and eliminating jagged and stringy artifacts at hard edges.

[0020] In summary, this invention significantly improves the image quality at the boundaries of moving objects in dynamic scene stitching under controllable computational overhead. The peak signal-to-noise ratio and structural similarity index within the boundary transition zone are superior to existing technologies, and visible boundary artifacts are effectively eliminated, demonstrating good practical value. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 This is a schematic diagram of the local mesh adaptive deformation system of the stitched camera based on dense optical flow field estimation according to the present invention; Figure 2 This is a schematic diagram illustrating the principle of image stitching in a real vehicle scene according to the present invention; Figure labels: A: Left camera image; B: Right camera image; C: Artifacts and black borders present in direct stitching; D: Adaptive local mesh deformation process; E: Final seamless stitching result. Detailed Implementation

[0022] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0023] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0024] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0025] Example 1: like Figure 1 As shown in the figure, the local mesh adaptive deformation system of the stitched camera based on dense optical flow field estimation in this embodiment of the invention is as follows: Figure 1 As shown, it includes the following modules: The optical flow field calculation module is used to acquire dense optical flow field data from the left and right views of the overlapping region, and to calculate the local gradient structure tensor of the optical flow field and its maximum eigenvalue distribution. The motion boundary detection module is used to extract single-pixel wide motion boundary lines representing motion boundaries based on the maximum eigenvalue distribution of the structure tensor through adaptive threshold segmentation and morphological thinning. The anisotropic deformation constraint module is used to reassign anisotropic deformation constraint matrix with strong normal constraints and weak tangential constraints to the mesh edges crossed by the boundary line, based on the spatial orientation of the motion boundary line. The two-degree-of-freedom control point construction module is used to generate two-degree-of-freedom control points that share the same initial coordinates but have independent foreground and background offsets at the grid control points traversed by the motion boundary line, without changing the original grid topology adjacency relationship. The dual-branch image fusion module is used to resample the original image according to the foreground offset and background offset respectively to obtain the foreground-aligned image and the background-aligned image. It also constructs a non-linear weight function based on the signed distance from the pixel to the motion boundary line to fuse and output the dual-branch image.

[0026] In this embodiment, the optical flow field calculation module acquires dense optical flow field data from the left and right views of the overlapping region, and calculates the local gradient structure tensor of the optical flow field and its maximum eigenvalue distribution as follows: The overlapping area image captured by the left camera is denoted as The overlapping area image captured by the right camera is denoted as A dense optical flow estimation algorithm based on iterative pyramids is used to calculate... arrive pixel-level displacement vector field ,in These are the pixel coordinates on the image plane. and These are the displacement components of the pixel in the horizontal and vertical directions, respectively.

[0027] To suppress the interference of image noise and optical flow estimation errors on gradient calculation, the optical flow field is first... Perform Gaussian smoothing filtering, filter kernel standard deviation Pixel. For the smoothed optical flow field, calculate each pixel. Spatial gradient at: ; Then construct the optical flow gradient structure tensor of this pixel. : ; Solve the structure tensor Maximum eigenvalue The maximum eigenvalue characterizes the maximum rate of change of the optical flow field at that pixel. The larger the value, the more drastic the spatial change of the optical flow field in the neighborhood of that pixel, that is, the more significant the displacement discontinuity boundary.

[0028] In this embodiment, the motion boundary detection module, based on the maximum eigenvalue distribution of the structure tensor, extracts the single-pixel wide motion boundary line representing the motion boundary through adaptive threshold segmentation and morphological thinning. The specific process is as follows: The distribution of the largest eigenvalue is regarded as a gray-level response map. A block-based adaptive thresholding strategy is applied to the grayscale response map: the image plane is divided into... A number of non-overlapping local blocks, among which This represents the number of blocks in the horizontal direction. This refers to the number of blocks in the vertical direction. Preferably, for Image of resolution, take , ;for Image of resolution, take , .

[0029] The binarization threshold for each local block is calculated independently using Otsu's method. .

[0030] If a certain pixel Greater than the threshold of its local block And simultaneously greater than the global adaptive threshold If so, then the pixel is marked as a candidate edge pixel.

[0031] The global adaptive threshold is determined by the mean and standard deviation of the entire response plot: ,in and They are respectively The mean and standard deviation of the distribution across the entire graph. This is an empirical coefficient used to control the sensitivity of the global threshold.

[0032] For resolution at to The image, The preferred value range is .

[0033] Non-maximum suppression is performed on the binary image composed of the candidate edge pixels, and then the Zhang-Suen parallel thinning algorithm is used for iterative thinning until the width of all edge lines is compressed to the single pixel level.

[0034] The set of single-pixel wide lines obtained after thinning is denoted as the motion boundary line set. Each of the movement boundary lines From its ordered pixel sequence express, This is an index for the arc length parameter along the boundary line.

[0035] In this embodiment, the specific process by which the anisotropic deformation constraint module reassigns anisotropic deformation constraint matrices with strong normal constraints and weak tangential constraints to the mesh edges traversed by the boundary lines, based on the spatial orientation of the motion boundary lines, is as follows: Constructing the initial stitching control grid ,in For the set of grid vertices, This is the set of mesh edges connecting adjacent vertices. The control mesh is a uniform quadrilateral mesh or a Delaunay triangular mesh. For images with a resolution higher than 20 megapixels, the preferred mesh vertex spacing is... Pixels; for images with a resolution below 5 megapixels, the preferred grid vertex spacing is... Pixel.

[0036] Traverse the grid edge set Each grid edge in Determine whether it is related to the set of motion boundary lines. It intersects with any of the boundary lines in the region.

[0037] Intersection determination is achieved by calculating the geometric intersection point between the grid edge line segment and the polyline segment formed by each continuous pixel point of the motion boundary line: if the line segment and the polyline segment have a common point, they are determined to be intersecting, and the intersection point is used as the position where the constraint is applied.

[0038] For the grid edge intersecting the motion boundary line Perform anisotropic deformation constraint reconstruction: Local tangent direction at the intersection By performing principal component analysis on several points (e.g., 3 pixels before and after) of the boundary line pixel sequence before and after the intersection point, we obtain: the eigenvector corresponding to the largest eigenvalue is taken and normalized, which is... Local normal direction Depend on The mesh is obtained by rotating it 90 degrees clockwise and normalizing it. An anisotropic deformation constraint matrix is ​​then assigned to the mesh edges. : ; in, Represents the tensor product of vectors. For normal constraint weights, The tangential constraint weight is set to be much larger than the tangential constraint weight to achieve anisotropic deformation control with strong normal constraints and weak tangential constraints.

[0039] Preferably, the ratio of the normal constraint weight to the tangential constraint weight satisfies More preferably, The effect of this constraint is directly reflected in the displacement: in the normal direction, the high-weight constraint makes the displacement projections on both sides of the boundary in the normal direction tend to be consistent (the difference is less than 10%). (pixels) to prevent visible cracks or overlaps perpendicular to the boundary; in the tangential direction, relative slip of several to tens of pixels is allowed on both sides of the boundary to preserve the displacement discontinuity between the moving object and the background.

[0040] For ordinary mesh edges that do not intersect any motion boundary lines, their deformation constraint matrix remains isotropic. ,in The constant is the standard elastic modulus. It is a second-order identity matrix.

[0041] In this embodiment, the specific process by which the dual-degree-of-freedom control point construction module generates dual-degree-of-freedom control points sharing the same initial coordinates but with independent foreground and background offsets at the grid control points traversed by the motion boundary line, without changing the original grid topology adjacency relationship, is as follows: Determine a grid vertex The rule for "grid vertices traversed by the motion boundary line" is: if there exists at least one vertex that is connected to the boundary line of the grid... Connected mesh edges intersect with the motion boundary line, or vertices. If the Euclidean distance from itself to any motion boundary line is less than one pixel, then the vertex is considered to be located on the motion boundary line, and a two-degree-of-freedom control point needs to be constructed for it.

[0042] For each vertex that meets the conditions, a two-degree-of-freedom control point data structure is constructed. The two-degree-of-freedom control point contains at least the following information: a unique node identifier that is consistent with the original mesh vertex; and an initial geometric reference coordinate. Determined by the vertex's original position on the image plane; a foreground control sub-offset. Used to record the displacement of the control point driven by the optical flow field of a moving object; a background control sub-offset. This is used to record the displacement of the control point under the background alignment constraint.

[0043] The foreground and background control sub-offsets are treated as two independent unknowns in the subsequent deformation solution, but they share the same node identifier and the same reference coordinates. This design does not alter the original control mesh. The adjacency matrix and topological connectivity are used to assemble the deformation energy matrix. In this process, the foreground controller and the background controller are treated as two independent degrees of freedom on the same grid vertex, thus avoiding the matrix singularity problem caused by topological reconstruction in the traditional grid splitting method.

[0044] In this embodiment, the deformation solution process based on the anisotropic deformation constraint matrix and the two-degree-of-freedom control points is as follows: Constructing the overall deformation energy function It consists of data items and regularization terms: ; in For the weight of the data item, This represents the weight of the regularization term. Preferably, The range of the ratio is The specific value is determined based on the image noise level and texture richness.

[0045] All unknown offsets are arranged into a column vector This includes the single offset of all ordinary vertices. and the foreground offset of all two-degree-of-freedom control points. and background offset .

[0046] Before defining the data items, the original dense optical flow field is first represented by the detected motion boundary lines. Segmented into foreground optical flow field and background light flow field Specifically, based on the signed distance of each pixel to the motion boundary line (the sign determination method is described in the dual-branch fusion module), pixels are classified into foreground or background regions. If there is significant global camera motion in the scene, a global homography transformation is first estimated using matching point pairs from the background region. The background optical flow field is uniformly transformed to the coordinate system of the reference image to eliminate background parallax or motion interference, thus obtaining the residual background optical flow. .

[0047] Data Items Measure the matching error between the deformed control points and the target image: ; For a normal vertex, the position after deformation is Target location The complete optical flow field at this vertex The mapping yields the following: For a two-degree-of-freedom control point, the deformed position of its foreground controller is... The target position is determined by the foreground optical flow field. The value at that point is mapped; the deformed position of its background controller is... The target position is determined by the background optical flow field. The value at that point is mapped. The background light flow field degenerates to a zero vector only when the scene has a completely static background and the camera is fixed. However, this is a simplified special case of this solution and does not constitute a limitation of the general implementation.

[0048] Confidence weight The confidence level of optical flow estimation within the neighborhood of the control point and the image texture richness are jointly determined: ; in The confidence level is calculated based on the optical flow matching cost, such as normalized cross-correlation. The texture weight is calculated based on the gradient magnitude of the neighborhood image at that point; the richer the texture, the higher the weight.

[0049] If the point is located in a weakly textured region (gradient magnitude less than half the mean of the entire image), then Adaptive reduction.

[0050] Regularization term The sum of the elastic potential energies of all grid edges: ; in For grid edges The deformation displacement difference vector. If both endpoints of the edge are ordinary vertices, then If two-degree-of-freedom control points are involved, and the edges are entirely within the same motion branch, then the offset of that branch is used uniformly. For mesh edges that cross motion boundary lines (with the two endpoints belonging to the foreground and background respectively), the displacement difference vector... Calculated from the offsets of the branches to which the two endpoints belong: If the endpoints If it is located on the foreground side, then use If it is located on the background side, then use Endpoints Similarly.

[0051] Subsequently, along the local normal of the boundary line and tangential Decompose: ; Normal components High weight Constraints are applied to make the normal displacements on both sides of the boundary tend to be uniform; tangential components Subject to low weight Constraints allow for free sliding.

[0052] Setting the partial derivatives of the energy function with respect to all unknown offsets to zero, we form a system of linear equations: ; in The deformation energy matrix is ​​symmetric and positive definite. A column vector consisting of all unknown offsets. This is a load vector driven by data terms. The offsets of each control point can be obtained by using Cholesky decomposition or the conjugate gradient method for stable solution.

[0053] Through the above solution, the deformation error of the moving region is effectively prevented from spreading across the moving boundary to the static region, while the strong normal constraint effectively prevents the problem of exposed black borders in the background caused by sub-pixel misalignment.

[0054] In this embodiment, the specific process by which the dual-branch image fusion module resamples the original image based on foreground and background offsets respectively, and constructs nonlinear weights based on signed distance for fusion output is as follows: For ordinary mesh vertices not crossed by the motion boundary line, their foreground offset and background offset are defined to be equal to the solved single offset. ,Right now For two-degree-of-freedom control points, their independent solutions are taken separately. and .

[0055] For each pixel The foreground mapping coordinates are calculated by interpolating the deformation of the grid cell in which the foreground is located. Mapping coordinates to the background Using the mapped coordinates, bicubic inverse resampling is performed on the original left and right views respectively to obtain the foreground alignment map. Align image with background .

[0056] Calculate the signed distance of each pixel to the nearest moving boundary line. The sign of the optical flow is determined by the motion properties on both sides of the boundary line: in simple scenes, the foreground (positive) and background (negative) sides are determined by comparing the average optical flow amplitude of their local neighborhoods; in complex scenes with multiple motions or camera motion, an unsupervised classification method based on optical flow vector clustering is used. Specifically, the sign of the optical flow vector clustering is taken from both sides of the motion boundary line. Pixel strip area (preferred) The optical flow vectors of all pixels within a given pixel are used as clustering samples to perform K-means clustering. After clustering, the average optical flow amplitude of the two clusters is calculated. The cluster with the larger amplitude is determined to be on the foreground side, and the cluster with the smaller amplitude is determined to be on the background side. Based on this determination, a signed distance is assigned to all pixels within the strip region: pixels belonging to the foreground side have a positive distance (...). ); pixels belonging to the background side have negative distance values ​​( The pixel distance on the boundary line is zero. For pixels outside the strip region, the symbol is propagated outward pixel by pixel: starting from the boundary line, the assignment is expanded outward based on the symbol of the already assigned pixels and their spatial adjacency relationships until all pixels in the entire image have obtained the correct symbol distance. The distance value is always the Euclidean shortest distance from the pixel to the boundary line.

[0057] This scheme uses a sigmoid function based on signed distance as the fusion weights, with continuously adjustable transition band width and steepness. The weight function is defined as follows: ; in To control the steepness of the transition zone, the preferred value range is: ; As a bias parameter, it is usually taken as... The final fused image is calculated using the following formula: ; This fusion method preserves the relative slippage on both sides of the motion boundary while eliminating jagged edges and black border defects at hard edges through a smooth transition.

[0058] In this embodiment, the system further includes a seam post-processing module, used to perform local color gain compensation and exposure consistency correction on the centerline region of the seam based on dual-branch fusion. Color gain compensation is performed only on both sides of the centerline of the seam. The compensation is performed within the pixel range, and the intensity gradually decreases to zero in the direction away from the seam. Preferably, for image, Pixel; for image, Pixels. This post-processing eliminates residual color differences caused by variations in exposure or white balance between the left and right cameras without interfering with the anisotropic deformation mechanism.

[0059] Example 2: As shown in Figure 2, this embodiment uses a real-world scenario of vehicle-mounted panoramic stitching as an example to fully demonstrate the entire implementation process from original image acquisition, optical flow calculation, motion boundary detection, anisotropic mesh constraint construction, two-degree-of-freedom control point generation, deformation solution to final bi-branch fusion. Its purpose is to enable those skilled in the art to clearly understand and reproduce the technical solution of this invention based on the disclosed hardware configuration, parameter values, calculation steps, and intermediate results. This embodiment specifically demonstrates how to derive the structure tensor, motion boundary line, anisotropic constraint matrix, and two-degree-of-freedom control point offset step-by-step from the optical flow neighborhood data in Table 1, ultimately obtaining an artifact-free stitched image.

[0060] In this embodiment, the application scenario and device configuration are as follows: Two forward-facing cameras, one on the left and one on the right, are mounted on the front bumper of the vehicle to capture images of the overlapping area in front of the vehicle. Left camera... and right camera All cameras used are Sony IMX490 RGB global shutter cameras; single camera resolution. Pixels, 30fps frame rate, 6mm lens focal length, 300mm optical center baseline distance, and approximately parallel optical axes. Processing platform: NVIDIA Jetson AGX Orin, 32GB RAM.

[0061] The current shooting scene is a vehicle stationary on a city road. A white sedan is crossing the seam from right to left in the right lane at a speed of about 15 km / h. The background shows street buildings and roadside trees.

[0062] First, obtain the left view synchronously. and right view An 8-bit grayscale image. The RAFT dense optical flow algorithm (20 iterations, 6 pyramid levels) was used to calculate the optical flow from... arrive pixel-level displacement vector field The optical flow field is smoothed using a Gaussian filter, and the standard deviation of the filter kernel is... Pixels, to obtain a smooth optical flow field .

[0063] Select pixels near the motion boundary of the seam area. Its smoothed optical flow The neighborhood values ​​are shown in Table 1.

[0064] Table 1. Smooth optical flow components in the neighborhood of pixel (1048,532) (unit: pixels)

[0065] The horizontal displacement abruptly changes from approximately -11.5 pixels (moving vehicle) to approximately -2.3 pixels (stationary background), exhibiting a significant discontinuity.

[0066] Then, the spatial gradient is calculated using central difference on the smoothed optical flow field. Taking pixel (1048, 532) as an example: ; ; ; ; Obtain the gradient vector: ; Constructing the optical flow gradient structure tensor : ; Substitute the values: ; ; ; therefore, ; Calculate matrix The eigenvalues ​​are obtained to find the largest eigenvalue. .

[0067] This value is much higher than the average response of the entire map. Standard deviation This indicates that the pixel is at a strong displacement discontinuity boundary.

[0068] Next, the motion boundary lines are extracted, as follows: The distribution of the largest eigenvalue is regarded as a gray-level response map. The image plane is divided into Each block is a local threshold calculated independently using Otsu's method. For a block containing pixels (1048, 532), .

[0069] Global adaptive threshold ,Pick ,have to .

[0070] because and The pixel is then marked as a candidate edge point.

[0071] After nonmaximum suppression and Zhang-Suen thinning, a single-pixel wide motion boundary line is obtained. One of the boundary lines... The length is 456 pixels, and its partially ordered pixel sequence is as follows: .

[0072] Next, a control mesh is constructed and anisotropic deformation constraints are applied, as follows: Construct a uniform quadrilateral control grid in the overlapping region (480-pixel wide column), with vertex spacing... Pixels. The grid has 21 horizontal vertices and 46 vertical vertices, for a total of 966 vertices.

[0073] Take grid edges endpoints , This side is a horizontal line segment. Boundary line The equation of the line is .Will Substitution Therefore, the line segment intersects the boundary line precisely at... .

[0074] exist Take 3 pixels before and after the intersection point: These points all lie on a straight line. Principal component analysis was performed on it, and the first principal component (tangent direction) is: ; Will Rotate 90° clockwise to obtain the local normal direction: ; For the edge Assign anisotropic deformation constraint matrix: ; Take normal constraint weights Tangential constraint weights .

[0075] The calculation yields: ; ; For ordinary mesh edges that do not intersect with the motion boundary line, the constraint matrix maintains an isotropic form. , .

[0076] Next, two-degree-of-freedom control points are constructed and foreground and background optical flow are segmented, as follows: Because of the vertex There are connected edges Intersecting with the motion boundary line, it is determined that it is a mesh vertex traversed by the motion boundary line. Two-degree-of-freedom control points are created for it: reference coordinates. and assign foreground control sub-offset and background control sub-offset .

[0077] Using boundary lines The original dense optical flow field is segmented. Optical flow vectors within 15-pixel-wide strip regions on both sides of the boundary are then subjected to K-means clustering. Cluster 1 has an average optical flow amplitude of 11.5 pixels and is identified as the foreground; Cluster 2 has an average amplitude of 2.2 pixels and is identified as the background.

[0078] In this embodiment, the camera is stationary, and there is no global motion in the background region. Therefore, the background optical flow field is the zero or small residual against a static background. The global homography matrix is ​​estimated using the matching points in the background region. Near-unit array, residual background optical flow Root mean square 0.18 pixels. Foreground optical flow field. It consists of the original optical flow of the segmented foreground region.

[0079] Since the background in this embodiment is completely static, the light flow of the foreground and background is naturally in the same reference frame, and there is no reference frame offset.

[0080] For scenes with global camera motion, the global motion components should be subtracted from both the foreground and background areas simultaneously to unify their reference values.

[0081] Then, the deformation energy function is constructed as follows: For example, in this embodiment, the overall deformation energy function is: ; Pick , .

[0082] Confidence It is determined by the optical flow matching NCC cost and the image gradient magnitude.

[0083] Combining all vertex offsets into a vector This includes 838 single offsets for ordinary vertices and 256 foreground and background offsets for two-DOF control points, totaling 1094 unknowns. For edges crossing boundaries, the displacement difference is calculated based on the branch to which the endpoints belong. For example, suppose the solution yields: ; side The displacement difference in the normal direction is strongly constrained, while free slip is allowed in the tangential direction, thus maintaining alignment of the boundary normals.

[0084] linear systems The offsets of all control points are obtained by using the Cholesky decomposition to achieve stability.

[0085] Finally, dual-branch resampling and fusion are performed, as follows: Foreground mapping for each pixel is obtained using bilinear interpolation of mesh vertex offsets. and background mapping and resampled to obtain and .

[0086] Calculate the signed distance from the pixel to the boundary line. Foreground weights are positive, and background weights are negative. The fusion weights use the Sigmoid function. ; Pick , The final merged image is: ; In this embodiment, the system throughput of GPU parallel pipeline processing can reach 30fps, with a single frame processing latency of approximately 82ms, meeting the near real-time stitching requirements for automotive applications. The fused panoramic image shows no visible cracks, ghosting, or black edges at the boundaries of the moving car. Furthermore, under static background, single moving vehicle, and normal lighting and texture conditions, this embodiment effectively eliminates stitching artifacts at the edges of moving objects in the stitching seam area.

[0087] Example 3: This embodiment aims to verify the actual technical effect of the present invention through multiple sets of comparative tests and ablation experiments, and to clarify the causal relationship between this effect and various key technical features. By setting different comparison schemes (removing anisotropic constraints, removing two-degree-of-freedom control points, traditional uniform mesh stitching, and the optimal stitching method), quantitative evaluation was performed on real-world scenario data under multiple working conditions. It can be clearly seen that the present invention significantly outperforms existing methods in four indicators: edge offset, local signal-to-noise ratio, artifact area, and temporal consistency. At the same time, ablation experiments confirm that anisotropic deformation constraints and two-degree-of-freedom control points are indispensable technical features for achieving the above-mentioned performance improvements.

[0088] Furthermore, this embodiment also performs sensitivity analysis on key parameters and provides an optimal parameter range to provide a basis for actual deployment.

[0089] Test settings The test dataset contains 12 sets of continuous image sequences of typical in-vehicle scenes, 100 frames per set, for a total of 1200 frames, with all images having a resolution of [missing information]. The overlap area is approximately 480 columns wide. Scenes included: static background + single moving vehicle (6 groups), camera translational motion + moving pedestrian (4 groups), and low-texture road surface + moving vehicle (2 groups). All comparison methods used the same optical flow input (RAFT) and the same control grid (24-pixel spacing). Evaluation metrics included: Average absolute offset of the edge (pixels): The corresponding points of the actual edge of the moving object are manually marked, and the average Euclidean distance between the edge position and the reference position after stitching is calculated.

[0090] Local PSNR (dB): Only the 30-pixel area on both sides of the seam line is calculated.

[0091] Artifact area ratio (%): The percentage of pixels with visible artifacts such as manually marked shadows and black borders in the total number of pixels in the overlapping area.

[0092] Temporal consistency: Standard deviation of pixel grayscale fluctuation in the stitching seam area of ​​consecutive frames (grayscale level 0–255).

[0093] First, the present invention solution (M1): Using the complete solution described in Example 1: Gaussian smoothing Block-based adaptive threshold coefficient Anisotropy constraint Two-DOF control points, Sigmoid fusion based on signed distance .

[0094] Then, compare the solutions: M2 (non-anisotropic): Change mesh edge constraints to isotropic. The rest is the same as M1.

[0095] M3 (No Two Degrees of Freedom): Boundary vertices do not create two-degree-of-freedom control points; only a single offset is used, and the constraints are the same as M1.

[0096] M4 (Traditional Uniform Mesh Stitching): Uses single-light flow field alignment, with no motion boundary distinction, and isotropic uniform mesh deformation.

[0097] M5 (Dynamic Optimal Suture Line): Finds the optimal suture line based on Sobel gradient magnitude, without mesh deformation, and fuses multiple frequency bands along the suture line.

[0098] Finally, in this embodiment, the average value and standard deviation of each indicator of the entire test set are shown in Table 2.

[0099] Table 2 Comparison of splicing quality among different schemes (mean ± standard deviation)

[0100] It should be noted that in a scene with a static background and moving vehicles, the edge offset of the M1 is as low as 0.18 pixels; in a scene with the camera itself panning, the M1 maintains an accuracy of 0.43 pixels, while the M4 increases to 3.4 pixels; in a scene with weak textured road surface, the local PSNR of the M1 is 35.1dB, which is significantly better than the 26.4dB of the M2.

[0101] For ablation analysis: After removing the anisotropic constraints, the normals could not be forcibly aligned, the edge offset increased by approximately 4.9 times, and the artifact area ratio increased by approximately 8.8 times, confirming the crucial role of anisotropic constraints in suppressing boundary misalignment and black edges. Removing the two-degree-of-freedom control points significantly increased both edge offset and artifacts, indicating that the two-degree-of-freedom control points are a necessary feature for correctly handling moving boundaries.

[0102] Regarding parameter sensitivity: Tests on a static background with a subset of moving vehicles show that Gaussian smoothing... Performance is stable around 1.0; constraint weight ratio The back edge offset converges to around 0.32 pixels; Sigmoid steepness. The artifact area ratio is less than 0.22% in the range of 0.5 to 3.0.

[0103] The above results verify the effectiveness of the technical solution of the present invention in various typical vehicle splicing scenarios, and the anisotropic deformation constraint and the two-degree-of-freedom control points are the key technical features for achieving this effect.

[0104] Example 4: This embodiment aims to illustrate the coping strategies of the present invention under non-ideal or abnormal operating conditions, demonstrating the engineering robustness and practical deployability of the technical solution. In practical applications, optical flow estimation may encounter situations such as insufficient local confidence, broken motion boundary lines due to noise, or difficulty in distinguishing foreground and background optical flow. This embodiment clarifies the specific backoff and compensation mechanisms adopted by the system under these boundary conditions, such as weight zeroing, morphological breakpoint connection, and mode degradation processing, thereby ensuring that the stitching system will not produce serious errors or crashes due to local anomalies. This helps those skilled in the art understand that the present invention is not only applicable to ideal scenarios but also has the ability to operate stably in real-world complex environments.

[0105] Specifically, when the optical flow matching confidence level is lower than a set threshold, the weight of the corresponding control point data item is adjusted. When set to zero, the deformation in this region is entirely driven by the smoothing term.

[0106] If the extracted motion boundary line has a gap of less than 5 pixels, optionally, a morphological dilation-thinning operation can be automatically performed to fill it.

[0107] When the displacement difference between the moving object and the background is so small that clustering cannot distinguish them, the system can optionally degenerate into a mode that does not use two-degree-of-freedom control points, construct only a single offset and retain anisotropic constraints to avoid deformation conflicts caused by incorrect segmentation.

[0108] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0109] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0110] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only one method, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0111] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0112] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0113] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0114] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A local mesh adaptive deformation system for stitched cameras based on dense optical flow field estimation, characterized in that, include: The optical flow field calculation module is used to obtain the dense optical flow field of the left and right views of the overlapping area. After Gaussian smoothing the dense optical flow field, the spatial gradient of each pixel is calculated and the optical flow gradient structure tensor is constructed. The maximum eigenvalue of the structure tensor is solved to obtain the maximum eigenvalue distribution. The motion boundary detection module is used to take the maximum feature value distribution as a grayscale response map, divide the image plane into multiple non-overlapping local blocks, independently calculate the binarization threshold in each local block and combine it with the global threshold for binarization, and perform non-maximum suppression and thinning on the binarization result to obtain a single-pixel wide motion boundary line. The anisotropic deformation constraint module is used to construct a control mesh on the image plane and determine whether each mesh edge intersects with the motion boundary line. For intersecting mesh edges, the local tangent and normal directions of the motion boundary line at the intersection point are obtained, and an anisotropic deformation constraint matrix is ​​constructed for the mesh edge. The constraint response of this matrix in the normal direction is greater than the constraint response in the tangent direction. For non-intersecting mesh edges, isotropic constraints are maintained. The two-degree-of-freedom control point construction module is used to construct two-degree-of-freedom control points at the grid vertices traversed by the motion boundary line. These control points share the same initial coordinates but have independent foreground and background offsets. The foreground and background offsets participate in the deformation solution as two independent degrees of freedom without changing the topological adjacency relationship of the original control grid. The dual-branch image fusion module is used to resample the original image based on the solved foreground offset and background offset to obtain a foreground alignment map and a background alignment map. Based on the signed distance of each pixel to the nearest moving boundary line, the module uses a sigmoid function of signed distance to calculate nonlinear fusion weights and performs weighted fusion on the two alignment maps to obtain the final stitched image.

2. The local mesh adaptive deformation system for stitched cameras based on dense optical flow field estimation according to claim 1, characterized in that, In the motion boundary detection module, the global threshold is determined by weighted summation of the mean and standard deviation of the entire grayscale response image, with the weighting coefficient ranging from 2.5 to 3.5; the number of local blocks is adaptively set according to the image resolution.

3. The local mesh adaptive deformation system for stitched cameras based on dense optical flow field estimation according to claim 1, characterized in that, In the anisotropic deformation constraint module, the tangent and normal directions of the moving boundary lines at the intersection points are obtained by principal component analysis of the boundary line pixel sequence; the anisotropic deformation constraint matrix is ​​jointly determined by the normal constraint weight and the tangential constraint weight, wherein the ratio of the normal constraint weight to the tangential constraint weight is not less than 100.

4. The local mesh adaptive deformation system for stitched cameras based on dense optical flow field estimation according to claim 1, characterized in that, In the two-degree-of-freedom control point construction module, the condition for determining the grid vertex through which the motion boundary line passes is: at least one of the grid edges connected to the vertex is determined to intersect the motion boundary line.

5. The local mesh adaptive deformation system for stitched cameras based on dense optical flow field estimation according to claim 1, characterized in that, The local mesh adaptive deformation system of the stitched camera also includes a deformation solving module, which is used to construct the energy function and solve for the offset of all control points; The deformation solving module uses the motion boundary line to divide the dense optical flow field into a foreground optical flow field and a background optical flow field. The division is based on the signed distance of each pixel to the motion boundary line. The energy function includes a data term and a regularization term; In the data terms, for a two-degree-of-freedom control point, the target position of its foreground controller is determined by the value of the foreground optical flow field at the corresponding coordinates, and the target position of its background controller is determined by the value of the background optical flow field at the corresponding coordinates. In the regularization terms, for a mesh edge that crosses the motion boundary line, the displacement difference between its two control points is decomposed into normal and tangential components along the boundary line normal and tangential directions. The normal component is constrained by a higher normal constraint weight, and the tangential component is constrained by a lower tangential constraint weight. By minimizing this energy function, we can solve for the single offset of all ordinary vertices and the foreground and background offsets of all two-degree-of-freedom control points.

6. The local mesh adaptive deformation system for stitched cameras based on dense optical flow field estimation according to claim 5, characterized in that, The data item also includes a local confidence weight for each control point, which is determined by the product of the optical flow estimation confidence and the image texture richness. The optical flow estimation confidence is calculated based on the normalized cross-correlation cost, and the image texture richness is calculated based on the gradient magnitude of the neighborhood of the control point. When the gradient magnitude of the neighborhood of the control point is less than half of the average gradient magnitude of the entire image, the local confidence weight of the control point is reduced.

7. The local mesh adaptive deformation system for stitched cameras based on dense optical flow field estimation according to claim 5, characterized in that, The energy function is formed by multiplying the data term and the regularization term by their respective weights and summing the results. The ratio of the data term weight to the regularization term weight ranges from 0.1 to 10.

0.

8. The local mesh adaptive deformation system for stitched cameras based on dense optical flow field estimation according to claim 1, characterized in that, In the dual-branch image fusion module: For ordinary mesh vertices, both their foreground and background offsets are taken as the single offset of that vertex; for two-degree-of-freedom control points, their independently solved foreground and background offsets are taken respectively. The total foreground offset and total background offset of each pixel are obtained by bilinear interpolation, and then the foreground mapping coordinates and background mapping coordinates are obtained. The foreground alignment map and background alignment map are obtained by bicubic inverse resampling. In the signed distance, the sign is determined as follows: the optical flow vectors of pixels within a preset width range on both sides of the motion boundary line are subjected to binary clustering. The side corresponding to the cluster with the larger average optical flow amplitude is determined as the foreground side and assigned a positive sign; the other side is determined as the background side and assigned a negative sign; the distance value is the Euclidean distance from the pixel to the nearest motion boundary line.

9. The local mesh adaptive deformation system for stitched cameras based on dense optical flow field estimation according to claim 8, characterized in that, The Sigmoid function has a steepness parameter that controls the steepness of the transition band and an offset parameter that adjusts the center of the transition band, wherein the steepness parameter ranges from 0.5 to 3.0.

Images