Image stitching method, device and product based on feature enhancement and coordinate perception

Abstract

This invention provides an image stitching method, device, and product based on feature enhancement and coordinate awareness, relating to the field of computer vision technology. The invention includes: multi-scale feature extraction and hierarchical fusion of a reference image and a target image; recalibrating channel weights obtained through global averaging and maximum statistical interactive coding; constructing a global correlation tensor from target enhancement features using a matching kernel, and obtaining a two-dimensional displacement representation through orientation-aware recalibration; regressing corner offsets based on the first two-dimensional displacement representation of the lower spatial resolution feature layer to generate a projection transformation matrix for coarse registration of target features; generating a second two-dimensional displacement representation based on the higher spatial resolution feature layer, regressing the local offsets of regular control points, and solving a nonlinear smooth transformation function; and mapping the target image onto a common canvas and fusing it with the reference image to obtain a stitched image. This invention can effectively improve matching representation and local alignment effects in parallax scenarios.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to an image stitching method, device, and product based on feature enhancement and coordinate awareness. Background Technology

[0002] Image stitching technology aims to map multiple images with overlapping areas to a common coordinate system and fuse the effective regions to obtain an image with a wider field of view. Existing image stitching methods mainly include geometric methods based on artificial features and learning methods based on neural networks. Among them, traditional geometric methods typically complete image registration through steps such as feature extraction, feature matching, outlier removal, and transformation estimation; while deep learning methods utilize neural networks to extract image features and estimate the corresponding geometric transformation parameters end-to-end.

[0003] In geometric transformation estimation, global homography is suitable for near-planar scenes or situations where the camera rotates around its optical center. However, in actual shooting, when there are significant depth-of-field changes, large parallax, or local motion in the scene, a single global transformation is insufficient to accurately align all areas, easily leading to ghosting or misalignment in areas with large parallax. To address this issue, nonlinear transformations such as local mesh deformation and thin-plate splines have been introduced to provide higher local degrees of freedom. However, the effectiveness of these local nonlinear transformations is highly dependent on the accuracy of the front-end feature matching results, and without appropriate constraints, they can easily lead to excessive deformation or distortion in non-overlapping areas.

[0004] Regarding feature representation and matching mechanisms in depth image stitching methods, while multi-scale features can take into account both semantic information and localization details of an image, the importance of each feature channel often differs after direct fusion of features from different scales. Existing channel enhancement methods typically only use single global statistical information, making it difficult to simultaneously reflect the overall response distribution and local salient responses of feature channels, resulting in insufficient feature representation after multi-scale fusion. Furthermore, while existing general correlation calculations can describe the similarity between image features, they do not adequately utilize the directional relationship between the target feature location and the spatial coordinates of the reference image. When faced with repetitive textures, low-texture scenes, or scenes with significant parallax, this can easily lead to scattered or ambiguous matching responses, thus affecting the estimation accuracy of subsequent transformation parameters.

[0005] In terms of transformation parameter estimation strategies, existing methods often struggle to effectively coordinate global motion and local deformation. If local nonlinear transformations are directly estimated, the network needs to handle both large-scale global motion and fine-grained local deformations simultaneously, greatly increasing the difficulty of parameter estimation. Conversely, if only global transformations are estimated, it is difficult to eliminate residual misalignments caused by parallax in non-planar regions.

[0006] Therefore, there is an urgent need in this field for an image stitching scheme that can improve the multi-scale feature representation and related matching quality, and coordinate global coarse registration with local refinement, so as to overcome the problems of poor alignment, easy deformation and ghosting in complex parallax scenarios of existing technologies. Summary of the Invention

[0007] This invention provides an image stitching method, device, and product based on feature enhancement and coordinate awareness.

[0008] According to one aspect of the present invention, an image stitching method based on feature enhancement and coordinate awareness is provided, comprising: acquiring at least one reference image and at least one target image; performing multi-scale feature extraction with shared weights on the reference image and the target image respectively to obtain corresponding multi-scale features; upsampling the features with lower spatial resolution in the multi-scale features and stitching them with adjacent features with higher spatial resolution along the channel dimension to form layered fusion features corresponding to each depth layer; acquiring the statistical descriptions of the global average statistics and the global maximum statistics of each layered fusion feature respectively, and performing cross-coding on the statistical descriptions of the global average statistics and the global maximum statistics to obtain channel weights, and recalibrating the corresponding layered fusion features with the channel weights to obtain enhancement features corresponding to each depth layer; normalizing the enhancement features corresponding to each depth layer in the reference image and the target image respectively to obtain reference enhancement features and target enhancement features corresponding to each depth layer; generating corresponding two-dimensional displacement representations based on the reference enhancement features and the target enhancement features as first two-dimensional displacement representations; and regressing based on the first two-dimensional displacement representations corresponding to the depth layers with lower spatial resolution to obtain... The process involves: offsetting corner points of the target image; generating a projection transformation matrix to describe the projection transformation relationship between the reference image and the target image based on the corner point offset; spatially mapping the target enhancement features corresponding to the depth layer with high spatial resolution using the projection transformation matrix to obtain coarse registration target features; generating a corresponding two-dimensional displacement representation as a second two-dimensional displacement representation based on the reference enhancement features corresponding to the depth layer with high spatial resolution and the coarse registration target features; regressing the local offsets of each regular control point in the target image space based on the second two-dimensional displacement representation; spatially mapping the regular control points using the projection transformation matrix to obtain initial deformation control points; adding the initial deformation control points and the local offsets point by point to obtain final control points; solving a nonlinear smoothing transformation function based on the regular control points and the final control points; determining a common canvas based on the spatial range of the reference image and the spatial range of the target image after nonlinear smoothing transformation using the nonlinear smoothing transformation function; mapping the target image to the common canvas using the nonlinear smoothing transformation function and then fusing it with the reference image to obtain a stitched image.

[0009] According to at least one embodiment of the present invention, an image stitching method based on feature enhancement and coordinate awareness generates a corresponding two-dimensional displacement representation based on the reference enhancement feature and the target enhancement feature, as a first two-dimensional displacement representation, comprising: extracting local neighborhoods from each spatial position of the target enhancement feature and rearranging them into multiple matching kernels, such that each matching kernel acts on the spatial range of the reference enhancement feature to obtain a global correlation tensor; aggregating the global correlation tensor along the horizontal and vertical directions respectively to generate horizontal coordinate weights and vertical coordinate weights; performing orientation-aware recalibration on the global correlation tensor based on the horizontal coordinate weights and vertical coordinate weights, and mapping the recalibration result to a two-dimensional displacement representation to obtain the first two-dimensional displacement representation.

[0010] According to at least one embodiment of the present invention, an image stitching method based on feature enhancement and coordinate awareness aggregates the global correlation tensor along the horizontal and vertical directions respectively to generate horizontal coordinate weights and vertical coordinate weights, comprising: aggregating the global correlation tensor along the width and height directions respectively to obtain a horizontal one-dimensional direction description and a vertical one-dimensional direction description; stitching the horizontal one-dimensional direction description and the vertical one-dimensional direction description together to obtain a stitching direction description; performing channel compression processing on the stitching direction description to obtain a compressed direction description; and splitting the compressed direction description to generate the horizontal coordinate weights and the vertical coordinate weights respectively.

[0011] According to at least one embodiment of the present invention, an image stitching method based on feature enhancement and coordinate awareness performs orientation-aware recalibration on the global correlation tensor based on the horizontal coordinate weights and the vertical coordinate weights, and maps the recalibration result to a two-dimensional displacement representation to obtain a first two-dimensional displacement representation. The method includes: multiplying the horizontal coordinate weights and the vertical coordinate weights element-wise with the global correlation tensor to obtain a coordinate-enhanced correlation tensor; and performing displacement mapping on the coordinate-enhanced correlation tensor to obtain a two-dimensional displacement representation to obtain the first two-dimensional displacement representation.

[0012] According to at least one embodiment of the present invention, an image stitching method based on feature enhancement and coordinate awareness is used to interactively encode the statistical descriptions of the global average statistics and the global maximum statistics to obtain channel weights. The method includes: stacking the statistical descriptions of the global average statistics and the global maximum statistics to construct statistical markers; performing interactive processing on the statistical markers to obtain interactive results; sequentially flattening and channel mapping the interactive results, and generating the channel weights through a gating function.

[0013] According to at least one embodiment of the present invention, an image stitching method based on feature enhancement and coordinate awareness upsamples features with lower spatial resolution in the multi-scale features and stitches them along the channel dimension with adjacent features with higher spatial resolution to form layered fusion features corresponding to each depth layer. The method includes: upsampling deep features in the multi-scale features to adjust them to the same spatial scale as mid-level features in the multi-scale features, and stitching the two together to form a mid-level fusion feature; upsampling the mid-level fusion feature to adjust it to the same spatial scale as shallow features in the multi-scale features, and stitching the two together to form a shallow-level fusion feature; and using the mid-level fusion feature and the shallow-level fusion feature as layered fusion features corresponding to each depth layer.

[0014] According to at least one embodiment of the present invention, an image stitching method based on feature enhancement and coordinate awareness recalibrates the corresponding layered fusion features with the channel weights to obtain enhanced features corresponding to each depth layer, including: multiplying the channel weights and the corresponding layered fusion features channel by channel to obtain a product result; adding the product result to the corresponding layered fusion features, stitching the results together, or normalizing the product result to obtain the enhanced features.

[0015] According to at least one embodiment of the present invention, the image stitching method based on feature enhancement and coordinate awareness, after obtaining the global correlation tensor, further includes: performing numerical adjustment processing on the global correlation tensor; wherein, the numerical adjustment processing includes temperature scaling processing, Softmax probability normalization processing, sparse normalization processing, or confidence truncation processing.

[0016] According to another aspect of the present invention, an electronic device is provided, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, such that the processor performs an image stitching method based on feature enhancement and coordinate awareness according to any embodiment of the present invention.

[0017] According to another aspect of the present invention, a readable storage medium is provided, wherein execution instructions are stored therein, which, when executed by a processor, are used to implement the image stitching method based on feature enhancement and coordinate awareness according to any embodiment of the present invention.

[0018] According to another aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements an image stitching method based on feature enhancement and coordinate awareness according to any embodiment of the present invention.

[0019] The present invention has the following beneficial effects: (1) Enhancing the matching and representation capabilities of multi-scale features. This invention proposes a dual statistical channel enhancement mechanism, which preserves the overall response distribution and significant local responses of feature channels through global average statistics and global maximum statistics, respectively. The two complementary statistical descriptions are interactively encoded before generating channel weights, which can adaptively recalibrate multi-scale fusion features. This effectively reduces the impact of unbalanced channel contributions after direct splicing of features of different scales on matching, enabling the network to more accurately focus on key features useful for the splicing task.

[0020] (2) Enhancing matching robustness and directional consistency in complex scenarios. This invention employs a coordinate-aware correlation matching strategy, using local blocks at each spatial location of the target feature as the matching kernel, and applying them to the complete spatial range of the reference feature to construct a global correlation tensor, thereby simultaneously representing local neighborhood content and global search relationships. Furthermore, the correlation tensor undergoes lateral and longitudinal coordinate orientation-aware recalibration, effectively enhancing the spatial consistency of directional correlation. When faced with repetitive textures, low textures, or significant parallax scenes, it can significantly suppress scattered or ambiguous matching responses and improve the accuracy of geometric parameter estimation.

[0021] (3) Perfectly balances global structural stability and local parallax alignment. This invention designs a global-local hierarchical geometric transformation mechanism. At the first scale (lower spatial resolution), the global projection relationship is estimated for coarse registration, and at the second scale (higher spatial resolution), the local residuals of the regular control points are estimated for refinement. This hierarchical strategy from coarse to fine means that local regression does not have to independently bear all the large-scale motion, effectively reducing the difficulty of parameter estimation and overcoming the shortcomings of a single global transformation in eliminating residual misalignment in non-planar regions and the tendency of direct estimation of local transformation to lead to excessive deformation in non-overlapping regions.

[0022] (4) Reduce computational redundancy and improve engineering deployment efficiency. In terms of network architecture, this invention uses a feature extractor with shared weights to process the reference image and the target image separately, and combines a hierarchical fusion structure with optional lightweight convolution operations. This design avoids the parameter redundancy caused by building independent networks for the two images, and provides an efficient engineering implementation path to reduce repeated calculations and the number of model parameters, effectively improving the inference efficiency and computational resource utilization of the image stitching algorithm in actual deployment environments. Attached Figure Description

[0023] The accompanying drawings illustrate exemplary embodiments of the invention and, together with the description thereof, serve to explain the principles of the invention. These drawings are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification.

[0024] Figure 1This is a flowchart illustrating an image stitching method based on feature enhancement and coordinate awareness according to an embodiment of the present invention.

[0025] Figure 2 This is a flowchart illustrating the method corresponding to step S6 according to an embodiment of the present invention.

[0026] Figure 3 This is a flowchart illustrating the method corresponding to step S610 according to an embodiment of the present invention.

[0027] Figure 4 This is a flowchart illustrating the method corresponding to step S620 according to an embodiment of the present invention.

[0028] Figure 5 This is a flowchart illustrating the method corresponding to step S630 according to an embodiment of the present invention.

[0029] Figure 6 This is a schematic diagram illustrating the execution process of an image stitching method based on feature enhancement and coordinate awareness according to an embodiment of the present invention.

[0030] Figure 7 This is a schematic diagram of an enhanced feature generation process according to an embodiment of the present invention.

[0031] Figure 8 This is a schematic diagram of a two-dimensional displacement characterization generation process according to an embodiment of the present invention.

[0032] Figure 9 This is a schematic diagram of a two-stage execution process according to an embodiment of the present invention.

[0033] Figure 10 This is a schematic diagram of a canvas construction and image fusion process according to an embodiment of the present invention.

[0034] Figure 11 This is a schematic structural block diagram of an electronic device employing a processor-based hardware implementation according to an embodiment of the present invention. Detailed Implementation

[0035] The present invention will now be described in further detail with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present invention are shown in the accompanying drawings.

[0036] It should be noted that, unless otherwise specified, the embodiments and features described in this invention can be combined with each other. The technical solution of this invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0037] Figure 1 A schematic diagram illustrating the overall flow of an image stitching method based on feature enhancement and coordinate awareness according to one embodiment of the present invention is shown. Figure 1 The method shown includes steps S1 to S9. This method can be executed by electronic devices such as mobile phones and tablets.

[0038] S1: Acquire at least one reference image and at least one target image.

[0039] The reference image is a base image placed in a common coordinate system, and the target image is an image mapped to the common coordinate system according to the estimated transformation. Their roles can be interchanged, and the invention is not limited to processing only two images. For multiple images, the process can be repeated based on adjacency, selected reference frames, or graph structures.

[0040] As one possible implementation, the reference image and the target image have at least partially overlapping regions, and both have been resized, numerically normalized, or color space converted according to the requirements of subsequent network input.

[0041] For example, the reference image and the target image can be resized to 512×512 pixels.

[0042] S2: Perform multi-scale feature extraction with shared weights on the reference image and the target image respectively to obtain the corresponding multi-scale features.

[0043] As one possible implementation, multi-scale feature extraction can employ a feature extraction network with shared parameters to process the reference image and the target image separately. By drawing feature branches from different depth layers of the feature extraction network, shallow features, mid-level features, and deep features are obtained respectively. It should be noted that the specific type of feature extraction network (e.g., VGG network, ResNet network, or other convolutional neural networks with hierarchical structures) can be adjusted according to the actual application requirements and is not specifically limited here.

[0044] S3: Upsample the features with lower spatial resolution in the multi-scale features and then stitch them together with the adjacent features with higher spatial resolution along the channel dimension to form the hierarchical fusion features corresponding to each depth layer.

[0045] As one possible implementation, the multi-scale features include three features at different depths: shallow features, mid-level features, and deep features. Among them, shallow features have the highest spatial resolution, and deep features have the lowest spatial resolution.

[0046] Based on the above implementation method, the specific implementation process of step S3 can be as follows: Upsample the deep features to the same spatial scale as the mid-layer features, and then stitch the upsampled deep features and mid-layer features along the channel dimension to form a mid-layer fusion feature. Upsample the mid-layer fusion feature to the same spatial scale as the shallow features, and then stitch the upsampled mid-layer fusion feature and shallow features along the channel dimension to form a shallow-layer fusion feature. Use the mid-layer fusion feature and the shallow-layer fusion feature as the layered fusion features corresponding to each depth layer (the two depth layers: mid-layer and shallow).

[0047] S4: Obtain the statistical descriptions of the global average statistics and the global maximum statistics of each layered fusion feature, and perform interactive encoding of the statistical descriptions of the global average statistics and the global maximum statistics to obtain the channel weights. Use the channel weights to recalibrate the corresponding layered fusion features to obtain the enhanced features corresponding to each depth layer.

[0048] Let F∈R be the hierarchical fusion feature obtained through step S3. (B×C×H×W) Where B represents the batch size, C represents the number of channels, H and W represent the height and width of the feature map, respectively, and R represents the set of real numbers. Global average statistics and global maximum statistics are performed on the hierarchical fusion feature F along the spatial dimensions (H and W dimensions) to obtain the statistical description of the global average statistics (average description z_avg∈R). (B×C) ) and the statistical description of the global maximum statistic (maximum description z_max∈R) (B×C) Global average statistics can reflect the overall response distribution of each channel across the entire spatial range. Global maximum statistics can retain the most significant local responses of each channel. The two serve as complementary statistical markers rather than being directly added at the input, thus providing a more comprehensive characterization of channel features and avoiding the problem of significant local responses being diluted by the overall average response.

[0049] One possible implementation involves interactively encoding global average statistics and global maximum statistics to obtain channel weights. This includes: stacking the average and maximum descriptions to construct a unified statistical label; inputting the statistical label into an interactive encoding unit for interactive processing to obtain the interactive result; sequentially performing a flattening operation and a channel mapping operation on the interactive result; and generating the final channel weights using a gating function (such as sigmoid).

[0050] The role of the interactive coding unit is to enable the two types of statistical descriptions (average description and maximum description) to influence each other and deeply integrate. For example, the interactive coding unit can be a multi-head self-attention encoder, a cross-attention encoder, a gated recurrent unit, a bilinear interactive unit, or any other network structure that can enable the two types of statistical descriptions to interact effectively.

[0051] For example, the channel mapping operation may include one or more layers, specifically a fully connected layer or a 1×1 convolutional layer. The gating function can be a sigmoid function, used to constrain the mapping result within the (0,1) interval, and the final generated channel weights w∈R (B×C×1×1) .

[0052] As one possible implementation, the corresponding layered fusion features are recalibrated using channel weights to obtain the enhanced features corresponding to each depth layer, including: multiplying the channel weights of the layered fusion features channel by channel to obtain the enhanced features.

[0053] It should be noted that this step does not necessarily require adding additional external residuals to the enhancement result. In the current preferred embodiment, the output is directly the product of the hierarchical fusion feature and the channel weights. In other alternative embodiments, the above product result can also be added to the hierarchical fusion feature, concatenated, or normalized before being used as the final enhancement feature output; this invention does not limit this to a single method.

[0054] S5: Normalize the enhancement features corresponding to each depth layer in the reference image and the target image respectively to obtain the reference enhancement features and target enhancement features corresponding to each depth layer.

[0055] S6: Generate a corresponding two-dimensional displacement representation based on the reference enhancement features and the target enhancement features, which serves as the first two-dimensional displacement representation.

[0056] Regarding step S6, it may include, for example: Figure 2 Steps S610 to S630 are shown.

[0057] S610: Extract local neighborhoods from each spatial location of the target augmented feature and rearrange them into a matching kernel group, so that each matching kernel in the matching kernel group acts on the spatial range of the reference augmented feature, thus obtaining the global correlation tensor.

[0058] Regarding step S610, it may include, for example: Figure 3 Steps S611 to S613 are shown.

[0059] S611: Perform channel normalization on the reference enhancement features and the target enhancement features.

[0060] S612: Extract local blocks of a set size from each spatial location of the target enhancement feature, and rearrange all local blocks into multiple matching kernels.

[0061] S613: Perform correlation calculations on each matching kernel within the spatial range of the reference enhancement features to obtain the global correlation tensor.

[0062] In one example, let the normalized reference augmentation feature and the target augmentation feature be Fr∈R. (B×C×H×W) Ft∈R (B×C×H×W) From each spatial location (u,v) of the target augmentation feature, extract a K×K×C local block P_(u,v), and rearrange all local blocks into H×W matching kernels. Perform correlation calculations on each matching kernel with the complete spatial range of Fr to obtain the global correlation tensor V∈R. (B×HW×H×W) Where K represents the scale of the local block. For example, K=3.

[0063] Unlike methods that compute relevance only at corresponding locations or within a limited search window, this embodiment preserves each target location index of the global relevance tensor in the channel dimension and corresponds to a reference spatial response map. Therefore, the object of subsequent coordinate attention processing is not ordinary image features, but a matching tensor with "target location—reference coordinates" semantics, capable of simultaneously representing local neighborhood content and global search relationships.

[0064] S620: Aggregate the globally relevant tensors along the horizontal and vertical axes respectively to generate horizontal coordinate weights and vertical coordinate weights.

[0065] Regarding step S620, it may include, for example: Figure 4 Steps S621 to S624 are shown.

[0066] S621: Aggregate the global related tensors along the width (horizontal) and height (vertical) directions respectively to obtain a one-dimensional description of the horizontal direction and a one-dimensional description of the vertical direction.

[0067] S622: Combine the horizontal one-dimensional description and the vertical one-dimensional description to obtain the combined direction description.

[0068] S623: Perform channel compression processing on the splicing direction description to obtain a compressed direction description.

[0069] S624: The compression direction description is split into horizontal coordinate weights and vertical coordinate weights respectively.

[0070] S630: Based on the horizontal and vertical coordinate weights, the global relevant tensor is recalibrated with orientation awareness, and the recalibration result is mapped to a two-dimensional displacement representation to obtain the first two-dimensional displacement representation.

[0071] Regarding step S630, it may include, for example: Figure 5 Steps S631 to S632 are shown.

[0072] S631: Multiply the horizontal coordinate weights and vertical coordinate weights element-wise with the global correlation tensor to obtain the coordinate-enhanced correlation tensor.

[0073] S632: Perform displacement mapping on the coordinate enhancement correlation tensor to obtain a two-dimensional displacement representation, thus obtaining the first two-dimensional displacement representation.

[0074] As one possible implementation, the translation mapping can be achieved through a translation mapping unit. Exemplarily, the translation mapping unit may employ convolutional mapping, linear mapping, multilayer perceptron, etc., without limitation.

[0075] It should be noted that, in the current preferred embodiment, Softmax probability normalization is not performed on the globally relevant tensor. In other alternative embodiments, operations such as temperature scaling, Softmax, sparse normalization, or confidence truncation can be selected according to the training stability requirements.

[0076] S7: The corner offset of the target image is obtained by regression based on the first two-dimensional displacement representation corresponding to the depth layer with lower spatial resolution. Based on the corner offset, a projection transformation matrix is ​​generated to describe the projection transformation relationship between the reference image and the target image. The target enhancement features corresponding to the depth layer with higher spatial resolution are spatially mapped using the projection transformation matrix to obtain coarse registration target features. Based on the reference enhancement features corresponding to the depth layer with higher spatial resolution and the coarse registration target features, a corresponding two-dimensional displacement representation is generated as the second two-dimensional displacement representation. The local offset of each regular control point in the target image space is obtained by regression based on the second two-dimensional displacement representation.

[0077] Regular control points can be the intersection points of a pre-defined regular grid within the target image space. For example, the target image space can contain 13×13 regular control points.

[0078] It should be noted that, based on the reference enhancement features and coarse registration target features corresponding to the depth layer with high spatial resolution, the corresponding two-dimensional displacement representation is generated. The process of generating the second two-dimensional displacement representation can refer to the first two-dimensional displacement representation generation process in step S6, and the target enhancement features in step S6 can be replaced by the coarse registration target features. The specific implementation details will not be elaborated here.

[0079] In one example, the target image contains four corner points, so the corner offsets of the target image obtained through regression include the corner offsets of all four corner points. Since each corner offset consists of two dimensions, an x-coordinate and a y-coordinate, each corner offset contains two parameters, and the four corner offsets contain a total of eight parameters. Based on the corner offsets represented by these eight parameters, combined with the original four corner coordinates of the target image, the point correspondence can be established and the projection transformation matrix with eight degrees of freedom can be solved.

[0080] S8: Spatial mapping of the regular control points using the projection transformation matrix yields the initial deformable control points. The initial deformable control points are then added point-by-point to the local offset to obtain the final control points. The nonlinear smooth transformation function is then solved based on the regular control points and the final control points.

[0081] For example, the nonlinear smoothing transformation function can be implemented using a thin plate spline (TPS) transformation.

[0082] As one possible implementation, after solving the nonlinear smoothing transform function based on the rule control points and the final control points, the method further includes sampling the target image and generating a mask representing the effective sampling region. In one example, the nonlinear smoothing transform function is used to sample pixels from the target image using bilinear interpolation to obtain a locally aligned target image. Simultaneously, to accurately define the effective pixel region of the transformed target image, a single tensor (i.e., a tensor where all elements are 1) with the same size as the target image is generated. This single tensor and the target image are synchronously input into the nonlinear smoothing transform function, using the same transform coefficients and interpolation method for spatial mapping. After mapping by the nonlinear smoothing transform function, the values ​​of the regions in the single tensor originally covered by the target image are preserved or smoothed by interpolation, while the values ​​of the blank background regions not covered by the target image remain 0. Thus, the mapped single tensor forms the mask representing the effective sampling region.

[0083] S9: Determine the common canvas based on the spatial extent of the reference image and the spatial extent of the target image after nonlinear smoothing transformation using a nonlinear smoothing transformation function. After mapping the target image to the common canvas using a nonlinear smoothing transformation function, merge it with the reference image to obtain the stitched image.

[0084] As one possible implementation, in determining the common canvas based on the spatial extent of the reference and target images after mapping using a nonlinear smoothing transformation function, the minimum and maximum coordinates of all boundary points in the horizontal (width) and vertical (height) directions can be jointly calculated based on the original boundary of the reference image and the boundary of the final deformed mesh (i.e., the mesh composed of final control points) of the target image after mapping using the nonlinear smoothing transformation function. This determines the size of the common canvas and the required coordinate translation, ensuring that all valid content of both images is fully contained and that all coordinates within the canvas are positive.

[0085] As one possible implementation, after mapping the target image to a common canvas using a non-linear smoothing transformation function and then fusing it with a reference image to obtain the stitched image, the reference image can be directly translated and placed to the corresponding position on the common canvas to form a reference canvas image. Simultaneously, the target image is mapped to the common canvas using a non-linear smoothing transformation function to form a target canvas image. During this process, a reference effective mask and a target effective mask are formed using the mask generated in step S8 to accurately distinguish the effective pixel areas of the image from the blank background areas of the canvas. Finally, based on the two canvas images and their corresponding effective masks, the overlapping areas are fused, while retaining their respective non-overlapping effective areas, thus obtaining the final stitched image.

[0086] For example, the fusion processing can be carried out using mask weighted fusion, multi-band fusion, seam selection, or learning-based fusion, etc., without limitation.

[0087] The above embodiments are described below with a specific example.

[0088] like Figure 6 As shown, a reference image 101 and a target image 102 with overlapping regions are first acquired, and the two images are input into a shared multi-scale feature extraction module 200 with shared weights. The obtained features are recalibrated by a dual statistical channel enhancement module 300 to obtain enhanced features for each depth layer. The first-scale enhanced features (i.e., the reference enhanced features and target enhanced features corresponding to the feature layer with lower spatial resolution) are used by a coordinate-aware correlation module 400 to form a first two-dimensional displacement representation, and the global projection transformation is estimated by a corner offset and global transformation module 500. The target enhanced features corresponding to the feature layer with higher spatial resolution are spatially mapped using a projection transformation matrix to obtain the coarsely registered target intermediate features (i.e., coarsely registered target features) 506. The second-scale enhanced features are combined with the target intermediate features (i.e., the reference enhanced features and coarsely registered target features corresponding to the feature layer with higher spatial resolution) 506, and then the local offset of the regular control points is estimated by a coordinate-aware correlation module 400 and a control point offset and local transformation module 600. The local transformation is obtained by a nonlinear smooth transformation solution module 607. The joint canvas and image fusion module 700 places the reference image 101 and the transformed target image 102 into a common canvas and merges them to output a stitched image 103.

[0089] like Figure 7As shown, for the input fusion feature F, the average description z_avg and the maximum description z_max are obtained through the global average statistics unit 301 and the global maximum statistics unit 302, respectively. The statistical label construction unit 303 constructs statistical labels from the two types of descriptions, and the interactive coding unit 304 performs interactive processing on the two types of statistical information. The interactive results are sequentially processed by the concatenation or flattening unit 305, the channel mapping unit 306, and the gating function unit 307 to generate channel weights w. The channel weights w are multiplied with the input fusion feature F channel by channel to obtain the output enhanced feature F′.

[0090] like Figure 8 As shown, the reference feature Fr and the target feature Ft are processed by channel normalization unit 401 and channel normalization unit 402, respectively. Local block extraction unit 403 extracts K×K×C local blocks from each spatial location of the target feature Ft, and local block rearrangement unit 404 rearranges the local blocks into a matching kernel group. The matching kernel group performs correlation calculations within the complete spatial range of the reference feature Fr through sliding correlation unit 405 to obtain the global correlation tensor V. The global correlation tensor V is processed by width aggregation unit 406 and height aggregation unit 407, and then by shared compression unit 408 to generate horizontal weights a_h and vertical weights a_w. The global correlation tensor V is recalibrated using the two-directional weights to obtain coordinate-enhanced correlation tensor V′409, and the two-dimensional displacement representation q is output by displacement mapping unit 410.

[0091] like Figure 9 As shown, in the first stage, the first matching representation (i.e., the first two-dimensional displacement representation corresponding to the depth layer with lower spatial resolution) 501 is input to the corner offset regression unit 502 to obtain the two-dimensional offsets of the four corner points. Based on the original four corners 503 and the offset four corners 504, the global transformation H505 is solved through direct linear transformation, and the target intermediate features are coarsely registered using the global transformation H505 to obtain the coarse registration features (i.e., coarse registration target features) 506. In the second stage, the control point offset ΔP602 is regressed based on the second matching representation (i.e., the second two-dimensional displacement representation) 601; the regular control point grid P603 is mapped by the global transformation H505 to form the initial deformed grid 604, which is then added to the control point offset ΔP602 to obtain the final control point P′606. Based on the regular control point grid P603 and the final control point P′606, the local transformation is obtained through the nonlinear smooth transformation solution module 607, and the target image is sampled to obtain the locally aligned image 608.

[0092] like Figure 10As shown, the joint boundary calculation unit 701 determines the minimum and maximum coordinates based on the boundaries of the reference image 101 and the final deformed mesh 606 of the target. The canvas boundary and coordinate translation unit 702 determines the common canvas and coordinate translation accordingly. The reference image placement unit 703 places the reference image 101 onto the common canvas, and the target image mapping unit 704 maps the target image onto the same canvas, forming a reference effective mask 705 and a target effective mask 706 respectively. The fusion unit 707 fuses the overlapping areas based on the two canvas images and their corresponding effective masks, while retaining their respective non-overlapping effective areas, resulting in the stitched image 103.

[0093] According to a further embodiment of the present invention, an electronic device is also provided. Figure 11 This diagram illustrates a schematic block diagram of an electronic device employing a processor-based hardware implementation according to an embodiment of the present invention. The hardware structure of the electronic device of the present invention can be implemented using a bus architecture. The bus architecture can include any number of interconnect buses and bridges, depending on the specific application and overall design constraints of the hardware. Bus 1100 connects various circuits including one or more processors 1200, memory 1300, and / or hardware modules. Bus 1100 can also connect various other circuits 1400 such as peripheral devices, voltage regulators, power management circuits, external antennas, etc. Bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one connection line is used in this diagram, but this does not indicate that there is only one bus or one type of bus. The memory 1300 stores a computer program. When the processor 1200 executes the computer program, the processor 1200 is able to execute the image stitching method based on feature enhancement and coordinate awareness according to the above embodiments of the present invention.

[0094] The present invention also provides a readable storage medium storing a computer program, which, when executed by a processor, is used to implement the methods described above. A "readable storage medium" can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples of a readable storage medium include: an electrical connection (electronic device) having one or more wires, a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable read-only memory (CDROM), etc.

[0095] This invention also provides a computer program product. The method of this invention can be implemented wholly or partially through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented wholly or partially in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed, the processes or functions of this invention are performed wholly or partially.

[0096] Computer programs or instructions can be stored in a readable storage medium or transferred from one readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The readable storage medium can be any available medium capable of access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; an optical medium, such as a digital video optical disc; or a semiconductor medium, such as a solid-state drive. The computer-readable storage medium can be a volatile or non-volatile storage medium, or it can include both volatile and non-volatile types of storage media.

[0097] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0098] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0099] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0100] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0101] In the description of this specification, the references to terms such as "one embodiment / mode," "some embodiments / modes," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, or characteristic described in connection with that embodiment / mode or example is included in at least one embodiment / mode or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment / mode or example. Moreover, the specific features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments / modes or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments / modes or examples described in this specification, as well as the features of different embodiments / modes or examples.

[0102] Those skilled in the art should understand that the above embodiments are merely for illustrating the present invention and are not intended to limit the scope of the invention. Those skilled in the art can make other changes or modifications based on the above disclosure, and these changes or modifications still fall within the scope of the present invention.

Claims

1. An image stitching method based on feature enhancement and coordinate awareness, characterized in that, include: Acquire at least one reference image and at least one target image; Multi-scale feature extraction with shared weights is performed on the reference image and the target image respectively to obtain the corresponding multi-scale features; The features with lower spatial resolution in the multi-scale features are upsampled and then spliced ​​with the adjacent features with higher spatial resolution along the channel dimension to form the layered fusion features corresponding to each depth layer. The statistical descriptions of the global average statistics and the global maximum statistics of each layer fusion feature are obtained respectively, and the statistical descriptions of the global average statistics and the global maximum statistics are interactively encoded to obtain the channel weights. The corresponding layer fusion features are recalibrated using the channel weights to obtain the enhancement features corresponding to each depth layer. Normalize the enhancement features corresponding to each depth layer in the reference image and the target image respectively to obtain the reference enhancement features and target enhancement features corresponding to each depth layer; Based on the reference enhancement features and the target enhancement features, a corresponding two-dimensional displacement representation is generated as the first two-dimensional displacement representation; The corner offset of the target image is obtained by regression of the first two-dimensional displacement representation corresponding to the depth layer with low spatial resolution; Based on the corner offset, a projection transformation matrix is ​​generated to describe the projection transformation relationship between the reference image and the target image; The projection transformation matrix is ​​used to spatially map the target enhancement features corresponding to the depth layer with high spatial resolution to obtain coarsely registered target features. Based on the reference enhancement features corresponding to the depth layer with high spatial resolution and the coarse registration target features, a corresponding two-dimensional displacement representation is generated as the second two-dimensional displacement representation; based on the second two-dimensional displacement representation, the local offset of each regular control point in the target image space is obtained by regression. The initial deformation control points are obtained by spatially mapping the regular control points using the projection transformation matrix. The initial deformation control points are added point by point to the local offset to obtain the final control points; the nonlinear smooth transformation function is solved based on the rule control points and the final control points. A common canvas is determined based on the spatial range of the reference image and the spatial range of the target image after nonlinear smoothing transformation by the nonlinear smoothing transformation function; the target image is mapped to the common canvas through the nonlinear smoothing transformation function and then fused with the reference image to obtain a stitched image.

2. The image stitching method based on feature enhancement and coordinate awareness as described in claim 1, characterized in that, Based on the reference enhancement features and the target enhancement features, a corresponding two-dimensional displacement representation is generated as the first two-dimensional displacement representation, including: Local neighborhoods are extracted from each spatial location of the target augmented feature and rearranged into multiple matching kernels, so that each matching kernel acts on the spatial range of the reference augmented feature to obtain a global correlation tensor; The globally relevant tensors are aggregated along the horizontal and vertical axes respectively to generate horizontal coordinate weights and vertical coordinate weights; Based on the horizontal coordinate weights and the vertical coordinate weights, the global correlation tensor is recalibrated with orientation awareness, and the recalibration result is mapped to a two-dimensional displacement representation to obtain the first two-dimensional displacement representation.

3. The image stitching method based on feature enhancement and coordinate awareness as described in claim 2, characterized in that, Aggregate the globally relevant tensors along the horizontal and vertical axes respectively to generate horizontal coordinate weights and vertical coordinate weights, including: The global relevance tensor is aggregated along the width and height directions respectively to obtain a horizontal one-dimensional description and a vertical one-dimensional description. The horizontal one-dimensional direction description and the vertical one-dimensional direction description are spliced ​​together to obtain the spliced ​​direction description. The splicing direction description is subjected to channel compression processing to obtain a compressed direction description; The compression direction description is split into horizontal coordinate weights and vertical coordinate weights.

4. The image stitching method based on feature enhancement and coordinate awareness as described in claim 2, characterized in that, The global correlation tensor is recalibrated based on the horizontal coordinate weights and the vertical coordinate weights, and the recalibration result is mapped to a two-dimensional displacement representation to obtain a first two-dimensional displacement representation, including: The horizontal coordinate weights and the vertical coordinate weights are multiplied element-wise with the global correlation tensor to obtain the coordinate-enhanced correlation tensor. The coordinate enhancement correlation tensor is subjected to displacement mapping to a two-dimensional displacement representation, thus obtaining the first two-dimensional displacement representation.

5. The image stitching method based on feature enhancement and coordinate awareness as described in claim 1, characterized in that, The channel weights are obtained by interactively encoding the statistical descriptions of the global average statistics and the global maximum statistics, including: The statistical descriptions of the global average statistics and the global maximum statistics are stacked to construct a statistical label; The statistical markers are interactively processed to obtain the interactive results; The interaction results are flattened and channel-mapped sequentially, and the channel weights are generated using a gating function.

6. The image stitching method based on feature enhancement and coordinate awareness as described in claim 1, characterized in that, The features with lower spatial resolution in the multi-scale features are upsampled and then concatenated along the channel dimension with adjacent features with higher spatial resolution to form hierarchical fused features corresponding to each depth layer, including: The deep features in the multi-scale features are upsampled and adjusted to the same spatial scale as the mid-scale features in the multi-scale features, and the two are spliced ​​together to form a mid-scale fused feature. The mid-level fusion feature is upsampled and adjusted to the same spatial scale as the shallow feature in the multi-scale feature, and the two are spliced ​​together to form a shallow fusion feature; The mid-layer fusion feature and the shallow-layer fusion feature are used as the layered fusion features corresponding to each depth layer.

7. The image stitching method based on feature enhancement and coordinate awareness as described in claim 1, characterized in that, The corresponding hierarchical fusion features are recalibrated using the channel weights to obtain the enhanced features corresponding to each depth layer, including: The channel weights are multiplied one by one with the corresponding hierarchical fusion features to obtain the product result; The enhanced feature is obtained by adding, concatenating, or normalizing the product result with the corresponding hierarchical fusion feature.

8. The image stitching method based on feature enhancement and coordinate awareness as described in claim 2, characterized in that, After obtaining the global correlation tensor, the method further includes: performing numerical adjustment processing on the global correlation tensor; wherein, the numerical adjustment processing includes temperature scaling processing, Softmax probability normalization processing, sparse normalization processing, or confidence truncation processing.

9. An electronic device, characterized in that, include: The memory stores execution instructions; as well as A processor that executes the execution instructions stored in the memory, causing the processor to perform the image stitching method based on feature enhancement and coordinate awareness as described in any one of claims 1 to 8.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the image stitching method based on feature enhancement and coordinate awareness as described in any one of claims 1 to 8.

Images