Image splicing method, image splicing equipment and storage medium

Abstract

The invention discloses an image splicing method, an image splicing device and a storage medium, and relates to the technical field of image processing, and the image splicing method comprises the steps of extracting feature points of a to-be-spliced image based on a detection threshold value corresponding to a splicing overlapping side of the to-be-spliced image, the detection threshold value of the splicing overlapping side is smaller than the detection threshold value of the non-overlapping side; determining a feature matching matrix corresponding to the feature points of the to-be-spliced image based on a bidirectional matching strategy; according to a target homography matrix obtained by iteration of effective matching point pairs of the feature matching matrix, executing an image projection action of the to-be-spliced image; and fusing the projected images based on a preset image fusion algorithm to obtain a target spliced image. Effective feature information is emphatically extracted in the key splicing area, invalid interference is reduced, high-quality effective matching points are screened out for matching and splicing, and the image splicing effect of a complex wearing scene is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to image stitching methods, image stitching devices and storage media. Background Technology

[0002] In the field of image stitching, cameras are typically calibrated first to establish a fixed coordinate system. However, in scenarios involving headphones equipped with cameras, the random placement of the left and right headphones makes it impossible to predetermine their relative positions, such as rotation angles, tilt angles, and spacing. Therefore, during the pairing process, the inability to establish stable geometric constraints between the left and right cameras makes it difficult to accurately unify the coordinate systems of the two viewpoints. This leads to numerous mismatches during feature point matching, resulting in poor image stitching quality.

[0003] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main objective of this application is to provide an image stitching method, an image stitching device, and a storage medium, aiming to solve the technical problem of poor image stitching effect without pre-calibration.

[0005] To achieve the above objectives, this application proposes an image stitching method, which includes: Feature points of the image to be stitched are extracted based on the detection threshold corresponding to the overlapping side of the image to be stitched, wherein the detection threshold of the overlapping side is less than the detection threshold of the non-overlapping side. The feature matching matrix corresponding to the feature points of the image to be stitched is determined based on a bidirectional matching strategy. Based on the target homography matrix obtained by iterating through the effective matching point pairs of the feature matching matrix, the image projection action of the image to be stitched is performed. The projected images are fused based on a preset image fusion algorithm to obtain the target stitched image.

[0006] In one embodiment, the image to be stitched includes a first image and a second image. Before the step of extracting feature points of the image to be stitched based on a detection threshold corresponding to the overlapping side of the stitching, wherein the detection threshold of the overlapping side is less than the detection threshold of the non-overlapping side, the image stitching method further includes: Determine the relative positions between the first image and the second image to be stitched together; The overlapping side between the first image and the second image is determined based on their relative positions.

[0007] In one embodiment, before performing the image projection operation on the image to be stitched based on the target homography matrix obtained by iterating through the effective matching point pairs of the feature matching matrix, the image stitching method further includes: The first homography matrix with the largest number of interior points is determined by a random sampling consistency algorithm based on a preset number of iterations. The target number of iterations for the random sampling consensus algorithm is determined based on the proportion of interior points in the first homography matrix. Based on the target number of iterations and the random sampling consistency algorithm, the second homography matrix of the effective matching point pairs is determined, wherein the second homography matrix has the largest number of interior points; The target homography matrix is ​​obtained by performing weighted least squares processing on all interior points of the second homography matrix.

[0008] In one embodiment, the step of determining the first homography matrix with the largest number of interior points using the random sampling consensus algorithm based on a preset number of iterations includes: Based on the matching point pairs of the feature matching matrix and the preset number of iterations, the estimated homography matrix of the random minimum sample set is calculated according to the random sampling consensus algorithm; Determine the first homography matrix among the estimated homography matrices that has the largest number of interior points of the projection transformation model.

[0009] In one embodiment, before performing the image projection operation on the image to be stitched based on the target homography matrix obtained by iterating through the effective matching point pairs of the feature matching matrix, the image stitching method further includes: Determine the geometric consistency verification results, confidence level, and / or vertical coordinate differences between the matching point pairs of the feature matching matrix; Delete the matching point pairs in the geometric consistency verification results that do not match the homography matrix with the most internal points, the matching point pairs with a confidence level less than the preset confidence level, and / or the matching point pairs with a vertical coordinate difference greater than the preset threshold, to obtain the effective matching point pairs; If the number of valid matching point pairs is greater than a preset number, the target homography matrix is ​​obtained by iterating through the valid matching point pairs of the feature matching matrix, and the image projection action of the image to be stitched is performed.

[0010] In one embodiment, after the step of extracting feature points of the image to be stitched based on the detection threshold corresponding to the overlapping side of the image to be stitched, wherein the detection threshold of the overlapping side is less than the detection threshold of the non-overlapping side, the image stitching method further includes: Determine the number of feature points in the image to be stitched; If the number of feature points is less than the first preset number, the detection threshold of the splicing overlap side is reduced based on the interval corresponding to the target number and the number of feature points. If the number of feature points is greater than the second preset number, the detection threshold of the splicing overlap side is increased based on the interval corresponding to the target number and the number of feature points.

[0011] In one embodiment, the step of determining the feature matching matrix corresponding to the feature points of the image to be stitched based on a bidirectional matching strategy includes: Determine the first set of nodes of the first image to be stitched together, and the second set of nodes of the second image to be stitched together; Determine the first feature matching matrix from the feature points of the first node set to the feature points of the second node set; Determine the second feature matching matrix from the feature points of the second node set to the feature points of the first node set; The feature matching matrix is ​​output based on the similarity matching results between the first feature matching matrix and the second feature matching matrix.

[0012] In one embodiment, after the step of fusing the projected images based on a preset image fusion algorithm to obtain the target stitched image, the image stitching method further includes: Determine the number of inliers, the proportion of inliers, the average reprojection error of inliers, and the height difference of matching point pairs in the target homography matrix; If the number of inliers is greater than a preset number, the proportion of inliers is greater than a preset proportion, the average reprojection error of the inliers is less than a preset error, and the height difference of the matching point pairs is less than a preset height, then the effective area of ​​the target stitched image is determined. The target output ratio of the target stitched image is determined based on the effective area, and the target stitched image is cropped according to the target output ratio to obtain a cropped image; The defective areas of the cropped image are identified, and the defective areas are repaired based on a preset algorithm to obtain the output image corresponding to the target stitched image.

[0013] In addition, to achieve the above objectives, this application also proposes an image stitching device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the image stitching method described above.

[0014] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the image stitching method described above.

[0015] One or more technical solutions proposed in this application have at least the following technical effects: A detection threshold lower than that on the non-overlapping side is set on the overlapping side of the images to be stitched, and feature points are extracted based on the detection threshold. Then, a feature matching matrix of the feature points is determined by combining a bidirectional matching strategy. Next, an image projection operation is performed based on the target homography matrix obtained after iteration of the effective matching point pairs of the matrix. Finally, the projected images are fused by a preset image fusion algorithm. In this way, effective feature information is accurately captured in the key stitching area of ​​the image, reducing the interference of invalid feature points in the non-overlapping area. At the same time, high-quality effective matching point pairs are selected, which greatly reduces the mismatch problem caused by the random relative position of the video module of the head-mounted device. It also achieves accurate unification of the coordinate systems of the two viewpoint images without the need to establish stable geometric constraints in advance. This makes up for the lack of geometric constraints caused by the randomness of the head-mounted device wearing position, solves the problem of poor image stitching effect caused by inconsistent coordinate systems and a lot of mismatches, and improves the stitching results of the head-mounted device in complex wearing scenarios. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the shooting range of the device used for the image stitching method of this application; Figure 2 This is a flowchart illustrating the first embodiment of the image stitching method of this application; Figure 3 This is a schematic diagram illustrating the selection of matching point pairs in the feature matching matrix of the image stitching method of this application; Figure 4 This is a flowchart illustrating the second embodiment of the image stitching method of this application. Figure 5 This is a flowchart illustrating the process of obtaining the homography matrix by iterating through effective matching point pairs in the image stitching method of this application. Figure 6 This is a schematic diagram of the cropping process after image stitching using the image stitching method of this application. Figure 7 This is a simplified flowchart illustrating the image stitching method combined with various embodiments of this application; Figure 8 This is a schematic diagram illustrating the image stitching effect of this application; Figure 9 This is another schematic diagram of the image stitching effect in this application; Figure 10 This is a schematic diagram of the device structure of the hardware operating environment involved in the image stitching method in the embodiments of this application.

[0019] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0020] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0021] The main solution of this application embodiment is: based on the detection threshold corresponding to the overlapping side of the image to be stitched, extract the feature points of the image to be stitched, wherein the detection threshold of the overlapping side is less than the detection threshold of the non-overlapping side; The feature matching matrix corresponding to the feature points of the image to be stitched is determined based on a bidirectional matching strategy. Based on the target homography matrix obtained by iterating through the effective matching point pairs of the feature matching matrix, the image projection action of the image to be stitched is performed. The projected images are fused based on a preset image fusion algorithm to obtain the target stitched image.

[0022] In this embodiment, for ease of description, the following description will focus on image stitching as the main process.

[0023] In the field of image stitching, cameras are typically calibrated first to establish a fixed coordinate system. However, in scenarios involving headphones equipped with cameras, the random placement of the left and right headphones makes it impossible to predetermine their relative positions, such as rotation angles, tilt angles, and spacing. Therefore, during the pairing process, the inability to establish stable geometric constraints between the left and right cameras makes it difficult to accurately unify the coordinate systems of the two viewpoints. This leads to numerous mismatches during feature point matching, resulting in poor image stitching quality.

[0024] This application provides a solution that sets a detection threshold on the overlapping side of the image to be stitched lower than that on the non-overlapping side, and extracts feature points based on the detection threshold to capture more effective feature information in the key stitching area of ​​the image, reducing the number of feature point detections in the non-overlapping area, thereby reducing the interference of invalid feature points. Subsequently, a feature matching matrix is ​​determined by combining a bidirectional matching strategy, which can further filter out high-quality effective matching point pairs, significantly reducing the mismatch problem caused by the random relative position of the video modules of the head-mounted device. Based on the target homography matrix obtained by iteratively obtaining the effective matching point pairs, the image projection action is performed, which can achieve precise unification of the coordinate systems of the two viewpoint images without the need to establish stable geometric constraints in advance, effectively making up for the lack of geometric constraints caused by the randomness of the head-mounted device wearing position. Finally, the projected image is fused by a preset image fusion algorithm, which significantly improves the coherence and integrity of the target stitched image, improves the problem of poor image stitching effect caused by inconsistent coordinate systems and many mismatches, and ensures that the head-mounted device can still output stable and reliable stitching results in complex wearing scenarios.

[0025] It should be noted that the execution subject in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or image stitching device capable of performing the above functions. The image stitching device can be the main control device of an image acquisition device, such as a smartphone or personal computer, which is a device equipped with a camera, like headphones or glasses. The image stitching device and the image acquisition device can also be the same device; that is, images are acquired through headphones, glasses, or other devices with cameras, and the image stitching process is completed locally on these devices.

[0026] The following description uses an earphone with a camera as the image acquisition device and a main control device connected to the earphone as the image stitching device to illustrate this embodiment and the following embodiments.

[0027] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0028] A top-down view of the user's head and the area being captured when the user is wearing headphones. Figure 1 As shown, the left earphone captures the left field of view, and the right earphone captures the right field of view. The face is located in the center of the image, but there is an overlapping area between the left and right images. Because the wearing positions of the left and right earphones are relatively random, there is no fixed coordinate system between them. Therefore, during the pairing process, it is impossible to establish a stable geometric constraint relationship between the left and right cameras. The coordinate systems of the two perspective images are difficult to unify accurately, and a large number of mismatches are likely to occur during feature point matching, resulting in poor image stitching effect.

[0029] Based on this, this application provides an image stitching method, please refer to... Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the image stitching method of this application.

[0030] In this embodiment, the image stitching method includes steps S10 to S40: Step S10: Extract feature points of the image to be stitched based on the detection threshold corresponding to the overlapping side of the image to be stitched.

[0031] In this embodiment, the images to be stitched include a first image captured by the first camera of the earphone and a second image captured by the second camera. The first and second cameras were captured at the same time but from different angles. The detection threshold is a parameter that controls the density of feature point extraction, used to filter feature points that meet the response value requirements. In image feature point extraction, feature points larger than the detection threshold are typically considered valid. The stitching overlap side refers to the intersection area of ​​the shooting ranges of the two earphone cameras, i.e., the core alignment area for image stitching. For the left image, the focus is on detecting feature points in the right area, and for the right image, the focus is on detecting feature points in the left area. The detection threshold on the stitching overlap side is lower than the detection threshold on the non-overlapping side, so as to detect more feature points in the key areas and reduce the number of feature points detected in non-key areas, thereby reducing the interference of irrelevant feature points in non-key areas.

[0032] Therefore, when determining the overlapping side of the stitching, the relative position between the first and second images to be stitched can be determined first. This can be done by determining the relative position based on the cameras corresponding to the first and second images, respectively, and then determining the overlapping side of the stitching between the first and second images based on this relative position. Specifically, when the relative position is left-right, and the first image is captured by the left camera and the second image is captured by the right camera, the overlapping side of the stitching between the first and second images can be determined based on this relative position. Figure 1 The image acquisition area shown indicates that the overlapping side of the first image is the right side, and similarly, the overlapping side of the second image is the left side. The proportion of the overlapping side can be determined based on the image overlap range between the video modules. For example, if the field-of-view overlap rate of the two cameras is 40%, then the overlapping side occupies 40% of the original image.

[0033] Optionally, in addition to setting up cameras on the left and right sides for image acquisition, the head-mounted device can also be equipped with two cameras at different heights, one above the other, to perform image stitching based on the images at different heights. Therefore, if the relative positions are vertical, the lower side of the upper image and the upper side of the lower image are used as the stitching overlap side. The determination of the stitching overlap side for other relative positions is similar, and will not be elaborated here.

[0034] Specifically, feature point extraction can be performed based on the SIFT algorithm or the ORB algorithm. In addition, other deep learning algorithms can also be used for processing, but this application does not limit the specific methods used.

[0035] As an optional implementation, the earphones simultaneously capture images using independent left and right ear camera systems. After connecting to a main control device, such as a smartphone, via a wireless communication protocol, at least two images to be stitched are transmitted to the main control device for processing. After image transmission, necessary preprocessing such as noise reduction and color space conversion can be performed. Upon receiving the images to be stitched, the main control device calculates the overlapping area of ​​the left and right camera images using the camera's built-in distance sensor and shooting parameters. It determines that the overlapping area occupies 30% of each image; that is, the rightmost 30% of the left image and the leftmost 30% of the right image are the overlapping areas. This area is crucial for stitching alignment, requiring the extraction of a sufficient number of high-quality feature points.

[0036] Therefore, when setting the detection threshold, the feature point detection threshold on the overlapping side of the stitching is lowered, while the detection threshold on the non-overlapping side is increased. This allows for the selection of more feature points with medium to high response values ​​on the overlapping side, while on the non-overlapping side, only key feature points with high response values ​​are retained at a high threshold, reducing interference from redundant low-quality points. Finally, a feature point detection algorithm based on deep learning is used to detect key feature points in both the left and right images. The feature point detection algorithm based on a deep convolutional neural network includes: an encoder network, which uses a convolutional layer structure to extract multi-level features from the input image; a detection head, which outputs the probability that each pixel location is a feature point; and a descriptor head, which generates high-dimensional descriptor vectors, such as 256-dimensional vectors, for the detected feature points.

[0037] Step S20: Determine the feature matching matrix corresponding to the feature points of the image to be stitched based on the bidirectional matching strategy.

[0038] In this embodiment, the bidirectional matching strategy refers to a strategy that filters valid matching pairs through bidirectional verification of forward and reverse matching, effectively eliminating one-way mismatches. The feature matching matrix is ​​a structured data matrix that stores the matching relationships of feature points, including information such as the index, similarity, and coordinates of the matching point pairs, providing data support for subsequent screening of valid point pairs and estimation of the homography matrix.

[0039] As an optional implementation, in the image feature matching process, a feature matching algorithm based on graph neural networks can be used to model feature point matching as a graph matching problem. The feature points of the left image are constructed as a node set A of the graph; the feature points of the right image are constructed as a node set B of the graph; the association strength between nodes is calculated through an attention mechanism; finally, a matching matrix is ​​output, representing the matching probability of each pair of feature points. Therefore, the first node set of the first image to be stitched and the second node set of the second image to be stitched can be determined first. Then, the first feature matching matrix from the feature points of the first node set to the feature points of the second node set is determined; simultaneously, the second feature matching matrix from the feature points of the second node set to the feature points of the first node set is determined; finally, the feature matching matrix is ​​output based on the similarity matching results between the first and second feature matching matrices.

[0040] For example, after extracting feature points from the left and right images, a node feature vector is constructed for each feature point. The node vector for each feature point in the left image contains a 32-dimensional ORB binary descriptor, 2-dimensional image coordinates (x, y), and 1-dimensional scale information, forming a 1×35 node feature vector. All node feature vectors from the left image feature points form the first node set A. Similarly, the node vectors for each feature point in the right image use the same structure, forming the second node set B. Finally, the two node sets are input into a pre-defined graph neural network model to complete the graph structure initialization before matching.

[0041] The graph neural network model performs inter-layer operations on a first node set A and a second node set B. Through linear transformation, it maps the node feature vectors of A and B to the same high-dimensional space, eliminating feature space differences. Then, it calculates the attention score between each node in A and all nodes in B using a self-attention mechanism, and normalizes this score using the Softmax function to obtain the matching probability of each node pair. Finally, it outputs the first feature matching matrix M1, where M1[i][j] represents the matching probability between the i-th node in the first node set and the j-th node in the second node set. Next, it performs reverse matching, repeating the above graph neural network operation process. Using the second node set B as the query end and the first node set A as the target end, it outputs the second feature matching matrix M2 through linear transformation, attention score calculation, and Softmax normalization, where M2[j][i] represents the matching probability between the j-th node in the second node set and the i-th node in the first node set.

[0042] Finally, the similarity results are filtered, and a bidirectional matching consistency threshold θ = 0.8 is set. Then, each element M1[i][j] of the first feature matching matrix M1 is traversed to find candidate matching pairs (i,j) where M1[i][j] ≥ θ. The corresponding element M2[j][i] in the second feature matching matrix M2 is verified to be ≥ θ. If it is satisfied, it is determined to be a valid matching pair; otherwise, it is discarded. Finally, the information of all valid matching pairs is stored row by row to generate the final feature matching matrix M. For example, when M1

[100]

[80] = 0.96, it means that the 100th node in the left image matches the 80th node in the right image with a 96% probability, while M1

[100] [other j] = 0.01-0.05, the probability of this point matching other nodes is extremely low. In the second feature matching matrix M2 obtained after reverse matching, M2

[80]

[100] =0.95, that is, the 80th node in the right figure and the 100th node in the left figure have a 95% matching probability. These two points meet the requirements and are stored as a valid matching point pair.

[0043] In another alternative implementation, similar matching pairs between feature descriptors can be quickly found by constructing a KD-tree or ball tree index through fast nearest neighbor search packet matching.

[0044] Step S30: Based on the target homography matrix obtained by iterating through the effective matching point pairs of the feature matching matrix, perform the image projection action of the image to be stitched.

[0045] In this embodiment, after obtaining the feature matching matrix, it is necessary to perform quality assessment and screening of the matching points to further eliminate matching point pairs with low accuracy and obtain valid matching point pairs. Specifically, confidence screening, geometric consistency testing, and / or height constraint screening can be performed on the matching point pairs in the feature matching matrix to select valid matching point pairs. These high-quality matching point pairs can then be used for subsequent stitching processing, improving the stitching effect.

[0046] Therefore, before step S20, a quality assessment and screening of matching points is required. Specifically, the geometric consistency verification results, confidence levels, and / or vertical coordinate differences between matching point pairs in the feature matching matrix can be determined first. Then, matching point pairs that do not match the homography matrix with the most internal points, matching point pairs with confidence levels lower than a preset confidence level, and / or matching point pairs with vertical coordinate differences greater than a preset threshold are deleted from the geometric consistency verification results, thus obtaining valid matching point pairs. The screening of valid matching point pairs can be performed individually, in pairs, or by performing all three screening actions together.

[0047] It should be noted that during the geometric consistency verification process, the Random Sample Consistency (RANSAC) algorithm is used for geometric verification. A minimum sample set is randomly selected from the matching point pairs using a random sampling method. Then, the homography matrix is ​​estimated based on this sample set. Inlier counts are performed, and subsequently, the number of matching points in the homography matrix that conform to the geometric relationship under the projection transformation model is calculated, i.e., the reprojection error is less than a threshold. This process is iterated a preset number of times, selecting the model with the most inliers. Finally, matching point pairs that do not conform to the optimal model are marked as outliers and removed. When matching based on confidence, the confidence score output by the matching algorithm is used for filtering. A confidence threshold is first set, and matching point pairs with confidence scores below the threshold are removed, while high-confidence matching point pairs are retained for subsequent processing. In height constraint filtering, for each pair of matching points, the difference in their vertical coordinates in the left and right images is calculated. Matching point pairs with vertical coordinate differences exceeding a threshold are then removed. Optionally, the preset threshold can be 0.1-0.2 times the image height. After filtering the valid matching point pairs, it is also necessary to check the number of the filtered matching point pairs. If the number is less than the minimum threshold, the matching is determined to be unsuccessful. If the number of valid matching point pairs is greater than the preset number, the subsequent processing continues, that is, the processing action of step S20 is executed to ensure that the number of real matching point pairs can meet the minimum number requirements of subsequent image stitching processing.

[0048] As an optional implementation method, the multiple matching point filtering process is as follows: Figure 3 As shown, after obtaining the preliminary matching result, i.e. the feature matching matrix, candidate matching point pairs are first filtered by confidence level using a threshold of 0.1-0.3. Then, the homography matrix is ​​estimated by validating random sampling using RANSAC set. The interior points are calculated using the homography matrix, and optimization is performed based on a preset number of iterations. Matching point pairs that do not conform to the optimal model are marked as outliers and removed. Finally, valid matching point pairs are checked. If the number N is greater than the preset number N_min, such as 10, it is considered qualified and further processing is performed. Otherwise, the splicing is directly judged as a failure.

[0049] Furthermore, after obtaining valid matching point pairs, projection processing of the images to be stitched can be performed based on the target homography matrix. The target homography matrix is ​​a 3×3 matrix that, after iterative optimization, accurately describes the projection transformation relationship between the two image planes, and the projection of points is achieved through homogeneous coordinate transformation.

[0050] As can be understood, image projection refers to the process of mapping all pixels of one image to the coordinate system of another image using a homography matrix. The core is to unify the coordinate systems of the two viewpoint images. During the projection process, one image, such as the one on the right, is usually selected as the target coordinate system, and the other image, the one on the left, is projected onto this coordinate system. For each pixel coordinate (x, y) in the left image, after applying the target homography matrix H, the corresponding position (x', y') in the coordinate system of the right image is obtained. Pixel resampling can be performed using bilinear interpolation or bicubic interpolation during this process.

[0051] Furthermore, after image stitching, it is necessary to calculate the size of the stitched image, that is, to calculate the smallest rectangular canvas that can accommodate all the content of the two images. For example, calculating the left... Figure 4 Calculate the coordinates of the projected corner points, and then calculate the right... Figure 4 The coordinates of each corner point are used to determine the smallest bounding rectangle of all corner points, which is the size of the output canvas.

[0052] Step S40: The projected images are fused based on a preset image fusion algorithm to obtain the target stitched image.

[0053] In this embodiment, after the images are projected, although the projected images are aligned, differences in brightness or texture due to lighting, sensor variations, etc., will result in noticeable seams when directly stitched together. Therefore, image stitching and fusion processing is required. The preset image fusion algorithm is a pre-set image stitching and fusion technique used to eliminate stitching seams in overlapping areas of the projected images. A multi-band fusion algorithm can be used for image fusion, including constructing an image pyramid, calculating fusion weights, layered fusion, and image reconstruction processing to obtain the target stitched image. Specifically, a multi-band fusion algorithm is used to achieve a natural transition. A fusion weight map is calculated for each image, and a distance transformation is performed on the effective area (non-black border area) of each image. The value of each pixel represents the distance to the nearest boundary. In overlapping areas, the closer to the boundary, the smaller the weight; the farther away, the larger the weight. Linear interpolation or S-curve interpolation can be used for processing.

[0054] For example, the left face image A and right face image B to be stitched have already undergone projection processing based on the target homography transformation matrix, and the two images are aligned to the same coordinate system. When using a multi-band fusion algorithm to achieve seamless stitching, the black borders of images A and B are first removed. Then, four Gaussian pyramids are constructed for each image. The 0th layer represents the original valid image, and layers 1-3 are downsampled and Gaussian blurred sequentially to cover different frequency band information. Next, a fusion weight map is calculated. The Euclidean distance from the valid region of image A to its right boundary is calculated, and the Euclidean distance from the valid region of image B to its left boundary is calculated. In the overlapping region, the distance values ​​of corresponding pixels in the two images are used as initial weights and normalized to a sum of 1. Finally, S-curve interpolation using the Sigmoid function is used to optimize the weight transition smoothly. Next, the four pyramid layers are fused layer by layer. The third layer, low frequency band, emphasizes image contour consistency fusion. The first and second layers, mid frequency band, retain texture and other details with weighted fusion. The zero layer, high frequency band, combines optimized weights to accurately fuse key edges. Finally, the fused pyramid image is reconstructed using an inverse Gaussian pyramid, outputting a complete stitched image with no black borders, no stitching marks, and natural transitions between texture and contour in overlapping areas, resulting in a coherent and unified visual effect.

[0055] This embodiment provides an image stitching method. By adjusting the threshold on the overlapping side of the stitching, sufficient feature points are extracted from the core stitching area, providing ample feature points for the matching process while reducing invalid interference in non-overlapping areas. Subsequently, a bidirectional matching strategy is used to further eliminate mismatched points, resulting in a feature matching matrix that ensures the quality of the matching matrix. The matching results of the feature matching matrix are then further filtered, and iterative processing is performed based on the filtered effective feature point pairs to obtain the target homography matrix for image projection stitching. Finally, projection and seamless fusion processing are performed, solving the problems of missing geometric constraints, inconsistent coordinate systems, and mismatches caused by the random wearing of image acquisition devices with different shooting angles. The final output target stitched image has a complete viewpoint, no obvious stitching seams, and clear details, effectively improving matching and iteration efficiency. This ensures that the head-mounted device can stably output high-quality panoramic stitching results in any wearing scenario, meeting users' needs for shooting large-scale scenes.

[0056] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 4 Before step S30, the image stitching method further includes steps S50 to S80: Step S50: Determine the first homography matrix with the largest number of interior points based on a random sampling consistency algorithm with a preset number of iterations; In this embodiment, after obtaining the feature matching matrix, the result of the feature matching matrix needs to be subjected to multiple filtering processes. During the multiple filtering process, the Random Sample Consensus (RANSAC) algorithm is used for geometric verification to filter out the homography matrix with the largest number of interior points.

[0057] Specifically, based on the matching point pairs of the feature matching matrix and a preset number of iterations, the estimated homography matrix of the random minimum sample set can be calculated using the random sampling consistency algorithm. This allows us to determine the first homography matrix in the estimated homography matrix that has the most interior points of the projection transformation model. It is understandable that the process of determining the first homography matrix with the most interior points is the same as the process of determining the geometric consistency verification results between the matching point pairs of the feature matching matrix. That is, after finding the first homography matrix with the most interior points through a preset number of iterations during the multiple matching point screening process, this matrix can be directly used to calculate the target homography matrix in subsequent iterations.

[0058] Optionally, to improve robustness, multiple candidate homography matrices can be estimated, and then the optimal model, i.e. the first homography matrix, can be selected from the multiple candidate homography matrices by considering the number of interior points, the average reprojection error, and the rationality of the geometric transformation.

[0059] Step S60: Determine the target number of iterations for the random sampling consensus algorithm based on the proportion of interior points in the first homography matrix.

[0060] It should be noted that when determining the target homography matrix based on effective matching point pairs, traditional RANSAC uses a fixed number of iterations for least squares matrix estimation, where edge inliers can interfere with the matrix fitting accuracy. For example, when the scene features are clear and the matching points are of high quality, a large amount of invalid computational data is generated during the preset number of iterations, wasting computing power. Conversely, when the scene is complex, such as in low light or with blurred textures leading to a low proportion of inliers, the preset number of iterations may be insufficient to find the globally optimal model, resulting in poor matrix estimation accuracy. The inlier proportion is the ratio of the number of inliers to the total number of matching point pairs used in the iterations.

[0061] Therefore, in this embodiment, when calculating the target homography matrix based on effective matching points, the RANSAC algorithm with a preset number of iterations is not used directly. Instead, the required number of iterations is dynamically calculated based on the previously sampled inlier ratio. During the iteration process, the number of iterations is adjusted based on the real-time updated inlier ratio, thereby ensuring a high inlier ratio while completing the iteration quickly.

[0062] Specifically, when adjusting the number of iterations based on the proportion of interior points, the calculation formula is as follows: , Where N is the number of iterations required for RANSAC, rounded up. p is the expected success probability, i.e., the probability of finding at least one full interior sample set after N iterations, usually taken as 0.99. ε is the proportion of interiors in the current optimal model (0 < ε < 1), which is equal to the number of interiors divided by the total number of matching points. s is the number of points sampled each time; for homography matrix estimation, s = 4 is usually used.

[0063] During the iteration process, the initial maximum number of iterations N_max = 10000 is set. Whenever a better model is found, the required number of iterations N is recalculated according to the above formula. The maximum number of iterations is updated to min(N, N_max) based on N. The process terminates early when the updated number of iterations is reached.

[0064] For example, suppose we randomly sample 3 times from 630 valid matching point pairs to obtain the initial optimal model. The number of inliers is 472, the total number of matching point pairs is 630, the inlier ratio is 472 / 630 ≈ 0.75, p = 0.99, and the minimum sample size is 4. Substituting the parameters into the formula, we get the numerator ln(0.01) ≈ 4.60517, in the denominator, 1-0.75 4 =1-0.31640625=0.68359375, and ln(0.68359375)≈ 0.37904. Finally, calculate N=( 4.60517)÷( 0.37904)≈12.15, rounded up to 13. Further, after 6 iterations, a better model is obtained, with the number of interior points increasing to 517, i.e., the interior point ratio = 517 / 630≈0.82; subsequent iterations yield N=8. Since 6 iterations have already been performed, only 8 iterations remain. 6 = 2 times, so only 2 iterations are needed to satisfy the requirement of p.

[0065] It should be noted that the above parameters are for illustrative purposes only and are not intended to limit this application.

[0066] This embodiment dynamically calculates the required number of iterations by using the proportion of interior points of the homography matrix sampled during the matching point screening process of the feature matching matrix. At the same time, the number of iterations is updated in real time during the iteration to avoid invalid iterations, thereby significantly reducing computing power consumption and ensuring real-time response of the splicing process.

[0067] Furthermore, in this embodiment, under the conditions of p=0.99 and s=4, the approximate number of iterations corresponding to different interior point ratios is calculated through actual experiments as shown in the table below:

[0068] Therefore, when there is sufficient experimental data, the number of iterations can also be determined by looking up a table.

[0069] Step S70: Based on the target number of iterations and the random sampling consensus algorithm, determine the second homography matrix of valid matching point pairs; In this embodiment, after determining the target number of iterations, the RANSAC algorithm iteratively calculates the effective matching point pairs based on the corresponding number of iterations to obtain the second homography matrix. It can be understood that the second homography matrix has the largest number of interior points, meaning its projection transformation model has the largest number of interior points.

[0070] Step S80: Perform weighted least squares processing on all interior points of the second homography matrix to obtain the target homography matrix.

[0071] In this embodiment, after obtaining the homography matrix of the effective matching point pairs, it is necessary to perform least squares calculations on all interior points. However, when traditional RANSAC estimates the matrix using least squares, it treats all interior points equally, which can cause edge interior points to interfere with the matrix fitting accuracy.

[0072] Therefore, in this embodiment, by calculating the reprojection error and matching confidence of the interior points, different weights are assigned to the interior points. Weighted least squares processing is then performed on all interior points based on these different weights, finally obtaining the target homography matrix corresponding to the effective matching point pairs. The matching confidence of the interior points can be read from the matching algorithm output.

[0073] When determining the weights of interior points based on reprojection errors, the source points need to be transformed using the currently estimated homography matrix H. The formula for calculating the distance between the transformed position and the actual target point position is as follows: .

[0074] in,( , ( ) represents the actual target point coordinates, , The coordinates are obtained after transforming the source point using H. ei is the reprojection error, in pixels.

[0075] Furthermore, the final weight of each interior point is determined by the following formula: .

[0076] Among them, w i c is the weight of the i-th interior point; i Let g(e) be the matching confidence of the i-th interior point. i） The conversion formula for the weighting factor based on reprojection error is as follows: .

[0077] For example, suppose there are 3 inliers after RANSAC, and their matching confidence and reprojection error are as follows:

[0078] It can be seen that point A has high matching quality, small error, and the highest weight (0.95). Point B has low matching quality; although the error is small, its weight is low (0.35). Point C has relatively high matching quality, but a larger error, and its weight is reduced (0.48). By setting different weights for the inliers, the error of high-weight points is given more attention when optimizing the homography matrix, while the error of low-weight points has a smaller impact. This results in a more accurate and robust estimation result, minimizing the sum of squared weighted projection errors of all inliers.

[0079] Furthermore, to aid in understanding the implementation process of the image stitching method in this embodiment, please refer to... Figure 5 , Figure 5 The process of calculating the target homography matrix is ​​illustrated. Specifically, after obtaining high-quality matching pairs, i.e., effective matching point pairs, RANSAC iterative processing is performed. During the loop, the homography matrix H is calculated by randomly sampling 4 pairs of points, and the reprojection error of all points is calculated. Then, the number of inliers with an error less than a preset threshold is counted. Subsequently, the optimal model is updated based on the proportion of inliers, and the iteration number is updated using the optimal model. Finally, the weight value of each inlier is determined based on the reprojection error, and weighted least squares optimization is performed on all inliers according to the weight values ​​to obtain the target homography matrix.

[0080] This embodiment provides an image stitching method that estimates the homography matrix using an improved RANSAC algorithm. It employs an algorithm with an adaptive iteration count, eliminating the fixed iteration count of traditional algorithms and reducing unnecessary computation. During model optimization, adaptive weights are assigned based on the reprojection error of each inlier to suppress interference from noisy or edge inliers on model parameter estimation. Subsequently, least-squares fitting optimization is performed on all valid inliers based on these weights to further refine the homography matrix parameters. This ensures the estimated homography matrix more closely reflects the true transformation relationships between images, providing higher fitting accuracy for subsequent projection alignment steps. This guarantees that images from different perspectives can be accurately aligned to the same coordinate system, achieving a high-quality stitching effect with no obvious traces, natural detail transitions, and overall visual coherence.

[0081] Based on any of the above embodiments, in the third embodiment of this application, the same or similar content as the above embodiments can be referred to the above description, and will not be repeated hereafter. On this basis, when performing feature point detection, the number of feature points detected is controlled by a detection threshold. Different detection thresholds correspond to different numbers of feature points, therefore it is necessary to ensure that there are enough feature points and avoid excessive redundant feature points.

[0082] Therefore, after step S10, it is also necessary to determine the number of feature points in the images to be stitched, that is, to determine the total number of feature points in all images to be stitched. If the number of feature points is less than the first preset number or greater than the second preset number, the detection threshold on the overlapping side of the stitching needs to be adjusted based on the range corresponding to the target number and the current number of feature points. The target number can be dynamically set based on actual needs.

[0083] Specifically, an adaptive threshold strategy can be used to dynamically adjust the detection threshold on the overlapping side of the stitching. When the number of feature points is too small (e.g., <500), the detection threshold on the overlapping side is lowered to detect more feature points; if the number of feature points is too large (e.g., >5000), the detection threshold on the overlapping side is increased to reduce the number of feature points. This process is repeated iteratively until the number of feature points is within a reasonable range. When updating the detection threshold based on the interval corresponding to the target number, the relationship between the current number and the extreme value of the interval can be determined. For example, if the interval is 1000-3000 and the current number is 500, the threshold is increased by 500 / 1000 = 2 times, or by 0.2, etc. The specific adjustment size of the detection threshold is not limited in this application.

[0084] This embodiment provides an image stitching method. After obtaining the feature points of the image to be stitched, the detection threshold of the overlapping side of the image is dynamically adjusted based on the current number and the actual required number through adaptive adjustment. This ensures that there are enough feature points for matching when performing image stitching in the future, while avoiding too many redundant feature points that would reduce matching efficiency.

[0085] Based on any of the above embodiments, in the fourth embodiment of this application, the content that is the same as or similar to the above embodiments can be referred to the above description, and will not be repeated hereafter. In addition, after step S40, steps S90-120 are also included: Step S90: Determine the number of interior points in the target homography matrix, the average reprojection error of the interior points, and the height difference of the matching point pairs.

[0086] In this embodiment, after obtaining the stitched image, the stitching result needs to be quality evaluated to determine whether it meets the preset quality standards. The evaluation methods include at least one of the following: matching point count check, interior point ratio check, geometric consistency evaluation, and high consistency check. If any one of the evaluation indicators fails to meet the standard, the stitching is deemed to have failed.

[0087] Specifically, during the matching point count check, the number of interior points ultimately used for homography matrix estimation is counted. If the number is less than a minimum threshold, the stitching is deemed a failure. In the geometric consistency assessment, the average reprojection error of all interior points is calculated. If the error exceeds a threshold, the stitching quality is deemed poor. In the interior point ratio assessment, the interior point ratio is calculated as the number of interior points divided by the initial number of matching points. If the ratio is lower than a threshold, the scene is deemed unsuitable for stitching. In the height consistency check, the average height difference between matching point pairs is calculated. If the difference exceeds a certain proportion of the image height, the stitching is deemed a failure.

[0088] By conducting quality assessments on the images, we ensure that the stitched output image meets the actual requirements.

[0089] Optionally, before quality assessment, global color adjustments can be performed on the stitched images to address color inconsistencies between the left and right images. Specifically, color statistical analysis can be used to analyze the color distribution characteristics of the overlapping areas of the left and right images, including brightness, saturation, and hue. Subsequently, a progressive adjustment method is used to gradually apply color adjustments outside the overlapping areas to ensure a natural and unified color across the entire image.

[0090] Step S100: If the number of inner points is greater than the preset number, the proportion of inner points is greater than the preset proportion, the average reprojection error of the inner points is less than the preset error, and the height difference of the matching point pairs is less than the preset height, then the effective area of ​​the target stitched image is determined.

[0091] Step S120: Determine the target output ratio of the target stitched image based on the effective area, and crop the target stitched image according to the target output ratio to obtain the cropped image.

[0092] In this embodiment, when the quality assessment is passed (i.e., the number of inliers is greater than a preset number, the proportion of inliers is greater than a preset proportion, the average reprojection error of the inliers is less than a preset error, and the height difference of the matching point pairs is less than a preset height), the stitched image needs to be intelligently cropped to remove invalid areas at the edges.

[0093] Specifically, the process of cropping and repairing imperfections in an image is as follows: Figure 6As shown, after obtaining the stitched image, it is necessary to first identify the effective content area in the image. Through effective area detection, black border detection, distortion detection, and morphological detection, the effective area mask is obtained to exclude areas with black borders, black and white areas, and areas with severe distortion.

[0094] As an optional implementation, when performing black border detection, regions with pixel values ​​close to 0 are identified and marked as invalid regions. When performing distortion detection, the degree of distortion in the edge regions is detected, and severely distorted regions are marked as invalid. For valid region extraction, non-invalid regions are marked as valid, and morphological operations such as erosion and dilation can be used to smooth the boundaries.

[0095] After obtaining the effective region, the output ratio of the target stitched image is determined so that the target stitched image can be cropped according to the output ratio. That is, the largest rectangular region in the effective region is extracted as the cropping target by using a dynamic programming algorithm. First, the image is scanned from top to bottom. For each row, the largest rectangle with that row as the bottom edge is calculated and the global largest rectangle is recorded.

[0096] When performing cropping, a cropping threshold parameter can be set to control the aggressiveness of the cropping. A smaller threshold retains more content but may include a few imperfections, while a larger threshold ensures image quality but may lose some edge content. The cropping threshold is set based on actual needs. Therefore, the cropping threshold parameter θ (range 0-1) is set as follows: θ=0 allows maximum cropping and ensures no black borders; θ=1 does not crop and retains all content; an intermediate value of θ strikes a balance between content preservation and quality assurance.

[0097] Step S120: Determine the defective areas of the cropped image and repair the defective areas based on a preset algorithm to obtain the output image corresponding to the target stitched image.

[0098] In this embodiment, after cropping, it is necessary to repair any small imperfections that may remain after cropping. Therefore, an image recognition algorithm can be used to scan several rows / columns of the image edge to identify imperfection areas such as black borders and color blocks, thus detecting whether small imperfections still exist at the cropped image edge. Subsequently, based on a preset algorithm, such as a neighborhood-based repair algorithm, information from surrounding pixels is used to fill in the imperfection areas using an image repair algorithm, or a texture synthesis-based repair method is used to sample textures from the surrounding areas, synthesize them, and fill them into the imperfection areas, finally obtaining the cropped image.

[0099] This embodiment provides an image stitching method. After obtaining the stitched image, the visual effect and practical value of the stitched image are further optimized by performing quality inspection, global color restoration, image cropping, and edge repair. This effectively solves problems such as color imbalance, edge defects, and composition redundancy that may exist in the stitched image. Finally, a high-quality stitched image with high-precision alignment, natural color transition, complete composition, and delicate details is output to meet the needs of subsequent image use.

[0100] For example, to help understand the implementation process of the image stitching method obtained by combining the above embodiments, please refer to... Figure 7 , Figure 7 A simplified flowchart of an image stitching method is provided. Specifically, taking headphones as an example, after acquiring the left and right camera images of the headphones, special point detection is performed using a deep learning algorithm. During the detection process, a region adaptive strategy is used to adjust the detection threshold on the overlapping side of the stitching. Subsequently, feature point matching is performed based on depth, and an attention mechanism is used to process the data to obtain a feature matching matrix. Then, the matching points are further filtered based on confidence and geometric constraints. After filtering, an improved RANSAC algorithm is used to estimate the homography matrix of the effective matching point pairs. Then, the quality of the stitching points is evaluated, and those that are not up to standard are considered to have failed the stitching. Next, image projection transformation is performed based on the calculated homography matrix, and image fusion processing is performed based on multi-band fusion to obtain the stitched image. After obtaining the stitched image, global color adjustment processing is performed on the stitched image. Then, intelligent cropping processing is performed using the extracted maximum rectangle, and edge repair is performed on the cropped image. Finally, the stitched and quality-repaired image is output.

[0101] Furthermore, an example process for image stitching and restoration is as follows: Figure 8 and Figure 9 As shown, feature points on the right side of the left image and feature points on the left side of the right image are extracted respectively. Based on the features, image stitching and subsequent repair processing are performed to obtain the final stitched image.

[0102] This application provides an image stitching device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the image stitching method described in the first embodiment above.

[0103] The following is for reference. Figure 10This document illustrates a structural schematic diagram of an image stitching device suitable for implementing embodiments of this application. The image stitching device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), portable media players (PMPs), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 10 The image stitching device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0104] like Figure 10 As shown, the image stitching device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the image stitching device. The processing unit 1001, the ROM 1002, and the RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the image stitching device to communicate wirelessly or wiredly with other devices to exchange data. Although image stitching devices with various systems are shown in the figures, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.

[0105] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0106] The image stitching device provided in this application, employing the image stitching method described in the above embodiments, can solve the technical problem of poor image stitching effect without pre-calibration. Compared with the prior art, the beneficial effects of the image stitching device provided in this application are the same as those of the image stitching method provided in the above embodiments, and other technical features of this image stitching device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0107] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0108] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0109] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the image stitching method described in the above embodiments.

[0110] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM, or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, radio frequency (RF), etc., or any suitable combination thereof.

[0111] The aforementioned computer-readable storage medium may be included in the image stitching device; or it may exist independently and not be assembled into the image stitching device.

[0112] The aforementioned computer-readable storage medium carries one or more programs that, when executed by the image stitching device, cause the image stitching device to: Feature points of the image to be stitched are extracted based on the detection threshold corresponding to the overlapping side of the image to be stitched, wherein the detection threshold of the overlapping side is less than the detection threshold of the non-overlapping side. The feature matching matrix corresponding to the feature points of the image to be stitched is determined based on a bidirectional matching strategy. Based on the target homography matrix obtained by iterating through the effective matching point pairs of the feature matching matrix, the image projection action of the image to be stitched is performed. The projected images are fused based on a preset image fusion algorithm to obtain the target stitched image.

[0113] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0114] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0115] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0116] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described image stitching method, which can solve the technical problem of poor image stitching effect without pre-calibration. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the image stitching method provided in the above embodiments, and will not be repeated here.

[0117] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. An image stitching method, characterized in that, The image stitching method includes: Feature points of the image to be stitched are extracted based on the detection threshold corresponding to the overlapping side of the image to be stitched, wherein the detection threshold of the overlapping side is less than the detection threshold of the non-overlapping side. The feature matching matrix corresponding to the feature points of the image to be stitched is determined based on a bidirectional matching strategy. Based on the target homography matrix obtained by iterating through the effective matching point pairs of the feature matching matrix, the image projection action of the image to be stitched is performed. The projected images are fused based on a preset image fusion algorithm to obtain the target stitched image.

2. The image stitching method as described in claim 1, characterized in that, The images to be stitched include a first image and a second image. Before the step of extracting feature points from the images to be stitched based on a detection threshold corresponding to the overlapping side of the stitching, wherein the detection threshold for the overlapping side is less than the detection threshold for the non-overlapping side, the image stitching method further includes: Determine the relative positions between the first image and the second image to be stitched together; The overlapping side between the first image and the second image is determined based on their relative positions.

3. The image stitching method as described in claim 1, characterized in that, Before the step of performing the image projection action on the image to be stitched, based on the target homography matrix obtained by iterating through the effective matching point pairs of the feature matching matrix, the image stitching method further includes: The first homography matrix with the largest number of interior points is determined by a random sampling consistency algorithm based on a preset number of iterations. The target number of iterations for the random sampling consensus algorithm is determined based on the proportion of interior points in the first homography matrix. Based on the target number of iterations and the random sampling consistency algorithm, the second homography matrix of the effective matching point pairs is determined, wherein the second homography matrix has the largest number of interior points; The target homography matrix is ​​obtained by performing weighted least squares processing on all interior points of the second homography matrix.

4. The image stitching method as described in claim 3, characterized in that, The steps of the random sampling consensus algorithm based on a preset number of iterations to determine the first homography matrix with the largest number of interior points include: Based on the matching point pairs of the feature matching matrix and the preset number of iterations, the estimated homography matrix of the random minimum sample set is calculated according to the random sampling consensus algorithm; Determine the first homography matrix among the estimated homography matrices that has the largest number of interior points of the projection transformation model.

5. The image stitching method as described in claim 1, characterized in that, Before the step of performing the image projection action on the image to be stitched, based on the target homography matrix obtained by iterating through the effective matching point pairs of the feature matching matrix, the image stitching method further includes: Determine the geometric consistency verification results, confidence level, and / or vertical coordinate differences between the matching point pairs of the feature matching matrix; Delete the matching point pairs in the geometric consistency verification results that do not match the homography matrix with the most internal points, the matching point pairs with a confidence level less than the preset confidence level, and / or the matching point pairs with a vertical coordinate difference greater than the preset threshold, to obtain the effective matching point pairs; If the number of valid matching point pairs is greater than a preset number, the target homography matrix is ​​obtained by iterating through the valid matching point pairs of the feature matching matrix, and the image projection action of the image to be stitched is performed.

6. The image stitching method as described in claim 1, characterized in that, After the step of extracting feature points of the image to be stitched based on the detection threshold corresponding to the overlapping side of the image to be stitched, wherein the detection threshold of the overlapping side is less than the detection threshold of the non-overlapping side, the image stitching method further includes: Determine the number of feature points in the image to be stitched; If the number of feature points is less than the first preset number, the detection threshold of the splicing overlap side is reduced based on the interval corresponding to the target number and the number of feature points. If the number of feature points is greater than the second preset number, the detection threshold of the splicing overlap side is increased based on the interval corresponding to the target number and the number of feature points.

7. The image stitching method as described in claim 1, characterized in that, The step of determining the feature matching matrix corresponding to the feature points of the image to be stitched based on a bidirectional matching strategy includes: Determine the first set of nodes of the first image to be stitched together, and the second set of nodes of the second image to be stitched together; Determine the first feature matching matrix from the feature points of the first node set to the feature points of the second node set; Determine the second feature matching matrix from the feature points of the second node set to the feature points of the first node set; The feature matching matrix is ​​output based on the similarity matching results between the first feature matching matrix and the second feature matching matrix.

8. The image stitching method as described in claim 1, characterized in that, After the step of fusing the projected images based on a preset image fusion algorithm to obtain the target stitched image, the image stitching method further includes: Determine the number of inliers, the proportion of inliers, the average reprojection error of inliers, and the height difference of matching point pairs in the target homography matrix; If the number of inliers is greater than a preset number, the proportion of inliers is greater than a preset proportion, the average reprojection error of the inliers is less than a preset error, and the height difference of the matching point pairs is less than a preset height, then the effective area of ​​the target stitched image is determined. The target output ratio of the target stitched image is determined based on the effective area, and the target stitched image is cropped according to the target output ratio to obtain a cropped image; The defective areas of the cropped image are identified, and the defective areas are repaired based on a preset algorithm to obtain the output image corresponding to the target stitched image.

9. An image stitching device, characterized in that, The image stitching device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the image stitching method as described in any one of claims 1 to 8.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the image stitching method as described in any one of claims 1 to 8.

Images