Image stitching methods, computer-readable storage media, and terminal devices thereof

Abstract

This invention relates to an image stitching method, its computer-readable storage medium, and a terminal device, comprising: acquiring at least two images to be stitched with overlapping regions, and determining a stitching matrix based on the images; transforming the overlapping regions of the images to be stitched to the same coordinate system based on the stitching matrix; calculating the similarity of each transformed overlapping region; continuously adjusting the stitching matrix based on the similarity of the overlapping regions, repeating steps S2 and S3, and selecting the stitching matrix corresponding to the highest similarity as the optimal stitching matrix; stitching the images to be stitched based on the optimal stitching matrix to obtain a stitched image. The key lies in introducing an evaluation mechanism to calculate the similarity of the overlapping regions of the transformed images to be stitched, reflecting the overall effect of the stitching under the given stitching matrix, and deriving the optimal stitching matrix with the highest similarity, thereby fundamentally optimizing and updating the stitching matrix—the source factor affecting the stitching effect—to achieve the best stitching result.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image stitching method. Background Technology

[0002] Image stitching is a technique that combines a set of images according to certain rules to create a large-field-of-view, high-resolution image. It is a crucial research area in image processing, with wide applications in industry, medicine, photography, and digital image processing. For example, in industry, image stitching is a prerequisite technology for machine vision operations such as size measurement, shape matching, and defect detection of large objects.

[0003] In existing technologies, the most commonly used image stitching algorithms are feature-matching methods based on feature points, such as SIFT, SURF, and ORB. These methods extract feature points from two images to be stitched, obtaining a stitching matrix (e.g., homography) to complete the image stitching. However, due to stitching errors and accuracy issues, defects such as stitching seams and ghosting may exist. To improve image stitching quality, existing technologies typically involve subsequent updates and fusion of the stitching seams.

[0004] However, in many fields such as industry and medicine, the requirements for splicing accuracy are even higher. Taking large industrial scenarios as an example, numerous workpieces and packages are scattered on conveyor belts. If the splicing accuracy is not high, it may lead to workpiece grabbing failure, coding errors, and incomplete display in large scenarios. Therefore, how to further improve splicing accuracy is a technical problem that urgently needs to be solved in the splicing field. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides an adaptive image stitching method, comprising:

[0006] S1: Obtain at least two images to be stitched that have overlapping regions, and determine the stitching matrix based on the images to be stitched;

[0007] S2: Based on the splicing matrix, transform the overlapping areas of the images to be spliced ​​to the same coordinate system;

[0008] S3: Calculate the similarity of each transformed overlapping region;

[0009] S4: Based on the similarity of the overlapping regions, continuously adjust the splicing matrix, repeat steps S2 and S3, and select the splicing matrix corresponding to the highest similarity as the optimal splicing matrix;

[0010] S5: Based on the optimal stitching matrix, stitch the images to obtain the stitched image.

[0011] Further, in step S1, at least two images to be stitched together with overlapping regions are obtained, specifically by directly obtaining the images to be stitched together; or including:

[0012] S11: Place a calibration plate at the position to be stitched and obtain the image to be stitched with the same calibration plate;

[0013] S12: Determine the splicing matrix by the mapping relationship of feature points in the calibration board;

[0014] S13: Remove the calibration plate and take a picture of the image to be stitched after removing the calibration plate.

[0015] Further, step S3 includes:

[0016] S31: Select the feature region of each transformed overlapping region;

[0017] S32: Calculate the similarity of the feature regions of each transformed overlapping region, and calculate the similarity of each transformed overlapping region.

[0018] Furthermore, prior to step S31, the following steps are also included:

[0019] S30: For each converted overlapping region, perform binarization to obtain the preprocessed overlapping region. Further, calculate the similarity using formula (2);

[0020] SSIM (x,y) =[I (x,y) ] α [C (x,y) ] β [S (x,y) ] γ

[0021]

[0022]

[0023]

[0024] Among them, I (x,y) For brightness similarity; C (x,y) For contrast similarity; S (x,y) α represents structural similarity; β, γ are weighting coefficients, all non-zero constants; u x u y Let σ represent the mean values ​​of the x-th and y-th images, respectively; x , σ y Let σ represent the variances of the x-th and y-th images, respectively; xy Let C1, C2, and C3 represent the covariance between the x-th and y-th images; C1, C2, and C3 are constants.

[0025] Furthermore, step S4, which involves continuously adjusting the splicing matrix, includes:

[0026] S41: Determine the adjustment range of the stitching matrix based on the image to be stitched and / or each overlapping area after conversion;

[0027] S42: Within the range of splicing matrix adjustment, continuously adjust the splicing matrix.

[0028] Furthermore, step S42 also includes:

[0029] S421: Gradually narrow the adjustment range of the splicing matrix based on the trend of similarity changes;

[0030] or / and,

[0031] S422: Adjust the parameters of the splicing matrix individually.

[0032] Furthermore, it also includes:

[0033] S6: Perform fusion processing on the seams of the stitched images.

[0034] On the other hand, the present invention also provides a computer-readable storage medium storing computer-executable program code; the computer-executable program code is used to execute any of the above-described image stitching methods.

[0035] On the other hand, the present invention also provides a terminal device, including a memory and a processor; the memory stores program code that can be executed by the processor; the program code is used to execute any of the above-described image stitching methods.

[0036] The image stitching method, computer-readable storage medium, and terminal device provided by this invention break with conventional methods. Instead of simply updating and merging the seams of the stitched images, it introduces an evaluation mechanism to calculate the similarity of overlapping areas in the converted images to be stitched. This reflects the overall effect of the stitching under the given stitching matrix, allowing for reverse derivation of the optimal stitching matrix with the highest similarity. This fundamentally optimizes the stitching matrix, the source factor affecting the stitching effect, resulting in the optimal stitched image under the optimal stitching matrix and achieving the best stitching effect. Attached Figure Description

[0037] Figure 1 A flowchart of one embodiment of the image stitching method of the present invention;

[0038] Figure 2 An example diagram of one embodiment of the images to be stitched together;

[0039] Figure 3 Example diagram of another embodiment of the images to be stitched;

[0040] Figure 4 for Figure 3 The example image to be stitched is shown after the stitching matrix transformation;

[0041] Figure 5 for Figure 3 An example illustration of the stitching effect of the images to be stitched under the unoptimized and updated stitching matrix;

[0042] Figure 6 for Figure 3 A schematic diagram of the overlapping areas of the images to be stitched after the stitching matrix transformation;

[0043] Figure 7 for Figure 3 A schematic diagram of the feature regions of the overlapping areas in the example images to be stitched after the stitching matrix transformation;

[0044] Figure 8 for Figure 3 A schematic diagram illustrating the stitching effect of the example images to be stitched under the optimized and updated stitching matrix;

[0045] Figure 9 for Figure 3 This is a schematic diagram illustrating the stitching effect after the seams of the images to be stitched together. Detailed Implementation

[0046] like Figure 1 As shown, an adaptive image stitching method according to the present invention is given, comprising:

[0047] S1: Acquire at least two images with overlapping areas to be stitched, and determine the stitching matrix based on the images. Specifically, the images to be stitched can be, but are not limited to, images taken by camera or other image acquisition devices at different positions and angles, as long as they have overlapping areas. The number, form, and content of these images can be arbitrarily set according to actual needs, such as... Figure 2 , 3 As shown, an example of images to be stitched is provided; however, the text and illustrations in this instruction manual are merely illustrative and not intended to be limiting. The above... Figure 2 , Figure 3 This example only provides an example with two images to be stitched. In real-world applications, there may be more images to be stitched. More specifically, the images to be stitched can be denoted as image 1, image 2, ..., image n, where n represents the number of images to be stitched. More specifically, the stitching matrix can be denoted as H = (H... 21 H 32 ...H (n)(n-1) ), where H (n)(n-1)This represents the transformation matrix from the nth image to the (n-1)th image. Of course, those skilled in the art will understand that, among the n images, the stitching matrix can be calculated using, but is not limited to, the first image as the reference image. It can also be calculated using any image as the reference image, or even a standard image, i.e., calculating the stitching matrix H in a standard coordinate system. For example, such as... Figure 2 As shown, the images to be stitched consist of two images, denoted as image 1 and image 2. Image 1 (left image) can be used as the reference image, but is not limited to it. The mapping relationship is found by comparing the pixel coordinates of the feature points in the overlapping region of the left image with the pixel coordinates of the corresponding feature points in the right image, thus obtaining the stitching matrix H = (H... 21 This involves using the transformation matrix to convert the pixel coordinates of the feature points in the right image to the coordinate system of the left image; alternatively, but not limited to, using the second image on the right as the reference image, the stitching matrix H = (H... 12 This involves transforming the pixel coordinates of feature points in the left image to the coordinate system of the right image using a transformation matrix. Alternatively, but not limited to, using a standard image as the reference image, denoted as image 0, the stitching matrix H = (H... 10 H 20 ), which is the transformation matrix that converts the pixel coordinates of feature points in the left and right images to the standard coordinate system.

[0048] More specifically, in addition to directly acquiring the image to be stitched through the aforementioned methods of shooting and receiving, the following considerations are also taken into account: In industrial settings, since workpieces typically lack distinct texture features, making it difficult to find representative feature points, step S1 may optionally, but is not limited to, including:

[0049] S11: Place a calibration plate at the position to be stitched and obtain images to be stitched with the same calibration plate; specifically, it is optional but not limited to first placing a calibration plate at the position to be stitched, and then taking images to be stitched with the same calibration plate at multiple positions or angles, as shown in the left and right images in 3.

[0050] S12: Determine the splicing matrix through the mapping relationship of feature points in the calibration board; specifically, the step of calculating the splicing matrix can be selected, but is not limited to, any method of calibration in the prior art. The step of calculating the splicing matrix is ​​not the inventive point of this invention and will not be described in detail here.

[0051] S13: Remove the calibration plate and capture the image to be stitched after removing the calibration plate. Specifically, optional but not limited to removing the calibration plate, you can capture the image to be stitched at the original location using an image acquisition device such as a camera.

[0052] In this embodiment, a step of adding a calibration board is given to deal with situations where the texture of the image to be stitched in an industrial setting is unclear and the extraction of feature points is relatively difficult. This step can more conveniently, quickly, efficiently and accurately determine the stitching matrix when feature point extraction is difficult. This step is not necessary and is only a preferred embodiment.

[0053] More specifically, the concatenated matrix, which can be, but is not limited to, a homology matrix (Homography), can be, but is not limited to, represented as:

[0054]

[0055] Among them, A 2×2 For rotation and scaling parameters; T 2×1 V is the translation parameter; T is the perspective parameter; s is a constant, usually taken as 1.

[0056] Specifically, in existing technologies, after obtaining the stitching matrix H, the images to be stitched can be transformed to the same coordinate system based on the stitching matrix to complete the stitching work. However, with... Figure 3 Taking the two images to be stitched as an example, under the transformation of the stitching matrix, Figure 3 The left image can be converted to Figure 4 The upper half of the left image; Figure 3 The image on the right can be converted to Figure 4 The lower half of the image on the right; pieced together Figure 5 The composite image shown. Those skilled in the art will observe... Figure 5 It can be observed that there are seams and ghosting. Most current algorithms eliminate these by fusing overlapping areas, such as using deep learning, multi-band fusion, or optimizing the seam location. This involves updating the seam position through fusion processing to optimize the splicing effect. The key to this invention lies in breaking with this conventional approach by updating and adjusting the splicing matrix itself, specifically through the following processing:

[0057] S2: Based on the stitching matrix, transform the overlapping regions of the images to be stitched to the same coordinate system; specifically, the following example uses... Figure 3-4 Using an industrial scenario as an example for explanation, the left image can be used as a reference image and remains unchanged; the right image is the image to be transformed. Through the splicing matrix H, the right image is transformed to the reference coordinate system of the left image, resulting in the image shown below. Figure 6 The transformed overlapping regions shown can be achieved using any form of calibration method in the prior art, which will not be elaborated here. Of course, as described in step S1, if there are multiple images to be stitched, the multiple images to be stitched are sequentially transformed to the same coordinate system through a transformation matrix, and the overlapping regions corresponding to each image to be stitched are obtained to obtain multiple transformed overlapping regions.

[0058] S3: Calculate the similarity S of each transformed overlapping region; specifically, such as... Figure 6 As shown, it is optional, but not limited to, calculating the similarity S between the overlapping regions in the left and right images. 12 More specifically, when the images to be stitched include not only Figure 3 When two such cases are shown, the similarity of multiple transformed overlapping regions can be calculated pairwise, such as S. 12 S 23 ...and then calculate the pairwise transformation matrix, which can be, but is not limited to, H = (H 21 H 32 ...H (n)(n-1) ).

[0059] More specifically, in certain industrial scenarios, such as Figure 6 As shown, when conveying workpieces on a conveyor belt, the workpieces are generally located in the middle of the conveyor belt. Therefore, the focus of technicians is generally on whether the middle of the conveyor belt has high similarity, that is, whether excellent splicing effect can be achieved, while whether the side areas of the conveyor belt have high similarity is less important and less of a concern. Therefore, step S3 may also optionally include, but is not limited to, including:

[0060] Specifically, you may, but are not limited to, marking the areas of interest in the scene within overlapping areas, such as areas with many workpieces, or areas where key landmarks or people are located in the image, to extract these areas as feature regions; more specifically, you may, but are not limited to, using windows such as squares or circles to extract these feature regions.

[0061] S32: Calculate the similarity of the feature regions of each transformed overlapping region; specifically, replace the overall similarity of the overlapping region with the local similarity of the feature region, and replace the complex and computationally intensive overall region with a local region that has obvious features and high attention.

[0062] In this embodiment, step S3 adds a step of selecting feature regions. Only the similarity of the feature regions of the overlapping regions is calculated, which replaces the similarity of the entire overlapping region. This avoids calculating the entire overlapping region, greatly reducing the range of the region to be calculated, i.e. the number of pixels to be compared. This greatly reduces the workload of calculation and can further improve the comparison efficiency and shorten the stitching time.

[0063] More specifically, this similarity can be determined, but is not limited to, by evaluation metrics related to similarity. For example, the SSIM value can be used, but is not limited to, as a similarity evaluation metric. Specifically, with... Figure 6 The example has two overlapping regions or Figure 7Taking the feature regions of two overlapping areas as an example, with the left image being the similarity image to be compared for the first image and the right image being the similarity image to be compared for the second image, the similarity between the two can be calculated using formula (1), but is not limited to:

[0064] SSIM (1,2) =[I (1,2) ] α [C (1,2) ] β [s (1,2) ] γ (1)

[0065] More specifically, if the images to be stitched include multiple images, the similarity can be calculated pairwise for each of the overlapping regions or feature regions of the overlapping regions after transformation. Specifically, the similarity can be calculated using formula (2) as follows:

[0066] SSIM (x,y) =[I (x,y) ] α [C (x,y) ] β [S (x,y) ] γ

[0067] in:

[0068]

[0069]

[0070]

[0071] Among them, I (X,y) For brightness similarity, the brightness of two images is compared using the pixel mean; C (x,y) For contrast similarity, the contrast of two images is compared using pixel variance; S (x,y) For structural similarity, the relationship between the structure and pixels of two images is compared by combining covariance; α, β, and γ are weighting coefficients, all non-zero constants, used to adjust I. (x,y) C (x,y) S (x,y) The relative importance; u x u y Let σ represent the mean values ​​of the x-th and y-th images, respectively; x , σ y Let σ represent the variances of the x-th and y-th images, respectively; xy Let Cx represent the covariance between the x-th and y-th images; C1, C2, and C3 are constants to ensure that Ix = ... (x,y) C (x,y) S (x,y)The stability of.

[0072] More specifically, optional but not limited to, by determining the constants in formulas (1) and (2), the range of SSIM is ensured to be within the interval [0,1]. The larger the SSIM value, the higher the similarity. In practical scenarios, the images to be stitched are used to obtain the stitching overlap area through the stitching matrix. If SSIM = 1, the overlap area is exactly the same, and the best stitching effect has been achieved. If SSIM ≠ 1, or even has a certain gap from 1, the method of this invention is required to continuously adjust the stitching matrix so that SSIM is infinitely close to 1, thus completing the fine-tuning process of the stitching matrix.

[0073] This embodiment provides a method for evaluating the similarity of each transformed overlapping region, using SSIM as the evaluation index and providing a specific formula for calculating the similarity. This allows for a quantitative comparison of the similarity of each overlapping region. However, using SSIM as the evaluation index is merely illustrative and not a limitation. Those skilled in the art will understand that other evaluation indices for similarity measurement may include, but are not limited to, Peak Signal-to-Noise Ratio (PSNR), Discrete Cosine Transform (pHash), etc.

[0074] More specifically, to reduce interference caused by different lighting conditions, when calculating the similarity of overlapping regions or feature regions of overlapping regions, optional preprocessing may be included, but is not limited to:

[0075] S30: For each transformed overlapping region or each feature region of the transformed overlapping region, perform binarization processing to obtain the preprocessed overlapping region or the preprocessed feature region. Specifically, taking... Figure 6 Taking the two overlapping regions shown as an example, you can optionally, but not be limited to, binarize the two overlapping regions using their pixel mean values. Pixels with pixel values ​​greater than the mean are set to 255, and pixels with values ​​less than the mean are set to 0. In this way, the two regions in the binarized image are either black or white. Based on this binarization process, the computational cost of similarity calculation can be further reduced, comparison time can be shortened, and stitching efficiency can be improved. More importantly, by superimposing the two binarized overlapping regions, it is clear that the black and white areas are regions with the same pixels in both images, while the gray areas are regions with different pixels in both images. For example, using... Figure 6 For example, based on step S3, the SSIM value of the two can be calculated to be 0.87, which is still some distance from 1. The splicing matrix H needs further improvement, as detailed below:

[0076] S4: Based on the similarity of the overlapping regions, continuously adjust the stitching matrix, repeat steps S2-S3, and select the stitching matrix corresponding to the highest similarity as the optimal stitching matrix. Specifically, this step of continuously adjusting the stitching matrix can be, but is not limited to, exhaustive enumeration, that is, exhaustively enumerating every possible combination of stitching matrices, repeating steps S2-S3, and based on each stitching matrix, transforming the overlapping regions of the images to be stitched to the same coordinate system, and calculating the similarity of each transformed overlapping region, thereby determining which stitching matrix yields the highest similarity; then, the stitching matrix corresponding to the highest similarity is selected as the optimal stitching matrix. In the experiment, this invention adjusted the similarity under the corresponding stitching matrix from 0.82 to 0.85, 0.87, and 0.92 sequentially to achieve the highest similarity.

[0077] More specifically, because exhaustive search has too many possibilities and is too slow, step S4 involves continuously adjusting the splicing matrix. This step can include, but is not limited to, the following:

[0078] S41: Determine the adjustment range of the stitching matrix based on the images to be stitched and / or each converted overlapping area. Specifically, the adjustment range can be arbitrarily set based on parameters such as the type of the images to be stitched (e.g., images of people, workpieces, interior scenes, exterior scenes, etc.), the application environment, and the purpose of stitching. For example, since the stitching matrix determined in step S1 generally meets certain stitching requirements and only requires fine-tuning, the adjustment range can be set as follows: rotation parameter ±2°; scaling parameter ±0.02; translation parameter ±1%, etc. More specifically, the adjustment range of the stitching matrix can also be related to the size and thickness of the workpiece. For example, taking the stitching matrix determined by the calibration plate in S1 as an example, this stitching matrix is ​​calculated for a certain type of workpiece. If the workpiece type is changed, such as changing to a workpiece of a different size and thickness, it will inevitably affect the stitching matrix, and the adjustment range of the stitching matrix needs to be adjusted accordingly. More specifically, the adjustment range of the stitching matrix can also be selected, but is not limited to, images related to overlapping regions. For example, with Figure 5 As can be seen from the example, under the action of the initial stitching matrix, the image below needs to be shifted to the left. Therefore, the amount of translation in the opposite direction of the X-axis is determined to be negative. It can be directly set to a negative number to determine the adjustment range of the stitching matrix.

[0079] S42: Within the adjustment range of the stitching matrix, continuously adjust the stitching matrix. Of course, at this point, within the adjustment range of the stitching matrix, we can still exhaustively enumerate every possible combination of the stitching matrix and continue to repeat steps S2-S3. Based on each stitching matrix, transform the overlapping regions of the images to be stitched to the same coordinate system, and calculate the similarity of each transformed overlapping region. From this, determine which stitching matrix yields the highest similarity; then, the stitching matrix corresponding to the highest similarity is the optimal stitching matrix.

[0080] In this embodiment, a preferred embodiment of continuously adjusting the stitching matrix in step S4 is given. The adjustment range of the stitching matrix is ​​determined according to the image to be stitched and / or each overlapping area after conversion. The adjustment area of ​​the stitching matrix is ​​limited to a range, which reduces the amount of computation of exhaustive search, and can further reduce the computational difficulty, shorten the computation time, and improve the stitching efficiency.

[0081] More specifically, in step S42, while continuously adjusting the splicing matrix within its adjustment range, some adjustment techniques can be employed to further reduce the computational load and time of exhaustive search. For example, step S42 may optionally include, but is not limited to, the following:

[0082] S421: Based on the trend of similarity changes, gradually narrow the adjustment range of the splicing matrix. Specifically, taking the adjustment of the translation parameter as an example, it is possible, but not limited to, moving in the first direction, such as the positive or negative direction, by adjusting the X-axis translation of the splicing matrix by a step size, such as 10 pixels. This can be done, but is not limited to, moving 10 pixels to the right in the positive direction as shown in the example, to obtain the current splicing matrix (in the example, the translation parameter T). 2×1 t in x Adjusted to t x +10); After steps S2 and S3, the similarity of the current stitched matrix is ​​calculated; it is determined whether the similarity of the current stitched matrix is ​​better than the similarity of the previous stitched matrix; if so, continue to move 10 pixels to the right in the positive direction... Repeat the operation until the current stitched matrix (in the example, the translation parameter T) is obtained. 2×1 t in x Adjusted to t x The similarity under the +w×10 (where w is the number of iterations) is worse than the similarity under the previous concatenated matrix, meaning that the similarity in the translation parameter t is lower. x The rightward shift in the positive direction has exceeded the ideal threshold. At this point, it's necessary to reverse the shift, moving back a small amount (in the example, 5 pixels, with a shift parameter T) in the negative X-axis direction with a second step size. 2×1 t in x Adjusted to t x +w×10 -5), then judge again whether the similarity under the current splicing matrix is ​​better than the similarity under the previous splicing matrix, until the optimal splicing matrix with the highest similarity is found.

[0083] In addition to the above, step S42 may optionally, but is not limited to, employ other adjustment techniques to further reduce the computational load and time of exhaustive search. Again, as an example, adjusting the stitching matrix H may optionally, but is not limited to, adjusting the rotation and size parameters, translation parameters, and perspective parameters of the stitching matrix H sequentially. For example, first adjust the translation parameters, keeping the rotation and size parameters and perspective parameters unchanged, until the translation parameters are adjusted to their optimal values; then fix the translation parameters at this optimal value, keeping the perspective parameters unchanged, and adjust the rotation and size parameters until they are adjusted to their optimal values; finally, fix the translation, rotation, and size parameters at their respective optimal values, and adjust the perspective parameters; until all parameters are adjusted to their optimal values, the optimal stitching matrix is ​​obtained.

[0084] This embodiment presents some techniques for adjusting the splicing matrix. Whether it is to continuously narrow the adjustment range of the splicing matrix based on the changing trend of similarity, or to adjust and optimize each parameter of the splicing matrix individually, these methods can further reduce the computational difficulty and workload, and improve the splicing effect.

[0085] S5: Based on the optimal stitching matrix, stitch the images to be stitched together to obtain the stitched image, such as... Figure 8 As shown. Comparison Figure 5 and Figure 8 It is evident that after fine-tuning the stitching matrix, the stitched image with the optimal stitching matrix exhibits less misalignment at the intersection of the two images, resulting in a superior stitching effect.

[0086] The adaptive image stitching method of this invention breaks with conventional methods and optimizes the stitching matrix adjustment, possessing at least the following advantages:

[0087] 1. Calculate the similarity of overlapping regions after transformation by the stitching matrix. This fully reflects the stitching effect of the images to be stitched under this stitching matrix. Select the stitching matrix corresponding to the highest similarity to obtain the image before stitching with the highest similarity. This improves the stitching accuracy of subsequent target images and minimizes the occurrence of stitching seams, ghosting, etc.

[0088] 2. Taking an industrial scenario as an example, using the feature points of a calibration board to determine the stitching matrix can further improve the accuracy of the stitching matrix. However, in existing technologies, if the calibration board is directly removed, the image with the calibration board... (Example) Figure 3 The stitching matrix determined by the model will definitely differ from the stitching matrix determined by the model without the calibration plate. If used directly, the stitching effect will be poor.

[0089] However, by employing the image stitching method of this invention, errors caused by the calibration plate itself can be reduced by fine-tuning the stitching matrix. Especially when the type of workpiece on the conveyor belt changes, existing technologies require recalibrating the calibration plate due to the thickness and size of the new workpiece. However, the image stitching method of this invention eliminates the need for recalibration. Fine-tuning the stitching matrix avoids inapplicability issues caused by workpiece thickness and size, significantly reducing the steps of photographing the calibration plate and recalculating the stitching matrix, thus greatly reducing waste of manpower, resources, and time.

[0090] 3. In actual industrial scenarios, image acquisition devices such as cameras may experience slight movements in position and angle due to environmental factors, human factors, etc. In existing technologies, these slight movements of image acquisition devices will seriously affect the stitching effect. Therefore, in existing technologies, production lines need to be recalibrated and the stitching matrix needs to be recalculated after a period of operation. This not only consumes a lot of manpower and resources but also requires downtime for coordination, which is equivalent to a maintenance process. However, the image stitching method of this invention can also solve this problem by fine-tuning the stitching matrix.

[0091] In summary, the image stitching method of this invention breaks with conventional methods. Instead of simply updating and merging the seams of the stitched images, it introduces an evaluation mechanism to calculate the similarity of overlapping areas in the converted images to be stitched. This allows for a comprehensive understanding of the stitching effect under the given stitching matrix, and by working backward, the optimal stitching matrix with the highest similarity is derived. This yields the optimal stitched image under the optimal stitching matrix, achieving the best stitching effect.

[0092] Of course, after going through steps S1-S5, the resulting stitched image is as follows: Figure 8 As shown, due to factors such as lighting and camera parallax, a gap can still be seen. Taking SSIM as a similarity evaluation index as an example, under the optimal stitching matrix in step S4, the similarity SSIM cannot be completely equal to 1, but can only be infinitely close to 1.

[0093] Therefore, the splicing method of the present invention may optionally include, but is not limited to, specific steps, and may optionally include, but is not limited to, using multi-band fusion near the splicing seam to eliminate this splicing seam effect, achieving a more natural visual transition between the two images, resulting in, as... Figure 9 The stitched image shown.

[0094] On the other hand, the present invention also provides a computer-readable storage medium storing computer-executable program code; the computer-executable program code is used to execute any of the above-described image stitching methods.

[0095] On the other hand, the present invention also provides a terminal device, including a memory and a processor; the memory stores program code that can be executed by the processor; the program code is used to execute any of the above-described image stitching methods.

[0096] For example, the program code can be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the program code in the terminal device.

[0097] The terminal device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the terminal device may also include input / output devices, network access devices, buses, etc.

[0098] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0099] The memory can be an internal storage unit of the terminal device, such as a hard drive or RAM. The memory can also be an external storage device of the terminal device, such as a plug-in hard drive, SmartMedia Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory can include both internal and external storage units of the terminal device. The memory is used to store the program code and other programs and data required by the terminal device. The memory can also be used to temporarily store data that has been output or will be output.

[0100] The aforementioned computer-readable storage medium and terminal device are created based on the aforementioned image stitching method. The combination of their technical features and their technical effects will not be elaborated further here. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification. It is worth noting that the order of the reference numerals in the above methods, such as S1-S5, is merely illustrative and does not specifically limit the execution order. Those skilled in the art will understand that any change in order that does not violate the technical concept of this invention is within the protection scope of this invention.

[0101] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A method of adaptive adjustment of image stitching, characterized in that, The method comprises the following steps: S1: obtaining at least two to-be-stitched images with overlapping regions, and determining an initial stitching matrix according to the to-be-stitched images; The to-be-stitched images are derived from a workpiece detection scene of a conveying belt in an industrial scene; S2: converting the overlapping regions of the to-be-stitched images to the same coordinate system according to the current stitching matrix; S3: calculating the similarity of each converted overlapping region; S4: continuously adjusting the stitching matrix according to the similarity of the overlapping regions, repeating steps S2 and S3, and selecting the stitching matrix corresponding to the highest similarity as the optimal stitching matrix; the method comprises the following steps: determining the adjustment range of the stitching matrix according to the to-be-stitched images or / and each converted overlapping region; continuously adjusting the stitching matrix within the adjustment range of the stitching matrix; returning to steps S2 and S3 to calculate the similarity after each adjustment; gradually narrowing the adjustment range of the stitching matrix according to the change trend of the similarity; or / and adjusting the parameters of the stitching matrix one by one; selecting the stitching matrix corresponding to the highest similarity as the optimal stitching matrix; S5: stitching the to-be-stitched images according to the optimal stitching matrix to obtain a stitched image.

2. The image stitching method of claim 1, wherein, In step S1, the at least two to-be-stitched images with overlapping regions are obtained, specifically by directly obtaining the to-be-stitched images; or the method comprises the following steps: S11: placing a calibration board at a to-be-stitched position to obtain to-be-stitched images with the same calibration board; S12: determining the stitching matrix through the mapping relationship of the feature points in the calibration board; S13: removing the calibration board and shooting the to-be-stitched images after the calibration board is removed.

3. The image stitching method of claim 1, wherein, Step S3 comprises the following steps: S31: selecting a feature region of each converted overlapping region; S32: calculating the similarity of the feature region of each converted overlapping region, which is the similarity of each converted overlapping region.

4. The image stitching method of claim 3, wherein, Before step S31, the method further comprises the following step: S30: performing binaryzation processing on each converted overlapping region to obtain a pre-processed overlapping region.

5. The image stitching method of claim 1, wherein, The similarity is calculated by using formula (2); （2） wherein, is a luminance similarity; is a contrast similarity; is a structural similarity; are weight coefficients, all being non-zero constants; respectively represent the mean of the xth image and the yth image; respectively represent the variance of the xth image and the yth image; represents the covariance of the xth image and the yth image; is a constant.

6. The image stitching method of claim 1, wherein, The adjustment range of the stitching matrix is determined according to the to-be-stitched images or / and each converted overlapping region; specifically: The initial adjustment range of the stitching matrix is determined according to the size, thickness and image of the overlapping region of the workpiece in the to-be-stitched images; The adjustment range of the stitching matrix is gradually narrowed according to the change trend of the similarity; specifically: If the current similarity is better than the previous similarity, the current step is continued in the current direction; if the current similarity is worse than the previous similarity, the step is reduced and adjusted in the opposite direction.

7. The image stitching method of claim 1, wherein, The parameters of the stitching matrix are adjusted one by one, including: The rotation and size parameters, translation parameters and perspective parameters of the stitching matrix are sequentially adjusted.

8. The image stitching method according to any one of claims 1 to 7, characterized in that, The method further comprises the following step: S6: performing fusion processing on the stitching seam of the stitched image.

9. A computer-readable storage medium, characterized in that, The computer executable program code is stored in the memory; the computer executable program code is used to execute the image stitching method of any one of claims 1-8.

10. A terminal device, comprising: The memory and the processor are included; the memory stores the program code executable by the processor; the program code is used to execute the image stitching method of any one of claims 1-8.

Images