An image stitching method, device, equipment and medium
By combining the improved SIFT algorithm and bidirectional matching algorithm with a Generative Adversarial Network (GAN) generator, the problems of noise and computational complexity in image stitching in industrial environments are solved, achieving efficient and accurate image stitching. The generated stitched images are closer to real-world scenes in terms of visual effects and detection applicability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHENGDU AJIAXI INTELLIGENT TECH CO LTD
- Filing Date
- 2025-05-29
- Publication Date
- 2026-05-15
AI Technical Summary
In industrial environments, image stitching faces challenges such as noise and blurring affecting feature matching accuracy, high computational complexity, and difficulty in meeting real-time requirements.
Images are acquired using industrial cameras, and feature points are extracted using an improved SIFT algorithm and a bidirectional matching algorithm. Image stitching is performed using a Generative Adversarial Network (GAN) generator, and wavelet transform and weight fusion are combined to eliminate ambiguous matches and improve matching accuracy and efficiency.
It improves the accuracy and efficiency of image stitching, generates stitched images that closely resemble real-world scenes, reduces stitching artifacts, lowers computational complexity, and meets industrial inspection needs.
Smart Images
Figure CN120807271B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of industrial production and testing technology, and provides an image stitching method, apparatus, equipment and medium. Background Technology
[0002] Currently, in the field of industrial production and inspection, in order to conduct comprehensive and high-precision monitoring of large workpieces or production scenarios, it is often necessary to stitch together multiple images captured by industrial cameras to obtain a complete large-size image. However, image stitching in industrial environments faces several challenges.
[0003] On the one hand, industrial sites present complex and variable lighting conditions, such as direct sunlight and shadows. Simultaneously, factors like mechanical vibrations and electromagnetic interference generated during production can lead to noise and blurriness in images captured by industrial cameras, significantly impacting the accuracy of image feature matching and increasing stitching errors. This results in stitched images prone to misalignment and blurring, severely affecting subsequent analysis and judgment. On the other hand, industrial production typically demands real-time and efficient inspection processes. Traditional image stitching algorithms have high computational complexity, consuming significant computing resources and time when processing large amounts of image data acquired by industrial cameras, making it difficult to meet the real-time requirements of industrial production.
[0004] Therefore, it is urgent to develop a method that can adapt to industrial camera application scenarios, effectively improve stitching accuracy, reduce computational complexity, and enable rapid image stitching. Summary of the Invention
[0005] This application provides an image stitching method, apparatus, device, and medium to solve the technical problems of low stitching accuracy and efficiency and high computational complexity in existing technologies.
[0006] On the one hand, an image stitching method is provided, the method comprising:
[0007] An industrial camera is used to acquire images of the target part, resulting in multiple captured images;
[0008] Feature extraction is performed on each of the multiple acquired images to obtain the feature point set corresponding to each of the multiple acquired images;
[0009] A bidirectional matching algorithm is used to match feature points in the feature point sets corresponding to each of the multiple acquired images to determine multiple matching point pairs.
[0010] Based on the multiple matching point pairs, image transformation is performed on the multiple acquired images to obtain multiple transformed images;
[0011] The multiple transformed images are input into a trained generator for image stitching to obtain the target stitched image; wherein, the generator is based on a generative adversarial network (GAN).
[0012] Optionally, the step of extracting features from the multiple acquired images to obtain feature point sets corresponding to each of the multiple acquired images includes:
[0013] An improved SIFT algorithm is used to extract features from the multiple acquired images to obtain the feature point sets corresponding to each of the multiple acquired images; wherein, the initial scale factor of the improved SIFT algorithm is 1.5, and the number of scale space layers for each group of acquired images is 2.
[0014] Optionally, the step of using a bidirectional matching algorithm to perform feature point matching on the feature point sets corresponding to each of the multiple acquired images to determine multiple matching point pairs includes:
[0015] Within the contour range of the target industrial part, for any two acquired images, a bidirectional matching method and the Random Sample Consensus (RANSAC) algorithm are used to sequentially match the corresponding feature point sets of the two acquired images to determine multiple matching point pairs corresponding to the two acquired images.
[0016] Optionally, the step of performing image transformation on the multiple acquired images based on the multiple matching point pairs to obtain multiple transformed images includes:
[0017] Distortion correction and affine transformation are sequentially performed on multiple matching point pairs corresponding to any two acquired images to obtain two transformed images corresponding to any two acquired images.
[0018] Optionally, the step of inputting the multiple transformed images into a trained generator for image stitching to obtain the target stitched image includes:
[0019] For any transformed image, the transformed image is decomposed into three layers of wavelet coefficients;
[0020] Determine the weights corresponding to the three wavelet coefficients respectively;
[0021] The three-layer wavelet coefficients and their respective weights are weighted and fused to obtain the fused wavelet coefficients.
[0022] Perform inverse wavelet transform on the fused wavelet coefficients to obtain the reconstructed image corresponding to any one of the transformed images;
[0023] The reconstructed images corresponding to each of the multiple transformed images are input into a trained generator for image stitching to obtain the target stitched image.
[0024] Optionally, the step of determining the weights corresponding to each of the three wavelet coefficients includes:
[0025] For the low-frequency wavelet coefficients, the weights of the low-frequency wavelet coefficients are determined based on the pixel gray-level variance of the target part region obtained from the regional feature energy.
[0026] For the high-frequency wavelet coefficients, the weights of the high-frequency wavelet coefficients are determined according to the edge gradient direction;
[0027] For the wavelet coefficients of the intermediate frequency layer, the weights of the wavelet coefficients of the intermediate frequency layer are determined based on the weights of the wavelet coefficients of the low frequency layer and the wavelet coefficients of the high frequency layer.
[0028] Optionally, the step of inputting the reconstructed images corresponding to each of the multiple transformed images into a trained generator for image stitching to obtain the target stitched image includes:
[0029] The reconstructed images corresponding to each of the multiple transformed images are input into a trained generator for image stitching to obtain a stitched image to be judged.
[0030] The stitched image to be judged is input into a trained discriminator for judgment to determine whether the stitched image to be judged is a real image; wherein, the discriminator is generated based on a generative adversarial network (GAN);
[0031] If it is determined that the image to be stitched is not a real image, then the target stitched image is output.
[0032] On the one hand, an image stitching device is provided, the device comprising:
[0033] The image acquisition unit is used to acquire images of the target part using an industrial camera, and obtain multiple acquired images;
[0034] The feature extraction unit is used to extract features from the multiple acquired images respectively, and obtain the feature point set corresponding to each of the multiple acquired images;
[0035] The feature point matching unit is used to perform feature point matching on the multiple acquired images based on the feature point sets corresponding to each of the multiple acquired images, and determine multiple matching point pairs;
[0036] An image transformation unit is used to perform image transformation on the multiple acquired images based on the multiple matching point pairs to obtain multiple transformed images;
[0037] An image stitching unit is used to input the multiple transformed images into a trained generator for image stitching to obtain a target stitched image; wherein, the generator is based on a generative adversarial network (GAN).
[0038] On one hand, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the methods described above.
[0039] On the one hand, a storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement any of the methods described above.
[0040] Compared with the prior art, the beneficial effects of this application are as follows:
[0041] In this application, firstly, an industrial camera can be used to acquire images of the target part to obtain multiple acquired images; then, feature extraction can be performed on each of the multiple acquired images to obtain the feature point set corresponding to each of the multiple acquired images; next, feature point matching can be performed on the multiple acquired images based on the feature point sets corresponding to each of the multiple acquired images to determine multiple matching point pairs; then, image transformation can be performed on the multiple acquired images based on the multiple matching point pairs to obtain multiple transformed images; finally, the multiple transformed images can be input into a trained generator for image stitching to obtain the target stitched image; wherein, the generator is based on a Generative Adversarial Network (GAN).
[0042] Based on this, in this application, since a bidirectional matching algorithm is used to match the feature point sets corresponding to each of the multiple acquired images, compared with the prior art, this application can eliminate ambiguous matches between feature points through bidirectional consistency verification, thereby improving the reliability of matching point pairs and further improving the accuracy of image stitching. In addition, since the image stitching generator is based on Generative Adversarial Network (GAN), compared with the prior art, the target stitched image generated by this application is not only closer to the real scene in terms of visual effect and industrial inspection applicability, reducing stitching traces, but also greatly improving the efficiency of image stitching and reducing computational complexity. Attached Figure Description
[0043] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0044] Figure 1 An electronic device provided in an embodiment of this application;
[0045] Figure 2This is a schematic diagram of an image stitching method provided in an embodiment of this application;
[0046] Figure 3 This is a schematic diagram of an image stitching device provided in an embodiment of this application.
[0047] The diagram is labeled as follows: 10-image stitching device, 101-processor, 102-memory, 103-I / O interface, 104-database, 30-image stitching device, 301-image acquisition unit, 302-feature extraction unit, 303-feature point matching unit, 304-image transformation unit, 305-image stitching unit. Detailed Implementation
[0048] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Unless otherwise specified, the embodiments and features in the embodiments of this application can be arbitrarily combined with each other. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.
[0049] Currently, in the field of industrial production and inspection, in order to conduct comprehensive and high-precision monitoring of large workpieces or production scenarios, it is often necessary to stitch together multiple images captured by industrial cameras to obtain a complete large-size image. However, image stitching in industrial environments faces several challenges.
[0050] On the one hand, industrial sites present complex and variable lighting conditions, such as direct sunlight and shadows. Simultaneously, factors like mechanical vibrations and electromagnetic interference generated during production can lead to noise and blurriness in images captured by industrial cameras, significantly impacting the accuracy of image feature matching and increasing stitching errors. This results in stitched images prone to misalignment and blurring, severely affecting subsequent analysis and judgment. On the other hand, industrial production typically demands real-time and efficient inspection processes. Traditional image stitching algorithms have high computational complexity, consuming significant computing resources and time when processing large amounts of image data acquired by industrial cameras, making it difficult to meet the real-time requirements of industrial production.
[0051] Based on this, this application provides an image stitching method. In this method, firstly, an industrial camera can be used to acquire images of the target part to obtain multiple acquired images; then, feature extraction can be performed on each of the multiple acquired images to obtain the feature point set corresponding to each of the multiple acquired images; next, feature point matching can be performed on the multiple acquired images according to the feature point set corresponding to each of the multiple acquired images to determine multiple matching point pairs; then, image transformation can be performed on the multiple acquired images according to the multiple matching point pairs to obtain multiple transformed images; finally, the multiple transformed images can be input into a trained generator for image stitching to obtain the target stitched image; wherein, the generator is generated based on a Generative Adversarial Network (GAN). Based on this, in this application, since a bidirectional matching algorithm is used to match the feature point sets corresponding to each of the multiple acquired images, compared with the prior art, this application can eliminate ambiguous matches between feature points through bidirectional consistency verification, thereby improving the reliability of matching point pairs and further improving the accuracy of image stitching. In addition, since the image stitching generator is based on Generative Adversarial Network (GAN), compared with the prior art, the target stitched image generated by this application is not only closer to the real scene in terms of visual effect and industrial inspection applicability, reducing stitching traces, but also greatly improving the efficiency of image stitching and reducing computational complexity.
[0052] After introducing the design concept of the embodiments of this application, the following is a brief introduction to the application scenarios to which the technical solutions of the embodiments of this application can be applied. It should be noted that the application scenarios described below are only for illustrating the embodiments of this application and are not intended to limit the scope. In specific implementation, the technical solutions provided by the embodiments of this application can be flexibly applied according to actual needs.
[0053] like Figure 1 As shown, this is an electronic device provided in an embodiment of the present application. Specifically, the electronic device can be an image stitching device 10.
[0054] The image stitching device 10 can stitch together images captured by industrial cameras, such as personal computers (PCs), servers, and laptops. The image stitching device 10 may include one or more processors 101, memory 102, I / O interfaces 103, and databases 104. Specifically, the processor 101 may be a central processing unit (CPU) or a digital processing unit, etc. The memory 102 may be volatile memory, such as random-access memory (RAM); the memory 102 may also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or the memory 102 may be any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. The memory 102 may be a combination of the above-mentioned memories. The memory 102 may store some program instructions of the image stitching method provided in the embodiments of this application. When these program instructions are executed by the processor 101, they can be used to implement the steps of the image stitching method provided in the embodiments of this application, so as to solve the technical problems of low stitching accuracy and efficiency and high computational complexity in the prior art. The database 104 may be used to store data such as acquired images, preprocessed images, feature point sets, transformed images and target stitched images involved in the solution provided in the embodiments of this application.
[0055] In this embodiment, the image stitching device 10 can obtain image stitching instructions through the I / O interface 103. Then, the processor 101 of the image stitching device 10 will solve the technical problems of low stitching accuracy and efficiency and high computational complexity in the prior art by following the code instructions of the image stitching method provided in this embodiment in the memory 102. In addition, the acquired image, preprocessed image, feature point set, transformed image and target stitched image can be stored in the database 104.
[0056] Of course, the methods provided in the embodiments of this application are not limited to... Figure 1 The application scenarios shown can also be used in other possible scenarios, and this application embodiment does not impose any limitations. Figure 1 The functions that the various devices in the application scenarios shown can achieve will be described in subsequent method embodiments, and will not be elaborated on here. Below, the methods of the embodiments of this application will be described in conjunction with the accompanying drawings.
[0057] like Figure 2 The diagram shown is a schematic representation of an image stitching method provided in an embodiment of this application. This method can... Figure 1 The image stitching device 10 in the image stitching device is used to perform the process. The specific process is described below.
[0058] Step 201: Use an industrial camera to acquire images of the target part and obtain multiple images.
[0059] In this embodiment of the application, on an industrial parts production line, an industrial camera can be used to capture images of the target parts at a fixed angle and spacing to obtain multiple captured images (part surface images) to ensure that adjacent images have sufficient overlapping area, which facilitates subsequent image stitching.
[0060] Step 202: Extract features from multiple acquired images to obtain feature point sets corresponding to each image.
[0061] In this embodiment of the application, in order to improve the fusion accuracy, image preprocessing can also be performed when extracting features from multiple acquired images to obtain the feature point sets corresponding to each of the multiple acquired images.
[0062] Specifically, firstly, bilateral filtering can be performed on multiple acquired images for noise reduction (setting the standard deviation of the spatial Gaussian kernel to 3 and the standard deviation of the grayscale Gaussian kernel to 0.1); then, the color images can be converted to grayscale images; next, adaptive histogram equalization can be performed on the grayscale images (setting the block size to 16×16 and the clip limit to 4.0) to enhance image contrast, highlight the surface texture of the parts and any possible defects, thereby obtaining multiple preprocessed images; finally, feature extraction can be performed on these multiple preprocessed images to obtain the feature point sets corresponding to each of the multiple acquired images.
[0063] Furthermore, since the scale space construction of the traditional SIFT algorithm usually uses fixed parameters (e.g., an initial scale factor of 1.6 and 3 scale space layers per group), and industrial parts have regular shapes (e.g., clear geometric contours and relatively simple textures), if the traditional SIFT algorithm with default parameters is used, a large number of redundant feature points may be generated.
[0064] Therefore, in order to make the SIFT algorithm more suitable for industrial scenarios, in this embodiment of the application, when performing feature extraction on multiple acquired images to obtain the feature point sets corresponding to each of the multiple acquired images, an improved SIFT algorithm can be used to perform feature extraction on multiple acquired images to obtain the feature point sets corresponding to each of the multiple acquired images; wherein, the initial scale factor of the improved SIFT algorithm is 1.5, and the number of scale space layers for each group of acquired images is 2.
[0065] Based on this, this application can achieve a denser scale space by using a smaller scale factor, thus capturing more refined local details (such as edges and corners) under the regular shapes of industrial parts, avoiding feature omissions caused by excessive scale jumps. Furthermore, this application can reduce computational load by decreasing the number of scale space layers in each set of acquired images, while balancing multi-scale feature representation and computational efficiency by limiting the total number of layers (3 sets). Consequently, under the regular shapes of industrial parts, it can avoid redundant or invalid features introduced by too many scale layers, focusing on feature extraction at key scales.
[0066] Step 203: Use a bidirectional matching algorithm to match the feature points of each of the multiple acquired images to determine multiple matching point pairs.
[0067] To improve image stitching efficiency, in this embodiment, when performing feature point matching, prior knowledge of the part's geometry can be used to constrain the feature points to a specific region, searching for matching points only within the part's outline. Furthermore, to enhance the reliability of matching pairs, this embodiment can also perform bidirectional matching and random sampling consistency checks on the feature points.
[0068] Specifically, within the outline of the target industrial part, for any two acquired images, a bidirectional matching method and a Random Sample Consensus (RANSAC) algorithm can be used to sequentially match the corresponding feature point sets of any two acquired images, thereby determining multiple matching point pairs corresponding to any two acquired images.
[0069] Furthermore, since the bidirectional matching method requires that feature points in image A have a unique matching point in image B, and that the corresponding point in image B also has a unique matching point back to image A, compared to the potential "one-to-many" ambiguity in traditional unidirectional matching (e.g., feature points in image A matching to feature points in image B), this application can eliminate approximately 20% of unidirectional ambiguous matching points through bidirectional consistency verification, thereby further improving the reliability of matching point pairs.
[0070] Furthermore, in this embodiment, the Random Sample Consensus (RANSAC) algorithm includes mechanisms such as random sampling, hypothesis testing, and iterative optimization to estimate optimal model parameters from observation data containing a large number of outliers. The execution steps of the RANSAC algorithm are as follows:
[0071] First, random sampling can be used, that is, the smallest sample set can be randomly selected from the pairs of matching points to be optimized in any two acquired images obtained after bidirectional matching (for example, affine transformation requires at least 4 pairs of matching points).
[0072] Then, we can assume a model, that is, a temporary transformation model can be computed based on a minimal sample set.
[0073] Next, the filtering can be verified. That is, a temporary transformation model can be used to verify all pairs of matching points to be optimized in any two acquired images, so as to filter out the interior points (correct matching points) that conform to the temporary transformation model.
[0074] Finally, iterative optimization can be performed. That is, through multiple iterations, the temporary transformation model with the most interior points can be found as the optimal solution. The minimum sample set corresponding to this optimal temporary transformation model is the multiple matching point pairs corresponding to any two acquired images. The maximum number of iterations is 300, and the threshold is 2 pixels.
[0075] Step 204: Based on multiple matching point pairs, perform image transformation on multiple acquired images to obtain multiple transformed images.
[0076] Because industrial cameras easily introduce geometric distortions such as radial and tangential distortion when capturing images, this application, in its implementation, when transforming multiple acquired images based on multiple matching point pairs to obtain multiple transformed images, can specifically perform a cascaded operation of distortion correction and affine transformation on multiple matching point pairs corresponding to any two acquired images. This transforms the two acquired images to the same coordinate system, thereby obtaining two transformed images corresponding to any two acquired images, achieving more comprehensive geometric correction. In the embodiments of this application, affine transformation can handle linear transformations such as translation, rotation, scaling, and shearing.
[0077] Furthermore, in the embodiments of this application, in addition to using "distortion correction + affine transformation" during image transformation, "parallel selection polynomial transformation" can also be used for some industrial application scenarios with extremely high image accuracy requirements and small changes in shooting angle. Specifically, firstly, control points can be manually marked on each acquired image; then, polynomial fitting can be performed based on the marked control points; next, the transformation parameters can be stored; finally, image transformation can be performed in batches.
[0078] Based on this, since polynomial transformation can more flexibly describe the geometric deformation of an image, this application can achieve more accurate image transformation by fitting control point pairs in the acquired image. This provides better transformation effects in scenarios such as the inspection of precision optical instrument components, making the stitched image more accurate in terms of size, shape, etc., meeting the demand for high-precision images, and providing a more accurate image basis for precision industrial production and inspection.
[0079] Step 205: Input multiple transformed images into the trained generator to stitch the images together to obtain the target stitched image.
[0080] In this embodiment, the generator is generated based on a Generative Adversarial Network (GAN).
[0081] Specifically, when inputting multiple transformed images into a trained generator for image stitching to obtain the target stitched image, firstly, for any given transformed image, it can be decomposed into three layers of wavelet coefficients; then, the weights corresponding to the low-frequency, high-frequency, and intermediate-frequency layers of wavelet coefficients can be determined separately; next, the three layers of wavelet coefficients and their corresponding weights can be weighted and fused (the low-frequency layer highlights high-energy regions, and the high-frequency layer strengthens edges with consistent directions) to obtain fused wavelet coefficients; then, the fused wavelet coefficients can be subjected to inverse wavelet transform to obtain the reconstructed image corresponding to any given transformed image; finally, the reconstructed images corresponding to the multiple transformed images can be input into the trained generator for image stitching to eliminate stitching seams, thereby obtaining a high-quality target stitched image that incorporates multi-scale features.
[0082] Therefore, this application can improve the smoothness of stitching by dynamically adjusting the image content (e.g., emphasizing low frequencies in texture areas and high frequencies in edge areas) to ensure complementary features at multiple scales.
[0083] Furthermore, in order to improve the smoothness of the stitching and obtain a high-quality stitched image, in this embodiment of the application, when determining the weights corresponding to the three wavelet coefficients respectively, the weight calculation method of multi-resolution weighted fusion can also be optimized.
[0084] Specifically, for low-frequency wavelet coefficients, the weights of the low-frequency wavelet coefficients can be determined based on the pixel gray-level variance of the target part region obtained from the regional feature energy calculation. The pixel gray-level variance reflects the energy distribution, with the energy of each region normalized to [0,1]. Higher energy (richer texture / detail) results in higher weights for the low-frequency wavelet coefficients.
[0085] Of course, in the low-frequency layer, in addition to regional feature energy, factors such as the shape complexity of the regions can also be considered to adjust the weights, so that the fused image can better balance the fusion effect of different regions while preserving the overall structure. Specifically, first, the regional feature energy is calculated as the basic weight; then, shape complexity features such as perimeter, area ratio, and number of holes can be extracted, normalized, and then used to generate a regional adjustment factor through the Sigmoid function; finally, the weight of the frequency layer wavelet coefficients = basic weight × regional adjustment factor. Based on this, if the regional adjustment factor > 1, the weight will be enhanced; if the regional adjustment factor < 1, the weight will be weakened to achieve differentiated weighting.
[0086] For example, in stitched images of large mechanical equipment, weights can be reasonably allocated between key component areas with complex shapes and auxiliary component areas with regular shapes, so that the stitched image can present the overall equipment structure while highlighting the details of key components.
[0087] For high-frequency wavelet coefficients, the weights of the high-frequency wavelet coefficients can be determined based on the edge gradient directions extracted by the Sobel operator. The edge gradient direction θ = arctan2(Gy, Gx) is quantized to [0, π), where Gy is the vertical operator and Gx is the horizontal operator.
[0088] Of course, in the high-frequency layer, in addition to basing on the edge gradient direction, the consistency of the edge gradient direction can also be enhanced by combining edge curvature information, etc., to more accurately enhance the texture and defect details of the part surface and improve the fusion quality. Specifically, firstly, the consistency of the edge gradient direction is calculated as the basic weight; then, the edge curvature (e.g., the extreme value of the second derivative) is extracted and normalized to generate a curvature adjustment factor; finally, the weight of the wavelet coefficients in the high-frequency layer = basic weight × curvature adjustment factor. Based on this, the weight of high curvature regions (sharp edges) is increased (e.g., multiplied by 1.2-1.5), while the weight of low curvature regions (smooth edges) is decreased (e.g., multiplied by 0.8-0.5), to achieve differentiated weighting of edges with different shapes, more accurately enhance the texture and defect details of the product surface, and improve the fusion quality.
[0089] For example, in the image fusion of circuit board solder joint inspection, defects such as minor cracks and poor solder joints can be clearly presented, thereby improving the ability to identify product surface quality problems in industrial inspection.
[0090] For the wavelet coefficients of the intermediate frequency layer, the weights of the intermediate frequency layer wavelet coefficients are determined based on the weights of the low-frequency layer wavelet coefficients and the high-frequency layer wavelet coefficients. Specifically, the geometric mean of the low-frequency layer weights and the high-frequency layer weights can be taken, or the weights can be adaptively fused according to w_mid = w_low·w_high. Here, w_mid is the weight of the intermediate frequency layer wavelet coefficients, w_low is the weight of the low-frequency layer wavelet coefficients, and w_high is the weight of the high-frequency layer wavelet coefficients.
[0091] In one possible implementation, to avoid problems such as misalignment, blurring, or deformation that occur in traditional stitching methods, in this embodiment of the application, when inputting the reconstructed images corresponding to multiple transformed images into a trained generator for image stitching to obtain the target stitched image, adversarial loss can be used to drive the generator to optimize stitching details (e.g., eliminate seams, enhance texture consistency) to improve the rationality of the global structure.
[0092] Specifically, firstly, the reconstructed images corresponding to multiple transformed images are input into a trained generator (used to generate a stitched image) for image stitching to obtain a stitched image to be judged; the generator is based on a generative adversarial network (GAN). Then, the stitched image to be judged is input into a trained discriminator (used to distinguish between real and stitched images) for discrimination to determine whether the stitched image is a real image; the discriminator is also based on a generative adversarial network (GAN). Finally, if it is determined that the stitched image to be judged is not a real image, the target stitched image is output; otherwise, the stitched image to be judged is determined to be a real image.
[0093] In this embodiment, the Generative Adversarial Network (GAN) structure adopts an encoder-decoder structure. Each layer of the encoder includes a convolutional layer (Conv), a batch normalization layer (BN), and a ReLU activation function; each layer of the decoder includes a transposed convolutional layer (TransConv), a batch normalization layer (BN), and a ReLU activation function, and is connected to the corresponding layer of the encoder via skip connections to preserve low-level detailed features.
[0094] Therefore, in this application, since the generator is based on a Generative Adversarial Network (GAN), the generated target stitched image is closer to the real scene in terms of visual effect and industrial inspection applicability, reducing stitching artifacts. For example, in image stitching for surface quality inspection of building materials, the stitched image is visually more natural with no obvious stitching boundaries; when detecting defects such as cracks and holes on the material surface, it can more accurately present defect features, improve the accuracy and reliability of industrial inspection, and provide high-quality stitched images for building material quality assessment.
[0095] In one possible implementation, the generator and discriminator need to be trained before image stitching is performed.
[0096] Specifically, first, the prepared industrial image dataset is preprocessed (denoising, alignment, etc.); then, a generator (U-Net) and a discriminator (two-branch PatchGAN) are constructed; next, training can be carried out in stages: first, the generator is pre-trained, and then bidirectional adversarial training is performed (combining adversarial loss, gradient consistency loss, and detection loss) to dynamically adjust the learning rate, iterating for 300-800 rounds until convergence, with fine-tuning using real-time data during the process.
[0097] Based on this, compared to existing technologies, this application abandons the traditional simple adversarial mode of GANs and adopts a bidirectional guided adversarial training algorithm. The generator relies on the feedback of the discriminator, and the discriminator also relies on the transmission of the generator, so that the stitched image not only meets the requirements of industrial inspection features, but also highlights the key features of the product. In addition, since the discriminator evaluates the stitched image from the perspectives of realism and industrial inspection applicability, this application can also improve the usability of the stitched image in industrial scenarios.
[0098] In one possible implementation, each industrial image in the industrial image dataset used for training can also be preprocessed.
[0099] Specifically, firstly, bilateral filtering can be used to denoise industrial images, and parameters can be dynamically adjusted based on local noise in the image to preserve details; secondly, the denoised industrial images can be standardized to unify the brightness, contrast, and color space of images from different cameras; finally, data augmentation operations such as random rotation, scaling, and cropping can be used to simulate changes in camera angle and distance, providing diverse and high-quality data adapted to industrial camera images for training.
[0100] In one possible implementation, the training parameters can also be set appropriately.
[0101] Specifically, the learning rate can be dynamically adjusted. A larger initial value helps the Generative Adversarial Network (GAN) model converge quickly, and then gradually decreases as training progresses and the loss function changes to prevent missing the optimal solution. For example, an initial value of 0.001 can be used, halved when the loss function decreases to less than 0.01. The number of iterations depends on the image complexity and model convergence; fewer iterations are needed for simple scenes, and more for complex scenes, balancing training efficiency and image quality.
[0102] In one possible implementation, the training strategy can also be optimized.
[0103] Specifically, L1 and L2 regularization constraints can be introduced to prevent overfitting and ensure model robustness. Batch normalization is used to normalize data before the neural network input, accelerating convergence and reducing gradient problems. The model is regularly evaluated using industrial detection metrics combined with PSNR and SSIM, and parameters and strategies are adjusted based on the results to ensure that the model adapts to different industrial scenarios and camera images, generating high-quality fused images to meet industrial needs.
[0104] In one possible implementation, to improve the quality of the stitched image, the target stitched image can be cropped after stitching to remove excess edges. Furthermore, an adaptive sharpening algorithm can be used to dynamically adjust the sharpening parameters based on the characteristics of the part image; for example, the sharpening intensity factor can be set to 0.8.
[0105] Furthermore, the quality of the stitched image can be evaluated by calculating the defect detection accuracy, peak signal-to-noise ratio (PSNR) value, and structural similarity (SSIM) value. If the defect detection accuracy reaches 95% or higher, the PSNR value reaches 38dB or higher, and the SSIM value reaches 0.96 or higher, it indicates that the stitched image can clearly display the surface defects of the parts and meets the industrial inspection requirements. Conversely, if it does not meet the industrial inspection requirements, the sharpening intensity factor needs to be further adjusted.
[0106] In summary, by employing a bidirectional matching algorithm to match feature points from the feature point sets corresponding to each of the multiple acquired images, this application can eliminate ambiguous matches between feature points through bidirectional consistency verification, thereby improving the reliability of matching point pairs and further enhancing the accuracy of image stitching. Furthermore, since the image stitching generator is based on a Generative Adversarial Network (GAN), the target stitched image generated by this application not only more closely resembles the real-world scene in terms of visual effects and industrial inspection applicability, reducing stitching artifacts, but also significantly improves the efficiency of image stitching and reduces computational complexity.
[0107] Based on the same inventive concept, embodiments of this application provide an image stitching device 30, such as... Figure 3 As shown, the image stitching device 30 includes:
[0108] Image acquisition unit 301 is used to acquire images of the target part using an industrial camera and obtain multiple acquired images;
[0109] The feature extraction unit 302 is used to extract features from multiple acquired images to obtain feature point sets corresponding to each of the multiple acquired images;
[0110] The feature point matching unit 303 is used to perform feature point matching on multiple acquired images based on the feature point sets corresponding to each of the multiple acquired images, and determine multiple matching point pairs;
[0111] The image transformation unit 304 is used to perform image transformation on multiple acquired images based on multiple matching point pairs to obtain multiple transformed images;
[0112] The image stitching unit 305 is used to input multiple transformed images into a trained generator for image stitching to obtain a target stitched image; wherein, the generator is generated based on a generative adversarial network (GAN).
[0113] Optionally, the feature extraction unit 302 is also used for:
[0114] An improved SIFT algorithm is used to extract features from multiple acquired images to obtain feature point sets corresponding to each image. The initial scale factor of the improved SIFT algorithm is 1.5, and the number of scale space layers for each group of acquired images is 2.
[0115] Optionally, the feature point matching unit 303 is also used for:
[0116] Within the contour range of the target industrial part, for any two acquired images, a bidirectional matching method and the Random Sample Consensus (RANSAC) algorithm are used to sequentially match the corresponding feature point sets of any two acquired images to determine multiple matching point pairs corresponding to any two acquired images.
[0117] Optionally, the image transformation unit 304 is also used for:
[0118] Distortion correction and affine transformation are performed sequentially on multiple matching point pairs corresponding to any two acquired images to obtain two transformed images corresponding to any two acquired images.
[0119] Optionally, the image stitching unit 305 is also used for:
[0120] For any transformed image, decompose it into three layers of wavelet coefficients;
[0121] Determine the weights corresponding to the wavelet coefficients of the three layers respectively;
[0122] The fused wavelet coefficients are obtained by weighting and fusing the 3-layer wavelet coefficients and their respective weights.
[0123] Perform inverse wavelet transform on the fused wavelet coefficients to obtain the reconstructed image corresponding to any transformed image;
[0124] The reconstructed images corresponding to the multiple transformed images are input into a trained generator for image stitching to obtain the target stitched image.
[0125] Optionally, the image stitching unit 305 is also used for:
[0126] For the low-frequency wavelet coefficients, the weights of the low-frequency wavelet coefficients are determined based on the pixel gray-level variance of the target part region obtained from the regional feature energy.
[0127] For the wavelet coefficients of the high-frequency layer, the weights of the wavelet coefficients are determined according to the edge gradient direction;
[0128] For the wavelet coefficients of the intermediate frequency layer, the weights of the wavelet coefficients of the intermediate frequency layer are determined based on the weights of the wavelet coefficients of the low frequency layer and the wavelet coefficients of the high frequency layer.
[0129] Optionally, the image stitching unit 305 is also used for:
[0130] The reconstructed images corresponding to multiple transformed images are input into a trained generator for image stitching to obtain a stitched image to be judged.
[0131] The stitched image to be judged is input into the trained discriminator for judgment to determine whether the stitched image to be judged is a real image; the discriminator is generated based on the generative adversarial network (GAN);
[0132] If it is determined that the image to be stitched is not a real image, then the target stitched image is output.
[0133] The image stitching device 30 can be used to perform... Figure 2 The method performed in the illustrated embodiment is as follows; therefore, the functions that each functional module of the image stitching device 30 can achieve can be referred to. Figure 2 The embodiments shown are described in detail below.
[0134] In some possible implementations, various aspects of the methods provided in this application can also be implemented as a program component comprising program code that, when run on a computer device, causes the computer device to perform the steps of the methods according to the various exemplary embodiments of this application described above. For example, the computer device may perform actions such as... Figure 2 The method performed in the illustrated embodiment.
[0135] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks. Alternatively, if the integrated unit of the present invention is implemented as a software functional module and sold or used as an independent part, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software part. This computer software part is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.
[0136] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.
[0137] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.
Claims
1. An image stitching method, characterized in that, The method includes: An industrial camera is used to acquire images of the target part, resulting in multiple captured images; Feature extraction is performed on each of the multiple acquired images to obtain the feature point set corresponding to each of the multiple acquired images; A bidirectional matching algorithm is used to match feature points in the feature point sets corresponding to each of the multiple acquired images to determine multiple matching point pairs. Based on the multiple matching point pairs, image transformation is performed on the multiple acquired images to obtain multiple transformed images; The multiple transformed images are input into a trained generator for image stitching to obtain a target stitched image; wherein, the generator is based on a Generative Adversarial Network (GAN); the step of inputting the multiple transformed images into the trained generator for image stitching to obtain the target stitched image includes: For any transformed image, the transformed image is decomposed into three layers of wavelet coefficients; The weights corresponding to the three wavelet coefficients are determined separately. The step of determining the weights corresponding to the three wavelet coefficients includes: for the low-frequency wavelet coefficients, determining the weights of the low-frequency wavelet coefficients based on the pixel grayscale variance of the target part region obtained from the regional feature energy; for the high-frequency wavelet coefficients, determining the weights of the high-frequency wavelet coefficients based on the edge gradient direction; and for the intermediate-frequency wavelet coefficients, determining the weights of the intermediate-frequency wavelet coefficients based on the weights of the low-frequency and high-frequency wavelet coefficients. The three-layer wavelet coefficients and their respective weights are weighted and fused to obtain the fused wavelet coefficients. Perform inverse wavelet transform on the fused wavelet coefficients to obtain the reconstructed image corresponding to any one of the transformed images; The reconstructed images corresponding to each of the multiple transformed images are input into a trained generator for image stitching to obtain the target stitched image.
2. The method as described in claim 1, characterized in that, The step of extracting features from the multiple acquired images to obtain the feature point set corresponding to each of the multiple acquired images includes: An improved SIFT algorithm is used to extract features from the multiple acquired images to obtain the feature point sets corresponding to each of the multiple acquired images; wherein, the initial scale factor of the improved SIFT algorithm is 1.5, and the number of scale space layers for each group of acquired images is 2.
3. The method as described in claim 1, characterized in that, The step of using a bidirectional matching algorithm to match feature points in the feature point sets corresponding to each of the multiple acquired images to determine multiple matching point pairs includes: Within the contour range of the target part, for any two acquired images, a bidirectional matching method and the Random Sample Consensus (RANSAC) algorithm are used to sequentially match the corresponding feature point sets of the two acquired images to determine multiple matching point pairs corresponding to the two acquired images.
4. The method as described in claim 3, characterized in that, The step of performing image transformation on the multiple acquired images based on the multiple matching point pairs to obtain multiple transformed images includes: Distortion correction and affine transformation are sequentially performed on multiple matching point pairs corresponding to any two acquired images to obtain two transformed images corresponding to any two acquired images.
5. The method as described in claim 1, characterized in that, The step of inputting the reconstructed images corresponding to each of the multiple transformed images into a trained generator for image stitching to obtain the target stitched image includes: The reconstructed images corresponding to each of the multiple transformed images are input into a trained generator for image stitching to obtain a stitched image to be judged. The stitched image to be judged is input into a trained discriminator for judgment to determine whether the stitched image to be judged is a real image; wherein, the discriminator is generated based on a generative adversarial network (GAN); If it is determined that the image to be stitched is not a real image, then the target stitched image is output.
6. An image stitching device, characterized in that, The device includes: The image acquisition unit is used to acquire images of the target part using an industrial camera, and obtain multiple acquired images; The feature extraction unit is used to extract features from the multiple acquired images respectively, and obtain the feature point set corresponding to each of the multiple acquired images; The feature point matching unit is used to perform feature point matching on the multiple acquired images based on the feature point sets corresponding to each of the multiple acquired images, and determine multiple matching point pairs; An image transformation unit is used to perform image transformation on the multiple acquired images based on the multiple matching point pairs to obtain multiple transformed images; An image stitching unit is used to input the multiple transformed images into a trained generator for image stitching to obtain a target stitched image; wherein, the generator is generated based on a Generative Adversarial Network (GAN); the step of inputting the multiple transformed images into the trained generator for image stitching to obtain the target stitched image includes: for any given transformed image, decomposing the given transformed image into three layers of wavelet coefficients; and determining the weights corresponding to each of the three layers of wavelet coefficients; wherein, the step of determining the weights corresponding to each of the three layers of wavelet coefficients includes: for the low-frequency layer wavelet coefficients, determining the pixel grayscale of the target part region obtained from the regional feature energy. The variance is used to determine the weights of the low-frequency wavelet coefficients; for the high-frequency wavelet coefficients, the weights are determined based on the edge gradient direction; for the mid-frequency wavelet coefficients, the weights are determined based on the weights of the low-frequency and high-frequency wavelet coefficients; the three wavelet coefficients and their respective weights are weighted and fused to obtain fused wavelet coefficients; inverse wavelet transform is performed on the fused wavelet coefficients to obtain the reconstructed image corresponding to any one of the transformed images; the reconstructed images corresponding to the multiple transformed images are input into a trained generator for image stitching to obtain the target stitched image.
7. An electronic device, characterized in that, The device includes: Memory, used to store program instructions; A processor is configured to invoke program instructions stored in the memory and execute the method described in any one of claims 1-5 according to the obtained program instructions.
8. A storage medium, characterized in that, The storage medium stores computer-executable instructions for causing a computer to perform the method described in any one of claims 1-5.