Image splicing method and device, equipment and medium

Abstract

The invention discloses an image splicing method and device, equipment and a medium, relates to the technical field of industrial production and detection, and is used for solving the technical problems of relatively low splicing precision and efficiency and relatively high calculation complexity in the prior art. The method comprises the following steps: respectively carrying out feature extraction on a plurality of acquired images to obtain feature point sets corresponding to the plurality of acquired images; performing feature point matching on the feature point sets corresponding to the plurality of acquired images by adopting a bidirectional matching algorithm, and determining a plurality of matching point pairs; according to the plurality of matching point pairs, performing image transformation on the plurality of acquired images to obtain a plurality of transformed images; inputting the plurality of transformed images into the trained generator for image splicing to obtain a target spliced image; wherein the generator is generated based on the GAN, so that the image splicing is carried out through the bidirectional matching algorithm and the generator generated based on the GAN, the splicing accuracy and efficiency are improved, and the calculation complexity is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of industrial production and detection, and provides an image stitching method, device, equipment and medium. BACKGROUND

[0002] At present, in the field of industrial production and detection, in order to comprehensively and accurately monitor large workpieces or production scenes, it is often necessary to stitch multiple images captured by an industrial camera to obtain a complete large-size image. However, image stitching in an industrial environment faces many challenges.

[0003] On the one hand, the industrial site has complex and variable lighting conditions, such as direct strong light and shadow obstruction, and factors such as mechanical vibration and electromagnetic interference generated during the production process, which can cause the images captured by the industrial camera to have noise and blur, greatly affecting the accuracy of image feature matching, making the stitching error significantly increase, and the stitched image prone to misalignment and blur, which seriously affects subsequent analysis and judgment. On the other hand, industrial production usually requires real-time and efficient detection processes, and traditional image stitching algorithms have high computational complexity, which consumes a large amount of computing resources and time when processing large amounts of image data collected by industrial cameras, making it difficult to meet the real-time requirements of industrial production.

[0004] Therefore, how to develop a method that can adapt to the application scenario of an industrial camera, effectively improve the stitching accuracy, reduce the computational complexity, and quickly stitch images is imminent. SUMMARY

[0005] The present application provides an image stitching method, device, equipment and medium, which solves the technical problems of low stitching accuracy and efficiency and high computational complexity in the prior art.

[0006] On the one hand, an image stitching method is provided, which comprises: using an industrial camera to collect images of a target part to obtain multiple collected images; performing feature extraction on the multiple collected images respectively to obtain a feature point set corresponding to each of the multiple collected images; using a bidirectional matching algorithm to perform feature point matching on the feature point set corresponding to each of the multiple collected images to determine a plurality of matching point pairs; performing image transformation on the multiple collected images according to the plurality of matching point pairs to obtain multiple transformed images; inputting the multiple transformed images into a trained generator for image stitching to obtain a target stitched image; wherein the generator is generated based on a generative adversarial network (GAN).

[0007] Optionally, the step of performing feature extraction on the plurality of acquisition images respectively to obtain a feature point set corresponding to each of the plurality of acquisition images comprises: The improved SIFT algorithm is adopted to perform feature extraction on the plurality of acquisition images respectively to obtain a feature point set corresponding to each of the plurality of acquisition images; wherein the initial scale factor of the improved SIFT algorithm is 1.5, and the number of scale space layers of each group of acquisition images is 2.

[0008] Optionally, the step of performing feature point matching on the feature point set corresponding to each of the plurality of acquisition images respectively by using the bidirectional matching algorithm to determine a plurality of matching point pairs comprises: Within the contour range of the target industrial part, for any two acquisition images, the bidirectional matching method and the random sample consensus (RANSAC) algorithm are used to perform feature point matching on the corresponding feature point sets of the any two acquisition images in sequence to determine a plurality of matching point pairs corresponding to the any two acquisition images.

[0009] Optionally, the step of performing image transformation on the plurality of acquisition images according to the plurality of matching point pairs to obtain a plurality of transformed images comprises: The plurality of matching point pairs corresponding to the any two acquisition images are sequentially subjected to distortion correction and affine transformation to obtain two transformed images corresponding to the any two acquisition images.

[0010] Optionally, the step of inputting the plurality of transformed images into the trained generator for image stitching to obtain a target stitched image comprises: For any one transformed image, the any one transformed image is decomposed into 3 layers of wavelet coefficients; The weights corresponding to the 3 layers of wavelet coefficients are determined respectively; The 3 layers of wavelet coefficients and the weights corresponding to the 3 layers of wavelet coefficients are weighted and fused to obtain fused wavelet coefficients; Inverse wavelet transformation is performed on the fused wavelet coefficients to obtain a reconstructed image corresponding to the any one transformed image; The reconstructed images corresponding to the plurality of transformed images are input into the trained generator for image stitching to obtain a target stitched image.

[0011] Optionally, the step of determining the weights corresponding to the 3 layers of wavelet coefficients respectively comprises: For low-frequency layer wavelet coefficients, the weight of the low-frequency layer wavelet coefficients is determined according to the pixel gray scale variance of the target part region obtained by region feature energy calculation; For high-frequency layer wavelet coefficients, the weight of the high-frequency layer wavelet coefficients is determined according to the edge gradient direction; For the wavelet coefficients of the middle frequency layer, weights of the wavelet coefficients of the middle frequency layer are determined according to the weights of the wavelet coefficients of the low frequency layer and the weights of the wavelet coefficients of the high frequency layer.

[0012] Optionally, the step of inputting the reconstructed images corresponding to the plurality of transformed images respectively into the trained generator to perform image stitching to obtain a target stitched image comprises: inputting the reconstructed images corresponding to the plurality of transformed images respectively into the trained generator to perform image stitching to obtain a target stitched image; inputting the target stitched image into the trained discriminator to perform discrimination to determine whether the target stitched image is a real image; wherein the discriminator is generated based on a generative adversarial network (GAN); if it is determined that the target stitched image is not a real image, outputting a target stitched image.

[0013] In one aspect, an image stitching device is provided, and the device comprises: an image acquisition unit configured to acquire a plurality of images of a target part by using an industrial camera; a feature extraction unit configured to extract features from the plurality of images respectively to obtain a plurality of feature point sets corresponding to the plurality of images respectively; a feature point matching unit configured to match feature points of the plurality of images according to the plurality of feature point sets corresponding to the plurality of images respectively to determine a plurality of matching point pairs; an image transformation unit configured to transform the plurality of images according to the plurality of matching point pairs to obtain a plurality of transformed images; an image stitching unit configured to input the plurality of transformed images into a trained generator to perform image stitching to obtain a target stitched image; wherein the generator is generated based on a generative adversarial network (GAN).

[0014] In one aspect, an electronic device is provided, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements any of the above methods when executing the computer program.

[0015] In one aspect, a storage medium is provided, which stores computer program instructions executable by a processor to implement any of the above methods.

[0016] Compared with the prior art, the present application has the following advantages: In the present application, first, an industrial camera can be used to collect images of the target part to obtain multiple collected images; then, feature extraction can be performed on the multiple collected images respectively to obtain feature point sets corresponding to the multiple collected images respectively; next, feature point matching can be performed on the multiple collected images according to the feature point sets corresponding to the multiple collected images respectively to determine multiple matched point pairs; then, image transformation can be performed on the multiple collected images according to the multiple matched point pairs to obtain multiple transformed images; finally, the multiple transformed images can be input into a trained generator for image stitching to obtain a target stitched image; wherein the generator is generated based on a generative adversarial network (GAN).

[0017] Based on this, in the present application, since the feature point matching is performed on the feature point sets corresponding to the multiple collected images respectively by using the bidirectional matching algorithm, compared with the prior art, the present application can eliminate the ambiguous matching between the feature points through bidirectional consistency verification to improve the reliability of the matched point pairs, thereby further improving the accuracy of image stitching, in addition, since the generator for image stitching is generated based on the generative adversarial network (GAN), compared with the prior art, the target stitched image generated by the present application not only is closer to the real scene in terms of visual effect and industrial detection applicability, reduces the stitching traces, but also can greatly improve the efficiency of image stitching and reduce the computational complexity. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments or the related art, the drawings needed to be used in the embodiments or the related art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to the provided drawings without any creative labor.

[0019] Figure 1 An electronic device provided by the embodiments of the present application; Figure 2 A schematic diagram of the image stitching method provided by the embodiments of the present application; Figure 3 A schematic diagram of the image stitching device provided by the embodiments of the present application.

[0020] In the figure, 10 is an image stitching device, 101 is a processor, 102 is a memory, 103 is an I / O interface, 104 is a database, 30 is an image stitching device, 301 is an image collection unit, 302 is a feature extraction unit, 303 is a feature point matching unit, 304 is an image transformation unit, and 305 is an image stitching unit. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application. The embodiments in the present application and the features in the embodiments can be combined with each other arbitrarily without conflict. And, although the logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in an order different from that here.

[0022] At present, in the field of industrial production and detection, in order to monitor large workpieces or production scenes comprehensively and with high precision, it is often necessary to splice multiple images captured by an industrial camera to obtain a complete large-size image. However, image splicing in an industrial environment faces many challenges.

[0023] On the one hand, the industrial field has complex and variable lighting conditions, such as direct strong light, shadow blocking, etc., and factors such as mechanical vibration and electromagnetic interference generated during the production process, which can cause the images captured by the industrial camera to have noise and blur, greatly affecting the accuracy of image feature matching, making the splicing error significantly increase, and the spliced image prone to misalignment, blur and other problems, seriously affecting subsequent analysis and judgment. On the other hand, industrial production usually requires real-time and efficient detection processes, and traditional image splicing algorithms have high computational complexity, which will consume a lot of computing resources and time when processing large amounts of image data collected by industrial cameras, making it difficult to meet the real-time requirements of industrial production.

[0024] Based on this, the embodiment of the present application provides an image stitching method, in which, first, an industrial camera can be used to collect images of a target part to obtain a plurality of collected images; then, the plurality of collected images can be respectively subjected to feature extraction to obtain a plurality of feature point sets corresponding to the plurality of collected images respectively; next, the plurality of collected images can be subjected to feature point matching according to the plurality of feature point sets corresponding to the plurality of collected images respectively to determine a plurality of matching point pairs; then, the plurality of collected images can be subjected to image transformation according to the plurality of matching point pairs to obtain a plurality of transformed images; finally, the plurality of transformed images can be input into a trained generator for image stitching to obtain a target stitched image; wherein the generator is generated based on a generative adversarial network (GAN). Based on this, in the present application, since the two-way matching algorithm is used to match the feature points in the plurality of feature point sets corresponding to the plurality of collected images respectively, compared with the prior art, the present application can eliminate the ambiguous matching between the feature points through two-way consistency verification to improve the reliability of the matching point pairs, thereby further improving the accuracy of image stitching, in addition, since the generator used for image stitching is generated based on the generative adversarial network (GAN), compared with the prior art, the target stitched image generated by the present application not only is closer to the real scene in terms of visual effect and industrial detection applicability, reduces the stitching traces, but also can greatly improve the efficiency of image stitching and reduce the computational complexity.

[0025] After introducing the design idea of the embodiment of the present application, the application scenarios to which the technical solution of the embodiment of the present application can be applied will be briefly introduced, and it should be noted that the following application scenarios are only used to illustrate the embodiment of the present application but not to limit. In the specific implementation process, the technical solution provided by the embodiment of the present application can be flexibly applied according to actual needs.

[0026] As shown in Figure 1 , an electronic device provided by the embodiment of the present application, which can be specifically an image stitching device 10.

[0027] The image stitching device 10 can stitch images collected by an industrial camera, and can be a personal computer (PC), a server, a laptop, or the like. The image stitching device 10 can include one or more processors 101, a memory 102, an I / O interface 103, and a database 104. Specifically, the processor 101 can be a central processing unit (CPU), or a digital processing unit, or the like. The memory 102 can be a volatile memory, such as a random-access memory (RAM), or a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD), or a solid-state drive (SSD), or any other medium capable of carrying or storing desired program codes in the form of instructions or data structures and accessible by a computer, but is not limited thereto. The memory 102 can be a combination of the above memories. The memory 102 can store part of the program instructions of the image stitching method provided in the embodiments of the present application, and the program instructions can be executed by the processor 101 to implement the steps of the image stitching method provided in the embodiments of the present application, to solve the technical problems of low stitching accuracy and efficiency and high computational complexity in the prior art. The database 104 can be used to store data such as collected images, preprocessed images, feature point sets, transformed images, and target stitched images involved in the solutions provided in the embodiments of the present application.

[0028] In the embodiments of the present application, the image stitching device 10 can obtain an image stitching instruction through the I / O interface 103, and then the processor 101 of the image stitching device 10 can solve the technical problems of low stitching accuracy and efficiency and high computational complexity in the prior art according to the code instructions of the image stitching method provided in the embodiments of the present application in the memory 102. In addition, the data such as collected images, preprocessed images, feature point sets, transformed images, and target stitched images can be stored in the database 104.

[0029] Of course, the method provided in the embodiments of the present application is not limited to the application scenarios shown in the above description, but can also be used in other possible application scenarios, which are not limited in the embodiments of the present application. For the functions that can be achieved by the devices in the application scenarios shown in the above description, the functions will be described in the subsequent method embodiments, and will not be described here in more detail. Figure 1 Of course, the method provided in the embodiments of the present application is not limited to the application scenarios shown in the above description, but can also be used in other possible application scenarios, which are not limited in the embodiments of the present application. For the functions that can be achieved by the devices in the application scenarios shown in the above description, the functions will be described in the subsequent method embodiments, and will not be described here in more detail. Figure 1 The functions that can be achieved by the devices in the application scenarios shown in the above description will be described in the subsequent method embodiments, and will not be described here in more detail. In the following, the method of the embodiments of the present application will be introduced with reference to the accompanying drawings.

[0030] As Figure 2 shown, a schematic diagram of an image stitching method provided by an embodiment of the present application, which can be executed by the image stitching device 10 in Figure 1 , and the flow of the method is introduced as follows.

[0031] Step 201: An industrial camera is used to collect images of the target part to obtain a plurality of collected images.

[0032] In the embodiment of the present application, on the industrial part production line, an industrial camera can be used to collect images of the target part at a fixed angle and interval to obtain a plurality of collected images (part surface images), so as to ensure that adjacent images have sufficient overlapping areas, and facilitate subsequent image stitching.

[0033] Step 202: Feature extraction is performed on the plurality of collected images respectively to obtain a plurality of feature point sets respectively corresponding to the plurality of collected images.

[0034] In the embodiment of the present application, in order to improve the fusion accuracy, when performing feature extraction on the plurality of collected images respectively to obtain a plurality of feature point sets respectively corresponding to the plurality of collected images, image preprocessing can also be performed.

[0035] Specifically, first, bilateral filtering denoising can be performed on the plurality of collected images respectively (set the spatial Gaussian kernel standard deviation to 3 and the gray value Gaussian kernel standard deviation to 0.1); then, the color image can be converted into a gray image; next, adaptive histogram equalization can be performed on the gray image (set the block size to 16x16 and the clip limit to 4.0) to enhance the image contrast and highlight the part surface texture and possible defects, thereby obtaining a plurality of preprocessed images; finally, feature extraction can be performed on the plurality of preprocessed images respectively to obtain a plurality of feature point sets respectively corresponding to the plurality of collected images.

[0036] Further, since the scale space construction of the traditional SIFT algorithm usually adopts fixed parameters (for example, the initial scale factor is 1.6 and the number of scale space layers of each group is 3), and the shape of the industrial part is regular (for example, the geometric contour is clear and the texture is relatively simple), if the traditional SIFT algorithm with default parameters is used, a large number of redundant feature points may be generated.

[0037] Therefore, in order to make the SIFT algorithm more suitable for industrial scenarios, in the embodiment of the present application, when performing feature extraction on the plurality of collected images respectively to obtain a plurality of feature point sets respectively corresponding to the plurality of collected images, an improved SIFT algorithm can be used to perform feature extraction on the plurality of collected images respectively to obtain a plurality of feature point sets respectively corresponding to the plurality of collected images; wherein the initial scale factor of the improved SIFT algorithm is 1.5 and the number of scale space layers of each group of collected images is 2.

[0038] Based on this, the application can make the scale space more dense through a smaller scale factor, more finely capture the local detail features (such as edges, corner points) under the regular shape of the industrial part, and avoid feature omission caused by too large scale jumps. In addition, the application can also reduce the number of scale space layers of each group of collected images to reduce the amount of calculation, while balancing feature multi-scale expression and calculation efficiency by limiting the total number of layers (3 groups). Further, under the regular shape of the industrial part, repeated or invalid features introduced by too many scale layers can be avoided, and feature extraction at key scales can be focused on.

[0039] Step 203: A bidirectional matching algorithm is used to match the feature points of each corresponding feature point set of the plurality of collected images to determine a plurality of matching point pairs.

[0040] In order to improve the efficiency of image stitching, in the embodiments of the application, when performing feature point matching, the geometric shape prior knowledge of the part can be used to regionally constrain the feature points, and only search for matching points within the part contour range. In order to improve the reliability of the matching pairs, in the embodiments of the application, bidirectional matching and random sample consensus verification can also be performed on the feature points.

[0041] Specifically, within the contour range of the target industrial part, for any two collected images, a bidirectional matching method and a random sample consensus (RANSAC) algorithm can be used to sequentially match the feature points of the corresponding feature point sets of the any two collected images, thereby determining a plurality of matching point pairs corresponding to the any two collected images.

[0042] Further, since the bidirectional matching method requires that the feature points of A image have a unique matching point in B image, and the reverse matching of the corresponding points of B image back to A image is also unique, compared to the "one-to-many" ambiguity that may exist in the traditional one-way matching (for example, A image feature point matching to B image), the application can eliminate about 20% of the one-way ambiguous matching points through bidirectional consistency verification, to further improve the reliability of the matching point pairs.

[0043] In addition, in the embodiments of the application, the random sample consensus RANSAC (random sample consensus) algorithm includes random sampling, hypothesis verification and iterative optimization mechanisms to estimate the optimal model parameters from observation data containing a large number of outliers. The execution steps of the random sample consensus RANSAC algorithm are as follows: First, random sampling can be performed, that is, a minimum sample set (for example, at least 4 matching point pairs are required for affine transformation) is randomly selected from the to-be-optimized matching point pairs of any two collected images obtained after bidirectional matching.

[0044] Then, a model can be assumed, i.e., a temporary transformation model can be calculated based on the minimum sample set.

[0045] Next, a screening can be verified, i.e., all the to-be-optimized matching point pairs of any two acquisition images can be verified by using the temporary transformation model to screen the inliers (correct matching points) conforming to the temporary transformation model.

[0046] Finally, iterative optimization can be performed, i.e., through multiple rounds of iteration, the temporary transformation model with the largest number of inliers can be found as the optimal solution, and the minimum sample set corresponding to the optimal temporary transformation model is the multiple matching point pairs corresponding to any two acquisition images. The maximum number of iterations is 300, and the threshold is 2 pixels.

[0047] Step 204: Perform image transformation on the multiple acquisition images according to the multiple matching point pairs to obtain multiple transformed images.

[0048] Since the industrial camera is prone to introduce radial distortion and tangential distortion when shooting images, in the present application, when performing image transformation on the multiple acquisition images according to the multiple matching point pairs to obtain multiple transformed images, the multiple matching point pairs corresponding to any two acquisition images can be sequentially subjected to a cascade operation of distortion correction and affine transformation, so as to transform the two acquisition images into the same coordinate system, thereby obtaining two transformed images corresponding to any two acquisition images, and achieving more comprehensive geometric correction. In the present embodiment, the affine transformation can process linear transformations such as translation, rotation, scaling, and shearing.

[0049] In addition, in the present embodiment, in addition to using "distortion correction + affine transformation" in image transformation, for some industrial application scenarios with extremely high image accuracy requirements and small shooting angle changes, "parallel selection polynomial transformation" can also be used. Specifically, first, control points can be manually labeled on each acquisition image; then, polynomial fitting can be performed according to the labeled control points; next, transformation parameters can be stored; finally, batch image transformation can be performed.

[0050] Based on this, since the polynomial transformation can more flexibly describe the geometric deformation of the image, the present application can realize more accurate image transformation by fitting the control points in the acquisition image, so as to provide better transformation effect in the scene of detecting precision optical instrument parts, so that the accuracy of the size and shape of the spliced image is higher, and the demand for high-precision images is met, providing more accurate image basis for precision industrial production and detection.

[0051] Step 205: Input the multiple transformed images into the trained generator for image stitching to obtain a target spliced image.

[0052] In the embodiments of the present application, the generator is generated based on a generative adversarial network (GAN).

[0053] Specifically, when inputting multiple transformed images into the trained generator to perform image stitching and obtaining a target stitched image, first, for any transformed image, the transformed image can be decomposed into three layers of wavelet coefficients; then, the weights corresponding to the three layers of wavelet coefficients, i.e., a low-frequency layer, a high-frequency layer, and an intermediate-frequency layer, can be determined respectively; next, the three layers of wavelet coefficients and the weights corresponding to the three layers of wavelet coefficients can be weighted and fused (the low-frequency layer highlights the high-energy area, and the high-frequency layer strengthens the edges with consistent directions) to obtain fused wavelet coefficients; then, the inverse wavelet transform can be performed on the fused wavelet coefficients to obtain a reconstructed image corresponding to any transformed image; finally, the reconstructed images corresponding to the multiple transformed images can be input into the trained generator to perform image stitching and eliminate the stitching seams, so as to obtain a high-quality target stitched image combining multi-scale features.

[0054] Therefore, the present application can dynamically adjust the image content (for example, the texture area focuses on low frequency, and the edge area focuses on high frequency) to ensure that the multi-scale features are complementary, thereby improving the stitching smoothness.

[0055] Further, in order to improve the stitching smoothness and obtain a high-quality stitched image, in the embodiments of the present application, when the weights corresponding to the three layers of wavelet coefficients are determined respectively, the weight calculation method of the multi-resolution weighted fusion can also be optimized.

[0056] Specifically, for the low-frequency layer wavelet coefficients, the weight of the low-frequency layer wavelet coefficients can be determined according to the pixel gray variance of the target part region obtained by the region feature energy calculation. The pixel gray variance is used to reflect the energy distribution, and each region energy is normalized to [0, 1]. The higher the energy (rich in texture and details), the higher the weight of the low-frequency layer wavelet coefficients.

[0057] Of course, in the low-frequency layer, in addition to the region feature energy, factors such as the shape complexity of the region can also be considered to adjust the weight, so that the fused image can better balance the fusion effect of different regions while retaining the overall structure. Specifically, first, the region feature energy is calculated as the basic weight; then, the shape complexity features such as the perimeter, the area ratio, and the number of holes can be extracted, normalized, and then a region adjustment factor is generated through a Sigmoid function; finally, the weight of the low-frequency layer wavelet coefficients = basic weight x region adjustment factor. Based on this, if the region adjustment factor > 1, the weight will be enhanced, and if the region adjustment factor < 1, the weight will be attenuated, to realize differentiated weighting.

[0058] For example, in the spliced image of large mechanical equipment, for the key component area with complex shape and the auxiliary component area with regular shape, the weights can be reasonably allocated to highlight the key component details while presenting the overall equipment structure.

[0059] For the high-frequency layer wavelet coefficients, the weights of the high-frequency layer wavelet coefficients can be determined according to the edge gradient direction extracted by the Sobel operator. The edge gradient direction θ = arctan2(Gy, Gx) is quantized to [0, π), Gy is the operator in the vertical direction, and Gx is the operator in the horizontal direction.

[0060] Of course, in the high-frequency layer, in addition to the edge gradient direction, the curvature information of the edge can also be combined to strengthen the consistency of the edge gradient direction, so as to more accurately strengthen the texture and defect details of the part surface and improve the fusion quality. Specifically, first, the consistency of the edge gradient direction is calculated as the basic weight; then, the edge curvature (for example, the extreme value of the second derivative) is extracted and normalized to generate a curvature adjustment factor; finally, the weight of the high-frequency layer wavelet coefficient = basic weight × curvature adjustment factor. Based on this, the high-curvature area (sharp edge) is enhanced in weight (for example, multiplied by 1.2-1.5), and the low-curvature area (smooth edge) is attenuated in weight (for example, multiplied by 0.8-0.5), so as to realize the differential weighting of different edge shapes, more accurately strengthen the texture and defect details of the product surface, and improve the fusion quality.

[0061] For example, in the image fusion of circuit board welding point detection, the fine cracks and virtual welding of the welding points can be clearly presented to improve the identification ability of the product surface quality problems in industrial detection.

[0062] For the intermediate frequency layer wavelet coefficients, the weight of the intermediate frequency layer wavelet coefficients can be determined according to the weight of the low-frequency layer wavelet coefficients and the weight of the high-frequency layer wavelet coefficients. Specifically, the geometric mean of the low-frequency layer weight and the high-frequency layer weight can be taken, or the weight adaptive fusion can be performed according to w_mid = w_low·w_high. Wherein, w_mid is the weight of the intermediate frequency layer wavelet coefficient, w_low is the weight of the low-frequency layer wavelet coefficient, and w_high is the weight of the high-frequency layer wavelet coefficient.

[0063] In one possible implementation, in order to avoid the problems such as misalignment, blur or deformation in the traditional splicing method, in the embodiment of the present application, when the reconstructed images corresponding to the plurality of transformed images are input into the trained generator to perform image splicing to obtain the target spliced image, the generator can be further driven by the adversarial loss to optimize the splicing details (for example, to eliminate the seams and enhance the texture consistency) to improve the global structure rationality.

[0064] Specifically, first, the reconstructed images corresponding to the plurality of transformed images respectively can be input into the trained generator (for generating a spliced image) to perform image splicing, to obtain a to-be-discriminated spliced image; wherein the generator is generated based on a generative adversarial network (GAN); then, the to-be-discriminated spliced image can be input into the trained discriminator (for distinguishing real images from spliced images) to perform discrimination, to determine whether the to-be-discriminated spliced image is a real image; wherein the discriminator is generated based on the generative adversarial network (GAN); finally, if it is determined that the to-be-discriminated spliced image is not a real image, the target spliced image is output; on the contrary, it is determined that the to-be-discriminated spliced image is a real image.

[0065] In the embodiment of the present application, the structure of the generative adversarial network (GAN) adopts an "encoder-decoder" structure. Each layer of the encoder includes a convolutional layer Conv, a batch normalization layer BN and a ReLU activation function; each layer of the decoder includes a transposed convolutional layer TransConv, a batch normalization layer BN and a ReLU activation function, and is connected to the corresponding layer of the encoder by a skip connection (Skip Connection) to preserve the bottom layer detail features.

[0066] Therefore, in the present application, since the generator is generated based on the generative adversarial network (GAN), the generated target spliced image is closer to the real scene in terms of visual effect and industrial detection applicability, and the splicing traces are weakened. For example, in the splicing of building material surface quality detection images, the spliced image is more natural in vision and has no obvious splicing boundary; when detecting defects such as cracks and holes on the surface of the material, the defect features can be more accurately presented, improving the accuracy and reliability of industrial detection, and providing high-quality spliced images for building material quality evaluation.

[0067] In a possible implementation, before image splicing is performed using the generator and the discriminator, the generator and the discriminator also need to be trained.

[0068] Specifically, first, the prepared industrial image dataset is preprocessed (denoising, alignment, etc.); then, the generator (U-Net) and the discriminator (double-branch PatchGAN) are constructed; next, the training can be divided into stages: first, the generator is pre-trained, and then bidirectional adversarial training (combined with adversarial loss, gradient consistency loss, and detection loss) is performed to dynamically adjust the learning rate, and the iteration is performed for 300-800 rounds until convergence, during which real-time data is fine-tuned.

[0069] Based on this, compared with the prior art, the application discards the traditional GAN simple confrontation mode, adopts a bidirectional guided confrontation training algorithm, the generator relies on the feedback of the discriminator, and the discriminator also relies on the transmission of the generator, so that the spliced image not only meets the requirements of industrial detection features, but also highlights the key features of the product. In addition, since the discriminator evaluates the spliced image from the authenticity and industrial detection applicability, the application can also improve the usability of the spliced image in the industrial scene. In a possible implementation, each industrial image in the industrial image data set for training can also be preprocessed.

[0070] Specifically, first, the industrial image can be subjected to bilateral filtering denoising, the parameters are dynamically adjusted according to the local noise of the image, and the details are preserved; then, the denoised industrial image can be standardized to unify the brightness, contrast and color space of different camera images; finally, random rotation, scaling, cropping and other data enhancement operations can be used to simulate camera angle and distance changes, to provide diverse and adaptive high-quality industrial camera image data for training.

[0071] In a possible implementation, the training parameters can also be reasonably set.

[0072] Specifically, the learning rate can be dynamically adjusted, an initial large value is set to help the generative adversarial network (GAN) model converge quickly, and the learning rate is gradually reduced with the change of the loss function to prevent missing the optimal solution, for example, the initial value is set to 0.001, and the learning rate is halved when the loss function decreases by less than 0.01. The number of iterations is determined according to the complexity of the image and the convergence status of the model, and the number of iterations is set to be less for simple scenes and more for complex scenes, to balance the training efficiency and image quality.

[0073] In a possible implementation, the training strategy can also be optimized.

[0074] Specifically, L1 and L2 regularization constraint parameters can be introduced to prevent overfitting and ensure model robustness. Batch normalization is used to normalize the data before the input of the neural network to accelerate convergence and reduce gradient problems. The model is evaluated periodically using industrial detection indicators combined with PSNR and SSIM, and the parameters and strategies are adjusted according to the results to ensure that the model adapts to different industrial scenes and camera images, generates high-quality fused images, and meets the industrial requirements.

[0075] In a possible implementation, to improve the quality of the spliced image, in the embodiment of the application, the target spliced image after splicing can also be cropped to remove the redundant edge portion. In addition, an adaptive sharpening algorithm can also be used to dynamically adjust the sharpening parameters according to the characteristics of the part image, for example, the sharpening intensity factor is set to 0.8.

[0076] Further, the quality of the spliced image can be evaluated by calculating the defect detection accuracy, a peak signal-to-noise ratio (PSNR) value, and a structural similarity (SSIM) value. If the defect detection accuracy is greater than 95%, the PSNR value is greater than 38 dB, and the SSIM value is greater than 0.96, it indicates that the spliced image can clearly display the surface defects of the part and meet the industrial detection requirements. Otherwise, if the industrial detection requirements are not met, the sharpening intensity factor needs to be continuously adjusted.

[0077] In summary, since the bidirectional matching algorithm is used to match the feature points in the respective feature point sets of the plurality of collected images, compared with the prior art, the present application can eliminate ambiguous matching between the feature points through bidirectional consistency verification to improve the reliability of the matched point pairs, thereby further improving the accuracy of image splicing. In addition, since the generator for image splicing is generated based on the generative adversarial network (GAN), compared with the prior art, the target spliced image generated by the present application not only has a more realistic visual effect and industrial detection applicability, reduces the splicing traces, but also greatly improves the efficiency of image splicing and reduces the computational complexity.

[0078] Based on the same inventive concept, an embodiment of the present application provides an image splicing device 30, as shown in the figure, which comprises: Figure 3 An image collection unit 301 is configured to collect images of a target part using an industrial camera to obtain a plurality of collected images. A feature extraction unit 302 is configured to extract features from the plurality of collected images respectively to obtain respective feature point sets of the plurality of collected images. A feature point matching unit 303 is configured to match feature points in the respective feature point sets of the plurality of collected images to determine a plurality of matched point pairs. An image transformation unit 304 is configured to transform the plurality of collected images based on the plurality of matched point pairs to obtain a plurality of transformed images. An image splicing unit 305 is configured to input the plurality of transformed images into a trained generator for image splicing to obtain a target spliced image. The generator is generated based on a generative adversarial network (GAN).

[0079] Optionally, the feature extraction unit 302 is further configured to: extract features from the plurality of collected images respectively using an improved SIFT algorithm to obtain respective feature point sets of the plurality of collected images. The initial scale factor of the improved SIFT algorithm is 1.5, and the number of scale space layers of each set of collected images is 2. ​

[0080] Optionally, the feature point matching unit 303 is further configured to: Within the contour range of the target industrial part, for any two collected images, the feature point matching unit 303 is configured to sequentially perform feature point matching on corresponding feature point sets of the any two collected images by using a bidirectional matching method and a random sample consensus (RANSAC) algorithm, and determine a plurality of matching point pairs corresponding to the any two collected images.

[0081] Optionally, the image transformation unit 304 is further configured to: The image transformation unit 304 is configured to sequentially perform distortion correction and affine transformation on the plurality of matching point pairs corresponding to the any two collected images, and obtain two transformed images corresponding to the any two collected images.

[0082] Optionally, the image stitching unit 305 is further configured to: The image stitching unit 305 is configured to decompose any one of the transformed images into three layers of wavelet coefficients. The image stitching unit 305 is configured to determine weights corresponding to the three layers of wavelet coefficients, respectively. The image stitching unit 305 is configured to perform weighted fusion on the three layers of wavelet coefficients and the weights corresponding to the three layers of wavelet coefficients, respectively, and obtain fused wavelet coefficients. The image stitching unit 305 is configured to perform inverse wavelet transformation on the fused wavelet coefficients, and obtain a reconstructed image corresponding to any one of the transformed images. The image stitching unit 305 is configured to input the reconstructed images corresponding to the plurality of transformed images into a trained generator for image stitching, and obtain a target stitched image.

[0083] Optionally, the image stitching unit 305 is further configured to: The image stitching unit 305 is configured to determine the weight of the low-frequency layer wavelet coefficients according to a pixel gray scale variance of the target part region obtained by the region feature energy calculation. The image stitching unit 305 is configured to determine the weight of the high-frequency layer wavelet coefficients according to an edge gradient direction. The image stitching unit 305 is configured to determine the weight of the intermediate-frequency layer wavelet coefficients according to the weight of the low-frequency layer wavelet coefficients and the weight of the high-frequency layer wavelet coefficients.

[0084] Optionally, the image stitching unit 305 is further configured to: The image stitching unit 305 is configured to input the reconstructed images corresponding to the plurality of transformed images into a trained generator for image stitching, and obtain a to-be-discriminated stitched image. The image stitching unit 305 is configured to input the to-be-discriminated stitched image into a trained discriminator for discrimination, and determine whether the to-be-discriminated stitched image is a real image; the discriminator is generated based on a generative adversarial network (GAN). If it is determined that the to-be-discriminated stitched image is not a real image, the image stitching unit 305 outputs the target stitched image.

[0085] The image stitching apparatus 30 can be configured to perform the method 100. Figure 2The method performed in the illustrated embodiment, and thus the functions that can be achieved by the various functional modules of the image stitching device 30, can be referred to Figure 2 The description of the illustrated embodiment will not be repeated.

[0086] In some possible implementation manners, each aspect of the method provided in the present application can also be implemented in the form of a program element, which includes program codes for causing a computer device to perform the steps of the method according to various exemplary embodiments of the present application described above in the specification when the program element is run on the computer device, for example, the computer device can perform the steps of the method according to various exemplary embodiments of the present application described above in the specification. Figure 2 The method performed in the illustrated embodiment.

[0087] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware, and the foregoing program can be stored in a computer readable storage medium, and the program performs the steps including the above-mentioned method embodiments when executed; and the foregoing storage medium includes mobile storage devices, read-only memories (ROMs), random access memories (RAMs), magnetic discs or optical discs and various media that can store program codes. Or, when the above-mentioned integrated units are implemented in the form of software functional modules and sold or used as independent parts, they can also be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application can be embodied in the form of software parts, which are stored in a storage medium and include a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the methods described in the various embodiments of the present application. The foregoing storage medium includes mobile storage devices, ROMs, RAMs, magnetic discs or optical discs and various media that can store program codes.

[0088] Although the preferred embodiments of the present application have been described, those skilled in the art can make additional changes and modifications to the embodiments once they know the basic inventive concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0089] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. An image stitching method, characterized in that: The method comprises: Use industrial cameras to capture images of target parts and obtain multiple captured images; Performing feature extraction on the multiple acquired images respectively to obtain feature point sets corresponding to the multiple acquired images; Performing feature point matching on the feature point sets corresponding to the plurality of collected images using a bidirectional matching algorithm to determine a plurality of matching point pairs; performing image transformation on the plurality of collected images according to the plurality of matching point pairs to obtain a plurality of transformed images; The multiple transformed images are input into a trained generator for image stitching to obtain a target stitched image; wherein the generator is generated based on a generative adversarial network (GAN).

2. The method according to claim 1, wherein The step of performing feature extraction on the plurality of collected images to obtain feature point sets corresponding to the plurality of collected images includes: An improved SIFT algorithm is used to perform feature extraction on the multiple acquired images to obtain feature point sets corresponding to the multiple acquired images; wherein the initial scale factor of the improved SIFT algorithm is 1.5, and the number of scale space layers of each group of acquired images is 2.

3. The method according to claim 1, wherein The step of performing feature point matching on the feature point sets corresponding to the plurality of collected images using a bidirectional matching algorithm to determine a plurality of matching point pairs includes: Within the contour range of the target industrial part, for any two acquired images, the bidirectional matching method and the random sampling consistency RANSAC algorithm are used to perform feature point matching on the corresponding feature point sets of any two acquired images in turn to determine multiple matching point pairs corresponding to the any two acquired images.

4. The method according to claim 3, wherein The step of performing image transformation on the plurality of collected images according to the plurality of matching point pairs to obtain a plurality of transformed images comprises: Distortion correction and affine transformation are sequentially performed on a plurality of matching point pairs corresponding to any two collected images to obtain two transformed images corresponding to the any two collected images.

5. The method according to claim 1, wherein The step of inputting the multiple transformed images into a trained generator for image stitching to obtain a target stitched image comprises: For any transformed image, decomposing the transformed image into three layers of wavelet coefficients; Determine the weights corresponding to the three layers of wavelet coefficients respectively; Performing weighted fusion on the three layers of wavelet coefficients and the weights corresponding to the three layers of wavelet coefficients to obtain fused wavelet coefficients; Performing inverse wavelet transform on the fused wavelet coefficients to obtain a reconstructed image corresponding to any one of the transformed images; The reconstructed images corresponding to the multiple transformed images are input into a trained generator for image stitching to obtain a target stitched image.

6. The method according to claim 5, wherein The step of respectively determining the weights corresponding to the three layers of wavelet coefficients comprises: For the low-frequency layer wavelet coefficients, determining the weights of the low-frequency layer wavelet coefficients according to the pixel grayscale variance of the target part area obtained by calculating the regional characteristic energy; For the high-frequency layer wavelet coefficient, determining the weight of the high-frequency layer wavelet coefficient according to the edge gradient direction; For the wavelet coefficients of the intermediate frequency layer, the weights of the wavelet coefficients of the intermediate frequency layer are determined according to the weights of the wavelet coefficients of the low frequency layer and the weights of the wavelet coefficients of the high frequency layer.

7. The method according to claim 5, wherein The step of inputting the reconstructed images corresponding to the plurality of transformed images into a trained generator for image stitching to obtain a target stitched image comprises: Inputting the reconstructed images corresponding to the plurality of transformed images into the trained generator for image stitching to obtain a stitched image to be judged; Inputting the spliced ​​image to be judged into a trained discriminator for discrimination to determine whether the spliced ​​image to be judged is a real image; wherein the discriminator is generated based on a generative adversarial network (GAN); If it is determined that the stitched image to be determined is not a real image, the target stitched image is output.

8. An image stitching device, characterized in that: The device comprises: An image acquisition unit is used to acquire images of target parts using an industrial camera to obtain multiple acquired images; A feature extraction unit, configured to perform feature extraction on each of the plurality of acquired images to obtain feature point sets corresponding to each of the plurality of acquired images; A feature point matching unit, configured to perform feature point matching on the plurality of collected images according to the feature point sets corresponding to the plurality of collected images, and determine a plurality of matching point pairs; an image transformation unit, configured to perform image transformation on the plurality of collected images according to the plurality of matching point pairs to obtain a plurality of transformed images; An image stitching unit is used to input the multiple transformed images into a trained generator for image stitching to obtain a target stitched image; wherein the generator is generated based on a generative adversarial network (GAN).

9. An electronic device, characterized in that: The device comprises: a memory for storing program instructions; A processor is configured to call program instructions stored in the memory and execute the method according to any one of claims 1 to 7 according to the obtained program instructions.

10. A storage medium, characterized in that: The storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the method according to any one of claims 1 to 7.

Images