Image intelligent stitching method and stitching system based on deep learning

Abstract

The present invention provides an image intelligent splicing method based on deep learning and a splicing system thereof, which belongs to the field of image processing technology. By constructing a deep convolutional neural network model, feature extraction and feature encoding are performed on each image to be spliced, and feature representation and context encoding of the image block are obtained. Then, the fusion weights of adjacent images to be spliced ​​are calculated in combination with the feature representations and context encodings of adjacent images to be spliced. Based on the fusion weights, adjacent images to be spliced ​​are fused using a gating mechanism to obtain a preliminary spliced ​​image. Finally, a panoramic optimization generation network is used to iteratively optimize and enhance the preliminary spliced ​​image to obtain a final spliced ​​image. The present invention can more accurately match and fuse the overlapping areas of adjacent images, thereby significantly improving the precision of image splicing, reducing splicing gaps and misalignment phenomena, and improving the clarity and contrast of the image, making the spliced ​​image more natural and realistic.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image intelligent splicing method and a splicing system thereof based on deep learning. Background Art

[0002] With the rapid development of deep learning technology, image stitching technology has been widely used in the fields of computer vision and image processing. The main goal of image stitching technology is to stitch multiple images with overlapping areas into a panoramic image to meet users' needs for a wider field of view or higher resolution images.

[0003] Traditional image stitching methods mainly rely on feature point matching and image transformation models. Although they have achieved certain results, they often have problems such as low matching accuracy and obvious stitching gaps when dealing with complex scenes, large viewing angle changes, and illumination changes. Therefore, it is of great theoretical and practical significance to study more robust and efficient image stitching methods.

[0004] Therefore, it is necessary to provide an image intelligent stitching method and a stitching system based on deep learning to solve the above technical problems. Summary of the invention

[0005] To solve the above technical problems, the present invention provides an image intelligent stitching method and a stitching system based on deep learning, which can more accurately match and fuse the overlapping areas of adjacent images, thereby significantly improving the accuracy of image stitching, reducing stitching gaps and misalignment, and improving the clarity and contrast of the image, making the stitched image more natural and realistic.

[0006] The present invention provides an image intelligent splicing method based on deep learning, and the splicing method comprises the following steps:

[0007] S1: Extract features of each image to be stitched based on a pre-built deep convolutional neural network, and perform feature encoding on the extracted features to obtain feature representation and context encoding of each image block;

[0008] S2: Determine adjacent images to be stitched, and calculate fusion weights of the adjacent images to be stitched by combining feature representations and context encoding of the adjacent images to be stitched;

[0009] S3: Based on the fusion weight, adjacent images to be stitched are fused using a gating mechanism to obtain a preliminary stitched image;

[0010] S4: Utilize a panoramic optimization generation network to iteratively optimize and enhance the preliminary stitched image to obtain a stitched image.

[0011] Preferably, step S1 specifically includes the following steps:

[0012] S101: Obtaining a set of images to be stitched I[I 1 ,I 2 ,……,I n ], where n is the number of images to be stitched, and the images to be stitched are preprocessed;

[0013] S102: Using a pre-built deep convolutional neural network to extract the pre-processed image to be stitched I n The features of each image to be stitched are obtained;

[0014] S103: Encode the features in the feature set of each image to be spliced ​​based on the global average pooling method to obtain the context code C corresponding to each image to be spliced i .

[0015] Preferably, step S2 specifically includes the following steps:

[0016] S201: Acquire adjacent images to be stitched based on a predefined stitching order;

[0017] S202: determining the positional relationship between adjacent images to be stitched together by using feature representations of the adjacent images to be stitched together;

[0018] S203: Calculate the fusion weight of adjacent images to be spliced ​​based on the positional relationship of adjacent images to be spliced ​​and context coding, wherein the calculation formula of the fusion weight is:

[0019] W i =f(Similarity)

[0020] Among them, W i is the fusion weight of the i-th image to be spliced, C i and C i-1 They represent the context encoding of adjacent images to be stitched, Similarity(·) is the similarity function, and Position(·) is the position influence function.

[0021] Preferably, step S3 specifically includes the following steps:

[0022] S301: normalizing the calculated fusion weights of all adjacent images to be stitched, and the normalization formula is:

[0023]

[0024] Among them, W i ' is the normalized fusion weight, j is the number of adjacent images to be stitched;

[0025] S302: Apply the Sigmoid conversion function to perform nonlinear transformation on the normalized fusion weights to obtain a gating signal corresponding to the fusion weights, wherein the conversion formula of the gating signal is:

[0026]

[0027] Among them, g(W i ') is the gate signal, e is the base, β is the steepness, and θ is the offset;

[0028] S303: adjusting the feature representations of all adjacent images to be stitched based on the gate signal to obtain adjusted feature representations of all adjacent images to be stitched, wherein the adjustment formula is:

[0029] F i '=g(W i ')·F i

[0030] Among them, F' i is the adjusted feature representation, F i is the feature representation before adjustment;

[0031] S304: performing weighted averaging on the adjusted feature representations of all adjacent images to be stitched together to generate a preliminary stitched image.

[0032] Preferably, step S4 specifically includes the following steps:

[0033] S401: inputting the preliminary stitching image into the panoramic optimization generation network, and extracting features from the preliminary stitching image;

[0034] S402: using a gradient descent algorithm to adjust the features extracted from the preliminary stitched image to obtain a feature map corresponding to the preliminary stitched image;

[0035] S403: Iteratively train the feature map using a generative adversarial network to obtain a spliced ​​image.

[0036] The present invention also provides an image intelligent splicing system based on deep learning, which is applied to an image intelligent splicing method based on deep learning. The splicing system includes:

[0037] A feature extraction and encoding module is used to extract features from each image to be spliced ​​based on a pre-built deep convolutional neural network, and to perform feature encoding on the extracted features to obtain feature representation and context encoding of each image block;

[0038] A fusion weight calculation module is used to determine adjacent images to be spliced, and calculate the fusion weights of the adjacent images to be spliced ​​by combining the feature representation and context coding of the adjacent images to be spliced;

[0039] A preliminary stitching image acquisition module, used for fusing adjacent images to be stitched using a gating mechanism based on the fusion weight to obtain a preliminary stitching image;

[0040] The stitched image generation module is used to iteratively optimize and enhance the preliminary stitched image by using a panoramic optimization generation network to obtain a stitched image.

[0041] Compared with the related art, the image intelligent splicing method and splicing system based on deep learning provided by the present invention have the following beneficial effects:

[0042] The present invention constructs a deep convolutional neural network model to extract and encode features of each image to be spliced, and obtains feature representation and context encoding of the image block. Then, the fusion weights of adjacent images to be spliced ​​are calculated in combination with the feature representations and context encodings of adjacent images to be spliced. Based on the fusion weights, the adjacent images to be spliced ​​are fused using a gating mechanism to obtain a preliminary spliced ​​image. Finally, the preliminary spliced ​​image is iteratively optimized and enhanced using a panoramic optimization generation network to obtain a final spliced ​​image. The present invention can more accurately match and fuse the overlapping areas of adjacent images, thereby significantly improving the accuracy of image splicing, reducing splicing gaps and misalignment, and improving the clarity and contrast of the image, making the spliced ​​image more natural and realistic. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A flowchart of the deep learning-based image intelligent splicing method provided by the present invention;

[0044] Figure 2 This is a schematic diagram of the module structure of the deep learning-based image intelligent stitching system provided by the present invention. DETAILED DESCRIPTION

[0045] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only the parts related to the present invention, rather than all structures, are shown in the accompanying drawings. In addition, the embodiments of the present invention and the features in the embodiments may be combined with each other without conflict.

[0046] It should also be noted that, for ease of description, only the part relevant to the present invention but not all the content is shown in the accompanying drawings. It should be mentioned before discussing the exemplary embodiments in more detail that some exemplary embodiments are described as the processing or method described as a flow chart. Although the flow chart describes each operation (or step) as being processed sequentially, many operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. When its operation is completed, the processing can be terminated, but it can also have additional steps not included in the accompanying drawings. The processing can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0047] Embodiment 1

[0048] The image intelligent splicing method based on deep learning provided by the present invention refers to Figure 1 As shown, the splicing method includes the following steps:

[0049] S1: Based on the pre-built deep convolutional neural network, features are extracted for each image to be spliced, and the extracted features are encoded to obtain feature representation and context encoding of each image block.

[0050] In this embodiment, first, each image to be stitched is input into a pre-trained deep convolutional neural network, which includes multiple convolutional layers and pooling layers to extract high-level feature information such as edges, textures, color distribution, etc. from the image to be stitched.

[0051] After feature extraction is completed, it is further processed by the autoencoder to obtain the feature representation of the image to be stitched. Then, these feature representations are processed using context coding technology to retain the information of a single image to be stitched, while increasing the influence of the surrounding environment to maintain spatial continuity and consistency.

[0052] Through feature extraction and encoding, each image to be stitched is converted into a high-dimensional and representative vector. At the same time, context encoding ensures that the features not only describe local details, but also contain contextual information of the surrounding environment, which helps subsequent seamless stitching.

[0053] S2: Determine adjacent images to be stitched, and calculate fusion weights of the adjacent images to be stitched by combining feature representations and context encoding of the adjacent images to be stitched.

[0054] In this embodiment, for each pair of adjacent images to be stitched, their feature representation and context coding are comprehensively considered, and based on the feature representation and context coding, a fusion weight for fusing the two images is calculated by a formula. The calculated fusion weight can ensure that when stitching, the transition of the image edges is more natural, reduce discontinuity or misalignment, and improve the overall visual quality of the stitched image.

[0055] S3: Based on the fusion weight, adjacent images to be stitched are fused using a gating mechanism to obtain a preliminary stitched image.

[0056] In this embodiment, the calculated fusion weights are used to control the flow of information through a gating mechanism to achieve fusion of adjacent images. Since the gating mechanism allows the network to selectively fuse information, such as emphasizing certain features and suppressing other inconsistent features, a better stitching effect is achieved to generate a preliminary stitched image, in which the boundaries of adjacent images to be stitched have been smoothly transitioned according to the calculated weights, thereby achieving image fusion based on content adaptability, avoiding obvious seams that may be caused by simple stitching, and improving the coherence and authenticity of the stitched image.

[0057] S4: Utilize a panoramic optimization generation network to iteratively optimize and enhance the preliminary stitched image to obtain a stitched image.

[0058] In this embodiment, the preliminary stitched image is taken as input and sent to a panoramic optimization generation network (a network based on GANs). The network consists of two parts: a generator and a discriminator. The generator attempts to generate more realistic stitched images, while the discriminator attempts to distinguish between real images and generated images. The two promote each other through adversarial training. During the iteration process, the network continuously adjusts the generated image to improve the overall quality and realism while maintaining content coherence until it converges to an optimal solution, which significantly improves the global consistency of the stitched image, including color balance, lighting matching and texture connection. The final image has almost no stitching traces, as if it was shot in a single continuous scene.

[0059] Specifically, step S1 specifically includes the following steps:

[0060] S101: Obtaining a set of images to be stitched I[I 1 ,I 2 ,……,I n ], where n is the number of images to be stitched, and the images to be stitched are preprocessed.

[0061] In this embodiment, all images to be stitched are collected from different sources and ensured to belong to different parts of the same scene, and then conventional preprocessing is performed, which includes but is not limited to resizing, grayscale correction and white balance adjustment, and denoising; wherein, resizing can ensure that all images have the same size, which is convenient for subsequent operations; grayscale correction and white balance adjustment make the color and brightness of each image consistent; and noise removal helps to improve image quality.

[0062] Through preprocessing, unnecessary differences between the images to be stitched are eliminated, providing a good data basis for subsequent feature extraction and improving the quality of feature representation and the accuracy of stitching.

[0063] S102: Using a pre-built deep convolutional neural network to extract the pre-processed image to be stitched I n features, and obtain the feature sets of each image to be stitched.

[0064] In this embodiment, the preprocessed images to be stitched are input into a pre-trained deep convolutional neural network. This network architecture includes multiple convolutional layers and pooling layers, where the convolutional layer is used to capture local features, and the pooling layer is used to reduce the dimension and retain important information. The extracted features reflect the main structure, texture, color distribution and other information of the image, providing rich information for subsequent feature encoding and stitching decisions.

[0065] S103: Encode the features in the feature set of each image to be spliced ​​based on the global average pooling method to obtain the context code C corresponding to each image to be spliced i .

[0066] In this embodiment, global average pooling is applied to convert the feature map output by the convolutional layer into a vector of fixed length while retaining important global information, which is beneficial to subsequent context encoding and fusion processing.

[0067] Specifically, a global average pooling operation is applied to the feature set of each image to be stitched, that is, all values ​​of each channel of the feature map are averaged to obtain a vector of fixed length. These vectors are the context encodings of the corresponding images, which carry the overall feature information of the image. At the same time, the averaging operation reduces unnecessary details and highlights the main features of the image. At the same time, the global average pooling operation not only reduces the dimension of feature representation and speeds up subsequent processing, but also strengthens the overall features of the image, which is conducive to maintaining spatial continuity and consistency.

[0068] Specifically, step S2 specifically includes the following steps:

[0069] S201: Acquire adjacent images to be stitched based on a predefined stitching order.

[0070] In this embodiment, before image stitching is performed, it is crucial to clarify which image blocks are adjacent and arranged in the correct order. The predefined stitching order can ensure that the system processes images in an orderly and efficient manner, avoiding confusion and incorrect alignment caused by disordered stitching.

[0071] Specifically, the predefined stitching order is the motion trajectory or coordinates of the image acquisition device, thereby determining the order of the images to be stitched, ensuring that adjacent images are correctly identified and sorted, ensuring the correct alignment of images in subsequent processing, simplifying the matching complexity in the stitching process, and improving stitching efficiency and accuracy.

[0072] S202: Determine the positional relationship between adjacent images to be stitched together by using feature representations of the adjacent images to be stitched together.

[0073] In this embodiment, accurate understanding of the positional relationship between adjacent images is crucial to achieving seamless stitching, which involves not only the continuity of image content but also the consistency of geometric transformations (rotation, scaling, translation) to ensure the realism of the stitched scene.

[0074] Specifically, the feature representations of adjacent images to be stitched (especially key features such as edges and textures) are used for comparison, and the key feature points are detected and described in the adjacent images to be stitched using the SIFT feature matching algorithm. These feature points are then matched between the adjacent images to find the corresponding relationship, and the RANSAC algorithm is used to estimate reliable geometric transformation parameters from the matching points, including rotation and translation information, so as to accurately determine the positional relationship between the images and provide an accurate alignment basis for subsequent seamless stitching.

[0075] S203: Calculate the fusion weight of adjacent images to be spliced ​​based on the positional relationship of adjacent images to be spliced ​​and context coding, wherein the calculation formula of the fusion weight is:

[0076] W i =f(Similarity)

[0077] Among them, W i is the fusion weight of the i-th image to be spliced, C i and C i-1 They represent the context encoding of adjacent images to be stitched, Similarity(·) is the similarity function, and Position(·) is the position influence function.

[0078] In this embodiment, in order to achieve a smooth transition when stitching adjacent images, especially at image edges and areas with large feature changes, it is necessary to fine-tune the weights according to the similarity and positional relationship of the image content to achieve the best stitching effect.

[0079] Specifically, the fusion weight is obtained through the calculation formula of the fusion weight, which ensures that the transition of the stitching area is more natural, reduces the discontinuity and visual abruptness caused by direct stitching, and improves the visual fluency and realism of the entire stitching image.

[0080] Specifically, step S3 specifically includes the following steps:

[0081] S301: normalizing the calculated fusion weights of all adjacent images to be stitched, and the normalization formula is:

[0082]

[0083] Among them, W i ' is the normalized fusion weight, j is the number of adjacent images to be stitched.

[0084] In this embodiment, normalization processing can ensure that the sum of the fusion weights of all adjacent images to be stitched is 1, which can ensure the rationality of weight distribution in the fusion process, avoid excessive weight of a certain part causing other parts of information to be ignored, and ensure that all image blocks can contribute fairly to the final stitching result according to their importance. By dividing by the sum of weights, each fusion weight is scaled to between 0 and 1, achieving balanced weight distribution.

[0085] S302: Apply the Sigmoid conversion function to perform nonlinear transformation on the normalized fusion weights to obtain a gating signal corresponding to the fusion weights, wherein the conversion formula of the gating signal is:

[0086]

[0087] Among them, g(W i ') is the gate signal, e is the base, β is the steepness, and θ is the offset.

[0088] In this embodiment, the Sigmoid function is applied to convert the normalized linear weights into nonlinear gating signals. The output range of the Sigmoid function is between (0, 1), which is suitable for use as a control signal because it can smoothly control the opening and closing of the information flow and enhance the control accuracy of feature fusion, especially for edges and transition areas that require fine adjustment.

[0089] Specifically, a higher β value makes the change of the gated signal more sensitive, and the setting can adjust the center position of the gated signal to adapt to different fusion requirements.

[0090] S303: adjusting the feature representations of all adjacent images to be stitched based on the gate signal to obtain adjusted feature representations of all adjacent images to be stitched, wherein the adjustment formula is:

[0091] F i '=g(W i ')·F i

[0092] Among them, F i ' is the adjusted feature representation, F i is the feature representation before adjustment.

[0093] In this embodiment, the selective enhancement or weakening of features is achieved through element-by-element multiplication of feature representation and gating signal, so as to dynamically adjust the feature strength of the image block according to the gating signal, emphasize or suppress certain features, and better match the transition area between adjacent images. The gating mechanism provides flexibility, so that the fusion process can be more adaptable to changes in image content and reduce unnatural transitions.

[0094] S304: performing weighted averaging on the adjusted feature representations of all adjacent images to be stitched together to generate a preliminary stitched image.

[0095] In this embodiment, the adjusted feature representation can better reflect the need for fusion between image blocks. The weighted average is to merge these adjusted features according to the instructions of their gating signals (i.e., fusion weights) to form a comprehensive feature representation, thereby generating a preliminary stitched image. This process ensures the coherence and overall consistency of the image content.

[0096] Specifically, step S4 specifically includes the following steps:

[0097] S401: Inputting a preliminary stitched image into the panoramic optimization generation network, and extracting features from the preliminary stitched image.

[0098] In this embodiment, before entering the optimization process of the generative adversarial network GANs, it is first necessary to input the preliminary stitched image into the panoramic optimization generation network and re-extract features therefrom. Since the preliminary stitched image may still have some subtle discontinuities or unnatural transitions, feature re-extraction can provide more detailed basic information for the subsequent optimization process, ensuring that the network can understand the image content and structure from a more updated perspective.

[0099] Specifically, the preliminary stitched image, as the input of the panoramic optimization generation network, will first enter a feature extraction layer, which contains multiple convolutional layers to capture deeper features from the preliminary stitched image, especially those related to global consistency, such as large-scale color harmony, texture consistency, etc. This process is similar to the feature extraction in S1, but is aimed at the stitched image, aiming to further refine and adjust the existing feature representation.

[0100] S402: Using a gradient descent algorithm to adjust the features extracted from the preliminary stitched image to obtain a feature map corresponding to the preliminary stitched image.

[0101] In this embodiment, the extracted features are modified in a fine-grained manner through a gradient descent algorithm to make them more conducive to generating a high-quality final stitched image, thereby ensuring that the feature map can reflect the image features that the generator needs to strengthen or weaken in order to produce more natural results in the subsequent generation process.

[0102] Specifically, using the gradient descent algorithm, the network calculates the gradient of the loss function with respect to the feature map, and then backpropagates based on this gradient to gradually adjust the weights in the feature map so that it changes in the direction of reducing the loss function. This is actually fine-tuning at the feature level, optimizing the image's detailed performance in terms of color, texture, etc., providing a more optimized starting point for the generator.

[0103] S403: Iteratively train the feature map using a generative adversarial network to obtain a spliced ​​image.

[0104] In this embodiment, through iterative training of generative adversarial networks (GANs), the adversarial mechanism between the generator and the discriminator can further improve the quality and realism of the image. The generator attempts to generate spliced ​​images that are difficult to distinguish from real images, while the discriminator is committed to accurately identifying which are real and which are generated by the generator. This mechanism forces the generator to continuously improve until the generated images meet very high standards in all aspects.

[0105] Specifically, in each iteration, the generator generates a more optimized version of the stitched image based on the adjusted feature map, which is then sent to the discriminator for evaluation. The discriminator outputs a probability value indicating how "real" the image looks. Based on the feedback from the discriminator, the parameters of the generator are fine-tuned through back propagation and gradient descent to generate an image that is closer to reality in the next round of iteration. This process is repeated until the quality of the stitched image is no longer significantly improved, that is, it reaches a convergence state, and finally a stitched image is obtained.

[0106] The working principle of the image intelligent stitching method based on deep learning provided by the present invention is as follows: the present invention constructs a deep convolutional neural network model to extract and encode features of each image to be stitched, and obtains feature representation and context encoding of the image block. Then, the fusion weights of adjacent images to be stitched are calculated in combination with the feature representation and context encoding of adjacent images to be stitched. Based on the fusion weights, the adjacent images to be stitched are fused using a gating mechanism to obtain a preliminary stitched image. Finally, the preliminary stitched image is iteratively optimized and enhanced using a panoramic optimization generation network to obtain a final stitched image. The present invention can more accurately match and fuse the overlapping areas of adjacent images, thereby significantly improving the accuracy of image stitching, reducing stitching gaps and misalignment, and improving the clarity and contrast of the image, making the stitched image more natural and realistic.

[0107] Embodiment 2

[0108] The present invention also provides an image intelligent splicing system based on deep learning, which is applied to an image intelligent splicing method based on deep learning, referring to Figure 2 As shown, the splicing system includes:

[0109] A feature extraction and encoding module is used to extract features from each image to be spliced ​​based on a pre-built deep convolutional neural network, and to perform feature encoding on the extracted features to obtain feature representation and context encoding of each image block;

[0110] A fusion weight calculation module is used to determine adjacent images to be spliced, and calculate the fusion weights of the adjacent images to be spliced ​​by combining the feature representation and context coding of the adjacent images to be spliced;

[0111] A preliminary stitching image acquisition module, used for fusing adjacent images to be stitched using a gating mechanism based on the fusion weight to obtain a preliminary stitching image;

[0112] The stitched image generation module is used to iteratively optimize and enhance the preliminary stitched image by using a panoramic optimization generation network to obtain a stitched image.

[0113] The working principle of the image intelligent splicing system based on deep learning provided by the present invention is as follows: a deep convolutional neural network model is constructed through a feature extraction and encoding module, and feature extraction and feature encoding are performed on each image to be spliced ​​to obtain feature representation and context encoding of the image block; then, the fusion weight of the adjacent images to be spliced ​​is calculated by combining the feature representation and context encoding of the adjacent images to be spliced ​​through a fusion weight calculation module; then, based on the fusion weight, the adjacent images to be spliced ​​are fused through the gating mechanism of the preliminary splicing image acquisition module to obtain a preliminary splicing image; finally, the preliminary splicing image is iteratively optimized and enhanced by using the panoramic optimization generation network in the splicing image generation module to obtain a final splicing image. The present invention can more accurately match and fuse the overlapping areas of adjacent images, thereby significantly improving the accuracy of image splicing, reducing splicing gaps and misalignment phenomena, and improving the clarity and contrast of the image, so that the spliced ​​image is more natural and realistic.

[0114] The present application is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0115] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0116] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device that includes a series of elements includes not only those elements, but also other elements that are not explicitly listed, or includes elements that are inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device that includes the elements.

Claims

1. The image intelligent splicing method based on deep learning is characterized by: The splicing method includes the following steps: S1: Extract features of each image to be stitched based on a pre-built deep convolutional neural network, and perform feature encoding on the extracted features to obtain feature representation and context encoding of each image block; S2: Determine adjacent images to be stitched, and calculate fusion weights of the adjacent images to be stitched by combining feature representations and context encoding of the adjacent images to be stitched; S3: Based on the fusion weight, adjacent images to be stitched are fused using a gating mechanism to obtain a preliminary stitched image; S4: using a panoramic optimization generation network to iteratively optimize and enhance the preliminary stitched image to obtain a stitched image; Wherein, step S3 specifically includes the following steps: S301: normalizing the calculated fusion weights of all adjacent images to be stitched, and the normalization formula is: Among them, W' i is the normalized fusion weight, j is the number of adjacent images to be stitched; S302: Apply the Sigmoid conversion function to perform nonlinear transformation on the normalized fusion weights to obtain a gating signal corresponding to the fusion weights, wherein the conversion formula of the gating signal is: Among them, g(W' i ) is the gating signal, e is the base, β is the steepness, and θ is the offset; S303: adjusting the feature representations of all adjacent images to be stitched based on the gate signal to obtain adjusted feature representations of all adjacent images to be stitched, wherein the adjustment formula is: F’ i =g(W’ i )·F i Among them, F' i is the adjusted feature representation, F i is the feature representation before adjustment; S304: performing weighted averaging on the adjusted feature representations of all adjacent images to be stitched together to generate a preliminary stitched image.

2. The image intelligent splicing method based on deep learning according to claim 1 is characterized in that: Step S1 specifically includes the following steps: S101: Obtain the image set to be stitched I[I1,I2,...,I n ], where n is the number of images to be stitched, and the images to be stitched are preprocessed; S102: Using a pre-built deep convolutional neural network to extract the pre-processed image to be stitched I n The features of each image to be stitched are obtained; S103: Encode the features in the feature set of each image to be spliced ​​based on the global average pooling method to obtain the context code C corresponding to each image to be spliced i .

3. The image intelligent splicing method based on deep learning according to claim 2 is characterized in that: Step S2 specifically includes the following steps: S201: Acquire adjacent images to be stitched based on a predefined stitching order; S202: determining the positional relationship between adjacent images to be stitched together by using feature representations of the adjacent images to be stitched together; S203: Calculate the fusion weights of the adjacent images to be spliced ​​by combining the positional relationship of the adjacent images to be spliced ​​and the context coding.

4. The image intelligent splicing method based on deep learning according to claim 3 is characterized in that: Step S4 specifically includes the following steps: S401: inputting the preliminary stitching image into the panoramic optimization generation network, and extracting features from the preliminary stitching image; S402: using a gradient descent algorithm to adjust the features extracted from the preliminary stitched image to obtain a feature map corresponding to the preliminary stitched image; S403: Iteratively train the feature map using a generative adversarial network to obtain a spliced ​​image.

5. A deep learning-based intelligent image stitching system, applied to a deep learning-based intelligent image stitching method as claimed in any one of claims 1 to 4, characterized in that: The splicing system includes: A feature extraction and encoding module is used to extract features from each image to be spliced ​​based on a pre-built deep convolutional neural network, and to perform feature encoding on the extracted features to obtain feature representation and context encoding of each image block; A fusion weight calculation module is used to determine adjacent images to be spliced, and calculate the fusion weights of the adjacent images to be spliced ​​by combining the feature representation and context coding of the adjacent images to be spliced; A preliminary stitching image acquisition module, used for fusing adjacent images to be stitched using a gating mechanism based on the fusion weight to obtain a preliminary stitching image; The stitched image generation module is used to iteratively optimize and enhance the preliminary stitched image by using a panoramic optimization generation network to obtain a stitched image.

Images