Image stitching processing method and device

Abstract

The invention discloses an image splicing processing method and device, which relate to the technical field of image processing, and include receiving a group of images to be spliced, correcting them, extracting the feature point positions of the corrected images and their corresponding sub-vectors, calculating the similarity scores between the feature points of different images, matching the feature points, establishing an image combination model based on the matched feature points, and obtaining a preliminary panoramic image; the invention uses the Euclidean distance to calculate the distance of feature description vectors, combines the two-way matching verification and the RANSAC algorithm to perform geometric consistency screening, effectively eliminating the possibility of mismatching, and finally adopts reinforcement learning to ensure that the matching pairs finally retained have high credibility.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image splicing processing method and device. Background Art

[0002] In the field of computer vision and image processing, image stitching technology, as an important means of information fusion, has been widely studied and developed in recent years. With the advancement of digital photography technology and the popularity of smart devices, users can easily obtain a large number of high-resolution images. However, a single image often cannot fully capture complex scenes or a wide field of view, which has prompted the need to stitch multiple images to form a panorama.

[0003] Existing image stitching solutions still have several limitations. First, for feature point matching, traditional methods usually only consider local geometric relationships, ignoring the influence of global contextual information, which easily leads to incorrect matching; in addition, in the process from preliminary panorama to final output, there is a lack of effective optimization mechanism to solve problems such as boundary discontinuity and uneven brightness, which affects the quality of the final generated image. Summary of the invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides an image stitching processing method to solve the problem that in the process from preliminary panoramic image to final output, there is a lack of effective optimization mechanism to solve problems such as discontinuous boundaries and uneven brightness, which affects the quality of the final generated image.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In a first aspect, the present invention provides an image stitching processing method, which comprises:

[0008] Receive a set of images to be stitched and perform correction;

[0009] Extract the feature point positions and their corresponding sub-vectors of the rectified image;

[0010] Calculate the similarity scores between feature points of different images and match the feature points;

[0011] An image combination model is established based on the matched feature points to obtain a preliminary panoramic image;

[0012] Based on the physical model, geometric correction is performed on the component images in the preliminary panorama to obtain a set of stitched images;

[0013] Perform global optimization and stitching on the stitched image set to output a complete panorama.

[0014] As a preferred solution of the image stitching processing method of the present invention, a group of images to be stitched are received and corrected, which specifically includes the following steps:

[0015] Perform white balance correction and color adjustment on images;

[0016] Perform histogram equalization and denoising filtering on the color adjusted image.

[0017] As a preferred solution of the image stitching processing method of the present invention, wherein: extracting the feature point positions of the rectified image and their corresponding sub-vectors specifically includes the following steps:

[0018] Normalize the rectified image and evaluate the content complexity of the local area in the image;

[0019] Adjust the density and scale of feature point extraction according to the content complexity;

[0020] Input the standardized image into a deep convolutional neural network to obtain a feature response map;

[0021] Extract the local maximum value from the feature response map as the feature point to form a set of feature points for each image;

[0022] A receptive field of fixed size is extracted around the feature points of each image, and a feature description vector of fixed dimension is generated based on the receptive field, and the feature description vectors of all images are integrated into a feature description vector set.

[0023] As a preferred solution of the image stitching processing method of the present invention, wherein: calculating the similarity scores between the feature points of different images and matching the feature points specifically includes the following steps:

[0024] The Euclidean distance is used to calculate the distance of the feature description vector of each image, and each feature point is matched according to the distance. The expression is:

[0025] ;

[0026] in, Represents the feature description vector and the feature description vector The Euclidean distance of Representing images Middle The feature description vector of feature points, Representing images Middle The feature description vector of feature points, represents the dimension of the feature description vector, Represents the feature description vector No. Dimensional component, Represents the feature description vector No. Dimensional component;

[0027] For the feature description vector of each feature point in the image, find the nearest neighbor matching point in the feature description vector set of other images based on the Euclidean distance;

[0028] Perform two-way matching verification to confirm whether the current feature point and the feature point of the current nearest neighbor matching point are each other's nearest neighbors, and obtain a preliminarily screened feature point matching pair;

[0029] Use RANSAC to verify the geometric consistency of feature point matching pairs and obtain the geometric consistency score;

[0030] Convert the Euclidean distance between feature description vectors into a similarity score;

[0031] A reward function is set according to the geometric consistency score and similarity score of the feature point matching pair to obtain a reward value;

[0032] Similarity scores are adjusted using nonlinear functions and scaling factors;

[0033] Apply the Sigmoid function to the adjusted similarity score to obtain a normalized similarity score;

[0034] The normalized similarity score is calculated with the reward value to obtain the retention contribution of the matching pair;

[0035] Integrate the retention contributions within the time interval to obtain the cumulative retention likelihood of the matching pair;

[0036] Traverse the feature points of each image, calculate the time integral of each matching pair, and sum the time integral results of each matching pair;

[0037] The retention probability of the matching pair is calculated based on the cumulative retention possibility of the matching pair and the time integral sum of each matching pair, and the expression is:

[0038] ;

[0039] in, Representing images Middle Feature points and images Middle The probability that a matching pair consisting of feature points will be retained in the final decision, Represents a matching pair The reward value, represents the Sigmoid function, represents the scaling factor, Represents the similarity score The function for nonlinear adjustment, Represents a matching pair The similarity score of represents the total number of feature points in image 1, represents the total number of feature points in image 2, represents the starting time of reinforcement learning, Indicates the end time of reinforcement learning;

[0040] Set a pairing threshold. When the probability of a matching pair being retained in the final decision is greater than the threshold, the pairing is retained.

[0041] As a preferred solution of the image stitching processing method of the present invention, wherein: an image combination model is established based on the matched feature points to obtain a preliminary panoramic image, which specifically includes the following steps:

[0042] Select a pair of adjacent images with the largest overlapping area from the image collection as the starting point for stitching;

[0043] Use SIFT algorithm to extract feature points of a pair of adjacent images with the largest overlapping area, and find the nearest neighbor matching pair of feature points of the two images;

[0044] For feature points, the homography matrix is ​​used to describe the geometric transformation relationship between the two images. Its expression is:

[0045] ;

[0046] in, Representing images and images The geometric transformation relationship of Representing images The two-dimensional coordinates of the feature points, Representing images The two-dimensional coordinates of the feature points, Representing images and images All matching feature point pairs between Representing images The two-dimensional coordinates of the feature points The projection position after transformation under the homography matrix, Indicates the optimal geometric transformation relationship;

[0047] Image According to the geometric transformation relationship between the two images, transform to image The coordinate system of is used, and the overlapping areas are linearly mixed and stitched to form an initial small-scale panoramic image;

[0048] The image with the largest overlap with the current panorama is selected from the remaining images, and the feature point extraction, matching and transformation matrix methods are repeated until the entire initial panorama is completed.

[0049] As a preferred solution of the image stitching processing method of the present invention, wherein: geometric correction is performed on the component images in the preliminary panoramic image based on the physical model to obtain a stitched image set, which specifically includes the following steps:

[0050] The focal length and principal point offset of the camera are obtained through the standard chessboard;

[0051] Use radial and tangential distortion models to describe lens distortion and map distorted pixels back to undistorted pixel coordinates;

[0052] By using the inverse mapping method to solve the distortion-free coordinates, each pixel in the image is traversed;

[0053] The interpolation method is used to fill the blank areas on the edge of the image caused by the distortion correction, and a set of geometrically corrected images is obtained.

[0054] As a preferred solution of the image stitching processing method of the present invention, the stitching image set is globally optimized and stitched to output a complete panoramic image, which specifically includes the following steps:

[0055] Based on the image set, the overlapping area of ​​each pair of adjacent images is calculated according to the homography matrix;

[0056] Assign a weight to each image in the overlapping area and perform weighted fusion on the pixel values ​​in the overlapping area;

[0057] For the fused boundary area, bilateral filtering is used to enhance the smoothing effect;

[0058] Calculate the brightness mean of each stitched image, and perform brightness normalization adjustment on each stitched image so that its brightness mean is consistent with the global brightness mean;

[0059] Adjust the hue and saturation of the image in the LAB color space and perform gamma correction on the adjusted image;

[0060] Repair the edge area of ​​the panorama and output a complete panorama.

[0061] In a second aspect, the present invention provides an image stitching processing device, comprising:

[0062] A correction module receives a set of images to be stitched and performs correction;

[0063] A feature extraction module extracts the feature point positions and their corresponding sub-vectors of the rectified image;

[0064] Feature matching module, which calculates the similarity scores between different feature points and matches the feature points;

[0065] The combination module builds an image combination model based on the matched feature points to obtain a preliminary panoramic image;

[0066] A geometric correction module, which performs geometric correction on the component images in the preliminary panorama based on a physical model to obtain a set of stitched images;

[0067] The panorama optimization module performs global optimization and stitching on the set of stitched images and outputs a complete panorama.

[0068] In a third aspect, the present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, any step of the image stitching processing method described in the first aspect of the present invention is implemented.

[0069] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, any step of the image stitching processing method described in the first aspect of the present invention is implemented.

[0070] The beneficial effects of the present invention are as follows: the distance of feature description vectors is calculated using Euclidean distance, and geometric consistency screening is performed by combining bidirectional matching verification and RANSAC algorithm, which effectively eliminates the possibility of mismatching, and finally reinforcement learning is used to ensure that the matching pairs retained in the end have high credibility; in addition, the overlapping area of ​​each pair of adjacent images is calculated based on the homography matrix, and a weight is assigned to each image in the overlapping area, and the pixel values ​​of the overlapping area are weighted fused. This method not only takes into account the relative contribution of each image, but also effectively avoids the color inconsistency problem that may be caused by simple linear mixing. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0072] Figure 1 Schematic diagram of the image stitching processing method in Example 1.

[0073] Figure 2 This is a schematic diagram of image feature point matching in Example 1. DETAILED DESCRIPTION

[0074] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0075] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0076] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0077] Example 1, reference Figure 1 and Figure 2 , is the first embodiment of the present invention, which provides an image stitching processing method, comprising the following steps:

[0078] S1. Receive a set of images to be stitched and perform correction.

[0079] The specific steps include:

[0080] S1. Perform white balance correction on the image by adjusting the gain factors of the red, green, and blue channels of the image to ensure that the pixel values ​​in the white area are close to the same, unify the color reproduction, and reduce the color cast caused by lighting conditions.

[0081] Normalizes the brightness and contrast of images to ensure consistent color distribution across all images and balances brightness and color differences between images.

[0082] The color-adjusted image is subjected to histogram equalization to enhance contrast, especially in dark areas, and denoising filtering is performed to reduce noise and improve image clarity.

[0083] It is further explained that the above processing of the image improves the color accuracy of the image, making the image look more natural. At the same time, it enhances the visual appeal of the image, makes the image more vivid and lively, and reveals details that were originally difficult to see, thereby enhancing the information content of the image.

[0084] S2. Extract the feature point positions of the rectified image and their corresponding sub-vectors.

[0085] The specific steps include:

[0086] S2.1. Standardize the rectified image and use the mean and variance of the image gradient to estimate the complexity of the local area. Due to the different complexity of the image content, directly and uniformly extracting feature points will lead to insufficient number of feature points in texture-rich areas, resulting in loss of details and too many redundant feature points in flat areas, wasting computing resources.

[0087] Therefore, the gradient mean and variance of the local area of ​​the image are used to determine whether it is of high complexity. When it is of high complexity, the density of feature points is increased and the local neighborhood window is reduced. When it is of low complexity, the density of feature points is reduced and the neighborhood window is expanded.

[0088] Further explanation: the coverage of feature points in key areas is improved, while redundant calculations are reduced, and overall performance is improved. It can also adapt to various scenes more flexibly, especially for environments with rich textures or changes, and its adaptability is significantly enhanced.

[0089] S2.2, input the normalized data into a deep convolutional neural network to obtain the probability distribution of whether each pixel is a key point, and use the convolution-based output layer to generate a feature response map;

[0090] Find local maximum points on the feature response graph. These points are the most representative positions in the image and are suitable as feature points. Filter out the most significant feature points and record their coordinates to form a set of feature points for each image.

[0091] It is further explained that it ensures that the selection of feature points is both representative and sufficiently discriminative, which improves the success rate of matching.

[0092] Define a receptive field around each feature point, and a fixed-size window (such as a 32×32 area), compressing the information in the receptive field into a vector of fixed dimension. Collect the feature description vectors in all images to form a unified feature description vector set, which is convenient for subsequent similarity calculation and matching.

[0093] Further explanation: the consistency of feature descriptions between different images is guaranteed, which is conducive to feature matching across images. In addition, the feature description vectors with fixed dimensions are easy to store and quickly retrieve, which improves processing efficiency.

[0094] S3. Calculate the similarity scores between feature points of different images and match the feature points.

[0095] The specific steps include:

[0096] S3.1. Use Euclidean distance to calculate the distance of feature description vectors in order to match feature points from different images. The expression is:

[0097] ;

[0098] in, Represents the feature description vector and the feature description vector The Euclidean distance of Representing images Middle The feature description vector of feature points, Representing images Middle The feature description vector of feature points, represents the dimension of the feature description vector, Represents the feature description vector No. Dimensional component, Represents the feature description vector No. Dimensional component.

[0099] For the feature description vector of the feature point of each image, find the feature point with the closest distance in the feature description vector set of other images according to the Euclidean distance, and use it as the nearest neighbor matching point.

[0100] To further illustrate, the Euclidean distance can be used to find the similarity between feature points, screen out possible matching feature point pairs, and provide a basis for subsequent matching.

[0101] S3.2. For each feature point and its nearest neighbor matching point, reverse verification is performed to check whether its nearest neighbor matching point is itself. Only when the two are each other's nearest neighbors can the matching pair be retained.

[0102] It is further explained that through two-way verification, one-way matching errors are avoided and the reliability of matching pairs is improved.

[0103] S3.3. Use RANSAC (random sampling consensus algorithm) to estimate the homography matrix of the initially screened matching pairs, use the homography matrix to project all matching points, obtain the projection error, set the judgment error to 3 pixels, if it meets the criteria, it is considered to be an inlier, repeat the sampling until the homography matrix with the most inliers is found, and retain the matching pairs that meet the geometric consistency. The ratio of the number of inliers to the total number of matching points is used as the geometric consistency score.

[0104] It is further explained that RANSAC can effectively eliminate mismatched points, improve geometric consistency, and enhance the globality of matching.

[0105] S3.4. Euclidean distance between feature description vectors Convert to similarity score , set the reward function based on the geometric consistency score and similarity score of the matching pair , get the reward value; the reward value state is the description vector and matching result (match / mismatch) of the current pair of feature points; the reward action is to choose whether to keep the current matching pair, and give rewards or penalties based on the geometric consistency and similarity scores of the matching pair.

[0106] Similarity scores are adjusted using nonlinear functions and scaling factors;

[0107] Apply the Sigmoid function to the adjusted similarity score to get the normalized similarity score ;

[0108] Introducing time points, based on the reward value of matching pairs , normalized similarity score , calculate the retention probability of the matching pair, which is expressed as:

[0109] ;

[0110] in, Representing images No. Feature points and images No. The probability that a matching pair consisting of feature points will be retained in the final decision, Represents a matching pair The reward value, represents the Sigmoid function, represents the scaling factor, Represents the similarity score The function for nonlinear adjustment, Represents a matching pair The similarity score of represents the total number of feature points in image 1, represents the total number of feature points in image 2, represents the starting time of reinforcement learning, Indicates the end time of reinforcement learning.

[0111] Set a pairing threshold. When the probability of a matching pair being retained in the final decision is greater than the threshold, the pairing is retained.

[0112] It is further explained that the pairing threshold can be adjusted according to the specific application scenario to flexibly control the number and accuracy of matching pairs.

[0113] S4. Building an image combination model based on the matched feature points to obtain a preliminary panoramic image.

[0114] The specific steps include:

[0115] S4.1. Select a pair of adjacent images with the largest overlapping area from the image collection through image metadata, and use this pair of images as the starting point for stitching, which are called images and images .

[0116] It is further explained that taking the image pair with the largest overlapping area as the starting point can ensure the reliability of the initial stitching and reduce the error accumulation in the subsequent stitching process.

[0117] Extract images using SIFT algorithm and images feature points of the two images and find matching point pairs between the feature point description vectors of the two images.

[0118] For feature points, the homography matrix is ​​used to describe the geometric transformation relationship between the two images. Its expression is:

[0119] ;

[0120] in, Representing images and images The geometric transformation relationship of Representing images The two-dimensional coordinates of the feature points, Representing images The two-dimensional coordinates of the feature points, Representing images and images All matching feature point pairs between Representing feature points The projection position after transformation under the homography matrix, Indicates the optimal geometric transformation relationship;

[0121] S4.2. Image According to the geometric transformation relationship between the two images, transform to image In the coordinate system, the projection distortion is reduced, and the overlapping areas are linearly mixed and stitched to form a small-scale panoramic image with smooth transition and reduce stitching marks.

[0122] The image with the largest overlap with the current panorama is selected from the remaining images, and feature point extraction, matching, geometric transformation relationship solution, image transformation and linear blending and stitching are performed again until the entire initial panorama is completed.

[0123] It is further explained that the image with the largest overlapping area is selected each time to ensure maximum information coverage during the stitching process.

[0124] S5. Based on the physical model, geometric correction is performed on the component images in the preliminary panoramic image to obtain a set of stitched images.

[0125] The specific steps include:

[0126] S5.1. Use a standard chessboard as a calibration plate and take multiple photos at different angles and positions. Using the corner point information in these images, the intrinsic parameter matrix can be accurately obtained, including the focal length and principal point offset of the camera.

[0127] To further illustrate, the standard chessboard provides known and easily detectable feature points, which can achieve very high calibration accuracy.

[0128] S5.2. The non-ideal characteristics of the camera lens will cause image distortion. Common distortions include radial distortion and tangential distortion. Therefore, radial distortion and tangential distortion are used to describe the nonlinear distortion characteristics of the camera lens.

[0129] Based on the radial and tangential distortions, the ideal pixel coordinates without distortion are obtained.

[0130] Furthermore, the separation of radial and tangential distortions makes the correction process more flexible.

[0131] S5.3. Use the inverse distortion model to iteratively solve the undistorted points. For each pixel in the image, calculate its corresponding undistorted ideal pixel coordinates. According to the distortion model, perform point-by-point mapping to generate an undistorted image. Pixel-by-pixel mapping ensures the accuracy of correction so that all pixels can be processed.

[0132] It is further explained that after distortion correction, the geometry of the image will be closer to the real scene.

[0133] S5.4. After distortion correction, some pixels may be mapped to the edge of the image or even outside the image, which will cause blank areas to appear in the corrected image.

[0134] Using nearest neighbor interpolation, the pixels in the blank area are filled with the nearest known pixel value, and then a set of geometrically corrected images is output.

[0135] It is further explained that the interpolation method effectively fills the blank areas at the edges of the corrected image, improves the integrity of the image, and makes the corrected image more natural without obvious blank or broken traces.

[0136] S6. Globally optimize and stitch the stitched image set to output a complete panoramic image.

[0137] The specific steps include:

[0138] S6.1. Using the homography matrix calculated previously, transform all pixels of one image into the space of another image through the homography matrix, and find the minimum circumscribed rectangle of the transformed image in the target image space, which is the overlapping area of ​​the two.

[0139] The weights are gradually adjusted according to the distance from the boundary of the overlapping area, so that the two images merge naturally at the junction.

[0140] It is further explained that accurate definition of overlapping areas is ensured, reducing uncertainty and errors in subsequent processing.

[0141] S6.2, Bilateral filtering is a filter that can both preserve edge details and smooth noise. It combines spatial proximity and pixel similarity, and uses bilateral filtering to enhance and smooth the fused boundary area.

[0142] It can be further explained that the use of bilateral filtering can smooth the noise while retaining important edge information and avoiding excessive blurring. At the same time, it improves the visual effect of the splicing boundary, making the entire image look more natural and smooth.

[0143] S6.2. For each stitched image, calculate the brightness mean, which is expressed as:

[0144] ;

[0145] in, represents the mean brightness of the image, Represents the total number of pixels in the image, Represents the two-dimensional coordinates of the pixel points in the image, Indicates The brightness value of the image.

[0146] Calculate the global brightness mean of all stitched images, which is expressed as:

[0147] ;

[0148] in, represents the global brightness mean, Indicates the total number of stitched images.

[0149] The brightness of each image is normalized to make its mean brightness consistent with the global mean brightness, eliminating the brightness difference between images caused by different lighting conditions.

[0150] S6.3. Convert the stitched image from the RGB color space to the LAB color space, where L is the brightness channel and AB are channels describing hue and saturation.

[0151] Adjust the A channel and the B channel, and convert the adjusted LAB image back to the RGB color space.

[0152] Perform gamma correction on the converted image to enhance the brightness and contrast of the image.

[0153] S6.4. Stitch the panorama, and use the cropping method to remove the incomplete edge areas after stitching, and retain the complete parts of the image. This improves the aesthetics and practicality of the panorama and avoids the incomplete areas affecting the visual effect.

[0154] After cropping, the complete panorama is exported.

[0155] This embodiment also provides an image stitching processing device, including:

[0156] A correction module receives a set of images to be stitched and performs correction;

[0157] A feature extraction module extracts the feature point positions and their corresponding sub-vectors of the rectified image;

[0158] Feature matching module, which calculates the similarity scores between different feature points and matches the feature points;

[0159] The combination module builds an image combination model based on the matched feature points to obtain a preliminary panoramic image;

[0160] A geometric correction module, which performs geometric correction on the component images in the preliminary panorama based on a physical model to obtain a set of stitched images;

[0161] The panorama optimization module performs global optimization and stitching on the set of stitched images and outputs a complete panorama.

[0162] This embodiment also provides a computer device, which is applicable to the image stitching processing method, including: a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions to implement the image stitching processing method proposed in the above embodiment.

[0163] The computer device may be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0164] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the image stitching processing method proposed in the above embodiment is implemented; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EPROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, referred to as EPROM), programmable read-only memory (Programmable Red-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0165] In summary, the present invention uses the Euclidean distance to calculate the distance of the feature description vector, combines the two-way matching verification and the RANSAC algorithm to perform geometric consistency screening, effectively eliminates the possibility of mismatching, and finally uses reinforcement learning to ensure that the matching pairs retained in the end have a high degree of credibility; in addition, the overlapping area of ​​each pair of adjacent images is calculated based on the homography matrix, and a weight is assigned to each image in the overlapping area, and the pixel values ​​of the overlapping area are weighted fused. This method not only takes into account the relative contribution of each image, but also effectively avoids the color inconsistency problem that may be caused by simple linear mixing.

[0166] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. An image stitching processing method, characterized in that: include, Receive a set of images to be stitched and perform correction; Extract the feature point positions and their corresponding sub-vectors of the rectified image; Calculate the similarity scores between feature points of different images and match the feature points; An image combination model is established based on the matched feature points to obtain a preliminary panoramic image; Based on the physical model, geometric correction is performed on the component images in the preliminary panorama to obtain a set of stitched images; Perform global optimization and stitching of the stitched image set to output a complete panorama; Calculate the similarity scores between the feature points of different images and match the feature points. The specific steps include: For the feature description vector of each feature point in the image, find the nearest neighbor matching point in the feature description vector set of other images based on the Euclidean distance; Perform two-way matching verification to confirm whether the current feature point and the feature point of the current nearest neighbor matching point are each other's nearest neighbors, and obtain a preliminarily screened feature point matching pair; Use RANSAC to verify the geometric consistency of feature point matching pairs and obtain the geometric consistency score; Convert the Euclidean distance between feature description vectors into a similarity score; A reward function is set according to the geometric consistency score and similarity score of the feature point matching pair to obtain a reward value; Similarity scores are adjusted using nonlinear functions and scaling factors; Apply the Sigmoid function to the adjusted similarity score to obtain a normalized similarity score; The normalized similarity score is calculated with the reward value to obtain the retention contribution of the matching pair; Integrate the retention contributions within the time interval to obtain the cumulative retention likelihood of the matching pair; Traverse the feature points of each image, calculate the time integral of each matching pair, and sum the time integral results of each matching pair; The retention probability of the matching pair is calculated based on the cumulative retention possibility of the matching pair and the time integral sum of each matching pair, and the expression is: ； in, Representing images Middle Feature points and images Middle The probability that a matching pair consisting of feature points will be retained in the final decision, Represents a matching pair The reward value, represents the Sigmoid function, represents the scaling factor, Represents the similarity score The function for nonlinear adjustment, Represents a matching pair The similarity score of represents the total number of feature points in image 1, represents the total number of feature points in image 2, represents the starting time of reinforcement learning, Indicates the end time of reinforcement learning; Set a pairing threshold. When the probability of a matching pair being retained in the final decision is greater than the threshold, the pairing is retained.

2. The image stitching processing method according to claim 1, characterized in that: Receive a set of images to be stitched and perform correction, which specifically includes the following steps: Perform white balance correction and color adjustment on images; Perform histogram equalization and denoising filtering on the color adjusted image.

3. The image stitching processing method according to claim 2, characterized in that: Extracting the feature point positions of the rectified image and their corresponding sub-vectors specifically includes the following steps: Normalize the rectified image and evaluate the content complexity of the local area in the image; Adjust the density and scale of feature point extraction according to the content complexity; Input the standardized image into a deep convolutional neural network to obtain a feature response map; Extract the local maximum value from the feature response map as the feature point to form a set of feature points for each image; A receptive field of fixed size is extracted around the feature points of each image, and a feature description vector of fixed dimension is generated based on the receptive field, and the feature description vectors of all images are integrated into a feature description vector set.

4. The image stitching processing method according to claim 3, characterized in that: Based on the matched feature points, an image combination model is established to obtain a preliminary panoramic image, which specifically includes the following steps: Select a pair of adjacent images with the largest overlapping area from the image collection as the starting point for stitching; Use SIFT algorithm to extract feature points of a pair of adjacent images with the largest overlapping area, and find the nearest neighbor matching pair of feature points of the two images; For feature points, the homography matrix is ​​used to describe the geometric transformation relationship between the two images. Its expression is: ； in, Representing images and images The geometric transformation relationship of Representing images The two-dimensional coordinates of the feature points, Representing images The two-dimensional coordinates of the feature points, Representing images and images All matching feature point pairs between Representing images The two-dimensional coordinates of the feature points The projection position after transformation under the homography matrix, Indicates the optimal geometric transformation relationship; Image According to the geometric transformation relationship between the two images, transform to image The coordinate system of is used, and the overlapping areas are linearly mixed and stitched to form an initial small-scale panoramic image; The image with the largest overlap with the current panorama is selected from the remaining images, and the feature point extraction, matching and transformation matrix methods are repeated until the entire initial panorama is completed.

5. The image stitching processing method according to claim 4, characterized in that: Based on the physical model, geometric correction is performed on the component images in the preliminary panorama to obtain a set of stitched images, which specifically includes the following steps: The focal length and principal point offset of the camera are obtained through the standard chessboard; Use radial and tangential distortion models to describe lens distortion and map distorted pixels back to undistorted pixel coordinates; By using the inverse mapping method to solve the distortion-free coordinates, each pixel in the image is traversed; The interpolation method is used to fill the blank areas on the edge of the image caused by the distortion correction, and a set of geometrically corrected images is obtained.

6. The image stitching processing method according to claim 5, characterized in that: The stitched image set is globally optimized and stitched to output a complete panorama, which specifically includes the following steps: Based on the image set, the overlapping area of ​​each pair of adjacent images is calculated according to the homography matrix; Assign a weight to each image in the overlapping area and perform weighted fusion on the pixel values ​​in the overlapping area; For the fused boundary area, bilateral filtering is used to enhance the smoothing effect; Calculate the brightness mean of each stitched image, and perform brightness normalization adjustment on each stitched image so that its brightness mean is consistent with the global brightness mean; Adjust the hue and saturation of the image in the LAB color space and perform gamma correction on the adjusted image; Repair the edge area of ​​the panorama and output a complete panorama.

7. An image stitching processing device, based on the image stitching processing method according to any one of claims 1 to 6, characterized in that: include, A correction module receives a set of images to be stitched and performs correction; A feature extraction module extracts the feature point positions and their corresponding sub-vectors of the rectified image; Feature matching module, which calculates the similarity scores between different feature points and matches the feature points; The combination module builds an image combination model based on the matched feature points to obtain a preliminary panoramic image; A geometric correction module, which performs geometric correction on the component images in the preliminary panorama based on a physical model to obtain a set of stitched images; The panorama optimization module performs global optimization and stitching on the set of stitched images and outputs a complete panorama.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the image stitching processing method described in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the image stitching processing method described in any one of claims 1 to 6 are implemented.

Images