Image stitching processing method and device

Abstract

The present application provides an image stitching processing method and device, which are applied to the field of image processing technology. The method includes: constructing multiple first feature point matching pairs; eliminating abnormal feature point matching pairs in the first feature point matching pairs to obtain second feature point matching pairs; eliminating abnormal feature point matching pairs in the second feature point matching pairs to obtain third feature point matching pairs; according to the distribution area of ​​the third feature point matching pairs in the image to be stitched, inserting new feature point matching pairs into the area outside the distribution area to obtain fourth feature point matching pairs; according to the fourth feature point matching pairs, generating alignment parameters for image stitching. By screening multiple feature points and enhancing the feature points in the areas where the feature points are not distributed in the image, the quality of feature point pairing is improved, and the image alignment effect obtained by calibrating the stitching using these feature point matching pairs is better and the ghosting is smaller.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image stitching processing method and device. Background Art

[0002] Due to the limitations of optical properties, the imaging field of view of a single lens is limited, and the expansion of the field of view is always accompanied by greater distortion. The independent imaging of multiple lenses divides the picture, which is not conducive to intelligent analysis based on panoramic images or videos.

[0003] In the related technology, image stitching technology is used to map multiple images obtained from different viewpoints into a unified coordinate domain to synthesize an image with a larger field of view, thereby realizing intelligent analysis of panoramic views or videos. Currently, in the process of stitching multiple images, factory calibration or on-site calibration is usually used to complete image stitching.

[0004] However, for scenes with rich depth of field and devices with large viewpoint spacing, factory calibration has difficulty in effectively handling the different parallaxes of these objects, which leads to misalignment or distortion when stitching images; and on-site calibration often results in poor stitching effects due to the uneven quality of feature points. Summary of the invention

[0005] The present application provides an image stitching processing method and device to solve at least one of the above technical problems.

[0006] According to a first aspect of the present application, an image stitching processing method is provided, comprising: constructing a plurality of first feature point matching pairs according to consistency recognition results between feature points of a plurality of images to be stitched; removing abnormal feature point matching pairs whose difference between the rotation angle of the line in the first feature point matching pair and the rotation angle of the line in other feature point matching pairs reaches a difference threshold according to the line rotation angle of each first feature point matching pair, to obtain a second feature point matching pair; removing abnormal feature point matching pairs that deviate from the fitting result according to the fitting result of the second feature point matching pair, to obtain a third feature point matching pair; inserting a new feature point matching pair into an area outside the distribution area according to the distribution area of ​​the third feature point matching pair in the image to be stitched, to obtain a fourth feature point matching pair; wherein the new feature point matching pair is determined based on the third feature point matching pair; generating a registration parameter for image stitching according to the fourth feature point matching pair, and performing stitching processing on the plurality of images to be stitched based on the registration parameter, to obtain a stitched image.

[0007] According to a second aspect of the present application, an image stitching processing device is provided, comprising: a matching pair construction module, which is configured to construct a plurality of first feature point matching pairs according to consistency recognition results between feature points of a plurality of images to be stitched; a feature removal module, which is configured to remove abnormal feature point matching pairs whose difference between the line rotation angle in the first feature point matching pair and the line rotation angle of other feature point matching pairs reaches a difference threshold according to the line rotation angle of each first feature point matching pair, so as to obtain a second feature point matching pair; the feature removal module is further configured to remove the abnormal feature point matching pairs according to the fitting result of the second feature point matching pairs. An abnormal feature point matching pair in the two feature point matching pairs that deviates from the fitting result is used to obtain a third feature point matching pair; a feature enhancement module is configured to insert a new feature point matching pair into an area outside the distribution area of ​​the third feature point matching pair in the image to be stitched, so as to obtain a fourth feature point matching pair; wherein the new feature point matching pair is determined based on the third feature point matching pair; a stitching processing module is configured to generate a registration parameter for image stitching based on the fourth feature point matching pair, and stitch the multiple images to be stitched based on the registration parameter to obtain a stitched image.

[0008] According to a third aspect of the present application, a server is provided, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the image stitching processing method in the first aspect above.

[0009] According to a fourth aspect of the present application, a computer-readable storage medium is provided, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to execute the image stitching processing method in the first aspect.

[0010] According to a fifth aspect of the present application, a computer program product is provided, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When the at least one processor executes the computer program, the image stitching processing method in the first aspect can be implemented.

[0011] The image stitching processing method and device provided by the embodiment of the present application construct a plurality of first feature point matching pairs according to the consistency recognition results between the feature points of a plurality of images to be stitched; according to the line rotation angle of each first feature point matching pair, remove the abnormal feature point matching pairs whose difference between the line rotation angle in the first feature point matching pair and the line rotation angle of other feature point matching pairs reaches the difference threshold, so as to obtain the second feature point matching pairs; according to the fitting result of the second feature point matching pair, remove the abnormal feature point matching pairs in the second feature point matching pair that deviate from the fitting result, so as to obtain the third feature point matching pairs; according to the distribution area of ​​the third feature point matching pair in the image to be stitched, insert a new feature point matching pair into the area outside the distribution area, so as to obtain the fourth feature point matching pair; wherein the new feature point matching pair is determined based on the third feature point matching pair; according to the fourth feature point matching pair, generate the registration parameters for image stitching, and stitch the plurality of images to be stitched based on the registration parameters, so as to obtain the stitched image. In this process, multiple feature point screening is used to strengthen the elimination of mismatched points. At the same time, feature point enhancement is performed in areas where feature points are not distributed in the image to solve the trend of changes in featureless areas that are significantly greater than reasonable values, thereby improving the alignment effect. The registration parameters generated by feature points after feature point screening and enhancement are used to constrain the overall transformation at the same time, solving the problem of stitching multiple cameras with variable structures and large viewpoint spacing. It can also be compatible with a variety of large parallax scenes, as well as objects at different distances in the same scene, to achieve good stitching effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0013] Figure 1 An application scenario diagram of the image stitching processing method provided in an embodiment of the present application;

[0014] Figure 2 A schematic diagram of the process flow of the image stitching processing method provided in the embodiment of the present application;

[0015] Figure 3 An example diagram of matching point pairs in an image;

[0016] Figure 4 This is an example diagram of the feature distribution of matching point pairs;

[0017] Figure 5 This is an example diagram of a feature point matching pair of an abnormal line rotation angle in an embodiment of the present application;

[0018] Figure 6 For Figure 3(c) is an example of a pair of matching points after removing abnormal line rotation angles;

[0019] Figure 7 For Figure 3 (c) is an example of feature distribution of some matching point pairs after removing abnormal line rotation angles;

[0020] Figure 8 This is an example diagram of the feature distribution and stitching effect of indoor close-up scenes;

[0021] Fig. 9 This is an example diagram of feature distribution after feature enhancement for 8 example scenes;

[0022] Fig.10 To use Fig. 9 An example of the stitching effect of matching point pairs for image stitching;

[0023] Fig.11 This is an example of the stitching result of the global model with rotation;

[0024] Fig.12 This is an example of the stitching result of the global model without rotation;

[0025] Fig.13 This is an example diagram of cylindrical projection in the embodiment of this application;

[0026] Fig.14 This is the image before distortion correction and cylindrical projection in the embodiment of the present application;

[0027] Fig.15 The image to be stitched after distortion correction and cylindrical projection in the embodiment of the present application;

[0028] Fig.16 For the embodiment of this application Fig.15 An example diagram of the effect of image stitching processing on the images to be stitched;

[0029] Fig.17 is the original image of the multi-camera;

[0030] Fig.18 This is a detailed enlargement of the original image;

[0031] Fig.19 The spliced ​​image and detail enlarged image are obtained by factory calibration of the original image in the related art;

[0032] Fig. 20 The spliced ​​image and detail enlarged image are obtained by using on-site calibration for the original image in the related art;

[0033] Fig.21 The spliced ​​image is obtained by calibrating the original image using this embodiment;

[0034] Fig. 22 for Fig.21 Zoomed-in detail of the stitched image;

[0035] Fig.23 A schematic diagram of the structure of an image stitching processing device provided in an embodiment of the present application;

[0036] Fig.24 A schematic diagram of the structure of a server provided in an embodiment of the present application.

[0037] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0038] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0039] To facilitate understanding of the embodiments of the present application, the following terms are first explained:

[0040] Image stitching: refers to the process of aligning images acquired by multiple lenses or imaging devices and stitching them into a large image.

[0041] Stitching calibration: refers to the process of stitching images and generating registration parameters. It can be calibrated at the factory or on-site.

[0042] Parallax: refers to the difference in direction when observing the same object from two points at a certain distance. It is manifested in the image as the position difference of the same object in two images.

[0043] Feature points: refers to the coordinates of pixels with unique local properties in the image and their descriptors. Through a specific algorithm, the matching degree between descriptors can be calculated to determine the pairing relationship between feature points. Excellent feature points should have high robustness and be less affected by factors such as lighting, scaling, and perspective differences.

[0044] Stitching line: refers to the line that divides the imaging content from two different lenses in the stitched image. It can be either a straight line or a curve.

[0045] In the field of image processing, using image stitching technology to map multiple images acquired from different viewpoints into a unified coordinate domain to synthesize an image with a larger field of view has always been an important research topic in camera multi-viewing technology. In related technologies, the calibration methods for image stitching are divided into two categories: factory calibration and on-site calibration. These two calibration methods have the following difficulties:

[0046] One of the difficulties is that multi-cameras have parallax. Objects at different distances in the scene have different parallax sizes in the left and right images. For the same device, the closer the object, the greater the parallax. For the same distance, the greater the distance between the device viewpoints, the greater the parallax. This means that it is impossible to use a homography matrix to describe the positional relationship between the two images. For scenes with rich depth of field and devices with large viewpoint distances, factory calibration is not feasible.

[0047] The second difficulty is that the on-site calibration environment is complex and changeable, and the accuracy of the registration is greatly affected by features such as texture. For some devices with non-fixed structures, users are often required to calibrate multiple times according to their usage needs. This means that the stitching calibration algorithm needs to be applicable to different scenarios without introducing complex parameters, which makes calibration difficult.

[0048] The third difficulty is that for devices with more than two eyes, when mapped to a unified coordinate system, the parameters calculated in multiple overlapping areas need to be accumulated, which can easily lead to cumulative errors. Since the effective range of the final video stitching is the largest inscribed rectangle, too large a cumulative error can easily lead to a serious loss of field of view and distortion of the image.

[0049] For example, in factory calibration, related technologies use a checkerboard to calibrate the camera's internal and external parameters, determine the mapping relationship between the original image captured by the camera and the reference plane, determine the image area corresponding to the original image on the reference plane and the corresponding overlap area between each camera based on the mapping relationship, and stitch the original images captured by the two cameras based on the coordinates of the overlap area in each original image, without relying on the feature points of the image overlap area in the actual scene. This technology is aimed at devices with fixed structures and small viewpoint spacing, and can achieve better stitching effects when the monitoring distance is far, but this method has significant limitations:

[0050] a) It is unable to resolve the parallax of objects at different distances in the same scene;

[0051] b) It can only be used for multi-eye devices with unchanged structure. If the structure changes, it needs to be recalibrated in the factory.

[0052] Another example is on-site calibration, which currently focuses on solving the parallax caused by non-pure rotation models. For example, in related technologies, the image to be stitched is divided into multiple grids, and local weights are calculated based on the matching feature points and the center position of the grid area. Each matching feature point is processed based on the local weight to obtain the reprojection error, and the rotation matrix corresponding to the grid area is optimized to obtain the regional rotation matrix. A global energy function is constructed based on the coordinates of each grid vertex for global optimization. In addition, there are some related technologies that optimize the stitching line to improve the visual effect. For example, weights are calculated based on the brightness component or chromaticity component of the pixel point. In the stitching path within the overlapping area, the stitching path with the smallest stitching line strength is determined as the best stitching line. This technology can make the stitching line avoid the area to be processed, ensure the natural stitching of the area to be processed, and improve the user's subjective experience of browsing the image. The limitations of these technologies are:

[0053] a) The stitching effect is strongly related to the quality of feature points, and the distribution of feature points in different scenes varies. When the quality of feature points is poor, the stitching effect is poor.

[0054] b) Block-by-block calculation of the rotation matrix For large viewpoint spacing and flat images, the viewing angles of the images differ significantly, making it more difficult to find common feature points during alignment and stitching, resulting in insufficient alignment effect;

[0055] c) Since the feature points are stable in static scenes, but may change in dynamic scenes due to camera movement, object motion, etc., such changes may mask the mismatching of feature points. Therefore, the visual effect optimization brought by the seam line is more obvious in static scenes, but weaker in dynamic scenes.

[0056] In view of the above problems, the embodiment of the present application provides an image stitching processing scheme for on-site calibration based on feature point screening and enhancement. Through multiple feature point screening, the elimination of mismatched points is strengthened, and feature point enhancement is performed on the area where feature points are not distributed in the image. This can effectively solve the trend of the featureless area in the large parallax device scene being significantly greater than the reasonable value due to the wide distribution range of the relative position and distance of the feature points, effectively improve the alignment effect, and use the registration parameters generated by the feature points after feature point screening and enhancement to constrain the overall transformation at the same time, solving the problem of multi-eye camera stitching with variable structure and large viewpoint spacing. Moreover, through the above-mentioned feature point screening and enhancement method, a good stitching effect can be achieved for objects of different distances in the same scene without adjusting the stitching algorithm parameters, and it has strong scene compatibility. In addition, in the process of generating the registration parameters, a thin plate spline function is used for local elastic deformation to improve the registration accuracy, and an interpolation transition is performed between the overlapping area and the non-overlapping area to ensure smoothness, further improving the image stitching effect.

[0057] Figure 1The application scenario diagram of the image stitching processing method provided in the embodiment of the present application is as follows: Figure 1 As shown, the scene may include an image acquisition device 101 and a server 102, wherein the image acquisition device 101 and the server 102 are communicatively connected, the image acquisition device 101 may be a binocular camera or a trinocular camera, or an image acquisition device with more lenses, or any device capable of multi-image acquisition or storage, for acquiring multiple images and transmitting them to the server 102 for image stitching processing; the server 102 is used to construct feature point matching pairs in multiple images, and after outliers are removed from these feature point matching pairs and feature point matching pairs are enhanced, alignment parameters for image stitching are generated using the feature point matching pairs after being removed and enhanced, and then multiple images are stitched using the alignment parameters, the server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Networks) In some embodiments, the cloud server may be an electronic device used to perform various operations of the above-mentioned servers, such as a smart phone, a laptop computer, etc.

[0058] Optionally, the server 102 may also be communicatively connected to the image analysis and processing device 103. After generating the stitched image, the server may transmit the stitched image to the image analysis and processing platform, and the image analysis and processing platform may perform intelligent analysis on the stitched image, such as target detection and tracking.

[0059] It should be noted that the communication connection between the server and the image acquisition device and the image analysis and processing platform can be a wired connection, connecting the devices through physical cables such as data cables and network cables, such as Universal Serial Bus (USB), High-Definition Multimedia Interface (HDMI), Ethernet and other interfaces to connect devices for data transmission. It can also be a wireless connection, communicating through radio wave signals, such as wireless fidelity (WiFi), Bluetooth, infrared and other wireless technologies to achieve communication between devices. Or it can be a remote connection, connecting devices through network technologies such as the Internet to achieve remote control and data transmission, such as cloud services, remote desktops, etc. It can also be a near field communication (NFC) connection, which realizes the connection between devices through near field communication technology, usually used for applications such as file transfer; low-power Bluetooth connection, used for short-distance communication connection between devices, such as the connection of smart bracelets, smart watches and other devices; wireless communication technology connection, used for low-speed, short-distance communication connection between devices, suitable for low-power IoT devices. This application does not specifically limit the specific method of communication connection between physical devices.

[0060] This application does not specifically limit the specific form and type of any of the above-mentioned physical devices.

[0061] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0062] Figure 2 The flowchart of the image stitching processing method provided in the embodiment of the present application is shown in FIG. 1 . The execution subject of the method may be the server in the above application scenario, such as Figure 2 As shown, the method includes steps S201-S205:

[0063] Step S201: construct a plurality of first feature point matching pairs according to the consistency recognition results between the feature points of a plurality of images to be stitched.

[0064] Among them, the images to be stitched can be multiple images captured by multi-eye cameras, such as two images captured by binocular cameras. In order to effectively improve the quality of image stitching, the images to be stitched can be corrected by image distortion and projected onto an image on a projection plane. Feature points are pixel coordinates and descriptors with unique local properties in an image. In this embodiment, an accelerated nonlinear scale space-based feature detection method (Accelerated Nonlinear ScaleSpace for Feature Detection, AKAZE) can be used to extract feature points to improve the extraction accuracy of feature points. The detailed process can be referred to the image to be stitched and the feature point extraction process in the embodiments below.

[0065] In this embodiment, taking two images to be stitched as an example, the consistency recognition result between the feature points of the two images can be to find the K nearest neighbor feature points (descriptors) in the other image for each feature point (descriptor) through the nearest neighbor algorithm (K-Nearest Neighbors, KNN), and can further combine the random sample consensus algorithm (RandomSample Consensus, RANSAC) to improve the accuracy to construct feature point matching pairs. It can be understood that the feature point matching pairs of images are usually matching point pairs with overlapping areas between the images to be stitched.

[0066] In some embodiments, in order to further improve the quality of feature points, the nearest neighbor algorithm KNN and RANSAC are combined to construct feature point matching pairs, and the RANSAC threshold is relaxed using a scaling factor to effectively improve the accuracy of feature point matching pairs while retaining as many matching pairs as possible. Specifically, the above step S201 constructs multiple first feature point matching pairs based on the consistency recognition results between the feature points of multiple images to be stitched, and may include the following steps:

[0067] Determining a plurality of potential feature point matching pairs according to the consistency recognition results between the feature points;

[0068] According to the random sampling consistency algorithm, the potential feature point matching pairs among the multiple potential feature point matching pairs whose deviation from the sampling model is greater than a preset threshold are filtered out; wherein the preset threshold is a threshold obtained by relaxing the initial threshold of the random sampling consistency algorithm according to a preset proportional factor.

[0069] Exemplarily, the descriptors of feature points in different images can be compared and the nearest neighbor method (KNN) can be used to find matching feature point pairs. This step may produce some mismatches, so this embodiment can eliminate bad matches according to the model actually required by the random sampling consensus algorithm (RANSAC) algorithm to ensure the accuracy of the match. For example, if the two images are preset to conform to the similarity transformation, three pairs of matching points are randomly selected, the similarity transformation matrix is ​​calculated, and the matrix is ​​applied to all matching points. According to the Euclidean distance between the actual matching point and the theoretical matching point, the points that conform to the matrix are screened according to the preset threshold, which are called inliers, and the number is recorded. This sampling process is repeated continuously to find the model with the largest number of inliers, which is used as the global transformation parameter.

[0070] It should be noted that the matching criterion of RANSAC is the deviation between the current matching pair and the preset sampling model. In the case of large parallax, it is impossible to describe it with a single deformation model, such as an affine model or a similarity model. If the conventional threshold is used, the matched feature points are concentrated in a small part of the picture. In order to further solve the problem of poor stitching effect caused by too few feature points, the richness of the number of feature points is improved. This embodiment uses a preset proportional factor (technical personnel in this field can adaptively determine it based on actual applications or prior data) to relax the initial threshold (i.e., the conventional threshold used in the relevant technology). The threshold relaxation formula can be y=ax, where y is the relaxed threshold, a is the proportional factor, where a>1, and x is the initial threshold. In some embodiments, other threshold relaxation methods can also be used. Combined with Figure 3 As shown, Figure 3 Part (a) is an example of the original KNN matching point pair. Figure 3 Part (b) shows an example of matching point pairs using RANSAC in related technologies with a conventional threshold (e.g., 1 to 5 pixels). Figure 3 Part (c) is an example of a matching point pair (i.e., a first feature point matching pair) with a loose threshold (eg, when the preset scale factor is 2, the loose threshold is 1 to 10 pixels) adopted in this embodiment, where the white lines are connecting lines of the feature point matching pairs.

[0071] In the related technology, feature point matching KNN and model fitting method RANSAC work well in most small parallax stitching devices because they meet an assumption that the two images are approximately a global transformation, and a small parallax will not interfere with the calculation of the overall transformation matrix. Therefore, the preset threshold of RANSAC can be restricted to be more strict, so that the matching pairs obtained are highly reliable. Even if there are a small number of filtered points, only very slight misalignment will occur locally, which will not affect the overall stitching effect. Therefore, the conventional threshold of RANSAC is restricted to be more strict, so that the matching pairs obtained are highly reliable. Even if there are a small number of filtered points, only very slight misalignment will occur locally, which will not affect the overall stitching effect.

[0072] Different image acquisition devices have different feature distributions. Figure 4 As shown, taking the (high-precision) image acquisition device installed at a high place as an example, its feature point distribution is as follows Figure 4 As shown in part (a), the ordinate is consistent with the ordinate of the image, the abscissa represents the distance between matching points, the white dots represent matching points, and the red line is the result of quadratic curve fitting (the quadratic curve fitting here is used as an example for easy observation).

[0073] For large parallax devices, since the distance distribution of feature points at different physical distances is relatively wide, if a strict screening threshold is used, a large number of correct matching points will be filtered out, resulting in the registration result being biased towards the global level and failing to ensure the local alignment effect. Therefore, this embodiment sets a loose threshold to enrich the number of feature point matching pairs. Taking the large parallax device targeted by this embodiment as an example, the distribution of scenes with rich features is as follows: Figure 4 As shown in the middle (b), the degree of dispersion is significantly higher than that of the small parallax device, while the feature distribution of some relatively flat scenes or close scenes is relatively messy, such as Figure 4 As shown in Figure (c), combined with the subsequent feature point abnormality elimination process, a relatively large number of feature point matching pairs with high accuracy can be obtained.

[0074] Step S202: according to the line rotation angles of each first feature point matching pair, remove abnormal feature point matching pairs whose line rotation angle difference with that of other feature point matching pairs reaches a difference threshold, to obtain second feature point matching pairs.

[0075] In the related art, after constructing a feature point matching pair, the feature point matching pair is used to perform image stitching. When there is an erroneous matching of feature points, the image stitching effect is poor.

[0076] This embodiment takes into account that in the final ideal spliced ​​image, the feature points usually present a horizontal offset but no vertical offset, which is reflected in the original image. At this time, the correct rotation angle has not been found, and at least it meets the requirement that there will be no excessive rotation error. Therefore, this embodiment uses the rotation angle of the line between the feature point matching pairs to eliminate abnormal feature point matching pairs. Specifically, the difference between the rotation angle of the line between each feature point matching pair and the rotation angle of the line between other feature point matching pairs is calculated, and a difference threshold is used (technical personnel in this field can adaptively determine the difference threshold based on actual applications or prior data) to eliminate feature point matching pairs with abnormal rotation angles, so that the line segments connected between the matching points will not cross, such as Figure 5 As shown in the figure, even though the lengths of the black matching line segments are different, this is a normal phenomenon that may be caused by parallax. The matching lines in the white circle need to be filtered out because the rotation angle is too large, thereby eliminating the feature point matching pairs with abnormal rotation angles in the feature point matching pairs. Figure 6 Compared to Figure 3 Part (c) shows the example after feature removal. Figure 7 Part (a) is compared to Figure 3 Example of feature distribution after the feature points are rotated by the abnormal line in (c).

[0077] Step S203: according to the fitting result of the second feature point matching pair, remove the abnormal feature point matching pairs in the second feature point matching pairs that deviate from the fitting result, to obtain a third feature point matching pair.

[0078] For multi-eye cameras, as long as there is a certain monitoring distance, the parallax distribution in the picture should be generally continuous, and the overall law from top to bottom should be in line with the law from far to near. There may be some changes in the depth of field in the middle, but the amplitude is small and the change is slow. For more special scenes, such as close-up and unconventional close objects before monitoring, if the sudden change in depth of field is forced to align, it may cause distortion, which will reduce the effect of splicing. In addition, when the width of the overlapping area of ​​multiple spliced ​​images is large, the depth of field may also produce different distributions in the horizontal direction, thereby producing discontinuous visual phenomena in the spliced ​​image. Therefore, this embodiment obtains the fitting result by fitting the distance between the feature point matching pairs by a quadratic function curve (in some embodiments, other fitting methods, such as linear model fitting methods, etc., can also be used, and this embodiment does not specifically limit the specific fitting process), and removes points that deviate too much from the fitting value to further improve the quality of the feature point matching pairs, and solve the problems of image distortion caused by forced mutation in the related technology.

[0079] In some embodiments, an error threshold is used to identify the deviation of a feature point matching pair from a fitting result, wherein the error threshold is determined based on the standard deviation of all feature point matching pairs and a preset maximum error, so as to improve the effectiveness of removing abnormal feature points. Specifically, the above step S203 removes abnormal feature point matching pairs that deviate from the fitting result in the second feature point matching pair based on the fitting result of the second feature point matching pair, and may include the following steps: for each feature point matching pair in the second feature point matching pair, obtain the error value of the feature matching pair for the fitting result, and if the error value reaches a preset error threshold, determine that the feature point matching pair is an abnormal feature point matching pair that deviates from the fitting result; wherein the error threshold is determined based on the standard deviation calculated from the error values ​​of all feature point matching pairs in the second feature point matching pair and the preset maximum error.

[0080] Exemplarily, for each feature point matching pair in the second feature point matching pair, the error value relative to the fitting result is calculated. This error value can be the geometric distance of the feature point in the image space, or other suitable error metrics, by collecting the error values ​​of all feature point matching pairs and calculating the standard deviation of these error values. The standard deviation is a statistic that represents the distribution range and discreteness of the error value, and the error threshold is determined based on the calculated standard deviation of the error value and the preset maximum error. The preset maximum error can be adaptively determined by a person skilled in the art in combination with practical applications or prior art (for example, it can be set to 10), indicating the maximum error range acceptable to the feature point matching pair. The error threshold can be determined in the following manner: error threshold = min (standard deviation × coefficient, 10). This coefficient can be an empirical value used to adjust the strictness of the error threshold, for example, it can be 3 times the standard deviation. For each feature point matching pair, compare its error value with the determined error threshold. If the error value of a feature point matching pair exceeds the error threshold, it will be marked as an abnormal feature point matching pair, and the identified abnormal feature point matching pair will be removed from the second feature point matching pair to obtain the third feature point matching pair, so as to eliminate the points that deviate too much from the fitting value.

[0081] In further examples of some embodiments, the fitting result of this embodiment can be obtained by fitting with a quadratic curve fitting function and corrected by a loss function, providing data support for effectively eliminating abnormal deviated feature points. Specifically, the method provided in this embodiment may also include the following steps: based on the quadratic curve fitting function, fitting the distribution of the distance between each feature point matching pair in the second feature point matching pair in the ordinate direction to determine the initial parameters of the quadratic curve fitting function and obtain the initial fitting result; for each feature point matching pair in the second feature point matching pair, obtaining the initial error value of the feature matching pair for the initial fitting result, and constructing a loss function based on the sum of the initial error values ​​of each feature point matching pair; according to the loss function, correcting the initial parameters to obtain the fitting result.

[0082] For example, it is assumed that the images to be stitched include image A and image B, where the feature point sets are P and B respectively. A and P B , contains a total of N pairs of feature point matching pairs (referred to as feature point pairs), any feature point pair is ,in ,set up The actual distance between , the fitting distance is , where y represents the vertical coordinate of the pixel point, k1, k2, and k3 are polynomial fitting coefficients (i.e., the parameters of the quadratic curve fitting function). The fitting formula example is as follows:

[0083]

[0084] The fitting method is to minimize the loss function, where the loss function is constructed based on the sum of the initial error values ​​of each feature point matching pair to adjust the parameters k1, k2, and k3 in the fitting formula to make the fitting formula more accurate. The example is as follows:

[0085]

[0086] After getting the fitting results, the points that deviate from the fitting function by more than three times the standard deviation need to be eliminated. Considering the cluttered distribution of feature points in some scenes, it is necessary to limit the maximum value of the standard deviation. When the three times the standard deviation is greater than 10, it is taken as 10. Let the outliers (i.e., abnormal feature points) to be eliminated, and the standard deviation is σ, then

[0087]

[0088] The calculation process of the standard deviation σ is as follows:

[0089]

[0090] Through the above technical solution, combined with the distribution of feature points in the above example graph, before and after the use of the fitting outlier removal strategy, such as Figure 7 As shown in part (b), Figure 7 By comparing part (a) in the figure, we can see Figure 7 The feature points in the middle (b) figure have almost no deviated feature points. By removing abnormal deviation values, the quality of feature point matching pairs is effectively improved.

[0091] Step S204: insert a new feature point matching pair into an area outside the distribution area of ​​the third feature point matching pair in the image to be stitched, to obtain a fourth feature point matching pair; wherein the new feature point matching pair is determined based on the third feature point matching pair.

[0092] Considering that compared with conventional image acquisition devices, which have a narrow distribution range and good consistency, the relative position and distance distribution range of large parallax devices may be concentrated in some areas, resulting in the extracted feature points being concentrated in some areas, while other areas of the image have almost no feature points, which may cause a large distortion in the final stitching result. For example, in some scenes, the sky and the ground are relatively flat, and it is almost impossible to correctly extract feature points. This results in the overall fitting judgment being almost entirely dependent on the middle area. If the depth of field in the middle area changes greatly, the curvature of the fitting curve will be extremely large, and eventually serious anomalies will occur in the stitching image. For example, in indoor close-up scenes, such as Figure 7 As shown, Figure 7 Part (a) is the characteristic distribution and its fitting results. Figure 7 Part (b) shows the stitching result of images stitched together using these feature point matching pairs.

[0093] In order to solve the problem that the extracted feature point matching pairs are unevenly distributed, resulting in abnormal stitching results, this embodiment inserts feature points in the undistributed area of ​​the image to be stitched, wherein the inserted new feature point matching pairs are determined based on the existing feature point matching pairs (i.e., the third feature point matching pairs). For example, if the undistributed area is the upper and lower ends of the image, the new feature point matching pairs inserted at the upper end can be consistent with the existing feature points at the top of the distribution area, that is, the inserted new feature point matching pairs are consistent with the existing feature points in terms of transformation (such as translation, rotation, scaling, etc.), so that the inserted feature points maintain the same change trend as the existing feature points in geometric transformation. For example, if the feature points at the top of the image need to undergo some transformation (such as translation or rotation) during the stitching process, then the inserted feature points will also undergo the same transformation.

[0094] By inserting feature points that are consistent with the transformation of existing feature points in areas without feature points, the geometric constraints of these areas can be strengthened, preventing unreasonable deformation during the stitching process, helping to maintain the stability of various parts of the image and avoiding distortion and distortion caused by the lack of feature points.

[0095] In some embodiments, inserting a new feature point matching pair into an area outside the distribution area of ​​the third feature point matching pair in the image to be stitched comprises: if the distribution area of ​​the third feature point matching pair is located in the middle area of ​​the image to be stitched, then inserting a new feature point matching pair into the upper area of ​​the image to be stitched based on the feature point matching pair located at the upper end of the third feature point matching pair; and, inserting a new feature point matching pair into the lower area of ​​the image to be stitched based on the feature point matching pair located at the lower end of the third feature point matching pair.

[0096] Continuing with the indoor scene (i.e. Figure 8 ) as an example, in order to ensure that the featureless areas above and below the picture do not produce a trend of change that is significantly greater than a reasonable value, this embodiment adopts a simple and efficient feature enhancement strategy: feature points are inserted at a certain number of pixels at the top and bottom, and their transformation is consistent with the top and bottom feature points (the feature point pairs located at the top and the smallest segment in the third feature point matching pair), where the top and bottom feature points represent the depth of field values ​​at the farthest or closest distance. This strategy will not affect the alignment of valid features, but adds strong constraints in the featureless area, effectively avoiding severe distortion. Figure 8 Corresponding scenes, the enhanced feature distribution is as follows Fig. 9 As shown in the figure, the optimized splicing effect is as follows Fig.10 As shown, it can be seen that the splicing effect is significantly optimized.

[0097] Next, the method of inserting a new feature point matching pair in the upper end area of ​​the image to be stitched based on the feature point matching pair located at the upper end among the third feature point matching pairs is further introduced: according to a first distance between the feature point matching pair located at the upper end among the third feature point matching pairs and the upper boundary of the image to be stitched, and a preset insertion interval distance, a first insertion number of feature point matching pairs to be inserted in the upper end area is determined; according to the first insertion number, a new feature point matching pair is inserted upward along the pixel vertical coordinate direction of the uppermost feature point matching pair at every insertion interval distance until the insertion number reaches the first insertion number, so as to insert a new feature point matching pair in the upper end area of ​​the image to be stitched.

[0098] Exemplarily, taking the upper left corner of the coordinate system of the image to be stitched as the origin (0, 0) as an example, the horizontal rightward is the positive direction of the x-axis, and the vertical downward is the positive direction of the y-axis. The upper boundary of the image corresponds to the horizontal line with a y coordinate of 0. Assuming that the pixels of the uppermost feature point matching pair are a (x1, y1) and b (x2, y2), the first distance, i.e., y=y1=y2 (using the horizontal translation of the image as an example for explanation, in some embodiments, y1 and y2 may also be unequal), reflects the longitudinal axis distance from the uppermost feature point matching pair in the image to be stitched to the image boundary. Since the feature point matching pair is at the uppermost end of the image to be stitched, it means that there are no feature points between the uppermost end and the upper boundary of the image. This embodiment inserts feature point pairs within the first distance range to solve the problem of lack of feature point matching pairs in some distribution areas in such a large parallax scene.

[0099] After calculating the first distance, the number of feature point matching pairs to be inserted can be calculated in combination with the preset insertion interval distance, such as the first insertion number n=y / y', where y' is the insertion interval distance, and the insertion interval distance can be determined according to the longitudinal axis distance between the currently identified feature point matching pairs (such as equal, or proportional), so that the inserted feature point matching pairs are more consistent with the identified feature point matching pairs, thereby making the distribution density of feature points in the image of the inserted feature point matching pairs more consistent, thereby further optimizing the image stitching effect. In some embodiments, the user can also make adaptive adjustments or determinations based on empirical values, and dynamically adjust the insertion interval distance according to the actual distribution of feature points to better adapt to the feature distribution of different images. It can be understood that for each insertion interval, the pixel horizontal coordinates of the inserted feature point matching pair are consistent with the pixel horizontal coordinates of the existing feature point matching pair, and the pixel vertical coordinates are (y- y'*1), (y- y'*2),..., (y- y'*n) respectively.

[0100] Correspondingly, the above steps of inserting a new feature point matching pair in the lower end region of the image to be spliced ​​based on the feature point matching pair located at the lower end among the third feature point matching pairs may include the following steps: determining the second insertion number of feature point matching pairs to be inserted in the lower end region according to the second distance between the feature point matching pair located at the lowest end among the third feature point matching pairs and the lower boundary of the image to be spliced, and the preset insertion interval distance; according to the second insertion number, inserting a new feature point matching pair every the insertion interval distance downward along the pixel ordinate direction of the feature point matching pair at the lowest end until the insertion number reaches the second insertion number, so as to insert a new feature point matching pair in the lower end region of the image to be spliced. For related instructions, please refer to the above-mentioned upper end feature point insertion process, which will not be described in detail here.

[0101] Through the above technical solution, the situation in which the feature points are concentrated in the middle area of ​​the image in a large parallax scene, while the feature points are lacking in the two end areas can be effectively solved. By calculating the number of feature point pairs to be inserted and inserting feature point pairs in the corresponding insertion intervals, the feature point pairs in the image can be enhanced, and the distribution of the inserted feature point pairs is more coordinated with the distribution of the existing feature point pairs, further improving the quality of the feature point matching pairs, thereby achieving the purpose of further optimizing the image stitching effect.

[0102] In some other embodiments, in addition to feature point enhancement for the upper and lower regions, feature point pairs can also be inserted for the corresponding undistributed regions based on other undistributed regions of the actual image. The insertion method is similar to the feature point insertion for the upper and lower regions mentioned above, and the relevant description will not be repeated.

[0103] Step S205 : generating a registration parameter for image stitching according to the fourth feature point matching pair, and performing stitching processing on the multiple images to be stitched based on the registration parameter to obtain a stitched image.

[0104] In this embodiment, the registration parameters can be obtained through the image stitching algorithm, which describes the mapping method of the image to be stitched to the stitching image, that is, the mapping relationship between the image to be stitched and the stitching common plane. In this embodiment, it can include the global geometric transformation matrix of the image, which can include translation parameters, rotation parameters, scaling parameters, etc. Next, with the help of hardware capabilities, the effect of real-time output of the stitching video code stream is achieved. The final stitching video is obtained by mapping two or more channels of images respectively and then combining the fusion algorithm. There are different types of fusion algorithms according to the device platform capabilities, which are not particularly limited in this embodiment.

[0105] The registration parameters generated by feature point matching pairs through multiple feature point screening and feature point enhancement improve the registration quality and are compatible with a variety of large parallax scenes, thereby solving the problem of stitching multiple cameras with variable structures and large viewpoint spacing, and achieving better stitching effects.

[0106] In some embodiments, in order to further improve the image stitching effect, taking into account that errors in actual projection may occur due to parallax and lens optical distortion errors, the alignment parameters of this embodiment may include not only a global geometric transformation matrix but also a local deformation function. The above-mentioned step S205 generates alignment parameters for image stitching according to the fourth feature point matching pair, and may include the following steps: generating a global geometric transformation matrix for indicating the geometric transformation relationship between the multiple images to be stitched according to the fourth feature point matching pair; for each feature point matching pair in the fourth feature point matching pair, obtaining the projection error of the feature point matching pair according to the geometric transformation matrix; and determining the local deformation function for correcting the local feature point matching pair according to the thin plate spline function and the projection error.

[0107] For devices with large parallax, due to the depth distribution law of the upper far and lower near, when calculating the global similarity matrix with rotation, rotation error may be introduced, and a certain amount of bending effect may be easily generated at the end of the transition zone. Fig.11 and Fig.12 These are the stitching results of the global model with rotation and the global model without rotation. This embodiment takes into account that the rotation tolerance of the stitching equipment hardware is not large, which can be solved by local deformation alignment. In addition, if it is to be expanded to multiple objects, the global similarity matrix without rotation can effectively avoid error accumulation and bring more robust results.

[0108] In this embodiment, after calculating the global geometric transformation matrix H (H describes the overall relationship between the feature point sets, and H can be a rigid, affine, or homography transformation, wherein the process of generating the global geometric transformation matrix using feature point matching pairs can refer to the relevant technical content, and the relevant description will not be repeated), the relationship between the feature points of the two images to be stitched can be expressed as follows: , and due to the error of parallax and lens optical distortion, the actual projection may have errors. Assuming p i 'It is p i Using the projection point calculated by the matrix, the projection error is In order to make the registration as accurate as possible and minimize the local distortion, the present embodiment uses the thin plate spline function (TPS) to construct the local deformation equation, which is a composite energy equation. The energy function is minimized to achieve the goal of local alignment but minimum overall deformation. For example, the energy equation contains two terms, one of which is the registration term J D , and the second is the smoothing term J S , which is used to balance the registration accuracy and deformation smoothness. There is a weight coefficient λ between these two items, which can be adjusted according to the actual situation. The local deformation equation example is as follows:

[0109]

[0110] in,

[0111]

[0112]

[0113] Where Ω represents the definition range of the coordinates, represents the Laplace operator, that is, calculating the second-order partial derivative of the projection error with respect to x and y.

[0114] In this embodiment, the TPS theory can be used to solve the solution to minimize the energy equation, thereby obtaining the mapping relationship between each point in the image to be stitched and the target image. Specifically, according to the TPS theory, the solution to minimize the energy equation is as follows:

[0115]

[0116] Among them, g(x,y) is the local deformation function, is the radial basis function, coefficient ω i and α can be solved using linear equations:

[0117]

[0118] Among them, K is the matrix calculated by the radial basis function, and λ is the regularization parameter used to control the smoothness of the fitting. is a constant matrix used for regularization, I is the unit matrix. P is the coordinate matrix of the feature points, is a coefficient vector containing all ω i , α is the coefficient vector of the linear term, containing , f is the vector of objective function values, usually the coordinates of feature points in the target image. 3×3 and 0 3×1 is a zero matrix and a zero vector, which are used to construct the right side of the linear equation system. According to the above formula, as long as the number of feature point matching pairs is greater than or equal to 4, the points at any position of the source image can be mapped to the target image.

[0119] Through the above technical solution, compared with the related art in which the global matrix H is not ideal, the global matrix H is directly adjusted (or by dividing the grid, each grid corresponds to a local matrix h), this embodiment uses local elastic deformation for pixel-level alignment, which can flexibly handle local geometric deformations while maintaining global consistency, thereby achieving high-precision image registration and stitching.

[0120] In a further example of this embodiment, this embodiment uses non-overlapping area interpolation transition to achieve smooth transition from overlapping area to non-overlapping area. Specifically, the registration parameter may also include an interpolation transition function; the above step S205 generates the registration parameter for image stitching based on the fourth feature point matching pair, and may also include the following steps: determining a transition area from the overlapping area to the non-overlapping area between the multiple stitched images; determining an interpolation transition function for smooth transition of pixel points in the transition area based on the local deformation function and the preset transition factor.

[0121] It can be understood that the overlapping area between multiple stitched images is the area formed by the feature point matching pair, and the non-overlapping area is the area outside the overlapping area. The transition area is a buffer area used to smoothly reduce the influence of the overlapping area to the non-overlapping area. The width d of the transition area is s It can be determined based on the maximum offset and a proportional constant K, as shown below:

[0122]

[0123] in, They represent the absolute value of the deformation of each feature point or pixel in the image. The deformation can be caused by the geometric transformation (such as translation, rotation, scaling, etc.) applied during the image registration process. is the absolute value of the deformation in the x-axis direction, is the absolute value of the deformation in the y-axis direction.

[0124] And according to the width of the transition zone, the preset transition factor η is determined, as shown in the following example:

[0125]

[0126] in, and is the coordinate point of the overlap area boundary.

[0127] Next, the interpolation transition function is determined according to the local deformation function and the transition factor. The interpolation transition function is implemented by gradually reducing the deformation function g(x,y) to 0. The example is as follows:

[0128]

[0129] When the point (x, y) gradually moves from the overlapping area to the non-overlapping area, η gradually changes from 1 to 0.

[0130] Through the above technical solution, the deformation of non-overlapping areas can be effectively processed during the image stitching process to ensure the smooth transition and visual consistency of the images, thereby achieving the purpose of further optimizing the image stitching effect.

[0131] In some embodiments, in order to further improve the image stitching effect, the image is subjected to distortion correction and projection in the feature extraction stage, and feature points are extracted in combination with a specific feature extraction algorithm to improve the matching accuracy of the feature points, and provide data support for the above-mentioned feature screening and enhancement. Specifically, before the above-mentioned step S201 constructs multiple first feature point matching pairs based on the consistency recognition results between the feature points of multiple images to be stitched, the following steps may also be included: obtaining multiple original images captured by multiple cameras; performing image distortion correction on the multiple original images to obtain multiple corrected images; projecting the multiple corrected images onto a common projection surface to obtain the multiple images to be stitched; and extracting all feature points in the multiple images to be stitched according to a preset feature extraction algorithm.

[0132] In this embodiment, multiple original images captured by the multi-eye camera may be acquired in real time, or multiple original images captured by the multi-eye camera may be acquired from an image storage device, and the specific acquisition process is not particularly limited.

[0133] After acquiring the original image, considering the correction of image distortion caused by the camera lens or other factors, this embodiment first performs distortion correction on the original image. Distortion mainly includes radial distortion and tangential distortion, wherein radial distortion refers to the stretching or compression of the area near the center of the image, and tangential distortion refers to the stretching or compression of the area near the edge of the image. Common distortion types also include barrel distortion and pincushion distortion, which are respectively manifested as a convex image center and a concave edge, and a concave image center and a convex edge.

[0134] Exemplarily, the image distortion correction process can use Zhang Zhengyou calibration method to shoot checkerboard images at different poses and distances to obtain multiple sets of nonlinear equations. The camera intrinsic parameter matrix (fx, fy, cx, cy) and distortion parameters (k1, k2, p1, p2, k3, k4, k5, k6) can be solved through a nonlinear optimization algorithm to obtain a corrected image after distortion correction.

[0135] Next, the multiple corrected images are projected onto a common projection surface, which can be a plane, a sphere, or a cylinder, depending on the application scenario and stitching requirements. The purpose of projection is to transform images from different perspectives into the same coordinate system for subsequent feature extraction and stitching processing. After the images to be stitched are obtained by projection, it is necessary to find the spatial relationship between the images to be stitched, which needs to be achieved through feature matching in the overlapping area. In this embodiment, key point features are extracted and matched. These feature points are some significant features in the image, such as corner points, edges or textures. The purpose of the preset feature extraction algorithm is to find these points and generate descriptors for them. These descriptors can be used to match the same objects or scenes in different images. Optionally, the feature point extraction algorithm can use SIFT, SURF, AKAZE to extract feature points that remain unchanged under different scales, rotations and lighting conditions. In this embodiment, AKAZE features are used to extract feature points. Even when the image conditions are not ideal, these feature points can be matched, thereby improving the robustness of stitching. After the feature point extraction is completed, the feature point matching process can be executed to obtain feature point matching pairs and perform subsequent feature point matching pair elimination and enhancement processing to complete the image stitching.

[0136] In a further example of this embodiment, in order to better maintain the geometric shape of the image when effectively processing an image with a large horizontal viewing angle, avoid perspective deformation, and obtain a better planar visual effect, this embodiment uses a cylindrical projection method to project the image to be spliced. Specifically, the common projection surface is a cylindrical image; projecting the multiple corrected images onto the common projection surface includes: projecting the pixels of the multiple corrected images onto the cylindrical image according to a cylindrical projection algorithm.

[0137] It can be understood that the goal of projection transformation is to project multiple images onto a common projection plane, and then perform registration and splicing to obtain a smooth splicing image. Cylindrical projection refers to projecting a plane image onto a cylinder, and then unfolding it to the image plane along a generatrix of the cylinder. In addition to using cylindrical projection transformation, other projection methods that can be selected include plane projection, spherical projection, and cubic projection, etc. The present invention chooses to use cylindrical projection transformation.

[0138] For the rectified image, the projection radius can be calculated as follows:

[0139]

[0140] Where W is the width of the corrected image, which can be directly obtained from the optical design parameters, and α is the field of view of the lens. Fig.13Figures (a) to (b) in the figure are schematic diagrams of cylindrical projection, where Figure (a) is a side view of the cylindrical projection, in which O1, x, y represent the coordinate system of the plane image, ABC are the vertices of the plane image, P and P' are two arbitrary points O in the plane image, X, Y, Z represent the spatial coordinate system of the cylinder, O is the coordinate origin and also the projection center point, Q and Q' are the projection points of P and P' on the cylinder; Figure (b) is a top view of the cylindrical projection, O represents the cylindrical coordinate origin and also the projection center point, w is the width of the plane image, f is the projection radius and also the camera focal length, P' and Q' is a set of original points and projection points, which has the same meaning as Figure (a), s is the projection point corresponding to the edge point of the original image, θ is the half field of view angle, and α is the projection angle corresponding to P'; Figure (c) is the expanded image after cylindrical projection, where O2, x, y represents the coordinate system of the cylindrical expanded image, Q and Q' represent the projection points, corresponding to Figure (a), but Figure a is still in a cylindrical state, and Figure c is in an expanded state. The reason why the corners are arc-shaped is because the projection center is point O and remains unchanged. When the points on the corners of the original plane are projected, the height will be smaller than the height of the projection of the upper and lower edge points in the middle.

[0141] Specifically, the image plane is first projected onto the cylinder, and then expanded to the image plane along a generatrix of the cylinder. The radius of the projection cylinder is the focal length of the camera f (in pixels), and the projection center is located at the origin of the coordinate system O. The image plane I to be projected is located at the coordinate system z=f, that is, tangent to the cylinder and intersects the Z axis at the center of the image. The width and height of the image are W and H respectively, then the coordinates of the point on the image plane I are , the horizontal half field of view of the image is:

[0142]

[0143] The width and height of the projected image are:

[0144]

[0145]

[0146] Then the pixel coordinates of the cylindrical projection point Q corresponding to point P are for:

[0147]

[0148]

[0149] Through the above cylindrical projection method, perspective deformation can be effectively avoided and a better planar visual effect can be obtained.

[0150] For example, the images before and after distortion correction and cylindrical projection are as follows: Fig.14 and Fig.15As shown in the figure, for this sample image, the final effect of stitching and calibration after combining the above-mentioned feature point multiple screening and enhancement, local deformation correction, etc. is as follows Fig.16 As shown, there is almost no splicing error and distortion, and a good splicing effect can be achieved.

[0151] To further verify the effectiveness of this embodiment, factory calibration, field calibration in related technologies, and field calibration based on multiple feature screening and enhancement provided by this embodiment are used to stitch images of the same multi-camera. The original image and the processed image are shown in Figure 2. Figure 17 to Figure 22 As shown, Fig.17 Parts (a) and (b) are two original images from the multi-view camera. Fig.18 Parts (a) and (b) in the middle are magnified details of two parts of the original image, corresponding to the real texture near and far. Fig.19 Part (a) is the stitched image after factory calibration. Fig.19 Part (b) in the middle is a magnified detail of the stitched image, where it can be seen that a double image appears on the distant building; Fig. 20 Part (a) is the spliced ​​image after on-site calibration in the related art. Fig. 20 Part (b) in the middle is a magnified detail of the stitched image, where we can see a ghost image in the middle of the grass. Fig.21 is a spliced ​​image after on-site calibration using this embodiment, Fig. 22 (a), (b), and (c) are three magnified details of the stitched image, and it can be seen that the details of the near and far are preserved without ghosting. It can be seen that this embodiment can accommodate the parallax of objects at different distances in the same scene for complex and changeable scene environments and devices with variable structures, with high alignment accuracy and no obvious stitching seams and ghosting.

[0152] In summary, the present embodiment is compatible with a multi-camera on-site stitching calibration method with large parallax. Through multiple feature point screening and enhancement, the elimination of mismatched points is strengthened, and the number of effective feature points is enriched. At the same time, the overall transformation is constrained to be compatible with a variety of large parallax scenes. The thin plate spline function is used for local elastic deformation to improve the alignment effect. The interpolation transition between the overlapping area and the non-overlapping area is performed to ensure smoothness, and the influence of parallax is better eliminated. It can be compatible with objects at different distances in the same scene without adjusting the algorithm parameters, and has strong compatibility in various scenes.

[0153] The present application also provides an image stitching processing device. Fig.23As shown, the device 2500 includes a matching pair construction module 2501, a feature elimination module 2502, a feature enhancement module 2503 and a stitching processing module 2504, wherein the matching pair construction module 2501 is configured to construct a plurality of first feature point matching pairs according to the consistency recognition results between the feature points of a plurality of images to be stitched; the feature elimination module 2502 is configured to eliminate the abnormal feature point matching pairs whose difference between the connection rotation angle of the first feature point matching pairs and the connection rotation angle of other feature point matching pairs reaches a difference threshold according to the connection rotation angle of each first feature point matching pair, so as to obtain the second feature point matching pairs; the feature elimination module 2502 is also configured to eliminate the abnormal feature point matching pairs whose difference between the connection rotation angle of the first feature point matching pairs and the connection rotation angle of other feature point matching pairs reaches a difference threshold according to the connection rotation angle of each first feature point matching pair; The fitting result of the second feature point matching pair is obtained by removing abnormal feature point matching pairs in the second feature point matching pair that deviate from the fitting result, so as to obtain a third feature point matching pair; a feature enhancement module 2503 is configured to insert a new feature point matching pair into an area outside the distribution area of ​​the third feature point matching pair in the image to be stitched, so as to obtain a fourth feature point matching pair; wherein the new feature point matching pair is determined based on the third feature point matching pair; a stitching processing module 2504 is configured to generate a registration parameter for image stitching according to the fourth feature point matching pair, and stitch the multiple images to be stitched based on the registration parameter to obtain a stitched image.

[0154] In one embodiment, the consistency recognition result is obtained based on a nearest neighbor algorithm; the matching pair construction module 2501 includes: a first determination unit, which is configured to determine multiple potential feature point matching pairs based on the consistency recognition result between the feature points; a filtering unit, which is configured to filter the potential feature point matching pairs among the multiple potential feature point matching pairs whose deviation from the sampling model is greater than a preset threshold based on a random sampling consistency algorithm; wherein the preset threshold is a threshold after relaxing the initial threshold of the random sampling consistency algorithm according to a preset proportional factor.

[0155] In one embodiment, the feature enhancement module 2503 includes: a first insertion unit, which is configured to insert a new feature point matching pair in the upper end area of ​​the image to be stitched based on the feature point matching pair located at the upper end of the third feature point matching pair if the distribution area of ​​the third feature point matching pair is located in the middle area of ​​the image to be stitched; and a second insertion unit, which is configured to insert a new feature point matching pair in the lower end area of ​​the image to be stitched based on the feature point matching pair located at the lower end of the third feature point matching pair.

[0156] In one embodiment, the first insertion unit is specifically configured to determine a first insertion number of feature point matching pairs to be inserted in the upper end region according to a first distance between the uppermost feature point matching pair among the third feature point matching pairs and the upper boundary of the image to be stitched, and a preset insertion interval distance; according to the first insertion number, a new feature point matching pair is inserted upward along the pixel vertical coordinate direction of the uppermost feature point matching pair at intervals of the insertion interval distance until the insertion number reaches the first insertion number, so as to insert a new feature point matching pair in the upper end region of the image to be stitched.

[0157] In one embodiment, the second insertion unit is specifically configured to determine a second insertion number of feature point matching pairs to be inserted in the lower end area based on a second distance between the feature point matching pair located at the lowermost end of the third feature point matching pairs and the lower boundary of the image to be stitched, and a preset insertion interval distance; according to the second insertion number, a new feature point matching pair is inserted downward along the pixel vertical coordinate direction of the feature point matching pair at the lowermost end at intervals of the insertion interval until the insertion number reaches the second insertion number, so as to insert a new feature point matching pair in the lower end area of ​​the image to be stitched.

[0158] In one embodiment, the feature removal module 2502 includes: an abnormal feature determination module, which is configured to obtain, for each feature point matching pair in the second feature point matching pair, an error value of the feature matching pair with respect to the fitting result, and if the error value reaches a preset error threshold, determine that the feature point matching pair is an abnormal feature point matching pair that deviates from the fitting result; wherein the error threshold is determined based on a standard deviation calculated from the error values ​​of all feature point matching pairs in the second feature point matching pair and a preset maximum error.

[0159] In one embodiment, the device also includes: an initial fitting module, which is configured to fit the distribution of the distance between each feature point matching pair in the second feature point matching pair in the vertical coordinate direction based on a quadratic curve fitting function to determine the initial parameters of the quadratic curve fitting function and obtain an initial fitting result; a loss function construction module, which is configured to obtain the initial error value of the feature matching pair for the initial fitting result for each feature point matching pair in the second feature point matching pair, and construct a loss function based on the sum of the initial error values ​​of each feature point matching pair; and a correction fitting module, which is configured to correct the initial parameters according to the loss function to obtain the fitting result.

[0160] In one embodiment, the registration parameters include a global geometric transformation matrix and a local deformation function; the stitching processing module 2504 includes: a matrix generation unit, which is configured to generate a global geometric transformation matrix for indicating the geometric transformation relationship between the multiple images to be stitched according to the fourth feature point matching pair; a projection error acquisition unit, which is configured to obtain the projection error of the feature point matching pair according to the geometric transformation matrix for each feature point matching pair in the fourth feature point matching pair; a local deformation determination module, which is configured to determine the local deformation function for correcting the local feature point matching pair according to the thin plate spline function and the projection error.

[0161] In one embodiment, the alignment parameters also include an interpolation transition function; the stitching processing module 2504 also includes: a transition area determination unit, which is configured to determine the transition area from the overlapping area to the non-overlapping area between the multiple stitched images; a transition function determination unit, which is configured to determine the interpolation transition function for smooth transition of pixel points in the transition area based on the local deformation function and a preset transition factor.

[0162] In one embodiment, the device also includes: an original image acquisition module, which is configured to acquire multiple original images collected by multiple cameras; a distortion correction module, which is configured to perform image distortion correction on the multiple original images to obtain multiple corrected images; a projection module, which is configured to project the multiple corrected images onto a common projection surface to obtain the multiple images to be stitched; and a feature point extraction module, which is configured to extract all feature points in the multiple images to be stitched according to a preset feature extraction algorithm.

[0163] In one implementation, the common projection surface is a cylindrical image; and the projection module is specifically configured to: project pixels of the plurality of corrected images onto the cylindrical image according to a cylindrical projection algorithm.

[0164] Fig.24 A schematic diagram of the structure of the server provided in the embodiment of the present application, such as Fig.24 As shown, the server includes: a processor 2602, and a memory 2601 that is in communication with the processor 2602; the memory 2601 stores computer-executable instructions; the processor 2602 executes the computer-executable instructions stored in the memory 2601 to implement the image stitching processing method in any of the above method embodiments. The memory 2601 and the processor 2602 are connected via a bus 2603.

[0165] The embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to execute the image stitching processing methods provided in the various embodiments described above.

[0166] The computer-readable storage medium mentioned above can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory, electrically erasable programmable read-only memory, erasable programmable read-only memory, programmable read-only memory, read-only memory, magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0167] Optionally, a readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0168] An embodiment of the present application also provides a computer program product, which includes a computer program. The computer program is stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When at least one processor executes the computer program, it can implement the technical solution provided by any of the above method embodiments.

[0169] It should be noted that the apparatus and related devices provided in the above embodiments are used to execute any image stitching processing method provided in the above method embodiments, and their implementation principles and technical effects are similar and will not be repeated here.

[0170] It is to be understood that the various numerical numbers involved in the embodiments of the present application are only for the convenience of description and are not intended to limit the scope of the embodiments of the present application. In the embodiments of the present application, the size of the sequence number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0171] In this application, "at least one" means one or more, and "more" means two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural.

[0172] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0173] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. An image stitching processing method, characterized in that: include: Determining a plurality of potential feature point matching pairs according to a consistency recognition result between feature points of a plurality of images to be stitched, wherein the consistency recognition result is obtained based on a nearest neighbor algorithm; According to the random sampling consistency algorithm, the potential feature point matching pairs whose deviation from the sampling model is greater than a preset threshold among the multiple potential feature point matching pairs are filtered to obtain multiple first feature point matching pairs; wherein the preset threshold is a threshold value obtained by relaxing the initial threshold value of the random sampling consistency algorithm according to a preset proportional factor; According to the connection rotation angle of each first feature point matching pair, the abnormal feature point matching pairs whose difference between the connection rotation angle of the first feature point matching pair and the connection rotation angle of other feature point matching pairs reaches the difference threshold are eliminated to obtain the second feature point matching pairs; According to the fitting result of the second feature point matching pair, abnormal feature point matching pairs that deviate from the fitting result are eliminated from the second feature point matching pairs to obtain third feature point matching pairs; According to the distribution area of ​​the third feature point matching pair in the image to be stitched, insert a new feature point matching pair into the area outside the distribution area to obtain a fourth feature point matching pair; wherein the new feature point matching pair is determined based on the third feature point matching pair; According to the fourth feature point matching pair, a registration parameter for image stitching is generated, and the plurality of images to be stitched are stitched based on the registration parameter to obtain a stitched image.

2. The method according to claim 1, characterized in that: The inserting a new feature point matching pair into an area outside the distribution area of ​​the third feature point matching pair in the to-be-joined image comprises: If the distribution area of ​​the third feature point matching pair is located in the middle area of ​​the image to be stitched, inserting a new feature point matching pair in the upper area of ​​the image to be stitched based on the feature point matching pair located at the upper end of the third feature point matching pair; and A new feature point matching pair is inserted into the lower end region of the image to be stitched based on the feature point matching pair located at the lower end of the third feature point matching pair.

3. The method according to claim 2, characterized in that The inserting a new feature point matching pair in the upper end region of the image to be stitched based on the feature point matching pair located at the upper end of the third feature point matching pair comprises: Determining a first insertion number of feature point matching pairs to be inserted into the upper end region according to a first distance between the feature point matching pair located at the uppermost end of the third feature point matching pairs and the upper boundary of the image to be stitched, and a preset insertion interval distance; According to the first insertion quantity, a new feature point matching pair is inserted upward along the pixel ordinate direction of the uppermost feature point matching pair at every insertion interval until the insertion quantity reaches the first insertion quantity, so as to insert a new feature point matching pair in the upper end area of ​​the image to be stitched.

4. The method according to claim 2, characterized in that: The inserting a new feature point matching pair in the lower end region of the image to be stitched based on the feature point matching pair located at the lower end of the third feature point matching pair comprises: Determine a second insertion number of feature point matching pairs to be inserted into the lower end region according to a second distance between a feature point matching pair located at the lowermost end of the third feature point matching pairs and a lower boundary of the image to be stitched, and a preset insertion interval distance; According to the second insertion quantity, a new feature point matching pair is inserted downward along the pixel ordinate direction of the lowermost feature point matching pair at every insertion interval until the insertion quantity reaches the second insertion quantity, so as to insert a new feature point matching pair in the lower end area of ​​the image to be stitched.

5. The method according to claim 1 or 2 or 3 or 4, characterized in that: The removing, according to the fitting result of the second feature point matching pair, abnormal feature point matching pairs that deviate from the fitting result in the second feature point matching pair, comprises: For each feature point matching pair in the second feature point matching pairs, obtaining an error value of the feature point matching pair with respect to the fitting result, and if the error value reaches a preset error threshold, determining that the feature point matching pair is an abnormal feature point matching pair that deviates from the fitting result; The error threshold is determined based on a standard deviation calculated from the error values ​​of all feature point matching pairs in the second feature point matching pairs and a preset maximum error.

6. The method according to claim 5, characterized in that Also includes: Based on the quadratic curve fitting function, fitting the distribution of the distance between each feature point matching pair in the second feature point matching pair in the ordinate direction to determine the initial parameters of the quadratic curve fitting function and obtain an initial fitting result; For each feature point matching pair in the second feature point matching pairs, obtaining an initial error value of the feature point matching pair with respect to the initial fitting result, and constructing a loss function according to the sum of the initial error values ​​of each feature point matching pair; According to the loss function, the initial parameters are modified to obtain the fitting result.

7. The method according to claim 1 or 2 or 3 or 4 or 6, characterized in that: The registration parameters include a global geometric transformation matrix and a local deformation function; the generating the registration parameters for image stitching according to the fourth feature point matching pair includes: generating a global geometric transformation matrix for indicating a geometric transformation relationship between the plurality of images to be stitched according to the fourth feature point matching pair; For each feature point matching pair in the fourth feature point matching pair, obtaining a projection error of the feature point matching pair according to the global geometric transformation matrix; A local deformation function for correcting the matching pairs of local feature points is determined according to the thin plate spline function and the projection error.

8. The method according to claim 1 or 2 or 3 or 4 or 6, characterized in that: Before constructing a plurality of first feature point matching pairs according to the consistency recognition results between the feature points of the plurality of images to be stitched, the method further includes: Acquire multiple original images collected by multiple cameras; Performing image distortion correction on the multiple original images to obtain multiple corrected images; Projecting the multiple corrected images onto a common projection surface to obtain the multiple images to be stitched; According to a preset feature extraction algorithm, all feature points in multiple images to be stitched are extracted.

9. An image stitching processing device, characterized in that: include: A matching pair construction module, which is configured to determine a plurality of potential feature point matching pairs according to a consistency recognition result between feature points of a plurality of images to be stitched, wherein the consistency recognition result is obtained based on a nearest neighbor algorithm; According to the random sampling consistency algorithm, the potential feature point matching pairs whose deviation from the sampling model is greater than a preset threshold among the multiple potential feature point matching pairs are filtered to obtain multiple first feature point matching pairs; wherein the preset threshold is a threshold value obtained by relaxing the initial threshold value of the random sampling consistency algorithm according to a preset proportional factor; A feature elimination module is configured to eliminate abnormal feature point matching pairs whose difference between the line rotation angle in the first feature point matching pair and the line rotation angle of other feature point matching pairs reaches a difference threshold according to the line rotation angle of each first feature point matching pair, so as to obtain second feature point matching pairs; The feature elimination module is further configured to eliminate abnormal feature point matching pairs that deviate from the fitting result in the second feature point matching pairs according to the fitting result of the second feature point matching pairs, so as to obtain third feature point matching pairs; A feature enhancement module, which is configured to insert a new feature point matching pair into an area outside the distribution area of ​​the third feature point matching pair in the image to be stitched, to obtain a fourth feature point matching pair; wherein the new feature point matching pair is determined based on the third feature point matching pair; A stitching processing module is configured to generate a registration parameter for image stitching according to the fourth feature point matching pair, and perform stitching processing on the multiple images to be stitched based on the registration parameter to obtain a stitched image.

Images