Image stitching method and system based on deep point line features
By employing an image stitching method based on depth point and line features, utilizing feature extraction networks and geometric coplanar feature matching, and combining mesh deformation constraints, the mismatch problem in image stitching is solved, achieving high-precision and realistic image stitching results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- WUHAN UNIV
- Filing Date
- 2024-01-12
- Publication Date
- 2026-06-26
AI Technical Summary
Existing image stitching techniques struggle to effectively achieve realistic results after image stitching, especially when dealing with regions with repeated textures where high feature descriptor similarity leads to mismatch issues.
An image stitching method based on depth point and line features is adopted. Point features, line features, and point-line feature descriptions are obtained through a feature extraction network. Feature matching is performed by combining geometric coplanar features, and image pixel fusion is performed using grid deformation constraints to output the stitched image result.
It improves the alignment accuracy and fidelity of image stitching, ensuring that the stitched image has a realistic visual effect and effectively solves the problem of mismatch in areas with repeated textures.
Smart Images
Figure CN117974472B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and in particular to an image stitching method and system based on depth point and line features. Background Technology
[0002] Image stitching can provide large-field-of-view, high-resolution images for various applications without losing image information, making it a hot research topic in computer vision. In remote sensing image processing, single images taken by satellites and aircraft have limited perspectives, requiring stitching to obtain high-resolution panoramic images. In the medical field, stitching together multiple medical images (such as X-rays, CT scans, MRI scans, etc.) can provide more comprehensive and accurate diagnostic information, helping doctors better understand the location and morphology of lesions. Furthermore, image stitching can be used to create various visual effects, such as placing virtual images in specific locations in the real world, or allowing virtual images to interact with real-world objects, enhancing the immersion and realism of VR.
[0003] Currently, most methods for image stitching are still in the early stages of processing and have failed to achieve a realistic effect after image stitching. Therefore, it is necessary to propose new image stitching methods. Summary of the Invention
[0004] This invention provides an image stitching method and system based on depth point and line features to address the shortcomings of existing technologies.
[0005] In a first aspect, the present invention provides an image stitching method based on depth point and line features, comprising:
[0006] Acquire scene images to be stitched together and obtain a feature extraction network;
[0007] The scene image to be stitched is input into the feature extraction network to extract point features, line features, and point-line feature descriptions;
[0008] Obtain the geometric coplanar features of the scene image to be stitched, and use the geometric coplanar features and the point and line features to perform feature matching to obtain the image stitching model;
[0009] Obtain the mesh deformation constraints of the scene image to be stitched, perform image pixel fusion of the mesh deformation constraints and the image stitching model, and output the stitched image result.
[0010] According to the present invention, an image stitching method based on depth point and line features is provided, which includes acquiring images of the scene to be stitched and obtaining a feature extraction network, comprising:
[0011] Multiple frames of scene images with partial overlap are acquired using image acquisition equipment;
[0012] The feature extraction network was trained using an open-source dot and line feature dataset.
[0013] According to the present invention, an image stitching method based on depth point and line features is provided, wherein the scene image to be stitched is input into the feature extraction network to extract point features, line features, and point and line feature descriptions, including:
[0014] The feature extraction network is defined as comprising a point classification subnetwork, a line classification subnetwork, and a feature description subnetwork;
[0015] Multiple geometric images of different types and the coordinates of points and lines in the geometric images are obtained. The feature extraction network is trained using the geometric images and the coordinates of points and lines in the geometric images to obtain a feature extraction network with initial point and line shape recognition capability.
[0016] The feature extraction network with initial point and line shape recognition capability is adjusted using several real images to obtain the point features, the line features, and the point and line feature descriptions.
[0017] According to the present invention, an image stitching method based on depth point and line features is provided, which obtains the geometric coplanar features of the scene image to be stitched, performs feature matching using the geometric coplanar features and the point and line features, and obtains an image stitching model, including:
[0018] Normalize the descriptors output by the feature extraction network, and calculate the dot product of any two feature vectors:
[0019] s=desc1·desc2,match=argmax(s),s>ε
[0020] Where desc1 and desc2 are descriptors of any two features to be matched, s is the similarity between any two descriptors, and ε is the similarity threshold.
[0021] The scene images to be stitched are divided into multiple uniform grids. A plane is formed by several locally adjacent grids. If no exact matching point exists in the current grid, an exact matching point is obtained within the plane. The transformation matrix H of the plane is estimated using at least four pairs of points to obtain the local correspondence between any two scene images to be stitched.
[0022]
[0023] Among them, (u i ,v i ) and (u′ i ,v′ i ) represents the pixel coordinates of the matching point, h i(i = 1, 2, ..., 9) are the parameters of the transformation matrix H;
[0024] Self-matching is performed on the detected straight lines in a single image. The straight lines are clustered according to descriptor similarity. The clustered straight lines are fitted to form a plane according to the principle of spatial proximity. The feature values of the plane are calculated using the points and lines on the plane. The plane matching is performed in different images based on the feature values. The projection matrix of the pairing plane is estimated using feature points. The straight lines are projected. The two straight lines that simultaneously satisfy the distance threshold and the descriptor similarity threshold after projection are taken as the pairing lines.
[0025] According to the present invention, an image stitching method based on depth point and line features is provided, wherein the planar feature values include:
[0026]
[0027] P = a1P1 + b1P2
[0028] Where P1 and P2 are the endpoints of a straight line, and P is another point on the straight line. P is represented by P1 and P2, and the coefficients are a1 and b1. The plane characteristic value CN is calculated using three straight lines that form a closed loop.
[0029] Correspondingly, the method for calculating planar similarity is as follows:
[0030]
[0031] Among them, CN a (r) represents the CN value calculated from the r-th line in image a, where CN b (r) represents the CN value calculated from the r-th line in image b.
[0032] According to the present invention, an image stitching method based on depth point and line features is provided, which obtains the mesh deformation constraint conditions of the scene image to be stitched, performs image pixel fusion of the mesh deformation constraint conditions and the image stitching model, and outputs the stitched image result, including:
[0033] The grid vertices of the scene image to be stitched are used to represent the coordinates of internal points, and the distance between any two points is calculated to obtain the point feature E. PointAlign :
[0034]
[0035] Among them, s j The weights are calculated based on the point matching scores; groups with higher matching confidence have greater weights, P′. j Indicates with P j Paired point coordinates, V j W represents the vertex coordinates of the grid where the point is located. jThese are the corresponding weighting coefficients;
[0036] Sample several points at equal intervals along each line, calculate the distance from the projection point to the line, and obtain the line feature E. LineAlign :
[0037]
[0038] Where i represents a line in the image, s i It is the line matching score, A i B i C i These represent the equation coefficients of the lines paired with this line. j represents the sampling point on each line. and These are the x and y coordinates of the grid vertices;
[0039] The governing equations for line deformation include:
[0040]
[0041] Among them, mor x and mor y The normal vector that forms the line. and It is a bilinear representation of two adjacent points on the line;
[0042] The deformation scaling E is obtained by the point spacing on each line. Stretch :
[0043]
[0044] in, This represents three adjacent points on a line;
[0045] The stitched image result is obtained by using singular value decomposition (SVD).
[0046] Secondly, the present invention also provides an image stitching system based on depth point and line features, comprising:
[0047] The acquisition module is used to acquire the scene images to be stitched together and obtain the feature extraction network.
[0048] The extraction module is used to input the scene image to be stitched into the feature extraction network to extract point features, line features, and point-line feature descriptions;
[0049] The matching module is used to obtain the geometric coplanar features of the scene image to be stitched, and to perform feature matching using the geometric coplanar features and the point and line features to obtain the image stitching model;
[0050] The stitching module is used to obtain the mesh deformation constraints of the scene image to be stitched, perform image pixel fusion of the mesh deformation constraints and the image stitching model, and output the stitched image result.
[0051] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the image stitching method based on depth point and line features as described above.
[0052] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image stitching method based on depth point and line features as described above.
[0053] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the image stitching method based on depth point and line features as described above.
[0054] The image stitching method and system based on depth point and line features provided by this invention utilizes the depth point and line features of the images to be stitched, performs feature extraction and matching on images with a certain degree of overlap, and then projects these images onto a selected reference plane based on the feature registration relationship, so that the common parts of different images are accurately aligned, and has the characteristics of high alignment accuracy and high restoration. Attached Figure Description
[0055] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0056] Figure 1 This is a flowchart illustrating the image stitching method based on depth point and line features provided by the present invention.
[0057] Figure 2 This is an image stitching framework diagram provided by the present invention;
[0058] Figure 3 This is the point and line extraction network diagram provided by the present invention;
[0059] Figure 4 This is a schematic diagram of a simple geometric shape provided by the present invention;
[0060] Figure 5 These are schematic diagrams of real-world scenes provided by this invention;
[0061] Figure 6 This is a schematic diagram of feature alignment and deformation control provided by the present invention;
[0062] Figure 7 This is a schematic diagram of the image stitching system based on depth point and line features provided by the present invention;
[0063] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation
[0064] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0065] Figure 1 This is a flowchart illustrating the image stitching method based on depth point and line features provided in an embodiment of the present invention, as shown below. Figure 1 As shown, it includes:
[0066] Step 100: Acquire the scene images to be stitched together and obtain the feature extraction network;
[0067] Step 200: Input the scene image to be stitched into the feature extraction network to extract point features, line features, and point-line feature descriptions;
[0068] Step 300: Obtain the geometric coplanar features of the scene image to be stitched, and use the geometric coplanar features and the point and line features to perform feature matching to obtain the image stitching model;
[0069] Step 400: Obtain the mesh deformation constraint conditions of the scene image to be stitched, perform image pixel fusion of the mesh deformation constraint conditions and the image stitching model, and output the stitched image result.
[0070] Specifically, this embodiment of the invention inputs several images with a certain degree of overlap, extracts and matches features from each, and then projects these images onto a selected reference plane based on feature registration relationships (the projection relationship is usually one or more homography transformation matrices), so that the common parts of the different images are accurately aligned. During this process, it is also necessary to preserve as much structural information as possible in non-overlapping areas to control image deformation, resulting in a good overall visual appearance of the stitched image. Therefore, the key to the stitching algorithm is to extract and match image features, then establish a projection model based on the feature matching relationships, and solve for the transformation parameters that enable accurate image alignment and minimize deformation.
[0071] like Figure 2 As shown, the image stitching process based on depth point and line features proposed in this embodiment of the invention includes:
[0072] Acquisition of multiple frames of images to be stitched together, and preparation of training data for the feature extraction network;
[0073] Deep neural networks are used to detect and describe point and line features in images.
[0074] High-precision feature matching is achieved using geometrically coplanar features and feature descriptors output by the network.
[0075] An image alignment model is established based on feature matching relationships, and image deformation is controlled by other constraints, such as mesh deformation constraints.
[0076] This invention utilizes the depth point and line features of the images to be stitched together, extracts and matches features of images with a certain degree of overlap, and then projects these images onto a selected reference plane based on the feature registration relationship, so that the common parts of different images are accurately aligned, which has the characteristics of high alignment accuracy and high restoration.
[0077] Based on the above embodiments, the process involves acquiring images of the scene to be stitched together and obtaining a feature extraction network, including:
[0078] Multiple frames of scene images with partial overlap are acquired using image acquisition equipment;
[0079] The feature extraction network was trained using an open-source dot and line feature dataset.
[0080] Specifically, in this embodiment of the invention, a mobile measuring vehicle or other image acquisition device is used to acquire multiple frames of scene images with a certain degree of overlap to be stitched together, and there needs to be a certain degree of distinction between objects in the stitched scene; a feature extraction network model is trained using an existing open-source dataset.
[0081] Based on the above embodiments, the scene image to be stitched is input into the feature extraction network to extract point features, line features, and point-line feature descriptions, including:
[0082] The feature extraction network is defined as comprising a point classification subnetwork, a line classification subnetwork, and a feature description subnetwork;
[0083] Multiple geometric images of different types and the coordinates of points and lines in the geometric images are obtained. The feature extraction network is trained using the geometric images and the coordinates of points and lines in the geometric images to obtain a feature extraction network with initial point and line shape recognition capability.
[0084] The feature extraction network with initial point and line shape recognition capability is adjusted using several real images to obtain the point features, the line features, and the point and line feature descriptions.
[0085] Specifically, the first step is network model design. Feature points and lines are pixels in an image whose gray levels change according to a certain pattern. Detecting feature points and lines involves extracting various gray levels, gradients, and other information around these pixels, and then determining whether the pixel is a feature. This can essentially be considered a pixel classification problem. Therefore, this embodiment of the invention divides the network into a point / line learning module and a pixel classification module, such as... Figure 3 As shown, the network on the left learns semantic features from the image, while the three branches on the right perform point classification, line classification, and feature description respectively based on the learned knowledge.
[0086] Then comes network training, which is divided into two steps. First, the program draws a large number of geometric images such as polygons, cubes, checkerboard patterns, and stars. Figure 4 As shown, the coordinates of points and lines in these images are recorded simultaneously. This data is used to train the network initially, enabling it to recognize simple point and line shapes.
[0087] Furthermore, the network model trained in the previous step is adjusted using real-world images, such as... Figure 5 As shown in the diagram, the model obtained in the previous step has a preliminary ability to extract points and lines. However, because the distribution of the data features it uses differs from that of the real world, the model has difficulty adapting to the domain when detecting real images and is prone to missing some potential features. Therefore, it needs to be retrained to optimize the weights. Since the network itself can already recognize some points and lines, we can first perform point and line detection multiple times on the training dataset and then use these points and lines as labels, eliminating the need for manual data annotation.
[0088] Based on the above embodiments, the point and line features obtained by traditional stitching algorithms often suffer from mismatch problems, mainly in areas of repeated textures where the similarity between feature descriptors is very high, making these features difficult to distinguish. Therefore, this invention proposes a feature matching method based on geometric coplanarity conditions, which can effectively match repeated texture features.
[0089] First, descriptor similarity point matching is performed. The cosine distance between feature vectors in the feature space can be used to measure their similarity. During matching, the descriptors output by the network are first normalized, and then the dot product of the feature vectors is calculated pairwise, using the following formula:
[0090] s=desc1·desc2,match=argmax(s),s>ε
[0091] Where desc1 and desc2 are descriptors of any two features to be matched, s is the similarity between any two descriptors, and ε is the similarity threshold.
[0092] Secondly, local homography constraint point matching is performed. The image is divided into a uniform grid, with each grid containing only a small amount of content. Several locally adjacent grids can be roughly approximated as a plane. All grids are traversed; if no exact matching point or other points exist in the current grid, an exact matching point is searched within this plane. Then, the transformation matrix H of this plane is estimated using at least four pairs of points, yielding the correspondence between the two images in that local area. The formula is:
[0093]
[0094] Among them, (u i ,v i ) and (u′ i ,v′ i ) represents the pixel coordinates of the matching point, h i (i = 1, 2, ..., 9) are the parameters of the transformation matrix H;
[0095] Next is coplanar feature line matching. Self-matching is performed on the detected lines in an image, clustering lines with high descriptor similarity into one class. Then, based on the spatial proximity principle, these lines are fitted into a plane, and it is determined which points lie on this plane. Then, a feature value is calculated using the points and lines on the plane, matching the planes with high similarity between the two images. Finally, the projection matrix of the paired plane is estimated using feature points, and the lines are projected. When the two projected lines simultaneously satisfy both the distance threshold and the descriptor similarity threshold, they are used as paired lines. The method for calculating the plane feature value CN is as follows:
[0096]
[0097] P = a1P1 + b1P2
[0098] Where P1 and P2 are the endpoints of a straight line, and P is another point on the straight line. P is denoted by P1 and P2, and the coefficients are a1 and b1. The plane characteristic value CN is calculated using three straight lines that form a closed loop.
[0099] The method for calculating planar similarity is as follows:
[0100]
[0101] Among them, CN a (r) represents the CN value calculated from the r-th line in image a, where CN b (r) represents the CN value calculated from the r-th line in image b.
[0102] Based on the above embodiments, the mesh deformation constraints of the scene image to be stitched are obtained, and the mesh deformation constraints and the image stitching model are fused pixel by pixel to output the stitched image result, such as... Figure 6As shown, it includes:
[0103] First, establish the point-line alignment equation, using the grid vertex coordinates to represent the coordinates of points within it. After transformation, the closer the distance between two points, the more accurate the alignment. The equation is:
[0104]
[0105] Among them, s j The weights are calculated based on the point matching scores; groups with higher matching confidence have greater weights, P′. j Indicates with P j Paired point coordinates, V j W represents the vertex coordinates of the grid where the point is located. j These are the corresponding weighting coefficients.
[0106] For line features, several points are sampled at equal intervals along each line. Each transformed point should fall on the corresponding line. Therefore, the energy function optimizes the distance from the projected point to the line, as shown in the formula:
[0107]
[0108] Where i represents a line in the image, s i It is the line matching score, A i B i C i These represent the equation coefficients of the lines paired with this line. j represents the sampling point on each line. and These are the x and y coordinates of the grid vertices.
[0109] The governing equations for line deformation include:
[0110] To avoid excessive image distortion, the transition between meshes after transformation should be as smooth and natural as possible; that is, the slope change between adjacent segments of a line feature should not be too large. The equation is:
[0111]
[0112] Among them, nor x and nor y The normal vector that forms the line. and It is a bilinear representation of two adjacent points on the line;
[0113] Scaling distortion refers to an abnormal change in the size of certain areas in an image, mainly manifested as stretching or compression in the horizontal direction. Scaling can be controlled by maintaining the spacing between points along each line, using the following formula:
[0114]
[0115] in, This represents three adjacent points on a line;
[0116] Since the above equations about V are all linear, the stitched image result can be obtained by using Singular Value Decomposition (SVD).
[0117] The image stitching system based on depth point and line features provided by the present invention is described below. The image stitching system based on depth point and line features described below can be referred to in correspondence with the image stitching method based on depth point and line features described above.
[0118] Figure 7 This is a schematic diagram of the image stitching system based on depth point and line features provided in an embodiment of the present invention, as shown below. Figure 7 As shown, it includes: a data acquisition module 71, an extraction module 72, a matching module 73, and a splicing module 74, wherein:
[0119] The acquisition module 71 is used to acquire the scene image to be stitched and obtain a feature extraction network; the extraction module 72 is used to input the scene image to be stitched into the feature extraction network and extract point features, line features, and point-line feature descriptions; the matching module 73 is used to obtain the geometric coplanar features of the scene image to be stitched, and perform feature matching using the geometric coplanar features and the point-line feature descriptions to obtain an image stitching model; the stitching module 74 is used to obtain the mesh deformation constraint conditions of the scene image to be stitched, perform image pixel fusion of the mesh deformation constraint conditions and the image stitching model, and output the stitched image result.
[0120] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8As shown, the electronic device may include a processor 810, a communication interface 820, a memory 830, and a communication bus 840. The processor 810, communication interface 820, and memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute an image stitching method based on depth point and line features. This method includes: acquiring an image of the scene to be stitched; obtaining a feature extraction network; inputting the image of the scene to be stitched into the feature extraction network to extract point features, line features, and point-line feature descriptions; obtaining the geometric coplanar features of the image of the scene to be stitched; performing feature matching using the geometric coplanar features and the point-line feature descriptions to obtain an image stitching model; obtaining the mesh deformation constraints of the image of the scene to be stitched; performing image pixel fusion of the mesh deformation constraints and the image stitching model; and outputting the stitched image result.
[0121] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0122] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image stitching method based on depth point and line features provided by the above methods. The method includes: acquiring a scene image to be stitched and obtaining a feature extraction network; inputting the scene image to be stitched into the feature extraction network to extract point features, line features, and point and line feature descriptions; obtaining the geometric coplanar features of the scene image to be stitched, performing feature matching using the geometric coplanar features and the point and line feature descriptions to obtain an image stitching model; obtaining the mesh deformation constraint conditions of the scene image to be stitched, performing image pixel fusion of the mesh deformation constraint conditions and the image stitching model, and outputting the stitched image result.
[0123] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the image stitching method based on depth point and line features provided by the above methods. The method includes: acquiring a scene image to be stitched and obtaining a feature extraction network; inputting the scene image to be stitched into the feature extraction network to extract point features, line features, and point and line feature descriptions; obtaining the geometric coplanar features of the scene image to be stitched, performing feature matching using the geometric coplanar features and the point and line feature descriptions to obtain an image stitching model; obtaining the mesh deformation constraint conditions of the scene image to be stitched, performing image pixel fusion of the mesh deformation constraint conditions and the image stitching model, and outputting the stitched image result.
[0124] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0125] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. An image stitching method based on depth point and line features, characterized in that, include: Acquire scene images to be stitched together and obtain a feature extraction network; The scene image to be stitched is input into the feature extraction network to extract point features, line features, and point-line feature descriptions; Obtain the geometric coplanar features of the scene image to be stitched, and use the geometric coplanar features and the point and line features to perform feature matching to obtain the image stitching model; Obtain the mesh deformation constraints of the scene image to be stitched, fuse the mesh deformation constraints and the image stitching model into image pixels, and output the stitched image result, including: The internal point coordinates are represented by the grid vertices of the scene image to be stitched, and the point features are obtained by calculating the distance between any two points. : in, The weights are calculated based on the point matching scores; groups with higher matching confidence have greater weights. Indicates and The coordinates of the paired points, This indicates the vertex coordinates of the grid where the point is located. These are the corresponding weighting coefficients; Sample several points at equal intervals along each line, calculate the distance from the projected points to the line, and obtain the line features. : Where i represents a line in the image, It is the line matching score. These represent the equation coefficients of the lines paired with this line. , This indicates the sampling points on each line. and These are the x and y coordinates of the grid vertices; The governing equations for line deformation include: in, and The normal vector that forms the line. and It is a bilinear representation of two adjacent points on the line; Deformation and scaling are achieved by adjusting the spacing between points on each line. : in, This represents three adjacent points on a line; The stitched image result is obtained by using singular value decomposition (SVD).
2. The image stitching method based on depth point and line features according to claim 1, characterized in that, Acquire images of the scene to be stitched together, and obtain a feature extraction network, including: Multiple frames of scene images with partial overlap are acquired using image acquisition equipment; The feature extraction network was trained using an open-source dot and line feature dataset.
3. The image stitching method based on depth point and line features according to claim 1, characterized in that, The scene image to be stitched is input into the feature extraction network to extract point features, line features, and point-line feature descriptions, including: The feature extraction network is defined as comprising a point classification subnetwork, a line classification subnetwork, and a feature description subnetwork; Multiple geometric images of different types and the coordinates of points and lines in the geometric images are obtained. The feature extraction network is trained using the geometric images and the coordinates of points and lines in the geometric images to obtain a feature extraction network with initial point and line shape recognition capability. The feature extraction network with initial point and line shape recognition capability is adjusted using several real images to obtain the point features, the line features, and the point and line feature descriptions.
4. The image stitching method based on depth point and line features according to claim 1, characterized in that, Obtain the geometric coplanar features of the scene image to be stitched, and perform feature matching using the geometric coplanar features and the point and line features to obtain an image stitching model, including: Normalize the descriptors output by the feature extraction network, and calculate the dot product of any two feature vectors: in, and Let be any two descriptors of features to be matched, and s be the similarity between any two descriptors. It is a similarity threshold; The scene image to be stitched is divided into multiple uniform grids. Several locally adjacent grids in the image are considered as a plane. If no exact matching point or other points exist in the current grid, an exact matching point is obtained within the plane. The transformation matrix of the plane is estimated using no fewer than four pairs of points. This yields the local correspondence between any two scene images to be stitched together: in, These are the pixel coordinates of the matching points. It is a transformation matrix The parameters, ; Self-matching is performed on the detected straight lines in a single image. The straight lines are clustered according to descriptor similarity. The clustered straight lines are fitted to form a plane according to the principle of spatial proximity. The feature values of the plane are calculated using the points and lines on the plane. The plane matching is performed in different images based on the feature values. The projection matrix of the pairing plane is estimated using feature points. The straight lines are projected. The two straight lines that simultaneously satisfy the distance threshold and the descriptor similarity threshold after projection are taken as the pairing lines.
5. The image stitching method based on depth point and line features according to claim 4, characterized in that, The planar feature values include: in, It is the endpoint of a straight line. It is another point located on that line, use It is indicated that the coefficient is The eigenvalues of a plane are calculated using three straight lines that form a closed loop. ; Correspondingly, the method for calculating planar similarity is as follows: in, Representing an image The Middle The CN value calculated from a straight line. Representing an image The Middle The CN value is calculated from a straight line.
6. An image stitching system based on depth point and line features, based on the image stitching method based on depth point and line features according to any one of claims 1 to 5, characterized in that, include: The acquisition module is used to acquire the scene images to be stitched together and obtain the feature extraction network. The extraction module is used to input the scene image to be stitched into the feature extraction network to extract point features, line features, and point-line feature descriptions; The matching module is used to obtain the geometric coplanar features of the scene image to be stitched, and to perform feature matching using the geometric coplanar features and the point and line features to obtain the image stitching model; The stitching module is used to obtain the mesh deformation constraints of the scene image to be stitched, perform image pixel fusion of the mesh deformation constraints and the image stitching model, and output the stitched image result.
7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the image stitching method based on depth point and line features as described in any one of claims 1 to 5.
8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the image stitching method based on depth point and line features as described in any one of claims 1 to 5.
9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the image stitching method based on depth point and line features as described in any one of claims 1 to 5.