Two-dimensional fragment splicing method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of image processing, and discloses a two-dimensional fragment splicing method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining a first fragment image and a second fragment image; detecting the first fragment image and the second fragment image, and respectively generating corner point sequence sets with ordered fragment outlines; on the basis of the angular point sequence set, taking angular points as vertexes, and constructing a triangle by every three adjacent angular points; determining a side length proportion set of each triangle, and generating a first triangle sequence set corresponding to the first fragment image and a second triangle sequence set corresponding to the second fragment image; and splicing the first fragment image and the second fragment image based on the first triangular sequence set and the second triangular sequence set. According to the method, the influence of a single angular point error on overall matching can be reduced, the problems of matching deviation and the like caused by weak textures, fragment overturning, inconsistent fragment proportions and the like can be solved, and an efficient, accurate and stable splicing effect is achieved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a two-dimensional fragment splicing method, device, computer equipment and storage medium. Background Art

[0002] The restoration of two-dimensional fragments plays an important role in image restoration, medical image stitching, archaeological artifact restoration, etc. In the past, the restoration and stitching of two-dimensional fragments were mostly done manually, which was very inefficient. However, with the development of computer technology, especially the rapid development of graphics, the use of computers to stitch two-dimensional fragments has begun to be widely used.

[0003] At present, the splicing of two-dimensional irregular fragments using computer technology is mainly based on contour splicing, that is, according to the matching contour curve segments found, they are automatically or interactively spliced ​​together. The splicing process mainly includes the following steps: (1) Calculate the features of the fragment contour points; (2) Match the contour point feature sets between the fragments to obtain the correspondence between the contour points; (3) Calculate the rigid body transformation matrix based on the correspondence between the contour points; (4) Transform the fragments according to the obtained transformation matrix and then directly splice them. Although this algorithm can solve the splicing and matching problem of irregular fragments to a certain extent, for some fragments, such as ceramics, paper, bronze artifacts and other materials, there are some problems that restrict the application of this method, such as: (1) There are often defects of varying degrees at the fracture; (2) The surface texture of the fragment material is weak, and the color features are not obvious or even have no color; (3) Due to lens distortion, changes in the position of the fragments, changes in the distance between the fragments and the lens, the resolution of the image is different after imaging; (4) The fragments may be flipped.

[0004] Specifically, defects will cause large differences in the corner points extracted from different images; weak texture will lead to the inability to match based on the pixel information of adjacent fragments; two fragments cannot be spliced ​​if they are flipped; changes in fragment position and camera distance will cause large deviations in the distance calculations of fragment outlines and fragment corners on different images, resulting in matching deviations or even failure to match. Summary of the invention

[0005] In view of this, the present invention provides a two-dimensional fragment splicing method, device, computer equipment and storage medium to solve the problem of unstable splicing effect of the contour-based fragment splicing method in the prior art.

[0006] In a first aspect, the present invention provides a two-dimensional fragment splicing method, the method comprising:

[0007] Acquire a first fragment image and a second fragment image, wherein the first fragment image and the second fragment image are both binary images;

[0008] Detecting the first fragment image and the second fragment image, respectively generating ordered corner point sequence sets corresponding to the fragment contours, wherein the corner point sequence sets include one-to-one corresponding corner point coordinates of each corner point;

[0009] Based on the set of corner point sequences, a triangle is constructed with corner points as vertices and every three adjacent corner points;

[0010] Determine a side length ratio set of each triangle, and generate a first triangle sequence set corresponding to the first fragment image and a second triangle sequence set corresponding to the second fragment image; wherein the side length ratio is a ratio between adjacent side lengths in a triangle;

[0011] The first fragment image and the second fragment image are spliced ​​based on the first triangle sequence set and the second triangle sequence set.

[0012] The present invention compares the corner points in the detected edge contour of the fragments with the contour points to obtain an ordered sequence of corner points, the corner points are accurately positioned, and the sorting results are stable and reliable. Furthermore, by constructing triangles according to the corner point sequence and then matching through the corner point triangles, it can not only reduce the impact of single corner point errors on the overall matching and improve the stability and accuracy of the matching, but also deal with problems such as matching deviations caused by weak textures, flipping of fragments, and inconsistent fragment proportions, thereby achieving an efficient, accurate, and stable splicing effect, and having significant technical innovation advantages. Moreover, the present invention provides a two-dimensional fragment splicing algorithm based on triangle corner points, which can be applied to the splicing and restoration of two-dimensional fragments of various materials, especially in the fields of ceramics, paper pieces, bronze artifacts, etc. It is of great significance.

[0013] In an optional implementation, stitching the first fragment image and the second fragment image based on the first triangle sequence set and the second triangle sequence set includes:

[0014] Determine a triangle ratio difference set, wherein the triangle ratio difference is a difference in side length ratio between a side length ratio set in the first triangle sequence set and a side length ratio set in the second triangle sequence set;

[0015] Based on the triangle ratio difference set, determine the first triangle and the second triangle corresponding to the minimum triangle ratio difference;

[0016] Determine whether the first triangle and the second triangle meet a preset pixel condition;

[0017] When a preset pixel condition is met, a first corner point corresponding to the first triangle and a second corner point corresponding to the second triangle are determined; the first corner point and the second corner point are used to splice the first fragment image with the second fragment image.

[0018] In this implementation, the side length ratio of the triangles in the two fragments is used as a condition to compare the difference in the side length ratio of all triangles between the two fragments. The triangle side length ratio difference is calculated accurately, and the threshold is set reasonably, which can effectively screen out triangle pairs that preliminarily meet the requirements. In addition, the pixel ratio calculation can accurately reflect the matching degree of the convex hull and concave hull shapes. The comprehensive geometric relationship judgment and pixel matching degree are judged to further accurately screen out corner points with high matching degree.

[0019] In an optional implementation, determining whether the first triangle and the second triangle meet a preset pixel condition includes:

[0020] Determine a first ratio of the number of pixels of a first preset color in the first triangle to the total number of pixels in the first triangle;

[0021] Determine a second ratio of the number of pixels of a second preset color in the second triangle to the total number of pixels in the second triangle;

[0022] Wherein, when the first preset color is black, the second preset color is white; when the first preset color is white, the second preset color is black;

[0023] determining a degree of match between the first ratio and the second ratio;

[0024] When the matching degree satisfies the preset matching degree, it is determined that the first triangle and the second triangle satisfy the preset pixel condition.

[0025] In this implementation, the corresponding pixel proportions in the enclosed triangles are comprehensively considered to jointly measure the matching degree of the corner points between the two fragments, so that the corner points with high matching degree can be accurately screened out.

[0026] In an optional implementation, stitching the first fragment image and the second fragment image based on the first triangle sequence set and the second triangle sequence set includes:

[0027] Determine a triangle ratio difference set, wherein the triangle ratio difference is a difference in side length ratio between a side length ratio set in the first triangle sequence set and a side length ratio set in the second triangle sequence set;

[0028] Determine, based on the triangle ratio difference set, a triangle pair that meets a preset ratio difference threshold, wherein the triangle pair includes a first triangle corresponding to the first fragment image and a second triangle corresponding to the second fragment image;

[0029] Determine whether the first triangle and the second triangle in the triangle pair meet a preset pixel condition;

[0030] When a preset pixel condition is met, determining a first corner point corresponding to the first triangle and a second corner point corresponding to the second triangle;

[0031] Regenerate all first corner points into a new first corner point sequence, and regenerate all second corner points into a new second corner point sequence;

[0032] Based on the first corner point sequence, a triangle is constructed for every three adjacent corner points to generate a new first triangle set;

[0033] Based on the second corner point sequence, a triangle is constructed for every three adjacent corner points to generate a new second triangle set;

[0034] Based on the first triangle set and the second triangle set, determining the number of matches between the first corner point and the second corner point;

[0035] Determine whether the number of matches of the first corner point or the second corner point meets the preset number of matches;

[0036] When the preset matching times are met, the corresponding first corner point or the second corner point is determined as a matching point, and the matching point is used to splice the first fragment image and the second fragment image.

[0037] In this embodiment, based on the paired triangle corner points between two fragments, the number of times each corner point is paired is counted, and the corner point pairs that appear more than or equal to the preset number of matches are retained. The retained corner point pairs will be used for fragment splicing. This method can improve the accuracy of fragment splicing.

[0038] In an optional implementation, after the matching point is determined, the method further includes:

[0039] Select three first corner points and three second triangular points corresponding to the first corner points from the matching points;

[0040] Determine a first angle and a first direction between three first corner points and a second angle and a second direction between three second triangular points;

[0041] Calculate an angle difference based on the first angle and the second angle;

[0042] When the angle difference satisfies a preset range, it is determined whether the first fragment image or the second fragment image needs to be flipped based on the first direction and the second direction.

[0043] In this implementation, by calculating the vector angle and judging the direction, it is possible to accurately judge whether the two fragments are in the same direction, with a high judgment accuracy, and can effectively avoid splicing errors caused by the reverse direction of the fragments.

[0044] In an optional implementation, after the matching point is determined, the method further includes:

[0045] Select three first corner points and three second triangulation points corresponding to the first corner points from the matching points; wherein the maximum index difference of the selected three first corner points needs to meet the preset index value;

[0046] Determine a first distance and a first edge vector formed between three first corner points; determine a second distance and a second edge vector formed between three second triangle points;

[0047] Based on the first side vector and the second side vector, determine the angle between each vector and the horizontal line;

[0048] Based on the first distance and the second distance, determining a distance ratio between the first distance and the second distance;

[0049] Determine whether the distance ratio meets the preset conditions;

[0050] When the preset conditions are met, the average rotation angle and the average scaling ratio of the first fragment image relative to the second fragment image or the second fragment image relative to the first fragment image are calculated based on the included angle and the distance ratio.

[0051] In this implementation, the calculated errors of the determined rotation angle and scaling ratio are small, and the rotation and scaling processing of fragment two is accurate, which can effectively ensure that the processed fragment two is highly consistent with fragment one in angle and ratio, achieving seamless splicing.

[0052] In an optional implementation, after determining the average rotation angle and the average scaling ratio, the method further includes:

[0053] Determine an image to be transformed, where the image to be transformed is the first fragment image or the second fragment image;

[0054] Increase the pixel area of ​​the image to be transformed;

[0055] Determine the center point of the image to be transformed and use the center point as the rotation center;

[0056] Based on the average rotation angle, calculate the rotation matrix;

[0057] Transform the image to be transformed based on the rotation matrix and the average scaling ratio;

[0058] Calculate the offset of the matching corner points based on the rotation matrix and average scaling;

[0059] Update the coordinates of the corner points in the corner point sequence set based on the matching corner point offsets.

[0060] In this implementation, the offset is further adjusted by rotating the image after the matrix and scaling transformation, so that the transition of the stitching line composed of the corner points can be smoother, effectively improving the stitching accuracy.

[0061] In a second aspect, the present invention provides a two-dimensional fragment splicing device, the device comprising:

[0062] An acquisition module, used for acquiring a first fragment image and a second fragment image, wherein the first fragment image and the second fragment image are both binary images;

[0063] A detection module, used for detecting the first fragment image and the second fragment image, and generating an ordered set of corner point sequences corresponding to the fragment contours, respectively, wherein the set of corner point sequences includes one-to-one corresponding corner point coordinates of each corner point;

[0064] A construction module, for constructing a triangle based on a set of corner point sequences, with the corner points as vertices and every three adjacent corner points;

[0065] A determination module, used to determine a side length ratio set of each triangle, and generate a first triangle sequence set corresponding to the first fragment image and a second triangle sequence set corresponding to the second fragment image; wherein the side length ratio is a ratio between adjacent side lengths in a triangle;

[0066] The splicing module is used for splicing the first fragment image and the second fragment image based on the first triangle sequence set and the second triangle sequence set.

[0067] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the two-dimensional fragment splicing method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0068] In a fourth aspect, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to execute the two-dimensional fragment splicing method of the first aspect or any corresponding embodiment thereof.

[0069] It should be noted that the two-dimensional fragment splicing device, computer device and computer-readable storage medium provided by the present invention correspond to the above-mentioned two-dimensional fragment splicing method. Therefore, for the beneficial effects of the two-dimensional fragment splicing device, computer device and computer-readable storage medium, please refer to the description of the corresponding beneficial effects of the two-dimensional fragment splicing method above, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0071] Figure 1 is a schematic flow chart of a two-dimensional fragment splicing method according to an embodiment of the present invention;

[0072] Figure 2 is a schematic diagram of a grayscale image and a binary image of two fragments according to an embodiment of the present invention;

[0073] Figure 3 is a schematic diagram of two fragment corner point images detected according to an embodiment of the present invention;

[0074] Figure 4 is a schematic diagram of the outlines of two fragments detected according to an embodiment of the present invention;

[0075] Figure 5 is a schematic diagram of ordered corner points according to an embodiment of the present invention;

[0076] Figure 6 is a schematic diagram of partial triangle construction according to an embodiment of the present invention;

[0077] Figure 7 is a schematic diagram of randomly selected triangle matching according to an embodiment of the present invention;

[0078] Figure 8 is a schematic diagram of the angle between the corner points of two fragment-matched triangles and a horizontal line and a scaling ratio according to an embodiment of the present invention;

[0079] Fig. 9 is a schematic diagram of an image obtained after the fragment 2 is rotated and scaled according to an embodiment of the present invention;

[0080] Fig.10 is a schematic diagram of two fragment images after matching corner points are aligned according to an embodiment of the present invention;

[0081] Fig.11 is a schematic diagram of a non-rotation and non-scaling fragment splicing image according to an embodiment of the present invention;

[0082] Fig.12 is a schematic diagram of a rotated fragment splicing image according to an embodiment of the present invention;

[0083] Fig.13 is a schematic diagram of a zoomed fragment splicing image according to an embodiment of the present invention;

[0084] Fig.14 is a schematic diagram of a rotated and scaled fragment splicing image according to an embodiment of the present invention;

[0085] Fig.15 is a schematic diagram of an image with rotation, scaling, and splicing of fragments with partial area defects according to an embodiment of the present invention;

[0086] Fig.16 is a schematic diagram of a fragmented image splicing process that is rotated, scaled, partially missing, and flipped according to an embodiment of the present invention;

[0087] Fig.17 is a structural block diagram of a two-dimensional fragment splicing device according to an embodiment of the present invention;

[0088] Fig.18 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0089] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0090] At present, the demand for high-precision and high-efficiency splicing technology continues to grow in the fields of cultural heritage restoration, image recognition, industrial production quality inspection, etc. However, the existing splicing technology has problems such as insufficient precision and limited scope of application. The present invention uses triangle corner point recognition to achieve efficient, accurate and stable splicing effects, which has significant technical innovation advantages.

[0091] In terms of development prospects, this technology is expected to be widely used in scenarios such as image restoration, image synthesis, intelligent manufacturing, and cultural relics restoration. Its promotion and application will greatly improve the automation and intelligence level of various industries and promote technological change. After industrialization, it is expected to effectively improve production efficiency, reduce costs, and form significant economic benefits. In addition, the technology of the present invention is also of great significance in terms of social values ​​such as cultural relics protection and cultural heritage, which helps to improve the technical level of the industry and promote the integrated innovation and development of culture and technology.

[0092] Specifically, according to an embodiment of the present invention, a two-dimensional fragment splicing method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0093] In this embodiment, a two-dimensional fragment splicing method is provided, which can be executed by a server, a terminal, a mobile terminal and other devices. Figure 1 is a flow chart of a two-dimensional fragment splicing method according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0094] Step S101 : acquiring a first fragment image and a second fragment image, wherein the first fragment image and the second fragment image are both binary images.

[0095] In this embodiment, it is necessary to select a suitable color background so that the color of the fragment has a high contrast with the background color, and use a camera to take pictures of each fragment in turn. Furthermore, in this embodiment, morphological operations including opening and closing operations, grayscale, and binarization are used to segment the fragment area from the background area, and the grayscale image and binary image of the fragment area are retained. In this embodiment, taking the splicing of two fragments as an example, the first fragment image and the second fragment image are both images after segmentation processing. Figure 2 As shown, a grayscale image and a binary image of a segmented first fragment image and a second fragment image are provided in this embodiment.

[0096] Step S102 , detecting the first fragment image and the second fragment image, and generating ordered corner point sequence sets corresponding to the fragment contours respectively, wherein the corner point sequence sets include one-to-one corresponding corner point coordinates of each corner point.

[0097] Specifically, this embodiment uses the cv2.goodFeaturesToTrack() function in the opencv library to detect the corner points of the edge contour of the fragment in the grayscale image. Figure 3 As shown in the figure (if it is necessary to explain, the fragment area in the figure is actually a binary black and white image, and other colors are used here to facilitate the labeling of the image). At this time, the corner points obtained are disordered. At the same time, the cv2.findContours() function is used to obtain the set of edge contour points of the fragment in the binary image. The detected contour image is referenced Figure 4 As shown, the edge contour point set is ordered. In this embodiment, the unordered corner points are sorted according to the ordered contour point set, that is, the ordered contour point coordinates are compared with the unordered corner point coordinates to obtain an ordered corner point sequence set. Figure 5 As shown in Figure 2, an ordered sequence of corner points can better match the corner points of two contours.

[0098] The method for sorting unordered corner points based on the edge contour point set is:

[0099] Assume that the unordered corner points are as follows: A1(x1,y1),A2(x2,y2)…A i(x i ,y i )…A m (x m ,y m ); the contour point set is as follows: B1(z1,w1),B2(z2,w2)…B i (z i ,w i )…B n (z n ,w n ), the contour point index is 1, 2, 3…n. The detailed positioning sorting method is as follows:

[0100] 1) The formula for calculating the distance between each corner point and all contour points is: D i =(x i -z i ) 2 +(y i -w i ) 2 , for each corner point, we get m distance sequences:

[0101] (D 11 D 12 LD 1n );(D 21 ,D 22 LD 2n );L;(D i1 ,D i2 LD in ); L(D m1 ,D m2 LD mn );

[0102] 2) For each corner point, select the distance with the shortest distance, assuming it is D1p and D1q, their corresponding indexes are p and q, and the index position of this corner point is: I = 0.5*(p+q), and finally the average index position sequence of all corner points is I1, I2, ... I i …I m ;

[0103] 3) Then sort by average index size from small to large to get J1, J2, ... J i ,…J m , this index sequence is the corner points sorted according to the ordered contour points. At this time, the corner points are ordered according to the fragment edge contour point set: C1(x1,y1),C2(x2,y2)…C i (x i ,y i )…C m (x m ,ym ), which is the set of corner point sequences.

[0104] In this embodiment, the cv2.goodFeaturesToTrack() function is used to detect the edge contour corner points of the grayscale image, and the cv2.findContours() function is used to detect the contour points of the binary image. The unordered corner points are sorted with the help of the ordered contour points to obtain an ordered corner point sequence, laying a solid foundation for subsequent precise matching.

[0105] Step S103, based on the corner point sequence set, with the corner points as vertices, a triangle is constructed with every three adjacent corner points.

[0106] After obtaining an ordered set of corner point sequences, every three adjacent corner points are combined into a triangle according to the coordinates of the corner point sequences in the fragments. Figure 6 shown.

[0107] In this embodiment, matching is performed based on corner point triangles. Triangles are used as a combination of the fewest corner points to form a polygon, which can reduce the impact of a single corner point error on the overall matching and improve the stability and accuracy of the matching.

[0108] Step S104, determining a side length ratio set of each triangle, generating a first triangle sequence set corresponding to the first fragment image and a second triangle sequence set corresponding to the second fragment image; wherein the side length ratio is the ratio between adjacent side lengths in a triangle.

[0109] After constructing the triangle, calculate the lengths of the three sides of the triangle and the ratio of the side lengths.

[0110] To join two fragments, we first need to find the joining point between the two fragments. In this embodiment, the triangle formed by the corner points of the two fragments is used for matching. First, a triangle requires the least corner points to form a polygon; second, if a corner point is wrong, the impact will not be too great.

[0111] First, match the triangles formed by two fragments according to the method of forming a triangle with three adjacent corner points. The matching method is as follows:

[0112] Assume that the two fragment corner points are: G1, G2...G i ...G m ; H1,H2,...H j ...H m ; Since the fragment outline is closed, the corner points should also be closed at the beginning and end. The closed corner point sequences are:

[0113] G 1, G2…G i …G m,G1,G2;H1,H2…H i …H n ,H1,H2; the triangle sequence composed of every three adjacent corner points is:

[0114] (G1,G2,G3),(G2,G3,G4),(G3,G4,G5)…(G i-1 ,G i ,G i+1 )…(G m ,G1,G2)

[0115] (H1,H2,H3),(H2,H3,H4),(H3,H4,H5)…(H j-1 ,H j ,H j+1 )…(H n ,H1,H2);

[0116] Since the present invention aims to solve the problem that when shooting fragments, two fragments may present a certain proportional relationship, resulting in different sizes of fragments, so the transformation relationship of triangle corner points cannot be directly used for calculation. It is necessary to use the similarity of triangles to determine whether the corner points of two fragments match.

[0117] Therefore, in this embodiment, the side length between each triangle corner point needs to be calculated, and the side length of each triangle is sorted from small to large, and the obtained side length sequence of each triangle is:

[0118] (a1,a2,a3),(a2,a3,a4),(a3,a4,a5)…(a i-1 ,a i ,a i+1 )…(a m ,a1,a2)

[0119] (b1,b2,b3),(b2,b3,b4),(b3,b4,b5)…(b j-1 ,b j ,b j+1 )…(b n ,b1,b2);

[0120] Further, calculate the ratio of the other two side lengths to the minimum side length. The calculation formula is as follows:

[0121] r (i,i-1) =a i / a i-1

[0122] r (i+1,i) =a i+1 / a i

[0123] t (j,j-1) =b j / b j-1

[0124] t (j+1,j) =b j+1 / b j ;

[0125] The formed side length ratio sequence, that is, the side length ratio set is:

[0126] (r (i,i-1) ,r (i+1,i) )

[0127] (t (j,j-1) ,t (j+1,j) ).

[0128] That is, the first triangle sequence set corresponding to the first fragment image in this embodiment includes: a triangle side length sequence and a side length ratio sequence; the second triangle sequence set corresponding to the second fragment image also includes: a triangle side length sequence and a side length ratio sequence.

[0129] Step S105 , stitching the first fragment image and the second fragment image based on the first triangle sequence set and the second triangle sequence set.

[0130] In this embodiment, by comparing the corner points in the detected edge contour of the fragments with the contour points, an ordered sequence of corner points can be obtained, the corner points are accurately positioned, and the sorting results are stable and reliable. Furthermore, by constructing triangles based on the corner point sequence and then matching through the corner point triangles, it is not only possible to reduce the impact of single corner point errors on the overall matching and improve the stability and accuracy of the matching, but also to deal with problems such as matching deviations caused by weak textures, fragment flipping, and inconsistent fragment proportions, thereby achieving an efficient, accurate, and stable splicing effect, and having significant technical innovation advantages. Moreover, the two-dimensional fragment splicing algorithm based on triangle corner points provided by the present invention can be applied to the splicing and restoration of two-dimensional fragments of various materials, especially in the fields of ceramics, paper pieces, bronze artifacts, etc., which is of great significance.

[0131] In some optional implementations, the above step S105, i.e., stitching the first fragment image and the second fragment image based on the first triangle sequence set and the second triangle sequence set, includes:

[0132] Step S1051 : determining a triangle ratio difference set, wherein the triangle ratio difference is a difference in side length ratio between a side length ratio set in a first triangle sequence set and a side length ratio set in a second triangle sequence set.

[0133] After determining the first triangle sequence set corresponding to the first fragment image and the second triangle sequence set corresponding to the second fragment image, the difference in the side length ratio of all triangles between the two fragments is calculated using the following calculation method:

[0134]

[0135] Step S1052: Determine the first triangle and the second triangle corresponding to the minimum triangle proportional difference based on the triangle proportional difference set.

[0136] Based on the first fragment, the triangles with the smallest difference with the triangles in the second fragment are obtained to form a preliminary triangle pair (the first triangle and the second triangle) that meets the requirements, without omission or duplication. Next, the preliminary triangle pair needs to be screened in the first step. In this embodiment, the threshold of the difference Diff can be limited to 0.2. The above process belongs to the judgment based on the geometric relationship between the corner points.

[0137] Step S1053, determining whether the first triangle and the second triangle satisfy a preset pixel condition.

[0138] That is, after making a set relationship judgment, further judgment can be made based on the pixel ratio relationship between the fragments.

[0139] Step S1054, when a preset pixel condition is met, determining a first corner point corresponding to the first triangle and a second corner point corresponding to the second triangle; the first corner point and the second corner point are used to splice the first fragment image with the second fragment image.

[0140] In this embodiment, the side length ratio of the triangles in the two fragments is used as a condition to compare the difference of the side length ratio of all triangles between the two fragments. The triangle side length ratio difference is calculated accurately, and the threshold is set reasonably, which can effectively screen out triangle pairs that preliminarily meet the requirements. In addition, the pixel ratio calculation can accurately reflect the matching degree of the convex hull and concave hull shapes. The comprehensive geometric relationship judgment and pixel matching degree are judged, and the corner points with high matching degree can be further accurately screened out.

[0141] In some optional implementations, the above step S1053, i.e., determining whether the first triangle and the second triangle meet a preset pixel condition, includes:

[0142] Determine a first ratio of the number of pixels of a first preset color in the first triangle to the total number of pixels in the first triangle;

[0143] Determine a second ratio of the number of pixels of a second preset color in the second triangle to the total number of pixels in the second triangle;

[0144] Wherein, when the first preset color is black, the second preset color is white; when the first preset color is white, the second preset color is black;

[0145] determining a degree of match between the first ratio and the second ratio;

[0146] When the matching degree satisfies the preset matching degree, it is determined that the first triangle and the second triangle satisfy the preset pixel condition.

[0147] Specifically, since the joints between the two fragments must be meshed with each other, and the image is already a pre-processed binary image, when the corner points of the triangle on one fragment are convex hull shapes, the matching triangle on the other fragment is concave hull shapes. The convex hull shape and the concave hull shape not only mesh with each other, but their colors also match each other. Therefore, the number of black and white pixels can be used to further determine whether the corner points of the triangles match each other. However, if there is a certain proportional transformation between the two fragments, the number of pixels cannot be used as a judgment condition. At this time, the ratio of black pixels in the convex hull and concave hull triangles to the total number of pixels in the entire triangle is needed to eliminate the proportional influence between the two fragments. The calculation formula is:

[0148] For the convex hull triangle, the black pixel ratio is: rate (pixel＝0) =(black (pixel＝0) ) / total (pixel) ;

[0149] For the concave triangle, the proportion of white pixels is: rate (pixel＝255) =(white (pixel＝255) ) / total (pixel) ;

[0150] The formula for judging the matching degree between the two is:

[0151] M=rate (pixel＝0) -rate (pixel＝255) In this embodiment, the preset matching degree can be limited to 0.1.

[0152] For example, see Figure 6As shown, taking triangle 17 in the first fragment image and triangle 11 in the second fragment image as examples, the first ratio of the number of white pixels in triangle 17 to the total number of pixels is 98%, and the first ratio of the number of black pixels in triangle 17 to the total number of pixels is 2%; the second ratio of the number of black pixels in triangle 11 to the total number of pixels is 98%, and the second ratio of the number of white pixels in triangle 11 to the total number of pixels is 2%. At this time, the difference between the first ratio of the number of white pixels in triangle 17 to the total number of pixels and the second ratio of the number of black pixels in triangle 11 to the total number of pixels is less than 0.1; at the same time, the difference between the first ratio of the number of black pixels in triangle 17 to the total number of pixels and the second ratio of the number of white pixels in triangle 11 to the total number of pixels is also less than 0.1, and it is determined that the first triangle and the second triangle meet the preset pixel condition.

[0153] Finally, if the corner points of two triangles meet both the geometric constraints and the pixel ratio constraints, the two triangles are considered to be correctly matched, which means that the corresponding corner points of the triangles are matched.

[0154] In this embodiment, the corresponding pixel proportions in the enclosed triangle are comprehensively considered to jointly measure the matching degree of the corner points between the two fragments, so that the corner points with high matching degree can be accurately screened out.

[0155] In some alternative implementations, the above step S105, i.e., stitching the first fragment image and the second fragment image based on the first triangle sequence set and the second triangle sequence set, includes:

[0156] Determine a triangle ratio difference set, wherein the triangle ratio difference is a difference in side length ratio between a side length ratio set in the first triangle sequence set and a side length ratio set in the second triangle sequence set;

[0157] Determine, based on the triangle ratio difference set, a triangle pair that meets a preset ratio difference threshold, wherein the triangle pair includes a first triangle corresponding to the first fragment image and a second triangle corresponding to the second fragment image;

[0158] Determine whether the first triangle and the second triangle in the triangle pair meet a preset pixel condition;

[0159] When a preset pixel condition is met, determining a first corner point corresponding to the first triangle and a second corner point corresponding to the second triangle;

[0160] Regenerate all first corner points into a new first corner point sequence, and regenerate all second corner points into a new second corner point sequence;

[0161] Based on the first corner point sequence, a triangle is constructed for every three adjacent corner points to generate a new first triangle set;

[0162] Based on the second corner point sequence, a triangle is constructed for every three adjacent corner points to generate a new second triangle set;

[0163] Based on the first triangle set and the second triangle set, determining the number of matches between the first corner point and the second corner point;

[0164] Determine whether the number of matches of the first corner point or the second corner point meets a preset number of matches;

[0165] When the preset matching times are met, the corresponding first corner point or the second corner point is determined as a matching point, and the matching point is used to splice the first fragment image and the second fragment image.

[0166] Specifically, in the previous implementation, the corresponding relationship between the corner points of two fragments is obtained by the triangle corner point method based on the judgment method of geometry and pixel ratio factors. Although the corner point matching pair obtained at this time has a relatively high accuracy, there may still be a problem of wrong matching. In this embodiment, on this basis, the matching pairs are further screened, and the screening method is as follows:

[0167] Assume that fragment 1 has G1, G2, G3, G4, G5, G6, G7, G8, G9, G 10 There are ten corner points, and the corner point pairs of the triangle are: (G1, G2, G3), (G2, G3, G4), (G3, G4, G5), (G4, G5, G6), (G5, G6, G7), (G6, G7, G8), (G7, G8, G9), (G8, G9, G 10 ),(G9,G 10 ,G1),(G 10 ,G1,G2).

[0168] Fragment 2 also has H1, H2, H3, H4, H5, H6, H7, H8, H9, H 10 There are ten corner points, and the corner point pairs of the triangle are: (H1, H2, H3), (H2, H3, H4), (H3, H4, H5), (H4, H5, H6), (H5, H6, H7), (H6, H7, H8), (H7, H8, H9), (H8, H9, H10, H11, H12, H13, H14, H15, H16, H17, H18, H19, H20, H21, H22, H23, H24, H25, H26, H27, H28, H29, H30, H31, H32, H33, H34, H35, H36 10 ),(H9,H 10 ,H1),(H 10 H1,H2).

[0169] Assume that fragment 1 has six corner points G3, G4, G5, G6, G7, and G8, and fragment 2 has six corner points H3, H4, H5, H6, H7, and H8 as matching points, and the remaining corner points are non-matching points.

[0170] Then the triangle formed by fragment 1 is: (G3, G4, G5), (G4, G5, G6), (G5, G6, G7), (G6, G7, G8);

[0171] Then the triangle formed by fragment 2 is: (H3, H4, H5), (H4, H5, H6), (H5, H6, H7), (H6, H7, H8).

[0172] If they can satisfy the matching conditions, ideally, the number of occurrences of each matching corner point is shown in Table 1:

[0173] Table 1

[0174] Matching Pairs (G3,H3) (G4,H4) (G5,H5) (G6,H6) (G7,H7) (G8,H8) Occurrence 1 2 3 3 2 1

[0175] Table 1 shows the number of times each matching point appears in an ideal situation. However, in actual situations, due to various reasons, the number of occurrences is less than the ideal number. Therefore, in actual operations, matching points with a matching number greater than or equal to 2 can be used as ideal matching points. According to the above criteria, (G4, H4), (G5, H5), (G6, H6), and (G7, H7) are selected as matching points. Then, according to the ordering relationship of the corner points of the two fragments, (G3, H3) and (G8, H8) are found as the starting and ending points of the matching corner points.

[0176] In this embodiment, based on the paired triangle corner points between two fragments, the number of times each corner point is paired is counted, and the corner point pairs that appear more than or equal to the preset number of matches are retained. The retained corner point pairs will be used for fragment splicing. This method can improve the accuracy of fragment splicing.

[0177] In some optional implementations, after the matching points are determined, the method further includes:

[0178] Select three first corner points and three second triangular points corresponding to the first corner points from the matching points;

[0179] Determine a first angle and a first direction between three first corner points and a second angle and a second direction between three second triangular points;

[0180] Calculate an angle difference based on the first angle and the second angle;

[0181] When the angle difference satisfies a preset range, it is determined whether the first fragment image or the second fragment image needs to be flipped based on the first direction and the second direction.

[0182] If two fragments are to be spliced, they must be on the front or back at the same time. However, some fragments have no obvious texture features on the surface and cannot be distinguished. If the two fragments are in reverse, the fragments need to be flipped 180 degrees before the fragments can be spliced. In this embodiment, it is first necessary to determine whether the two fragments are in the same direction, and the method is as follows:

[0183] Among the screened matching corner points, three points are randomly selected to form two groups of matching triangles. Figure 7 As shown, the specific method is as follows:

[0184] Assume that the coordinates of the matching triangle of fragment 1 are: C1(x1,y1), C2(x2,y2), C3(x3,y3);

[0185] Assume that the coordinates of the matching triangles of fragment 2 are: P1(z1,w1),P2(z2,w2),P3(z3,w3);

[0186] Due to the flipping of the image, although the corresponding triangles can still be matched, it is not possible to judge intuitively. Therefore, it is necessary to judge whether the two groups of fragments are in the same or opposite directions based on the positive or negative angle of the vector formed by the corresponding points between the two fragments. The specific judgment steps are:

[0187] Fragment 1:

[0188] C1C2=(x2-x1,y2-y1)

[0189] C2C3=(x3-x2,y3-y2)

[0190] C3C1=(x1-x3,y1-y3);

[0191] Fragment 2:

[0192] P1P2=(z2-z1,w2-w1)

[0193] P2P3=(z3-z2,w3-w2)

[0194] P3P1=(z1-z3,w1-w3);

[0195] The angle and direction between each vector are calculated by using the dot product, cross product and cosine value calculation formulas in the Python related library functions:

[0196] Fragment 1: Angle1, Dir1, Angle2, Dir2, Angle3, Dir3;

[0197] Fragment 2: Angle4, Dir4, Angle5, Dir5, Angle6, Dir6;

[0198] Angle mark =abs(Angle1-Angle4)+abs(Angle2-Angle5)+abs(Angle3-Angle6).

[0199] First, determine whether the angle difference is within an allowable range. Assume that the preset range is Angle mark <5. If it is satisfied, it further indicates that the matching point between the two fragments is correct. However, the direction of the angle is needed to determine whether the two fragments are flipped. The judgment basis is:

[0200] Dir>0: the vector angle is clockwise;

[0201] Dir<0: the vector angle is clockwise;

[0202] Dir=0:vector angles are collinear;

[0203] At this time, according to the directions of fragment one and fragment two, if the vector angles are in the same direction, it means that the two fragments are in the same direction; conversely, if the vector angles are in opposite directions, it means that the two fragments are in opposite directions.

[0204] If the two fragments are in the same direction, no processing is required. However, if the two fragments are in opposite directions, fragment 2 needs to be flipped 180 degrees along the perpendicular bisector so that fragment 2 and fragment 1 are in the same direction. The original corner points are also flipped and transformed. The x-axis coordinates of the corner points of fragment 2 are subtracted from the length of the fragment image to obtain the new corner point coordinates of the flipped image without re-detection.

[0205] In this embodiment, three pairs of corner points are randomly selected to form new triangle pairs. If the angles between the vectors formed by the three points are in the same direction, it means that the two fragments are both front or back; if the angles are in opposite directions, it means that one of the two fragments is front and the other is back. If they are in reverse directions, the second fragment needs to be flipped 180 degrees along the perpendicular bisector, and its corner points are also transformed accordingly. By calculating the vector angle and judging the direction, it is possible to accurately judge whether the two fragments are in the same direction, with a high judgment accuracy, and can effectively avoid splicing errors caused by reverse fragments.

[0206] In some alternative implementations, after the matching points are determined, the method further includes:

[0207] Select three first corner points and three second triangulation points corresponding to the first corner points from the matching points; wherein the maximum index difference of the selected three first corner points needs to meet the preset index value;

[0208] Determine a first distance and a first edge vector formed between three first corner points; determine a second distance and a second edge vector formed between three second triangle points;

[0209] Based on the first side vector and the second side vector, determine the angle between each vector and the horizontal line;

[0210] Based on the first distance and the second distance, determining a distance ratio between the first distance and the second distance;

[0211] Determine whether the distance ratio meets the preset conditions;

[0212] When the preset conditions are met, the average rotation angle and the average scaling ratio of the first fragment image relative to the second fragment image or the second fragment image relative to the first fragment image are calculated based on the included angle and the distance ratio.

[0213] In the previous implementation, the matching corner points are further screened. Although the matching corner points found at this time are very accurate, the erroneous matching corner points with a very small probability must still be excluded. Since the corner point pairs of the two fragments are needed to calculate the transformation matrix in the end, the screened corner point pairs need to be selected again to ensure the correctness of the match.

[0214] To calculate the precise angle and ratio of the two fragments, in this embodiment, it is necessary to randomly extract these matching points again, and ensure that the difference of the randomly extracted corner point index is large, that is, to ensure that the distance between the three extracted corner points is as far as possible. There are two considerations here. First, the various parameters between adjacent corner points are calculated in the previous steps. If there is a defect in the middle of the spliced ​​contour line, the triangle parameters formed by the corner points on both sides of the defect can be used to determine whether both sides of the defect are all from the second fragment; second, using paired points that are farther away to calculate the angle and ratio between the two fragments will result in smaller errors.

[0215] The specific method is as follows:

[0216] Assume that the six corner points of fragment one G3, G4, G5, G6, G7, and the six corner points of fragment two H3, H4, H5, H6, H7, and H8 are matching points, and the remaining corner points are non-matching points.

[0217] Suppose the first pair of matching points extracted are: (G3, H3), (G4, H4), (G5, H5), and the index order is 3, 4, 5, and the maximum index difference is 2, which does not meet the conditions, and the matching point pair must be randomly extracted again. Suppose the re-extracted points are (G3, G5), (G5, G8), (G5, G8), and the indexes are 3, 5, 8, and the maximum index difference is 5, which meets the conditions. At this time, these three pairs of matching points are used to calculate the rotation and scaling relationship between the two fragments.

[0218] The edges corresponding to the three points G3, G5, and G8 of the first fragment form vectors G3, G5, G5, G8, and G5, G8; the edges corresponding to the three points H3, H5, and H8 of the second fragment form vectors H3, H5, H5, H8, and H5, H8. At this time, they are three groups of vectors. To calculate the angle, an initial axis is required. In this embodiment, the right half axis of the horizontal line of the image can be used as the starting angle. Figure 8 As shown, the calculations are done using the math library in the Python programming language.

[0219] 1) Angle calculation:

[0220] Calculate the x and y components of a vector:

[0221] d x =x2-x1

[0222] d y =y2-y1;

[0223] Further, use the atan2 function to calculate the angle (radians):

[0224] angle rad =math.atan2(dy,dx);

[0225] 2) Distance ratio calculation:

[0226] The distance calculation formula is: D = ((x2-x1) 2 +(y2-y1) 2 ) 0.5 ;

[0227] Finally, the angles formed by the three vectors formed by the three corner points of the two fragments and the horizontal axis are:

[0228] P (G3,G5) , P (G5,G8) , P (G5,G8) ;Q (H3,H5) , Q (H5,H8) , Q (H8,H3) .

[0229] The distance ratio is: R ((G3,G5),(H3,H5)) , R ((G3,G5),(H5,H8)) , R ((G3,G5),(H8,H3)) .

[0230] At this time, it is necessary to ensure that the maximum and minimum ratios of the three side lengths are not too different, otherwise it is considered to be a problematic corner point pair and a new corner point triangle needs to be randomly extracted. If the difference in the side length ratio is small, that is, it meets the preset conditions and is considered to be within the error tolerance range, then calculate the average rotation angle and average scaling ratio of the second fragment relative to the first fragment. The calculation formula is as follows:

[0231] Average rotation angle: S mean =((P (G3,G5) -Q (H3,H5) )+(P (G5,G8) -Q (H5,H8) )+(P (G5,G8) -Q (H8,H5) ))*1 / 3;

[0232] Average scaling: R mean =(R ((G3,G5),(H3,H5)) +R ((G3,G5),(H5,H8)) +R ((G3,G5),(H8,H3)) )*1 / 3.

[0233] In this embodiment, the calculated errors of the determined rotation angle and scaling ratio are small, and the rotation and scaling processing of the fragment 2 is accurate, which can effectively ensure that the processed fragment 2 is highly consistent with the fragment 1 in angle and ratio, thereby achieving seamless splicing.

[0234] In some optional implementations, after determining the average rotation angle and the average scaling ratio, the method further includes:

[0235] Determine an image to be transformed, where the image to be transformed is the first fragment image or the second fragment image;

[0236] Increase the pixel area of ​​the image to be transformed;

[0237] Determine the center point of the image to be transformed and use the center point as the rotation center;

[0238] Based on the average rotation angle, calculate the rotation matrix;

[0239] Transform the image to be transformed based on the rotation matrix and the average scaling ratio;

[0240] Calculate the offset of the matching corner points based on the rotation matrix and average scaling;

[0241] Update the coordinates of the corner points in the corner point sequence set based on the matching corner point offsets.

[0242] In this embodiment, still taking the second fragment needing to be processed as an example, the second fragment is processed according to the rotation angle and proportional relationship between the found fragments.

[0243] Before rotating and scaling an image, you need to add white areas to the top, bottom, left, and right of the image, that is, add a background area to prevent the fragmented part of the image from disappearing after the image is rotated, affecting subsequent stitching. In this embodiment. There are two ways to add the size of the white area: one is to observe the size of the fragment area with the human eye and select a suitable angle; the other method is to select the distance of the longest straight line based on the position of the corner points of the fragments, and then estimate it based on the rotation angle. Both methods are acceptable. Experiments have shown that when the pixels of the white area added around the image are large, it will not affect the final stitching of the image. In this embodiment, a fixed value of 150 can be selected. It should be noted that if 150 pixels of white area are added to the top, bottom, left, and right of the image, then the corresponding corner point coordinates also need to be added by 150.

[0244] The specific steps for image rotation and scaling are as follows:

[0245] 1) Obtain the center point of the image. In this embodiment, the center of the image is used as the rotation center:

[0246] (h,w) = image.shape[:2];

[0247] Calculate the rotation center: Center = (w / 2, h / 2);

[0248] 2) Calculate the rotation matrix, counterclockwise rotation:

[0249] M=cv2.getRotationMatrix2D(center,S(mean),S(mean));

[0250] 3) Apply the rotation matrix to the coordinate points:

[0251] Rota_points=cv2.transform(cords.reshape(1,-1,2),M).reshape(-1,2);

[0252] 4) Apply the rotation matrix to the coordinate points:

[0253] Rota_image=cv2.warpAffine(image,M,(w,h));

[0254] After the above steps, the new image is processed according to the relative angle and scaling ratio between the two fragments. Fig. 9As shown. However, there is still a position offset, which is specifically manifested on the x-axis and y-axis. Therefore, it is necessary to further use the corresponding corner points to calculate their offsets. Since the corner points on fragment 2 have been rotated and scaled, they cannot be directly involved in the calculation. Therefore, it is necessary to rotate and scale the corner points on fragment 2 to obtain the new corner point coordinate positions.

[0255] Assume that the coordinates of the corner points of fragment 1 are: G3(x3,y3), G5(x5,y5), G8(x8,y8);

[0256] The coordinates of the corner points of fragment 2 are H3(z3,w3), H5(z5,w5), and H8(z8,w8). Since a white area was added to fragment 2, the coordinates of the corner points of fragment 2 are increased by 150. The same rotation and scaling as the image are performed. The formula for calculating the offset of the matching corner points of the two fragments is as follows:

[0257] offset (x) =((x 3- z3)+(x 5- z5)+(x 8- z8)) / 3;

[0258] offset (y) =((y 3- w3)+(y 5- w5)+(y 8- w8)) / 3.

[0259] After the rotation and scaling transformation, the two fragments are respectively added with the upper, lower, left and right offsets of the two fragments through the obtained coordinate offsets, so that the positions of the corresponding corner points of the two fragments are the same, and the overall image size is the same, which is convenient for subsequent stitching using image fusion methods.

[0260] The specific steps are as follows:

[0261] Assume that the width and height of fragment 1 are (w1, h1), and the width and height of fragment 2 are (w2, h2). If the offset of the x or y axis is a positive number, then the corresponding offset offset(x) or offset(y) is added to the x or y axis of fragment 2; if the offset of the x or y axis is a negative number, then the corresponding offset offset(x) or offset(y) is added to the x or y axis of fragment 1. Assume that the width and height of fragment 1 and fragment 2 are (w3, h3) and (w4, h4) respectively, and finally make the coordinates of the paired corner points of fragment 1 and fragment 2 consistent.

[0262] After the above operation, the corner coordinates of the two fragments are consistent, but the image sizes are generally different, that is, their widths and heights are different. In order to facilitate subsequent image fusion and stitching, it is necessary to further fill the bottom and right sides of the image with white so that the corner coordinates will not be changed. The specific method is as follows:

[0263] w max =(w3,w4)

[0264] h max =(h3,h4)

[0265] offset ( w 3) =w max -w3

[0266] offset ( h 3) =h max -h3

[0267] offset ( w 4) =w max -w4

[0268] offset ( h 4) =h max -h4.

[0269] According to the offset of the two fragment images obtained above, the white area can be filled at the bottom and right of the image. This not only makes the coordinate position of the corner point pairs of the two fragments the same, but also makes the width and height of the two fragment images consistent. The aligned fragment images refer to Fig.10 shown.

[0270] After maintaining the conditions of corner point and image alignment, we can further use average weight fusion to obtain the average fused image. Specifically, we use the cv2.addWeighted(image_1,0.5,image_2,0.5,0) function to achieve the stitching and fusion of the two images, so that the transition of the stitching line composed of the corner points is smoother.

[0271] In this embodiment, the offset is further adjusted by rotating the image after the matrix and scaling transformation, so that the transition of the stitching line composed of the corner points can be smoother, thereby effectively improving the stitching accuracy.

[0272] Reference Fig.11 As shown, it is a fragment splicing image without rotation and scaling; refer to Fig.12 As shown, it is a rotated spliced ​​image of fragments; Fig.13 As shown, it is a scaled fragment splicing image; refer to Fig.14As shown, it is a rotated and scaled spliced ​​image of fragments; refer to Fig.15 As shown, it is a fragment splicing image that is rotated, scaled, and has some areas missing; refer to Fig.16 As shown, the fragmented splicing image is rotated, scaled, partially missing, and flipped.

[0273] The invention proposes a fragment splicing algorithm based on triangle corner points, which is independent of fragment texture and has good adaptability and versatility even if the fragments have partial outline defects, scaling and other problems.

[0274] For the grayscale image and binary image of the fragmented image, the present invention uses the cv2.goodFeaturesToTrack() function to detect the corner points in the edge contour of the grayscale image; and uses cv2.findContours() to detect the contour points in the binary image. Since the contour points are ordered and the corner points are disordered, the disordered corner points are sorted using the ordered contour points to obtain the ordered corner points. In addition, in order to splice the fragments of different scaling ratios, the adjacent corner point triangles are used to sort from small to large to obtain the side length ratio relationship. The angle is not applicable here because the angle has a large error when the corner points tend to a straight line. At the same time, the triangle corner points can obtain the parameter information between non-adjacent corner points, which helps to obtain more matching information on limited corner points. Further, the present invention uses the side length ratio of the corner point triangle for matching based on the geometric relationship of the corner point triangle; in addition, since the image is a binary image at this time, which is either black or white, the convex hull and concave hull shapes formed by the two fragmented triangles are also used for matching. Since the number of black back pixels enclosed inside matches each other, and considering the relationship between scaling, the ratio of the number of black and white pixels contained in the convex hull and concave hull shapes to the total area is used for calculation to remove the impact of image scaling. Based on the above geometric relationship and pixel ratio, the matching corner points of the two fragments are screened.

[0275] Since there will be repeated pairing points when the triangles are paired, the present invention also utilizes this feature to retain the pairing points that appear more than or equal to 2 times, and further find the starting point and end point of the matching points according to the index order of the pairing points. This step can further screen the previous matching points to ensure the accuracy of the matching points. After the matching points are screened again, the present invention uses a random sampling method to randomly select three points, so that the three points with a longer distance are as far as possible, and the error is reduced. At the same time, whether the corner points on both sides of the defective part come from the same fragment is detected. Then, according to the coordinates of the corner points of the two fragments with a longer distance, the horizontal right half axis of the image is used as the starting point of the corner point, and the rotation angle of each corner point on the two fragments relative to the starting point is calculated respectively, and the corresponding side length ratio is obtained at the same time. Compared with the first fragment, the second fragment is rotated and scaled according to the obtained rotation angle and scaling ratio, and the original corner point is also transformed in the same way. According to the calculated offset of the corresponding corner points of fragment one and fragment two after the transformation, the two images are filled with white areas up and down, so that fragment one and fragment two are finally obtained, and the corner point coordinate positions are the same, and the height and width of the entire image are also the same. Only when the two are aligned can the fusion stitching method be used to make the transition between the two fragments smoother.

[0276] In the present embodiment, a two-dimensional fragment splicing device is also provided, which is used to implement the above-mentioned embodiments and preferred embodiments, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0277] This embodiment provides a two-dimensional fragment splicing device, such as Fig.17 As shown, the device comprises:

[0278] An acquisition module 201 is used to acquire a first fragment image and a second fragment image, wherein the first fragment image and the second fragment image are both binary images;

[0279] The detection module 202 is used to detect the first fragment image and the second fragment image, and respectively generate an ordered set of corner point sequences corresponding to the fragment contours, wherein the set of corner point sequences includes the corresponding corner point coordinates of each corner point;

[0280] A construction module 203 is used to construct a triangle based on the corner point sequence set, with the corner points as vertices and every three adjacent corner points;

[0281] The determination module 204 is used to determine a side length ratio set of each triangle, and generate a first triangle sequence set corresponding to the first fragment image and a second triangle sequence set corresponding to the second fragment image; wherein the side length ratio is a ratio between adjacent side lengths in a triangle;

[0282] The stitching module 205 is configured to stitch the first fragment image and the second fragment image based on the first triangle sequence set and the second triangle sequence set.

[0283] The two-dimensional fragment splicing device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0284] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0285] The embodiment of the present invention also provides a computer device having the above Fig.17 The two-dimensional fragment splicing device shown.

[0286] See also Fig.18 , Fig.18 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Fig.18 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig.18 A processor 10 is taken as an example.

[0287] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0288] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0289] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0290] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0291] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0292] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0293] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A two-dimensional fragment splicing method, characterized in that: The method comprises: Acquire a first fragment image and a second fragment image, wherein the first fragment image and the second fragment image are both binary images; Detecting the first fragment image and the second fragment image, respectively generating ordered corner point sequence sets corresponding to the fragment contours, wherein the corner point sequence sets include one-to-one corresponding corner point coordinates of each corner point; Based on the corner point sequence set, taking the corner points as vertices, a triangle is constructed with every three adjacent corner points; Determine a side length ratio set of each triangle, and generate a first triangle sequence set corresponding to the first fragment image and a second triangle sequence set corresponding to the second fragment image; wherein the side length ratio is a ratio between adjacent side lengths in a triangle; The first fragment image and the second fragment image are stitched together based on the first triangle sequence set and the second triangle sequence set.

2. The method according to claim 1, characterized in that The step of stitching the first fragment image and the second fragment image based on the first triangle sequence set and the second triangle sequence set includes: Determine a triangle ratio difference set, wherein the triangle ratio difference is a difference in side length ratio between the side length ratio set in the first triangle sequence set and the side length ratio set in the second triangle sequence set; Based on the triangle ratio difference set, determine a first triangle and a second triangle corresponding to a minimum triangle ratio difference; Determine whether the first triangle and the second triangle meet a preset pixel condition; When the preset pixel condition is met, a first corner point corresponding to the first triangle and a second corner point corresponding to the second triangle are determined; the first corner point and the second corner point are used to splice the first fragment image with the second fragment image.

3. The method according to claim 2, characterized in that The determining whether the first triangle and the second triangle meet a preset pixel condition includes: Determine a first ratio of the number of pixels of a first preset color in the first triangle to the total number of pixels in the first triangle; Determine a second ratio of the number of pixels of a second preset color in the second triangle to the total number of pixels in the second triangle; Wherein, when the first preset color is black, the second preset color is white; when the first preset color is white, the second preset color is black; determining a degree of match between the first ratio and the second ratio; When the matching degree satisfies a preset matching degree, it is determined that the first triangle and the second triangle satisfy a preset pixel condition.

4. The method according to claim 1, characterized in that: The step of stitching the first fragment image and the second fragment image based on the first triangle sequence set and the second triangle sequence set includes: Determine a triangle ratio difference set, wherein the triangle ratio difference is a difference in side length ratio between the side length ratio set in the first triangle sequence set and the side length ratio set in the second triangle sequence set; Based on the triangle ratio difference set, determining a triangle pair that meets a preset ratio difference threshold, the triangle pair including a first triangle corresponding to the first fragment image and a second triangle corresponding to the second fragment image; Determine whether the first triangle and the second triangle in the triangle pair meet a preset pixel condition; When the preset pixel condition is met, determining a first corner point corresponding to the first triangle and a second corner point corresponding to the second triangle; Regenerate a new first corner point sequence from all the first corner points, and regenerate a new second corner point sequence from all the second corner points; Based on the first corner point sequence, construct a triangle for every three adjacent corner points to generate a new first triangle set; Based on the second corner point sequence, a triangle is constructed for every three adjacent corner points to generate a new second triangle set; Based on the first triangle set and the second triangle set, determining the number of matches between the first corner point and the second corner point; Determine whether the matching times of the first corner point or the second corner point meets a preset matching times; When the preset number of matches is met, the corresponding first corner point or the second corner point is determined as a matching point, and the matching point is used to splice the first fragment image and the second fragment image.

5. The method according to claim 4, characterized in that After the matching point is determined, the method further includes: Selecting three of the first corner points and three second triangular points corresponding to the first corner points from the matching points; Determine a first angle and a first direction between the three first corner points and a second angle and a second direction between the three second triangle points; Calculating an angle difference based on the first angle and the second angle; When the angle difference satisfies a preset range, it is determined whether the first fragment image or the second fragment image needs to be flipped based on the first direction and the second direction.

6. The method according to claim 4, characterized in that After the matching point is determined, the method further includes: Selecting three first corner points and three second triangulation points corresponding to the first corner points from the matching points; wherein the maximum index difference of the selected three first corner points needs to satisfy a preset index value; Determine a first distance and a first edge vector formed between the three first corner points; determine a second distance and a second edge vector formed between the three second triangle points; Based on the first side vector and the second side vector, determining an angle between each vector and a horizontal line; Based on the first distance and the second distance, determining a distance ratio between the first distance and the second distance; Determining whether the distance ratio meets a preset condition; When a preset condition is met, based on the included angle and the distance ratio, an average rotation angle and an average scaling ratio of the first fragment image relative to the second fragment image or of the second fragment image relative to the first fragment image are calculated.

7. The method according to claim 6, characterized in that After determining the average rotation angle and the average scaling ratio, the method further includes: Determine an image to be transformed, where the image to be transformed is the first fragment image or the second fragment image; increasing the pixel area of ​​the image to be transformed; Determine the center point of the image to be transformed, and use the center point as the rotation center; Based on the average rotation angle, calculating a rotation matrix; Transforming the image to be transformed based on the rotation matrix and the average scaling ratio; Calculate the matching corner point offset based on the rotation matrix and the average scaling; The coordinates of the corner points in the corner point sequence set are updated based on the matching corner point offsets.

8. A two-dimensional fragment splicing device, characterized in that: The device comprises: An acquisition module, used for acquiring a first fragment image and a second fragment image, wherein the first fragment image and the second fragment image are both binary images; A detection module, used for detecting the first fragment image and the second fragment image, and generating an ordered set of corner point sequences corresponding to the fragment contours, respectively, wherein the set of corner point sequences includes one-to-one corresponding corner point coordinates of each corner point; A construction module, configured to construct a triangle based on the corner point sequence set, with the corner points as vertices and every three adjacent corner points; a determination module, configured to determine a side length ratio set of each triangle, and generate a first triangle sequence set corresponding to the first fragment image and a second triangle sequence set corresponding to the second fragment image; wherein the side length ratio is a ratio between adjacent side lengths in a triangle; The splicing module is used for splicing the first fragment image and the second fragment image based on the first triangle sequence set and the second triangle sequence set.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the two-dimensional fragment splicing method described in any one of claims 1-7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the two-dimensional fragment splicing method described in any one of claims 1-7.

Images