A two-dimensional fragment splicing method, device, computer device and storage medium

Abstract

The present invention relates to the technical field of image processing, and discloses a two-dimensional fragment splicing method, device, computer device and storage medium. The method includes: obtaining a first fragment image and a second fragment image; detecting the first fragment image and the second fragment image, and respectively generating a set of corner point sequences with ordered fragment contours; based on the set of corner point sequences, using the corner points as vertices, constructing a triangle for every three adjacent corner points; determining a set of side length ratios for each triangle, and generating a first triangle sequence set corresponding to the first fragment image and a second triangle sequence set corresponding to the second fragment image; splicing the first fragment image and the second fragment image based on the first triangle sequence set and the second triangle sequence set. The present invention can not only reduce the influence of a single corner point error on the overall matching, but also cope with problems such as matching deviations caused by weak texture, fragment flipping, inconsistent fragment ratios, etc., and achieves an efficient, accurate and stable splicing effect.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to a two-dimensional fragment splicing method, device, computer device and storage medium. Background Art

[0002] The restoration of two-dimensional fragments plays an important role in fields such as image restoration, medical image splicing, and archaeological cultural relic restoration. In the past, the restoration and splicing of two-dimensional fragments were mostly carried out manually, with very low efficiency. However, with the development of computer technology, especially the rapid development in computer graphics, the method of using a computer to process two-dimensional fragment splicing has begun to be widely applied.

[0003] Currently, the two-dimensional irregular fragment splicing using computer technology is mainly based on contour splicing, that is, according to the found mutually matching contour curve segments, and then automatically or interactively splicing them together. The splicing process mainly includes the following steps: (1) calculating the characteristics of the fragment contour points; (2) matching the set of contour point characteristics between fragments to obtain the correspondence between contour points; (3) calculating the rigid body transformation matrix according to the correspondence between contour points; (4) transforming the fragments according to the obtained transformation matrix, and then directly splicing. Although this algorithm can solve the splicing and matching problem of irregular fragments to a certain extent, for some fragments, such as ceramics, paper, bronze cultural relics and other materials, there are some problems that restrict the application of this method, such as: (1) there are often different degrees of defects at the fracture; (2) the surface texture of the fragment material is weak, the color characteristics are not obvious or even there is no color; (3) due to reasons such as lens distortion, the change of fragment position, and the change of the distance between the fragment and the lens, after imaging, the resolutions of the images are different; (4) the fragments may be flipped.

[0004] Specifically, the defect will cause a large difference in the corner points extracted from different images; the weak texture will cause the inability to match based on the pixel information of adjacent fragments; if two fragments are flipped, they cannot be spliced; the change of fragment position and the distance from the camera will cause a large deviation in the distance between the fragment contour and the fragment corner points calculated on different images, and finally there is a matching deviation, or even no matching. Summary of the Invention

[0005] In view of this, the present invention provides a two-dimensional fragment splicing method, device, computer device and storage medium to solve the problem that the splicing effect of the existing contour-based fragment splicing method is unstable.

[0006] In a first aspect, the present invention provides a two-dimensional fragment splicing method, and the method includes:

[0007] Obtain a first fragmented image and a second fragmented image, where both the first fragmented image and the second fragmented image are binary images;

[0008] Detect the first fragmented image and the second fragmented image, and respectively generate an ordered set of corner point sequences corresponding to the fragmented contours. The set of corner point sequences includes the corner point coordinates corresponding to each corner point one by one;

[0009] Based on the set of corner point sequences, with the corner points as vertices, construct a triangle for every three adjacent corner points;

[0010] Determine the set of side length ratios of each triangle, and generate a first triangle sequence set corresponding to the first fragmented image and a second triangle sequence set corresponding to the second fragmented image; where the side length ratio is the ratio between adjacent side lengths in the triangle;

[0011] Stitch the first fragmented image and the second fragmented image based on the first triangle sequence set and the second triangle sequence set.

[0012] By comparing the corner points in the detected fragmented edge contour with the contour points, the present invention can obtain an ordered set of corner points. The corner points are accurately positioned and the sorting result is stable and reliable. Further, by constructing triangles based on the corner point sequences and then performing matching through the corner point triangles, not only can the influence of a single corner point error on the overall matching be reduced, the stability and accuracy of the matching be improved, but also problems such as matching deviations caused by weak texture, flipping of the fragments, inconsistent fragment ratios, etc. can be addressed, achieving an efficient, accurate, and stable stitching effect, and having significant technological innovation advantages. Moreover, a two-dimensional fragment stitching algorithm based on triangle corner points provided by the present invention can be applied to the two-dimensional fragment stitching and restoration work of various materials, and is particularly significant in the fields of ceramics, paper, bronze cultural relics, etc.

[0013] In an optional implementation manner, stitching the first fragmented image and the second fragmented image based on the first triangle sequence set and the second triangle sequence set includes:

[0014] Determine the set of triangle ratio differences, where the triangle ratio difference is the difference in side length ratios between the set of side length ratios in the first triangle sequence set and the set of side length ratios in the second triangle sequence set;

[0015] Based on the set of triangle ratio differences, determine the first triangle and the second triangle corresponding to the minimum triangle ratio difference;

[0016] Judge whether the first triangle and the second triangle meet the preset pixel conditions;

[0017] When the preset pixel condition is satisfied, determine the first corner point corresponding to the first triangle and the second corner point corresponding to the second triangle; the first corner point and the second corner point are used to splice the first fragmented image and the second fragmented image.

[0018] In this embodiment, the ratio of the side lengths of the triangles formed in the two fragments is used as a condition, and the differences in the ratios of the side lengths of all triangles between the two fragments are compared respectively. The calculation of the triangle side length ratio difference is accurate, and the threshold setting is reasonable, which can effectively screen out the triangle pairs that initially meet the requirements. In addition, the pixel ratio calculation can accurately reflect the matching degree of the convex hull and concave hull shapes. By comprehensively judging the geometric relationship and pixel matching degree, the corner points with high matching degree can be further accurately screened out.

[0019] In an alternative embodiment, determining whether the first triangle and the second triangle satisfy the preset pixel condition includes:

[0020] Determine a first ratio of the number of first preset color pixels in the first triangle to the total number of pixels in the first triangle;

[0021] Determine a second ratio of the number of second preset color pixels in the second triangle to the total number of pixels in the second triangle;

[0022] Wherein, when the first preset color is black, the second preset color is white; when the first preset color is white, the second preset color is black;

[0023] Determine the matching degree between the first ratio and the second ratio;

[0024] When the matching degree meets the preset matching degree, determine that the first triangle and the second triangle satisfy the preset pixel condition.

[0025] In this embodiment, by comprehensively considering the pixel ratio situation corresponding to the enclosed triangles, the matching degree of the corner points between the two fragments can be jointly measured, and the corner points with high matching degree can be accurately screened out.

[0026] In an alternative embodiment, splicing the first fragmented image and the second fragmented image based on the first triangle sequence set and the second triangle sequence set includes:

[0027] Determine a set of triangle ratio differences, where the triangle ratio difference is the difference in the side length ratios between the set of side length ratios in the first triangle sequence set and the set of side length ratios in the second triangle sequence set;

[0028] Based on the set of triangle ratio differences, determine the triangle pairs that meet the preset ratio difference threshold, and the triangle pairs include the first triangle corresponding to the first fragmented image and the second triangle corresponding to the second fragmented image;

[0029] Determine whether the first triangle and the second triangle in the triangle pair satisfy a preset pixel condition;

[0030] When the preset pixel condition is satisfied, determine the first corner point corresponding to the first triangle and the second corner point corresponding to the second triangle;

[0031] Regenerate all the first corner points into a new first corner point sequence and regenerate all the second corner points into a new second corner point sequence;

[0032] Based on the first corner point sequence, construct a triangle for every three adjacent corner points to generate a new set of first triangles;

[0033] Based on the second corner point sequence, construct a triangle for every three adjacent corner points to generate a new set of second triangles;

[0034] Based on the first triangle set and the second triangle set, determine the matching times of the first corner point and the second corner point;

[0035] Judge whether the matching times of the first corner point or the second corner point satisfy a preset matching times;

[0036] When the preset matching times is satisfied, determine the corresponding first corner point or second corner point as a matching point, and the matching point is used to splice the first fragment image and the second fragment image.

[0037] In this embodiment, according to the paired triangle corner points between two fragments, the pairing times of each corner point are counted, and the corner point pairs with the occurrence times greater than or equal to the preset matching times are reserved. The reserved corner point pairs will be used for fragment splicing, and this method can improve the accuracy of fragment splicing.

[0038] In an optional embodiment, after determining the matching points, it further includes:

[0039] Select three first corner points and three second triangle points corresponding to the first corner points from the matching points;

[0040] Determine the first angle and the first direction between the three first corner points and the second angle and the second direction between the three second triangle points;

[0041] Calculate the angle difference based on the first angle and the second angle;

[0042] When the angle difference satisfies a preset range, judge whether the first fragment image or the second fragment image needs to be flipped based on the first direction and the second direction.

[0043] In this embodiment, through vector included angle calculation and direction judgment, it can accurately judge whether two fragments are in the same direction, and the judgment accuracy rate is high, which can effectively avoid splicing errors caused by fragment reversal.

[0044] In an alternative embodiment, after determining the matching points, the method further includes:

[0045] Select three first corner points and three corresponding second triangular points from the matching points; wherein, the maximum index difference of the three selected first corner points needs to satisfy a preset index value;

[0046] Determine the first distance and the first side vector formed between the three first corner points; determine the second distance and the second side vector formed between the three second triangular points;

[0047] Based on the first side vector and the second side vector, determine the angle between each vector and the horizontal line;

[0048] Based on the first distance and the second distance, determine the distance ratio between the first distance and the second distance;

[0049] Judge whether the distance ratio satisfies a preset condition;

[0050] When the preset condition is satisfied, based on the angle and the distance ratio, calculate the average rotation angle and the average scaling ratio of the first fragment image relative to the second fragment image or the second fragment image relative to the first fragment image.

[0051] In this embodiment, the calculated errors of the determined rotation angle and scaling ratio are small, and the rotation and scaling processing of the second fragment is precise, which can effectively ensure that the processed second fragment is highly consistent with the first fragment in terms of angle and ratio, and achieve seamless splicing.

[0052] In an alternative embodiment, after determining the average rotation angle and the average scaling ratio, the method further includes:

[0053] Determine the image to be transformed, and the image to be transformed is the first fragment image or the second fragment image;

[0054] Increase the pixel area of the image to be transformed;

[0055] Determine the center point of the image to be transformed, and use the center point as the rotation center;

[0056] Calculate the rotation matrix based on the average rotation angle;

[0057] Transform the image to be transformed based on the rotation matrix and the average scaling ratio;

[0058] Calculate the matching corner point offset based on the rotation matrix and the average scaling;

[0059] Update the corner point coordinates in the corner point sequence set based on the matching corner point offset.

[0060] In this embodiment, the image after rotation matrix and scaling ratio transformation is further adjusted by an offset amount, so that the stitching line formed by the corner points can be made smoother, effectively improving the stitching accuracy.

[0061] In a second aspect, the present invention provides a two-dimensional fragment stitching device, which includes:

[0062] An acquisition module for acquiring a first fragment image and a second fragment image, wherein both the first fragment image and the second fragment image are binary images;

[0063] A detection module for detecting the first fragment image and the second fragment image, respectively generating an ordered set of corner point sequences corresponding to the fragment contours, and the set of corner point sequences includes the corner point coordinates corresponding to each corner point one by one;

[0064] A construction module for constructing a triangle with every three adjacent corner points as vertices based on the set of corner point sequences;

[0065] A determination module for determining the set of side length ratios of each triangle, generating a first triangle sequence set corresponding to the first fragment image and a second triangle sequence set corresponding to the second fragment image; wherein, the side length ratio is the ratio between adjacent side lengths in the triangle;

[0066] A stitching module for stitching the first fragment image and the second fragment image based on the first triangle sequence set and the second triangle sequence set.

[0067] In a third aspect, the present invention provides a computer device, including: a memory and a processor, which are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to execute the two-dimensional fragment stitching method according to the first aspect or any corresponding embodiment thereof.

[0068] In a fourth aspect, the present invention provides a computer-readable storage medium, which stores computer instructions for causing a computer to execute the two-dimensional fragment stitching method according to the first aspect or any corresponding embodiment thereof.

[0069] It should be noted that since the two-dimensional fragment stitching device, computer device, and computer-readable storage medium provided by the present invention correspond to the above two-dimensional fragment stitching method. Therefore, for the beneficial effects of the two-dimensional fragment stitching device, computer device, and computer-readable storage medium, please refer to the description of the corresponding beneficial effects of the two-dimensional fragment stitching method above, and will not be repeated here. Description of the Drawings

[0070] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the accompanying drawings required for the description of the specific embodiments or the prior art. Obviously, the accompanying drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can also be obtained based on these drawings.

[0071] Figure 1 is a schematic flowchart of a two-dimensional fragment splicing method according to an embodiment of the present invention;

[0072] Figure 2 is a schematic diagram of the grayscale image and binary image of two fragments according to an embodiment of the present invention;

[0073] Figure 3 is a schematic diagram of the corner point images of two fragments detected according to an embodiment of the present invention;

[0074] Figure 4 is a schematic diagram of the contours of two fragments detected according to an embodiment of the present invention;

[0075] Figure 5 is a schematic diagram of ordered corner points according to an embodiment of the present invention;

[0076] Figure 6 is a schematic diagram of the construction of partial triangles according to an embodiment of the present invention;

[0077] Figure 7 is a schematic diagram of randomly selected triangle matching according to an embodiment of the present invention;

[0078] Figure 8 is a schematic diagram of the included angle between the corner points of two fragments and the horizontal line and the scaling ratio during matching according to an embodiment of the present invention;

[0079] Figure 9 is a schematic diagram of the image obtained after rotating and scaling Fragment 2 according to an embodiment of the present invention;

[0080] Figure 10 is a schematic diagram after aligning the matching corner points of two fragment images according to an embodiment of the present invention;

[0081] Figure 11 is a schematic diagram of the spliced image of fragments without rotation and scaling according to an embodiment of the present invention;

[0082] Figure 12 is a schematic diagram of the spliced image of rotated fragments according to an embodiment of the present invention;

[0083] Figure 13 is a schematic diagram of the spliced image of scaled fragments according to an embodiment of the present invention;

[0084] Figure 14 Schematic diagram of a rotated and scaled fragment splicing image according to an embodiment of the present invention;

[0085] Figure 15 Schematic diagram of a rotated, scaled, and partially missing fragment splicing image according to an embodiment of the present invention;

[0086] Figure 16 Schematic diagram of a rotated, scaled, partially missing, and flipped fragment splicing image according to an embodiment of the present invention;

[0087] Figure 17 Block diagram of the structure of a two-dimensional fragment splicing device according to an embodiment of the present invention;

[0088] Figure 18 Schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed implementation manners

[0089] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0090] Currently, in the fields of cultural heritage restoration, image recognition, industrial production quality inspection, etc., the demand for high-precision and high-efficiency splicing technologies continues to grow. However, existing splicing technologies have problems such as insufficient accuracy and limited application scope. In the present invention, triangle corner recognition is used to achieve an efficient, accurate, and stable splicing effect, with significant technological innovation advantages.

[0091] In terms of development prospects, this technology is expected to be widely applied in scenarios such as image restoration, image synthesis, intelligent manufacturing, and cultural relic restoration. Its popularization and application will greatly improve the automation and intelligence levels of various industries and promote technological transformation. After industrialization, it is expected to effectively improve production efficiency, reduce costs, and form significant economic benefits. In addition, the technology of the present invention also has important significance in social values such as cultural relic protection and cultural inheritance, which helps to improve the technical level of the industry and promote the integrated innovation development of culture and technology.

[0092] Specifically, according to an embodiment of the present invention, an embodiment of a two-dimensional fragment splicing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0093] In this embodiment, a two-dimensional fragment splicing method is provided, which can be executed by devices such as a server, a terminal, and a mobile terminal. Figure 1 It is a flowchart of the two-dimensional fragment splicing method according to an embodiment of the present invention, as Figure 1 shown. The process includes the following steps:

[0094] Step S101, obtain a first fragment image and a second fragment image, where both the first fragment image and the second fragment image are binary images.

[0095] In this embodiment, it is necessary to select a suitable color background so that the fragment color has a high contrast with the background color, and use a photographing device to photograph each fragment in turn. Further, in this embodiment, morphological operations such as opening and closing operations, grayscale conversion, and binarization are used to segment the fragment area and the background area, and the grayscale image and the binary image of the fragment area are retained. In this embodiment, taking the splicing of two fragments as an example, both the first fragment image and the second fragment image are images after segmentation processing. Refer to Figure 2 shown, which are the grayscale image and the binary image of a first fragment image and a second fragment image after segmentation provided by this embodiment.

[0096] Step S102, detect the first fragment image and the second fragment image, and respectively generate an ordered set of corner point sequences corresponding to the fragment contours, where the set of corner point sequences includes the corner point coordinates corresponding to each corner point one by one.

[0097] Specifically, in this embodiment, the cv2.goodFeaturesToTrack() function in the opencv library is used to detect the corner points of the fragment edge contour in the grayscale image. Refer to Figure 3 shown (it should be noted that the fragment area in the attached drawing is actually a binary black and white image, and other colors are used here for the convenience of image marking and explanation). At this time, the obtained corner points are disordered. At the same time, the cv2.findContours() function is used to obtain the set of fragment edge contour points in the binary image. The detected contour image is referred to Figure 4 shown, and the set of edge contour points is ordered. In this embodiment, according to the ordered set of contour points, the disordered corner points are sorted, that is, the ordered contour point coordinates are compared with the disordered corner point coordinates, so as to obtain an ordered set of corner point sequences. Refer to Figure 5 shown. Only the ordered set of corner point sequences can better match the corner points of the two contours.

[0098] The method for sorting the disordered corner points based on the set of edge contour points is as follows:

[0099] Suppose the obtained disordered corner points are as follows: A1(x1,y1), A2(x2,y2)…A i(x i , y i )…A m (x m , y m );The set of contour points is as follows: B1(z1, w1), B2(z2, w2)…B i (z i , w i )…B n (z n , w n ). The contour point indices are 1, 2, 3…n. The detailed positioning and sorting method is as follows:

[0100] 1) The formula for calculating the distance between each corner point and all contour points is: D i =(x i - z i ) 2 +(y i - w i ) 2 . For each corner point, m distance sequences are obtained as:

[0101] (D 11 , D 12 , LD 1n );(D 21 , D 22 , LD 2n );L;(D i1 , D i2 , LD in );L(D m1 , D m2 , LD mn );

[0102] 2) For each corner point, select the minimum distance, assumed to be D1p and D1q, and their corresponding indices are p and q. The index position of this corner point is: I = 0.5*(p + q). Finally, the average index position sequence of all corner points is I1, I2,…I i …I m ;

[0103] 3) Then sort them in ascending order according to the average index size to get J1, J2,…J i ,…J m . This index sequence is the corner points sorted according to the ordered contour points. At this time, the corner points are ordered corner points according to the set of fragment edge contour points: C1(x1, y1), C2(x2, y2)…C i (x i , y i )…C m (x m , ym ), that is, the set of corner point sequences.

[0104] In this embodiment, the cv2.goodFeaturesToTrack() function is used to detect the corner points of the edge contour of the grayscale image, and the cv2.findContours() function is used to detect the contour points of the binary image. The ordered contour points are used to sort the disordered corner points, so as to obtain an ordered corner point sequence, laying a solid foundation for subsequent accurate matching.

[0105] Step S103, based on the set of corner point sequences, with the corner points as vertices, construct a triangle for every three adjacent corner points.

[0106] After obtaining the ordered set of corner point sequences, according to the corner point sequence coordinates in the fragment, every three adjacent corner points are combined into a triangle, and the constructed triangle is referred to Figure 6 as shown.

[0107] In this embodiment, matching is performed based on the corner point triangles. As the combination with the fewest corner points to form a polygon, the triangle can reduce the impact of a single corner point error on the overall matching and improve the stability and accuracy of the matching.

[0108] Step S104, determine the set of side length ratios of each triangle, and generate the first triangle sequence set corresponding to the first fragment image and the second triangle sequence set corresponding to the second fragment image; wherein, the side length ratio is the ratio between adjacent side lengths in the triangle.

[0109] After constructing the triangle, calculate the side lengths and side length ratios of the three sides of this triangle.

[0110] To splice two fragments, first find the splicing points between the two fragments. In this embodiment, the triangles formed by the corner points of the two fragments are used for matching. One reason is that the triangle is the combination with the fewest corner points required to form a polygon; the other reason is that the impact of an incorrect corner point will not be too large.

[0111] First, according to the method of forming triangles with three adjacent corner points, make the triangles formed between the two fragments for matching. The matching method is as follows:

[0112] Suppose the corner points of the two fragments are respectively: G1, G2...G i ...G m ; H1, H2,...H j ...H m ; Since the fragment contour is closed, the corner points at the beginning and end should also be closed. The closed corner point sequences are respectively:

[0113] G 1, G2…G i …G m, G1, G2; H1, H2…H i …H n , H1, H2; The triangle sequences formed by every three adjacent corner points are:

[0114] (G1, G2, G3), (G2, G3, G4), (G3, G4, G5)…(G i-1 , G i , G i+1 )…(G m , G1, G2)

[0115] (H1, H2, H3), (H2, H3, H4), (H3, H4, H5)…(H j-1 , H j , H j+1 )…(H n , H1, H2);

[0116] Since what the present invention aims to solve is that when photographing fragments, two fragments may present a certain proportional relationship, resulting in different sizes of the fragments, so the transformation relationship of the triangle corner points cannot be directly used for calculation, and the similarity of triangles needs to be used to determine whether the corner points of two fragments match.

[0117] Therefore, in this embodiment, it is also necessary to calculate the side lengths between every triangle corner point, sort each triangle side length from smallest to largest, and the obtained sequence of each triangle side length is:

[0118] (a1, a2, a3), (a2, a3, a4), (a3, a4, a5)…(a i-1 , a i , a i+1 )…(a m , a1, a2)

[0119] (b1, b2, b3), (b2, b3, b4), (b3, b4, b5)…(b j-1 , b j , b j+1 )…(b n , b1, b2);

[0120] Furthermore, calculate the ratios of the other two side lengths to the minimum side length respectively, and the calculation method formula is as follows:

[0121] r (i,i-1) = a i / a i-1

[0122] r (i+1,i) = a i+1 / a i

[0123] t (j,j-1) = b j / b j-1

[0124] t (j+1,j) = b j+1 / b j ;

[0125] The formed sequence of side length ratios, that is, the set of side length ratios is:

[0126] (r (i,i-1) , r (i+1,i) )

[0127] (t (j,j-1) , t (j+1,j) ).

[0128] That is, the first triangle sequence set corresponding to the first fragment image in this embodiment includes: the triangle side length sequence and the side length ratio sequence; the second triangle sequence set corresponding to the second fragment image also includes: the triangle side length sequence and the side length ratio sequence.

[0129] Step S105, based on the first triangle sequence set and the second triangle sequence set, splice the first fragment image and the second fragment image.

[0130] In this embodiment, by comparing the corner points and contour points in the detected fragment edge contour, an ordered corner point sequence can be obtained. The corner points are accurately positioned and the sorting result is stable and reliable. Further, triangles are constructed according to the corner point sequence, and then matched through the corner point triangles. This can not only reduce the influence of individual corner point errors on the overall matching, improve the stability and accuracy of the matching, but also cope with problems such as matching deviations caused by weak texture, fragment flipping, inconsistent fragment ratios, etc., achieving an efficient, accurate and stable splicing effect, and having significant technological innovation advantages. Moreover, a two-dimensional fragment splicing algorithm based on triangle corner points provided by the present invention can be applied to the two-dimensional fragment splicing and restoration work of various materials, especially has important significance in the fields of ceramics, paper, bronze cultural relics, etc.

[0131] In some alternative embodiments, the above step S105, that is, based on the first triangle sequence set and the second triangle sequence set to splice the first fragment image and the second fragment image, includes:

[0132] Step S1051, determine the set of triangle ratio differences, where the triangle ratio difference is the difference in side length ratios between the set of side length ratios in the first triangle sequence set and the set of side length ratios in the second triangle sequence set.

[0133] After determining the first set of triangle sequences corresponding to the first fragmented image and the second set of triangle sequences corresponding to the second fragmented image as described above, calculate the difference in the side length ratios of all triangles between the two fragments. The calculation method is as follows:

[0134]

[0135] Step S1052: Based on the set of triangle ratio differences, determine the first triangle and the second triangle corresponding to the minimum triangle ratio difference.

[0136] Taking the first fragment as the basis, respectively obtain the triangles in the first fragment with the smallest difference from the triangles in the second fragment, and form a set of triangle pairs (the first triangle and the second triangle) that initially meet the requirements, without omission and without repetition. Next, the initial triangle pairs need to be screened for the first step. In this embodiment, the threshold of the difference Diff can be limited to 0.2. The above process belongs to making a judgment based on the geometric relationship between the corner points.

[0137] Step S1053: Determine whether the first triangle and the second triangle meet the preset pixel conditions.

[0138] That is to say, after making the set relationship judgment, further judgment can also be made according to the pixel ratio relationship between the fragments.

[0139] Step S1054: When the preset pixel conditions are met, determine the first corner point corresponding to the first triangle and the second corner point corresponding to the second triangle; the first corner point and the second corner point are used to splice the first fragmented image and the second fragmented image.

[0140] In this embodiment, the ratio of the side lengths of the triangles formed in the two fragments is used as a condition, and the differences in the side length ratios of all triangles between the two fragments are compared respectively. The calculation of the triangle side length ratio difference is accurate, and the threshold setting is reasonable, which can effectively screen out the triangle pairs that initially meet the requirements. In addition, the pixel ratio calculation can accurately reflect the matching degree of the convex hull and concave hull shapes. Making a judgment by combining the geometric relationship judgment and the pixel matching degree can further accurately screen out the corner points with a high matching degree.

[0141] In some alternative embodiments, the above step S1053, that is, determining whether the first triangle and the second triangle meet the preset pixel conditions, includes:

[0142] Determine the first ratio of the number of first preset color pixels in the first triangle to the total number of pixels in the first triangle;

[0143] Determine the second ratio of the number of second preset color pixels in the second triangle to the total number of pixels in the second triangle;

[0144] Among them, when the first preset color is black, the second preset color is white; when the first preset color is white, the second preset color is black;

[0145] Determine the matching degree between the first ratio and the second ratio;

[0146] When the matching degree meets the preset matching degree, it is determined that the first triangle and the second triangle meet the preset pixel conditions.

[0147] Specifically, since the splicing part between the two fragments must be meshed with each other, and the image is already a pre-processed binary image, when the corner point of the triangle on one fragment is in the shape of a convex hull, the triangle on the other fragment that matches it is in the shape of a concave hull. The convex hull shape and the concave hull shape not only mesh with each other, but also their colors match each other. Therefore, the number of black and white pixels can be used to further determine whether the triangle corner points match each other. However, if there is a certain proportional transformation between the two fragments, the number of pixels cannot be used as a judgment condition. At this time, the ratio of the number of black pixel points in the convex hull and concave hull triangles to the total number of pixel points in the whole triangle is required to eliminate the proportional influence between the two fragments. The calculation formula is:

[0148] For the convex hull triangle, the proportion of black pixels is: rate (pixel＝0) =(black (pixel＝0) ) / total (pixel) ;

[0149] For the concave hull triangle, the proportion of white pixels is: rate (pixel＝255) =(white (pixel＝255) ) / total (pixel) ;

[0150] The formula for judging the matching degree between the two is:

[0151] M = rate (pixel＝0) - rate (pixel＝255) , and in this embodiment, this preset matching degree can be limited to 0.1.

[0152] Illustrate with an example, refer to Figure 6As shown, taking triangle 17 in the first fragmented image and triangle 11 in the second fragmented image as examples, the first ratio of the number of white pixel points in triangle 17 to the total number of pixel points is 98%, and the first ratio of the number of black pixel points in triangle 17 to the total number of pixel points is 2%; the second ratio of the number of black pixel points in triangle 11 to the total number of pixel points is 98%, and the second ratio of the number of white pixel points in triangle 11 to the total number of pixel points is 2%. At this time, the difference between the first ratio of the number of white pixel points in triangle 17 to the total number of pixel points and the second ratio of the number of black pixel points in triangle 11 to the total number of pixel points is less than 0.1; at the same time, the difference between the first ratio of the number of black pixel points in triangle 17 to the total number of pixel points and the second ratio of the number of white pixel points in triangle 11 to the total number of pixel points is also less than 0.1. At this time, it is determined that the first triangle and the second triangle meet the preset pixel condition.

[0153] Finally, if the corner points of the two triangles simultaneously meet the geometric limit and the pixel ratio limit, it is considered that the two triangles are correctly matched, which means that the corresponding corner points of the triangles are matched.

[0154] In this embodiment, by comprehensively considering the pixel ratio of the corresponding triangles in the enclosed area, the matching degree of the corner points between the two fragments can be jointly measured, and the corner points with high matching degree can be accurately screened out.

[0155] In some alternative embodiments, step S105 above, that is, splicing the first fragmented image and the second fragmented image based on the first triangle sequence set and the second triangle sequence set, includes:

[0156] Determine a set of triangle ratio differences, where the triangle ratio difference is the difference in the side length ratio between the set of side length ratios in the first triangle sequence set and the set of side length ratios in the second triangle sequence set;

[0157] Based on the set of triangle ratio differences, determine pairs of triangles that meet the preset ratio difference threshold, and each pair of triangles includes the first triangle corresponding to the first fragmented image and the second triangle corresponding to the second fragmented image;

[0158] Judge whether the first triangle and the second triangle in the pair of triangles meet the preset pixel condition;

[0159] When the preset pixel condition is met, determine the first corner point corresponding to the first triangle and the second corner point corresponding to the second triangle;

[0160] Regenerate all the first corner points into a new first corner point sequence and regenerate all the second corner points into a new second corner point sequence;

[0161] Based on the first corner point sequence, construct a triangle for every three adjacent corner points to generate a new set of first triangles;

[0162] Based on the second corner point sequence, construct a triangle for every three adjacent corner points to generate a new set of second triangles;

[0163] Based on the first set of triangles and the second set of triangles, determine the matching times between the first corner points and the second corner points;

[0164] Judge whether the matching times of the first corner points or the second corner points meet the preset matching times;

[0165] When the preset matching times are met, determine the corresponding first corner points or second corner points as matching points, and the matching points are used to splice the first fragmented image and the second fragmented image.

[0166] Specifically, in the previous embodiment, through the triangle corner point method and the judgment method based on geometric and pixel ratio factors, the corresponding relationship between the two fragmented corner points is obtained. Although the corner point matching pairs obtained at this time have relatively high accuracy, there may still be problems of incorrect pairing. In this embodiment, on this basis, further screening of the matching pairs is carried out, and the screening method is as follows:

[0167] Suppose Fragment 1 has ten corner points G1, G2, G3, G4, G5, G6, G7, G8, G9, G 10 The triangle corner point pairs formed are: (G1, G2, G3), (G2, G3, G4), (G3, G4, G5), (G4, G5, G6), (G5, G6, G7), (G6, G7, G8), (G7, G8, G9), (G8, G9, G 10 ), (G9, G 10 , G1), (G 10 , G1, G2).

[0168] Fragment 2 also has ten corner points H1, H2, H3, H4, H5, H6, H7, H8, H9, H 10 The triangle corner point pairs formed are: (H1, H2, H3), (H2, H3, H4), (H3, H4, H5), (H4, H5, H6), (H5, H6, H7), (H6, H7, H8), (H7, H8, H9), (H8, H9, H 10 ), (H9, H 10 , H1), (H 10 , H1, H2).

[0169] Suppose Fragment 1 has six corner points G3, G4, G5, G6, G7, and G8, and the six corner points H3, H4, H5, H6, H7, and H8 of Fragment 2 are matching points, and the remaining corner points are non-matching points.

[0170] Then the triangles formed by Fragment 1 are: (G3, G4, G5), (G4, G5, G6), (G5, G6, G7), (G6, G7, G8);

[0171] Then the triangles formed by Fragment 2 are: (H3, H4, H5), (H4, H5, H6), (H5, H6, H7), (H6, H7, H8).

[0172] If the matching conditions can be satisfied with each other, ideally, the number of occurrences of each matching corner point is as shown in Table 1:

[0173] Table 1

[0174] Matching pair (G3, H3) (G4, H4) (G5, H5) (G6, H6) (G7, H7) (G8, H8) Occurrence times 1 2 3 3 2 1

[0175] The above Table 1 shows the number of occurrences of each matching point under ideal conditions. However, in actual situations, due to various reasons, the number of occurrences is less than the ideal number. Therefore, in actual operations, the matching points with a matching number greater than or equal to 2 times can be used as ideal matching points. According to the above criteria, (G4, H4), (G5, H5), (G6, H6), and (G7, H7) are selected as matching points, and then according to the corner point sorting relationship of the two fragments, (G3, H3) and (G8, H8) are found as the starting point and ending point of the matching corner points.

[0176] In this embodiment, according to the corner points of the paired triangles between the two fragments, the number of times each corner point is paired is counted, and the corner point pairs with the number of occurrences greater than or equal to the preset matching number are retained. The retained corner point pairs will be used for fragment splicing, and this method can improve the accuracy of fragment splicing.

[0177] In some alternative embodiments, after determining the matching points, it further includes:

[0178] Select three first corner points and three corresponding second corner points from the matching points;

[0179] Determine the first angle, first direction between the three first corner points, and the second angle, second direction between the three second corner points;

[0180] Calculate the angle difference based on the first angle and the second angle;

[0181] When the angle difference satisfies the preset range, determine whether the first fragment image or the second fragment image needs to be flipped based on the first direction and the second direction.

[0182] If two fragments are to be joined, it is necessary to ensure that both fragments are on the front or back side at the same time. However, some fragments have no obvious texture features on their surfaces and cannot be distinguished. If the two fragments are in the reverse direction, the fragments need to be flipped by 180 degrees before the fragment joining can continue. In this embodiment, first, it is necessary to determine whether the two fragments are in the same direction. The method is as follows:

[0183] Among the filtered matching corner points, randomly select 3 points to form two sets of matching triangles, as shown in Figure 7 The specific method is as follows:

[0184] Suppose the coordinates of the matching triangle of fragment one are: C1(x1,y1), C2(x2,y2), C3(x3,y3);

[0185] Suppose the coordinates of the matching triangle of fragment two are: P1(z1,w1), P2(z2,w2), P3(z3,w3);

[0186] Due to the flipping of the image, although the corresponding triangles can still be matched, it cannot be visually judged. Therefore, it is necessary to judge whether the two sets of fragments are in the same or reverse direction based on the positive or negative of the included angle of the vectors formed by the corresponding points between the two fragments. The specific judgment steps are as follows:

[0187] Fragment one:

[0188] C1C2 = (x2 - x1, y2 - y1)

[0189] C2C3 = (x3 - x2, y3 - y2)

[0190] C3C1 = (x1 - x3, y1 - y3);

[0191] Fragment two:

[0192] P1P2 = (z2 - z1, w2 - w1)

[0193] P2P3 = (z3 - z2, w3 - w2)

[0194] P3P1 = (z1 - z3, w1 - w3);

[0195] Through the calculation formulas of the dot product, cross product and cosine value in the relevant library functions of python, calculate the angle and direction between each vector:

[0196] Fragment one: Angle1, Dir1, Angle2, Dir2, Angle3, Dir3;

[0197] Fragment two: Angle4, Dir4, Angle5, Dir5, Angle6, Dir6;

[0198] Angle mark = abs(Angle1 - Angle4) + abs(Angle2 - Angle5) + abs(Angle3 - Angle6).

[0199] First, determine whether the angle difference is within an allowable range. Assume the preset range is Angle mark <5. If satisfied, further illustrate that the matching points between the two fragments are correct. However, at this time, the direction of the angle needs to be used to determine whether the two fragments are flipped. The judgment basis is as follows:

[0200] Dir > 0: The vector angle is in the clockwise direction;

[0201] Dir < 0: The vector angle is in the clockwise direction;

[0202] Dir = 0: The vector angle is collinear;

[0203] At this time, according to the directions in Fragment 1 and Fragment 2, if the vector angles are in the same direction, it means that the two fragments are in the same direction; on the contrary, if the vector angles are in the opposite direction, it means that the two fragments are in the opposite direction.

[0204] If the two fragments are in the same direction, no processing is required. However, if the two fragments are in the opposite direction, Fragment 2 needs to be flipped 180 degrees along the perpendicular bisector so that Fragment 2 and Fragment 1 become in the same direction. The original corner points are also transformed by flipping. Subtract the x-axis of the corner points of Fragment 2 from the lengths of the fragment images respectively to obtain the new corner point coordinates of the flipped image, and there is no need to re-detect.

[0205] In this embodiment, randomly select three of the retained corner point pairs to form a new triangle pairing. If the included angle directions between the vectors formed by the three points are the same, it means that both fragments are on the front or the back; if the included angle directions are opposite, it means that one fragment is on the front and the other is on the back. If they are in the opposite direction, the second fragment needs to be flipped 180 degrees along the perpendicular bisector, and its corner points are also transformed accordingly. Through vector included angle calculation and direction judgment, it can accurately determine whether the two fragments are in the same direction, with a high judgment accuracy rate, and can effectively avoid splicing errors caused by fragment reversal.

[0206] In some alternative embodiments, after determining the matching points, it further includes:

[0207] Select three first corner points and three second triangular points corresponding to the first corner points from the matching points; among them, the maximum index difference of the three selected first corner points needs to meet the preset index value;

[0208] Determine the first distance formed between three first corner points and the first side vector; determine the second distance formed between three second triangular points and the second side vector;

[0209] Based on the first side vector and the second side vector, determine the angle between each vector and the horizontal line;

[0210] Based on the first distance and the second distance, determine the distance ratio between the first distance and the second distance;

[0211] Determine whether the distance ratio meets the preset condition;

[0212] When the preset condition is met, based on the angle and the distance ratio, calculate the average rotation angle and the average scaling ratio of the first fragment image relative to the second fragment image or the second fragment image relative to the first fragment image.

[0213] In the previous embodiment, the matching corner points were further screened. Although the matching corner points found at this time are very accurate, the extremely low-probability incorrect matching corner points still need to be excluded. Since the transformation matrix finally needs to be obtained using the corner point pairs of the two fragments, it is necessary to further select the best from the screened corner point pairs again to ensure the correctness of the matching.

[0214] To calculate the accurate angle and ratio relationship between the two fragments, in this embodiment, these matching points need to be randomly selected again, and it is ensured that the index differences of the randomly selected corner points are large, that is, the distances between the three selected corner points are as far as possible. There are two considerations here. One is that all the previous calculations are for the various parameters between adjacent corner points. If there is a defect in the middle of the splicing contour line, it can be judged whether both sides of the defective part all come from the second fragment through the triangle parameters formed by the corner points on both sides of the defective part; the other is that using the paired points with a larger distance to calculate the angle and ratio relationship between the two fragments results in a smaller error.

[0215] The specific method is as follows:

[0216] Suppose there are six corner points G3, G4, G5, G6, G7, G7 of the first fragment and six corner points H3, H4, H5, H6, H7, H8 of the second fragment as matching points, and the rest of the corner points are non-matching points.

[0217] Suppose the first pair of matching points drawn is: (G3, H3), (G4, H4), (G5, H5). At this time, the index order is 3, 4, 5 in sequence, and their maximum index difference is 2, which does not meet the condition, and the matching point pairs must be randomly selected again. Suppose the re-selected ones are (G3, G5), (G5, G8), (G5, G8), and the indices are 3, 5, 8 in sequence, and their maximum index difference is 5, which meets the condition. At this moment, these three pairs of matching points are used to calculate the rotation and scaling relationship between the two fragments.

[0218] For the first fragment, the sides corresponding to the three points G3, G5, and G8 form vectors G3, G5, G5, G8, and G5, G8. For the second fragment, the sides corresponding to the three points H3, H5, and H8 form vectors H3, H5, H5, H8, and H5, H8. At this time, they are three groups of vectors. To calculate the angles, an initial axis is required. In this embodiment, the right semi-axis of the horizontal line of the image can be used as the starting angle, with reference to Figure 8 As shown, use the math library of the python programming language for calculation.

[0219] 1) Angle calculation:

[0220] Calculate the x and y components of the vector:

[0221] d x = x2 - x1

[0222] d y = y2 - y1;

[0223] Furthermore, use the atan2 function to calculate the included angle (in radians):

[0224] angle rad = math.atan2(dy, dx);

[0225] 2) Distance ratio calculation:

[0226] The distance calculation formula is: D = ((x2 - x1) 2 + (y2 - y1) 2 ) 0.5 ;

[0227] Finally, the angles formed by the three vectors formed by the three corner points of each of the two fragments with the horizontal axis are:

[0228] P (G3,G5) ,P (G5,G8) ,P (G5,G8) ; Q (H3,H5) ,Q (H5,H8) ,Q (H8,H3) .

[0229] The distance ratios are: R ((G3,G5),(H3,H5)) ,R ((G3,G5),(H5,H8)) ,R ((G3,G5),(H8,H3)) .

[0230] At this time, it is necessary to ensure that the ratio between the maximum and minimum of the three side lengths does not differ too much. Otherwise, the corner point pair is considered problematic and new corner point triangles need to be randomly selected again. If the ratio of the side lengths has a small difference, that is, it meets the preset conditions, it is considered within the allowable error range. At this time, calculate the average rotation angle and average scaling ratio of the second fragment relative to the first fragment. The calculation formulas are as follows:

[0231] Average rotation angle: S mean = ((P (G3,G5) - Q (H3,H5) ) + (P (G5,G8) - Q (H5,H8) ) + (P (G5,G8) - Q (H8,H5) )) * 1 / 3;

[0232] Average scaling ratio: R mean = (R ((G3,G5),(H3,H5)) + R ((G3,G5),(H5,H8)) + R ((G3,G5),(H8,H3)) ) * 1 / 3.

[0233] In this embodiment, the calculated errors of the determined rotation angle and scaling ratio are small, and the rotation and scaling processing of the second fragment is precise, which can effectively ensure that the processed second fragment is highly consistent with the first fragment in terms of angle and ratio, achieving seamless splicing.

[0234] In some alternative embodiments, after determining the average rotation angle and the average scaling ratio, it further includes:

[0235] Determine the image to be transformed, where the image to be transformed is the first fragment image or the second fragment image;

[0236] Increase the pixel area of the image to be transformed;

[0237] Determine the center point of the image to be transformed and use the center point as the rotation center;

[0238] Calculate the rotation matrix based on the average rotation angle;

[0239] Transform the image to be transformed based on the rotation matrix and the average scaling ratio;

[0240] Calculate the matching corner point offset based on the rotation matrix and the average scaling;

[0241] Update the corner point coordinates in the corner point sequence set based on the matching corner point offset.

[0242] In this embodiment, still taking the second fragment that needs to be processed as an example, the second fragment is processed according to the found rotation angle and ratio relationship between the fragments.

[0243] Before rotating and scaling the image, it is necessary to add white areas to the top, bottom, left, and right of the image, that is, add background areas, to prevent the image size of the fragmented part from disappearing after rotation and affecting subsequent stitching. In this embodiment, there are two methods for the size of the added white area: one is to observe the size of the fragmented area with the human eye and select a suitable angle; the other method is to select the distance of the longest straight line according to the corner point positions of the top, bottom, left, and right of the fragment, and then estimate according to the rotation angle. Both methods are acceptable. Experiments have shown that when the pixel value of the white area added to the four sides of the image is very large, it will not affect the final stitching of the image. In this embodiment, a fixed value of 150 can be selected. It should be noted that if 150 pixels of white area are added to the top, bottom, left, and right of the image respectively, the corresponding corner point coordinates also need to be increased by 150.

[0244] The specific steps for image rotation and scaling are as follows:

[0245] 1) Obtain the center point of the image. In this embodiment, the center of the image is used as the rotation center:

[0246] (h, w) = image.shape[:2];

[0247] Calculate the rotation center: Center = (w / 2, h / 2);

[0248] 2) Calculate the rotation matrix, rotating counterclockwise by degrees:

[0249] M = cv2.getRotationMatrix2D(center, S(mean), S(mean));

[0250] 3) Apply the rotation matrix to the coordinate points:

[0251] Rota_points = cv2.transform(cords.reshape(1, -1, 2), M).reshape(-1, 2);

[0252] 4) Apply the rotation matrix to the coordinate points:

[0253] Rota_image = cv2.warpAffine(image, M, (w, h));

[0254] After the above steps, according to the relative angle and scaling ratio between the two fragments, the new image obtained is referred to Figure 9As shown. However, there is still a positional offset, specifically on the x-axis and y-axis. Therefore, it is necessary to further use the corresponding corner points to calculate their offsets. Since the corner points on Fragment 2 have been rotated and scaled, they cannot be directly involved in the calculation. So, the corner points on Fragment 2 need to be rotated and scaled to obtain the new corner point coordinate positions.

[0255] Suppose the corner point coordinates of Fragment 1 are: G3(x3, y3), G5(x5, y5), G8(x8, y8);

[0256] The corner point coordinates of Fragment 2 are H3(z3, w3), H5(z5, w5), H8(z8, w8); Since a white area was added to Fragment 2 before, the corner point coordinates of Fragment 2 here are all increased by 150 accordingly, and at the same time, the same rotation and scaling changes as the image were performed. The formula for calculating the offset of the matching corner points of the two fragments is as follows:

[0257] offset (x) = ((x 3- z3) + (x 5- z5) + (x 8- z8)) / 3;

[0258] offset (y) = ((y 3- w3) + (y 5- w5) + (y 8- w8)) / 3.

[0259] For the two fragments after rotation and scaling transformation, by using the obtained coordinate offsets, add the offsets of the top, bottom, left, and right of the two fragments respectively, so that the positions of the corresponding corner points of the two fragments are the same, and at the same time, the overall image sizes are the same, which is convenient for subsequent stitching using the image fusion method.

[0260] The specific steps are as follows:

[0261] Suppose the width and height of Fragment 1 are (w1, h1), and the width and height of Fragment 2 are (w2, h2). If the offset on the x or y axis is positive, then add the corresponding offset offset(x) or offset(y) to the x or y axis on Fragment 2; if the offset on the x or y axis is negative, then add the corresponding offset offset(x) or offset(y) to the x or y axis on Fragment 1. Suppose the final widths and heights of Fragment 1 and Fragment 2 are (w3, h3) and (w4, h4) respectively, and finally make the corresponding paired corner point coordinates of Fragment 1 and Fragment 2 consistent.

[0262] After the above operations, the corner coordinates of the two fragments are the same, but their image sizes are generally different, that is, their widths and heights are different. To facilitate subsequent image fusion and stitching, it is necessary to further perform white filling on the lower and right sides of the images, which will not change the corner coordinates. The specific method is as follows:

[0263] w max =(w3, w4)

[0264] h max =(h3, h4)

[0265] offset ( w 3) = w max - w3

[0266] offset ( h 3) = h max - h3

[0267] offset ( w 4) = w max - w4

[0268] offset ( h 4) = h max - h4.

[0269] According to the above, the offsets of the two fragment images can be obtained. By filling white areas on the lower and right sides of the images, not only can the side lengths of the coordinate positions of the corner pairs of the two fragments be made the same, but also the widths and heights of the two fragment images can be kept consistent. The aligned fragment images are shown in reference to Figure 10 as shown.

[0270] After maintaining the conditions of corner and image alignment, average weight fusion can be further used to obtain the averaged fused image. Specifically, the cv2.addWeighted(image_1, 0.5, image_2, 0.5, 0) function is used to achieve the stitching and fusion of the two images, making the stitching line formed by the corners smoother.

[0271] In this embodiment, the images after rotation matrix and scaling ratio transformation are further adjusted for offset, which can make the stitching line formed by the corners smoother and effectively improve the stitching accuracy.

[0272] Refer to Figure 11 as shown, which is the fragment stitching image without rotation and scaling; refer to Figure 12 as shown, which is the rotated fragment stitching image; refer to Figure 13 as shown, which is the scaled fragment stitching image; refer to Figure 14As shown, it is a fragmented mosaic image that has been rotated and scaled; refer to Figure 15 As shown, it is a fragmented mosaic image that has been rotated, scaled, and has some areas missing; refer to Figure 16 As shown, it is a fragmented mosaic image that has been rotated, scaled, has some areas missing, and has been flipped.

[0273] A fragmented mosaic algorithm based on triangle corner points proposed by the present invention does not rely on fragment texture. Even if there are problems such as partial defects in the contour and scale scaling of the fragments, it has good adaptability and generality.

[0274] For the grayscale image and binary image of the fragmented image, the present invention uses the cv2.goodFeaturesToTrack() function to detect the corner points in the edge contour of the grayscale image; uses cv2.findContours() to detect the contour points in the binary image. Since the contour points are ordered and the corner points are disordered, the ordered contour points are used to sort the disordered corner points to obtain ordered corner points. In addition, in order to splice fragments of different scaling ratios, adjacent corner point triangles are used and sorted from small to large to obtain the side length ratio relationship. Angles are not used here because the error is very large when the corner points tend to be on a straight line. At the same time, the triangle corner points can obtain the parameter information between non-adjacent corner points, helping to obtain more matching information with limited corner points. Further, based on the geometric relationship of the corner point triangle, the present invention uses the side length ratio of the corner point triangle for matching; in addition, since the image is a binary image at this time, either black or white, the meshing relationship between the convex hull and concave hull shapes formed by two fragment triangles is also used for matching. Since the number of black background pixels enclosed inside is mutually matched, and considering the relationship of the scaling ratio, the ratio of the number of black and white pixels contained in the convex hull and concave hull shapes to the total area is calculated to remove the influence of image scaling. Through the above geometric relationship and pixel ratio, the matching corner points of the two fragments are screened.

[0275] Due to triangle pairing, there will be duplicate pairing points. The present invention also utilizes this feature to retain the pairing points that appear two or more times. At the same time, according to the index order of the pairing points, the starting point and the ending point of the matching points are further found. This step can further screen the previous matching points to ensure the accuracy of the matching points. For the matching points that have been screened again, the present invention uses the method of random sampling to randomly select three points, and tries to select three points with relatively large distances to reduce the error and at the same time detect whether the corner points on both sides of the defective part come from the same fragment. Then, according to the coordinates of the corner points with relatively large distances on the two fragments, with the right half-axis of the image horizontal as the starting point of the corner points, the rotation angle of each corner point on the two fragments relative to the starting point is calculated respectively, and the corresponding side length ratio is obtained at the same time. Compared with the first fragment, according to the obtained rotation angle and scaling ratio, the second fragment is rotated and scaled, and the original corner points are also transformed in the same way. According to the offset amount of the corresponding corner points of the first fragment and the second fragment after transformation, the two images are filled with white areas up, down, left and right, so that finally the first fragment and the second fragment are obtained, and the corner point coordinate positions are the same, and the height and width of the whole image are also the same. Only when the two are aligned can the fusion splicing method be used to make the splicing line of the two fragments smoother.

[0276] In this embodiment, a two-dimensional fragment splicing device is also provided. This device is used to implement the above-mentioned embodiments and preferred implementation manners, and those that have been described will not be repeated. As used hereinafter, the term "module" can be a combination of software and / or hardware that can achieve a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible and contemplated.

[0277] This embodiment provides a two-dimensional fragment splicing device, as Figure 17 shown, the device includes:

[0278] An acquisition module 201, configured to acquire a first fragment image and a second fragment image, wherein both the first fragment image and the second fragment image are binary images;

[0279] A detection module 202, configured to detect the first fragment image and the second fragment image, and respectively generate an ordered set of corner point sequences corresponding to the fragment contours. The set of corner point sequences includes the corner point coordinates corresponding to each corner point one by one;

[0280] A construction module 203, configured to construct a triangle with each three adjacent corner points as vertices based on the set of corner point sequences;

[0281] A determination module 204, configured to determine a set of side length ratios of each triangle, and generate a first triangle sequence set corresponding to the first fragment image and a second triangle sequence set corresponding to the second fragment image; wherein, the side length ratio is the ratio between adjacent side lengths in a triangle.

[0282] A splicing module 205, configured to splice the first fragment image and the second fragment image based on the first triangle sequence set and the second triangle sequence set.

[0283] The two-dimensional fragment splicing device in this embodiment is presented in the form of functional units. Here, the unit refers to an ASIC circuit, a processor and a memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0284] The further function descriptions of the above-mentioned various modules and units are the same as those in the corresponding embodiments above, and will not be elaborated here.

[0285] An embodiment of the present invention further provides a computer device having the above Figure 17 shown two-dimensional fragment splicing device.

[0286] Please refer to Figure 18 , Figure 18 which is a schematic structural diagram of a computer device provided by an optional embodiment of the present invention. As shown in Figure 18 , the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including a high-speed interface and a low-speed interface. Each component communicates with each other using different buses and can be installed on a common motherboard or installed in other ways as needed. The processor can process instructions executed within the computer device, including instructions stored in the memory or on the memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (such as an array of servers, a set of blade servers, or a multi-processor system). Figure 18 One processor 10 is taken as an example in

[0287] The processor 10 can be a central processor, a network processor, or a combination thereof. Among them, the processor 10 can further include a hardware chip. The above hardware chip can be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The above programmable logic device can be a complex programmable logic device, a field programmable gate array, a generic array logic, or any combination thereof.

[0288] Among them, the memory 20 stores instructions that can be executed by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiments.

[0289] The memory 20 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system and application programs required for at least one function; the data storage area can store data created according to the use of the computer device and the like. In addition, the memory 20 may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some alternative embodiments, the memory 20 may optionally include a memory remotely provided with respect to the processor 10, and these remote memories can be connected to the computer device through a network. Examples of the above network include but are not limited to the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0290] The memory 20 may include volatile memory, for example, random access memory; the memory may also include non-volatile memory, for example, flash memory, a hard disk, or a solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0291] The computer device further includes a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0292] The embodiments of the present invention also provide a computer-readable storage medium. The method according to the embodiments of the present invention can be implemented in hardware, firmware, or be implemented as computer code that can be recorded on a storage medium, or be implemented as computer code originally stored in a remote storage medium or a non-transitory machine-readable storage medium and downloaded through a network and will be stored in a local storage medium, so that the method described herein can be stored in such software processed on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk, or a solid-state drive, etc.; further, the storage medium may also include a combination of the above types of memory. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code, and when the software or computer code is accessed and executed by the computer, the processor, or the hardware, the method shown in the above embodiments is implemented.

[0293] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A two-dimensional fragment splicing method, characterized in that, The method includes: Obtain a first fragmented image and a second fragmented image, where both the first fragmented image and the second fragmented image are binary images; Detect the first fragmented image and the second fragmented image, and respectively generate an ordered set of corner point sequences corresponding to the fragmented contours. The set of corner point sequences includes corner point coordinates corresponding to each corner point one by one; Based on the set of corner point sequences, with the corner points as vertices, construct a triangle for every three adjacent corner points; Determine the set of side length ratios of each triangle, and generate a first triangle sequence set corresponding to the first fragmented image and a second triangle sequence set corresponding to the second fragmented image; where the side length ratio is the ratio between adjacent side lengths in the triangle; Stitch the first fragmented image and the second fragmented image based on the first triangle sequence set and the second triangle sequence set.

2. The method according to claim 1, wherein The stitching of the first fragmented image and the second fragmented image based on the first triangle sequence set and the second triangle sequence set includes: Determine the set of triangle ratio differences, where the triangle ratio difference is the difference in side length ratios between the set of side length ratios in the first triangle sequence set and the set of side length ratios in the second triangle sequence set; Based on the set of triangle ratio differences, determine the first triangle and the second triangle corresponding to the minimum triangle ratio difference; Judge whether the first triangle and the second triangle meet the preset pixel condition; When the preset pixel condition is met, determine the first corner point corresponding to the first triangle and the second corner point corresponding to the second triangle; the first corner point and the second corner point are used to stitch the first fragmented image and the second fragmented image.

3. The method according to claim 2, wherein The judgment of whether the first triangle and the second triangle meet the preset pixel condition includes: Determine the first ratio of the number of first preset color pixels in the first triangle to the total number of pixels in the first triangle; Determine the second ratio of the number of second preset color pixels in the second triangle to the total number of pixels in the second triangle; Wherein, when the first preset color is black, the second preset color is white; when the first preset color is white, the second preset color is black; Determine the matching degree between the first ratio and the second ratio; When the matching degree meets the preset matching degree, determine that the first triangle and the second triangle meet the preset pixel condition.

4. The method according to claim 1, wherein The stitching of the first fragmented image and the second fragmented image based on the first triangle sequence set and the second triangle sequence set includes: Determine the set of triangle ratio differences, where the triangle ratio difference is the difference in side length ratios between the set of side length ratios in the first triangle sequence set and the set of side length ratios in the second triangle sequence set; Based on the set of triangular ratio differences, determine triangle pairs that meet a preset ratio difference threshold, where the triangle pairs include a first triangle corresponding to the first fragmented image and a second triangle corresponding to the second fragmented image; Determine whether the first triangle and the second triangle in the triangle pair meet a preset pixel condition; When the preset pixel condition is met, determine a first corner point corresponding to the first triangle and a second corner point corresponding to the second triangle; Regenerate all the first corner points into a new first corner point sequence and regenerate all the second corner points into a new second corner point sequence; Based on the first corner point sequence, construct a triangle for every three adjacent corner points to generate a new set of first triangles; Based on the second corner point sequence, construct a triangle for every three adjacent corner points to generate a new set of second triangles; Based on the first triangle set and the second triangle set, determine the matching times of the first corner point and the second corner point; Determine whether the matching times of the first corner point or the second corner point meet a preset matching times; When the preset matching times is met, determine the corresponding first corner point or second corner point as a matching point, and the matching point is used to splice the first fragmented image and the second fragmented image.

5. The method according to claim 4, characterized in that, After determining the matching points, it further includes: Select three of the first corner points and three second triangular points corresponding to the first corner points from the matching points; Determine a first angle and a first direction among the three first corner points and a second angle and a second direction among the three second triangular points; Calculate an angle difference based on the first angle and the second angle; When the angle difference meets a preset range, determine whether the first fragmented image or the second fragmented image needs to be flipped based on the first direction and the second direction.

6. The method according to claim 4, wherein After determining the matching points, it further includes: Select three of the first corner points and three second triangular points corresponding to the first corner points from the matching points; wherein, the maximum index difference of the selected three first corner points needs to meet a preset index value; Determine a first distance and a first side vector formed among the three first corner points; determine a second distance and a second side vector formed among the three second triangular points; Based on the first side vector and the second side vector, determine the angle between each vector and the horizontal line; Based on the first distance and the second distance, determine the distance ratio between the first distance and the second distance; Determine whether the distance ratio meets a preset condition; When the preset condition is met, calculate the average rotation angle and the average scaling ratio of the first fragmented image relative to the second fragmented image or the second fragmented image relative to the first fragmented image based on the angle and the distance ratio.

7. The method according to claim 6, wherein After determining the average rotation angle and the average scaling ratio, it further includes: Determine the image to be transformed, and the image to be transformed is the first fragmented image or the second fragmented image; Increase the pixel area of the image to be transformed; Determine the center point of the image to be transformed, and use the center point as the rotation center; Calculate the rotation matrix based on the average rotation angle; Transform the image to be transformed based on the rotation matrix and the average scaling ratio; Calculate the matching corner point offset based on the rotation matrix and the average scaling ratio; Update the corner point coordinates in the corner point sequence set based on the matching corner point offset.

8. A two-dimensional fragment splicing device, characterized in that, The device includes: An acquisition module, configured to acquire a first fragment image and a second fragment image, where both the first fragment image and the second fragment image are binary images; A detection module, configured to detect the first fragment image and the second fragment image, and respectively generate an ordered set of corner point sequences corresponding to the fragment contours, where the set of corner point sequences includes corner point coordinates corresponding to each corner point one by one; A construction module, configured to construct a triangle with every three adjacent corner points as vertices based on the set of corner point sequences; A determination module, configured to determine the set of side length ratios of each triangle, and generate a first triangle sequence set corresponding to the first fragment image and a second triangle sequence set corresponding to the second fragment image; where the side length ratio is the ratio between adjacent side lengths in the triangle; A splicing module, configured to splice the first fragment image and the second fragment image based on the first triangle sequence set and the second triangle sequence set.

9. A computer device, characterized in that, Includes: A memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to execute the two-dimensional fragment splicing method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions, and the computer instructions are used to cause the computer to execute the two-dimensional fragment splicing method according to any one of claims 1-7.

Images