A method and system for generating panoramic images of cylindrical murals

Abstract

The present invention proposes a method for generating a panoramic image of a cylindrical mural, comprising the following steps: Step 1: using a three-dimensional reconstruction method to restore the original three-dimensional scene of the target object; Step 2: reading and processing the data in the PLY file; Step 3: calculating the parameter information of the ideal cylindrical surface closest to the cylindrical surface through a sampling consistency algorithm; Step 4: mapping the model to the ideal cylindrical surface according to the obtained parameter information; Step 5: segmenting the triangular mesh of the closed model and unfolding it to a plane; Step 6: finally projecting it into a panoramic digital image stored in a BigTIFF file. The present invention solves the problem that the traditional method can only handle the preservation of planar murals but cannot preserve murals on curved surfaces such as cylinders.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer digital image processing, and specifically relates to computer vision, and specifically includes a method for generating a three-dimensional cylindrical image by stitching images of cave murals captured by a camera and expanding the three-dimensional cylindrical image. Background Art

[0002] Cave paintings are a cultural treasure of mankind, but due to their immovability, it is not easy to preserve them in a mobile form. Since the last century, cultural relics protection personnel can only prevent the paintings from being further eroded by nature through rescue restoration. At the same time, cultural relics protection personnel also use copying as a means of backup preservation. This method requires extremely high technical skills of the staff and is time-consuming. In recent years, with the development of science and technology, the use of computers for digital preservation has been used in cultural relics protection work. According to the data indicators released by the State Administration of Cultural Heritage [1], the sampling resolution of the first-level orthophoto map cannot be lower than 300DPI. It is impossible to find a camera with such a high resolution to take a panoramic image. Therefore, the idea of ​​image stitching was introduced into the protection of cave paintings. Image stitching refers to the process of collecting source images, dividing them into blocks, and finally obtaining a set of images with overlapping regional relationships, and stitching them together. According to research and investigation, many practitioners still use manual stitching, which requires the staff to enlarge the area and compare the stitching pixel by pixel. Even so, there will still be obvious stitching seams, and it is still time-consuming and laborious. Therefore, how to use computers to automatically perform recognition and splicing tasks is a meaningful research direction.

[0003] Image stitching can be divided into two categories: one is stitching in two-dimensional image space; the other is stitching in three-dimensional image space. Nowadays, the two-dimensional image stitching method is widely used in the field of panorama generation. The earliest image registration algorithm based on Fourier-Mellin[2] transform was adopted, followed by image matching method based on phase correlation angle[3], and SIFT[4] features were widely used in image stitching. The method of stitching in three-dimensional image space is mainly based on three-dimensional reconstruction, such as the mural digitization framework based on three-dimensional reconstruction proposed in reference[5], the stele digital rubbing generation system proposed in reference[6], and the three-dimensional reconstruction algorithm based on non-Lambertian photometric stereo proposed in reference[7].

[0004] However, the above methods are unable to generate panoramic images of large cylindrical murals. Therefore, the uniqueness of the present invention is that it can generate panoramic images of the texture of cylindrical murals. Summary of the invention

[0005] The method for generating a panoramic image of a cylindrical mural proposed in the present invention processes a captured image set, extracts image feature points using SIFT, uses CHNN to match feature points, uses incremental SFM to restore sparse point cloud scenes, uses depth maps to fuse densified point clouds, performs three-dimensional surface modeling through visual information, performs texture mapping based on Markov random fields, and uses a maximum likelihood consistency algorithm to fit and calculate parameter information of an ideal cylindrical surface that is closest to an actual cylindrical surface. The parameter information is then used to map the actual cylindrical surface to an ideal cylindrical surface through the method proposed in the present invention, and then the mapping method proposed in the present invention is used to unfold the ideal cylindrical surface into a three-dimensional plane, and finally a panoramic image is obtained using the projection method proposed in the present invention.

[0006] The method for generating a panoramic image of a cylindrical mural of the present invention comprises the following steps:

[0007] Step 1: Use a 3D reconstruction method to restore the original 3D scene of the target object;

[0008] Step 2: Read the data in the PLY file and process it;

[0009] Step 3: Calculate the ideal cylindrical surface parameter information closest to the cylindrical surface through a sampling consistency algorithm;

[0010] Step 4: Map the model onto the ideal cylindrical surface according to the obtained parameters;

[0011] Step 5: Use the corresponding algorithm to segment the triangular mesh of the closed model and unfold it onto the plane;

[0012] Step 6: Finally, the panoramic digital image is projected and stored in a BigTIFF file.

[0013] The cylindrical mural panoramic image of the present invention is an image that can reflect the original texture detail information of the mural.

[0014] The processing in step 1 of the present invention is as follows: firstly, the captured image set is processed, image feature points are extracted using SIFT, feature point matching is performed using CHNN[8], sparse point cloud scene is restored using incremental SFM[9], point cloud is densified using depth map fusion

[10] , three-dimensional surface modeling is performed using visual information

[11] , texture mapping is performed based on Markov random field

[12] , and finally a three-dimensional texture image is obtained. The method described in step 1 is secondary development based on the existing algorithm, which is for code integration and code development of the original open source support library.

[0015] The three-dimensional texture image is the restored original three-dimensional scene.

[0016] The processing in step 2 of the present invention is:

[0017] The data of a 3D image saved in PLY format is divided into three types: ① 3D point coordinates, ② texture coordinates, and ③ triangle patch relationship index. The 3D point coordinates are defined as:

[0018] P=(x,y,z)

[0019] Where x is the position of the P coordinate on the x-axis; y is the position of the P coordinate on the y-axis; z is the position of the P coordinate on the z-axis. The texture coordinates are the coordinates of the corresponding triangle patch on the texture image, which are defined as:

[0020] T=(u,v)

[0021] Among them, u is the u-axis value of T in the texture coordinate system; v is the v-axis value of T in the texture coordinate system.

[0022] The triangle patch relationship index includes three three-dimensional vertex coordinates and three texture coordinates, which are defined as:

[0023] I=(P1,P2,P3,T1,T2,T3)

[0024] Among them, P1 is the coordinate of the first three-dimensional point; P2 is the coordinate of the second three-dimensional point; P3 is the coordinate of the third three-dimensional point; T1 is the coordinate of the first texture point; T2 is the coordinate of the second texture point; T3 is the coordinate of the third texture point.

[0025] Read the data into memory and confirm the file integrity;

[0026] The three-dimensional image is an actual cylindrical three-dimensional image generated by three-dimensional reconstruction, which is a cylindrical three-dimensional image with an irregular surface.

[0027] The processing in step 3 of the present invention is: calculating parameter information of an ideal cylindrical 3D image that is closest to the input actual cylindrical 3D image; the parameter information includes the radius and axis information of the cylindrical surface, and an ideal cylindrical surface can be determined by using these contents, which is defined as:

[0028] D=(x0,y0,z0,l,m,n,r)

[0029] Among them, any point on the axis (x0, y0, z0) together with the axis direction (l, m, n) can determine an axis, and r is the radius.

[0030] The processing in step 4 is: coordinate transformation of the actual cylindrical surface and the ideal cylindrical surface is performed so that the actual cylindrical surface I cylinder The central axis of axis Its reference point P0 (x0, y0, z0) coincides with the Z axis and the origin O of the world coordinate system respectively. The central axis l is calculated by the following formula axis Direction vector The angle θ with the z-axis and perpendicular to and A vector of is the unit vector of the Z axis, and the calculation formula is:

[0031]

[0032]

[0033] The rotation matrix R calculated according to the Rodriguez rotation formula rotation and the translation matrix R translation , where the translation matrix is ​​the offset from the axis to the origin of the coordinate system. The relationship between the coordinates of the actual cylindrical surface before and after the transformation can be obtained, as shown in the formula:

[0034] I' cylinder =R rotation ×R translation ×I cylinder

[0035]

[0036] Where γ represents the value of (1-cosθ); where p x 、p y and p z Respectively, vector The x, y, and z coordinate components of .

[0037] Then, according to the matrix after coordinate transformation, we can find the mapping relationship between the actual cylindrical surface and the points on the ideal cylindrical surface, which can be solved by two formulas:

[0038]

[0039] x' 2 +y' 2 =r 2

[0040] The equations of the straight line connecting the point on the actual cylindrical surface and the intersection of the plane perpendicular to the axis and the axis, and the equation of the circle perpendicular to the axis where the point on the ideal cylindrical surface is located; y' is the y coordinate of the mapping point on the ideal cylindrical surface, and x' is the x coordinate of the mapping point on the ideal cylindrical surface. The solution to the equation can map the actual cylindrical surface to the ideal cylindrical surface.

[0041] The processing in step 5 is as follows: The purpose of this part is to find a plane tangent to the cylinder and to match the points on the plane with the points on the cylinder one by one. The relationship conforms to the following formula:

[0042]

[0043] Among them, α is and The angle between two vectors; O circle is the center of the circle on the circular plane perpendicular to the coordinate axis where point P is located; Q is the intersection of the circular plane and the tangent line, and the tangent line is the intersection of the plane tangent to the cylinder and the cylindrical surface; x P' is the x coordinate of the mapping point P' on the three-dimensional plane, y P' is the y coordinate of the mapping point P' on the three-dimensional plane, z P is the z coordinate of point p on the ideal cylindrical surface, z P' is the z coordinate of the mapping point P' on the three-dimensional plane.

[0044] After calculation, a plane in three-dimensional space can be obtained, on which the original cylindrical shape is retained, which is the relationship of the triangular facets.

[0045] The processing in step 6 is as follows: first, the points on the three-dimensional space plane image are reduced to two-dimensional points using the principal component analysis PCA (Principal component analysis) algorithm; the texture is projected onto an image with 32-bit floating-point data as the unit of a single triangular mesh, and the image is still continuous in the mathematical sense; finally, the image is sampled with an image of actual resolution size and eight-bit integer as the data type to obtain a data image, i.e., a panoramic image.

[0046] The BigTIFF format is an uncompressed storage format capable of storing images larger than 4 GB in size.

[0047] Based on the above method, the present invention also proposes a cylindrical mural panoramic image generation system, including: a memory and a processor; a computer program is stored in the memory, and when the computer program is executed by the processor, the above method is implemented.

[0048] The innovation of the present invention is that it can process cave murals located on a cylindrical surface and save the cylindrical murals as two-dimensional digital images through calculation, thus solving the problem that the traditional method can only process the preservation of flat murals but cannot preserve murals on curved surfaces such as cylinders. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a graphical analysis of the parameters of the cylinder equation in the present invention.

[0050] Figure 2 Illustration of how the cylinder rotation parameters are calculated.

[0051] Figure 3 This is a diagram showing the mapping of points on the actual cylindrical surface to the ideal cylindrical surface.

[0052] Figure 4Illustration of the method of unfolding an ideal cylindrical surface into a three-dimensional plane.

[0053] Figure 5 A projection generation method for panoramic images.

[0054] Figure 6 Generate a flow chart for panoramic images DETAILED DESCRIPTION

[0055] The invention is further described in detail with reference to the following specific implementation examples and drawings. The process, conditions, experimental methods, etc. for implementing the invention, except for the contents specifically mentioned below, are common knowledge and common common sense in the art and are not particularly limited by the present invention.

[0056] The present invention relates to the field of digital image processing and computer vision, and its specific content is a method for generating a panoramic image of a mural on a cylindrical surface in a cave. The method consists of the following two parts: ① image stitching of the block-collected images of the cave murals taken by the camera generates a three-dimensional cylindrical image, and ② unfolding the three-dimensional cylindrical image. The first part uses multi-view stereo geometry to restore the original three-dimensional information of the target scene object in the image set to obtain a three-dimensional image with a texture pattern. The second part unfolds the three-dimensional cylindrical image, which is a method for unfolding the three-dimensional cylindrical image generated in the previous part, which is divided into four steps: ① calculating the ideal cylindrical surface parameter information closest to the cylindrical surface through a sampling consistency algorithm, ② mapping the model to the ideal cylindrical surface according to the obtained parameters, ③ then segmenting the triangular mesh of the closed model through the corresponding algorithm and unfolding it to the plane, ④ finally projecting it into a panoramic digital image.

[0057] Example

[0058] Step 1: Use a 3D reconstruction method to restore the original 3D scene of the target object;

[0059] Step 2: Read the data in the PLY file and process it;

[0060] Step 3: Calculate the ideal cylindrical surface parameter information closest to the cylindrical surface through a sampling consistency algorithm;

[0061] Step 4: Map the model onto the ideal cylindrical surface according to the obtained parameters;

[0062] Step 5: Use the corresponding algorithm to segment the triangular mesh of the closed model and unfold it onto the plane;

[0063] Step 6: Finally, the panoramic digital image is projected and stored in a BigTIFF file.

[0064] First, we introduce in detail the seven parameter values ​​obtained by the sampling consistency algorithm in step 3. Figure 1As shown, in step 4, the model is first moved to the origin of the coordinate system so that P0 coincides with the origin of the coordinate system. Figure 2 As shown, the rotated model is described as follows Figure 3 The mapping method shown uses parameter values ​​to calculate each three-dimensional point on the cylindrical model, and can map the points on the actual cylindrical surface to the ideal cylindrical surface.

[0065] Then in step 5, the arc length formula of the circle is used to calculate the corresponding relationship mentioned above, and the points on the ideal cylindrical model are mapped to the three-dimensional plane in the space through the corresponding relationship. Finally, in step 6, the obtained plane model is mapped to the image stored in the continuous data format in units of triangular faces, and then sampled to obtain the image in the discrete data format, and stored in the BigTIFF file format.

[0066] References

[0067] [1] State Administration of Cultural Heritage of the People’s Republic of China. WW / T 0082-2017 Technical Specifications for Digital Mapping of Ancient Architectural Murals[S]. Beijing: Cultural Relics Publishing House, 2017.

[0068] [2]Anuta P E.Spatial registration of multispectral and multitemporaldigital imagery using fast Fourier transform techniques[J].IEEE transactionson Geoscience Electronics, 1970,8(4):353-368.

[0069] [3]Kuglin C D.The phase correlation image alignment methed[C] / / Proc.Int.Conference Cybernetics Society.1975:163-165.

[0070] [4]Lowe D G.Distinctive image features from scale-invariant keypoints[J].International journal of computer vision,2004,60(2):91-110.

[0071] [5]Liu Liming. Research on ultra-high resolution mural rapid digitization technology based on 3D reconstruction[D]. Zhejiang University, 2016.

[0072] [6] Zhang Yuanlin. Generation technology and application of digital rubbings of inscriptions based on three-dimensional models[D]. Zhejiang University, 2018.

[0073] [7] Li Min. Research on 3D reconstruction based on non-Lambertian photometric stereo[D]. Zhejiang University, 2020.

[0074] [8]Cheng J, Leng C, Wu J, et al. Fast and accurate image matching with cascade hashing for 3d reconstruction[C] / / Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2014:1-8.

[0075] [9]Snavely N,Seitz SM,Szeliski R.Photo tourism:exploring photocollections in3D.ACM Trans Graph 25(3):835-846[J].ACM Transactions onGraphics,2006,25(3):835-846.

[0076]

[10] Shen S.Accurate multiple view 3d reconstruction using patch-basedstereo for large-scale scenes[J].IEEE transactions on image processing, 2013, 22(5):1901-1914.

[0077]

[11] Jancosek M,Pajdla T.Multi-view reconstruction weakly-supported surfaces[C] / / CVPR 2011.IEEE,2011:3121-3128.

[0078] The protection content of the present invention is not limited to the above embodiments. Without departing from the spirit and scope of the inventive concept, changes and advantages that can be thought of by those skilled in the art are included in the present invention and are protected by the attached claims.

Claims

1. A method for generating a panoramic image of a cylindrical mural, characterized in that: The steps include: Step 1: Use a 3D reconstruction method to restore the original 3D scene of the target object; The 3D reconstruction method in step 1 is to first process the captured image set, extract image feature points using SIFT, match feature points using CHNN, restore sparse point cloud scenes using incremental SFM, densify point clouds using depth map fusion, perform 3D surface modeling through visual information, perform texture mapping based on Markov random field, and finally obtain a 3D texture image; Step 2: Read the data in the PLY file and process it; In the step 2, the data of the three-dimensional image of the PLY file is divided into: three-dimensional point coordinates, texture coordinates, and triangle patch relationship indexes; The three-dimensional point coordinates are defined as: P = (x, y, z) Where x is the position of the P coordinate on the x-axis; y is the position of the P coordinate on the y-axis; z is the position of the P coordinate on the z-axis; The texture coordinates are the coordinates of the corresponding triangle patch on the texture image, which are defined as: T=(u,v) Wherein, u is the u-axis value of T in the texture coordinate system; v is the v-axis value of T in the texture coordinate system; The triangle patch relationship index includes three three-dimensional vertex coordinates and three texture coordinates, which are defined as: I=(P1, P2, P3, T1, T2, T3) Among them, P1 is the coordinate of the first 3D point; P2 is the coordinate of the second 3D point; P3 is the coordinate of the third 3D point; f1 is the coordinate of the first texture point; T2 is the coordinate of the second texture point; T3 is the coordinate of the third texture point; Processing the data includes reading the data into memory and confirming the integrity of the file; The three-dimensional image is an actual cylindrical three-dimensional image generated by three-dimensional reconstruction, which is a cylindrical three-dimensional image with an irregular surface; Step 3: Calculate the ideal cylindrical surface parameter information closest to the cylindrical surface through a sampling consistency algorithm; Step 4: Map the model onto the ideal cylindrical surface according to the obtained parameter information; Step 5: Split the triangular mesh of the closed model and unfold it onto a plane; Step 6: Finally, the panoramic digital image is projected and stored in a BigTIFF file.

2. The method for generating a panoramic image of a cylindrical mural according to claim 1, characterized in that: The step 3 comprises the following steps: Calculate parameter information of an ideal cylindrical three-dimensional image that is closest to the input actual cylindrical three-dimensional image; The parameter information includes the radius and axis information of the cylindrical surface. An ideal cylindrical surface is determined by using the parameter information, which is defined as: D=(x0,y0,z0,l,m,n,r) Among them, any point on the axis (x0, y0, z0) and the axis direction (l, m, n) together determine an axis, and r is the radius.

3. The method for generating a panoramic image of a cylindrical mural according to claim 1, characterized in that: The step 4 comprises the following steps: The coordinates of the actual cylindrical surface and the ideal cylindrical surface are transformed so that the actual cylindrical surface I cylinder The central axis of axis Its reference point P0 (x0, y0, z0) coincides with the Z axis and the origin O of the world coordinate system respectively; The central axis l is calculated by the following formula axis Direction vector The angle θ with the z-axis and perpendicular to and A vector of is the unit vector of the Z axis, and the calculation formula is: The rotation matrix R calculated according to the Rodriguez rotation formula rotation and the translation matrix R translation , where the translation matrix is ​​the offset from the axis to the origin of the coordinate system, and the relationship between the coordinates of the actual cylindrical surface before and after the transformation is obtained, as shown in the formula: I′ cylinder =R rotation ×R translation ×I cylinder Where γ represents the value of (1-cosθ); where p x 、p y and p z Respectively, vector The x, y, and z coordinate components of Then, according to the matrix after coordinate transformation, we find the mapping relationship between the actual cylindrical surface and the points on the ideal cylindrical surface, which can be solved by two formulas: x′ 2 +y′ 2 =r 2 are respectively the equation of the straight line connecting the point on the actual cylindrical surface and the intersection of the plane perpendicular to the axis and the axis, and the equation of the circle perpendicular to the axis where the point on the ideal cylindrical surface is located; where y′ is the y coordinate of the mapping point on the ideal cylindrical surface, and x′ is the x coordinate of the mapping point on the ideal cylindrical surface; The solution to the equation maps the actual cylindrical surface to the ideal cylindrical surface.

4. The method for generating a panoramic image of a cylindrical mural according to claim 1, characterized in that: The step 5 comprises the following steps: Find a plane tangent to the cylinder and match the points on the plane with the points on the cylinder one by one. The relationship conforms to the following formula: Among them, α is and The angle between two vectors; O circle is the center of the circle on the circular plane perpendicular to the coordinate axis where point P is located; Q is the intersection of the circular plane and the tangent line, and the tangent line is the intersection of the plane tangent to the cylinder and the cylindrical surface; x P′ is the x coordinate of the mapping point P′ on the three-dimensional plane, y P′ is the y coordinate of the mapping point P′ on the three-dimensional plane, z P is the z coordinate of point p on the ideal cylindrical surface, z P′ is the z coordinate of the mapping point P′ on the three-dimensional plane; After calculation, a plane in three-dimensional space can be obtained, on which the triangular face relationship of the original cylindrical shape is retained.

5. The method for generating a panoramic image of a cylindrical mural according to claim 1, characterized in that: The step 6 comprises the following steps: First, the points on the three-dimensional space plane image are reduced to two-dimensional points using the principal component analysis PCA algorithm; Project the texture onto an image of 32-bit floating-point data in units of a single triangle mesh. At this time, the image is still continuous in the mathematical sense. Finally, the image is sampled with the actual resolution size of eight bits as the data type to obtain a data image, namely a panoramic image.

6. The method for generating a panoramic image of a cylindrical mural according to claim 1, characterized in that: The BigTIFF format is an uncompressed storage format capable of storing images larger than 4 GB in size.

7. A cylindrical mural panoramic image generation system, characterized in that: include: Memory and processor; The memory stores a computer program, and when the computer program is executed by the processor, the method according to any one of claims 1 to 6 is implemented.

Images