Image scanning method, device, equipment, and storage medium

Abstract

An image scanning method and apparatus, a device, and a storage medium, wherein the method comprises: acquiring point cloud data of a scanning scene and an initial scanning image of the scanning scene (101); performing surface detection on the point cloud data to obtain initial surface data (102); optimizing the surface represented by the initial surface data to obtain target surface data (103); and correcting the pixel coordinates of pixel points in the initial scanning image according to the three-dimensional coordinates in the target surface data to obtain a target scanning image (104).

Description

Technical Field

[0001] The embodiments of the present application relate to an image processing method, and more particularly to an image scanning method and apparatus, device, and storage medium. Background Art

[0002] Document scanning technology based on image information can be integrated into mobile terminals such as mobile phones, which is easy to carry and use. Document scanning technology based on image information requires texture, boundary and other information to calculate the transformation matrix, so this technology is not suitable for scanning documents without boundaries or with little texture.

[0003] Time of flight (TOF) sensors are not affected by changes in lighting and object textures, and can reduce costs while meeting accuracy requirements. Using TOF's three-dimensional data to assist in obtaining the three-dimensional information of the target object and calculating the transformation matrix can make document scanning independent of image information, thereby greatly improving the scope of application of document scanning.

[0004] However, the current image scanning technology based on TOF data cannot cope with the situation where the target object in the scanning scene is curved. Summary of the invention

[0005] In view of this, embodiments of the present application provide an image scanning method and apparatus, a device, and a storage medium.

[0006] The technical solution of the embodiment of the present application is implemented as follows:

[0007] In a first aspect, an embodiment of the present application provides an image scanning method, the method comprising: acquiring point cloud data of a scanning scene and an initial scanning image of the scanning scene; performing surface detection on the point cloud data to obtain initial surface data; optimizing the surface represented by the initial surface data to obtain target surface data; and correcting the pixel coordinates of pixel points in the initial scanning image according to the three-dimensional coordinates in the target surface data to obtain a target scanning image.

[0008] In a second aspect, an embodiment of the present application provides an image scanning device, comprising: a data acquisition module, used to acquire point cloud data of a scanning scene and an initial scanning image of the scanning scene; a surface detection module, used to perform surface detection on the point cloud data to obtain initial surface data; a surface optimization module, used to optimize the surface represented by the initial surface data to obtain target surface data; and an image correction module, used to correct the pixel coordinates of pixel points in the initial scanning image according to the three-dimensional coordinates in the target surface data to obtain a target scanning image.

[0009] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the steps of any image scanning method described in the embodiment of the present application are implemented.

[0010] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any image scanning method described in the embodiments of the present application.

[0011] In an embodiment of the present application, after obtaining the point cloud data of the scanning scene and the initial scanning image of the scanning scene, the electronic device performs surface detection on the point cloud data, and optimizes the initial surface data obtained by detection to obtain target surface data; in this way, on the one hand, the electronic device can correct the pixel coordinates in the initial scanning image according to the three-dimensional coordinates in the target surface data, so as to obtain a more accurate and detailed scanning result, that is, the target scanning image; on the other hand, compared with plane detection, surface detection on point cloud data can be applied to more scanning scenarios, for example, it can scan the surfaces of cylinders and nearly cylindrical objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is a schematic diagram of the implementation process of the image scanning method of the embodiment of the present application;

[0013] Figure 2 A schematic diagram showing the comparison between the initial scan image and the target scan image in an embodiment of the present application;

[0014] Figure 3 This is a schematic diagram of the column characteristics detected by the embodiment of the present application;

[0015] Figure 4 This is a schematic diagram of the mesh division results of the embodiment of the present application;

[0016] Figure 5 A reference diagram of an embodiment of the present application;

[0017] Figure 6 This is a schematic diagram of radial optimization based on a cubic bounding box according to an embodiment of the present application;

[0018] Figure 7 Schematic diagram of radial optimization based on Poisson Surface Reconstruction (PSR) according to an embodiment of the present application;

[0019] Figure 8 This is a schematic diagram of radial optimization based on dual-camera data fusion in an embodiment of the present application;

[0020] Fig. 9This is a schematic diagram of the implementation flow of another image scanning method according to an embodiment of the present application;

[0021] Fig.10 This is a schematic diagram of the structure of an image scanning device according to an embodiment of the present application;

[0022] Fig.11 A hardware entity schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the specific technical solution of the present application will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application, but are not used to limit the scope of the present application.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0025] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0026] It should be pointed out that the terms "first\second\third" involved in the embodiments of the present application are merely to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0027] The embodiment of the present application provides an image scanning method, which can be applied to electronic devices, and the electronic devices can be devices with information processing capabilities such as mobile phones, tablet computers, laptop computers, desktop computers, robots, drones, etc. The functions implemented by the image scanning method can be implemented by calling program codes by a processor in the electronic device, and of course the program codes can be stored in a computer storage medium. It can be seen that the electronic device at least includes a processor and a storage medium.

[0028] Figure 1 Schematic diagram of the implementation process of the image scanning method of the present application embodiment. Figure 1 As shown, the method at least includes the following steps 101 to 104:

[0029] Step 101: Acquire an initial scan image of a scan scene and point cloud data of the scan scene.

[0030] The target object scanned by the electronic device is, for example, a label on a bottle surface, a poster on a cylinder, a curved surface of a book, etc. It can be understood that the bottle surface or the cylindrical surface itself is a curved surface.

[0031] During implementation, the electronic device can collect data on the scanning scene through the TOF sensor, and then perform preliminary filtering on the sensor data output by the TOF sensor through the processor, and transform the filtered sensor data into three-dimensional coordinates in the camera coordinate system to obtain the point cloud data.

[0032] In some embodiments, the preliminary filtering includes denoising the characteristics of the sensor data output by the TOF sensor. The denoising method, for example, point cloud removal based on a position threshold, that is, removing points in the sensor data whose positions are greater than a threshold (e.g., 7 meters); or point cloud removal based on mutual distance, that is, removing points whose average distance to the surrounding point clouds is greater than the surrounding average of other points.

[0033] For the initial scan image, the electronic device may photograph the scan scene through a red, green, and blue (RGB) sensor to obtain the initial scan image.

[0034] In summary, the electronic device includes at least a TOF sensor and an RGB sensor, wherein the TOF sensor is used to obtain point cloud data of a scanning scene, and the RGB sensor is used to photograph the scanning scene to obtain an initial scanning image.

[0035] Step 102: Perform surface detection on the point cloud data to obtain initial surface data.

[0036] In some embodiments, the electronic device may use a near-cylinder as a fitting model and determine the approximate shape in the point cloud data through cylinder detection to obtain initial surface data. The so-called near-cylinder includes a cylinder with equal upper and lower bottom surfaces and a truncated cone with unequal upper and lower bottom surfaces.

[0037] Step 103: Optimize the surface represented by the initial surface data to obtain target surface data.

[0038] The surface represented by the initial surface data is usually a relatively rough surface. In order to obtain a more refined surface, in an embodiment of the present application, the electronic device optimizes the surface represented by the initial surface data. For example, when the surface is the side of a cylinder, the surface is meshed to obtain a mesh surface, and then the vertices of each mesh on the mesh surface are radially optimized to obtain the target surface data.

[0039] Step 104: Correct the pixel coordinates of the pixel points in the initial scanned image according to the three-dimensional coordinates in the target curved surface data to obtain a target scanned image.

[0040] It is understandable that in most cases, the initial scanned image obtained by the electronic device is an image that does not meet the application conditions. For example, Figure 2 As shown, the initial scanned image, that is, the image before correction is image 20, in which the target object 201 is tilted. After correction, the front view scan result of the target object 201 can be obtained, that is, Figure 2 A target scan image 202 is shown.

[0041] In some embodiments, the electronic device may implement step 104 through steps 307 and 308 of the following embodiments to obtain a target scan image.

[0042] In an embodiment of the present application, after obtaining the point cloud data of the scanning scene and the initial scanning image of the scanning scene, the electronic device performs surface detection on the point cloud data, and optimizes the initial surface data obtained by detection to obtain target surface data; in this way, on the one hand, the electronic device can correct the pixel coordinates in the initial scanning image according to the three-dimensional coordinates in the target surface data, so as to obtain a more accurate and detailed scanning result, that is, the target scanning image; on the other hand, compared with plane detection, surface detection on point cloud data can be applied to more scanning scenarios, for example, it can scan the surfaces of cylinders and nearly cylindrical objects.

[0043] The present application embodiment further provides an image scanning method, the method comprising at least the following steps 201 to 205:

[0044] Step 201, acquiring point cloud data of a scanning scene and an initial scanning image of the scanning scene;

[0045] Step 202: Perform cylinder shape detection on the point cloud data to obtain multiple characteristic parameter values ​​of the target object.

[0046] In some embodiments, the target object may be a cylinder. Figure 3 As shown, the characteristic parameter values ​​of the cylinder 30 include: the center position p0 of the bottom surface of the cylinder, the axial direction n0, the radius r, the tangential range (θ1, θ2) and the axial range (h); wherein θ1 is the angle between the tangential line 31 and the reference line, and θ2 is the angle between the tangential line 32 and the reference line; the axial range h is the height of the cylinder. In some embodiments, the electronic device can use a RANdom SAmple Consensus (RANSAC) cylinder detection algorithm to obtain these parameter values.

[0047] Step 203, determining the initial surface data from the point cloud data according to the multiple characteristic parameter values;

[0048] Step 204, optimizing the surface represented by the initial surface data to obtain target surface data;

[0049] Step 205: Correct the pixel coordinates of the pixel points in the initial scanned image according to the three-dimensional coordinates in the target curved surface data to obtain a target scanned image.

[0050] In an embodiment of the present application, the electronic device performs cylinder shape detection on point cloud data to obtain multiple characteristic parameter values ​​of the cylinder; and determines the initial surface data from the point cloud data based on the multiple characteristic parameter values; thus, compared with complex fitting models, using a near-cylinder as a fitting model can greatly simplify the algorithm while losing less accuracy, reduce the amount of calculation, and reduce the algorithm overhead cost; and compared with plane detection methods, it can cover more target objects and adapt to more user scenarios.

[0051] The present application embodiment further provides an image scanning method, the method at least comprising the following steps 301 to 308:

[0052] Step 301, acquiring point cloud data of a scanning scene and an initial scanning image of the scanning scene;

[0053] Step 302, performing a cylindrical shape detection on the point cloud data to obtain multiple characteristic parameter values ​​of the target object; wherein the shape of the target object is a cylinder, and the multiple characteristic parameter values ​​of the target object are, for example, Figure 3 As shown, it includes the center position p0 of the bottom surface of the cylinder, the axial direction n0, the radius r, the tangential range (θ1, θ2) and the axial range (h).

[0054] Step 303: determining initial surface data from the point cloud data according to the multiple characteristic parameter values.

[0055] by Figure 2 Taking the image 20 shown as an example, the initial surface data is the point cloud data of the target object 201 in the image 20 .

[0056] Step 304 , meshing the surface represented by the initial surface data according to the axial direction, the axial range, the tangential direction, the tangential range and the specific meshing interval to obtain N meshes, where N is an integer greater than 0.

[0057] The specific meshing interval includes the meshing interval in the axial direction and the meshing interval in the tangential direction. The meshing intervals in these two directions can be the same or different. The meshing result, such as Figure 4The mesh surface 401 shown has N meshes 402. It can be understood that the size of the mesh division interval determines the mesh density of the mesh surface to a certain extent, that is, the value of N. The larger the mesh division interval, the fewer meshes are obtained, and the rougher the optimization result; the smaller the mesh division interval, the more meshes are obtained, and the finer the optimization result, but the algorithm complexity will also be higher.

[0058] Step 305 , searching for an optimal solution for each vertex of each mesh according to the three-dimensional coordinates of the sampling points in the initial surface data, to obtain the optimal three-dimensional coordinates of the corresponding vertex.

[0059] In some embodiments, an optimal solution is searched for each vertex along the radial direction in which it is located to obtain the optimal three-dimensional coordinates of the corresponding vertex. In an embodiment of the present application, three optimal solution search methods are provided. For example, using steps 405 to 407 in the following embodiment, the optimal solution is searched in the search space where the radial vector of the vertex is located; for another example, using steps 505 to 508 in the following embodiment, that is, using a radial optimization method based on PSR to obtain the optimal solution; for another example, using steps 605 to 608 in the following embodiment, that is, using a radial optimization method based on at least dual-camera data fusion to obtain the optimal solution.

[0060] Step 306: determine the optimal three-dimensional coordinates of each vertex as the target surface data.

[0061] It can be understood that what is obtained through step 306 is an optimized mesh surface, which is a surface where the coordinates of each vertex of each mesh are the optimal three-dimensional coordinates. In other words, the target surface data includes the optimal three-dimensional coordinates of each vertex of each mesh.

[0062] Step 307: Determine the transformation relationship between the target surface data and the reference surface data according to the optimal three-dimensional coordinates of each vertex in the target surface data and the three-dimensional coordinates of the corresponding vertex in the reference surface data.

[0063] In fact, the electronic device determines the position transformation relationship between the optimized mesh surface and the reference surface through step 307. The reference surface data can be pre-configured. For example, the reference surface data represents a plane viewed from the front, such as Figure 5 The reference surface 50 is shown, wherein the reference surface data includes the three-dimensional coordinates of each vertex of each mesh in the reference surface 50 .

[0064] In the embodiment of the present application, the transformation relationship can be represented by a transformation matrix, a transformation matrix group, or a free mapping relationship.

[0065] Step 308: Correct the pixel coordinates of the pixel points in the initial scanned image according to the transformation relationship to obtain a target scanned image.

[0066] In an embodiment of the present application, the surface represented by the initial surface data is meshed, and only the vertices of the mesh are searched for the optimal solution, which can greatly reduce the amount of calculation for surface optimization.

[0067] The present application embodiment further provides an image scanning method, the method at least comprising the following steps 401 to 410:

[0068] Step 401, acquiring point cloud data of a scanning scene and an initial scanning image of the scanning scene;

[0069] Step 402, performing cylinder shape detection on the point cloud data to obtain multiple characteristic parameter values ​​of the target object; wherein the shape of the target object can be a cylinder, and the multiple characteristic parameter values ​​of the target object include the center position p0 of the bottom surface of the cylinder, the axial direction n0, the radius r, the tangential range (θ1, θ2) and the axial range (h).

[0070] Step 403, determining the initial surface data from the point cloud data according to the multiple characteristic parameter values;

[0071] Step 404, meshing the surface according to the axial direction, the axial range, the tangential direction, the tangential range and a specific meshing interval to obtain N meshes, where N is an integer greater than 0;

[0072] Step 405, determining the radial vector where the j-th vertex of the ith grid is located according to the position of the surface where the j-th vertex of the ith grid is located, the radius of the cylinder and the center position of the bottom surface of the cylinder; wherein i is an integer greater than 0 and less than or equal to N, and j is an integer greater than 0 and less than or equal to 4;

[0073] Step 406: Determine a search space from the initial surface data according to the radial vector.

[0074] In some embodiments, the search space may adopt a cubic bounding box as shown in formula (1):

[0075] L<[RT]p<U(1);

[0076] In formula (1), L and U are the lower boundary point and upper boundary point in the bounding box coordinate system, respectively, p is the three-dimensional coordinate of the midpoint of the initial surface data, and [RT] is the augmented transformation matrix from world coordinates to bounding box coordinates.

[0077] When implementing step 406, the electronic device may use the jth vertex as the center point of the cubic bounding box, and determine the cubic bounding box from the initial surface data according to the axial length, the tangential grid division interval and the size of the radial vector.

[0078] For example, Figure 6 As shown, the center point of the bounding box 60 is the jth vertex, the length of the bounding box is the radius r of the cylinder, the axial length (ie, the height) is r / 2, and the width is the grid division interval in the tangential direction.

[0079] Step 407: Determine the optimal three-dimensional coordinates of the j-th vertex according to the three-dimensional coordinates of the sampling points in the search space.

[0080] In some embodiments, the electronic device may determine the three-dimensional coordinates of the center of gravity of the search space based on the three-dimensional coordinates of the sampling points in the search space; project the three-dimensional coordinates of the center of gravity onto the radial vector to obtain the optimal three-dimensional coordinates of the j-th vertex.

[0081] For example, the search space {p i |p i ∈BOX} g The three-dimensional coordinates of:

[0082]

[0083] In formula (2), p i is the three-dimensional coordinate of the point in the search space, and K is a constant.

[0084] Then, according to the following formula (3), the calculated center of gravity p g The three-dimensional coordinates are projected to the radial vector where the j-th vertex is located Thus, the optimal three-dimensional coordinate p of the j-th vertex is obtained. * .

[0085]

[0086] Wherein, p0 represents the three-dimensional coordinates of the j-th vertex before optimization.

[0087] It should be noted that the determination of the optimal three-dimensional coordinates of each vertex can be achieved through the above steps 405 to 407.

[0088] Step 408, determining the optimal three-dimensional coordinates of each vertex as the target surface data;

[0089] Step 409, determining a transformation relationship between the target curved surface data and the reference surface data according to the three-dimensional coordinates of each vertex in the target curved surface data and the three-dimensional coordinates of the corresponding vertex in the reference surface data;

[0090] Step 410: Correct the pixel coordinates of the pixel points in the initial scanned image according to the transformation relationship to obtain a target scanned image.

[0091] The present application embodiment further provides an image scanning method, the method at least comprising the following steps 501 to 511:

[0092] Step 501, acquiring point cloud data of a scanning scene and an initial scanning image of the scanning scene;

[0093] Step 502, performing cylinder shape detection on the point cloud data to obtain multiple characteristic parameter values ​​of the target object; wherein the shape of the target object can be a cylinder, and the multiple characteristic parameter values ​​of the target object include the center position p0 of the bottom surface of the cylinder, the axial direction n0, the radius r, the tangential range (θ1, θ2) and the axial range (h).

[0094] Step 503, determining the initial surface data from the point cloud data according to the multiple characteristic parameter values;

[0095] Step 504, meshing the surface according to the axial direction, the axial range, the tangential direction, the tangential range and a specific meshing interval to obtain N meshes, where N is an integer greater than 0;

[0096] Step 505: Perform Poisson surface reconstruction according to the three-dimensional coordinates of the sampling points in the initial surface data to obtain an isovalued surface.

[0097] In some embodiments, the electronic device may use a PSR algorithm to obtain the isosurface. The surface of an object is estimated based on the assumption that the surface of an object in the real world is continuous. The PSR algorithm can eliminate the influence of point cloud measurement errors on the results to a certain extent and restore the surface of the real object.

[0098] Step 506, determining the radial vector where the j-th vertex of the ith grid is located according to the position of the surface where the j-th vertex of the ith grid is located, the radius of the cylinder and the center position of the bottom surface of the cylinder; wherein i is an integer greater than 0 and less than or equal to N, and j is an integer greater than 0 and less than or equal to 4;

[0099] Step 507, determining the intersection point between the radial vector and the isosurface;

[0100] Step 508: Determine the three-dimensional coordinates of the intersection point on the isosurface as the optimal three-dimensional coordinates of the j-th vertex.

[0101] like Figure 7 As shown, before radial optimization, the electronic device first uses the PSR algorithm to process the initial surface data to generate a corresponding mesh, and then detects the intersection point of the generated Mesh with the radial line where the j-th vertex is located, and takes the three-dimensional coordinates of the Mesh corresponding to the intersection point as the optimization result, that is, the optimal three-dimensional coordinates of the j-th vertex.

[0102] The PSR algorithm has the characteristic of being able to recover the surface of objects in the real world from point clouds with a lot of noise. PSR is an algorithm derived from the fact that the surface of real objects is smooth and continuous. Therefore, it conforms to the scanning characteristics of real objects, and the restored surface is closer to the real value. Therefore, the use of the PSR algorithm here can effectively remove the influence of noise or false detection, thereby reducing the influence of deviation values ​​in the point cloud.

[0103] It should be noted that the input of the PSR algorithm is the initial surface data, and the output is Mesh. The implementation steps of the PSR algorithm may include the following steps S1 to S4: S1, point cloud normal estimation of the initial surface data; S2, spatial meshing of the initial surface data, such as the octave method; S3, finding the optimal surface to meet the surface continuity and the estimated normal; S4, outputting the optimized optimal surface to form a Mesh grid.

[0104] It should also be noted that the determination of the optimal three-dimensional coordinates of each vertex is achieved through the above steps 506 to 508.

[0105] Step 509, determining the optimal three-dimensional coordinates of each vertex as the target surface data;

[0106] Step 510, determining a transformation relationship between the target curved surface data and the reference surface data according to the three-dimensional coordinates of each vertex in the target curved surface data and the three-dimensional coordinates of the corresponding vertex in the reference surface data;

[0107] Step 511, correcting the pixel coordinates of the pixel points in the initial scanned image according to the transformation relationship to obtain a target scanned image.

[0108] The present application embodiment further provides an image scanning method, the method at least comprising the following steps 601 to 611:

[0109] Step 601, acquiring point cloud data of a scan scene and an initial scan image of the scan scene;

[0110] Step 602, perform cylinder shape detection on the point cloud data to obtain multiple characteristic parameter values ​​of the target object; wherein the shape of the target object is a cylinder, and the multiple characteristic parameter values ​​of the target object include: the center position p0 of the bottom surface of the cylinder, the axial direction n0, the radius r, the tangential range (θ1, θ2) and the axial range (h).

[0111] Step 603, determining the initial surface data from the point cloud data according to the multiple characteristic parameter values;

[0112] Step 604, meshing the surface according to the axial direction, axial range, tangential direction, tangential range and specific meshing interval to obtain N meshes, where N is an integer greater than 0;

[0113] Step 605, back-project the three-dimensional coordinates of the k-th sampling point on the radial vector where the j-th vertex of the i-th grid is located onto the imaging plane of each camera to obtain the corresponding pixel coordinates; wherein i is an integer greater than 0 and less than or equal to N, j is an integer greater than 0 and less than or equal to 4, and k is an integer greater than 0;

[0114] Step 606, determining the area block of each pixel coordinate on the image captured by the corresponding camera according to the specific sampling window;

[0115] Step 607, determining the correlation between each of the region blocks;

[0116] Step 608, when the correlation degree does not satisfy the specific condition, the three-dimensional coordinates of the next sampling point of the radial vector are back-projected onto the imaging plane of each camera respectively until the determined correlation degree satisfies the specific condition, and the three-dimensional coordinates of the corresponding sampling point are determined as the optimal three-dimensional coordinates of the j-th vertex.

[0117] Steps 605 to 608 provide another optimization method, namely a radial optimization method based on at least dual-camera data fusion, which can further improve the robustness.

[0118] In some embodiments, the jth vertex may be used as a starting point to optimize the cost function, thereby obtaining the optimal three-dimensional coordinates of the jth vertex. Figure 8 As shown, the three-dimensional coordinates p of the j-th vertex before optimization j As the starting point, the gradient descent method or LM algorithm is used to optimize the cost function shown in formula (4):

[0119] p * =argmin p (R(pp j)+α*C(W(π0(p j )),W(π1(p j ))))(4);

[0120] In the formula, p * is the optimal solution, i.e., the optimal three-dimensional coordinates of the j-th vertex, p is the position of the search point in the initial surface data, i.e., the three-dimensional coordinates of the k-th sampling point, R is a regular function, such as the L2 regular function, π0 and π1 are back-projection functions of the dual-camera spatial coordinates to the pixel coordinates, W is a sampling window function, such as sampling of 3*3 square pixels or 7*7 square pixels of the back-projection point's neighbors, C is a cross-correlation function of the neighboring area blocks of the projection point in the dual-camera image, and the cross-correlation function can adopt functions such as NCC or ZNCC, and α is a proportional coefficient used to adjust the dependence on TOF data.

[0121] It should be noted that, for the case of three or more cameras, as shown in formula (5), the form of the cost function can remain consistent, and the difference lies in the calculation of the cross-correlation function:

[0122] p * =argmin p (R(pp j )+α*C(W(π0(p j )),W(π1(p j )),W(π2(p j )...))(5);

[0123] That is, by adding the projection terms of the third and fourth cameras in equation (4), the input of the function C for calculating the cross-correlation becomes multiple, and the calculation method of the cross-correlation function can adopt the sum or average value of the results of the correlation between two cameras.

[0124] Step 609, determining the optimal three-dimensional coordinates of each vertex as the target surface data;

[0125] Step 610, determining a transformation relationship between the target curved surface data and the reference surface data according to the three-dimensional coordinates of each vertex in the target curved surface data and the three-dimensional coordinates of the corresponding vertex in the reference surface data;

[0126] Step 611, correcting the pixel coordinates of the pixel points in the initial scanned image according to the transformation relationship to obtain a target scanned image.

[0127] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0128] In the shape detection algorithm based on TOF data in the embodiment of the present application, a near-cylinder is used as a fitting model for the case of a target object with a curved surface. The approximate shape in the point cloud data is first determined by cylinder detection, and then the radial direction of the cylinder is optimized to obtain a more refined curved surface result.

[0129] Compared with complex model fitting, it can greatly simplify the algorithm, reduce the amount of calculation, and reduce the algorithm overhead cost while losing less accuracy; compared with the plane detection method, it can cover more target objects and adapt to more user scenarios.

[0130] In some embodiments, the document scanning system includes a TOF sensor, an RGB sensor, a transformation parameter generation module and an image transformation module, and the functions of the transformation parameter generation module and the image transformation module can be completed by a processor; wherein,

[0131] The transformation parameter generation module is used to calculate the sensor data output by the TOF sensor to generate the transformation parameters required by the image transformation module. The transformation parameters can be one of the following: transformation matrix, transformation matrix group, free mapping relationship.

[0132] The image transformation module is used to transform the RGB data obtained from the RGB sensor (i.e., an example of the initial scan image) according to the transformation parameters generated by the transformation parameter generation module, so as to obtain a transformed front view scan result, i.e., the target scan image.

[0133] In some embodiments, the transformation parameter generation module includes a cylinder detection unit, a mesh division unit, a cylinder radial optimization unit, a free surface generation unit and a transformation parameter generation unit; wherein,

[0134] The cylinder detection unit can use the RANSAC cylinder detection algorithm;

[0135] The meshing unit can be used to divide the target area of ​​the cylindrical surface into regular meshes with equal intervals in the tangential and axial directions;

[0136] The cylindrical radial optimization unit can be used to search for the optimal solution for each vertex of the divided mesh in the radial direction of the cylinder;

[0137] The search for the optimal solution can be done by one of the following methods:

[0138] (1) Find the point with the largest number of point clouds in the radial upward range as the optimal solution;

[0139] (2) A method of calculating the intersection of the radial line and the grid reconstructed from the surface of the whole point cloud in the radial direction as the optimal solution;

[0140] (3) Use dual-camera data fusion to perform calculations to optimize the back-projection cost function.

[0141] The free-form surface generation unit is used to take all the optimal solutions in the radial direction as vertices, perform topological connections according to the topological structure of the mesh division, and form a free-form surface mesh.

[0142] The transformation parameter generation unit is used to generate transformation parameters according to the free-form surface mesh and the correction target parameters. According to the spatial position of the free-form surface mesh and the set spatial position of the correction result (i.e. the three-dimensional coordinates of the vertices in the reference surface data), a transformation matrix or a transformation matrix group or a free mapping relationship and other transformation parameters are generated.

[0143] The present application embodiment further provides an image scanning method, Fig. 9 Schematic diagram of the implementation process of the image scanning method of the present application embodiment. Fig. 9 As shown, at least the following steps 901 to 907 may be included:

[0144] Step 901, preliminarily filtering the sensor data output by the TOF, and then transforming the preliminarily filtered sensor data into three-dimensional coordinates in the camera coordinate system to obtain three-dimensional point cloud data;

[0145] In some embodiments, the preliminary filtering includes denoising the characteristics of the sensor data output by the TOF, the denoising method being, for example, point cloud removal based on a position threshold, or point cloud removal based on mutual distance;

[0146] Step 902, perform cylinder detection on the three-dimensional point cloud data to obtain cylinder fitting parameters, that is, multiple characteristic parameter values ​​of the cylinder, wherein the parameters include: the center position p0 of the bottom surface of the cylinder, the axial direction n0, the radius r, the tangential range (θ1, θ2) and the axial range (h). In some embodiments, the RANSAC cylinder detection algorithm can be used to obtain these parameters.

[0147] Step 903, dividing the target area on the cylindrical surface into regular grids with equal intervals in the tangential and axial directions;

[0148] Step 904, searching for the optimal solution for each vertex of the divided grid along the radial direction of the cylinder;

[0149] Step 905, according to the original mesh topology, updating the coordinate position to the optimal position obtained by radial optimization, forming an optimized free mesh surface;

[0150] Step 906, generating corresponding transformation parameters according to the coordinate correspondence relationship;

[0151] For example, when the transformation parameters are relatively uniform, a single homography matrix can be used; when the transformation is relatively complex, a coordinate pair relationship can be directly established to generate an interpolation function for subsequent image transformation.

[0152] Step 907, using the generated transformation parameters to transform the input image (ie, the initial scanned image) to obtain a corrected front view result, ie, the target scanned image.

[0153] Based on the foregoing embodiments, an embodiment of the present application provides an image scanning device, which includes the modules included and can be implemented by a processor in a computer device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0154] Fig.10 Schematic diagram of the structure of the image scanning device according to the embodiment of the present application. Fig.10 As shown, the device 100 includes a data acquisition module 101, a curved surface detection module 102, a curved surface optimization module 103 and an image correction module 104, wherein:

[0155] The data acquisition module 101 is used to acquire point cloud data of a scanning scene and an initial scanning image of the scanning scene;

[0156] A surface detection module 102 is used to perform surface detection on the point cloud data to obtain initial surface data;

[0157] A surface optimization module 103 is used to optimize the surface represented by the initial surface data to obtain target surface data;

[0158] The image correction module 104 is used to correct the pixel coordinates of the pixel points in the initial scanned image according to the three-dimensional coordinates in the target curved surface data to obtain a target scanned image.

[0159] In some embodiments, the surface detection module 102 is used to: perform cylindrical shape detection on the point cloud data to obtain multiple characteristic parameter values ​​of the target object; and determine the initial surface data from the point cloud data according to the multiple characteristic parameter values.

[0160] In some embodiments, the shape of the target object is a cylinder, and the multiple characteristic parameter values ​​include the axial direction, axial range, tangential direction and tangential range of the cylinder; the surface optimization module 103 is used to: mesh the surface represented by the initial surface data according to the axial direction, the axial range, the tangential direction, the tangential range and the specific mesh division interval to obtain N grids, where N is an integer greater than 0; search for the optimal solution for each vertex of each grid according to the three-dimensional coordinates of the sampling points in the initial surface data to obtain the optimal three-dimensional coordinates of the corresponding vertex; and determine the optimal three-dimensional coordinates of each vertex as the target surface data.

[0161] In some embodiments, the multiple characteristic parameter values ​​also include the radius of the cylinder and the center position of the bottom surface of the cylinder; the surface optimization module 103 is used to: determine the radial vector where the jth vertex of the ith grid is located based on the position of the surface where the jth vertex of the ith grid is located, the radius of the cylinder and the center position of the bottom surface of the cylinder; wherein i is an integer greater than 0 and less than or equal to N, and j is an integer greater than 0 and less than or equal to 4; determine the search space from the initial surface data based on the radial vector; determine the optimal three-dimensional coordinates of the jth vertex based on the three-dimensional coordinates of the sampling points in the search space.

[0162] In some embodiments, the search space adopts a cubic bounding box, and the surface optimization module 103 is used to:

[0163] The j-th vertex is used as the center point of the cubic bounding box, and the cubic bounding box is determined from the initial surface data according to the axial length, the tangential grid division interval and the size of the radial vector.

[0164] In some embodiments, the surface optimization module 103 is used to: determine the three-dimensional coordinates of the center of gravity of the search space based on the three-dimensional coordinates of the sampling points in the search space; project the three-dimensional coordinates of the center of gravity onto the radial vector to obtain the optimal three-dimensional coordinates of the j-th vertex.

[0165] In some embodiments, the surface optimization module 103 is used to: perform Poisson surface reconstruction according to the three-dimensional coordinates of the sampling points in the initial surface data to obtain an isosurface; determine the radial vector at which the j-th vertex of the ith grid is located according to the position of the surface at which the j-th vertex of the ith grid is located, the radius of the cylinder and the center position of the bottom of the cylinder; wherein i is an integer greater than 0 and less than or equal to N, and j is an integer greater than 0 and less than or equal to 4; determine the intersection point of the radial vector and the isosurface; and determine the three-dimensional coordinates of the intersection point on the isosurface as the optimal three-dimensional coordinates of the j-th vertex.

[0166] In some embodiments, the surface optimization module 103 is used to: back-project the three-dimensional coordinates of the k-th sampling point on the radial vector where the j-th vertex of the ith grid is located onto the imaging plane of each camera to obtain the corresponding pixel coordinates; wherein i is an integer greater than 0 and less than or equal to N, and j is an integer greater than 0 and less than or equal to 4; determine the area block of each of the pixel coordinates on the image captured by the corresponding camera according to a specific sampling window; determine the degree of correlation between each of the area blocks; if the degree of correlation does not meet a specific condition, back-project the three-dimensional coordinates of the next sampling point of the radial vector onto the imaging plane of each camera, until the determined degree of correlation meets the specific condition, and determine the three-dimensional coordinates of the corresponding sampling point as the optimal three-dimensional coordinates of the j-th vertex.

[0167] In some embodiments, the image correction module 104 is used to: determine the transformation relationship between the target surface data and the reference surface data based on the optimal three-dimensional coordinates of each vertex in the target surface data and the three-dimensional coordinates of the corresponding vertex in the reference surface data; and correct the pixel coordinates of the pixel points in the initial scanned image based on the transformation relationship to obtain the target scanned image.

[0168] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0169] It should be noted that in the embodiment of the present application, if the above-mentioned image scanning method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable an electronic device (which can be a mobile phone, a tablet computer, a laptop computer, a desktop computer, a robot, a drone, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0170] Correspondingly, an embodiment of the present application provides an electronic device, Fig.11 A hardware entity diagram of an electronic device according to an embodiment of the present application is shown in FIG. Fig.11As shown, the hardware entity of the electronic device 110 includes: a memory 111 and a processor 112, wherein the memory 111 stores a computer program that can be run on the processor 112, and the processor 112 implements the steps of the image scanning method provided in the above embodiment when executing the program.

[0171] The memory 111 is configured to store instructions and applications executable by the processor 112, and can also cache data to be processed or processed by the processor 112 and various modules in the electronic device 110 (for example, image data, audio data, voice communication data, and video communication data), which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0172] Correspondingly, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the image scanning method provided in the above embodiment are implemented.

[0173] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0174] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0175] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0176] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as: multiple modules or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or modules can be electrical, mechanical or other forms.

[0177] The modules described above as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules; they may be located in one place or distributed on multiple network units; some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0178] In addition, all functional modules in the embodiments of the present application may be integrated into one processing unit, or each module may be a separate unit, or two or more modules may be integrated into one unit; the above-mentioned integrated modules may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0179] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0180] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium, including a number of instructions to enable an electronic device (which can be a mobile phone, tablet computer, laptop computer, desktop computer, robot, drone, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, magnetic disks or optical disks.

[0181] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0182] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0183] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0184] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0185] Industrial Applicability

[0186] In an embodiment of the present application, after obtaining the point cloud data of the scanning scene and the initial scanning image of the scanning scene, the electronic device performs surface detection on the point cloud data, and optimizes the initial surface data obtained by detection to obtain target surface data; in this way, on the one hand, the electronic device can correct the pixel coordinates in the initial scanning image according to the three-dimensional coordinates in the target surface data, so as to obtain a more accurate and detailed scanning result (i.e., the target scanning image); on the other hand, compared with plane detection, surface detection on point cloud data can be applied to more scanning scenarios, for example, it can scan the surfaces of cylinders and near-cylindrical objects.

Claims

1. An image scanning method, the method comprising: Acquire an initial scanned image of a scanned scene and point cloud data of the scanned scene; Performing surface detection on the point cloud data to obtain initial surface data; Optimizing the surface represented by the initial surface data to obtain target surface data; Correcting the pixel coordinates of the pixel points in the initial scanned image according to the three-dimensional coordinates in the target curved surface data to obtain a target scanned image; The step of performing surface detection on the point cloud data to obtain initial surface data includes: Performing cylinder shape detection on the point cloud data to obtain multiple characteristic parameter values ​​of the target object; Determining the initial surface data from the point cloud data according to the multiple characteristic parameter values; The shape of the target object is a cylinder, the multiple characteristic parameter values ​​include the axial direction, axial range, tangential direction and tangential range of the cylinder, and the surface represented by the initial surface data is optimized to obtain the target surface data, including: According to the axial direction, the axial range, the tangential direction, the tangential range and the specific grid division interval, the surface represented by the initial surface data is grid-divided to obtain N grids, where N is an integer greater than 0; According to the three-dimensional coordinates of the sampling points in the initial surface data, an optimal solution search is performed for each vertex of each grid to obtain the optimal three-dimensional coordinates of the corresponding vertex; The optimal three-dimensional coordinates of each vertex are determined as the target surface data.

2. The method according to claim 1, wherein the plurality of characteristic parameter values ​​further include the radius and the center position of the cylinder, and the step of searching for an optimal solution for each vertex of each grid according to the three-dimensional coordinates of the sampling points in the initial surface data to obtain the optimal three-dimensional coordinates of the corresponding vertex comprises: Determine the radial vector where the j-th vertex of the i-th grid is located according to the position of the surface where the j-th vertex of the i-th grid is located, the radius of the cylinder and the center position of the bottom surface of the cylinder; wherein i is an integer greater than 0 and less than or equal to N, and j is an integer greater than 0 and less than or equal to 4; Determining a search space from the initial surface data according to the radial vector; The optimal three-dimensional coordinates of the j-th vertex are determined according to the three-dimensional coordinates of the sampling points in the search space.

3. The method according to claim 2, wherein the search space adopts a cubic bounding box, and the step of determining the search space from the initial surface data according to the radial vector comprises: The j-th vertex is used as the center point of the cubic bounding box, and the cubic bounding box is determined from the initial surface data according to the axial length, the tangential grid division interval and the size of the radial vector.

4. The method according to claim 2, wherein determining the optimal three-dimensional coordinates of the j-th vertex according to the three-dimensional coordinates of the sampling points in the search space comprises: Determining the three-dimensional coordinates of the center of gravity of the search space according to the three-dimensional coordinates of the sampling points in the search space; The three-dimensional coordinates of the center of gravity are projected onto the radial vector to obtain the optimal three-dimensional coordinates of the j-th vertex.

5. The method according to claim 1, wherein the plurality of characteristic parameter values ​​further include the radius and the center position of the cylinder, and wherein the optimal solution search is performed on each vertex of each grid according to the three-dimensional coordinates of the sampling points in the initial surface data to obtain the optimal three-dimensional coordinates of the corresponding vertex, comprising: Reconstructing the Poisson surface according to the three-dimensional coordinates of the sampling points in the initial surface data to obtain an isovalued surface; Determine the radial vector where the jth vertex of the i-th grid is located according to the position of the surface where the jth vertex of the i-th grid is located, the radius of the cylinder and the center position of the bottom surface of the cylinder; wherein i is an integer greater than 0 and less than or equal to N, and j is an integer greater than 0 and less than or equal to 4; determining an intersection point between the radial vector and the isosurface; The three-dimensional coordinates of the intersection point on the isovalue surface are determined as the optimal three-dimensional coordinates of the j-th vertex.

6. The method according to claim 1, wherein searching for an optimal solution for each vertex of each mesh according to the three-dimensional coordinates of the sampling points in the initial surface data to obtain the optimal three-dimensional coordinates of the corresponding vertex comprises: Back-project the three-dimensional coordinates of the k-th sampling point on the radial vector where the j-th vertex of the i-th grid is located onto the imaging plane of each camera to obtain the corresponding pixel coordinates; where i is an integer greater than 0 and less than or equal to N, j is an integer greater than 0 and less than or equal to 4, and k is an integer greater than 0; Determine, according to a specific sampling window, a region block of each pixel coordinate on an image acquired by a corresponding camera; Determining the degree of correlation between each of the region blocks; When the correlation degree does not satisfy a specific condition, the three-dimensional coordinates of the next sampling point of the radial vector are back-projected onto the imaging plane of each camera respectively until the determined correlation degree satisfies the specific condition, and the three-dimensional coordinates of the corresponding sampling point are determined as the optimal three-dimensional coordinates of the j-th vertex.

7. The method according to any one of claims 1 to 6, wherein the pixel coordinates of the pixel points in the initial scanned image are corrected according to the three-dimensional coordinates in the target surface data to obtain the target scanned image, comprising: Determining a transformation relationship between the target curved surface data and the reference surface data according to the optimal three-dimensional coordinates of each vertex in the target curved surface data and the three-dimensional coordinates of the corresponding vertex in the reference surface data; According to the transformation relationship, the pixel coordinates of the pixel points in the initial scanned image are corrected to obtain a target scanned image.

8. An image scanning device, comprising: A data acquisition module, used to acquire point cloud data of a scanning scene and an initial scanning image of the scanning scene; A surface detection module, used to perform surface detection on the point cloud data to obtain initial surface data; A surface optimization module, used for optimizing the surface represented by the initial surface data to obtain target surface data; An image correction module, used for correcting the pixel coordinates of the pixel points in the initial scanned image according to the three-dimensional coordinates in the target curved surface data to obtain a target scanned image; The step of performing surface detection on the point cloud data to obtain initial surface data includes: Performing cylinder shape detection on the point cloud data to obtain multiple characteristic parameter values ​​of the target object; Determining the initial surface data from the point cloud data according to the multiple characteristic parameter values; The shape of the target object is a cylinder, the multiple characteristic parameter values ​​include the axial direction, axial range, tangential direction and tangential range of the cylinder, and the surface represented by the initial surface data is optimized to obtain the target surface data, including: According to the axial direction, the axial range, the tangential direction, the tangential range and the specific grid division interval, the surface represented by the initial surface data is grid-divided to obtain N grids, where N is an integer greater than 0; According to the three-dimensional coordinates of the sampling points in the initial surface data, an optimal solution search is performed for each vertex of each grid to obtain the optimal three-dimensional coordinates of the corresponding vertex; The optimal three-dimensional coordinates of each vertex are determined as the target surface data.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor implements the steps of the image scanning method according to any one of claims 1 to 7 when executing the program.

10. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the image scanning method according to any one of claims 1 to 7.

Images