Method, device and equipment for reconstructing three-dimensional model of object and medium

Abstract

The invention discloses a three-dimensional model reconstruction method and device of an object, equipment and a medium. The method comprises the steps that two-dimensional images of a target object at multiple shooting angles are acquired, and an original three-dimensional model of the target object is generated according to the two-dimensional images; segmenting the original three-dimensional model to obtain a plurality of planar quadrangles, and determining a standard region mask according to the type of the original three-dimensional model; according to the plane quadrangles and the standard region masks, a homography transformation matrix is obtained through calculation; and converting the image pixel points in each standard region mask into plane point cloud data by using each homography transformation matrix, and obtaining a three-dimensional reconstruction model of the target object. According to the technical scheme, the two-dimensional image is converted into the three-dimensional point cloud data through the homography transformation matrix, the three-dimensional model reconstructed through the method is not affected by the camera pose, image distortion and the like, and the three-dimensional model of the actual target object can be output more stably.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method, device, equipment and medium for reconstructing a three-dimensional model of an object. Background Art

[0002] Three-dimensional reconstruction in the field of computer vision is mainly achieved by taking multiple photos of an object or a scene from different perspectives and then constructing a three-dimensional geometric structure. This technology has wide application value in many fields, such as the three-dimensional visualization of cities and the 3D digital preservation of cultural relics.

[0003] However, in the prior art, for visual 3D reconstruction technology, sparse reconstruction and dense reconstruction are mainly used to restore and reconstruct 3D geometric structures. However, due to inaccurate camera pose calculation, errors in image distortion correction, and unclear areas of some images, the 3D model generated using the above method may have inaccurate model shape and size. Summary of the invention

[0004] Based on this, the present invention provides a method, device, equipment and medium for reconstructing a three-dimensional model of an object to solve the problem of inaccurate shape and size of the reconstructed three-dimensional model.

[0005] In a first aspect, an embodiment of the present invention provides a method for reconstructing a three-dimensional model of an object, the method comprising:

[0006] Acquire two-dimensional images of the target object taken at multiple shooting angles, and generate an original three-dimensional model of the target object according to each of the two-dimensional images;

[0007] The original three-dimensional model is segmented to obtain a plurality of plane quadrilaterals, and a standard region mask corresponding to each plane quadrilateral is determined in each two-dimensional image according to the type of the original three-dimensional model;

[0008] According to each planar quadrilateral in the three-dimensional model and the standard region mask corresponding to each planar quadrilateral, a homography transformation matrix for mapping from the two-dimensional image to the three-dimensional model is calculated;

[0009] The image pixels in each standard region mask are converted into plane point cloud data by using each homography transformation matrix, and a three-dimensional reconstructed model of the target object is obtained according to each plane point cloud data.

[0010] In a second aspect, an embodiment of the present invention further provides a device for reconstructing a three-dimensional model of an object, the device comprising:

[0011] An original three-dimensional model generation module is used to obtain two-dimensional images of the target object taken at multiple shooting angles, and generate an original three-dimensional model of the target object according to each of the two-dimensional images;

[0012] A standard region mask determination module is used to segment the original three-dimensional model to obtain a plurality of planar quadrilaterals, and determine the standard region mask corresponding to each planar quadrilateral in each two-dimensional image according to the type of the original three-dimensional model;

[0013] A homography transformation matrix calculation module, used to calculate a homography transformation matrix for mapping from a two-dimensional image to a three-dimensional model according to each planar quadrilateral in the three-dimensional model and a standard region mask corresponding to each planar quadrilateral;

[0014] The three-dimensional reconstruction model building module is used to convert the image pixels in each standard area mask into plane point cloud data using each homography transformation matrix, and obtain the three-dimensional reconstruction model of the target object based on each plane point cloud data.

[0015] In a fourth aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute a method for reconstructing a three-dimensional model of an object as described in any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement a method for reconstructing a three-dimensional model of an object as described in any embodiment of the present invention when executed.

[0020] The technical solution of the embodiment of the present invention avoids the currently commonly used sparse reconstruction and dense reconstruction methods for reconstructing three-dimensional models, and instead converts a two-dimensional image into three-dimensional point cloud data by utilizing a homography transformation matrix, thereby reconstructing the generated original three-dimensional model. The three-dimensional model reconstructed by the above method is not affected by camera pose and image distortion, and can more stably output a three-dimensional model of the actual target object.

[0021] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 is a flow chart of a method for reconstructing a three-dimensional model of an object provided in Embodiment 1 of the present invention;

[0024] Figure 2 It is a two-dimensional image reference image of a target object at a certain shooting angle applicable to the embodiment of the present invention;

[0025] Figure 3 is an original region mask reference image corresponding to the current two-dimensional image applicable to the embodiment of the present invention;

[0026] Figure 4 is a flow chart of another method for reconstructing a three-dimensional model of an object provided in accordance with the second embodiment of the present invention;

[0027] Figure 5 is a schematic structural diagram of a three-dimensional model reconstruction device for an object provided in Embodiment 3 of the present invention;

[0028] Figure 6 It is a structural schematic diagram of an electronic device for implementing a method for reconstructing a three-dimensional model of an object according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] Embodiment 1

[0032] Figure 1 This is a flow chart of a method for reconstructing a three-dimensional model of an object provided in the first embodiment of the present invention. This embodiment is applicable to the case of reconstructing a three-dimensional geometric structure of a target object. The method can be executed by a three-dimensional model reconstruction device of an object, and the device can be implemented in the form of hardware and / or software. Figure 1 As shown, the method includes:

[0033] S110, acquiring two-dimensional images of the target object captured at multiple shooting angles, and generating an original three-dimensional model of the target object according to each of the two-dimensional images.

[0034] In the embodiment of the present invention, since the target object is a three-dimensional solid geometric structure, a lot of information will be lost when photographing the object with a three-dimensional solid geometric structure as a flat two-dimensional image, and the 3D model cannot be restored by relying solely on a two-dimensional image from a single shooting angle. Therefore, obtaining two-dimensional images of the target object from multiple shooting angles can ensure that the entire surface of the target object is captured.

[0035] Optionally, generating an original three-dimensional model of the target object according to each of the two-dimensional images may include:

[0036] Based on prior knowledge, a three-dimensional model of the target object in each two-dimensional image is constructed by a custom method to obtain an artificial three-dimensional model of the target object; or

[0037] According to each of the two-dimensional images, a three-dimensional point cloud model of the target object is generated using a three-dimensional reconstruction technology, and the three-dimensional point cloud model is adjusted using an approximate reconstruction method to obtain a non-artificial three-dimensional model of the target object.

[0038] In an embodiment of the present invention, there are two ways to generate an original three-dimensional model of a target object. First, an artificial three-dimensional model refers to making a preliminary estimate of the size of the target object based on common sense or experience, and using geometric modeling software to customize the shape, size and coordinate position of the object. If the artificial three-dimensional model is determined to be a cuboid, it is necessary to further determine the length, width and height of the cuboid and the coordinates of the eight vertices of the cuboid to obtain an artificial three-dimensional model of the target object.

[0039] Secondly, the non-artificial 3D model refers to the process of inputting multiple 2D images taken at different shooting angles, and using existing 3D reconstruction technologies, such as the depth estimation model and special point correction, to calculate the camera pose corresponding to each 2D image, and output the 3D point cloud model of the target object, and reconstruct the target object in the 3D point cloud model approximately, and adjust the shape of the object so that the size and position of the object are consistent with the actual state of the target object, so as to obtain the non-artificial 3D model of the target object. There is no restriction on which method is selected to generate the original 3D model.

[0040] S120, segmenting the original three-dimensional model to obtain a plurality of planar quadrilaterals, and determining a standard region mask corresponding to each planar quadrilateral in each two-dimensional image according to the type of the original three-dimensional model.

[0041] The segmentation process of the original three-dimensional model refers to the use of a large computer vision model to divide the surface of the original three-dimensional model into many closely connected quadrilateral faces, that is, multiple planar quadrilaterals. For the original three-dimensional models of two types, namely artificial three-dimensional models and non-artificial three-dimensional models, the region mask of each parallelogram is determined in different ways. Among them, the mask is a technology for specifying a specific area or pixel in an image, and is usually a matrix of the same size as the original image. The region mask in the embodiment of the present invention is a mask image with a marking value of 1 under the current parallelogram area.

[0042] For ease of understanding, the reference image of the two-dimensional image of the target object at a certain shooting angle is as follows Figure 2 As shown, the reference image of the original area mask corresponding to each plane quadrilateral identified according to the two-dimensional image is shown in Figure 3 .

[0043] Furthermore, according to the type of the original three-dimensional model, determining the standard region mask corresponding to each planar quadrilateral in each two-dimensional image may include:

[0044] If the generated three-dimensional model is an artificial three-dimensional model, directly identifying the original region masks corresponding to each planar quadrilateral in each two-dimensional image;

[0045] Each original region mask in each two-dimensional image is linearly calibrated to obtain a plurality of standard region masks.

[0046] In the embodiment of the present invention, since the size of the artificial three-dimensional model and the coordinate positions of each corner point are known, the original region mask of each plane quadrilateral can be directly obtained, and the original region mask ( Figure 3 ) performs straight line detection. The purpose of straight line detection is to correct the edge of the original area mask into a standard quadrilateral. The generated standard quadrilateral area is called the standard area mask.

[0047] Optionally, and according to the type of the original three-dimensional model, determining the standard area mask corresponding to each plane quadrilateral in each two-dimensional image may also include:

[0048] If the generated three-dimensional model is a non-artificial three-dimensional model, then in the original three-dimensional model, the three-dimensional coordinates of the corner points of each quadrilateral in each plane quadrilateral are obtained;

[0049] According to the shooting angle of each two-dimensional image, determine the camera intrinsic parameters and camera pose corresponding to each two-dimensional image;

[0050] According to the camera intrinsic parameters and camera poses corresponding to each two-dimensional image, the three-dimensional coordinates of each corner point of each plane quadrilateral are projected into each two-dimensional image to obtain the two-dimensional image coordinates of each corner point of the quadrilateral;

[0051] A plurality of standard region masks are obtained according to the two-dimensional image coordinates of each quadrilateral corner point projected in each two-dimensional image.

[0052] In an embodiment of the present invention, since the non-artificial three-dimensional model is generated by three-dimensional reconstruction technology, and the three-dimensional reconstruction technology relies on two-dimensional images at multiple shooting angles, when obtaining the standard area mask of the non-artificial three-dimensional model, the camera pose and camera intrinsic parameters at the shooting angle corresponding to each two-dimensional image should be determined first, and then the three-dimensional coordinates are converted into two-dimensional image coordinates. According to the two-dimensional coordinates of the four corner points under each planar quadrilateral, multiple standard area masks corresponding to each planar quadrilateral can be constructed.

[0053] S130 , calculating a homography transformation matrix for mapping from the two-dimensional image to the three-dimensional model according to each planar quadrilateral in the three-dimensional model and a standard region mask corresponding to each planar quadrilateral.

[0054] The homography transformation matrix is ​​used to describe the transformation relationship between two plane quadrilaterals. It is a 3×3 homogeneous matrix that maps each plane quadrilateral from a two-dimensional image to a three-dimensional model. That is, the homography transformation matrix can be used to project plane quadrilaterals at different viewing angles to the viewing angle of another plane, and the reconstruction of the three-dimensional model is completed by splicing all the planes.

[0055] S140, using each homography transformation matrix, converting the image pixels in each standard area mask into plane point cloud data, and obtaining a three-dimensional reconstructed model of the target object according to each of the plane point cloud data.

[0056] Point cloud data is a data type used to represent three-dimensional objects. Unlike images, which are ordered arrangements of pixels on a two-dimensional plane, point cloud data is an unordered collection in three-dimensional space, that is, the order of the points does not affect the shape representation of the overall point set. Planar point cloud data consists of several points in three-dimensional space, and a point contains at least coordinate values, normal coordinates, and color information of the point.

[0057] Among them, there is a homography transformation matrix between every two standard area masks, which is used to convert the two-dimensional pixels in one of the standard area masks into point cloud data in three-dimensional space. At this time, the point cloud data is assigned corresponding pixel colors, thereby obtaining a three-dimensional model consistent with the target object.

[0058] In an embodiment of the present invention, two-dimensional images of a target object taken at multiple shooting angles are obtained, and an original three-dimensional model of the target object is generated according to each of the two-dimensional images; the original three-dimensional model is segmented to obtain multiple plane quadrilaterals, and according to the type of the original three-dimensional model, a standard area mask corresponding to each plane quadrilateral is determined in each two-dimensional image; according to each plane quadrilateral in the three-dimensional model and the standard area mask corresponding to each plane quadrilateral, a homography transformation matrix for mapping from the two-dimensional image to the three-dimensional model is calculated; using each homography transformation matrix, the image pixel points in each standard area mask are converted into plane point cloud data, and according to each of the plane point cloud data, a three-dimensional reconstructed model of the target object is obtained. The technical solution of the embodiment of the present invention avoids the currently commonly used sparse reconstruction and dense reconstruction methods for reconstructing three-dimensional models, but instead converts the two-dimensional image into three-dimensional point cloud data by using the homography transformation matrix, thereby reconstructing the generated original three-dimensional model. The three-dimensional model reconstructed by the above method is not affected by camera posture and image distortion, and can output the three-dimensional model of the actual target object more stably.

[0059] Embodiment 2

[0060] Figure 4 This is a flow chart of another method for reconstructing a three-dimensional model of an object provided in the second embodiment of the present invention. This embodiment is based on the first embodiment and is refined. Figure 4 As shown, the method includes:

[0061] S410: Acquire two-dimensional images of the target object captured at multiple shooting angles, and generate an original three-dimensional model of the target object according to each of the two-dimensional images.

[0062] S420, segmenting the original three-dimensional model to obtain a plurality of planar quadrilaterals, and determining a standard region mask corresponding to each planar quadrilateral in each two-dimensional image according to the type of the original three-dimensional model.

[0063] S430: Acquire the target plane quadrilateral currently being processed, and acquire a target standard area mask corresponding to the target plane quadrilateral.

[0064] S440: For each corner point of the target plane quadrilateral, obtain the three-dimensional coordinates of the corner point on the three-dimensional model.

[0065] In an embodiment of the present invention, the standard area mask corresponding to the currently processed plane quadrilateral is first determined, and then the four target corner points of the target plane quadrilateral are extracted through the standard area mask. Combined with the two-dimensional image corresponding to the target plane quadrilateral, the positions of the four target corner points in the artificial or non-artificial three-dimensional model are determined, and then the three-dimensional position coordinates of the target corner points are determined.

[0066] S450, using the unit orthogonal vector group that matches the target plane quadrilateral, calculate the two-dimensional projection coordinates of the three-dimensional coordinates of each quadrilateral corner point on the two-dimensional plane.

[0067] The unit orthogonal vector group is composed of two unit direction vectors on the plane where the current target plane quadrilateral is located, which are perpendicular to the normal vector and perpendicular to each other, and can be expressed as and The three-dimensional coordinates of any target corner point are marked as (x, y, z), according to formula (1): The two-dimensional projection coordinates (x ′ ,y ′ ).

[0068] S460: Calculate a homography transformation matrix corresponding to the target standard region mask according to the two-dimensional image coordinates of each quadrilateral corner point in the target standard region mask and the two-dimensional projection coordinates of each quadrilateral corner point.

[0069] The homography transformation matrix is ​​a 3×3 matrix, denoted as H 3×3 , and when solving the homography transformation matrix, it is necessary to use the 2D image coordinates and 2D projection coordinates of the same target corner point to calculate the homography transformation matrix. The 2D image coordinates are the 2D coordinates of the target corner point in the standard region mask, recorded as (px, py), and the 2D projection coordinates are the projection coordinates (x) of the 3D coordinates of the target corner point obtained in S250 on the current plane. ′ ,y ′), both of which are expressions of the Cartesian coordinate system, that is, they only have two-dimensional representations. Therefore, it is necessary to increase the dimension of the two coordinates and add a homogeneous coordinate while ensuring that the coordinate value remains unchanged to make it three-dimensional, which will participate in the calculation without affecting the final result.

[0070] In the embodiment of the present invention, homogeneous coordinates are used to represent two-dimensional coordinates in three dimensions. Specifically, an additional variable can be added to the end of a two-dimensional coordinate to construct a homogeneous coordinate. For example, the two-dimensional image coordinate (px, py) becomes (px, py, 1) after dimension upgrading; the two-dimensional projection coordinate becomes (x ′ ,y ′ ,1), then use formula (2): The homography transformation matrix H of the target corner point in the plane can be obtained 3×3 .

[0071] Among them, if the camera internal parameter is recorded as K, the camera pose is recorded as w2c, and the three-dimensional coordinates of the target corner point under any plane quadrilateral in the original three-dimensional model are recorded as (x, y, z), then the coordinates of the two-dimensional image (px, py) satisfy formula (3): in,

[0072] S470 , returning to execute S230 , until the homography transformation matrix of each standard region mask corresponding to the target plane quadrilateral is determined.

[0073] Since the degree of freedom of the homography transformation matrix is ​​8, and one corner point can obtain two sets of equations according to the above formula (2), at least four points are required under a plane quadrilateral to obtain H 3×3 That is, by obtaining the 2D image coordinates and 2D projection coordinates of the four corner points of the target plane quadrilateral, a set of homography transformation matrices of the standard region mask corresponding to the target plane quadrilateral can be obtained. Return to perform the above operation until the homography transformation matrix of each standard region mask corresponding to the target plane quadrilateral is calculated.

[0074] S480 , obtaining a target standard region mask currently being processed, and obtaining a target homography transformation matrix corresponding to the target standard region mask.

[0075] S490, inversely generate plane point cloud data corresponding to the target standard area mask according to each image pixel point in the target standard area mask, the target homography transformation matrix, the unit orthogonal vector group and the unit normal vector.

[0076] When each homography transformation matrix is ​​determined, the two-dimensional projection coordinates of the target corner point on any plane quadrilateral in the three-dimensional model can be calculated according to formula (2), which is recorded as (x ′ ,y′ ), mark the three-dimensional coordinates of the target corner point as (x, y, z), and project the target corner point to the unit normal vector In, it must satisfy: Among them, z plane It can be understood as the distance from the plane where the point is located to the normal vector, which is a constant. plane After determination, by obtaining the normal vectors v1, v2, and v3 in three directions of the plane where the target corner point is located, it can be calculated according to formula (3): Reversely calculate the three-dimensional point cloud data of any point on the plane quadrilateral.

[0077] Optionally, after reversely generating the plane point cloud data corresponding to the target standard area mask, the method may further include:

[0078] Acquire a target two-dimensional image matching the target standard area mask;

[0079] According to the pixel color of each image pixel in the target standard area mask in the target two-dimensional image, a color value is assigned to each point cloud point in the plane point cloud data.

[0080] For ease of understanding, a reference diagram of the target standard area mask in an embodiment of the present invention is shown in Figure 3 , which is a grayscale image. By matching the target standard area mask with the target two-dimensional image, the color information of each pixel in the target area mask can be determined. After the pixel coordinates are converted into point cloud data using the homography transformation matrix, the color information carried in the point cloud data is extracted and the corresponding color is assigned to each point cloud point, and finally the reconstructed three-dimensional model of the target object is obtained.

[0081] In an embodiment of the present invention, different ways of determining standard area masks for artificial three-dimensional models and non-artificial three-dimensional models are first described. First, the two-dimensional coordinates of the four corner points in each plane quadrilateral are obtained and upgraded to homogeneous coordinates to calculate the homography transformation matrix. Then, the homography transformation matrix is ​​used to inversely generate the plane point cloud data corresponding to the target standard area mask. Finally, by assigning colors to the pixel points, a three-dimensional model that fits the target object is obtained. The embodiment of the present invention provides a three-dimensional reconstruction method based on two original three-dimensional models under an artificial three-dimensional model and a non-artificial three-dimensional model for the target object. Compared with the existing model reconstruction method, it does not require a large amount of sample data. While efficiently completing the three-dimensional model reconstruction of the target object, it maximizes the processing accuracy and reconstruction effect.

[0082] Embodiment 3

[0083] Figure 5 This is a schematic diagram of the structure of a three-dimensional model reconstruction device for an object provided in Embodiment 3 of the present invention. Figure 5 As shown, the device comprises:

[0084] The original 3D model generation module 510 is used to obtain 2D images of the target object taken at multiple shooting angles, and generate an original 3D model of the target object according to each of the 2D images;

[0085] The standard region mask determination module 520 is used to segment the original three-dimensional model to obtain a plurality of planar quadrilaterals, and determine the standard region mask corresponding to each planar quadrilateral in each two-dimensional image according to the type of the original three-dimensional model;

[0086] A homography transformation matrix calculation module 530, configured to calculate a homography transformation matrix for mapping from a two-dimensional image to a three-dimensional model according to each planar quadrilateral in the three-dimensional model and a standard region mask corresponding to each planar quadrilateral;

[0087] The three-dimensional reconstruction model building module 540 is used to convert the image pixels in each standard area mask into plane point cloud data using each homography transformation matrix, and obtain the three-dimensional reconstruction model of the target object based on each plane point cloud data.

[0088] In an embodiment of the present invention, the original three-dimensional model generation module 510 obtains two-dimensional images of the target object taken at multiple shooting angles, and generates an original three-dimensional model of the target object based on each of the two-dimensional images; the standard area mask determination module 520 segments the original three-dimensional model to obtain multiple planar quadrilaterals, and determines the standard area masks corresponding to each planar quadrilateral in each two-dimensional image according to the type of the original three-dimensional model; the homography transformation matrix calculation module 530 calculates the homography transformation matrix for mapping from the two-dimensional image to the three-dimensional model according to each planar quadrilateral in the three-dimensional model and the standard area masks corresponding to each planar quadrilateral; the three-dimensional reconstruction model construction module 540 uses each homography transformation matrix to convert the image pixels in each standard area mask into planar point cloud data, and obtains the three-dimensional reconstructed model of the target object according to each of the planar point cloud data. The technical solution of the embodiment of the present invention avoids the currently commonly used sparse reconstruction and dense reconstruction methods for reconstructing three-dimensional models, and instead converts a two-dimensional image into three-dimensional point cloud data by utilizing a homography transformation matrix, thereby reconstructing the generated original three-dimensional model. The three-dimensional model reconstructed using the above method is not affected by camera pose and image distortion, and can output a three-dimensional model of the actual target object more stably.

[0089] Based on the above embodiments, the original 3D model generation module 510 includes:

[0090] An artificial three-dimensional model construction unit, used to perform a custom construction of a three-dimensional model of a target object in each two-dimensional image based on prior knowledge to obtain an artificial three-dimensional model of the target object; or

[0091] The non-artificial three-dimensional model construction unit is used to generate a three-dimensional point cloud model of the target object according to each of the two-dimensional images using three-dimensional reconstruction technology, and adjust the three-dimensional point cloud model using an approximate reconstruction method to obtain a non-artificial three-dimensional model of the target object.

[0092] Based on the above embodiments, the standard area mask determination module 520 includes:

[0093] an original region mask identification unit, configured to directly identify original region masks corresponding to respective planar quadrilaterals in respective two-dimensional images if the generated three-dimensional model is an artificial three-dimensional model;

[0094] The linear calibration unit is used to perform linear calibration on each original region mask in each two-dimensional image to obtain a plurality of standard region masks.

[0095] Based on the above embodiments, the standard area mask determination module 520 further includes:

[0096] A first three-dimensional coordinate acquisition unit, configured to acquire the three-dimensional coordinates of corner points of each quadrilateral in each plane quadrilateral in the original three-dimensional model if the generated three-dimensional model is a non-artificial three-dimensional model;

[0097] A camera posture determination unit, used to determine the camera intrinsic parameters and camera posture corresponding to each two-dimensional image according to the shooting angle of each two-dimensional image;

[0098] A two-dimensional image coordinate determination unit is used to project the three-dimensional coordinates of each corner point of each quadrilateral in each plane quadrilateral into each two-dimensional image according to the camera intrinsic parameters and camera pose corresponding to each two-dimensional image, so as to obtain the two-dimensional image coordinates of each corner point of the quadrilateral;

[0099] The standard region mask generating unit is used to obtain a plurality of standard region masks according to the two-dimensional image coordinates of each quadrilateral corner point projected in each two-dimensional image.

[0100] Based on the above embodiments, the homography transformation matrix calculation module 530 includes:

[0101] A target standard region mask acquisition unit, used to acquire a target plane quadrilateral currently being processed, and acquire a target standard region mask corresponding to the target plane quadrilateral;

[0102] A second three-dimensional coordinate acquisition unit, configured to acquire, for each quadrilateral corner point of the target plane quadrilateral, the three-dimensional coordinates of the quadrilateral corner point on the three-dimensional model;

[0103] A two-dimensional projection coordinate calculation unit, used to calculate the two-dimensional projection coordinates of the three-dimensional coordinates of each quadrilateral corner point on the two-dimensional plane using a unit orthogonal vector group matching the target plane quadrilateral;

[0104] A homography transformation matrix solving unit, used for calculating a homography transformation matrix corresponding to the target standard area mask according to the two-dimensional image coordinates of each quadrilateral corner point in the target standard area mask and the two-dimensional projection coordinates of each quadrilateral corner point;

[0105] The loop judgment unit is used to return to execute the operation of obtaining a target standard area mask corresponding to the target plane quadrilateral until the homography transformation matrix of each standard area mask corresponding to the target plane quadrilateral is determined.

[0106] Based on the above embodiments, the 3D reconstruction model building module 540 includes:

[0107] A target homography transformation matrix acquisition unit, used to acquire a target standard region mask currently being processed, and acquire a target homography transformation matrix corresponding to the target standard region mask;

[0108] The plane point cloud data generating unit is used to reversely generate the plane point cloud data corresponding to the target standard area mask according to each image pixel point, the target homography transformation matrix, the unit orthogonal vector group and the unit normal vector in the target standard area mask.

[0109] On the basis of the above embodiments, it further includes: a color assignment unit, which is used to obtain a target two-dimensional image matching the target standard area mask after reversely generating the plane point cloud data corresponding to the target standard area mask;

[0110] According to the pixel color of each image pixel in the target standard area mask in the target two-dimensional image, a color value is assigned to each point cloud point in the plane point cloud data.

[0111] A three-dimensional model reconstruction device for an object provided in an embodiment of the present invention can execute a three-dimensional model reconstruction method for an object provided in any embodiment of the present invention, and has corresponding functional modules and beneficial effects of the execution method.

[0112] Embodiment 4

[0113] Figure 6A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0114] like Figure 6 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0115] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0116] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a method for reconstructing a three-dimensional model of an object.

[0117] That is: obtaining two-dimensional images of the target object taken at multiple shooting angles, and generating an original three-dimensional model of the target object according to each of the two-dimensional images;

[0118] The original three-dimensional model is segmented to obtain a plurality of plane quadrilaterals, and a standard region mask corresponding to each plane quadrilateral is determined in each two-dimensional image according to the type of the original three-dimensional model;

[0119] According to each planar quadrilateral in the three-dimensional model and the standard region mask corresponding to each planar quadrilateral, a homography transformation matrix for mapping from the two-dimensional image to the three-dimensional model is calculated;

[0120] The image pixels in each standard region mask are converted into plane point cloud data by using each homography transformation matrix, and a three-dimensional reconstructed model of the target object is obtained based on each plane point cloud data.

[0121] In some embodiments, a method for reconstructing a three-dimensional model of an object may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the method for reconstructing a three-dimensional model of an object described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute a method for reconstructing a three-dimensional model of an object in any other appropriate manner (e.g., by means of firmware).

[0122] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0123] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0124] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0125] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0126] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0127] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS servers.

[0128] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0129] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for reconstructing a three-dimensional model of an object, characterized in that: include: Acquire two-dimensional images of the target object taken at multiple shooting angles, and generate an original three-dimensional model of the target object according to each of the two-dimensional images; The original three-dimensional model is segmented to obtain a plurality of plane quadrilaterals, and a standard region mask corresponding to each plane quadrilateral is determined in each two-dimensional image according to the type of the original three-dimensional model; According to each planar quadrilateral in the three-dimensional model and the standard region mask corresponding to each planar quadrilateral, a homography transformation matrix for mapping from the two-dimensional image to the three-dimensional model is calculated; The image pixels in each standard region mask are converted into plane point cloud data by using each homography transformation matrix, and a three-dimensional reconstructed model of the target object is obtained according to each plane point cloud data.

2. The method according to claim 1, characterized in that Generating an original three-dimensional model of the target object according to each of the two-dimensional images includes: Based on prior knowledge, a three-dimensional model of the target object in each two-dimensional image is constructed by a custom method to obtain an artificial three-dimensional model of the target object; or According to each of the two-dimensional images, a three-dimensional point cloud model of the target object is generated using a three-dimensional reconstruction technology, and the three-dimensional point cloud model is adjusted using an approximate reconstruction method to obtain a non-artificial three-dimensional model of the target object.

3. The method according to claim 2, characterized in that And according to the type of the original three-dimensional model, a standard area mask corresponding to each plane quadrilateral is determined in each two-dimensional image, including: If the generated three-dimensional model is an artificial three-dimensional model, directly identifying the original region masks corresponding to each planar quadrilateral in each two-dimensional image; Each original region mask in each two-dimensional image is linearly calibrated to obtain a plurality of standard region masks.

4. The method according to claim 2, characterized in that: And according to the type of the original three-dimensional model, a standard area mask corresponding to each plane quadrilateral is determined in each two-dimensional image, including: If the generated three-dimensional model is a non-artificial three-dimensional model, then in the original three-dimensional model, the three-dimensional coordinates of the corner points of each quadrilateral in each plane quadrilateral are obtained; According to the shooting angle of each two-dimensional image, determine the camera intrinsic parameters and camera pose corresponding to each two-dimensional image; According to the camera intrinsic parameters and camera poses corresponding to each two-dimensional image, the three-dimensional coordinates of each corner point of each quadrilateral in each plane quadrilateral are projected into each two-dimensional image to obtain the two-dimensional image coordinates of each corner point of the quadrilateral; A plurality of standard region masks are obtained according to the two-dimensional image coordinates of each quadrilateral corner point projected in each two-dimensional image.

5. The method according to claim 1, characterized in that According to each planar quadrilateral in the three-dimensional model and the standard region mask corresponding to each planar quadrilateral, a homography transformation matrix for mapping from the two-dimensional image to the three-dimensional model is calculated, including: Obtaining a target plane quadrilateral currently being processed, and obtaining a target standard area mask corresponding to the target plane quadrilateral; For each quadrilateral corner point of the target plane quadrilateral, obtaining the three-dimensional coordinates of the quadrilateral corner point on the three-dimensional model; Using a unit orthogonal vector group matching the target plane quadrilateral, calculate the two-dimensional projection coordinates of the three-dimensional coordinates of each quadrilateral corner point on the two-dimensional plane; Calculate a homography transformation matrix corresponding to the target standard region mask according to the two-dimensional image coordinates of each quadrilateral corner point in the target standard region mask and the two-dimensional projection coordinates of each quadrilateral corner point; Return to execute the operation of obtaining a target standard region mask corresponding to the target plane quadrilateral, until the homography transformation matrix of each standard region mask corresponding to the target plane quadrilateral is determined.

6. The method according to claim 1, characterized in that Using each homography transformation matrix, the image pixels within each standard region mask are converted into planar point cloud data, including: Obtaining a target standard region mask currently being processed, and obtaining a target homography transformation matrix corresponding to the target standard region mask; Plane point cloud data corresponding to the target standard area mask is inversely generated according to each image pixel point in the target standard area mask, the target homography transformation matrix, the unit orthogonal vector group and the unit normal vector.

7. The method according to claim 6, characterized in that After reversely generating the plane point cloud data corresponding to the target standard area mask, the method further includes: Acquire a target two-dimensional image matching the target standard area mask; According to the pixel color of each image pixel in the target standard area mask in the target two-dimensional image, a color value is assigned to each point cloud point in the plane point cloud data.

8. A device for reconstructing a three-dimensional model of an object, characterized in that: include: An original three-dimensional model generation module is used to obtain two-dimensional images of the target object taken at multiple shooting angles, and generate an original three-dimensional model of the target object according to each of the two-dimensional images; A standard region mask determination module is used to segment the original three-dimensional model to obtain a plurality of planar quadrilaterals, and determine the standard region mask corresponding to each planar quadrilateral in each two-dimensional image according to the type of the original three-dimensional model; A homography transformation matrix calculation module, used to calculate a homography transformation matrix for mapping from a two-dimensional image to a three-dimensional model according to each planar quadrilateral in the three-dimensional model and a standard region mask corresponding to each planar quadrilateral; The three-dimensional reconstruction model building module is used to convert the image pixels in each standard area mask into plane point cloud data using each homography transformation matrix, and obtain the three-dimensional reconstruction model of the target object based on each plane point cloud data.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute a method for reconstructing a three-dimensional model of an object as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement a three-dimensional model reconstruction method of an object according to any one of claims 1 to 7 when executed.

Images