Method carried out by a computer for generating a 3D model using low GSD images

Abstract

A method carried out by a computer for generating a 3D model of an object, the method (100) comprising: obtaining a plurality of first 2D images (2A, 2B) which reproduce respective parts of the object (200), obtaining one or more groups of second 2D images (3A, 3B), wherein for each group of second images: the second images of the same group reproduce respective zones of a portion of the object, each second image being at least partially coincident with another second image of the same group, the second images have a respective second ground sample distance which is less than the ground sample distances of the first images, and at least one of the first images and at least one of the second images of the same group reproduce the same element (ei) of the object. For each group of second images, the method comprises: determining a scaling factor between the first images and the second images of the same group on the basis of the element which is reproduced in at least one of the first images and at least one of the second images of this group, resizing the second images of the group of second images by applying the scaling factor and generating a third image (3) which reproduces the portion of the object by merging the second resized images by means of a stitching algorithm which comprises a transformation function which correlates the points of the second resized images with the points of the third image. The method further comprises storing in a data-storage unit (4) the third image of each group of second images in addition to the plurality of first images, and generating a 3D model (5) of the object in a virtual space by means of a photogrammetry technique on the basis of the plurality of first images and the third image of each group of second images.

Description

[0001] Method carried out by a computer for generating a 3D model using low GSD images Description

[0002] The present invention relates to a method carried out by a computer for generating a 3D model of an object and related system, computer program and computer-readable medium.

[0003] The invention finds particular, though not exclusive, application in the technical field relevant to the reconstruction of 3D representations of scenes, such as a building construction, from two-dimensional images using photogrammetry techniques.

[0004] Generally, said two-dimensional images are captured by cameras with a relatively large field of view and have sufficient overlap to effectively reconstruct the 3D representation of a scene. These cameras are generally positioned relatively far away from the scene to be reconstructed.

[0005] The Applicant has found that two-dimensional images obtained from cameras located relatively far away from the scene to be reconstructed generally suffer from poor or otherwise unsatisfactory definition in order for a detailed analysis of one or more details of the scene to be carried out.

[0006] In particular, the Applicant has noted that the impossibility of carrying out detailed image analyses would result in the inability to carry out assessments of the conditions of an object in a scene, such as the state of preservation / deterioration of a pile of a bridge or overpass. Patent application US 20170289447 Al discloses a method and apparatus for creating a 3D model by means of high-definition images obtained from a first camera with a first angle of view, and panoramic images obtained from a second camera with a second angle of view greater than the first angle, as well as by means of data on the position and orientation of the first camera. The technical solution described in US 20170289447 also involves the use of an aircraft comprising both the first and second camera.

[0007] However, the Applicant noted that the technical solution described in US 20170289447 requires the use of a specific device comprising two different types of cameras, as well as systems designed to provide positioning / orientation information for the cameras, which inevitably increases the complexity of the solution itself in terms of implementation cost, data-storage space and computation.

[0008] Furthermore, the need to have information on the positioning / orientation of the cameras prevents the use of the solution described in US 20170289447 in those contexts where such information is not available or not sufficiently precise.

[0009] Patent applications US 2019 / 026955 Al and US 2021 / 264666 Al constitute further disclosures pertaining to the technical scope of the present invention. In detail, patent application US 2019 / 026955 Al concerns an image processing method, display device and inspection system, while patent application US2021 / 264666 Alconcerns a method and system for collecting photogrammetric data and processing the data to generate 3D models.

[0010] Further prior art relevant to the technical scope of the present invention is the following article: NOAH SNAVELY ET AL: "Scene Reconstruction and Visualization From Community Photo Collections", PROCEEDINGS OF THE IEEE, IEEE; NEW YORK, US, vol. 98, n. 8, 1 August 2010, pages 1370-1390, XP011311248, ISSN: 0018-9219.

[0011] The Applicant therefore understood the opportunity to develop a method carried out by a computer, a system, a computer program and a computer-readable medium for generating a 3D model correlated to detailed images without the use of camera positioning and / or orientation information.

[0012] The technical problem underlying the present invention is thus to provide a method carried out by a computer for generating a 3D model, a system, a computer program and a computer-readable medium that are structurally and functionally conceived to at least partially obviate one or more of the drawbacks complained of with reference to the aforementioned prior art. Within the scope of this problem, a purpose of the present invention is to provide a method carried out by a computer, a system, a computer program and a computer-readable medium that enables the generation of a 3D model correlated to detailed images.

[0013] Within the scope of this problem, a purpose of the present invention is to provide a computer-implemented method for generating a 3D model, system, computer program and computer-readable medium having a broader field of application than known solutions.

[0014] This problem is solved and at least one of these purposes is at least partially achieved by the invention by means of a method carried out by a computer for generating a 3D model, a system, a computer program and a computer-readable medium according to one or more of the respective appended claims.

[0015] In the present disclosure, as well as in the appended claims, certain terms and expressions are deemed to have, unless otherwise expressly indicated, the meaning expressed in the following definitions.

[0016] The term "3D model" refers to a three-dimensional model.

[0017] The term "2D image" refers to a two-dimensional image.

[0018] The term "ground sample distance", GSD, refers to the distance, expressed for example in mm, between the centre of two adjacent pixels measured on an object in a 2D image, particularly a photo.

[0019] The term "definition" of an object / portion of an object / element reproduced in an image refers to the degree of accuracy with which the true relative distances of any portion of the image can be measured. It should be noted that greater image definition implies a better ability to measure such distances.

[0020] In particular, a first image with a higher GSD than a second image has a lower definition than the second image.

[0021] The expression "resizing to scale" means reducing the size of, for example, an image while maintaining its proportions.

[0022] In a first aspect thereof, the present invention concerns a method carried out by a computer for generating a 3D model of an object. In the following, the method carried out by a computer will also be referred to as "method" for simplicity.

[0023] Preferably, the object corresponds to a building construction, or a part thereof, such as: a pile or girder of a bridge or overpass, or a wall of a tunnel or building.

[0024] The method involves obtaining a plurality of first 2D images (also referred to as "first images" for simplicity) which reproduce respective parts of the object. Each first image is at least partially coincident with another first image.

[0025] Preferably, a first image coincides with another first image at least by 30%, more preferably by 50%.

[0026] Preferably, a first image coincides with another first image at least by 50% along a first direction and at least by 30% along a second direction perpendicular to the first direction. Preferably, the first direction coincides with a horizontal direction while the second direction coincides with a vertical direction, or vice versa.

[0027] Each first image also has a relative first ground sample distance.

[0028] Preferably, the first images are captured by a first device equipped with a camera, the latter preferably having a wide-angle lens.

[0029] Preferably, the plurality of first images is such that the object is reproduced in full.

[0030] The method also comprises obtaining one or more groups of second 2D images (also referred to as "second images" for simplicity).

[0031] For each group of second images:

[0032] • second images of the same group reproduce respective areas of a portion of the object, each second image being at least partially coincident with another second image of the same group,

[0033] • the second images have a respective second ground sample distance which is less than the first ground sample distances of the first images, and

[0034] • at least one of the first images and at least one of the second images in the same group reproduce the same element of the object.

[0035] Thus, the portion of the object represented by each group of second images comprises a relative element reproduced by at least one of the first images and at least one of the second images.

[0036] Preferably, the portion of the object corresponds to a circumscribed surface of the object itself.

[0037] Preferably, different groups of second images represent different respective portions of the object.

[0038] Preferably, the portion of the object represented by each group of second images is also represented by the plurality of first images.

[0039] It should be noted that the second images, having a respective second ground sample distance which is less than the first ground sample distances of the first images, allow a portion of the object to be reproduced at a higher definition than the same portion reproduced by the first images.

[0040] By way of example, the second ground sample distance is less than or equal to 0.2 mm / px. Preferably, the second ground sample distance of a second image is at least an order of magnitude smaller than the first ground sample distance of a first image.

[0041] Preferably, a second image coincides with another second image at least by 30%, more preferably by 40%.

[0042] Preferably, a second image coincides with another second image by at least 40% along a first direction and by at least 40% along a second direction perpendicular to the first direction. Preferably, the first direction coincides with a horizontal direction while the second direction coincides with a vertical direction, or vice versa.

[0043] Preferably, the second images are captured by a second device provided with a camera. Preferably, the method comprises selecting the aforementioned element from at least one of the first images and at least one of the second images.

[0044] Element selection can be carried out manually via an input sent by a user to a data- processing device, or automatically by the data-processing device itself.

[0045] The identification of the common element can be carried out through the execution by a data-processing device of a known algorithm based on techniques of recognising objects in images and comparing them between different images until a common element is identified. An example of a recognition and comparison technique is the Siamese network. If the object is a building construction, the above-mentioned element may correspond to an element added to the building construction itself, such as a drainage joint, graffiti, etc For each group of second images, the method comprises:

[0046] • determining a scaling factor between the first images and the second images of the same group on the basis of the element which is reproduced in at least one of the first images and at least one of the second images of this group,

[0047] • resizing the second images of said group of second images by applying said scaling factor, and

[0048] • generating a third image which reproduces the portion of the object by merging the second resized images by means of a stitching algorithm which comprises a transformation function which correlates the points of the second resized images with the points of the third image.

[0049] Therefore, the method involves generating a third image (two-dimensional image) for each group of second images, wherein the third image reproduces the portion of the object represented by the respective group of second images.

[0050] The portion of the object reproduced by each third image has greater definition than that of the same portion reproduced by the first images due to the fact that the second images have a respective second ground sample distance which is less than the first ground sample distances of the first images.

[0051] Stitching algorithms are well known in the literature and will therefore not be detailed below.

[0052] The method also comprises:

[0053] • storing in a data-storage unit the third image of each group of second images in addition to the plurality of first images, and

[0054] • generating a 3D model of the object in a virtual space by means of a photogrammetry technique on the basis of the plurality of first images and the third image of each group of second images.

[0055] Preferably, the photogrammetry technique comprises at least one of Meshroom, COLMAP, Pix4D and Metashape.

[0056] Preferably, the method comprises storing the 3D model in said data-storage unit.

[0057] The above method therefore makes it possible to obtain a relationship between the 3D model of the object and the third image of each group of second images.

[0058] In fact, the third images were obtained by applying a stitching algorithm to the second images, the second images having a ground sample distance which is less than the first images.

[0059] In other words, the method according to the invention allows to achieve a correlation between the second images and the 3D model obtained by stitching and using the third image in the photogrammetric reconstruction. An analysis performed on the second images can then be projected onto the 3D model.

[0060] The method according to the invention preferably comprises determining the position and / or orientation of one or more cameras to capture and generate each of the second images based on the correlation between the second images and the 3D model.

[0061] Information on the positioning and / or orientation of the cameras is thus determined and not required as input data to the method as is the case with the solutions described in the prior art.

[0062] This allows to avoid the use of devices designed to generate positioning and / or orientation information for cameras. As a result, the field of application of the method according to the invention is broadened with respect to known solutions, since the method according to the invention enables the generation of a 3D model on the basis of detailed images and not also of objects for which the precise positioning and / or orientation of the cameras cannot be obtained by means of known devices, for example due to adverse environmental conditions.

[0063] In a second aspect thereof, the present invention concerns a system comprising a data- processing device configured to carry out the method according to the invention.

[0064] The system also comprises a data-storage unit operatively connected to the data- processing device to store the 3D model of the object generated by carrying out the method.

[0065] In a third aspect thereof, the present invention concerns a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method according to the invention.

[0066] In a fourth aspect thereof, the present invention concerns a computer-readable medium comprising instructions which, when executed by a computer, cause the computer to carry out the method according to the invention.

[0067] In at least one of the above aspects, the present invention may also have at least one of the preferred features set out hereinafter.

[0068] In at least one embodiment of the present invention, the method comprises obtaining the position, preferably the Cartesian position, in particular expressed in pixels, of a circumscribed area within a second image belonging to a group of said one or more groups of second 2D images.

[0069] Preferably, the pixel position of the circumscribed area is determined with respect to a reference system fixed relative to the relevant second image.

[0070] The method can involve identifying the circumscribed area within a second image, e.g. automatically or manually, before determining its position.

[0071] In at least one embodiment of the present invention, the method comprises:

[0072] • determining the position of the circumscribed area in the third image which is generated for the group of second 2D images which contain the circumscribed area, on the basis of the relevant scaling factor and transformation function,

[0073] • determining the position of the circumscribed area in virtual space by means of a ray casting technique on the basis of the 3D model and the position of the circumscribed area in the third image,

[0074] • storing in the data-storage unit the position of the circumscribed area in virtual space and / or displaying by means of a display the 3D model and the circumscribed area in virtual space.

[0075] Ray casting techniques are well known in the literature and will therefore not be detailed below.

[0076] These features allow the circumscribed area of a second 2D image to be associated with the virtual space of the 3D model.

[0077] In addition, storing the position of the circumscribed area in virtual space and / or displaying the 3D model and the circumscribed area in virtual space makes information about the circumscribed area with respect to the virtual space available for subsequent analysis.

[0078] According to an embodiment of the invention, the second images of at least one group of second images (preferably of each group of second images) have the same second ground sample distance.

[0079] This feature is particularly useful for simplifying the determination of the scaling factor used to generate the third image.

[0080] In at least one embodiment of the present invention, the step of determining the scaling factor comprises scaling a second image reproducing the aforementioned element and comparing the dimensions of the element reproduced in the second scaled image with those of the element reproduced in a first image.

[0081] Preferably, this resizing is repeated until the dimensions of the element reproduced in the second image coincide with those of the element reproduced in a first image, so that the element in the second image preferably overlaps perfectly with the same element contained in a first image.

[0082] In at least one embodiment of the present invention, the scaling factor corresponds to the scale resizing factor of a second image reproducing said element such that the element reproduced by said second image has the same dimensions as the element reproduced in a first image.

[0083] This provides the scaling factor to be applied to the second images of the relevant group of second images, so that dimensional consistency between the resized second images and the first images is achieved.

[0084] In at least one embodiment of the present invention, the method comprises geometrically projecting the circumscribed area onto the 3D model.

[0085] This provides information on the position of the circumscribed area on the 3D model, thus making it possible to find out on which element of the 3D model the circumscribed area is projected and deduce other information from this relationship.

[0086] In the event that the circumscribed area identifies a crack, the projection of this area in the space of the 3D model makes it possible, for example, to determine whether this crack belongs to a bridge girder and whether it could jeopardise the stability of the latter.

[0087] In at least one embodiment of the present invention, the step of determining the position of the circumscribed area in the third image comprises resizing said circumscribed area by applying the scaling factor and correlating the points of the resized circumscribed area to the points of the third image by means of the above-described transformation function. Preferably, the scaling factor applied corresponds to that determined for the group of second images comprising the circumscribed area.

[0088] The characteristics and advantages of the present invention will be better understood from the detailed description of preferred embodiments thereof, illustrated by way of nonlimiting example with reference to the accompanying drawings, in which:

[0089] • Figure 1 is a representative flowchart of a method carried out by a computer for generating a 3D model of an object according to an embodiment of the invention,

[0090] • Figures 2A and 2B schematically show two first images of an object corresponding to a pile of an overpass,

[0091] • Figures 3A and 3B schematically show two second images of a group of second images reproducing respective areas of a portion of the object depicted in Figures 2A and 2B,

[0092] • Figure 4 schematically shows a first image of the object depicted in Figures 2A and 2B comprising an element corresponding to a portion of a rainwater drain pipe,

[0093] • Figure 5 schematically shows a third image including the portion of the rainwater drain pipe shown in Figure 4,

[0094] • Figure 6 schematically shows a 3D model generated by the method in Figure 1,

[0095] • Figure 7 is a flowchart representing further steps of the method in Figure 1,

[0096] • Figure 8 schematically shows the determination of a circumscribed area within a second image and its projection onto a 3D model, and

[0097] • Figure 9 is a schematic view of a system according to an embodiment of the invention.

[0098] With reference to the appended figures, 100 denotes a method carried out by a computer for generating a three-dimensional model of an object 200.

[0099] The object 200 shown in the figures corresponds to a pile of an overpass.

[0100] With particular reference to Figure 1, the method 100 comprises a step 101 of obtaining a plurality of first two-dimensional images 1A,1B,„. reproducing respective parts of the object 200, each first image being at least partially coincident with another first image. Figures 2A and 2B show two first images 1A,1B of the object 200.

[0101] The method 100 further comprises a step 102 of obtaining N groups of second two- dimensional images 2A,2B,..., N being greater than 1, wherein the i-th group of second images satisfies the following characteristics, for each i from 1 to N:

[0102] • the second images 2A,2B,„. of the i-th group reproduce respective zones of an i-th portion of the object 200, each second image being at least partially coincident with another second image of the i-th group,

[0103] • the second images 2A,2B,... have the same ground sample distance, which is smaller than the ground sample distance of the first images 1A,1B,..., and

[0104] • at least one of the first images 1A,1B,... and at least one of the second images 2A,2B,... of the i-th group reproduce the same element Ei of the object 200.

[0105] Figures 3A and 3B show two second images 2A,2B of a group of second images.

[0106] The element Ei is preferably easily identifiable between the second images 2A,2B,... and may correspond, for example, to the portion of a rainwater drain pipe enclosed in the box shown in Figure 4.

[0107] For each i-th group of second images, the method 100 further comprises a step 103 in which a scaling factor Qi is determined between the first images 1A,1B,... and the second images 2A,2B,... of the i-th group based on the element Ei. Step 103 also involves resizing these second images 2A,2B,... by applying the scaling factor Qi.

[0108] In detail, step 103 may comprise resizing down to scale the second image reproducing the element Ei and comparing the dimensions of the element Ei reproduced in said second image with those of the element Ei reproduced in a first image, wherein the scaling factor Qi corresponds to the scale resizing factor of the second image reproducing the element Ei such that the element reproduced by such second image has the same dimensions as the element Ei reproduced in a first image.

[0109] In addition, for each i-th group of second images, the method 100 comprises a step 104 in which a third image 3 is generated which reproduces the i-th portion of the object 200 by combining the second images resized by a stitching algorithm. The stitching algorithm comprises a relevant transformation function Ti that correlates the points of the second resized images to the points of the third image 3.

[0110] Figure 5 shows an example of the third image 3 including the element ej corresponding to the portion of the rainwater drain pipe in Figure 4.

[0111] Thus, the method 100 comprises a step 105 of storing in a data-storage unit 4 the third image 3 of each group of second images 2A,2B,... in addition to the plurality of first images 1A,1B,..., and a step 106 of generating a three-dimensional model 5 of the object 200 in a virtual space by means of a photogrammetry technique on the basis of the plurality of first images 1A,1B,... and the third image 3 of each group of second images 2A,2B,....

[0112] It should be noted that the GSD (ground sample distance) of the first images does not necessarily have to be the same as these images do not participate in the stitching step, which is simplified by images taken all at the same distance and with the same resolution, but only in the photogrammetry step for creating the 3D model.

[0113] Figure 6 shows the three-dimensional model 5 in which the real cameras are also shown, arranged according to the position and orientation taken to capture the first images 1A,1B,..., as well as the virtual cameras arranged according to a position and orientation such that the third images 3 can be generated.

[0114] Indeed, using known photogrammetry techniques, it is possible to estimate with great accuracy the position and orientation of both the real cameras that captured the first images and the virtual cameras that generated the third images 3.

[0115] With particular reference to Figure 7, the method 100 may further comprise the step 107 of obtaining the position of a circumscribed area 6 within a second image belonging to one of the N groups of second two-dimensional images. This step preferably involves determining the circumscribed area 6, e.g. concrete spalling on pile of an overpass or a crack, and then calculating its position. The position of the circumscribed area can be indicated by a polygon, i.e. by a series of vertices ordered in the reference system of the second image.

[0116] Subsequently, the method 100 may further comprise a step 108 in which the position of the circumscribed area 6 in the third image 3 generated for the group of second images containing the circumscribed area is determined, based on the relative scaling factor Qi and transformation function Xi.

[0117] In particular, in order to determine the position of the circumscribed area 6 in the third image 3, step 108 involves resizing the circumscribed area 6 by applying the scaling factor Qi-

[0118] In a step 109, the method 100 can thus relate the points of the rescaled circumscribed area to the points of the third image 3 by means of the transformation function Xi.

[0119] The method 100 can determine in a step 110 the position of the circumscribed area 6 in virtual space by means of a ray casting technique on the basis of the three-dimensional model 5 and the position of the circumscribed area 6 in the third image 3. It should be noted that the ray casting technique uses the position and orientation of a virtual camera that would have been able to generate the third image 3.

[0120] In step 110, the method 100 may store in the data-storage unit 4 the position of the circumscribed area 6 in the virtual space and / or display by means of a display 7 the three- dimensional model 5 and the circumscribed area 6 in the virtual space, in particular by geometrically projecting the circumscribed area 6 onto the three-dimensional model 5.

[0121] Figure 8 schematically shows the determination of the circumscribed area 6, its resizing by means of the scale factor Qi, the identification of the position of the circumscribed area 6 in the third image 3 and the projection of the circumscribed area 6 onto the three- dimensional model 5.

[0122] Figure 9 schematically shows a system 300 comprising a data-processing device 8 configured to execute the method 100 and the data-storage unit 4 operatively connected to the data-processing device 8 to store the three-dimensional model of the object 200 generated by the execution of the method 100.

[0123] Preferably, the system 300 also comprises the display 7 through which the three- dimensional model 5 and the circumscribed area 6 in virtual space can be visualized.

[0124] Finally, according to the invention there is provided a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method 100 as well as a computer-readable medium comprising instructions which, when executed by a computer, cause the computer to carry out the method 100.

Claims

Claims1. A method carried out by a computer for generating a 3D model of an object, the method (100) comprising:• obtaining a plurality of first 2D images (2A, 2B) which reproduce respective portions of the object (200), each first image being at least partially coincident with another first image and having a relevant first ground sample distance,• obtaining one or more groups of second 2D images (3A, 3B), wherein for each group of second images:• the second images of the same group reproduce respective zones of a portion of the object, each second image being at least partially coincident with another second image of the same group,• the second images have a respective second ground sample distance which is less than the first ground sample distances of the first images, and• at least one of the first images and at least one of the second images of the same group reproduce the same element (ei) of said object, for each group of second images, the method comprising:• determining a scaling factor between the first images and the second images of the same group on the basis of the element which is reproduced in at least one of the first images and at least one of the second images of this group,• resizing the second images of the group of second images by applying the scaling factor and• generating a third image (3) which reproduces the portion of the object by merging the second resized images by means of a stitching algorithm which comprises a transformation function which correlates the points of the second resized images with the points of the third image, the method further comprising:• storing in a data-storage unit (4) the third image of each group of second images in addition to the plurality of first images, and• generating a 3D model (5) of the object in a virtual space by means of a photogrammetry technique on the basis of the plurality of first images and the third image of each group of second images.

2. A method according to claim 1, further comprising:• obtaining the position of a circumscribed area (6) inside a second image which belongs to a group of the one or more groups of second 2D images,• determining the position of the circumscribed area in the third image, which is generated for the group of second 2D images containing the circumscribed area, on the basis of the relevant scaling factor and transformation function,• determining the position of the circumscribed area in virtual space by means of a ray casting technique on the basis of the 3D model and the position of the circumscribed area in the third image, and• storing in the data-storage unit the position of the circumscribed area in virtual space and / or displaying by means of a display (7) the 3D model and the circumscribed area in virtual space.

3. A method according to either of the preceding claims, wherein the second images of at least one group of second images have the same second ground sample distance.

4. A method according to any one of the preceding claims, wherein the step of determining the scaling factor comprises:• resizing the scale of a second image which reproduces the element,• comparing the dimensions of the reproduced element in the second resized image with those of the reproduced element in a first image.

5. A method according to any one of the preceding claims, wherein the scaling factor corresponds to the scale resizing factor of a second image which reproduces theelement so that the element reproduced by the second image has the same dimensions as the element reproduced in a first image.

6. A method according to any one of claims 2 to 5, wherein the method comprises geometrically projecting the circumscribed area onto the 3D model.

7. A method according to any one of claims 2 to 6, wherein the step of determining the position of the circumscribed area in the third image comprises:• resizing the circumscribed area by applying the scaling factor,• correlating the points of the resized circumscribed area with the points of the third image by means of the transformation function.

8. A system (300) comprising:• a data-processing device (8) which is configured to carry out the method (100) according to any one of claims 1 to 7,• a data-storage unit (4) operatively connected to the data-processing device for storing the 3D model of the object which is generated by carrying out the method.

9. A computer program comprising instructions which, when the program is carried out by a computer, cause the computer to carry out the method (100) according to any one of claims 1 to 7.

10. A computer-readable medium comprising instructions which, when carried out by a computer, cause the computer to carry out the method (100) according to any one of claims 1 to 7.

Images