Processing images of an environment

Abstract

A method for processing images depicting an environment, comprises accessing an image database of images depicting the environment, accessing a point cloud corresponding to the environment, accessing camera pose information for each image in the image database, computing a depth map for each image in the image database using the point cloud and camera pose information and matching each depth map to the point cloud such that points in the depth map correspond to points in the point cloud. For each depth map, filtering the depth map is done to remove points in the depth map which are occluded by other points and produce a filtered depth map. The filtered depth maps are used to select images from the image database.

Description

SP3204PROCESSING IMAGES OF AN ENVIRONMENTTECHNICAL FIELD

[0001] The present invention relates to image processing.BACKGROUND

[0002] Image data of an environment are routinely obtained to monitor the status of the environment and detect issues or anomalies. For example, the environment may include equipment susceptible to damage or degradation over time. Images are obtained for example with handheld devices, drones or robots and are typically reviewed by a human inspector who identifies and reports any issues with equipment in the environment.

[0003] The embodiments described below are provided by way of example only and are not limiting of implementations which solve any or all of the disadvantages of known image processing methods.SUMMARY

[0004] This summary is provided to present a selection of concepts disclosed herein in a simplified form, which are described in more detail below. This summary is not intended to identify key features or essential features of the claimed subject matter nor is it intended to be used to limit the scope of the claimed subject matter.

[0005] Described herein is a computer-implemented method for processing images depicting an environment. The method comprises accessing an image database of images depicting the environment, accessing a point cloud corresponding to the environment, and accessing camera pose information for each image in the image database. A depth map for each image in the image database is computed using the point cloud and camera pose information. The method comprises matching each depth map to the point cloud such that points in the depth map correspond to points in the point cloud. Matching the depth map to the point cloud means that the images can be processed based on the point cloud which is a three-dimensional representation of the physical environment. Each depth map is filtered to remove points in the depth map which are occluded by other points to produce a filtered depth map. This means that only points (which correspond to point cloud points) which are in view remain in the filtered depth map. For each image in the image database, the points from the point cloud which are in view in the image are determined using the corresponding filtered depth map. Therefore, for each image there is an associated plurality of points from the point cloud which are in view. Points in view are used to select images from the image database. A plurality of images is selected form the image database. Reducing the number of images to be processed and / or analysed further (rather than processing the entire image database) means that computational resources including memory and processing resources are saved. Methods described herein comprise selecting a first image from the image database wherein the first selected image has a maximum number of points from the point cloud in view. Further images are iteratively selected. For each image in a plurality of remaining images, the number of points from the point cloud which are in view in the image andSP3204 which are not in view in any already selected image is determined. An image is selected from the plurality of remaining images. This is the image with the maximum number of points from the point cloud which are in view in the image and which are not in view in any already selected image. Image selection is stopped when a predetermined amount of points in the point cloud are in view in the selected images. This means that the number of images selected for further processing is reduced while ensuring that a suitable amount of the environment (represented by the point cloud) is depicted in the selected images thus improving the quality of image processing. In this way, downstream image processing tasks which rely on the environment being depicted in the selected images become more efficient without loss of performance.

[0006] In some examples, the method further comprises using machine vision to detect one or more anomalous portions of the environment in one or more images in the image database. For each anomalous portion of the environment detected in an image, methods comprise defining an anomalous portion of the image associated with the anomalous portion of the environment. This allows an anomalous portion of the image which depicts an anomaly in the environment to be identified and processed. A depth map for the anomalous portion of the image is accessed and points in the point cloud corresponding to the anomalous portion of the image are identified using the depth map for the anomalous portion of the environment. It is determined whether one or more points in the point cloud corresponding to the anomalous portion of the image are in view in one or more of the selected images. If one or more points in the point cloud corresponding to the anomalous portion of the environment are not in view in one or more of the selected images, the image depicting the anomalous environment portion is added to the selected images. This means that the selected images include depictions of all or almost all of the anomalous portions of the environment detected by machine vision and therefore the quality of image processing is improved.

[0007] In various examples machine vision comprises image segmentation, object detection and classification. Combining image segmentation, object detection and classification means that machine vision is improved and anomalous portions of the environment are more accurately detected.

[0008] In some examples, computing a depth map for each image in the image dataset comprises using a segmentation model to determine one or more blurred portions of the image and / or one or more out of focus portions of the image. Using a segmentation model allows pixels to be identified as blurred or out of focus in an accurate manner. Pixels corresponding to the one or more blurred portions and / or one or more out of focus portions of the image are excluded from the depth map. By excluding these pixels from the depth map, sections of the environment represented by point cloud points corresponding to the pixels will not be considered to be in view. This improves the quality of the selected images because they will not contain depictions of the environment which are blurred or out of focus and thus unsuitable for further processing. As aSP3204 result performance of downstream image processing tasks using the selected images is improved and / or made more efficient.

[0009] Various methods described herein comprise excluding one or more points from the depth map wherein depth is greater than a predetermined depth. This improves image processing because distant portions of the environment (those with larger depths) appear smaller and more blurred in the image making those pixels more difficult to process for example by anomaly detection. If distant parts of the environment are not considered to be in view by excluding corresponding points from the depth map, image processing is improved in performance and / or efficiency.

[0010] In some scenarios, the plurality of remaining images includes every image which has not been selected. This means that for every image which has not yet been selected, the number of points which are in view in the image and which are not in view in any already selected image is determined. Image processing is improved because the image with the maximum points in view (but not in view in any selected images) of the remaining images may then be selected at each iteration. The selected images are added to with the image which will improve coverage of the environment the most at each iteration.

[0011] In some other scenarios, the plurality of remaining images includes a predetermined number of the images which have not been selected. In examples the predetermined number is 16 or 32. The number of points in view which are not in view in any already selected image is determined only for this predetermined number of images. This saves computational resources such as processing time.

[0012] In various examples, the point cloud corresponding to the environment is obtained using photogrammetry. Photogrammetry provides an efficient and accurate way to obtain a point cloud. Additionally or alternatively, the point cloud corresponding to the environment is obtained using laser scanning. Laser scanning also provides an efficient and accurate way to obtain a point cloud.

[0013] Various methods described herein comprise computing a depth map for each image in the image database using ray casting to obtain depth information for pixels in the image. Using ray casting means that depth information can be accurately obtained and is efficient to compute using image processing hardware such as graphics processors.

[0014] In some examples, matching each depth map to the point cloud comprises obtaining a three-dimensional location for each point in the depth map, and using a nearest neighbor search to match the point in the depth map to one or more points in the point cloud. Matching each depth map to the point cloud allows point cloud points corresponding to points / pixels in the depth map to be identified in order to determine which point cloud points are in view. Using a depth map and a nearest neighbor search provides an efficient and accurate way to associate point cloud points with depth map points or pixels, in other words to match a depth map to the point cloud.

[0015] In further examples, the nearest neighbor search comprises using a search radius based on a physical size of each pixel in the depth map. This means that a smaller radius is usedSP3204 for pixels with smaller depth values, and a larger radius is used for pixels with larger depth values. This makes matching a depth map to the point cloud more accurate because point cloud points which are too far away, in other words outside the search radius are not matched to a depth map point / pixel.DESCRIPTION OF THE DRAWINGS

[0016] The present description will be better understood from the following detailed description read in light of the accompanying drawings, wherein:FIG. 1 is a schematic diagram showing an environment and camera locations;FIG. 2 is a schematic diagram showing an environment, a point cloud, an image and a depth map;FIG. 3 is a schematic diagram showing ray casting;FIG. 4 is a schematic diagram of a method for selecting images of an environment;FIG. 5 is a schematic diagram showing validation of an image selection process;FIG. 6 is a schematic diagram depicting a machine vision pipeline;FIG. 7 is a flow diagram of a method for processing images depicting an environment; FIG. 8 is a flow diagram of a method for processing images depicting an environment; andFIG. 9 illustrates an example computing-based device in which image processing methods are implemented.Corresponding reference numerals are used throughout the figures, where appropriate, to indicate similar features.DETAILED DESCRIPTION

[0017] The following description is presented in connection with the appended drawings and is intended as a description of the present examples to enable a person skilled in the art to make and use the invention. The description is not intended to represent the only forms in which the present examples are constructed or utilized. The present invention is not limited to the embodiments described herein and various modifications to the disclosed embodiments will be apparent to those skilled in the art.

[0018] As described above, environments containing equipment such as insulated piping are routinely monitored in order to check the status of the environment and detect damage or degradation over time. For example cracks, corrosion, vegetation, missing studs and / or bolts, ice, or deterioration of materials may occur in the environment. Images of the environment are acquired and processed to detect such issues. Images are obtained of the environment for example using handheld devices, drones or robots. A drone may fly around the environment, taking images. Alternatively, a person may navigate the environment with a handheld device or a robot may navigate the environment taking images. The images may be visible light images, infrared images, images from optimal gas imaging (OGI) combined with RGB (red green blue) cameras, or any other suitable type of image . Many images are obtained and stored in a database in order to obtain good coverage of the environment. In some cases many thousands or hundreds of thousands of images are obtained and stored, for example some projects captureSP3204 around two hundred thousand images of an environment. The images are then used to monitor the status of the environment. Images are analyzed and often reviewed by a human inspector who identifies and reports any issues with equipment in the environment.

[0019] The present technology relates to processing of images of an environment. In particular, the present technology allows images to be selected from an image database for further use. Reducing the number of images makes image processing more efficient, saving computational resources such as memory and processing resources. Furthermore reducing the number of images to be reviewed by a human inspector saves time and makes the human review process more accurate as more time can be spent on each image.

[0020] The disclosed technology allows images to be selected from the database of images of an environment while ensuring that suitable coverage of the environment is maintained. This is done by accessing a point cloud depicting the environment and iteratively selecting images from a plurality of images of the environment until a predetermined number of points in the point cloud are in view in at least one of the selected images. A point of a point cloud is in view in an image when the point cloud point is depicted in the image. The number of point cloud points in view in each image is determined by, for a given image, computing a depth map for the image and matching the depth map to the point cloud. When the depth map is computed by rendering points from the point cloud from a camera location, hidden or occluded points may be removed to determine the point cloud points in view. Various examples achieve good quality of image processing or analysis of the plurality of selected images because images are selected until a predetermined amount of points from the point cloud are in view in the selected images. Efficiency is gained by reducing the number of images which are used in further processing, while ensuring quality of the further processing. The disclosed methods ensure that no significant portion of the environment is missed (i.e. not in view) in the selected images.

[0021] FIG. 1 is a schematic diagram showing an environment 102 and camera locations. The environment 102 is imaged using camera 110 from various camera locations such as location 106 shown as circular dots. In the example shown in FIG. 1 , the camera locations such as Iocation106 lie along a path 104 of a drone to which camera 110 is attached. The drone flies along path 104 taking an image at regular intervals. The images are stored in a database of images depicting environment 102. The methods and systems disclosed herein relate to selection of images in the image database in order to reduce the number of images which are further processed thereby saving memory and / or power. In FIG. 1 the selected images are taken from locations such as location 108 which are shown as circular dots with boxes around them. Only some of the images (i.e. some but not all) from the image database are selected for further processing and how images are selected is explained in detail below with reference to FIG. 2, FIG. 3 and FIG. 4.

[0022] FIG. 2 is a schematic diagram showing an environment, a point cloud, an image and a depth map. FIG. 2a shows a physical environment 202 which is an example of physical environment 102. Environment 202 includes three pipes 204, 206 and 208 which are representative of equipment contained in the environment which is monitored and susceptible toSP3204 damage and deterioration over time. For example, pipes can develop cracks, rust and / or corrosion, bolts holding the pipes together may become missing, ice may form on the pipes or vegetation may grow on the pipes. Also, insulated pipes can become dented or cracks can develop in the insulation which introduces a risk of water leaking between the insulation and the pipe. This leads to problems with integrity of the equipment. In other examples, a physical environment may further include: more pipes, insulated equipment, insulation, storage tanks, machinery, pumps, computer equipment, or other objects.

[0023] FIG. 2b is a schematic diagram of a point cloud 212 depicting the environment 202. A point cloud is a discrete set of data points which each have a position in three-dimensional (3D) space. Point cloud 212 represents objects in environment 202 including pipes 204, 206 and 208. Point clouds such as point cloud 212 are produced for example using photogrammetry or using 3D scanners such as laser scanners. Photogrammetry involves using multiple overlapping images from different positions and angles to produce the point cloud. A 3D scanner such as a laser scanner can also be used to produce the point cloud. Reflections of the laser from the surface of objects are used to determine distance of objects from the scanner and generate the points in the point cloud. In some cases the point cloud 212 is obtained from a depth camera. In further cases point clouds are produced using Gaussian Splatting, or Trilinear Point Splatting (TRIPS). These methods only use images to create a point cloud reconstruction of the environment.

[0024] FIG. 2c is representative of an example image 214 from an image database 218 of images depicting the environment 202. The image shows pipes 204, 206 and 208. From the perspective of the camera pose corresponding to image 214, pipes 204 and 206 are in front of pipe 208. Only some points from point cloud 212 are in view in image 214. For example, points in the point cloud located on pipe 208 behind pipes 204 and 206 are not in view. Points at the front of pipes 204 and 206 are in view in image 214 but points on the back of the pipes are not in view. Image 214 is an example of one image from a database of images depicting the environment. In the image database are a plurality of images, such as many images taken from different camera locations, and each image has a group of points from the point cloud in view i.e. depicted in the image. In some examples, the image database 218 comprises a photogrammetry dataset used to generate a point cloud such as point cloud 212. In various scenarios all or most of the point cloud is in view across the image database containing many images. Often, the same point in the point cloud is in view in multiple images which differ only slightly in camera location. Disclosed herein are various methods which allow for the selection of a plurality of images from the image database in order to reduce the number of images which are processed further by a downstream application and / or which are stored in memory. There are a predetermined number of points from the point cloud in view in the selected images to maintain good image coverage of the environment in the selected images.

[0025] FIG. 2d represents a depth map 216 corresponding to image 214. A depth map is a type of image which contains depth information about the distance of a surface of an object in the environment to the camera viewpoint. In the example of FIG. 2d, the color of each point in aSP3204 depth map corresponds to distance from the camera viewpoint, with darker colors representing points closer to the viewpoint. Pipes 204 and 206 are closer to the camera than pipe 208 and therefore are represented by darker shading in the schematic depth map 214. Each point in depth map 216 corresponds to a pixel in image 214 Depth maps such as depth map 216 are computed for each image in the image database of images depicting the environment using the point cloud depicting the environment 202 as well as camera pose information for each image. Camera pose information represents the camera viewpoint and contains three-dimensional location information and three-dimensional orientation information about the position of the camera at the time an image is taken. In various examples, a depth map is computed for each image in the image dataset using the point cloud and camera pose information using ray casting.

[0026] FIG. 3 is a schematic diagram of ray casting which is used in some examples to compute a depth map for each image in the image database. Camera 310 has a six-dimensional pose including three-dimensional position information and three-dimensional orientation information at a time when image 314 is captured. The camera pose may be obtained from sensors in the camera such as global positioning sensors, accelerometers, gyroscopes or other sensors. Camera orientation and pose can also be obtained with a Structure-from-Motion (SfM) and Multi-View Stereo (MVS) pipeline such as COLMAP, which has the advantage of not relying on a hardware sensor. In the schematic diagram shown in FIG. 3, image 314 is an image of a sphere 312 which is present in the environment. Points 316 and 318 are points on the surface of the sphere 312. The points 316, 318 are points in a point cloud depicting the environment containing the sphere 312. The surface of the sphere is represented by points. The points 316 and 318 are a subset of the points on the surface of the sphere. Examples of pixels in image 314 are shown at 306, 308. Depth data are obtained for every pixel in image 314 such as pixels 306, 308 using ray casting. A theoretical ray from the camera 310 to a pixel (such as 306, 308) projected from the image into the environment is shown in FIG. 3. The first point in the point cloud depicting the environment which is encountered by the ray is used to determine the depth coordinate (distance from the camera) for the pixel. For pixel 308, the first point encountered by the corresponding ray is point 318 on the surface of sphere 312. For pixel 306, the first point encountered by the corresponding ray is point 316 on the surface of sphere 312.

[0027] In various scenarios a depth map is rendered by projecting the points from the point cloud onto an image plane using an intrinsic matrix. The intrinsic matrix represents internal parameters of the camera such as the focal length and principal point. It is a transformation matrix for transforming from 3-dimensional coordinates of point cloud points to 2-dimensional coordinates on the image plane. Where multiple point cloud points are projected to the same pixel, the smallest distance or depth is used as the depth value for that pixel. In examples, hidden point removal may be used to filter points from the point cloud before rendering the depth map and remove points which are occluded by points closer to the camera position. Alternatively or additionally, a depth map may be created using a stereo camera. A point in 3d space can be converted to a 2d pixel-coordinate using the intrinsic matrix (containing the camera specific information such as focal length and principal point) and the extrinsic matrix (containing theSP3204 camera pose). This results in a normalized pixel-coordinate, which is then scaled to a pixelcoordinate using the image resolution.

[0028] In various examples, when computing a depth map from an image in the image database, one or more portions of the image are excluded from the depth map. In scenarios, the portions which are excluded comprise one or more pixels. Portions of the image may be excluded from the depth map because they are blurred or out of focus or for another reason. Excluding these portions from the depth map means that those pixels from the image are not matched to points in the point cloud and are not considered to be in view in that image. This means that the selected images include images where a predetermined amount of points from the point cloud are in view and also are not blurred or out of focus. The quality of the selected images is therefore improved.

[0029] In order to detect portions of images in the image database to exclude when computing the depth map, a segmentation model is used in various example scenarios. The segmentation model is configured to classify each pixel in an image, for example as “in focus”, “out of focus”, “blurred” or as other classifications. Any suitable segmentation model may be used including but not limited to thresholding, clustering, histogram segmentation methods, edge detection, motion blur or others.. In example scenarios, pixels in an image which are classified as “blurred” or “out of focus” are excluded when computing the depth map corresponding to that image. In further examples, pixels are excluded from the depth map for another reason.

[0030] In other examples, additionally or alternatively, individual images may be classified as blurred by convolving the image with the Laplacian kernel. The variance of the response is computed, and if the variance is determined to be below a pre-defined threshold the image is classified as blurred. Images classified as blurred may be excluded from processing in order to improve the quality of selected images.

[0031] Some pixels in the depth maps have large depth values. In other words, some points in the depth maps are far away from the camera at the time when the image is taken and appear as objects in the distance in the corresponding image. These are more difficult to process for example because objects appear smaller and more blurred in the image and it is more difficult to detect anomalous parts of the objects. Therefore, image processing is improved if distant objects are not considered to be in view. Various example image processing methods comprise excluding one or more points from the depth map wherein depth is greater than a predetermined depth. The predetermined depth is also called a cut-off depth and is calculated based on ground sampling distance and / or physical pixel size. Ground sampling distance (GSD) is related to image resolution and is the distance between pixel centers measured on the ground. For example, if the GSD is one meter, the physical locations corresponding to the pixels are one meter apart. By determining a cut-off depth based on GSD and / or physical pixel size, image portions with suitable resolution are included in the depth map.

[0032] FIG. 4 is a schematic diagram of a method for selecting images of an environment. The method of FIG. 4 may be computer-implemented. The method comprises accessing an image database of images depicting the environment, accessing a point cloudSP3204 corresponding to the environment, accessing camera pose information for each image in the image database, and computing a depth map for each image in the image database using the point cloud and camera pose information as described above. Depth maps 416 are obtained. Optionally, one or more portions may have been excluded from the depth maps as described above on account of being blurred, out of focus, because of the pixel depth values being larger than a cut-off distance, or for another suitable reason.

[0033] Points in depth maps 416 such as depth map 216 correspond to pixels in an image such as image 214. In order to determine which points in the point cloud such as 212 are in view in each image, each depth map is matched to the point cloud such that points in the depth map correspond to points in the point cloud 402. Substantially each point or pixel in the depth map is matched to one or more points in the point cloud. Matching a point in a depth map to a point in a point cloud means identifying a point in the point cloud corresponding to the pixel in the depth map. For example, if the pixel in the image is in a region of the image depicting the surface of a pipe then the matched point cloud point will be located on the surface of the pipe.

[0034] In various scenarios, points in the point cloud corresponding to the pixel in the depth map are nearest neighbours in terms of three-dimensional location. Therefore, in various examples matching each depth map to the point cloud comprises obtaining a three-dimensional location for each point in the depth map, and using a nearest neighbor search to match points in the depth map to points in the point cloud. A three-dimensional location for each point in the depth map is determined using camera pose information, camera intrinsic parameters (such as focal length, resolution and the principal point) and depth data. In various examples the nearest neighbor search comprises using a k-d tree or any heuristic nearest neighbor search.

[0035] The nearest neighbor search in some scenarios comprises using a search radius for each pixel or point in the depth map. A search radius means that only space around the 3D location of the depth map point is included in the search for a point cloud point. The space around the 3D location is defined by the search radius. Any point cloud points which are further from the 3D location of the depth map point than the search radius are excluded from the search. In various scenarios the search radius depends on the physical size of the pixel. For example, the radius is determined by dividing the depth value of the pixel by the focal length of the camera lens. This results in a smaller radius for pixels with smaller depth values, and a larger radius for pixels with larger depth values. A cut-off depth is used based on a ground sampling distance requirement. Using a search radius improves image processing by making matching the depth map to the point cloud more accurate because point cloud points which are too far away are not found by the nearest neighbor search..

[0036] At block 404 in FIG. 4, the depth maps are filtered by removing hidden points from the depth map. Removing hidden points means that points (from the point cloud) of the matched depth map which are occluded by other points are not included in the depth map. During computation of the matched depth maps, point cloud points are matched to pixels in the depth map based on distance between the depth map points and the point cloud points. This means that some point cloud points are included in the matched depth map which are occluded by otherSP3204 points, for example because they are not located at the surface of an object. . Hidden point removal may be performed for example by extracting points that reside on the convex hull of a transformed point cloud, or by any other suitable method. The depth maps, once matched with the point cloud and filtered (and optionally having had some portions or pixels removed) are used to determine the points from the point cloud which are in view in each image in the image database. Points remaining in the filtered depth map are the points in view in the corresponding image. Images are selected based on the points from the point cloud in view as described with reference to image selection 408 shown in FIG. 4.

[0037] First, a first image is selected 412. The first selected image is the image with a maximum number of points in view. The number of points in view in each image is calculated at block 410. The images may then be ranked according to the number of points they depict from the point cloud and a highest ranking image selected as the one having the maximum number of points in view. Further images are iteratively selected based on the number of points in view in the remaining (unselected) images which are not already in view in the selected images. For each image in a plurality of remaining images, a point number is determined. Point number is the number of points from the point cloud which are in view in the image and which are not included in any already selected image. The image with the maximum point number in the plurality of remaining images is selected. For example, during selection of the second image, an image in a plurality of remaining images has X points from the point cloud in view. Of those X points, Y points are also in view in the first selected image. The point number is X minus Y. The image with the highest point number in the plurality of remaining images is selected as the second image. In order to select the third image points from the point cloud in view which are not in view in the first or second image are determined and counted.

[0038] In order to select second and subsequent images, image selection 408 comprises determining a number of points in view which are not in view in any already selected image 414 for a plurality of remaining images. The image with a maximum number of points in view which are not in view in any already selected image is selected 418. The steps 414 and 418 are repeated until a predetermined amount of points from the point cloud are in view in the selected images 420.

[0039] Image selection 408 continues until a predetermined amount of points in the point cloud are included in the filtered depth maps corresponding to the selected images. In other words, the selected images when combined have a predetermined amount of points from the point cloud in view. This means that suitable image coverage of the environment is obtained while the number of images selected for further processing is reduced. For example, the predetermined amount of points in the point cloud could be expressed as a percentage such as 80%, 90%, 95% or 99%. Particular example image selection methods use 95% or 99%. Alternatively or additionally, the predetermined amount of points in the point cloud could be a number of points.

[0040] The skilled person understands that the method of FIG. 4 is workable in scenarios where “maximum” is replaced by another quantity which still gives good working results; rather than using maximum as a criterion another criterion is used such as the 90thcentile, the 80thcentile, a quantity which is in a top 10% of ranked items, or other criterion. Selected points are points in view in the image which are not in view on any already selected image. At block 418 in FIG. 4, instead of selecting the image with the maximum number of selected points, an image is selected which has a quantity of selected points where the quantity is in a neighbourhood of the maximum. Where the images are ranked by number of selected points, the selected image may be in a top 10% of the ranked images. Where the method of FIG. 4 refers to each of an item the skilled person understands that the process is workable where some of the items are omitted from the operation stated as involving each item. For example, at block 414 the point number may not be determined for every remaining image, rather the point number may only be determined for some but not all of the remaining images.

[0041] As described above, point number is determined for a plurality of remaining images. In some examples, the plurality of remaining images includes every image which has not been selected. In other examples, the plurality of remaining images includes a predetermined number of images which have not been selected. In these examples, a number of images are randomly selected from the totality of the remaining images. The point number is determined only for the random selection of remaining images instead of for every remaining image. This saves computational resources such as processing time. If N is the total number of images in an image database , then the total compute time scales as O(N * N) where N is large if the plurality of remaining images includes every image which has not been selected. This is because the compute time varies as N + (N-1) + (N-2) + (N-3) + ... which equals N * (N-1 ) / 2 If, instead, k images are randomly selected from the total number of remaining images then the total compute time scales as O(N * k). In various scenarios, 16 or 32 images are randomly selected at each iteration and k is equal to 16 or 32.

[0042] In the examples described above with reference to FIG. 3, image selection 408 stops when a predetermined amount (such as a predetermined number) of points in the point cloud are in view in the selected images 420. Because the stopping criterion is based on coverage of the point cloud, points from the point cloud are sometimes not in view in the selected images. This means that some points which are of interest for example because they relate to an anomalous part of the environment are not selected. Optional validation of image selection 408 means that all points from the point cloud which relate to an anomalous part of the environment are in view in the selected images, as described with reference to FIG. 5.

[0043] FIG. 5 is a schematic diagram showing validation of an image selection process which includes using machine vision to detect one or more anomalous parts of an environment such as environment 202, 502. Environment 502 includes pipes 504, 506 and 508. In some scenarios environment 502 corresponds to environment 202 at a later time once one of the pipes has developed a crack. A crack is shown on pipe 504 in FIG. 5a. Anomalous portions of the environment are identified in images of the environment from the image database 218 using machine vision. One or more anomalous portions of the environment may be identified in one or more images.SP3204

[0044] FIG. 5b shows an image 514 in which the crack on pipe 504 can be seen and detected by machine vision. An anomalous portion of the image associated with the anomalous portion of the environment is defined in image 514. The anomalous portion of the image may be a bounding box such as bounding box 510 which surrounds the crack in pipe 504. The bounding box may be a rectangle such as bounding box 510 or any other suitable shape such as a circle or square. In various examples, the edges of the bounding box are determined based on distance from the anomalous portion of the environment. Additionally or alternatively, the anomalous portion of the image associated with the anomalous portion of the environment is determined based on a segmentation map. The segmentation map is generated from a segmentation model using the image. In these examples, the segmentation model identifies pixels in the image which are classified as corresponding to the anomalous portion of the environment and the anomalous portion of the image comprises these pixels.

[0045] For each anomalous environment portion in an image, points in the point cloud corresponding to the anomalous environment portion are identified as an anomalous portion of the image. Pixels within the anomalous portion of the image (for example within the bounding box 510) are matched with points in the point cloud depicting the environment in order to identify points in the point cloud corresponding to the anomalous portion of the environment. This is done by accessing a depth map corresponding to the anomalous image portion. The depth map corresponding to the portion of the image may be a portion of one of the depth maps 416 in FIG. 4. Each pixel in the depth map corresponding to the anomalous image portion has a depth value representing distance from the camera. Points from the point cloud corresponding to the anomalous portion of image are identified by matching points in the depth map to the point cloud. In some examples the matched depth maps are filtered to remove hidden points from the matched depth map. This is similar to the methods described in more detail with reference to FIG. 4 above. Referring again to FIG. 5b, points from the point cloud corresponding to bounding box 510 are represented as crosses 512.

[0046] The validation further comprises determining whether the anomalous portion of the environment is in view in the selected images. Various example validation methods include determining whether one or more points in the point cloud corresponding to the anomalous portion of the image (for example points 510 in FIG. 5b) are in view in one or more of the selected images 520, 420. Point cloud points in view in an image are determined using an image pixel to point cloud mapping which is obtained for example using the nearest neighbor search described above, following determining the 3-dimensional location of points in the depth map. In various scenarios, points from the point cloud which are in view in the selected images may be accessed from the filtered depth maps obtained at block 404 in FIG. 4. Points from the point cloud which are in view in the selected images may be obtained as a plurality of point indices. If each point corresponding to the anomalous portion of the environment is in view in one or more of the selected images, then the validation step is considered to be successful and the image in which the anomaly was identified is not added to the selected images. However, if one or more pointsSP3204 corresponding to the anomalous portion of the environment are not in view in the selected images 520, the image in which the anomaly is identified is added to the selected images 520.

[0047] As described above, machine vision is used to identify one or more anomalous portions of the environment. Any suitable method for machine vision may be used. An example machine vision pipeline is depicted in FIG. 6 which includes object detection 604, semantic segmentation 602 and classification 606. An image 614 is input into the machine vision pipeline. Semantic segmentation 602 partitions the model into regions or segments. Each pixel in the image is categorized into a segment associated with a class or object. Semantic segmentation model 602 identifies pixels in image 614 which correspond to one or more anomalous portions of the environment, for example the segmentation model identifies corrosion or vegetation in the environment. Additionally or alternatively, semantic segmentation model 602 identifies pixels which correspond to areas in the environment where anomalous portions of the environment are likely to be found. For example, the segmentation model may detect equipment in the environment such as pipes 204, 206, 208. Semantic segmentation allows crops of the image to be taken which surround areas in the image which are likely to include anomalous portions of the environment. Examples of semantic segmentation model 602 include segment anything model SAM, SwinT ransformer with a Mask2Former head, or a multi-modal model such as visual program distillation VPD or any other suitable model.

[0048] Object detection module 604 detects instances of semantic objects in the input image 614. Examples of objects which may be detected are pipes, insulation, storage units or other objects. Object detection is performed on either or both of the full input image 614 or crops of the input image 614 generated using the results of semantic segmentation 602. Examples of suitable object detection models include but are not limited to: a convolutional neural network model, YOLO (you only look once) object detection model, EfficientDet, CoDETR, ConvNeXt-V2, a SwimTransformer. Other suitable object detection models may be used.

[0049] Classification model 606 classifies the part or parts of the image 614 which correspond to one or more anomalous portions of the environment with a label corresponding to a type of issue. For example, classification 606 may result in a part of the image being labeled as “vegetation”, “ice”, or “crack”. In further examples, the classification model 606 also categorizes the severity of the issues for example into categories of high, medium and low severity. In various examples, classification is performed on image crops generated using the results of semantic segmentation 602. Examples of suitable classification models include a multi-layer perceptron neural network, support vector machine, random forest, or other machine learning classifier. In various examples, the classification model is based on a Vision Transformer or the ConvNeXt-V2 backbone.

[0050] FIG. 7 is a flow diagram of a method for processing images depicting an environment. The method comprises accessing an image database of images depicting the environment 702 such as image database 218 in FIG. 2, accessing a point cloud such as point cloud 212 corresponding to the environment 704, and accessing camera pose information for each image in the image database 706. A depth map such as 216 is computed for each image inSP3204 the image database using the point cloud and camera pose information 708. The method further includes matching each depth map to the point cloud such that points in the depth map correspond to points in the point cloud 710 and filtering each depth map using hidden point removal to remove points in the depth map which are occluded by other points 712. The filtered depth maps are used to determine the points from the point cloud which are in view in each image in the image database 714. A plurality of images are selected from the image database for example using image selection 408 in FIG. 4. Image selection comprises selecting a first image from the image database, wherein the first selected image has a maximum number of points from the point cloud in view 716. Further images are iteratively selected. For each image in a plurality of remaining images, the iterative selection includes determining the number of points from the point cloud which are in view in the image and which are not in view in any already selected image 718. The image with a maximum number of points in view which are not in view in any already selected image is selected 720. At 722, it is determined whether a predetermined amount of points in the point cloud are in view in the selected images. If there are less than the predetermined amount of points from the point cloud in view, then the image selection continues. If the predetermined amount of points from the point cloud are in view, the image selection ends 724.

[0051] FIG. 8 is a flow diagram of a method for processing images depicting an environment. The method depicted in FIG. 8 is a validation method for the method shown in FIG.7. The method in FIG. 8 comprises using machine vision such as the machine vision in FIG. 6 to detect one or more anomalous portions of the environment in images in the image database such as database 218, 802. For each anomalous portion of the environment detected in an image, an anomalous portion of the image such as 510 in FIG. 5 associated with the anomalous portion of the environment is defined 804. A depth map for the anomalous portion of the image is accessed 806 and points in the point cloud corresponding to the anomalous portion of the image using a depth map for the anomalous portion of the image are identified 808. At 810, the method includes determining whether one or more points in the point cloud corresponding to the anomalous portion of the image are in view in one or more of the selected images. If one or more points in the point cloud corresponding to the anomalous portion of the environment are not in view in one or more of the selected images, the image is added to the selected images 812. If not, then the image is not added to the selected images and the method moves on to the next detected anomalous portion of the environment.

[0052] FIG. 9 illustrates an example computing-based device in which image processing methods described herein are implemented. Computing-based device 900 comprises one or more processors 924 which are microprocessors, controllers, or any other suitable type of processors for processing computer executable instructions to control the operation of the device in order to process images depicting an environment. In some examples, for example where a system on a chip architecture is used, the processors 924 include one or more fixed function blocks (also referred to as accelerators) which implement a part of the methods disclosed herein in hardware (rather than software or firmware). Platform software comprising an operating system 908 or any other suitable platform software is provided at the computing-based device to enable applicationSP3204 software 912 to be executed on the device. A data store 910 holds for example image database 922, depth maps 932, selected images 926, camera pose information 928, point cloud 930 or any type of data suitable for processing images depicting an environment. Image selection software 904, machine vision software 902 and validation software 934 are also stored in memory 918 to be used to process images depicting an environment.

[0053] The computer executable instructions are provided using any computer-readable media that are accessible by computing based device 900. Computer readable media include, for example, computer storage media such as memory 918 and communications media. Computer storage media, such as memory 918, include volatile and non-volatile, removable and nonremovable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or the like. Computer storage media includes, but is not limited to, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM), electronic erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that is used to store information for access by a computing device. In contrast, communication media embody computer readable instructions, data structures, program modules, or the like in a modulated data signal, such as a carrier wave, or other transport mechanism. As defined herein, computer storage media does not include communication media. Therefore, a computer storage medium should not be interpreted to be a propagating signal per se. Although the computer storage media (memory 918) is shown within the computing-based device 900 it will be appreciated that the storage is, in some examples, distributed or located remotely and accessed via a network or other communication link (e.g. using communication interface 920). The computing-based device 900 also comprises an input / output controller 906 arranged to output display information to a display device 916 which may be separate from or integral to the computing-based device 900. The input / output controller 906 is also arranged to receive and process input from one or more devices, such as a user input device (e.g. a mouse, keyboard, camera, microphone or other sensor).

[0054] The term ‘computer’ or ‘computing-based device’ is used herein to refer to any device with processing capability such that it executes instructions. Those skilled in the art will realize that such processing capabilities are incorporated into many different devices and therefore the terms ‘computer’ and ‘computing-based device’ each include personal computers (PCs), servers, mobile telephones (including smart phones), tablet computers, set-top boxes, media players, games consoles, personal digital assistants, wearable computers, and many other devices.

[0055] The methods described herein are performed, in some examples, by software in machine readable form on a tangible storage medium e.g. in the form of a computer program comprising computer program code means adapted to perform all the operations of one or more of the methods described herein when the program is run on a computer and where the computerSP3204 program may be embodied on a computer readable medium. The software is suitable for execution on a parallel processor or a serial processor such that the method operations may be carried out in any suitable order, or simultaneously.

[0056] Those skilled in the art will realize that storage devices utilized to store program instructions are optionally distributed across a network. For example, a remote computer is able to store an example of the process described as software. A local or terminal computer is able to access the remote computer and download a part or all of the software to run the program.Alternatively, the local computer may download pieces of the software as needed, or execute some software instructions at the local terminal and some at the remote computer (or computer network). Those skilled in the art will also realize that by utilizing conventional techniques known to those skilled in the art that all, or a portion of the software instructions may be carried out by a dedicated circuit, such as a digital signal processor (DSP), programmable logic array, or the like.

[0057] Any range or device value given herein may be extended or altered without losing the effect sought, as will be apparent to the skilled person.

[0058] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above.Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

[0059] It will be understood that the benefits and advantages described above may relate to one embodiment or may relate to several embodiments. The embodiments are not limited to those that solve any or all of the stated problems or those that have any or all of the stated benefits and advantages. It will further be understood that reference to ‘an’ item refers to one or more of those items.

[0060] The operations of the methods described herein may be carried out in any suitable order, or simultaneously where appropriate. Additionally, individual blocks may be deleted from any of the methods without departing from the scope of the subject matter described herein. Aspects of any of the examples described above may be combined with aspects of any of the other examples described to form further examples without losing the effect sought.

[0061] The term ‘comprising’ is used herein to mean including the method blocks or elements identified, but that such blocks or elements do not comprise an exclusive list and a method or apparatus may contain additional blocks or elements.

[0062] It will be understood that the above description is given by way of example only and that various modifications may be made by those skilled in the art. The above specification, examples and data provide a complete description of the structure and use of exemplary embodiments. Although various embodiments have been described above with a certain degree of particularity, or with reference to one or more individual embodiments, those skilled in the art could make numerous alterations to the disclosed embodiments without departing from the scope of this specification.

Claims

SP3204CLAIMSWhat is claimed is:

1. A method for processing images depicting an environment (102), the method comprising: accessing an image database (218) of images depicting the environment; accessing a point cloud corresponding to the environment; accessing camera pose information for each image in the image database (218); computing a depth map for each image in the image database (218) using the point cloud and camera pose information; matching (402) each depth map to the point cloud such that points in the depth map correspond to points in the point cloud; for each depth map, filtering the depth map to remove points in the depth map which are occluded by other points and produce a filtered depth map; using the filtered depth maps to determine, for each image in the image database, the points from the point cloud which are in view in the image; selecting a plurality of images, wherein the image selection (408) comprises: selecting a first image from the image database (218), wherein the first selected image has a maximum number of points from the point cloud in view; iteratively selecting further images by: for each image in a plurality of remaining images, determining the number of points from the point cloud which are in view in the image and which are not in view in any already selected image; selecting, from the plurality of remaining images, the image with a maximum number of points from the point cloud which are in view in the image and which are not in view in any already selected image; and stopping the image selection when a predetermined amount of points in the point cloud are in view in the selected images.

2. The method of claim 1 further comprising: using machine vision to detect one or more anomalous portions of the environment in one or more images in the image database; for each anomalous portion of the environment detected in an image: defining an anomalous portion of the image associated with the anomalous portion of the environment; accessing a depth map for the anomalous portion of the image; identifying points in the point cloud corresponding to the anomalous portion of the image using a depth map for the anomalous portion of the image; determining whether one or more points in the point cloud corresponding to the anomalous portion of the image are in view in one or more of the selected images; andPage 17 of 20SP3204 if one or more points in the point cloud corresponding to the anomalous portion of the environment are not in view in one or more of the selected images, adding the image to the selected images.

3. The method of claim 2 wherein using machine vision to detect one or more anomalous portions of the environment comprises image segmentation, object detection and classification.

4. The method of any preceding claim wherein computing a depth map for each image in the image dataset comprises: using a segmentation model to determine one or more blurred portions of the image and / or one or more out of focus portions of the image; excluding pixels corresponding to the one or more blurred portions and / or one or more out of focus portions of the image from the depth map.

5. The method of any preceding claim further comprising excluding one or more points from the depth map wherein depth is greater than a predetermined depth.

6. The method of any preceding claim wherein the plurality of remaining images includes every image which has not been selected.

7. The method of any preceding claim wherein the plurality of remaining images includes a predetermined number of the images which have not been selected.

8. The method of claim 7 wherein the predetermined number is 16 or 32.

9. The method of any preceding claim wherein the point cloud corresponding to the environment is obtained using photogrammetry.

10. The method of any preceding claim wherein the point cloud corresponding to the environment is obtained using laser scanning.11 . The method of any preceding claim wherein computing a depth map for each image in the image database comprises using ray casting to obtain depth information for pixels in the image.

12. The method of any preceding claim wherein matching each depth map to the point cloud comprises obtaining a three-dimensional location for each point in the depth map, and using a nearest neighbor search to match the point in the depth map to one or more points in the point cloud.

13. The method of claim 12 wherein the nearest neighbor search comprises using a search radius based on a physical size of each pixel in the depth map..Page 18 of 20SP320414. An apparatus comprising: a processor; a memory storing instructions that, when executed by the processor, perform any of the methods of claims 1 to 13.

15. A computer storage medium having computer-executable instructions that, when executed by a computing system, direct the computing system to perform any of the methods of claims 1 to 13.Page 19 of 20

Images