Image analysis

Abstract

The present invention provides, a means for assisting in the analysis of constructional plans by identifying representational symbols within the image and grouping similar symbols as likely representi

Description

FIELD OF THE INVENTION The present invention relates to image analysis. BACKGROUND ART Image analysis is a large and complex field which faces a number of practical difficulties. At present, there is no known generic image analysis method which is general enough to be applied to any task, compared to the ability of the human vision system. These problems have therefore been ameliorated by developing image analysis systems which are optimised for a specific task, whether that be a simple one such as recognising a bar code or a more complex one such as identifying a person from their facial features. In both cases, the problem has been made manageable by constraining it to a specific context - in these instances bar codes and faces. By doing this, the system analysing the images can be taught what to expect. The context addressed by the inventors in this case has been the analysis of constructional plans. These represent a plan for a construction or development project and are usually a combination of representational drawings showing an intended physical layout to a chosen scale, over which is presented a number of icons or symbols which represent elements which are to be fitted, installed or otherwise provided at a specific location on the layout. These images are often prepared by (or on behalf of) a creative lead on the project and circulated to one or more service providers who will be invited to submit an estimated cost for the work. Typically, at this stage the providers have not been contracted to provide the service and are thus preparing the estimate on a speculative basis. Thus, speed and accuracy are important in the estimation process; speed since the time spent is not being paid for and accuracy since an over-estimate of the elements which need to be purchased and fitted will lead to an uncompetitive estimate, whereas an underestimate will lead to being awarded a contract which cannot be completed at the expected level of profit (or which may be loss-making). The providers also have no control over the format of the image and need to work with what they have been provided. The image files in question may use a range of standard symbols to represent the elements, but may use bespoke symbols, thus meaning the system cannot be taught to look for specific symbols. They may also include text elements within the image file, intended to give instructions to human readers but having the effect of obscuring symbolic representations in the image when rendered. SUMMARY OF THE INVENTION The present description sets out, in its different aspects, a number of processing steps which assist in the analysis of such images. They may be used together, to identify the different parts of the image file, identify representational symbols within the image, and group similar symbols as likely representing the same concept. Alternatively, one or more aspects may be used independently of others. They are principally regarded as useful in the analysis of constructional plans, but may find application in other contexts. The aspects all contribute towards the identification and / or classification of representational symbols within constructional plans. This allows, at its simplest level, the different groups of symbols to be added up thereby to identify the nature and number of elements which are to be fitted, installed or otherwise provided at a specific location on the layout, assisting with the estimation process by quickly providing a firm basis for the estimate. Other forms of analysis can also be carried out once the image has been interpreted in this way. The present invention relates to an image analysis method, comprising the steps of receiving an image file for analysis, the image file representing a constructional plan and including graphical elements and text elements, the text elements being stored as at least one of text information, text bounding boxes, and / or as text presented in graphical form, identifying a plurality of graphical elements within the image file, grouping the identified graphical elements by (i) selecting a first identified graphical element and a second identified graphical element, (ii) comparing the first and second identified graphical elements and determining a level of similarity between them, (iii) if the level of similarity is greater than a preset threshold, placing the second identified graphical element into a group that includes the first identified graphical element and any other identified graphical elements already grouped with the first identified graphical element, and repeating steps (i) to (iii) until all graphical elements have been compared and / or placed into a group. Once this process is complete, each group will contain a set of graphical elements that are within the preset level of similarity of at least one other member of that group. The group can then be considered to represent all instances of a specific element. It is worth noting that not all of the members of the group will necessarily be within the preset level of similarity of all of the other members of the group; for example element A may be similar to element B, and B similar to C, but A and C may be outside the preset level of similarity. This allows for the possibility that none of A, B and C are perfect representations of what might be referred to as the Platonic ideal of that element, with each differing from the ideal in different respects. Nevertheless, given a sufficiently large sample and with the preset level set appropriately, all instances can be recognised reliably in this way. The "image file" received for analysis may be a pure image file such as a bitmap or vector image, or it may be a file combining image and text elements such as a pdf (portable document file). Generally, a user of the system needs to work with whatever their client provides to them. Combined files such as pdf files may be processed prior to analysis to extract text and image elements, correlate them, identify text information within the image elements of the file, and the like. Character recognition steps may be carried out by a conventional OCR ("optical character recognition") process, many of which are available at the present date. Alternatively, or in addition, it may be a bespoke process aimed at identifying specific letter or letter-like characters within the image and presented in a graphical format. Conventional OCR processes often validate their identifications against a dictionary or other library, and often against grammatical rules, in order to improve their detection accuracy especially in relation to similar letters / symbols such as '1', '1', '1' and the like. This tends to incline their detection towards large blocks of text, which is not necessarily as accurate as might be desired for this context. The stored dictionary list ideally includes the words "legend", "legends", "symbol", "symbols", "key" and "keys". If at least one of these key words appears in the text elements, the initial potential legend area can be identified as the area occupied by that key word. The present invention also provides an image analysis system comprising a processor and a data storage element, the processor having access to the data storage system on which an image file can be stored, and including instructions for the processor to access the image file and carry out one or more of the above methods. The system can also comprise an optical capture device, arranged to capture a digital image of a plan and store it in the data storage system for access by the processor. Suitable optical capture systems include a camera and / or a scanner. Other aspects of the description are claimed in application no. 2309583.9, from which this application was divided. BRIEF DESCRIPTION OF THE DRAWINGS An embodiment of the present invention will now be described by way of example, with reference to the accompanying figures in which; Figure 1 shows selection of initial candidates for embedded annotations within the drawing; Figure 2 shows correction of object orientation; Figure 3 shows a first step in merging of contours to form a single, new, larger contour; Figure 4 shows a second step in merging of contours to form a single, new, larger contour; Figure 5 shows a third step in merging of contours to form a single, new, larger contour; Figure 6 illustrates legend extraction; Figure 7 illustrates area of interest extraction; Figure 8 illustrates border removal to assist with area of interest extraction; Figure 9 shows the perimeter of an external contour; Figure 10 shows the perimeter of an internal contour; Figure 11 shows merging of candidate objects based on shared contours; Figure 12 shows a commonly used shape; Figure 13 shows candidate bounding boxes typically detected in relation to the shape of figure 12; Figure 14 shows an example of bounding boxes (containing candidate symbols) which need to be assessed as to whether to merge them or keep them separate; Figure 15 shows possible candidates for merged or separate bounding boxes of figure 14; and Figure 16 shows the group of symbols obtained from the legend. DETAILED DESCRIPTION OF THE EMBODIMENTS The present invention operates on suitable computing hardware which is available in many forms and which the skilled reader will be familiar with. The hardware comprises at least a processor and a data storage element, but may comprise other elements such as a display to show the results, a keyboard, mouse or touchscreen to accept user input and direction, or the like. As a preliminary step, the drawing needs to be loaded into the data storage. In our preferred embodiment, this is done using the initPDF function, which accepts a filepath and provides a number of files derived from the pdf ("portable document file") file containing the drawing, including the image version of the PDF, the stripped version of the PDF, a set of initial (filtered) bounding boxes and original (unfiltered) bounding boxes, embedded text boxes, embedded annotation boxes, and legend keyword locations. To prepare the pdf file that is received for analysis, the content of the text and drawing data is extracted using the PyMuPDF module. Words located within the relevant file are checked against a dictionary to check that the text blocks are valid and that nonsense is not being acquired, and annotations are checked for their size. To create the stripped version of the image, as well as retrieve the filtered and unfiltered bounding boxes we iterate through the result of page.get_drawings() (an inbuilt function in the PyMuPDF module, see https: / / pymupdf.readthedocs.io / en / latest / page.html#Page.get_drawings). We filter for colour, size and width-to-height ratio. The stripped version of the image is created by redrawing the embedded data found within the pdf onto a new blank page. There are four types of objects; lines, quads, rectangles and curves. Using the original image, we create a "background removed" version of the image -this differs from the "stripped" version. Operating on the premise that (visually) the background of a drawing is the grey area, if an extremely thin black line (according to the pdf data) appears grey it is still drawn as it will remain unfiltered. The background removal however will pick up that it looks grey and filter it out. As embedded text does not present with any colour or styling information, we must then filter out false positive embedded annotations and text boxes. This is done by iterating over each box, viewing a sample of the removed background version of the image and resizing each box to ignore any grey text that is present. This prevents incorrect merging of annotations later in the process. In the example shown in figure 1, initial candidates for embedded annotations within the drawing 10 are shown in within boxes 12. The result of the above operation removed grey text (e.g. CDU, 12, m2) but additionally constrains boxes to the minimum required size. After this initial preparation step, detection of relevant elements take place. We begin by taking the initial bounding rectangles from the PDF data and filtering out any boxes that purely contain background. In this case we look at both the stripped and background removed version of the image and check that there is actually an object present in the candidate bounding box. This result is then passed to the beginning of our bounding box processing module. We filter out any detections that are extremely thin (lines), small or large, i.e. below a present dimensional threshold. In the case that multiple objects are found within the bounding box we constrain the output box to the perimeter of the large object found. Next, we correct the orientation of each symbol in preparation for merging. We also filter out any thin rectangles in this process. Figure 2 illustrates the difference - box 14 represents a standard detection of the graphical element 16 whereas box 18 represents an orientation-corrected bounding box. These are then fed into the contour NMS function. This seeks, for each box, to: - break down the centre point, the width and height and the angle of the box - given a certain intersection threshold, shrink down the width and height accordingly (i.e., if our threshold was 0.7 we would shrink the width and height to 70% its original value) - draw the new resultant box on a mask. If two contours overlap, they are then merged to form a single, new, larger contour. Once this is complete, the system performs an external contour extraction process on the mask to find a new set of bounding boxes. These are then increased in size to reverse the previous shrinking step, such as by the inverse of the initial threshold, or the inverse of the square root of the initial threshold. This can be illustrated using the example image of figure 3, which shows the image 20 to be inputted into the function, along with the three bounding boxes 22, 24, 26 that were provided through its embedded data. As can be seen, these do not always correspond accurately to the actual shape of the image element. After shrinking the area, we can derive the mask 28 shown in figure 4, enabling us to merge boxes 22, 24, 26 and derive a resulting bounding box 30 that encloses the masked area, shown in figure 5. These bounding boxes are converted back into the format x,y,x2,y2 identifying their diagonal corners for the remainder of the process. We then move to identification of the legend, as well as the symbols extracted from the legend, the area of interest (AOI, i.e. the area of the image which contains the relevant constructional diagram that is to be analysed), and any relevant text boxes. We begin with running Tesseract OCR over the pdf in order to extract blocks of text as well as specific legend keywords that may exist in the text. This ensures that any text that is not found in the PDF data is still extracted via OCR. Text boxes are extracted using a combination of the OCR results and the embedded text. We filter OCR text by size using a histogram method, ensuring we remove any outliers that exist outside two standard deviations from the mean; such instances are likely to be false recognitions. The results of this are added to a mask. Separately, the embedded text is also added to its own mask, then the two are combined. Contour detection is then run to extract the final text boxes, and the final mask is also kept for later. With text extracted, the legend can now be extracted. For each key word previously found we first look in its surrounding area in attempt to find a detection - which may be a text item or an image element, but preferably an image element. After finding this detection, we create a search area and iteratively expand as criteria are met. In each iteration, we inspect the new area and look for if detections exist, or if it is filled with text. Figure 6 illustrates this process. Iteration starts with key words that are most likely to be associated with a legend area. For example, in the English language these might be the words "legend", "legends", "symbol", "symbols", "key" and "keys". Once a candidate legend area is found, we check the number of detections present in the legend area and consider any candidates with a count of less than 10 detections a false positive; these are likely to be clusters of text within the image area. We then overlap the legends area onto a mask and run contour extraction to ensure any overlapping areas are combined into a single legend. The symbols are then extracted from the legend and rotated at an angle of 45 degrees for later use. Area of interest detection begins with border removal. Fundamentally, what we assume in this process is that the AOI has some noticeable structure around the symbols i.e., the symbols aren't just floating across the page with a white background. Figure 7 shows one step of the process applied to an example in which there are two true AOIs, but both are contained within a non-linear border. To handle this, we remove the outline of the biggest contour on the page. This process involves creating a mask of the contour, finding its outline via a Laplacian filter then using the result to remove the border. This results in the borderless image of figure 8 in which we can extract AOIs via contour detection. After capturing each individual AOI we run a filter to check the number of detections present, as well as checking the percentage of the AOI that is text. After finding the AOIs, legends and textboxes we filter all detections by these areas, to ensure we are only capturing symbols of interest. Following on from finding the AOI, we can run the computer vision element to capturing symbols. Contour detection is run over the AOI to pick up any shapes that might have been missed. These can include flat annotations (characters without any associated data in the PDF) in addition to flat symbols. This process also aids with completing symbols that have not fully been captured by the previous stages. These contours are passed through a size filter and then the bounding box processing module. The contours are then grouped, using the method outlined later in this document, and symbols that belong to a group with (we prefer) fewer than 3 detections and greater than 100 are filtered out to account for false positives and text. The final boxes are produced by rerunning the bounding box processing module over the combined list of boxes provided by both the contour detection and the PDF data. The data provided by the previous stages enables us to combine each annotation with its symbol and merge fragmented symbols. We do this in three ways, Internal contour matching, Legend Comparison, and Euclidean distance matching. Internal contours are the lowest level objects in an image with nested shapes. This is illustrated in figures 9 and 10; in figure 9, A represents the perimeter of the external contour whereas in figure 10, B represents the perimeter of the internal contour. To get to the "true" lowest level contours we first perform edge detection using canny lines filtering. Imagine now that this shape is fragmented into two different boxes, both of which overlap, but not enough to have been merged by the stage 2 merging algorithm (figure 11). In this case, both boxes share an internal contour B in figure 11. By creating a dictionary representing the relationship of contours to the detections we can find candidates for merging. The data structure relates contours to detections in the form {contour 1: [boxl,box2,box3], contour 2:[box4,box5]....} which allows us to filter each value of the dictionary into lists of boxes that need merging. When we have established boxes that require merging, we combined the lists into chains. This means if we have a series of boxes that requires merging, but with the same box shares in two places, we concatenate both lists, i.e., if I have [boxl,box2] and [box2,box3] from the previous stage we combined these to create [boxl,box2,box3]. If a chain has less than three elements then we go ahead and merge, doing a final check that we haven't captured more than one complete shape before continuing. The other way in which we use internal contours is to ensure complete coverage of certain shapes. For example, this symbol shown in figure 12 is very commonly used in electrical drawings. However, it is often the case that the bounding boxes for this symbol are irregular. They are typically in one of two forms as shown in figure 13, a single box enclosing the circular region only, or fragmented boxes - one for the circular area and another for the tail. The latter case is solved by the previously-described method of matching internal contours to boxes, but the former requires a different solution. We can analyse which contours cross the border of the bounding box by inspecting the external contours coordinates. With this, we can create a list of all contours in the bounding box that cross the border, and expand the border until these contours are completely enclosed. The next process for annotations and overlapping bounding boxes is comparing all symbol detections against the legend. Consider the case shown in figure 14; in this case we have three overlapping bounding boxes for two separate symbols. The theory is that there must be a correct form for this combination, either nothing is merged, or the centre box joins with either the left or right. To test the optimum configuration, we write out possible overlapping combinations for a given symbol, as demonstrated in figure 15. The test subject here is the "E", with a symbol to its left and a symbol to its right. There are in this case 4 possibilities: either the candidate remains unmerged, it is merged to the left, it is merged to the right or it is merged to both. At this point we take each of the above images and test them with the symbols in the legend (figure 16), after applying rotations as necessary to each symbol in the legend. It is worth noting at this point that inclusion within the group of false positives within the legend symbols is not an issue, as these will simply provide a low score for anything we compare with, will therefore not affect this validation process, and will eventually be discarded. The testing procedure uses a pretrained sentence transformer with the clip-ViT-B-32 model weights in order to test pairs of images and return a similarity score. The complete documentation for this process, together with illustrative examples can be found in the git repository https: / / gsthub.CQm / UKPLab / sentence-transfOHners / tree / rnastei7examptes / agj)hgitjpns / ima^ to which the skilled reader is referred for a fuller understanding of this process, and the contents of which are incorporated by reference. By running the comparison for all members, we can establish groups based on a given similarity threshold. Creating groups of the legend and all possible merging combinations allows us to find which are the true combinations by analysing which groups contain symbols found in the legend. It is also worth noting that, while these methods help us handle drawn annotations (as overlapping boxes can be annotations as well as other detections), the process has not actually needed to distinguish between the two. So far, we have been able to treat everything with the same brush; it is the next stage which is focused on separating drawn annotations from detections. Next, we seek to separate all drawn text from all symbols. The advantage of drawn text is that it is always a single character drawn at a time, so we can take that as a fundamental boundary condition to the problem. The proposed solution is as follows: - Iterate through all candidates - Test each individual candidate with an OCR engine and associate each candidate with a detection accuracy - Run the sentence transformer grouping algorithm in order to collect all text into a single group - Use the number of individual detections in a group (hit rate) alongside the average accuracy of the group to classify groups of text vs groups of symbols To iterate through all images, we must first write them out as images to disk. Here we use the stripped version of the image. We then group all detections into their nearly-identical duplicate versions, at a suitable threshold (we use 97.5%), and adjusting for rotation of 90- degree increments, storing which bounding box is related to which group. For every image we then add a border, then resize to 30x30 pixels. This ensures that each single character is an optimum size for the OCR engine. In order to take advantage of GPU acceleration, the "Easy OCR" (https: / / github.com / JaidedAI / EasyOCR) readtext_batched was used, which allows the user to input groups of images to be processed in parallel. Applying four rotations and ensuring the threshold for text discovery is set low, we gather the results together by group for both the hit rate and the average accuracy. Iterating through each group, we append the original detection to the annotations variable if the hit rate is above 90% and the average accuracy is above 90% or if the hit rate is above 40% and the average accuracy is above 98%. We finalise the annotations by doing basic filtering based on size, removing any abnormally large annotations. We also use this method combined with contour detection to establish where flat annotations may exist. Annotation matching is then carried out via a simple distance measurement, fis we have now separated out annotations from detections, we can check the number of detections that exist in the radius of an annotation. The centre of all annotations are found and are compared to all detections centres. The minimum is then extracted, along with the number of detections that exist within a predefined threshold distance from the annotation. If there aren't a large number of detections nearby, and the minimum distance is smaller than the threshold then we create a pair bond between that annotation and the detection. We then iterate through pairs and find the new resultant box from that pair merging. It is also worth noting that we track the content for all annotations, so for any embedded annotations that are merged their new boxes still are linked to the content of the annotation. The final process is to group the resulting boxes together. The process of grouping with sentence transformers has been previously described, however not the inner workings of how we achieve the result. The process begins with the exported detections, written out to the disk. These detections are picked up for "cleaning" which comprises tightening the bounding box to the symbol, adding a fixed border to the entire image and attempting to correct the orientation of all images. We do this to ensure more consistency between images. We also (provided no annotation is present in the bounding box) remove all but the largest external contour in the image, meaning we remove any clutter that might surround the symbol of interest. The next stage is to produce the rotations for each symbol. We create a record of the rotated version and source file, resulting in a structure that looks similar to: { boxl_rotationO: boxl, boxl_rotation90: boxl, boxl_rotation 180: boxl } This is used later for the recombination of groups. We are then ready to encode and compare images for their similarity. As before, this is using sentence transformers on the OpenAI CLIP model. Note, that the current SOTA of this industry is considered to be BERT, developed by Google, Inc. BERT is an NLP model developed specifically for understanding the context in sentences. As described in the paper https: / / arxiv.org / abs / 1908.10084 (to which the skilled reader is referred for a fuller understanding of this process, and the contents of which are incorporated by reference) the sentence-transformer module is a python framework for BERT. We iterate through each individual 1-1 comparison and begin building the network of connected, similar images. We only create a pair bond if the content (if any) of the detection's annotation matches in both cases and if the similarity score is over a pre-stated threshold (in this case 97.5%). Using these individual pair bonds, we collect all chains of pairs and bring them together into one group or list, and also write the content of that group's detections out to a file for later use. Once groups are established, we can look for cases where the same parental symbol has rotated copies of itself spread across multiple groups. If the same parent exists in multiple groups we simply recombine, then as there is no need for the rotated duplicates at this point, we remove them all from the groups and the folders. The leftover or orphan symbols from this grouping process are what we refer to as "singles". The final step to this process is writing out all groups, including the singles, to the disk into a designated folder. It is this folder structure which is used to create the data required for the front-end API. Thus, to summarise, this approach attempts to handle the problem of estimation based on a constructional drawing differently. It aims to use a combination of deep learning, data mining and computer vision to replace the logic that a human would subconsciously process in order to arrive at task completion. The legend matching, grouping, text and annotation extraction all are (at their core) techniques that use AI technology to understand information without the user needing to interfere. It is through these techniques that we have achieved what we believe is the first system that is near-independent, that requires an estimator to review its work rather than provide a tool that fundamentally aids - as is currently the SOTA. It will of course be understood that many variations may be made to the abovedescribed embodiment without departing from the scope of the present invention. Abbreviations and Terminology: AOI Area of interest Stripped Image Redrawn PDF data OCR Optical Character Recognition GPU Graphics Processing Unit CLIP Contrastive Language-Image Pretraining SOTA State of the art BERT Bidirectional Encoder Representations from Transformers

Claims

1. An image analysis method, comprising the steps of;receiving an image file for analysis, the image file representing a constructional plan and including graphical elements and text elements, the text elements being 5 stored as at least one of text information, text bounding boxes, and / or as textpresented in graphical form;identifying a plurality of graphical elements within the image file;grouping the identified graphical elements by:i. selecting a first identified graphical element and a second identified10 graphical element;ii. comparing the first and second identified graphical elements and determining a level of similarity between them;iii. if the level of similarity is greater than a preset threshold, placing the second identified graphical element into a group that includes the first 15 identified graphical element and any other identified graphical elementsalready grouped with the first identified graphical element;repeating steps (i) to (iii) until all graphical elements have been compared and / or placed into a group.

2. An image analysis method according claim 1 in which, prior to grouping, the graphical20 elements are rotated to allow comparison in a uniform orientation.

3. An image analysis system comprising a processor and a data storage element, theprocessor having access to the data storage system on which an image file can be stored, and including instructions for the processor to access the image file and carry out the method of claim 1 or claim 2 on the image file.25 4. An image analysis system according to claim 3 further comprising an optical capturedevice arranged to capture a digital image of a plan and store it in the data storage system for access by the processor.

5. An image analysis system according to claim 4 in which the optical capture system is one of a camera and a scanner.

186. An image analysis system comprising a processor and a data storage element, the processor having access to the data storage system on which an image file can be stored, and including instructions for the processor to access the image file and carry out the method of any one of claims 1 to 5 on the image file.5 7. An image analysis system according to claim 6 further comprising an optical capturedevice arranged to capture a digital image of a plan and store it in the data storage system for access by the processor.

8. An image analysis system according to claim 7 in which the optical capture system is one of a camera and a scanner.10