Anomaly detection systems and processes

Abstract

An anomaly detection system comprises an illumination system and an image acquisition device, wherein the illumination system comprises at least one light source configured to illuminate a surface in a directional manner, the illumination system is configured to selectively illuminate the surface from a selected direction when viewed in plan view of the surface and / or at a selected illumination angle relative to the plane of the surface, and the image acquisition device is configured to image the surface while the surface is illuminated from a selected direction and / or at a selected illumination angle relative to the plane of the surface.

Description

[Technical Field]

[0001] This disclosure relates to the detection of abnormalities in structures, particularly, but not limited to, the detection of cracks and other defects on the surface of structures, which may be concrete surfaces. [Background technology]

[0002] The detection and characterization of anomalies such as defects, cracks, deterioration, and damage are crucial in many industries. For example, the detection and characterization of anomalies in civil and building structures, particularly concrete structures, is becoming increasingly important due to growing environmental and economic pressures to extend the lifespan of aging infrastructure.

[0003] Manual visual inspection is a common approach to inspecting for anomalies. Manual inspection offers many advantages, including human flexibility and the ability to make adaptable, real-time decisions. However, it also has disadvantages, such as cost, reproducibility, safety, availability, training time, and the potential for human error.

[0004] Automated inspection systems and data analysis technologies can address many of the shortcomings of manual inspection, potentially providing a safe, remote, consistent, and time-efficient alternative. However, anomalies are diverse and can be found under a wide range of conditions and circumstances. Some automated solutions may lack the adaptability necessary to reliably identify, evaluate, and characterize anomalies across the diversity of anomaly types and environments.

[0005] At least one example described herein seeks to improve automated inspection systems and methods. [Overview of the project]

[0006] The inventors have confirmed that directional illumination can offer significant advantages in the automatic identification of anomalies on surfaces, and that these advantages can be further enhanced by using specific methods of applying directional illumination. Directional illumination generally occurs when viewed in a plan view of the surface, with light incident on the surface in a specific direction and / or angle. This is in contrast to diffuse illumination, which has no preferred direction. Described herein are examples in which light is projected onto a surface at intentionally varied angles and directions to improve the contrast between anomalies and the background. This is in contrast to ambient or diffuse illumination, where light is projected onto the surface from all directions and angles at once. Furthermore, the inventors have identified methods of processing surface images to enhance anomaly detection and / or characterization in combination with directional illumination techniques, but not to the extent particularly limited.

[0007] At least one aspect of this disclosure is defined in the independent claim. Preferred features are defined in the dependent claim.

[0008] According to a first example of the present disclosure, an anomaly detection system comprises an illumination system and an image acquisition device, wherein the illumination system comprises at least one light source configured to illuminate a surface in a directional manner, the illumination system is configured to selectively illuminate the surface from one or more selected directions and / or at one or more selected illumination angles relative to the plane of the surface when viewed in plan view of the surface, and the image acquisition device is configured to image the surface from one or more directions while the surface is illuminated from the one or more selected directions and / or at one or more selected illumination angles relative to the plane of the surface.

[0009] The illumination system may be configured to illuminate a surface from any of several directions and / or from any of several illumination angles relative to the surface plane. The illumination system may be a non-diffuse illumination system. However, in some examples, the illumination system may be configured to selectively illuminate a surface with diffuse light to collect one or more images of an illuminated surface from one or more selected directions and / or one or more selected illumination angles relative to the surface plane, and to collect one or more other images of a surface illuminated with diffuse light. Each light source may be configured to emit light at a limited range of angles. Each light source may be configured to emit a light profile centered on the optical axis of the light source. Each light source may be configured to preferentially emit light in a particular direction, whether or not it is centered on the optical axis of the light source. Each light source may comprise at least one LED, such as a white LED or an RGB LED, and optionally more LEDs. However, in other examples, different types of light sources may be used. LEDs or other light emitters may be arranged in an array, such as a linear array. Light emitters in an array may be individually controllable, for example, to selectively illuminate individual light emitters in the array or a subset of light emitters in the array.

[0010] The lighting system may include a support. The image acquisition device may be mounted on the support or supported by the support. One or more light sources may be supported by the support, for example, directly or indirectly mounted on the support.

[0011] At least one or each of the light sources may be attached to the support via, for example, at least one manipulator for manipulating the light sources to adjust the position and / or orientation of each light source. Each light source may be attached to the support via at least one respective manipulator.

[0012] At least one manipulator may comprise an arm such as a robotic arm or an articulated arm that may be a joint arm. At least one arm may comprise at least one and optionally a plurality of joints. The arm may be movable at each joint under the action of a drive mechanism such as a servo motor or other motor, an actuator such as a pneumatic or hydraulic actuator, a piston, and / or the like. The arm may comprise at least one or more rigid parts. At least one joint may be configured to rotate at least one joint relative to at least one other joint and / or a support. At least one of the manipulators may comprise a rotor configured to rotate each light source or otherwise change its orientation. At least one of the manipulators may be configured to reposition each light source, for example, to move at least one light source to a different position.

[0013] The manipulator may be controllable by a controller such that, for example, at least one light source is selectively and controllably movable and / or reorientable under the control of the controller. The manipulator may be controllable by a controller to control and / or change the direction and / or illumination angle at which light from each light source impinges on a surface. Although the manipulator can be beneficially used, other means may be used to control and / or change the direction and / or illumination angle at which light from each light source impinges on a surface. For example, selective illumination of a surface by light sources in different orientations may be used, or one or more light sources may be movable and / or reorientable by a robotic entity, a crawler, a drone, or other transport and / or reorientation mechanism.

[0014] The lighting system may include a plurality of light sources. At least one or each light source may be configured to illuminate a surface from a different direction with respect to at least one of each of the other light sources, for example, without repositioning or reorienting the light source. At least one or each light source may be mounted within the lighting system, for example, on a support, so as to face a different direction with respect to at least one or each of the other light sources. Each light source may be mounted on a different manipulator, for example, on a different arm. Each light source may be independently or individually movable and / or reorientable. Each light source may be independently or individually selectively activatable to provide illumination and may be operable to stop operating to turn off the illumination from said light source. Each light source may be independently or individually selectively activatable to selectively illuminate a surface from different directions, for example, from different sides of the surface when viewed in plan view and / or at different illumination angles with respect to the plane of the surface, and to stop operating to turn off the illumination from that light source, while different images or portions of video are being collected.

[0015] Although directional and / or angular illumination of the surface has been described, in addition to directional and / or angular illumination, other distinguishable lighting devices may be used, for example, different color / spectral components (e.g., while illuminating with light of different wavelengths, different images of the surface are collected), different types of illumination (e.g., illuminated by two or more of LED, fluorescent, halogen or laser light), different projection methods or patterns (e.g., spot, dome, line and / or the like that may be selected to conform to the features), different polarizations, different neutral densities, other different filter effects, and / or the like.

[0016] The illumination system may be configured to illuminate the surface with selected colors and / or selected wavelengths or wavelength bands, which may be selected according to the material of the surface or the type of material or anomaly on the surface. For example, the image may be selectively illuminated with blue light, which is particularly suitable for imaging red corrosion. As another example, illumination with green light may be used to reduce or ignore the effects of plants or other organic growth such as moss or algae on the surface. The anomaly acquisition system may include a shroud to at least partially block sunlight, general room lighting, background diffuse lighting and / or similar ambient or background light. The image acquisition device may be mounted on a fixed support or shroud. The illumination system may be mounted on a fixed support or shroud. The shroud may include one or more facets, for example, the shroud may be multifaceted. The shroud may include at least three, for example, four or more sides. The shroud may define an inner surface that can define a recess, for example, a concave surface. The inner surface may include a back and one or more side walls. The side walls may extend around the rear wall, for example, completely around it. The side walls may extend diagonally or perpendicularly to the rear wall. One or more facets may be provided on the inner side walls of the shroud. At least one image acquisition device may be provided in or on the rear wall. At least one light source may be mounted on the inner surface of the shroud or provided in other ways. At least one light source may be provided on one or more or on each facet. At least one light source, for example, at least one LED or strip or array of LEDs, may be provided on two or more or on each of each facet. At least one or each facet may have a different angle with respect to at least one or each other facet. Facets may extend around at least one, for example, two or more or each side of the side wall. Light sources may be provided on at least one or at least one or each facet of at least one or each side of the shroud. In this way, the surface may be selectively illuminated from different directions and / or from different angles with respect to the surface plane by selectively activating different light sources provided on different facets on the inner surface of the shroud.At least part or all of the inner surface of the shroud may be black and / or matte, for example, matte black. This may reduce reflection, improve the directionality of illumination, and avoid significant diffuse illumination.

[0017] In an alternative configuration, light sources, for example, at least one per side, may be mounted on an electric rail or other control system to move the light sources, for example, to change the direction and / or angle of illumination. In another variation, the light sources may be mounted on at least one manipulator within the shroud.

[0018] The image acquisition device may be controllable, for example, by a controller, to capture at least one image or video of a surface. The images may include grayscale and / or color images. The illumination system may be configured to selectively illuminate the surface from different directions, e.g., from different sides of the surface when viewed in plan view, and / or at different illumination angles relative to the plane of the surface, while different parts of the image or video are being acquired. The illumination system may be configured to selectively illuminate the surface with different types of illumination. Different types of illumination may include diffuse illumination and / or hard (non-diffuse) illumination. Each part of the image or video may show a surface illuminated from different directions, e.g., from different sides when viewed in plan view, and / or at different illumination angles relative to the plane of the surface, and / or with different types of illumination. For example, each image of at least one of multiple parts of multiple images or videos may show a surface illuminated from different directions that are opposite and / or perpendicular to each other when viewed in plan view of the surface. Each portion of an image or video may show a surface illuminated from at least two, three, four, or eight or more sides when viewed in a plan view of the surface, for example, a surface illuminated from the left, right, top, and / or bottom when viewed in a plan view of the surface. At least one image or at least a portion of a video may be of a surface illuminated in one direction when viewed in a plan view of the surface, another portion of another image or video may be of a surface illuminated from different directions when viewed in a plan view of the surface, and one or more further images or portions of a video may each be of a surface illuminated from one or more further different directions when viewed in a plan view of the surface. The image acquisition device may be controlled, for example by a controller, to capture at least one of several images or videos of a surface sequentially illuminated from different directions, for example, at least two, optionally at least three, at least four, or at least eight or more surfaces sequentially illuminated from different directions. Different portions of different images or videos may show a surface illuminated from different directions from at least two, optionally at least three, or at least four different directions.

[0019] The illumination system may be configured to dynamically change the direction in which a surface is illuminated, for example, by moving and / or reorienting each of one or more light sources, or by providing different light sources oriented in different directions and / or at different angles to the surface plane, and selectively illuminating only one or a subset of the light sources to change the direction and / or angle of illumination of the surface. The illumination system may also be configured to dynamically change the direction in which a surface is illuminated based on the quality value of the resulting image, for example, in order to maximize, optimize, or improve the quality value of the acquired image. The quality value may be a defined measure of image quality and / or any other metric indicating the system's ability to distinguish anomalies such as uncertainty, training efficiency, etc.

[0020] The lighting system may be configured to illuminate the surface at an angle. For example, the lighting system may be configured to position the light source at an angle to the surface plane when illuminating the surface. For example, the lighting system may be configured to position the light source so that its optical axis is at an angle to the surface plane when illuminating the surface. The light source may be positioned, or the lighting system may be configured to position the light source, to illuminate the surface at an angle to the surface plane in the range of 0° to 90°, preferably 5° to 50°, for example, 20° to 50°, 30° to 40°, etc. For example, the lighting system may be configured to position the light source so that its optical axis is at an angle in the range of 5° to 50° to the surface plane when illuminating the surface. The lighting system may be configured to illuminate the surface from different lighting angles to the surface plane, or at the same lighting angle to the surface plane, when different parts of an image or video are collected. The lighting system may be configured to reorient the light source to illuminate the surface from different angles to the surface plane when different parts of an image or video are collected. Each part of the image or video may show a surface illuminated from different angles to the surface plane.

[0021] The lighting system may be configured to dynamically change the illumination angle relative to the plane of the surface being illuminated, for example by moving and / or reorienting one or more or each of the light sources. The lighting system may also be configured to dynamically change the illumination angle relative to the plane of the surface being illuminated, based on the quality value of the resulting image, for example, to maximize the quality value of the resulting image.

[0022] One or more image acquisition devices may be digital, for example, digital cameras. The image acquisition devices may include RGB cameras, infrared cameras, hyperspectral cameras, multispectral cameras, UV cameras, and / or similar. One or more image acquisition devices may include CMOS, CCD, or other suitable image acquisition sensors.

[0023] One or more image acquisition devices may be configured to image a surface from at least one of one or more directions, one or more angles, and / or one or more different positions, for example, while an illumination system is illuminating the surface. One or more image acquisition devices may be reconfigurable, or otherwise configured to image the surface from different directions and / or different angles with respect to the plane of the surface. One or more image acquisition devices may be configured to collect at least one, for example, multiple images of the surface while it is illuminated by an illumination system, and at least one of the images may be collected by at least one of the one or more image acquisition devices from at least one of different directions, different positions, different types of illumination, and / or different angles with respect to the plane of the surface, relative to at least one other image acquisition device that collects another image. At least one of the images may be collected by at least one image acquisition device that is facing a different direction, positioned at a different location, and / or oriented at a different angle, relative to at least one other image acquisition device that collects another image. At least one image may be collected by at least one image collecting device, which is reconfigurable to be facing in different directions, positioned in different locations, and / or oriented at different angles with respect to a surface plane, and is operable to collect different images while facing in different directions, positioned in different locations, and / or oriented at different angles.

[0024] In the example, different images of multiple images may be of surfaces illuminated from opposing directions, and yet another image may be of surfaces in different opposing directions from at least one pair, which may be perpendicular to at least one other pair of opposing directions shown in the image. Images of surfaces illuminated from different directions may be under non-diffuse (hard) illumination. At least one other image may be of a surface under diffuse illumination. In one example, the images may include images of a surface taken when illuminated with non-diffuse (hard) light from the up, down, left, and right directions, and images of a surface taken under diffuse light, and all images are optionally grayscale images.

[0025] The image acquisition device may be mounted to a support via one or more control devices, such as actuators, robot arms, one or more rotatable joints and / or similar. The control devices may be configured, for example, to reposition and / or reorient at least one of the one or more image acquisition devices to change the direction and / or angle in which at least one image acquisition device images a surface. The control devices may include, for example, powered control devices equipped with motors such as stepper motors or servo motors, actuators, pneumatic or hydraulic actuators, piston arrays, electroactive polymer actuators, etc., which may be controllable by a controller. The control devices may be configured, for example, to reposition and / or reorient at least one of the one or more image acquisition devices in response to a command from a controller. The control devices may be configured to adjust at least one of the pitch, roll and / or yaw of at least one image acquisition device. At least one or each image acquisition device may be reconfigurable individually and / or separately, for example, under the action of the associated control devices.

[0026] At least one or each image acquisition device may be reconfigurable individually and / or separately.

[0027] One or more image acquisition devices may be configured to acquire depth, surface normals, or other 3D data. One or more image acquisition devices may be configured for photometric stereo, stereoscopic imaging, or other 3D imaging. The anomaly detection system may comprise multiple image acquisition devices, each of which may be configured to image the surface from at least one different direction, a different position, and / or a different angle to the plane of the surface in order to image the surface, for example, to acquire depth, surface normals, or other 3D data. For example, other 3D imaging may include stereophotogrammetry, infrared imaging (such as that used in Microsoft's Kinect system), LIDAR, RADAR, magnetic or electric field sensors, laser scanners, and / or similar.

[0028] An anomaly detection system may be configured to collect images for use in photometric stereo image processing, for example, to determine normals to a surface at multiple locations. The anomaly detection system may be configured to collect at least one or more images of a surface from a fixed viewpoint, each of which images is captured under different lighting conditions. The number of images collected may be one or more, for example, 1 to 100 images, for example, 5 to 20 images, 10 images, etc. This number of images has been shown to provide a good trade-off between accuracy and processing efficiency. The anomaly detection system may be configured to determine surface normals for at least a portion of a surface from at least one or more images by providing at least one or more images as input to an appropriate process, which may be a white-box process or a black-box process. A white-box process may include one or more defined equations or other mathematical processes that together derive surface normals for at least a portion of a surface from the input images. The defined equations or other mathematical processes may be known in the field of photometric stereo technology. The black box method may include one or more appropriate machine learning or artificial intelligence models, such as well-trained machine learning or artificial intelligence models, or self-learning models such as unsupervised learning models, semi-supervised learning models, self-supervised learning models, reinforcement learning models, or adversarial models.

[0029] The anomaly detection system may be configured to determine a 3D model or mesh of at least a portion of the surface, which may be done, for example, by using the determined normals, by using photometric stereo, stereoscopic images, or other 3D images. For example, the 3D data of the surface that may be generated from the determined normals and / or from multiple pairs of stereoscopic images or other 3D images may be stitched, merged, or otherwise combined to form a 3D model of the mesh. As an example, the determined normals may be used as input to a further black-box or white-box process to derive a 3D model or mesh from the determined normals. The further black-box or white-box process may be similar to those described above.

[0030] Multiple pairs of stereoscopic images or other 3D images may include images of at least a portion of a surface taken while the surface is illuminated in one or more selected directions and / or at one or more selected illumination angles relative to the surface plane. That is, a 3D model or mesh of at least a portion of a surface may be formed from multiple pairs of stereoscopic images, with different pairs of stereoscopic images taken while the surface is illuminated in different directions and / or at different illumination angles relative to the surface plane. Optionally, at least one of the pairs of stereoscopic images may include an image of at least a portion of the surface under diffuse illumination.

[0031] An anomaly detection system may be configured to detect and / or characterize anomalies from a 3D model or mesh. Characterizing an anomaly from a 3D model or mesh may include determining one or more geometric properties of the anomaly, such as depth, volume, length, width, or other extent. For example, characterizing an anomaly from a 3D model or mesh may include determining the volume and / or height of an anomaly such as delamination or interlayer delamination.

[0032] An anomaly detection system may be configured to predict areas of a surface that are more likely to experience anomalies in the future by analyzing a 3D model or mesh to identify features that indicate potential future anomalies, such as localized surface elevation or changes in surface height. These features may be stored in a library defining the characteristics of potential future anomalies. To identify areas of a surface that may exhibit potential future anomalies, the anomaly detection system may be configured to compare portions of a 3D model or mesh of a surface with these features to identify areas of the surface that match any of the characteristics of potential future anomalies. However, this disclosure is not limited thereto, and other suitable techniques for identifying areas of a surface that may exhibit potential future anomalies will be apparent based on this disclosure.

[0033] The anomaly detection system may be configured to generate a visualization of a 3D model or mesh, which may highlight any anomalies. The anomaly detection system may be configured to process at least one image to pinpoint the location of anomalies, for example using a black-box process. The anomaly detection system may be configured to translate the location of anomalies in at least one image to a location within the 3D model or mesh. Alternatively, the 3D model or mesh may be processed to pinpoint the location of anomalies within the 3D model or mesh, for example using a black-box process.

[0034] At least one of the multiple image acquisition devices may be configured to image a different wavelength range or be of a different type than at least one of the other image acquisition devices. At least one or each of the image acquisition devices may include or be configured to accept filters such as polarizing filters or spectral filters. Different image acquisition devices among the multiple image acquisition devices may have or be provided with different filters for at least one or each of the image acquisition devices, for example, to transmit different polarizations or different spectral ranges.

[0035] An anomaly detection system may include or be configured to communicate with an analysis system. The analysis system may be a controller or be contained within a controller. The analysis system may be a separate analysis system from the controller or may be provided with one, and it may be a local or remote analysis system. The analysis system may be distributed between the controller and the external analysis system. The analysis system may be configured to analyze at least one or more images or videos to analyze different parts of different images or videos of the surface when illuminated from different directions, in order to analyze any anomaly on the surface. Analysis of any anomaly on a surface may include at least one of identifying any anomaly and / or characterizing any anomaly. Identifying any anomaly may include detecting the presence or other of any anomaly. Characterizing an anomaly may include identifying the anomaly, such as the location of the anomaly on the surface; classifying the anomaly, for example, determining the type of anomaly; determining the extent of the anomaly, for example, determining one or more dimensions or ranges of the anomaly, such as the length, width, depth, shape or area of ​​the surface associated with any anomaly, and / or at least one of the same.

[0036] The analysis system may be configured to determine confidence values ​​for at least one or each step of the analysis, for example, confidence that a detected anomaly exists, confidence that the anomaly is located at a specified location, confidence that the anomaly is of a specified type, confidence that the anomaly is within a specified range, and / or one or more of the same. The analysis system may be configured to provide confidence values ​​determined for at least one of the above steps (for example, to assist in operator decision-making) and / or to make some decisions based at least in part on confidence values ​​determined for at least one of the above steps.

[0037] The analysis system may be configured to flag that further image acquisition is required based on a determined confidence value. For example, if the determined confidence value falls below a set or pre-set threshold, the analysis system may be configured to flag that further image acquisition is required. The flag indicating that further image acquisition is required can be provided to the controller, for example, so that the controller can acquire one or more further images of the surface in response to the flag, and / or change at least one of the following: the direction in which the surface is illuminated, the illumination angle of the surface relative to the plane of the surface, and the direction, position, and / or angle in which the surface is imaged by the image acquisition device. The controller may be configured to display or provide the flag to the operator.

[0038] The analysis system may be configured to aggregate or combine multiple portions of an image or video, where each portion of the image or video shows a surface illuminated in a different direction and / or at a different illumination angle relative to the other portions of the image or video.

[0039] The analysis system may be configured to distinguish between parts of an image or video representing anomalies, such as regions, blocks, or pixels, from parts of an image or video representing the background, such as regions, blocks, or pixels, such as a concrete surface, which is the majority, expected, and / or regular surface. The analysis system may also be configured to segment parts of an image or video representing anomalies, such as pixels or video, from parts of an image or video representing the background, such as pixels or video.

[0040] The analysis system may be configured to include and / or implement one or more models that may be machine learning models.

[0041] At least one of the models may be a discriminative model, or may include one. The discriminative model may be configured to identify the presence of anomalies. For example, the discriminative model may be configured to identify a portion of an image or video that represents an anomaly relative to a portion of an image or video that represents a background. The discriminative model may be configured to identify a region of a surface containing the identified anomaly. For example, the discriminative model may be configured to assign one or more bounding boxes around a specified portion of an image or video that represents an anomaly. The discriminative model may be a neural network, preferably (but not required), such as a convolutional neural network, including a region-based convolutional neural network (R-CNN). The discriminative model may include a two-part or two-shot machine learning model. One part of the discriminative model may be configured to detect features of interest relative to the background and / or include a region proposal network. Another part of the discriminative model may include a classification model for classifying whether a feature of interest belongs to a portion of an image or video that represents an anomaly. The classification model may include a convolutional neural network (CNN), etc.

[0042] The discriminant model may be at least partially trained using trained data or training data, which may include, but are not limited to, manually annotated trained data or manually assigned descriptive data, and may also be images of surfaces of similar materials having, for example, one or more parts of an image or video representing an anomaly that has been pre-identified, for example, manually pre-identified, and / or artificially generated and / or automatically labeled trained data. The discriminant model may be configured to take at least one image or at least part of a video as input, and may be trained to identify at least one part of an image or combination of images or at least part of a video representing an anomaly, and optionally apply one or more bounding boxes around it. At least one discriminant model may be trained to at least roughly or approximately detect and / or locate at least one part of an image or at least part of a video representing an anomaly, for example, within the limits of the accuracy of one or more bounding boxes. The one or more bounding boxes may optionally be of a set size or a set size, and may be quadrilaterals. The discriminant model may be, for example, a fast and / or computationally efficient machine learning model compared to at least one other machine learning model.

[0043] The analysis system may be configured to extract or isolate at least one portion of an image or at least one portion of a video within an arbitrary bounding box. The analysis system may be configured to identify at least one portion of an image or at least one portion of a video outside the bounding box as a background, for example as a normal surface, for example by labeling it in metadata, or in other ways.

[0044] At least one other model in the model may be a locator and / or classification model, or may include one. The locator and / or classification model may be a machine learning model, or may include one. The locator and / or classification model may be configured to take as input at least one portion of an image or at least one portion of a video within an arbitrary bounding box, which has been identified, for example, by a discriminator model. The locator and / or classification model may be trained to locate and / or classify at least one portion of an image or at least one portion of a video within a bounding box, e.g., a block, which represents at least one portion of anomaly.

[0045] The locator and / or classification model may be more accurate and / or have higher resolution than at least one discriminative machine learning model. The locator and / or classification model may be more computationally intensive and / or slower than the discriminative machine learning model. The locator and / or classification model may be configured to identify one or more blocks that include at least one portion of an image or at least one portion of a video that represents an anomaly. Each block of the one or more blocks may be smaller than the bounding box and, for example, have fewer pixels. Each block may represent multiple pixels. Each block may be a set or preset size and may contain a set or preset number of pixels. Each block may be the same size. Thus, the blocks output by the locator and / or classification model may be a subset of the bounding boxes output by the discriminative model. The locator and / or classification model may be trained by transfer learning. The locator and / or classification model may be trained using training data. For example, training data with manually annotated or manually assigned descriptive training data, such as images of the surface of similar material with one or more parts of an image or video representing anomalies that have been previously identified, for example, manually identified. The locator and / or classification model may include a convolutional neural network (CNN) such as a VGG-16 neural network, but other models or variations of models may be used. For example, in another example, the YOLO algorithm may be used. Additionally or alternatively, different channels and / or different numbers of channels may be input to the locator and / or classification model.

[0046] Multiple channels may be provided as inputs to the locator and / or classification model. At least some or each of the input channels may include images of the surface under different lighting conditions, for example, but not limited to, images of the surface illuminated from one or more different directions when viewed in plan view of the surface, and / or images of the surface illuminated at one or more different lighting angles relative to the plane of the surface. The images may include grayscale and / or color images. For example, the input to the locator and / or classification model may have three channels corresponding to red, green, and blue in a color image, but may have more, fewer, or different channels, for example, channels containing images corresponding to one or more of the following: different lighting directions, different lighting angles, different lighting types, different directions, positions and / or angles of the image acquisition device and / or similar. Input channels to the locator and / or classification model corresponding to different lighting directions may include images of the surface illuminated from one or more different sides of the surface.

[0047] For example, the input channels may include images of the surface illuminated from at least one, two, or more pairs of opposing sides relative to a plan view of the surface. For example, at least one or more channels may include images of the surface while it is illuminated from at least one or more, or each of, the left, right, top, and / or bottom of the surface when viewed in a plan view of the surface. Different types of illumination may include diffuse illumination and / or hard (non-diffuse) illumination. In a specific example, the locator and / or classification model may include multiple input channels, each of which corresponds to an image collected while the surface is illuminated from different directions, e.g., from opposing sides relative to a plan view of the surface, with one corresponding to diffuse illumination or hard illumination, and at least one other input channel corresponding to an image collected by illumination of the surface under the other of hard illumination or diffuse illumination. As an example, the locator and / or classification model may include five input channels, which may include channels corresponding to illumination of the surface from at least two pairs of opposing directions, such as top, bottom, left, and right, relative to the surface when viewed in a plan view under hard illumination, and input channels corresponding to the surface when illuminated by diffuse illumination.

[0048] The analysis system may be configured to perform pixel-level segmentation, for example, to segment one or more anomalies from the rest of the surface. At least one image can be used as input to the pixel-level segmentation. Pixel-level segmentation may include applying a suitable white-box model to at least one image to segment one or more anomalies from the rest of the surface.

[0049] The analysis system may be configured to apply a two-stage layered machine learning model approach. The analysis system may be configured to roughly locate at least one portion of an image or video or at least one portion of a video that represents an anomaly, and to provide one or more bounding boxes around the locably located at least one portion of the image or video or at least one portion of the video that represents the anomaly. The rough location of at least one portion of an image or video or at least one portion of a video that represents an anomaly may be performed using at least one locator machine learning model. The analysis system may be configured, for example, to use at least one locator and / or classification machine learning model to more accurately identify portions, e.g., blocks, within one or more bounding boxes that represent an anomaly for at least one portion of an image or video. Any blocks within one or more bounding boxes of at least one portion of an image or video that have not been identified as representing at least one portion of an anomaly may be designated as background, i.e., normal or expected portions of the surface.

[0050] The analysis system may be configured to perform one or more white-box or classical image processing methods. One or more white-box or classical image processing methods may be non-machine learning and non-artificial intelligence techniques, mathematical filters, mathematical transformations, and / or similar. The white-box or classical technique may include a segmentation algorithm for segmenting pixels representing anomalies from pixels representing the background. The white-box or classical technique may include, for example, edge detection for detecting the edges of any anomalies. The white-box or classical technique may be configured to process the output from one of the machine learning models, for example, a locator and / or a classification machine learning model. The white-box or classical technique may be applied to blocks output by the locator and / or classification machine learning model and may be configured to further refine blocks identified as containing at least one portion of an image or at least one portion of a video representing an anomaly.

[0051] The analysis system may be a hybrid analysis system that includes both one or more machine learning or black-box models and white-box or classical techniques. The black-box model may include a trained or statistical model rather than, for example, a manually defined absolute mathematical relationship. The white-box or classical technique may be configured to refine the output of one or more machine learning models or to accept it as input.

[0052] The above configuration is advantageous in that a two-stage machine learning model is applied followed by the application of white-box or classical techniques, but any combination of any order of these techniques may be applied. For example, only one machine learning model may be applied, or only white-box or classical techniques may be applied, or a combination of one machine learning model and white-box techniques, or white-box techniques may be applied before one or both machine learning models. Therefore, either black-box or machine learning techniques and / or either white-box or classical techniques, or other, may be applied in any order, or the functions described in relation to any technique may be combined into a common model or step, or certain models or steps may be omitted, repeated, or otherwise modified.

[0053] The analysis system may be configured to accept multiple image or video segments as input. Each image of one of the multiple image or video segments may show a surface illuminated from different directions and / or illuminated at different illumination angles relative to the plane of the surface when viewed in a planar view of the surface. Each image of one of the multiple image or video segments may, but is not limited to, show a surface illuminated from different directions that may be opposite and / or perpendicular to each other. Each image or video segment may show a surface illuminated from at least two, three or all of the left, right, top and / or bottom directions when viewed in a planar view of the surface, and / or include additional or other directions. Multiple image or video segments showing different parts of different images or videos that show a surface illuminated from different directions and / or illuminated at different illumination angles relative to the plane of the surface when viewed in a planar view of the surface can be analyzed by the analysis. The analysis system may be configured to combine the results of analyzing multiple image or video portions, for example, the outputs of a discrimination model and / or at least one locator and / or a classification model and / or at least one white-box or classical technique. For example, combining may include applying an "OR" function, such as a bitwise OR function, to the analysis results of multiple images, but other forms of combining may be used. For example, an alternative form of combining may include summing the intensities of the outputs of a locator and / or a classification model from images illuminated in multiple directions to arrive at a probability heatmap showing where anomalies are located.

[0054] The analysis system may be configured to dynamically determine an optimal or preferred configuration, which includes at least one of the following: the optimal or preferred direction and / or angle of surface illumination, the optimal direction, position and / or angle of the image acquisition device, the optimal or preferred color or wavelength of illumination and / or similar. The analysis system may be configured to dynamically determine an optimal or preferred configuration by collecting multiple images of the surface illuminated from different directions and / or illumination angles and / or different directions, positions and / or angles of the image acquisition device, and determining the quality value of each image. The illumination system may be controllable, for example, in response to a controller, to dynamically illuminate the surface from the direction and / or angle of surface illumination and / or different directions, positions and / or angles of the image acquisition device determined to be optimal or preferred. The illumination system may be controllable to dynamically rearrange and / or reorient one or more light sources and / or dynamically rearrange, reorient or use one or more image acquisition devices in the determined optimal or preferred configuration to dynamically illuminate the surface from the direction and / or angle of surface illumination. The analysis system may be configured to dynamically determine the optimal or preferred illumination direction and / or angle for each of the multiple surfaces.

[0055] An anomaly detection system utilizes directional illumination to illuminate a surface. Thus, the anomaly detection system may be configured to utilize the illumination system to illuminate a surface in different directions and / or at different illumination angles relative to the surface plane when different portions of an image or video are being collected by an image acquisition device. An analysis system may analyze each of the multiple portions of the image or video, each different portion of the image or video showing a surface illuminated from a different direction relative to other images or portions of the video. The analysis system can analyze each portion of the image or video to identify and / or characterize surface anomalies, for example, using at least one machine learning model and / or white-box or classical techniques. In this way, the identification and / or characterization of surface anomalies can be enhanced.

[0056] The analysis system may be configured to process multiple image or video portions of a surface, where different image or video portions are collected by one or more image acquisition devices that are oriented in different directions, located in different places, and / or oriented at different angles. Different images of the multiple images may be collected by the same or different image acquisition devices. The analysis system may be configured to process multiple image or video portions of a surface, where different image or video portions are of the surface illuminated in different directions and / or at different illumination angles relative to the surface plane. The analysis system may be configured to process multiple image or video portions to determine the surface normal of the surface from the multiple image or video portions, for example, from multiple image or video portions showing the surface illuminated in different directions and / or at different illumination angles relative to the surface plane. The analysis system may be configured to determine the surface normal of the surface using photometric stereo techniques (e.g., https: / / en.wikipedia.org / wiki / Photometric_stereo). The surface normals of a surface may be used to form a surface map and / or a 3D model or mesh of at least a portion of the surface, which may be the aforementioned 3D model or mesh. The surface normals may be derived from at least one image, for example, using the photometric stereo technique described above.

[0057] A second example of the present disclosure is a method for identifying and / or characterizing an anomaly on a surface, the method comprising the steps of providing an illumination system and at least one image acquisition device, the illumination system comprising at least one light source configured to illuminate the surface in a directional manner, the method further comprising the steps of using the illumination system to selectively illuminate the surface from at least one selected direction and / or at one or more selected illumination angles relative to the plane of the surface when viewed in plan view of the surface, and using the at least one image acquisition device to image the surface from at least one direction while the surface is illuminated from the at least one selected direction and / or at one or more selected illumination angles relative to the plane of the surface.

[0058] The method may include providing an anomaly detection system according to the first example, and the illumination system and image acquisition device are included in the anomaly detection system.

[0059] The method may include illuminating a surface from any of a plurality of directions and / or from any of a plurality of illumination angles with respect to the plane of the surface. The method may also include illuminating a surface from different directions and / or from different angles with respect to the plane of the surface by manipulating at least one light source using at least one manipulator as described in relation to the first embodiment and / or by selectively starting and stopping selected light sources that are facing different directions and / or at different angles.

[0060] The method may include, in addition to directional and / or angular illumination, the use of other distinguishable illumination devices, such as illuminating a surface with different color / spectral components (e.g., collecting different images of the surface while it is illuminated with light of different wavelengths), different types of illumination (e.g., illumination by two or more LEDs, fluorescent, halogen or laser light), different projection methods or patterns (e.g., spots, domes, lines and / or similar, which may be selected to suit the features), different polarizations, different neutral densities, other different filtering effects, diffuse and non-diffuse illumination, and / or similar.

[0061] The method may include illuminating the surface with a selected color and / or a selected wavelength or wavelength band, which may be selected according to the material of the surface or the type of material or anomaly on the surface. The color or wavelength or wavelength range of the light used for illumination may be selected to highlight or, alternatively, reduce or mitigate anomalies or materials on the surface. For example, the image may be selectively illuminated with blue light, which may be particularly suitable for imaging red corrosion, as red corrosion may be emphasized in the image. As another example, illumination with green light may be used to reduce, mitigate or ignore the effects of plants or other organic growth such as moss or algae on the surface.

[0062] The method may include selectively illuminating a surface from different directions, for example, from different sides of the surface when viewed in plan view, and / or at different illumination angles relative to the plane of the surface, while different portions of the image or video are being collected.

[0063] This method may include dynamically changing the direction in which the surface is illuminated, for example, to maximize, optimize, or improve the quality value of the resulting image, based on the quality value of the resulting image.

[0064] This method may also include arranging a light source to illuminate the surface at an angle to the plane of the surface in the range of 0° to 90°, preferably in the range of 5° to 50°, for example, in the range of 20° to 50°, 30° to 40° (including both extreme values), etc.

[0065] This method may include collecting multiple images of a surface, at least one of which may be collected by at least one of one or more image acquisition devices from a different direction, a different position and / or a different angle with respect to the plane of the surface relative to at least one other image.

[0066] A third example of the present disclosure is a computer program product, when implemented on a controller of an anomaly detection system such as the anomaly detection system of the first embodiment, configured to cause the controller to control an illumination system to selectively illuminate a surface from a selected direction and / or at a selected illumination angle relative to the plane of the surface, when viewed in plan view of the surface, and the image acquisition device is configured to image the surface while the surface is illuminated from the selected direction and / or at the selected illumination angle relative to the plane of the surface. The computer program product may be configured to cause the controller to implement the method of the second example.

[0067] A fourth example of the present disclosure is a method for analyzing an image of a surface to determine anomalies on the surface, the method comprising applying a hybrid analysis which includes applying one or more machine learning or black-box models to an image of the surface to determine blocks of the image that contain at least a portion of the anomalies, and applying one or more classical or white-box techniques to blocks of the image that have been determined to contain at least a portion of the anomalies, thereby distinguishing between portions of the block that do not represent anomalies and portions of the block that represent anomalies.

[0068] This method may include analyzing at least one image or video to analyze different parts of different images or videos of the surface when illuminated from different directions, in order to analyze any anomaly on the surface. Analysis of any anomaly on the surface may include at least one of identifying any anomaly and / or characterizing any anomaly. Identifying any anomaly may include detecting the presence or other of any anomaly. Characterizing an anomaly may include identifying the anomaly, such as its location on the surface; classifying the anomaly, for example, determining the type of anomaly; determining the extent of the anomaly, for example, determining one or more dimensions or ranges of the anomaly, such as the length, width, depth, shape or area of ​​the surface associated with any anomaly, and / or at least one of the same.

[0069] The method may include determining confidence values ​​for at least one or each step of the analysis, for example, confidence that a detected anomaly exists, confidence that the anomaly is located at a specified location, confidence that the anomaly is of a specified type, confidence that the anomaly is within a specified range, and / or similar. The method may include providing confidence values ​​determined for at least one of the steps above (for example, to assist in operator decision-making) and / or making decisions based at least in part on confidence values ​​determined for at least one of the steps above.

[0070] The method may include flagging that further image acquisition is required based on a determined confidence value, for example, if the determined confidence value falls below a set or pre-set threshold, the method may include flagging that further image acquisition is required. The flag indicating that further image acquisition is required may be provided to the controller, for example, so that the controller can acquire one or more further images of the surface in response to the flag, and / or change at least one of the following: the direction in which the surface is illuminated, the illumination angle of the surface relative to the plane of the surface, the direction, position, and / or angle in which the surface is imaged by the image acquisition device. The method may include acquiring further images with at least one of the following changed: the direction in which the surface is illuminated, the illumination angle of the surface relative to the plane of the surface, and the direction, position, and / or angle in which the surface is imaged by the image acquisition device. The method may include displaying or otherwise providing the flag to the operator.

[0071] The method may include aggregating or combining multiple images or portions of the video, wherein each image of the multiple images or portions of the video shows a surface illuminated in a different direction and / or at a different illumination angle relative to the other images or portions of the video.

[0072] The method may include distinguishing a portion of an image or video representing an anomaly, such as an area, block, or pixel, from a portion of an image or video representing the background, such as an area, block, or pixel, such as a concrete surface, which is a larger, expected, and / or regular surface. The method may also include segmenting a portion of an image or video representing an anomaly, such as a pixel or video, from a portion of an image or video representing the background, such as a pixel or video.

[0073] This method may include implementing one or more models that may be machine learning models.

[0074] At least one of the models may be, or may include, an identification model. The identification model may be configured to identify a region of a surface containing an identified anomaly. For example, the identification model may be configured to assign one or more bounding boxes around a specified portion of an image or a portion of a video that represents an anomaly. The identification model may be, or include, at least one or any or all of the features of the identification models described above in relation to the first example.

[0075] This method may include extracting or isolating at least one portion of an image or at least a portion of a video within an arbitrary bounding box determined by an identification model. This method may also include identifying at least one portion of an image or at least a portion of a video outside the bounding box as a background, for example as a normal surface, for example by labeling it in metadata or otherwise.

[0076] At least one other model in the model may be a locator and / or classification model, or may include them. The locator and / or classification model may be at least one, any, or all features of the locator and / or classification model described above in relation to the first example, or may include them. The locator and / or classification model may be configured to take as input at least one portion of an image or at least a portion of a video within an arbitrary bounding box, which has been identified, for example, by a discrimination model. The locator and / or classification model may be trained to locate and / or classify at least one portion of an image or at least a portion of a video within a bounding box, e.g., a block, which represents at least a portion of anomaly.

[0077] The locator and / or classification model may be a machine learning model or may include a machine learning model. The locator and / or classification model may be configured to take as input at least one portion of an image or at least one portion of a video within an arbitrary bounding box, which has been identified, for example, by a discriminator model. The locator and / or classification model may be trained to locate and / or classify at least one portion of an image or at least one portion of a video within a bounding box, e.g., a block, which represents at least one portion of anomaly.

[0078] This method may include applying a two-layer machine learning model approach. This method may include roughly localizing at least one portion of an image or video or at least one portion of a video that represents an anomaly, and providing one or more bounding boxes around the localized at least one portion of the image or video or at least one portion of the video that represents the anomaly. The rough localization of at least one portion of an image or video or at least one portion of a video that represents an anomaly may be performed using at least one locator machine learning model. This method may include, for example, using at least one locator and / or classification machine learning model to more accurately identify portions, e.g., blocks, within one or more bounding boxes of at least one portion of an image or video that represent an anomaly for at least one portion of an image or video. Any blocks within one or more bounding boxes of at least one portion of an image or video that have not been identified as representing at least one portion of an anomaly may be designated as background, i.e., normal or expected portions of the surface.

[0079] The method may include performing one or more white-box or classical image processing methods. One or more white-box or classical image processing images may be at least one, any, or all features of the white-box or classical image processing methods described above in relation to the first example, or may include them. The white-box or classical technique may include, for example, edge detection for detecting edges of any anomalies. The white-box or classical method may be configured to process the output from one of the machine learning models, for example, a locator and / or classification machine learning model. The white-box or classical method may be applied to blocks output by the locator and / or classification machine learning model and may be configured to further refine blocks identified as containing at least one portion of an image or at least a portion of a video that represents an anomaly.

[0080] This method may involve using a hybrid analytical system that includes both one or more machine learning or black-box models and white-box or classical techniques. The black-box models may include, for example, trained or statistical models rather than manually defined absolute mathematical relationships. The white-box or classical techniques may be configured to refine or accept the output of one or more machine learning models as input.

[0081] The above configuration is advantageous in that a two-stage machine learning model is applied followed by the application of white-box or classical techniques, but any combination of any order of these techniques may be applied. For example, only one machine learning model may be applied, or only white-box or classical techniques may be applied, or a combination of one machine learning model and white-box techniques, or white-box techniques may be applied before one or both machine learning models. Therefore, either black-box or machine learning techniques and / or either white-box or classical techniques, or other, may be applied in any order, or the functions described in relation to any technique may be combined into a common model or step, or certain models or steps may be omitted, repeated, or otherwise modified.

[0082] This method may include receiving multiple image or video portions as input. Each image of each portion of the multiple image or video may show a surface that is illuminated from different directions and / or illuminated at different illumination angles relative to the plane of the surface when viewed in a planar view of the surface. Each image of each portion of the multiple image or video may, but is not limited to, show a surface that is illuminated from different directions that may be opposite and / or perpendicular to each other. Each image or video portion may show a surface that is illuminated from at least two, three or all of the left, right, top and / or bottom directions when viewed in a planar view of the surface, and / or include additional or other directions. Multiple image or video portions showing different portions of different images or videos that show surfaces that are illuminated from different directions and / or illuminated at different illumination angles relative to the plane of the surface when viewed in a planar view of the surface can be analyzed by analysis. This method may be configured to combine the results of the analysis of multiple image or video portions, for example, the output of a discrimination model and / or at least one locator and / or a classification model and / or at least one white box or classical technique. For example, combining may involve applying an "OR" function, such as a bitwise OR function, to the analysis results of multiple images, but other forms of combining may be used. For example, an alternative form of combining may involve summing the intensities of locator and / or classification model outputs from images illuminated in multiple directions to arrive at a probability heatmap showing where anomalies are located.

[0083] The method may include dynamically determining an optimal or preferred configuration. The optimal or preferred configuration may include at least one of the following: the optimal or preferred direction and / or angle of surface illumination, and / or the optimal direction, position and / or angle of the image acquisition device, and / or the optimal color or wavelength of illumination. Determining the optimal or preferred configuration may include collecting multiple images of the surface illuminated with different directions and / or illumination angles, and / or different directions, positions and / or angles of the image acquisition device, and / or different colors or wavelengths, and determining the quality value of each image. The method may include controlling the illumination system to dynamically illuminate the surface from the direction and / or angle of surface illumination and / or from different directions, positions and / or angles of the image acquisition device that are determined to be optimal or preferred. The method may include dynamically repositioning and / or reorienting one or more light sources and / or controlling the illumination system to dynamically reposition, reorient, or use one or more image acquisition devices in the determined optimal or preferred arrangement to dynamically illuminate the surface from a direction and / or angle of illumination of the surface. The method may also include dynamically determining the optimal or preferred direction and / or angle of illumination for each of a plurality of surfaces.

[0084] The method may include processing portions of multiple images or videos of a surface, where different portions of the image or video represent the surface illuminated in different directions and / or at different illumination angles relative to the surface plane. The method may also include processing portions of multiple images or videos to determine the surface normals of a surface from the portions of the multiple images or videos, for example, from portions of multiple images or videos that show the surface illuminated in different directions and / or at different illumination angles relative to the surface plane. The method may also include determining the surface normals of a surface using photometric stereo techniques (e.g., https: / / en.wikipedia.org / wiki / Photometric_stereo). Photometric stereo techniques may be or may include black-box or white-box processes. The method may also include forming a map of surface normals representing at least a portion of the surface and / or using the determined surface normals to form a 3D model or mesh of at least a portion of the surface. The method may also include detecting and / or characterizing anomalies from the map of surface normals and / or the 3D model or mesh. This method includes generating a visualization of a 3D model or mesh, which can highlight any anomalies. The method may also include identifying one or more anomalies and / or regions from the 3D model or mesh and / or surface normal map that are likely to develop in the future. In response to the identification of anomalies, the method may include generating an alarm, warning, or flag.

[0085] A fifth embodiment of this disclosure is an analytical system configured to carry out the method of the fourth embodiment.

[0086] A sixth embodiment of this disclosure is a computer program product configured, when implemented in a computer-implemented analysis system, to cause the analysis system to implement the method of the fourth embodiment.

[0087] In relation to any particular embodiment of the present invention, the individual features and / or combinations of features defined above or below in accordance with any embodiment of the present disclosure may be used separately or individually, alone or in combination with any other defined features in any other aspect or embodiment of the present invention.

[0088] Furthermore, the present invention is intended to cover apparatus configured to implement any of the apparatus features described herein, in relation to methods and / or methods for using or manufacturing any of the apparatus features described herein.

[0089] Any method described above or below in this specification can be performed by one or more programmable processors that execute a computer program that performs the functions of the present invention by manipulating input data and generating outputs. Method steps can also be performed by dedicated logic circuits, such as FPGAs (Field Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits) or other customized circuits. Processors suitable for executing computer programs include CPUs and microprocessors, as well as any one or more processors. Generally, a processor receives instructions and data from read-only memory or random-access memory or both. Essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer includes one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or is operably coupled to receive data from them, transfer data from them, or both. Information media suitable for realizing computer program instructions and data include all forms of non-volatile memory, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory may be supplemented or incorporated by dedicated logic circuits.

[0090] To provide user interaction, this method may be implemented on a device having a screen for displaying information to the user, such as a CRT (cathode ray tube), plasma, LED (light-emitting diode), or LCD (liquid crystal display) monitor, and an input device that allows the user to provide input to the computer, such as a keyboard, touchscreen, mouse, or trackball. Other types of devices may be used, for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and user input may be received in any form, including acoustic input, voice input, or tactile input.

[0091] Any method described herein can be implemented on a device such as a mobile or network-enabled device that comprises or is configured to implement a controller or processing system of the first embodiment. The device may be, for example, a mobile phone, smartphone, PDA, tablet computer, laptop computer, or included therein. The controller or processing system can be implemented by a suitable program or application (app) that runs on the device. The device may include at least one processor, such as a central processing unit (CPU), mathematical coprocessor (MCP), graphics processing unit (GPU), tensor processing unit (TPU), and / or similar. The at least one processor may be a single-core or multi-core processor. The device may include memory and / or other data storage, which can be implemented on DRAM (dynamic random access memory), SSD (solid state drive), HDD (hard disk drive), or other suitable magnetic, optical, and / or electronic memory devices. The at least one processor and / or memory and / or data storage can be located locally and provided, for example, to a single device or multiple devices communicating in a single location, or distributed to several local and / or remote devices. The device may include communication modules, such as wireless and / or wired communication modules. The communication modules may be configured to communicate via cellular communication networks, Wi-Fi, Bluetooth®, ZigBee®, near-field communication (NFC), IR, satellite communications, other internet-enabled networks and / or similar. The communication modules may be configured to communicate via Ethernet® or other wired networks or connections, via telecommunications networks such as POTS, PSTN, DSL, ADSL, optical carrier lines and / or ISDN links or networks, via the cloud and / or the internet, or via other suitable data transmission networks.The communication module may be configured to communicate via optical communication such as optical wireless communication (OWC), free-space optical communication, or Li-Fi, or via optical fiber and / or similar. The device and / or controller, or at least one processor or processing unit, may be configured to communicate with a remote server or data store via the communication module. The controller or processing unit may include, or be implemented using, at least one processor, memory and / or other data storage, and / or the device's communication module. [Brief explanation of the drawing]

[0092] Refer to the accompanying drawings for a better understanding of this disclosure and to show how embodiments may be carried out. [Figure 1] A schematic diagram of the anomaly detection system is shown. [Figure 2] Figure 1 shows a plan view (planar perspective) of the anomaly detection system. [Figure 3] Figure 1 shows a perspective view of the anomaly detection system. [Figure 4] Figure 1 shows a schematic diagram of a part of the anomaly detection system. [Figure 5] Figure 1 is a schematic diagram of the image acquisition device for the anomaly detection system. [Figure 6] Figure 5 is a schematic diagram showing the field of view of the image acquisition device. [Figure 7] Examples of surfaces containing anomalies illuminated from different directions and angles are shown. [Figure 8] This is a contour map showing the optimal illumination direction and angle for an example surface. [Figure 9] This shows the histogram of the optimal quality image scores for diffuse and optimally directional illumination conditions. [Figure 10] Figure 1 shows a flowchart of the analysis method for analyzing images collected using the system. [Figure 11] The effect of applying white-box analysis to images collected using the system shown in Figure 1 is illustrated. [Figure 12]Figure 10 shows an example of the output at multiple stages of the method. [Figure 13] This chart shows the accuracy of crack detection for different lighting directions. [Figure 14] This chart shows the accuracy of crack detection for different illumination angles t relative to the surface. [Figure 15] This is a chart showing the overall accuracy of crack detection.

[0093] [Figure 16] This shows confusion values ​​for different methods of combining results obtained from different images of a surface illuminated from different directions and angles. [Figure 17] An alternative anomaly detection system is shown, along with an internal diagram. [Figure 18] An alternative anomaly detection system is shown, along with an internal diagram. [Figure 19] The alternative anomaly detection system is shown, along with the reverse / back view. [Modes for carrying out the invention]

[0094] In the following detailed description, specific embodiments of the subject matter of the present invention will be illustrated with reference to the accompanying drawings, which constitute part of this specification. These embodiments are described in sufficient detail so that those skilled in the art can implement them, and it should be understood that other embodiments can be utilized and structural, logical, and electrical modifications can be made without departing from the scope of the subject matter of the present invention. Such embodiments of the subject matter of the present invention are not intended, merely for convenience, to spontaneously limit the scope of this application to any single invention or inventive concept, and where multiple inventions are actually disclosed, they can be referred to individually and / or collectively by the term “invention.”

[0095] Therefore, the following description should not be interpreted as restrictive, and the scope of the subject matter of the present invention is defined by the appended claims and their equivalents.

[0096] In the following embodiments, similar components are labeled with the same reference number.

[0097] In the following embodiments, the terms data storage device or memory are intended to encompass any computer-readable storage medium and / or device (or collection of data storage media and / or devices). Examples of data storage devices include, but are not limited to, optical discs (e.g., CD-ROMs, DVD-ROMs, etc.), magnetic discs (e.g., hard disks, floppy disks, etc.), memory circuits (e.g., EEPROMs, solid-state drives, random-access memory (RAM), etc.), and / or similar devices.

[0098] As used herein, unless otherwise required by context, “equip,” “include,” “have,” and their grammatical variations are not intended to be exhaustive. They are intended to allow for the possibility of further additions, components, requirements, or steps.

[0099] The functions or algorithms described herein are implemented in one or more embodiments in hardware, software, or a combination of software and hardware. The software includes computer-executable instructions stored in a computer-readable carrier medium such as memory or other types of storage devices. Furthermore, the functions described may correspond to modules, which may be software, hardware, firmware, or any combination thereof. Multiple functions may be performed in one or more modules as needed, and the embodiments described are merely examples. The software may be performed in a digital signal processor, ASIC, FPGA, microprocessor, microcontroller, dedicated electrical circuit, or other types of processing device, or a combination thereof.

[0100] Furthermore, this specification refers to a processing module. A processing module may include software and / or be implemented at least partially using software. A processing module may include hardware for processing and / or be implemented at least partially using software, and the hardware for processing may include one or more devices, and the system of devices may include multiple processing units which may be distributed or localized, which may include different processing units in different devices, or which may all be contained in a single device. A processing unit may include one or more processors which may be multicore processors or single-core processors, ASICs, one or more hardware-programmable devices (e.g., FPGAs), electrical, electronic, or other logic circuits, and / or any combination thereof.

[0101] Specific embodiments will be described with reference to the drawings.

[0102] Figures 1 to 4 show an anomaly detection system 5, which includes an illumination system 10 and an image acquisition device 15 in the form of a digital camera. Both the illumination system 10 and the image acquisition device are mounted on a rigid support 20. The illumination system 10 is configured to illuminate a surface 25, such as a concrete surface, from multiple angles and / or directions relative to the plane of the surface 25. The image acquisition device 15 is configured to capture images of the surface 25 (which may be included in different parts of a video). Different images among the multiple images show the surface illuminated from different directions and / or different angles relative to the plane of the surface 25.

[0103] The lighting system 10 includes multiple light sources, each light source being attached to the end of a corresponding robotic arm extending from the support 20.

[0104] In this example, the support 20 is a rigid support in the form of a right-angled steel plate and can be attached to a control system for operating the support 20. In this example, the control system includes a robotic arm, such as a six-axis robotic arm (not shown). However, other forms of rigid support and / or control systems can also be used (or no control system can be used).

[0105] In this example, each light source 30a-30d is preferably in the form of an elongated strip of LEDs, but the disclosure is not limited thereto.

[0106] Each robotic arm 35a-35d to which each light source 30a-30d is attached is a three-joint servo motor driven arm, but again, the disclosure is not limited thereto, and other means can be used to operate the light sources to a desired directional and angular relationship with the surface 25. The position, direction and angle of each light source 30a-30d are individually and / or independently adjustable. That is, in the illustrated example, each robotic arm 35a-35d is individually and independently adjustable to operate the associated light source 30a-30d to the desired angle and position with respect to the surface 25.

[0107] Specifically, each robot arm 35a-35d comprises a plurality of elongated rigid parts 40a, 40b. One end of the first rigid part 40a is connected to the support 20 via a first articulated joint 45a. The other end of the first rigid part 40a is connected to the end of the second rigid part 40b via a second articulated joint 45b. Each light source 30a-30d is attached to the other end of the second rigid part 40b via a third articulated joint 45c. The first, second and / or third articulated joints 45a, 45b, 45c are rotatable on one, two, or three axes under the action of a suitable controllable force mechanism such as a motor (e.g., a servo motor), pneumatic actuator, or hydraulic actuator. For example, the third articulated joint 45c is rotatable at the angle at which light from the light sources 30a-30d is incident on the surface 25 (e.g., the angle Θ of the optical axis of the light sources 30a-30d with respect to the plane of the surface 35 shown in Figure 4). LEach light source 30a-30d may be configured to rotate to change the direction in which the surface 25 is illuminated when viewed in a plan view of the surface. The direction in which the surface 25 is illuminated can be changed by arranging the light sources 30a-30d so that they face different directions and selectively activating different light sources 30a-30d. For example, the light sources can be arranged so that each light source 30a-30d is perpendicular to at least one other light source 30a-30d, and then activated sequentially to collect an image when each light source 30a-30d is illuminating the surface 25.

[0108] The motion provided by each robotic arm 25a-25d is specifically shown in Figure 4. The three joints 45a-45c include the shoulder 45a, elbow 45b, and wrist 45c, which can be adjusted using servo motors or the like to change the orientation of the light source units 30a-30d and the angle of incidence Θ of light on the surface 25. L The proximity P can also be changed. By considering arms 25a-25d as vectors, the horizontal component is as follows:

[0109] A + L1cosΘ1 = L2cosΘ2 + PcosΘ L (1) The vertical component is as follows:

[0110] D=L1sin Θ1+L2sin Θ2+Psin Θ3(2) Here, L1 and L2 are the lengths of the arms, A is the distance from the center of the camera lens of the image acquisition device 15 to the shoulder joint 45a, and D is the working distance of the camera of the image acquisition device 15 or the distance from the shoulder joint 45a to the surface 25. Θ1 and Θ2 are the angles of the rigid parts 40a and 40b with respect to the horizontal, and Θ3 is the angle of the light source 30a-30d with respect to the horizontal from the optical axis. L is the angle of incidence of light from light sources 30a-30d on the surface 25 (for example, the angle of the optical axis of the light source 25 with respect to the plane of the surface 25). In a particular example, the lengths of the rigid parts 40a and 40b range from 100 mm to 500 mm, and the angles of the shoulder joint 45a and the elbow joint 45b are -90, respectively. o0 ≤ Θ1 ≤ 90 o and 0 o ≤ Θ2 ≤ 180 o is restricted to operate within the range. In this way, for a set of inputs of the desired illumination angle Θ L , proximity P, and working distance D, for example, by using an appropriate solution such as the least squares method or other minimization techniques, solutions for the joint angles Θ1, Θ2, and Θ3 can be determined from, for example, the above equations (1) and (2).

[0111] The support 20 can be moved, for example, translated, by a steering system in order to manipulate the illumination system 10 in the necessary relationship with the surface 25. The robotic arms 35a - 35d can be operated to finely adjust the light sources 30a - 30d to positions where the surface 25 can be illuminated from different directions and / or angles required with respect to the plane of the surface 25. Then, the light sources are sequentially activated and deactivated (or optionally combined if diffused light is required) so that the image collection device 15 can collect a plurality of images, and each image shows that the surface 25 is illuminated from different directions (for example, left, right, up, down when viewed in the plan view of the surface 25) and / or different illumination angles Θ L (for example, angles in the range of Θ L = 10° to 50°).

[0112] The steering system, servo motors, or each joint 45a - 45c of the robotic arms 35a - 35d and other suitable controllable actuating mechanisms for operating the light sources 35a - d can be controlled by the controller 50. The controller 50 includes a processor 55, a data storage device 60, and a communication module 65 for communicating control commands to the steering system, servo motors, and light sources 35a - d to operably control the illumination system 15. The controller 50 can operate the illumination system 15 in response to manual input via an appropriate user interface 70 or according to an automatic control that may include a plurality of rules, instructions, or other operating parameters, a machine learning model, etc., or a mixture of both manual and automatic control.

[0113] A specific apparatus for positioning light sources 30a-30d relative to surface 25 is shown with reference to Figure 1-4 and described above, but it will be understood that the possibilities are not limited to this and other arrangements are also possible. For example, a surface 25 such as a concrete surface may be illuminated in different directions and / or at different illumination angles Θ with respect to the plane of surface 25. L Other devices may also be used to position the light sources 30a-30d to selectively illuminate from a particular area. For example, although four light sources 30a-30d are shown, it is also possible to use other numbers of light sources 30, e.g., one, two, three, or four or more light sources 30. Robot arms 35a-35d are provided for manipulating the light sources 30a-30d to a desired position and / or angle relative to the surface 25, but different arrangements for positioning and orienting the light sources 30a-30d are also possible. For example, robot arms 35a-35d may not be provided, and one or more light sources can be attached to the support 20 via joints such as uniaxial, biaxial, or triaxial rotary joints. Furthermore, although robot arms 34a-35d having three joints 45a, 45b, 45c and two rigid parts 40a, 40b are shown and described, it is also possible to use other types of robot arms, for example, robot arms having more or fewer rigid parts 40a, 40b and / or more or fewer joints 45a-45c. In another example, a segmented arm can be used. Thus, although a particularly beneficial and flexible configuration of the lighting system 10 is shown in Figures 1 to 4, it will be understood that other configurations of the lighting system can also be used.

[0114] Furthermore, while the acquisition of images of the surface 25 illuminated at different angles and / or directions is described and shown, other distinguishable illumination configurations can be used in addition to directional and / or angular illumination, for example, different color / spectral components (e.g., acquiring different images of the surface while illuminated with light of different wavelengths), different types of illumination (e.g., illumination by two or more LED, fluorescent, halogen or laser light), different projection methods or patterns (e.g., spots, domes, lines and / or similar, which may be selected to suit the features), different polarizations, different neutral densities, other different filtering effects, diffuse and non-diffuse (hard) illumination, and / or similar. However, in certain cases of concrete and some other forms of the surface 25, imaging the surface 25 in different illumination directions and / or angles has been found to be particularly effective in enhancing the identification and characterization of anomalies such as cracks and other defects.

[0115] In examples where this has proven particularly effective, the different images of the multiple images may be images of a surface illuminated from at least two pairs of different opposing directions (e.g., up, down, left, and right in a planar view) under non-diffuse light, or at least one image of a surface taken under diffuse light, and some or all of the images may be grayscale images.

[0116] The image acquisition device 15 can operate to acquire images (or video that can be considered as a series of images). The image acquisition device 15 may include one or more digital cameras. Advantageously, the image acquisition device may be configured to have a minimum spatial resolution of 0.3 mm or less, for example, 0.1 mm or less. In this way, cracks and other defects of the size most likely to require further treatment can be adequately identified. The image acquisition device 15 may include any suitable camera, such as a forward-facing IR (FLIR) camera. Examples of suitable configurations of the image acquisition device 15 are shown in Figures 5 and 6. The image acquisition device 15 includes an imaging sensor 75, such as a CMOS or CCD pixel array, an aperture 80, and a focusing lens 85 for converting received light into electrical output. In a particular example, the imaging sensor 75 is a 5472 × 3648 pixel array FLIR sensor, and the lens is an 8 mm focal length lens, together providing a feature resolution of 0.1 mm or less at a working distance D of 350 mm and a field of view FoV of 574 mm × 383 mm. However, to achieve the required feature resolution, components with different properties and / or other configurations of the components may be used.

[0117] Depth of field (DoF) defines the distance at which an object remains in focus.

[0118]

number

[0119]

number

[0120] Equation (3) shows that a higher F-number results in a higher DoF, but this comes with trade-offs such as a decrease in image capture speed (due to reduced exposure of the image sensor). In particular, a high F-number may cause diffraction effects in the image, while a low F-number may cause blurring of the image edges.

[0121] F-number and distance F from the center of the lens to the depth of field (DoF) d This can be selected so that no identifiable diffraction effect occurs at an acceptable level of edge blurring. For example, in the particular example above, f=8 and F d At 250mm, diffraction effects and edge blurring are acceptable, and it is possible to clearly image objects at lens-to-object distances from 200mm to 350mm.

[0122] The camera / imaging sensor 75 can be configured to allow for acceptable imaging. To compensate for lens image distortion, calibration images, such as images of checkerboard patterns of known sizes at various distances and angles, can be captured. Distortion correction coefficients can be calculated from these images using commonly available techniques, such as scripts based on the OpenCV camera calibration module in Python.

[0123] White balance and exposure settings are automatically calculated and adjusted by the camera's onboard algorithm. However, this is a slow process compared to the rapid changes in lighting conditions generated by the lighting system 10. Alternatively, having preset values ​​for exposure and white balance for each lighting condition can ensure that exposure changes are quick and consistent.

[0124] The inventors have identified that the exposure setting is particularly dependent on the illumination angle when the surface 25 is illuminated from a single direction. Illumination angle Θ L Required exposure setting E ΘL teeth,

[0125]

number

[0126] Experimental results demonstrating the efficiency improvements achievable through the above-described use of directional lighting are provided in Figures 7 to 9. To produce these results, loads were applied to reinforced concrete slabs to induce various cracks on the surface at different lengths, widths, and directions. The cracks ranged in width from 0.1 mm to 1 mm and in length from 10 mm to 500 mm. Images of the sample surface were captured at a working distance of D = 250 mm, resulting in an FoV of approximately 500 × 375 mm.

[0127] In total, 12 different 5472×3648 pixel images of surface 25, which is the concrete surface of the slab, were acquired. Each region was captured with unidirectional illumination from the directions O=L, R, U, D, A, where U, D, L, R, and A are illumination from above, below, left, right, and diffused (or omnidirectional) when viewed in a plan view of surface 25. In other words, the L, U, R, and D directions are four directions in which each direction is perpendicular to the preceding and succeeding directions. For each illumination direction, the incident light angle Θ L The angles ranged from 10° to 50° in 10° increments. An example of a slab area between various lighting conditions is shown in Figure 7, and the inset compares a single 224×224 block in a larger image. Qualitatively, it can be seen that this block shows enhanced shading at the lower angle of 10° compared to 50°.

[0128] Image quality achieved under various lighting conditions was evaluated using the BRISQUE method (Mittal, Moorthy, and Bovik (2012), No-reference image quality assessment in the spatial domain, IEEE Transactions on Image Processing, 21(12), 4695-4708). Its contents are incorporated here as if fully described, and it is a non-reference metric previously used to evaluate the quality of images of cracked concrete (Kim, Choi, Hu, Lee, and Serfa Juan (2021), Multivariate analysis of concrete images using thermography and edge detection, Sensor 21(21), its contents are also incorporated here as if fully described.

[0129] The BRISQUE score varies significantly depending on the position within the directional illumination image. Therefore, each image is divided into 224x224 pixel sub-images (blocks), and a separate BRISQUE score is calculated for each block. The incident illumination angle is initially Θ L =50 o It is fixed to each lighting direction B dir =B U、D、L、R、A A BRISQUE score is calculated for each block. Here, U, D, L, R, and A represent illumination from the top, bottom, left, and right directions, and diffuse (omnidirectional) illumination, respectively, when viewed from a planar perspective of the surface. In this approach, the quality value Q for each direction of directional illumination is used. dir This can be defined as follows:

[0130]

number

[0131] Figure 8 shows a typical color contour map of the optimal lighting direction O overlaid on an image of concrete cracks. This map shows that while diffuse lighting works well in most areas, directional lighting, particularly from the left or right, clearly provides benefits in some areas.

[0132] Once the optimal illumination direction O is found, the optimal incident illumination angle Θ is determined. L Q is the quality value. ΘL By defining it as follows, the direction can be found in a similar manner to the approach described above.

[0133]

number

[0134] Figure 9 shows the diffused B diff , and optimized, B(O,Θ L The image shows a histogram of the BRISQUE score for all blocks during illumination and associated kernel density estimation (KDE). This indicates that optimized illumination conditions provide an overall improvement in image quality.

[0135] Having confirmed that optimized directional illumination significantly improves the image of surface 25, thereby making it more suitable for identifying anomalies such as cracks and other defects, the inventors also seek to further improve anomaly detection by using an enhanced image analysis method to analyze the surface image to identify and characterize anomalies. The process for analyzing the image of surface 25 is shown in Figure 10.

[0136] The process in Figure 10 is a hybrid image analysis process that uses both machine learning or AI technology (so-called "black box" technology) and classical technology (so-called "white box" technology). This hybrid process, which combines both black box and white box technologies, has been found to be particularly effective in processing images of surface 25 to identify anomalies such as cracks and other defects. The process may be executed by the controller 50 shown in Figures 1 and 2, or by external processing resources that may be local or remote to the controller 50, for example, in a server, cloud computing resources, central monitoring equipment or other suitable system, or the method may be executed by a combination of two or more of the controller 50, local processing resources and / or remote processing resources, for example, in a distributed computing configuration.

[0137] In step 1005, one or more images 1010 of the surface 25 are received. Advantageously, the received image 1010 may include at least one image 1010 from a plurality of images, where different images of the plurality of images show the surface while illuminated in different directions and / or at different angles relative to the plane of the surface 25 obtained from the anomaly detection system 5 shown in relation to Figures 1 to 7 and described above. In one possible example, the method uses, for example, the process described above to obtain the best quality value Q dir and / or Q ΘLAt least an image 1010 showing the surface 25 illuminated from a direction and / or angle relative to the surface is processed. In another example, multiple images showing the surface 25 illuminated from different directions and / or angles are processed, and the results from the different images are combined using, for example, "OR" or other appropriate join logic.

[0138] In step 1015, image 1010 of surface 25 is input to a discriminator model configured to detect and roughly locate features (cracks or other anomalies, etc.) against a background (i.e., plain concrete in this example). A particularly effective machine learning model that can be used for the discriminator model is a region-based convolutional neural network (R-CNN), which is a relatively fast neural network. The discriminator model is configured to take a complete image 1010 of surface 25, process it to identify and roughly locate portions of the image that represent anomalies such as cracks, and place bounding boxes 1020 around the identified regions as representing each identified anomaly 1025. Thus, the output from the discriminator model is image 1030, which includes the input image 1010 with bounding boxes 1020 placed around the anomalies 1025.

[0139] In a specific example, the discriminant model is a two-shot region-based CNN (R-CNN) that includes a region-proposed network (RPN) that detects features of interest relative to the background, and a CNN classifier network that subsequently classifies the features of interest as either anomalies or belonging to the background. A specific example of a suitable R-CNN is Tensorflow's Faster-R-CNN Inception v2 COCO Abadi et al. (2015), whose content is fully incorporated by reference and trained via transfer learning on manually annotated concrete crack images, which are identified, collected, and manually annotated images subject to directional and / or diffuse illumination.

[0140] The discrimination model is a relatively fast operating model, and for example, compared to the locator and / or classification models described later, it offers higher computational efficiency and faster output.

[0141] According to this method, in step 1035, the output 1030 of the identification model is divided into blocks of a set or pre-set size. In this example, the blocks are 224x224 pixels, but they may be any other suitable size. The result is an image 1040 formed from the image 1030 output by the identification model, but divided into multiple blocks.

[0142] In step 1045, image 1040 is effectively cropped by classifying the boxes within image 1040 into boxes within the bounding box 1020 and boxes outside the bounding box 1020. Specifically, any block within the bounding box 1020 with a duplicate of 1 is labeled "in" and passed on to the next step for further processing, while any block outside the bounding box is labeled "background" for all its pixels in step 1047 (and is also optionally retained). The labeled image obtained from this process is shown as 1050.

[0143] The boxes of image 1050 within bounding box 1020 are input to the locator and / or classification model in step 1055. The locator and / or classification model acts on each block within bounding box 1020 to more accurately locate and classify the blocks within bounding box 1020 that represent at least a portion of the anomaly. In particular, the locator and / or classification model is configured to identify blocks within bounding box 1020 that overlap with the anomaly.

[0144] In one example, particularly useful locators and / or classification models include convolutional neural networks such as VGG16-CNN. The locators and / or classification models are run on each block within the bounding box 1020 to determine whether the block contains any anomalies. If a block contains any anomalies, the block is passed on for pixel-level analysis, for example, by classical or "white-box" analysis. If it is determined that the block does not contain any anomalies (e.g., a negative detection result), in step 1047, all pixels within these boxes are labeled as "background" and set aside.

[0145] More specifically, the purpose of the locator and / or classification model is to filter blocks within a bounding box (e.g., 224x224 blocks) (determined by the discrimination model) into "positive" and "negative" categories. In this case, "positive" blocks are those with cracks detected within them, while "negative" blocks are those without cracks within the bounding box.

[0146] Discriminative models are highly computationally efficient, even more so than locators and / or classification models. Therefore, using a discriminative model before locators and / or classification models can reduce the computational load generated by the locators and / or classification models, thereby improving processing time by reducing the number of predictions required.

[0147] The VGG-16 model used in some of the examples herein is described in Simonyan & Zisserman (2014) Very deep convolutional networks for large-scale image recognition.arXiv, and its contents are incorporated herein by reference as if they were fully described herein. It uses 13 convolutional layers and 5 pooling layers input to three fully connected layers. In this example, the VGG-16 model is trained on a wide dataset obtained by training data showing directional illumination and / or diffuse illumination using transfer learning. The VGG-16 model has been found to be particularly effective for this process, but this disclosure is not limited thereto, and other suitable machine learning or other models may be used.

[0148] The output 1060 of the locator model and / or classification model is a box from image 1050 representing part of the anomaly. This is provided as input to the pixel-level analysis model in step 1065. In this example, the pixel-level analysis model uses classical or "white-box" techniques to refine the box representing part of the anomaly into pixels representing the anomaly, such as cracks. Classical or "white-box" techniques may include thresholding, edge detection, and / or similar.

[0149] More specifically, the input to the pixel-level model includes blocks (e.g., 224x224 blocks) previously labeled as "positive" for cracks by a locator and / or classification model (e.g., the VGG-16 algorithm). This hybrid approach, which performs classical or white-box analysis by reducing the number of blocks filtered by machine learning or AI models such as the R-CNN and VGG-16 models mentioned above, improves the accuracy of the final binary image output because it can completely remove background noise that would have been apparent in the "negative" blocks. It can also remove false positive pixels caused by features that simply resemble cracks but can be filtered by machine learning models (such as R-CNN and VGG-16 models) due to, for example, surface markings.

[0150] A suitable example of a classic or “white-box” technique that can be used for pixel-level analysis is the edge detector spatial region method used for crack detection, as described in Dorafshan, Thomas, & Maguire (2019) in Construction and Building Materials, 186, 1031-1045, which is incorporated herein by reference as if fully described herein. Other white-box techniques can be applied to other types of anomalies. Advantageously, the approach proposed in Dorafshan can be enhanced by (1) applying pre-processing contrast enhancement to each image, and (2) using a 3x3 Laplacian kernel as the edge detector (although different kernel sizes and edge detector types can be used instead of the above). Pre-processing enhancement may include, for example, image multiplication of diffuse and directional images, anisotropic diffusion to add a slight blur to areas in the image that are considered “non-edges,” and / or exposure correction (which is not currently done but may be beneficial to include).

[0151] Figure 11 illustrates a classical or "white-box" technique that does not use the previous filtering by a machine learning model (such as the R-CNN or VGG-16 model mentioned above). The image shown in Figure 11 is a full-surface image, but in this example, it is understood that the classical or white-box technique is applied only to blocks that are determined to contain anomaly 1060 outputs from the locator model and / or classification model applied in step 1055. The classical or white-box crack detection image processing technique is applied individually to each directional illumination image (right, left, up, down) 1070a, 1070b, 1070c, and 1070d, respectively, generating segmented images 1075a, 1075b, 1075c, and 1075d for each illumination direction, with pixels labeled "anomaly" or "background". Next, the images segmented for each distinct directional illumination direction are combined using other appropriate techniques, such as bitwise OR operations or summing and normalizing the images. For example, if a pixel appears as part of a crack in all images, it has a confidence value of 1. If a pixel appears as part of a crack in half of the image, it has a confidence value of 0.5, and so on.

[0152] The resulting combined image 1080 highlights the edges of the crack in all lighting directions. Denoising can be used to remove false positives. For example, in the case of the bitwise OR combination described above, this may involve removing groups of pixels that are not part of the anomaly. In the case of the image summation (confidence method), this may involve removing lower confidence pixels that are not part of a pixel group containing higher confidence pixels, although other techniques may also be used.

[0153] As described above, for input images filtered by a discriminative model and / or locator and / or classification model, this process is performed only on selected blocks of the image, not the entire image. This produces a much more accurate combined output image 1080 with most of the noise removed.

[0154] Figures 12 to 16 show experimental data demonstrating the effectiveness of the directional illumination method and system described herein. Figure 12(a) shows a sample area of ​​concrete surface 25 used as a test sample. Surface 25 includes two anomalies 1205 and 1210 in the shape of cracks and a surface line 1215 that resembles a crack. Figure 12(a) also shows bounding boxes 1020a, 1020b, and 1020c determined for each crack 1205, 1210 and surface line 1215 by an identification model (step 1015 in Figure 10). In this case, the identification model is an R-CNN with a confidence threshold set to 40%. In particular, surface 25 includes a surface marking 1220 in the upper left corner. The identification model successfully identified that these markings are not cracks and filtered them out. However, if classical or white-box methods had been applied directly to this image without prior filtering using a discrimination model, these markings would have resulted in false positives and been incorrectly identified as anomalies. Similarly, the cast marking 1225 in the lower left corner would likely have resulted in a false positive anomaly detection if the image had been fed directly as input to a classical / white-box method without prior filtering using a discrimination model. Therefore, the hybrid approach described herein can not only reduce the computational load and improve detection speed, but also reduce the false positive rate and improve accuracy.

[0155] Images from each illumination direction (i.e., up, down, left, and right in a plan view of surface 25) are supplied to the identification model, and the associated bounding boxes 1020a, 1020b, and 1020c and corresponding images 1030a, 1030b, and 1030c are output.

[0156] The output 1030 of the discrimination model (e.g., R-CNN) for each illumination direction is acquired separately. Subsequently, all blocks (and any overlaps) within the bounding boxes 1020a, 1020b, and 1020c are classified as "in" and sent to the locator and / or classification model (e.g., VGG-16 model) for classification. In this example, 32.5% of the blocks were determined as "in" by the locator and / or classification model (e.g., VGG-16 model), saving a considerable amount of processing time. In this example, a false positive bounding box 1020c was generated on the surface line 1215 created by casting resembling a crack. This is removed if the required confidence level of the discrimination model is set above 80%. However, since the discrimination model is the first step, it is preferable to have a recall rate higher than the precision to ensure that nothing is missed. For this reason, a low confidence bar is set.

[0157] The output of the discrimination model is input to the locator and / or classification model, where blocks within the bounding box that do not contain crack features are filtered out. The remaining blocks that the locator and / or classification model determine to contain some anomalies are output for classical / white-box processing. In this case, according to the prediction of the locator and / or classification model, the false positive bounding box 1020c does not contain any crack feature blocks. Figure 12(b) shows the output of the blocks filtered by the locator and / or classification model, and Figure 12(c) shows the final binary output with the application of classical / white-box techniques.

[0158] The area under the curve (AUC) of the precision-recall (PR) plot can be used to evaluate the accuracy of a discriminative model (in this case, a faster R-CNN). A higher AUC value corresponds to a model with higher accuracy. Figure 13 shows the results for three test samples with directional illumination in each direction. For each sample, it can be seen that one of the images showing the directionally illuminated surface 25 has the highest accuracy. However, the direction that produces the highest accuracy is sample-dependent and may depend, for example, on the anomaly in the problem and how the light illuminates the anomaly. (For example, Q mentioned above) dir If the image with the most appropriate or best illumination direction (using the values) is determined and used, it may be possible to achieve results that are significantly better than those achieved under diffuse conditions.

[0159] To show the effect of the illumination angle on the plane of surface 25, a corresponding analysis was performed, and the results shown in Figure 14 were obtained. From this, 20 o from 50 o More specifically, 30 o from 40 o It can be seen that the angle gave the best results. In the next step, the lighting angle will be 40 o Only images that met the following criteria were used to analyze the effectiveness of the locator and / or classification model and classical / white-box techniques.

[0160] Figure 15 shows the variation in accuracy of the locator and / or classification model (in this case, a VGG-16 neural network). Accuracy is evaluated against manually labeled ground truth. This indicates that the appropriate use of directional illumination (particularly by selecting the "best" direction or aggregating illumination from different directions) can achieve greater advantages than diffuse illumination (sample "A" in Figure 15).

[0161] Figure 16 shows the confusion matrix of predicted and true labels for (a) the prior art method Dorfshan et al., (2018) Comparison of deep convolutional neural networks and edge detectors for image-based crack detection in concrete, Construction and Building Materials, 186, 1031-1045, and (b) directional images concatenated by the above bitwise OR (logical disjunction) operation. These results show that the prediction of background pixels (the majority of pixels) is very accurate regardless of the method used. However, while the conventional Dorafshan method achieved an accuracy of 54% in predicting crack pixels, the bitwise OR concatenation of directional illumination images achieved an accuracy of 69%.

[0162] From the above, it can be seen that the use of directional illumination (direction and / or illumination angle when viewed in a plan view of surface 25) can provide a significant improvement over diffuse imaging. However, the optimal direction may vary depending on the sample. Therefore, in order to dynamically select the illumination direction for a given sample, Q dir It may be beneficial to determine the best direction for illuminating the form using quality values ​​such as Q. Similarly, while the angle does not seem to depend much on the sample, for example, quality value Q ΘL Based on this, there may be room to dynamically select the optimal angle to use.

[0163] The use of the robot arm-based system 5 shown in Figures 1 to 4 allows for easy manipulation of the light sources 30a-30d to the required position and orientation in order to provide a desired illumination direction and angle to a selected portion of the surface 25. However, variations of the system 5 in Figures 1 to 4 are also possible. For example, different numbers of robot arms 35a-d and light sources 30a-d can be used, such as one, two, three, five, or more robot arms 35a-d and light sources 30a-d. Furthermore, the robot arms 25a-d may have more or fewer joints 45a-c and / or rigid parts 40a-b than those shown in Figures 1 to 4. In fact, the light sources 30a-d do not need to be mounted on the robot arms, and other mechanisms can be used to manipulate or otherwise provide the light sources 30a-d to the required position and orientation in order to provide illumination of the surface 25 in the desired direction and angle.

[0164] For example, alternative configurations for imaging the surface of an object to be inspected while selectively illuminating the surface from different directions and / or orientations relative to the surface plane are shown in Figures 17 to 19.

[0165] In this example, no robotic arm is used (however, the entire device can be mounted on a robotic arm or other suitable manipulator if necessary). System 5' includes an illumination system 10' and an image acquisition device 15' in the form of a digital camera. Both the illumination system 10' and the image acquisition device 15' are mounted on a rigid support 20' in the form of a concave shroud in this example, but other examples may include different shapes or arrangements. The illumination system 10' is configured to illuminate a surface 25', such as a concrete surface, from multiple directions and / or angles relative to the plane of the surface 25. The image acquisition device 15' is configured to capture images of the surface 25' (which may be included in different parts of a video). Different images among the multiple images show the surface illuminated from different directions and / or different angles relative to the plane of the surface 25'.

[0166] In this example, the support 20' in the form of a concave shroud is polyhedral, that is, at least the inner surface of the support 20' comprises multiple facets 31', 32', 33', 34', and at least one of the different facets 31', 32', 33', 34' faces in a different direction from the other facets 31', 32', 33', 34'.

[0167] The lighting system 10' comprises multiple light sources 30', each different light source 30' mounted on facets 31', 32', 33', and 34' at different angles on the inner surface of the concave support 20'. Furthermore, the support 20' has a multifaceted shape, such as two, three, four, or more faces (in this example, approximately square in plan view). Different light sources 30' are also provided on different sides (and optionally different facets on different sides) of the inner surface of the support 20'. The inner surface of the concave support 20' is open at one end. In this way, the system 5' can be positioned to have an open end with respect to the surface 25' being tested, and different light sources 30' on different sides and / or different facets of the inner surface of the support can be selectively activated and deactivated to provide illumination at different times. The image capture device 15' can capture a portion of an image or video of the surface 25', and different portions of the image or video show the surface 25' illuminated by different light sources 30' on different sides and / or different facets 31', 32', 33', 34', thereby illuminating the surface 25' from different directions and / or angles relative to the plane of the surface 25'. In this way, the illuminated surface 25' can be illuminated from different directions and / or angles relative to the plane of the surface in a plan view by simply selectively starting and stopping the selected light sources 30' on different sides and / or facets 31', 32', 33', 34', without necessarily requiring a robotic arm (although it can be used to operate the entire system 5' if necessary). The inner surface of the shroud can be matte black or some other non-reflective finish to advantageously ensure directional illumination and avoid diffuse illumination. The concave shroud shape of the support 20' allows for better exclusion of diffuse light from the external environment, such as room lighting, sunlight, and / or similar sources, from the image.

[0168] It will be understood that variations of the specific example above are also possible. For example, while the above example can be used to generate a 2D combined output image, the above technique can be used in conjunction with an image acquisition device configured to collect images suitable for photometric stereo analysis and / or other 3D imaging techniques, from which 3D data representing surface normals and / or surfaces can be obtained.

[0169] In the example, any of the above anomaly detection systems may be configured to collect images for use in conjunction with photometric stereo image processing, for example, to determine normals to a surface at multiple locations. In this case, the multiple images of the surface can be collected from a fixed viewpoint, and each of the multiple images is captured under different lighting conditions. The number of images can be at least one, for example, 1 to 100 images, for example, 5 to 20 images, for example, 10 images. This number of images has been shown to provide a good trade-off between accuracy and processing efficiency.

[0170] Any of the anomaly detection systems described above may be configured to determine a 3D model or mesh of at least a portion of the surface, which can be used, for example, by photometric stereo, stereoscopic, or other 3D imaging using the determined normals. For example, the 3D surface data generated from the determined normals can be stitched, merged, or otherwise combined to form a 3D model of the mesh.

[0171] Surface 3D data can be stitched, merged, or otherwise combined to form a 3D model or mesh of a portion of the surface, which can then be output as a visualization.

[0172] Anomalies can be highlighted in a 3D model or mesh, and the 3D model or mesh can be manipulated to further investigate and / or characterize the anomalies, or to predict future anomalies (e.g., based on local elevation of the ground surface or local changes in ground surface height over time). The images can be color or grayscale. In another example, a surface normal map can be created, which can optionally be used to visualize the surface and / or characterize any anomalies. The 3D model or mesh may be partially created or supplemented using other 3D imaging techniques such as stereophotogrammetry, infrared imaging (such as that used in Microsoft's Kinect system), LIDAR, RADAR, magnetic or electric field sensors, or laser scanners.

[0173] While various arrangements of different illumination directions and / or different illumination angles, and optionally other distinguishable illumination arrangements are described above, in examples where this has proven particularly effective, the different images of multiple images may be images of a surface illuminated from at least two pairs of different opposing directions (e.g., up, down, left, and right in a plan view) under non-diffuse light, or at least one image of a surface captured under diffuse light, and some or all of the images may optionally be grayscale images. Different channels, for example, each channel may contain images of the surface under different illumination directions or angles and / or different types of illumination (e.g., diffuse or non-diffuse light), or images or data obtained therefrom, and can be used as input to a pixel segmentation process, for example, to segment pixels representing anomalies from pixels representing the rest of the surface.

[0174] In the example, the surface may be illuminated with a selected color and / or a selected wavelength or wavelength band, which may be selected according to the material of the surface or the type of material or anomaly on the surface. For example, the color or wavelength or wavelength range of light used for illumination may be selected to emphasize or highlight anomalies or materials on the surface, or alternatively to reduce or mitigate them. As an example, the image may be selectively illuminated with blue light, which may be particularly suitable for imaging red corrosion, as red corrosion may be emphasized in the image. As another example, illumination with green light may be used to reduce, mitigate or ignore the effects of plants or other organic growth such as moss or algae on the surface.

[0175] Therefore, the drawings provided herein are for illustrative purposes only, and the scope of protection is defined by the claims.

[0176] The method steps of the present invention can be performed by one or more programmable processors that execute a computer program that performs the functions of the present invention by manipulating input data and generating outputs. The method steps can also be performed by dedicated logic circuits, such as FPGAs (Field Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits) or other customized circuits. Processors suitable for executing computer programs include CPUs and microprocessors, as well as any one or more processors. Generally, a processor receives instructions and data from read-only memory or random-access memory or both. Essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer includes one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or is operably coupled to receive data from them, transfer data from them, or both. Information media suitable for realizing computer program instructions and data include all forms of non-volatile memory, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory may be supplemented or incorporated by dedicated logic circuits.

[0177] To provide interaction with the user, the present invention may be implemented on a device having a screen for displaying information to the user, such as a CRT (cathode ray tube), plasma, LED (light-emitting diode), or LCD (liquid crystal display) monitor, and an input device to which the user can provide input to the computer, such as a keyboard, touchscreen, mouse, or trackball. Other types of devices may be used, for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and the input from the user may be received in any form, including acoustic input, voice input, or tactile input.

Claims

1. Lighting system and, At least one image acquisition device and Equipped with, The illumination system comprises at least one light source configured to illuminate a surface in a directional manner, and the illumination system is configured to selectively illuminate the surface from one or more selected directions when viewed in plan view of the surface, and / or at one or more selected illumination angles with respect to the plane of the surface. An anomaly detection system comprising at least one image acquisition device configured to image the surface while the surface is illuminated from one or more selected directions and / or at one or more selected illumination angles relative to the plane of the surface.

2. With an additional support structure, An anomaly detection system according to claim 1, wherein at least one or each of the light sources is attached to the support via at least one manipulator for manipulating the light sources to adjust the position and / or orientation of each light source.

3. The anomaly detection system according to claim 2, wherein the at least one manipulator comprises a robotic arm.

4. At least one arm is provided with at least one joint, The abnormality detection system according to claim 3, wherein the arm is movable at each joint by the action of a drive mechanism.

5. The anomaly detection system according to any one of claims 2 to 4, wherein the at least one manipulator is controllable by a controller to selectively change the illumination angle and / or direction of light incident on the surface from each light source.

6. The lighting system comprises multiple light sources that can be configured to face in different directions, An anomaly detection system according to any one of claims 1 to 5, wherein each light source can be independently or individually selected to provide illumination and can be deactivated to turn off illumination from the light sources.

7. The at least one image acquisition device is controllable to capture a plurality of images or videos of the surface, The anomaly detection system according to any one of claims 1 to 6, wherein the illumination system is configured to selectively illuminate the surface from different sides of the surface as viewed in plan view, and / or at different illumination angles with respect to the plane of the surface, while different portions of images or videos are being collected.

8. The one or more image acquisition devices are configured to image the surface from at least one of different angles, different positions, and / or different directions with respect to the plane of the surface. The one or more image acquisition devices are configured to acquire a plurality of images or videos of the surface while the surface is illuminated by the illumination system. An anomaly detection system according to any one of claims 1 to 7, wherein at least one of the aforementioned images can be collected by the one or more image acquisition devices from at least one of different angles, different positions and / or different directions with respect to the plane of the surface, with respect to those used to collect at least one other image.

9. The anomaly detection system according to any one of claims 1 to 8, wherein the lighting system is configured to selectively illuminate the surface with one or more of the following: different colors, different spectral components or wavelengths of light, different types of illumination, different projection methods or patterns, different polarizations, different neutral densities, and / or different filtering effects.

10. The anomaly detection system according to any one of claims 1 to 9, wherein the illumination system is configured to dynamically change the direction in which the surface is illuminated based on the quality value of the resulting image in order to optimize the quality value of the collected image.

11. An anomaly detection system according to any one of claims 1 to 10, wherein the light source is positioned to illuminate the surface at an angle in the range of 5° to 50° with respect to the plane of the surface, or the illumination system is configured to position the light source.

12. The anomaly detection system according to any one of claims 1 to 11, further comprising an analysis system for identifying and / or characterizing an anomaly on the surface from at least one image of the surface collected by the image acquisition device, or configured to communicate with the analysis system.

13. The anomaly detection system according to claim 12, wherein the analysis system is configured to distinguish the portion of the image or video representing an anomaly from the portion of the image or video representing the background.

14. The analysis system is configured to include and / or implement one or more models. An anomaly detection system according to claim 13, wherein the one or more models include an identification model configured to identify a portion of the image or video representing the anomaly and to assign one or more bounding boxes around the identified portion of the image or video representing the anomaly, and / or one or both of a locator and / or classification model configured to locate and / or classify at least one block of the image or at least a portion of the video within the bounding box representing at least a portion of the anomaly.

15. The anomaly detection system according to claim 14, wherein the identification model comprises a region-based convolutional neural network (R-CNN), and the locator and / or classification model comprises a VGG-16 neural network.

16. The anomaly detection system according to claim 14 or 15, wherein the analysis system is configured to implement one or more whitebox or classical image processing methods configured to further refine the block identified as containing the portion of at least one of the images or at least a portion of the video that represents the anomaly.

17. The analysis system is configured to combine the outputs of a discrimination model and / or at least one locator and / or a classification model and / or at least one white-box or classical technique obtained from a portion of multiple images or videos. An anomaly detection system according to any one of claims 12 to 16, wherein a portion of the plurality of images or videos that is different from a portion of the images or videos shows the surface illuminated from different directions when viewed in plan view of the surface and / or the surface illuminated at different illumination angles with respect to the plane of the surface.

18. Anomaly detection system according to any one of claims 12 to 17, wherein the analysis system is configured to dynamically determine an optimal or preferred configuration, the optimal or preferred configuration comprising one or more arbitrary or preferred directions and / or angles of illumination of the surface, the optimal direction, position and / or angle of the image acquisition device, and / or the optimal or preferred color of the wavelength of illumination, the analysis system is configured to determine the optimal configuration by collecting a plurality of images of the surface which are one or more of images illuminated from different directions and / or angles of illumination, images collected from different positions and / or angles of the image acquisition device, and / or images illuminated with different colors of the wavelength of illumination, and to control the illumination system to the configuration determined to be optimal or preferred.

19. Equipped with a concave shroud, An anomaly detection system according to any one of claims 1 to 18, wherein at least one image acquisition device and one or more light sources of the illumination system are mounted inside the shroud.

20. An anomaly detection system according to any one of claims 1 to 19, configured to determine at least partially a 3D model or mesh of a surface from images collected by an image acquisition system and / or surface normals derived therefrom.

21. A method for identifying and / or characterizing surface abnormalities, The steps include providing a lighting system and at least one image acquisition device, The lighting system comprises at least one light source configured to illuminate a surface in a directional manner, and the method is The steps of using the lighting system to selectively illuminate the surface from at least one selected direction when viewed in plan view of the surface, and / or at one or more selected lighting angles with respect to the plane of the surface, and A method further comprising the step of imaging the surface from at least one direction using at least one image acquisition device while the surface is illuminated from at least one selected direction and / or at one or more selected illumination angles relative to the plane of the surface.

22. A computer program product, when implemented in a controller of an anomaly detection system according to any one of claims 1 to 20, is configured to cause the controller to control the illumination system to selectively illuminate the surface from at least one selected direction when viewed in plan view of the surface, and / or at one or more selected illumination angles with respect to the plane of the surface, wherein the at least one image acquisition device is configured to image the surface from at least one direction while the surface is illuminated from the at least one selected direction and / or at one or more selected illumination angles with respect to the plane of the surface.

23. A method for determining anomalies on a surface by analyzing an image of the surface, the method comprising applying a hybrid analysis to the image of the surface, which includes applying one or more machine learning or black-box models to the image of the surface to determine blocks of the image that contain at least a portion of the anomaly, and applying one or more classical or white-box techniques to the blocks of the image that have been determined to contain at least a portion of the anomaly, thereby distinguishing between portions of the block that do not represent the anomaly and portions of the block that represent the anomaly.

24. A computer implementation analysis system configured to implement the method described in claim 23.

25. A computer program product configured, when implemented in a computer-implemented analysis system, to cause the analysis system to implement the method described in claim 23.

Images