Interactive Measurement Based on Object's 3D Representation

Abstract

The present invention discloses a method, system, and computer program product for rendering a visual representation of a target object using a selection of a first position on a surface of the target object. A reference surface is determined based on the selected position on the surface of the target object. A graphical object covering the visual representation of the target object is rendered. The position of the graphical object in the graphical user interface is based on the reference surface. The graphical object has a perimeter including perimeter positions. The perimeter positions among the perimeter positions are mapped to a second point on a three-dimensional representation of the target object. The distance between a first point and the second point on the three-dimensional representation of the target object is less than or equal to a second distance. The first point on the three-dimensional representation is associated with the first position.

Description

[0001] Related Applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 341,696, filed on May 13, 2022, under 35 U.S.C. § 119(e), the entire content of which is hereby incorporated by reference in its entirety. Technical Field

[0003] This application relates to systems and methods for interactive measurement based on three-dimensional representations of objects. Background Art

[0004] Three-dimensional objects (e.g., surfaces) can be imaged and presented in a graphical user interface (GUI). For example, a two-dimensional image (e.g., a picture) of an object can be presented in the GUI. Alternatively, a three-dimensional image (e.g., a point cloud view) of an object can be presented in the GUI. A point cloud view can be generated by detecting a plurality of positions on the outer surface of the object using 3D imaging techniques such as structured light projection, stereoscopy, photogrammetry, time-of-flight, etc. In some embodiments, a plurality of images of an external object can be combined to generate a portion of the three-dimensional image. The point cloud can be used to generate a 3D computer-aided design (CAD) model of a manufacturing part for measurement in an animation or other application. Summary of the Invention

[0005] Generally, systems and methods are provided for interactive measurement of three-dimensional representations of objects detected by a monitoring device. Various aspects of the disclosed subject matter can provide one or more of the following capabilities.

[0006] A method includes rendering a visual representation of a target object in a graphical user interface display space. The method may further include receiving a first user input indicating a selection of a first position on the surface of the target object based on a first user interaction with the visual representation. The method may further include determining a reference surface based on selections of a plurality of positions on the surface of the target object. The plurality of positions are within a first distance from the first position on the surface of the target object. The method may further include rendering a graphical object in the graphical user interface that overlays the visual representation of the target object. The position of the graphical object in the graphical user interface is based on the determined reference surface. The graphical object has a perimeter that includes a plurality of perimeter positions. The perimeter positions among the plurality of perimeter positions are mapped to a second point in a three-dimensional coordinate system of the three-dimensional representation of the target object. The distance between a first point and the second point in the three-dimensional coordinate system is less than or equal to a second distance. The first point in the three-dimensional coordinate system is associated with the first position.

[0007] A non-transitory computer program product (i.e., a physically embodied computer program product) storing instructions is also described, which causes at least one data processor to perform the operations herein when the instructions are executed by one or more data processors of one or more computing systems. Similarly, a computer system is also described, which may include one or more data processors and a memory coupled to the one or more data processors. The memory may temporarily or permanently store instructions that cause at least one processor to perform one or more of the operations described herein. Additionally, the method may be implemented by one or more data processors within a single computing system or by one or more data processors distributed between two or more computing systems. Such computing systems may be connected via one or more connections, including connections over a network (e.g., the Internet, a wireless wide area network, a local area network, a wide area network, a wired network, etc.), via direct connections between one or more of the multiple computing systems, etc., and may exchange data and / or commands or other instructions, etc.

[0008] These and other capabilities disclosed will be more fully understood after reviewing the following drawings, detailed description, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] These and other features will be more readily understood from the following detailed description in conjunction with the accompanying drawings, in which:

[0010] Figure 1 An example of a system according to an embodiment presented herein is shown;

[0011] Figure 2 is a flowchart of an exemplary process according to an embodiment presented herein;

[0012] FIG. 3 to Figure 12 shows an example graphical user interface (GUI) including a visual representation of an imaging target according to an embodiment presented herein;

[0013] Figure 13 shows an exemplary GUI that allows user input indicating the radius of a graphical object according to an embodiment presented herein;

[0014] Figure 14 shows an exemplary GUI that allows user input indicating a multiplier value that can be used to determine the radius of a graphical object according to an embodiment presented herein;

[0015] Figure 15 shows an exemplary imaging device according to an embodiment presented herein; and

[0016] Figure 16 depicts a block diagram showing an example of a computing system according to some exemplary embodiments.

[0017] In practice, like reference numerals represent like structures, features, or elements. Detailed Description

[0018] This application relates to the interactive measurement of a three-dimensional representation of a target object. The target object (e.g., an industrial machine) can be imaged by a monitoring device, and the imaging data can be presented to a user for inspection in a graphical user interface (GUI) of a detection system. For example, a user can inspect the industrial machine and identify any damage or the possibility of damage during the operation of the industrial machine. The imaging data can be presented as a two-dimensional (2D) image and / or a three-dimensional (3D) image of the target object. The images can be displayed in an interactive setting. For example, the interactive setting can enable the generation of graphical objects that overlay the 2D / 3D images of the target object. The graphical objects can allow for improved inspection of the industrial machine. For example, the interactive setting can enable the comparison of the features of the industrial machine with the features of the graphical objects when inspecting the target object, the industrial machine.

[0019] In some specific implementations, the imaging data can be used to construct a three-dimensional (3D) representation of the target object. The 3D representation can allow mapping of positions (or pixels) in the 2D image to points (e.g., x, y, z coordinates) in the 3D image. Graphical objects generated in the 2D image (e.g., defined by multiple positions (or pixels) in the 2D image) can be mapped to multiple points or its surface in the 3D representation of the target object (e.g., each position in the 2D image can be mapped to a point in the 3D representation of the surface of the target object). In some specific implementations, a circle in the 3D representation can be mapped to an elliptical region in the 2D image (e.g., based on the perspective of the camera used to capture the 2D image). In some specific implementations, the interactive setting can generate recommendations for actions based on the mapped characterization of the target object, including defect detection. The action can be automatically triggered to be executed by the monitoring device or the repair device to remedy the detected defect of the target object.

[0020] As an introduction, Figure 1 FIG. shows an example of a system 100 according to some exemplary embodiments. Figure 1 The illustrated system 100 includes a server system 102, an industrial plant 103, a client device 104, and a network 106. The industrial plant 103 can include one or more imaging targets 108A, 108B, imaging devices 110A, sensors 110B, and a repair device 112. The imaging targets 108A, 108B can include industrial machines that can be operatively coupled to each other.

[0021] The server system 102 can be communicatively coupled via the network 106 to components of the industrial plant 103 (such as imaging targets 108A, 108B, imaging devices 110A, sensors 110B, and repair devices 112) and to a plurality of clients including, for example, the client device 104. The server system 102 can be configured to receive data (such as images, detected data, and metadata) associated with the imaging targets 108A, 108B from the imaging device 110A and / or the inspection device 110B.

[0022] The server system 102 includes a characterization engine 114 and a database 116. The characterization engine 114 can be configured to be a server device representing various forms of servers, including but not limited to web servers, application servers, proxy servers, network servers, and / or server pools. Generally, the characterization engine 114 receives requests for image processing services (such as the detection and characterization of defects in imaging targets 108A, 108B (such as pipes, pipeline networks, and / or machines)) and provides such services via the network 106 to any number of devices (such as the imaging device 110A, the repair device 112, and the client device 104). The characterization engine 114 can run an instance of the corresponding executable code (such as an.exe file) included in the kernel of the database 116. The kernel of the database 116 can also include other executable code (such as an.exe file) configured to run the characterization engine 114 or other functions of the server system 102. In some embodiments, the executable code can be a computer program that has been compiled into machine language (such as binary code) and is thus capable of being directly executed by a data processor. The characterization engine 114 is communicatively coupled to the database 116.

[0023] The database 116 can be any type of database, including for example a relational database, a non - structured query language (NoSQL) database, an in - memory database, a graph database, a cloud database system environment, etc. The database 116 can store a local library that can include documents 118 and a learning model 120, which can be used by the characterization engine 114 to detect and characterize defects of the imaging targets 108A, 108B. The documents 118 can include training documents (e.g., past repair decisions associated with past recommendations of detected defects), executable files (e.g., executable files used by the image processing application and the defect characterization application to automatically detect and characterize defects in the imaging targets 108A, 108B), and recommendation setting files (e.g., including threshold parameters for automatically characterizing defects). The documents 118 can be stored by the database 116 as a multi - dimensional file, such as an excel file. The excel file can allow a user to edit the document 118, such as modifying dimensions (rows and / or columns within a single or multiple labels), adding items to one or more dimensions of the document 118, and / or modifying the data type within a specific dimension of the document 118. The documents 118 can include any type of electronic document, which includes for example structured data in the form of key - value pairs. The key in the key - value pair can be a string value that identifies the corresponding value (e.g., the threshold parameter value for automatically characterizing defects).

[0024] In some exemplary embodiments, the learning model 120 can be stored in a table, which can be dynamically extended to include additional objects (defect classifications) when the client device 104 creates a defect - related event with an object that is not yet part of the learning model 120. For example, although the learning model 120 is described as including a first set of defect parameters (e.g., location, size, material gain / loss, shape), the characterization engine 114 can receive an indication from the client device 104 to add a new defect parameter (e.g., defect time progression) to the defect characterization. In response to adding the new defect parameter to the defect characterization, the characterization engine 114 can extend one or more tables storing the learning model 120 to accommodate the defect characterization.

[0025] The client device 104 can be and / or include any type of processor- and memory-based device, such as, for example, a cellular phone, a smartphone, a tablet computer, a laptop computer, a desktop computer, a workstation, a personal digital assistant (PDA), a network appliance, a camera, an enhanced general packet radio service (EGPRS) mobile phone, a media player, a navigation device, an email device, a game console, or a suitable combination of any two or more of these devices or other data processing devices. The client device 104 can include any combination of fixed and variable computing components. Although not shown, in some specific implementations, multiple client devices 104 including different computing system configurations (such as different operating systems, different processing capabilities, different hardware components, and / or other differences) can simultaneously receive defect characterization results from the server system 102, including defect repair recommendations from the characterization engine 114. The client device 104 can be configured to display a visual representation of the imaging target (as described in reference Figures 2 to 14 ), as well as the defect characterization results received from the server system 102, on a graphical user interface (GUI) 122. The GUI 122 can include an input interface, an output interface, and a communication interface. The GUI 122 can be configured to receive user input associated with the displayed image of the imaging target (target object), such as via a user input (e.g., a touchscreen display, a keyboard, a keypad, a mouse, a button, a switch, a microphone, a camera, etc.). In some examples, the communication interface 122 allows the client device 104 to receive information from and / or provide information to another device. In some examples, the communication interface 122 includes an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, interface, a cellular network interface, etc.

[0026] As Figure 1 shown, the client device 104 can be communicatively coupled to the server system 102 via the network 106 to enable monitoring of the automatically performed defect characterization process. The network 106 can be any wired and / or wireless network, including, for example, a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), the Internet, a cellular network, a telephone network (e.g., the PSTN), or a suitable combination thereof that connects any number of communication devices, mobile computing devices, fixed computing devices, server systems, etc.

[0027] In a context example, the imaging device 110A (inspection device) may be configured to scan components (e.g., compressor blades) of the imaging targets 108A, 108B (e.g., compressors) according to a scan plan (defining scan settings such as imaging positions) that may be stored by the database 116. The imaging device 110A and / or the sensor 110B may include any device configured to generate images and other data (e.g., position data) using a non-destructive beam, such as an x-ray imaging device, a computed tomography imaging device, and any other type of imaging device and sensor, the non-destructive beam that may penetrate the imaging target up to a set depth and may record responses of the interaction of the beam with the imaging target. The imaging device 110A and / or the sensor 110B may include a camera, a lidar sensor, a radar sensor, an ultrasonic sensor, or other sensing devices. The imaging device 110A may transmit the acquired images to the characterization engine 114, which may automatically process the images of the scanned areas of the imaging targets 108A, 108B according to the plan to detect any potentially existing defects. In some embodiments, the imaging device 110A may adjust its position (e.g., in response to user input) relative to a portion (e.g., blade) of the imaging targets 108A, 108B (e.g., compressors). The imaging device 110A may be integrated into the repair device 112 and / or coupled to the repair device 112 to enable the performance of the characterization and repair processes, as described with reference to FIG. 3. The characterization engine 114 may use a defect characterization application to characterize the detected defects, the defect characterization application including one or more machine learning algorithms for automatically characterizing defects in the imaging targets 108A, 108B. The characterization engine 114 may generate recommendations for the client device 104 to approve the repair of the detected defects characterized as critical that may be performed by the repair device 112.

[0028] In some exemplary embodiments, the characterization engine 114 may generate and / or update at least one or more defect characterization parameters used by a learning model stored by the database 116 based on feedback received from the client device 104. For example, in response to detecting the start of a repair event for a detected defect automatically assigned for monitoring, the characterization engine 114 may update the learning model 120 to increment the critical level of the defect, including combinations of detected defect parameters manually assigned for repair. Alternatively and / or additionally, in response to detecting the replacement of a repair event with a monitoring event for a detected defect automatically assigned to the repair event, the characterization engine 114 may update the learning model 120 to lower the critical level of the defect, including combinations of detected defect parameters manually assigned for monitoring. In this way, the characterization engine 114 may query the database 116 to update the learning model 120 and / or apply the learning model to generate one or more recommendations to be displayed by the client device 104.

[0029] As noted, when the characterization engine 114 receives an indication to create a repair event from the client device 104, the characterization engine 114 may respond by at least applying the learning model 120 to generate recommendations for adding the imaging targets 108A, 108B to the repair event for the client device 104. Thus, the characterization engine 114 may generate recommendations for adding the imaging targets 108A, 108B to the repair event based at least on the defect characterization of the defects detected in the imaging targets 108A, 108B. For example, the characterization engine 114 may generate recommendations for adding the imaging targets 108A, 108B to the repair event based at least on the defect parameters exceeding a critical threshold. Alternatively and / or additionally, the characterization engine 114 may generate recommendations for adding the imaging targets 108A, 108B to the repair event based at least on a combination of defect parameters exceeding respective critical thresholds. Refer to Figure 2 for further details of processes that may be performed by the exemplary system 100.

[0030] Now refer to Figure 2 , which is a flowchart of an exemplary process 200 of rendering a graphical object that overlays a visual representation of a target object in a GUI. The exemplary process 200 may be performed by the defect characterization system 100 described in Figure 1 .

[0031] At 202, a visual representation of a portion of the target object may be rendered in the graphical user interface. The visual representation may include a 2D image of the target image, a 3D image of the target image (e.g., a portion of an industrial machine or industrial component), or both. Images may be received by the server system or the client device from an imaging device configured to image at least a portion of the target object according to an imaging plan. In some embodiments, the imaging plan may include a description of the type of imaging associated with the imaging target type. The imaging device may be configured to generate and transmit data representing a plurality of images (e.g., stereoscopic images) of the target object and optionally metadata associated with the images (e.g., imaging positions relative to fiducial points included in or external to the imaging target). The visual representation may be generated by processing one or more images of the target object. The visual representation may include a point cloud image of the target object. The visual representation may include three-dimensional information of the surface of the target object (e.g., via a 3D representation of the surface of the target object). The 3D representation may be determined based on a plurality of images of the target object. The 3D representation includes a plurality of points on the surface of the target object.

[0032] At 204, a first user input indicating a selection of a first location on the surface of the target object may be received by the client device. The first user input is based on a first user interaction with the visual representation within the GUI of the client device (e.g., by the user clicking on the first location in the 2D image).

[0033] At 206, a reference surface can be determined in the 3D representation of the target object. In some embodiments, the reference surface can be determined based on the selection of a plurality of positions within a first distance from a first position on the surface of the target object. In some embodiments, the first distance can be a distance in the 2D image (e.g., pixels). In some embodiments, the first distance can be a distance in the 3D coordinate system (e.g., millimeters or inches). In some embodiments, the first position can be mapped (or projected) to a first point in the 3D representation of the target object. A plurality of points in the 3D representation that are within the first distance (e.g., a predetermined radius) and on the surface of the target object can be automatically identified. The selected plurality of points are mapped to the above-mentioned plurality of positions (e.g., in the 2D image). The plurality of points can be processed to determine the reference surface. For example, linear regression (e.g., fuzzy logic, neural networks, piecewise linear regression, polynomial fitting, and other pattern recognition and plane estimation techniques) can be used to process the points, and the linear regression can be used to calculate the best-fit plane of the plurality of points. In some embodiments, the reference surface can be determined based on the selection of a plurality of positions on the surface of the target object. For example, at least a first position, a second position, and a third position among the plurality of positions can be selected based on a second user input, a third user input, and a fourth user input, respectively. The first position, the second position, and the third position can be mapped to a first point, a second point, and a third point, respectively. A reference surface including the first point, the second point, and the third point can be determined in the 3D representation of the target object (e.g., using linear regression).

[0034] At 208, a graphical object that renders a visual representation covering the target object may be rendered. The graphical object may be covered based on the determined reference surface. The graphical object may have a perimeter including a plurality of perimeter positions. The perimeter positions may be mapped to perimeter points in the 3D representation. For example, each perimeter position may be mapped to a specific point in the 3D representation. The distance between a first point (e.g., determined based on a first user input selecting a first position) and a second point on the three-dimensional representation of the target object may be less than or equal to a second distance (e.g., indicating the radius of the graphical object). In other words, a plurality of points on the surface of the target object may be defined in the 3D coordinate system and may be used to render the 3D representation. A 3D circle may be defined on the reference surface in the 3D coordinate system. The 3D circle may have a center point and a radius determined based on the first user input and the second user input. The graphical object representing the 3D circle may be covered on the 3D representation (e.g., drawn on top of the 3D representation or rendered as part of the 3D representation). It may also be determined based on projecting one or more points on the perimeter of the 3D circle into the 2D image space according to the position and orientation of the camera used to capture the 2D image in the 3D coordinate system and its optical characteristics (e.g., field of view, optical distortion, pixel size, etc.). The graphical object covered on the 2D image and / or 3D image may appear as a circular shape (e.g., a circle or an ellipse) depending on the viewing perspective of the camera relative to the reference surface. In some embodiments, the graphical object covered on the 3D image may appear as a spherical shape (e.g., a sphere) depending on the viewing perspective of the camera relative to the reference surface.

[0035] In some embodiments, the points may include a first point and a second point. The first point may be the center of a graphical object that is displayed as a circle (which is mapped to the graphical object in the 2D image) in the 3D representation, and the second point may be a point on the perimeter of the circle. In some embodiments, both the first point and the second point may be located on the perimeter of the circle such that the distance between the first point and the second point is the diameter of the graphical object that is displayed as a circle, as Figure 13 described.

[0036] At 210, a second user input indicating a first distance may be received by the client device. The second user input may include values of the measurement plane and / or parameters (e.g., radius or diameter) of a graphical object (e.g., a circle) to be added covering the image of the target object. For example, the user input may include the placement of three cursors on the target surface in the representation, which may be used to determine the best-fit plane using a plurality of 3D points from adjacent positions in the 3D representation associated with the three cursor positions. In some embodiments, the second input may indicate the selection of a second position on the surface of the target object. The first distance (e.g., the diameter of the circle) may be determined based on the distance between a first point (which is the mapping of the first position in the 3D representation) and a second point (which is the mapping of the second position in the 3D representation).

[0037] At 212, a second parameter of a graphical object (e.g., a circle) overlaid on an image of a target object can be determined by using a second user input. For example, a first point (which is the center of the circle) and a first parameter value (e.g., a radius or a diameter) can be processed to determine a second point on the perimeter of the circle. In some embodiments, processing the first point (which is the center of the circle) and the first parameter value (e.g., the radius or the diameter) can include adding a measurement plane mask to the image of the target object, as Figure 6 described in

[0038] At 214, the target object is characterized by the client device or the server system based on the parameters of the graphical object. In some specific implementations, the characterization of the target object (e.g., an industrial machine such as a jet engine) may include comparing the parameters with associated thresholds. For example, the characterization of the target object (e.g., an industrial machine such as a jet engine) may include determining the edge profile of the industrial machine or a part thereof for performing the comparison with the threshold. The edge profile can be compared with a specific predetermined criterion that defines the characterization of a fault-free surface (e.g., a surface without defects). For example, the radius of the edge profile can correspond to a minimum radius value and a maximum radius value that define the structural integrity of the target object. For example, in aviation, debris passing through a jet engine can impact the edge of a compressor blade, causing damage associated with edge deformation, resulting in a radius outside the control radius range. Typically, the radius assessment is not performed until after the "blending" process. The damage is usually evaluated by its size and location on the blade to determine whether it requires blending. If so, blending is performed, and then the radii of the entry, exit, and central portions of the blended area can be checked against the minimum and / or maximum radius requirements. The damage caused can lead to stress concentration, which increases the risk of crack formation on the blade. The crack may eventually cause a part of the blade to break off, resulting in increased damage. The characterization of the target object may include classifying the target object as damaged (if one or more parameters are outside the set range and / or threshold) or fault-free (if all parameters are within the set range and / or below the set threshold). In some specific implementations, the critical level of the defect is determined by determining whether the detected defect is critical, medium, or insignificant. The classification of the critical level can be based on determining whether one or more defect parameters are above the threshold. In some specific implementations, if any of the defect parameters is above the corresponding first threshold, then the defect is identified as critical. The corresponding defect parameter threshold can depend on the type of material constituting the imaging target, the type of asset defining the imaging target, and the critical level associated with the imaging target. For example, the defect parameter threshold for a peripheral industrial asset can be higher than the defect parameter threshold for a node industrial asset whose function enables the functions of other industrial assets. In some specific implementations, if multiple defect parameters are above the corresponding second threshold, then the defect is identified as critical. For example, if the detected percentage of material change (loss or gain) along a dimension exceeding the second dimension threshold is above the corresponding second threshold, then the defect is identified as critical. As another example, if the change (change over time) of any of the defect parameters exceeds the change rate threshold, indicating an increase in defect severity, then a defect previously classified for a monitoring category can be reallocated to the critical category. In some specific implementations, the classification of the defect can be performed by a learning model trained on a set of images and user input for confirming and / or adjusting the classification.

[0039] At 216, in response to determining that the target object is damaged, an operation to be performed by the repair device can be selected by the client device or the server system. The operation can be selected based on a characterization and a repair plan that is recognized as matching the determined characterization and is compatible with the target object. In the example of a damaged blade of a jet engine, the repair device can include a grinding tool that can be inserted to grind away the damaged area, thereby creating a smooth edge profile that can reduce stress concentration (e.g., known as "blending"). In some cases, the engine manufacturer can specify a minimum allowable radius along the blended blade edge to ensure that the stress concentration is below an acceptable limit. Existing borescopes do not provide a practical way to determine whether the repaired (blended) blade meets the radius requirement. In some embodiments, the acceptability of the blended blade edge can be determined based on a comparison of the edge profile of the blended blade with a graphical object (e.g., a circle, an ellipse, etc.) by repeating at least a portion of the exemplary process 200. For example, the graphical object can be generated (e.g., by steps 202-108) and overlaid on an image of the blade (e.g., a 2D image of the blade). The graphical object can be an ellipse in the 2D image, which can be mapped to a circle in the 3D representation of the blade. The graphical object can be moved on the image of the blade (e.g., by clicking and dragging). In some embodiments, the perimeter of the graphical object can be placed on the blended blade edge, and the curvature of the perimeter of the graphical object can be compared with the curvature of the blended blade edge. For example, a perimeter position in the perimeter of the graphical object can be placed at a position on the blended edge (e.g., the position where blending starts in the blade). In some embodiments, the radius of the graphical object (or the projection of the graphical object in the 3D representation) can be determined based on the depth of the blend in the blade (e.g., can be a predetermined multiple determined by a multiplication value of the depth, as Figure 14 described). In some embodiments, if the curvature of the blended blade edge is less than the curvature of the perimeter of the graphical object, the blended blade can be considered not suitable for operation. In some embodiments, multiple graphical objects can be generated in the graphical user interface. For example, a first graphical object centered above the blade can be generated, and a second graphical object not centered above the blade can be generated. Referring to FIGS. 3 to Figure 14 provides an example of a representation of a graphical object.

[0040] Figure 3A and Figure 3B show an exemplary representation of at least one graphical object overlaid on a 2D image of the blade, including a measurement plane. Figure 3A shows an exemplary representation of a graphical object overlaid on a 2D image of the blade. The center of the graphical object is located on the blade. Figure 3BAn exemplary representation of two graphical objects overlaid on a 2D image of a blade is shown. The center of the first graphical object is located on the blade, while the center of the second graphical object is not located on the blade. The two graphical objects have different radii.

[0041] Figure 4 An exemplary representation including a 2D image of a blade and a point cloud image is shown. Figure 5 An example of generating a reference surface is shown by selecting three positions on the surface of a blade with an added measurement plane, and the reference surface can be referred to as a measurement plane. Figure 6 Shows a portion on the surface of the blade within a predetermined distance from the Figure 5 reference plane generated in Figure 7 and Figure 8 An exemplary representation of a point cloud image of a blade and a graphical object is shown, where the 3D circle represented by the graphical object is located on the reference surface determined using three positions on the surface of the blade selected in Figure 5 Figures 9 to 12 An exemplary representation of a 2D image of a blade is shown, where the graphical object and / or the reference surface are overlaid on the image of the blade.

[0042] Figure 13 An exemplary GUI that allows providing a user input indicating the radius of the circle represented by the graphical object is shown. Figure 14 An exemplary GUI that allows providing a user input indicating a multiplication value that can be used to determine the radius of the circle represented by the graphical object is shown. For example, a point-to-line measurement can be used to determine the depth of a blend. The multiplication value can be multiplied by the depth of the blend in the blade to determine the radius of the circle represented by the graphical object. In some specific implementations, the GUI can allow providing a user input that selects the result of an existing measurement (such as a point-to-line measurement) that will be used as a factor in determining the radius of the circle represented by the graphical object.

[0043] Figure 15 An exemplary imaging device including a borescope 1500 according to the embodiments presented herein is shown. The exemplary borescope 1500 can include a portable device with a limited amount of memory and storage space. The exemplary borescope 1500 can be configured to capture and generate a large amount of data during an inspection. The data generated by the exemplary borescope 1500 can include 2D images and / or 3D volume images with corresponding 3D data, measurement results, human or computer-generated annotations and indications, menu-driven inspection (MDI) metadata, and other metadata associated with the inspection, as referenced Figures 1 to 14 as described. The data captured from one or more borescopes 1500 can be continuously synchronized to a remote server system (e.g., the server system 102 referenced Figure 1 and managed from the borescope 1500.​

[0044] The endoscope 1500 may include a control unit 1502 (or controller), a catheter segment 1504, a flexible joint movement segment 1506, and a head segment 1508. In one embodiment, the segments 1504, 1506, 1508 may have different lengths and may be integral with each other or may be detachable from each other. As shown, the catheter segment 1504 may be adapted to be inserted into various different targets, such as inside a turbomachine, equipment, a pipe, a conduit, an underwater location, a turn, a bend, inside or outside an aircraft system, etc. The endoscope 1500 may include a probe driver coupled to the catheter segment 1504. The probe driver may include a motor (not shown) configured to translate and / or rotate one or more of the segments 1504, 1506, 1508 (e.g., to facilitate insertion of the probe head 1508 into an imaging target). Additionally or alternatively, the orientation / position of a portion of the head segment 1508 (e.g., a camera, a light source, etc.) may be varied to obtain an image of an inspection area (e.g., an RGB image, an IR image, etc.). The control unit 1502 may include a control unit housing 1510, a controller 1512, an orientation input terminal 1514, and a screen 1516. The controller 1512 may include a processor and a readable memory having computer-readable instructions executable by the processor to actuate the endoscope 1500. The computer-readable instructions may include an inspection plan based on which the endoscope 1500 or a portion thereof (e.g., the catheter segment 1504, the flexible joint movement segment 1506, and the head segment 1508) may be translated / rotated (e.g., by the probe driver). In some specific implementations, the operation of the probe driver may be based on a control signal (e.g., generated by the controller based on the inspection plan / user input via a GUI on the screen 1516, a computing device, etc.).

[0045] The controller 1512 may be communicatively coupled to the control unit 1502 via one or more signals. The controller 1512 may also be disposed within the control unit housing 1510 or may be disposed outside the control unit housing 1510. In some specific implementations, the orientation input terminal 1514 may be configured to receive user input (e.g., direction control) for the control unit 1502 for actuation of the endoscope 1500. The screen 1516 may display visual information received by a camera (including an optical sensor) disposed in the head segment 1508, which may allow the user to better guide the endoscope 1500 using the orientation input terminal 1514. The orientation input terminal 1514 and the screen 1516 may be communicatively coupled to the controller 1512 via one or more signals, which may be a hardwired connection or a wireless signal (such as WI-FI or Bluetooth). In one specific implementation, inspection data and / or notifications (e.g., notifications based on inspection data as described above) may be provided on the screen 1516.

[0046] The catheter segment 1504 may include a tubular housing that includes a proximal end and a distal end. The tubular housing may be a flexible member along its entire length or may be rigid at the proximal end and become more flexible as it travels distally along the length of the catheter segment 1504. In some embodiments, the tubular housing may be formed of a non-porous material to prevent contaminants from entering the endoscope 1500 via the catheter segment 1504. The control unit 1502 may be disposed at the proximal end of the tubular housing, and the flexible joint movement segment 1506 may be disposed at the distal end of the tubular housing. The flexible joint movement segment 1506 may include a flexible neck and a gasket. The flexible neck may be disposed at the distal end of the tubular housing and be actuatable in the Y-Z plane. The flexible neck may be wrapped in a non-porous material to prevent contaminants from entering the endoscope 1500 via the flexible joint movement segment 1506. The head segment 1508 may include a head assembly. The head assembly may include one or more light sources (e.g., LEDs or a fiber optic bundle having a lamp at the proximal end), a camera (or cameras, such as a visible light camera, an IR camera, etc.), and one or more sensors that may be configured to collect data about the surrounding environment. The camera of the endoscope 1500 may provide images and videos suitable for inspection to the screen 1516 of the control unit 1502. When the head segment 1508 is disposed in a location with low or no light, the light source may be used to provide illumination. The sensors may record data, including temperature data, distance data, clearance data (e.g., the distance between a rotating element and a stationary element), flow data, etc. During use, the flexible joint movement segment 1506 and the probe driver may be controlled, for example, by control inputs (e.g., relative control gestures, physical manipulation devices) from the orientation input 1514 and / or control signals generated by the controller 1512. The orientation input may be a joystick, a cross key, a touchpad, a trackball, an optical sensor, or a touch screen on the screen 1516. The orientation input 1514 may also be a similar device located outside the control unit housing 1510 and connected by wire or wirelessly. In particular, a set of control inputs may be used to control the flexible joint movement segment 1506 and / or the probe driver 1509. The flexible joint movement segment 1506 may be steered or "bent" in various dimensions, while the catheter segment 1504 may be translated and / or rotated using any combination of actuators and wires disposed within the control unit 1502 to adjust the orientation (e.g., positioning) of the head segment 1508. In some specific implementations, the control input / direction input 1514 may be generated by the controller based on an inspection plan.

[0047] The actuator can be an electric, pneumatic, or ultrasonic-operated motor or solenoid, shape memory alloy, electroactive polymer, dielectric elastomer, polymer muscle material, or other material. For example, the flexible joint movement segment 1506 and the probe driver 1509 can enable the head segment 1508 to move in the X-Y plane, X-Z plane, and / or Y-Z plane. In fact, the orientation input 1514 can be used to perform control actions suitable for setting the head segment 1508 at various angles (such as the depicted angle α). In this way, the head segment 1508 can be positioned to visually inspect the desired location. Once the head segment 1508 is in the desired position, the camera can be operated to obtain, for example, a still visual image or a continuous visual image, which can be displayed on the screen 1516 of the control unit 1502 and can be recorded by the endoscope 1500. In an embodiment, the screen 1516 can be a multi-touch touch screen that uses capacitive technology, resistive technology, infrared grid technology, etc. to detect the touch of a stylus and / or one or more human fingers. Additionally or alternatively, the acquired visual images can be transmitted to a separate storage device for later reference.

[0048] As Figure 16 shown, the computing system 1600 can include a processor 1610, a memory 1620, a storage device 1630, and an input / output device 1640. The processor 1610, the memory 1620, the storage device 1630, and the input / output device 1640 can be interconnected via a system bus 1650. The processor 1610 is capable of processing instructions for execution within the computing system 1600. Such executable instructions can be implemented by one or more components of, for example, the server system 102 and / or the client device 104 described with reference to Figure 1 In some exemplary embodiments, the processor 1610 can be a single-threaded processor. Alternatively, the processor 1610 can be a multi-threaded processor. The processor 1610 is capable of processing instructions stored in the memory 1620 and / or the storage device 1630 to render graphical information for a user interface provided via the input / output device 1640.

[0049] The memory 1620 is a computer-readable medium, such as volatile or non-volatile, that stores information within the computing system 1600. For example, the memory 1620 may store data structures representing a configuration object database. The storage device 1630 is capable of providing persistent storage for the computing system 1600. The storage device 1630 may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, or other suitable persistent storage device. The input / output device 1640 provides input / output operations for the computing system 1600. In some exemplary embodiments, the input / output device 1640 includes a keyboard and / or a pointing device. In various embodiments, the input / output device 1640 includes a display unit for displaying a graphical user interface.

[0050] According to some exemplary embodiments, the input / output device 1640 may provide input / output operations for a network device. For example, the input / output device 1640 may include an Ethernet port or other networking ports to communicate with one or more wired and / or wireless networks (e.g., local area network (LAN), wide area network (WAN), Internet).

[0051] In some exemplary embodiments, the computing system 1600 may be used to execute various interactive computer software applications that can be used to organize, analyze, and / or store data in various formats. Alternatively, the computing system 1600 may be specifically configured to execute software applications. These applications may perform various full detection functions, such as, for example, a planning function (e.g., generating, managing, editing spreadsheet documents, word processing documents, and / or any other objects, etc.), a computing function, a communication function, etc. The applications may include various plug-in functions or may be stand-alone computing products and / or functions. When activated within an application, the function may be used to generate a user interface provided via the input / output device 1640. The user interface may be generated by the computing system 1600 and presented to a user (e.g., on a computer screen monitor, etc.).

[0052] One or more aspects or features of the subject matter described herein can be implemented in specially configured digital electronic circuitry, integrated circuitry, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), computer hardware, firmware, software, and / or combinations thereof. These various aspects or features can include the implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, the implementation being specific or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device. The programmable system or computing system can include clients and servers. The clients and servers are remote from each other and typically interact via a communication network. The relationship of client and server arises from running computer programs on the respective computers and having a client-server relationship to each other.

[0053] These computer programs (which may also be referred to as programs, software, software applications, applications, components, or code) include machine instructions for a programmable processor and can be implemented in a high-level procedural and / or object-oriented programming language and / or in assembly language / machine language. As used herein, the term “machine-readable medium” refers to any computer program product, apparatus, and / or device, such as a disk, optical disk, memory, and programmable logic device (PLD), that provides machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term “machine-readable signal” refers to any signal that provides machine instructions and / or data to a programmable processor. The machine-readable medium can non-transitorily store such machine instructions, such as a non-transitory solid state memory or a magnetic hard disk drive or any equivalent storage medium would do. Alternatively or additionally, the machine-readable medium can store such machine instructions in a transient manner, such as a processor cache or other random access memory associated with one or more physical processor cores would do.

[0054] To provide interaction with a user, one or more aspects or features of the subject matter described herein may be implemented on a computer having a display device (such as, for example, a cathode ray tube (CRT), a liquid crystal display (LCD), or a light emitting diode (LED) monitor) for displaying information to the user and a keyboard and a pointing device (such as, for example, a mouse or a trackball) by which the user may provide input to the computer. Other kinds of devices may also be used to provide interaction with the user. For example, the feedback provided to the user may be any form of sensory feedback, such as, for example, visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. Other possible input devices include a touch screen or other touch-sensitive devices, such as a single-point or multi-point resistive or capacitive touchpad, speech recognition hardware and software, an optical scanner, an optical pointer, a digital image capture device, and associated interpretation software, etc.

[0055] In the foregoing specification and claims, phrases such as “at least one” or “one or more” may appear after a conjunctive list of elements or features. The term “and / or” may also appear in a list of two or more elements or features. Unless otherwise implied in the context in which the phrase is used or explicitly contradicted by that context, such phrases are intended to mean either any one of the individually listed elements or features or any combination of the listed elements or features with any other of the listed elements or features. For example, the phrases “at least one of A and B,” “one or more of A and B,” and “A and / or B” each are intended to mean “A alone, B alone, or A and B together.” Similar interpretations are intended for lists including three or more items. For example, the phrases “at least one of A, B, and C,” “one or more of A, B, and C,” and “A, B, and / or C” each are intended to mean “A alone, B alone, C alone, A and B together, A and C together, B and C together, or A and B and C together.” The term “based on” as used above and in the claims is intended to mean “at least partially based on” such that features or elements not recited are also permissible.

[0056] As used herein, a “user interface” (also referred to as an interactive user interface, a graphical user interface, or a user interface or UI) may refer to a web-based interface that includes data fields and / or other control elements for receiving input signals or providing electronic information and / or for providing information to a user in response to any received input signal. The control elements may include dials, buttons, icons, selectable areas, or other perceivable markers presented via the UI that initiate a data exchange for the device presenting the UI when interacting with these control elements (such as, for example, clicking, touching, selecting, etc.). Languages such as HyperText Markup Language (HTML), FLASH TM 、JAVATM ,.NET TM , web services, or Rich Site Summary (RSS) technologies to implement the UI, in whole or in part. In some particular implementations, the UI may be included in a stand-alone client (e.g., thick client, fat client) configured to communicate (e.g., send or receive data) according to one or more of the aspects described. This communication may be to or from a medical device or server with which it communicates.

[0057] As used herein, the term "determine" encompasses a wide variety of actions. For example, "determine" may include calculating, computing, processing, deriving, generating, obtaining, looking up (e.g., looking up in a table, database, or other data structure), ascertaining, etc., via a hardware element without user intervention. Additionally, "determine" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), etc., via a hardware element without user intervention. "Determine" may include parsing, selecting, picking, establishing, etc., via a hardware element without user intervention.

[0058] As used herein, the term "provide" encompasses a wide variety of actions. For example, "provide" may include storing a value in a location in a storage device for subsequent retrieval, directly transmitting the value to a receiver via at least one wired or wireless communication medium, transmitting or storing a reference to the value, etc. "Provide" may also include encoding, decoding, encrypting, decrypting, checking, validating, etc., via a hardware element.

[0059] As used herein, the term "message" encompasses a wide variety of formats for conveying (e.g., transmitting or receiving) information. A message may include a machine-readable collection of information, such as an XML document, a fixed-field message, a comma-separated message, etc. In some particular implementations, a message may include a signal for transmitting one or more representations of the information. Although stated in the singular, it will be understood that a message may be composed, transmitted, stored, received, etc., in multiple parts.

[0060] As used herein, the term "correspond" or "corresponding" encompasses a structural, functional, quantitative, and / or qualitative correlation or relationship between two or more objects, data sets, information, etc., preferably, where the correspondence or relationship may be used to translate one or more of the two or more objects, data sets, information, etc., such that they appear the same or equal. One or more of a threshold, a range of values, fuzzy logic, pattern matching, a machine learning evaluation model, or a combination thereof may be used to evaluate the correspondence.

[0061] In some specific implementations, the generated or detected data can be forwarded to a "remote" device or location, where "remote" refers to a location or device different from the location or device where the program is executed. For example, a remote location can be another location in the same city (e.g., an office, a laboratory, etc.), another location in a different city, another location in a different state, another location in a different country, etc. Thus, when an item is indicated as "away from" another item, this means that the two items can be in the same room but separated, or at least in different rooms or different buildings, and can be separated by at least one mile, ten miles, or at least one hundred miles. "Conveying" information means transmitting the data representing the information as an electrical signal through a suitable communication channel (e.g., a dedicated or public network). "Forwarding" an item means any means of getting the item from one location to the next, whether by physically transporting the item or otherwise (where possible), and in the case of data at least includes physically transporting the medium carrying the data or conveying the data. Examples of communication media include radio or infrared transmission channels and network connections to another computer or networked device, as well as the Internet or information including email transmissions and information recorded on websites, etc.

[0062] The subject matter described herein can be embodied in systems, devices, methods, and / or articles, depending on the desired configuration. The specific implementations set forth in the foregoing description do not represent all specific implementations consistent with the subject matter described herein. Instead, they are only some examples consistent with aspects of the described subject matter. Although a few variations have been described in detail above, other modifications or additions are possible. Specifically, features and / or variations other than those set forth herein can also be provided. For example, the above specific implementations can relate to various combinations and sub - combinations of the disclosed features and / or combinations and sub - combinations of several additional features disclosed above. Additionally, the logical flows depicted in the figures and / or described herein do not necessarily need to be in the particular order or sequence shown to achieve the desired result. Other specific implementations are within the scope of the following claims.

Claims

1. A method for interactive measurement of a three-dimensional representation of a target object, the method comprising: Rendering a visual representation of the target object in a graphical user interface display space; Receiving a first user input indicative of a selection of a first position on the surface of the target object based on a first user interaction with the visual representation; Determining a measurement plane based on selections of a plurality of positions on the surface of the target object, wherein the plurality of positions are within a first distance from the first position on the surface of the target object, and wherein the measurement plane extends beyond the boundary of the target object; And Rendering, in the graphical user interface, a graphical object that overlays the visual representation of the target object, wherein a position of the graphical object in the graphical user interface is based on the measurement plane, and wherein the graphical object includes a first point associated with the first position; and Characterizing the target object based on the graphical object and a specified threshold.

2. The method according to claim 1, further comprising: Receiving data representing a plurality of images of the target object from a videoscope; And Determining a three-dimensional representation of the target object, the three-dimensional representation including a plurality of points on the surface of the target object, wherein the visual representation of the target object is based on the plurality of images of the target object and / or the three-dimensional representation of the target object.

3. The method according to claim 2, wherein the plurality of images are stereoscopic images of the target object.

4. The method according to claim 2, further comprising receiving a second user input indicative of a second distance.

5. The method according to claim 4, further comprising receiving a second user input indicative of a selection of a second position on the surface of the target object; and Determining the second distance based on the first position and the second position.

6. The method according to claim 1, wherein the first point is a projection of the first position on the surface of the target object.

7. The method according to claim 1, wherein the visual representation includes a two-dimensional image of the target object.

8. The method according to claim 1, wherein the visual representation includes a three-dimensional representation of the target object.

9. The method according to claim 1, wherein the target object includes an industrial device.

10. The method according to claim 4, wherein the plurality of positions are mapped to a plurality of points in a three-dimensional coordinate system of the three-dimensional representation, and wherein each of the plurality of positions corresponds to a specific three-dimensional point.

11. The method according to claim 10, wherein the graphical object is mapped to a circle in the three-dimensional coordinate system, and wherein the second distance indicates a radius of the circle.

12. The method according to claim 11, wherein the circle lies on the measurement plane.

13. The method according to claim 10, wherein the graphical object is an ellipse in the visual representation of the target object.

14. A system for interactive measurement of a three-dimensional representation of a target object, the system comprising: At least one data processor; A memory coupled to the at least one data processor, the memory storing instructions for causing the at least one data processor to perform operations, the operations including: Rendering a visual representation of a target object in a graphical user interface display space; Receiving a first user input indicative of a selection of a first position on a surface of the target object based on a first user interaction with the visual representation; Determining a measurement plane based on selections of a plurality of positions on the surface of the target object, wherein the plurality of positions are within a first distance from the first position on the surface of the target object, and wherein the measurement plane extends beyond a boundary of the target object; and Rendering a graphical object in the graphical user interface that overlays the visual representation of the target object, wherein a position of the graphical object in the graphical user interface is based on the measurement plane, and wherein the graphical object includes a first point associated with the first position; and Characterizing the target object based on the graphical object and a specified threshold.

15. A computer program product comprising a machine-readable medium storing instructions that, when executed by at least one programmable processor, cause the at least one programmable processor to perform operations, the operations including: Rendering a visual representation of a target object in a graphical user interface display space; Receiving a first user input indicative of a selection of a first position on a surface of the target object based on a first user interaction with the visual representation; Determining a measurement plane based on selections of a plurality of positions on the surface of the target object, wherein the plurality of positions are within a first distance from the first position on the surface of the target object, and wherein the measurement plane extends beyond a boundary of the target object; And Rendering a graphical object in the graphical user interface that overlays the visual representation of the target object, wherein a position of the graphical object in the graphical user interface is based on the measurement plane, and wherein the graphical object includes a first point associated with the first position; and Characterizing the target object based on the graphical object and a specified threshold.

Images