Interactive 3D representation of an object

Abstract

A method includes receiving data representative of a target surface in three dimensions. It includes rendering a first visual representation and a second visual representation in a graphical user interface display space, the first visual representation including a two-dimensional image of a first portion of the target surface, the second visual representation including a three-dimensional representation of at least a subset of the first portion of the target surface included in the first visual representation. It includes receiving a first user input indicating selection of a first location on the target surface via a cursor based on a first user interaction with the three-dimensional representation. It includes rendering a first graphical object at a first target location in the three-dimensional representation and a second graphical object at a second target location in the two-dimensional image. The first target location and the second target location indicate the first location on the target surface.

Description

[0001] Related Applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 956,465, filed on January 2, 2020, under 35 U.S.C.§119(e), the entire content of which is hereby incorporated by reference in its entirety. Background Art

[0003] Three-dimensional objects (e.g., surfaces) can be imaged and presented in a graphical user interface (GUI). For example, a two-dimensional image (e.g., a picture) of the object can be presented in the GUI. Alternatively, a three-dimensional image (e.g., a point cloud view) of the object can be presented in the GUI. A point cloud view can be generated by detecting a plurality of positions on the outer surface of the object using 3D imaging techniques such as structured light projection, stereoscopy, photogrammetry, time-of-flight, etc. In some embodiments, multiple images of an external object can be combined to generate a portion of a three-dimensional image. The point cloud can be used to generate a 3D computer-aided design (CAD) model of a manufactured part in an animation, for measurement, etc. Summary of the Invention

[0004] Aspects of the disclosed subject matter can provide one or more of the following capabilities.

[0005] A method includes receiving data representative of a target surface extending in three dimensions. The method further includes rendering a first visual representation and a second visual representation in a display space of a graphical user interface, the first visual representation including a two-dimensional image of a first portion of the target surface, and the second visual representation including a three-dimensional representation of at least a subset of the first portion of the target surface included in the first visual representation. The method further includes receiving a first user input indicating selection of a first position of the target surface based on a first user interaction with the three-dimensional representation. The method further includes rendering a first graphical object at a first target position in the three-dimensional representation and a second graphical object at a second target position in the two-dimensional image. The first target position and the second target position indicate the first position of the target surface.

[0006] One or more of the following features can be included in any feasible combination.

[0007] In one embodiment, the method further includes receiving a second user input indicating selection of two positions of the target surface based on a second user interaction with the three-dimensional representation. The method further includes rendering a numerical value (measurement result) in the graphical user interface, which can be one or more values indicating the distance between the two positions.

[0008] In one embodiment, the second user interaction includes: selecting a first position on a three-dimensional representation (e.g., via a mouse, via a touch screen, etc.); and selecting a second position on the three-dimensional representation (e.g., via a mouse, via a touch screen, etc.), wherein the first position and the second position indicate two locations. In another embodiment, the method further includes receiving a third user input indicating three positions (e.g., a first position, a second position, and a third position) that select a target surface based on a third user interaction with the three-dimensional representation. The method also includes indicating a portion of the target surface rendered in the three-dimensional representation that lies on or adjacent to a reference plane that includes the three selected positions. These positions are indicated by a first set of three graphical objects having a predetermined color in a two-dimensional image (e.g., a graphical object for each position) and a second set of three graphical objects having the predetermined color in the three-dimensional representation.

[0009] In one embodiment, the method further includes: rendering a third graphical object indicating the reference plane in the three-dimensional representation; and rendering a fourth graphical object indicating the distance between a target position on the three-dimensional representation and the reference plane in the three-dimensional representation. In another embodiment, the method includes receiving a fourth user input indicating movement of a first graphical object along a first path in the three-dimensional representation based on a fourth user interaction with the first graphical object. The method also includes rendering movement of a second graphical object along a second path in the two-dimensional image, wherein the second path is determined by identifying pixel coordinates in the two-dimensional image that are associated with 3D surface points traversed along the first path.

[0010] A non-transitory computer program product (i.e., a physically embodied computer program product) storing instructions is also described, which when executed by one or more data processors of one or more computing systems, causes at least one data processor to perform the operations herein. Similarly, a computer system is also described, which may include one or more data processors and a memory coupled to the one or more data processors. The memory may temporarily or permanently store instructions that cause at least one processor to perform one or more of the operations described herein. Additionally, the method may be implemented by one or more data processors within a single computing system or by one or more data processors distributed between two or more computing systems. Such computing systems may be connected via one or more connections, including connections over a network (e.g., the Internet, a wireless wide area network, a local area network, a wide area network, a wired network, etc.), via direct connections between one or more of the multiple computing systems, etc., and may exchange data and / or commands or other instructions, etc.

[0011] These and other capabilities disclosed will be more fully understood after review of the following figures, detailed description, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] These and other features will be more readily understood from the following detailed description in conjunction with the accompanying drawings, in which:

[0013] Figure 1 is a flowchart of an exemplary method presented in an interactive graphical user interface having a two-dimensional image and a three-dimensional representation of the two-dimensional image;

[0014] Figure 2 illustrates an exemplary graphical user interface (GUI) that includes a two-dimensional image of a portion of a target surface and a three-dimensional representation of that portion of the target surface shown in the two-dimensional image;

[0015] Figure 3 illustrates Figure 2 an exemplary implementation of the GUI in

[0016] Figure 4 illustrates Figure 2 an exemplary implementation of the GUI in

[0017] Figures 5A to 5C illustrates an Figure 2 exemplary user interaction with a graphical object in the GUI of

[0018] Figure 6 illustrates an exemplary switching between various measurements; and

[0019] Figure 7 illustrates an exemplary GUI including a zoom window. DETAILED DESCRIPTION

[0020] Three-dimensional objects (e.g., surfaces) can be imaged, and the imaging data can be presented to a user for inspection in a graphical user interface (GUI) of an inspection system. The imaging data can be presented as two-dimensional (2D) images and / or three-dimensional (3D) images. For example, a structured light system including a camera, a white light illuminator, and a structured light pattern projector can be used. By enabling the white light illuminator, the camera can capture a 2D white light image of the target surface. The camera can also capture a 2D image of the structured light pattern projected onto the target surface, which can then be used to calculate the 3D (x, y, z) coordinates associated with each pixel in the 2D white light image. The inspection system can present the 2D white light image in the GUI. These 2D images and / or 3D images can be interactive and can allow the user to inspect the surface (e.g., measure distances between positions on the surface, view the surface from various viewpoints, etc.). In some embodiments, both a 2D image and a 3D image of the surface can be presented in the GUI simultaneously. The 3D image can be a three-dimensional representation of a portion of the surface shown in the 2D image. The user can interact with the 2D image (e.g., move a cursor on the 2D image (via a mouse, via a touch screen, etc.), select a portion of the 2D image, zoom in on a portion of the 2D image, etc.) and can generate a corresponding interaction in the 3D representation. For example, the user can move the cursor along a path on the 2D image, and the inspection system can generate a graphical object (e.g., another cursor) that can move along a path in the 3D representation. Alternatively, the user can select a path in the 2D image via interaction with a touch screen, which can cause the generation of a graphical object (e.g., another cursor) that can move along a path in the 3D representation. This path can be determined by identifying the pixel coordinates in the 3D representation associated with the path traversed in the 2D image.

[0021] It can be desirable for the user to interact directly with the 3D representation (also referred to as the 3D point cloud view) rather than indirectly via the 2D image. For example, the user may want to select a position on the imaged surface (e.g., for measurement), and the 3D representation can provide the user with increased visibility of the surface profile and / or improved spatial accuracy in selecting positions compared to the 2D image. Although the user can change the position of the cursor on the 2D image by viewing the corresponding movement of a graphical object in the 3D representation (which corresponds to traversing above the imaged surface), this process can be cumbersome and inefficient. The present application provides an improved inspection system that allows the user to interact directly with the 3D representation and generate a corresponding interaction in the 2D image (e.g., direct interaction via a mouse, interaction via a touch screen, etc.).

[0022] Figure 1A flowchart of an exemplary method of presenting a two-dimensional image having a target surface and a three-dimensional representation of the target surface shown in the two-dimensional image. At 102, data characterizing a target surface extending in three dimensions may be received (e.g., by a computing device in a detection system). This data may be generated by imaging the target surface (e.g., a weld connecting two surfaces) and may include three-dimensional information of the target surface (e.g., the contour of the target surface). The data characterizing the target surface may be stored in a storage device (e.g., remotely stored in a server) and transferred to a computing device.

[0023] Return Figure 1 , at 104, a first visual representation of a two-dimensional image including at least a first portion of the target surface may be rendered in the GUI, and a second visual representation including a three-dimensional representation of the first portion of the target surface shown in the two-dimensional image included in the first visual representation. Figure 2 An exemplary GUI 200 having a first visual representation 202 is shown, the first visual representation including a 2D image of a portion of the target surface. The GUI 200 also includes a second visual representation 204, the second visual representation including a three-dimensional representation of the portion of the target surface shown in the two-dimensional image in the first visual representation 202. The first / second visual representations 202 / 204 may include graphical objects that may indicate selection of a location (e.g., graphical object 210), lines connecting graphical objects, measurements (e.g., measurement 208), etc.

[0024] In some embodiments, a graphical object (such as a cursor) may be used to select a location on the viewing surface to determine the geometric dimensions of a feature on the viewing surface. For example, to measure the height of a weld connecting two metal plates, a point-to-plane measurement may be used, where three datum plane cursors may be placed to select three datum plane positions on one plate, and a fourth offset cursor may be placed to select an offset position on the far edge of the weld. The 3D coordinates associated with the datum plane positions may be used to determine the datum plane, and the distance along a direction perpendicular to the datum plane from the plane to the 3D coordinates associated with the offset position may be calculated and displayed as a measurement result. Lines connecting the three datum plane cursors may also be added to distinguish them from the offset cursor. Lines may also be added to represent the projection from the offset position back to the datum plane. Other types of measurements (such as length, area, point-to-plane distance, etc.) may be performed using different numbers of cursors and including different types of graphical objects to indicate different relationships or associated functions.

[0025] In some specific implementations, 3D coordinates are not considered when rendering graphical objects on top of 2D images or 3D representations, such that in the displayed GUI, the pixels of the 2D image or 3D representation are partially or completely occluded by the graphical objects without determining any 3D relationships. In another implementation, the graphical object is defined in 3D space, and its relationship with the observed target surface and other graphical objects in the 3D space affects the way it appears in the GUI. For example, a sphere 214 shown as part of a 3D representation 204 may have assigned central 3D coordinates and a defined radius. If the target surface is rendered from a viewpoint from which a first portion of the sphere 214 is closer to the viewpoint than any portion of the target surface, and a second portion of the sphere 214 is farther from the viewpoint than a portion of the target surface, the second portion may be partially or completely occluded by the surface, and the first portion may partially or completely occlude some portions of the target surface. Thus, the sphere 214 appears to have a non-zero volume and appears to be part of the 3D scene, rather than simply being a 2D graphical object rendered on top of the 3D representation 204.

[0026] In some specific implementations, the graphical objects shown in the 3D representation 204 are defined by one or more triangles, which are rendered by a graphics processing unit (GPU). For example, multiple triangles can be used to form the shape of a graphical object or a portion thereof (such as a sphere, an annulus, etc.). The graphical object can be oriented relative to a reference plane (e.g., a reference plane selected by the user). For example, the graphical object can be oriented relative to a normal vector associated with the reference plane.

[0027] In some specific implementations, the detection system can calculate 3D coordinates and render a 3D representation in the GUI based on the calculated 3D coordinates. For example, the calculated 3D coordinates can be used to define the vertices of 3D triangles, and then the vertices can be processed using a graphics processing unit (GPU) to render a 3D image from the perspective of a virtual camera, the position of which relative to the target surface can be freely assigned. In some specific implementations, the 3D coordinates can be associated with the pixels of a 2D image and can indicate the position of a point on the target surface associated with the 2D pixel. The detection system can generate a set of 3D surface points in the 3D representation, where each 3D surface point includes one of a plurality of 3D coordinates and a color based on the color of the associated 2D image pixel.

[0028] In some specific implementations, the graphical object can be interactive (e.g., the user can click on the graphical object, drag the graphical object onto a 2D / 3D image, etc.). For a given graphical object in the first / second visual representation (e.g., the sphere 214 in the 3D representation), the detection system can generate a corresponding graphical object in an adjacent visual representation. For example, the detection system can generate a graphical object 216 in the 2D image that corresponds to the graphical object 214 in the 3D representation. Additionally, the graphical object 214 and the graphical object 216 can be recognized (e.g., using a circle, a sphere, etc.).

[0029] The visual characteristics of the graphical object can indicate the state of the graphical object (e.g., active, inactive, movable, etc.). For example, as Figure 2 shown, the graphical object 208 that provides the measurement result can be in an enabled state as indicated by the blue outline. At this time, the graphical object 214 can be represented by a solid sphere rendered as part of a 3D scene, and the graphical object 216 can be represented by a circle drawn on top of the 2D image. This can indicate that the cursor is inactive. When interacting with the cursor (e.g., by placing the interactive cursor on the marker 214 / 216), the marker can become active. Figure 3 An exemplary embodiment of the GUI 200 with the activated markers 214 / 216 is shown. For example, the activated marker 214 in the 3D representation can be represented by a sphere with an annulus around the sphere. The corresponding marker 216 in the 2D image can be represented by a blue circle (e.g., larger in size compared to Figure 2 the inactive green circle).

[0030] Figure 4 Another exemplary embodiment of the GUI 200 is shown, where the markers 214 / 216 are in a movable state. This indicates that the markers 214 / 216 can be moved in the 3D representation / 2D image. The markers in the movable state can be presented by a visual representation different from that of the markers in the inactive state or the active state. For example, as Figure 4 shown, the marker 214 in the movable state is represented by a smaller sphere (compared to the active state) and a larger annulus around the sphere (compared to the active state). The graphical object (e.g., the marker 214) can switch between different states (e.g., based on input from the user).

[0031] Return Figure 1, at 106, a first user input indicating a first position of a target surface can be received (the "first input mode" of the detection system). This reception can be based on user interaction with the 3D representation. For example, the user can move and place the cursor at position 222 in the 3D representation and select position 220 (e.g., by clicking, touching the touch screen of the GUI 200, etc.). Alternatively, the user can move and place the cursor at position 220 in the 3D representation and select an existing graphical object 214. The user interaction can include selecting multiple positions on the 3D representation (e.g., by multiple clicks of the mouse, multiple touches on the touch screen, etc.).

[0032] In some specific embodiments, the user can indicate a desired new position or a change in position of a graphical object via one or more of touch, mouse drag, joystick movement, arrow keys, zoom window press, etc. Based on the new position of the graphical object, the detection system can determine the corresponding position in the 2D image (e.g., if the selection is made in the 3D representation) or in the 3D representation (e.g., if the selection is made in the 2D image). For example, the new position selected by the user can be in the screen pixel coordinates of the GUI 200 (e.g., 2D pixel coordinates). Each screen pixel coordinate can be associated with a ray in the coordinate system of the 3D representation. The detection system can calculate this ray based on the selected screen pixel coordinates and the orientation of the 3D representation. The 3D representation of the target surface (or a part thereof) can be formed using multiple triangles, and the calculated ray can collide with the 3D representation of the target surface at one of the triangles.

[0033] Additionally or alternatively, some embodiments may allow a user, when operating in different input modes, to use the same user input to rotate a 3D representation for viewing from different advantageous positions and to adjust the position of a 3D graphical object (e.g., sphere 214). For example, in a first input mode, a user input (such as a mouse drag while holding down the left mouse button, a touchscreen drag, a joystick push, or pressing a key on the keyboard) may cause a change in the position of a 3D graphical object that is in a movable state on the surface of an object shown in a second visual representation 204, while in a second input mode, these same user inputs may cause a rotation of the 3D representation, with no change in the position of any 3D graphical object on the surface of the object. To transition from the first input mode (3D graphical object movement) to the second input mode (rotation), the user may click or tap or press a designated button at a position at least a minimum distance from any 3D graphical object, which will cause any graphical object that is in a movable state to transition to an active or inactive state. To transition from the second input mode (rotation) to the first input mode (3D graphical object movement), the user may click or tap within a maximum distance of the 3D graphical object, at which point the graphical object will enter a movable state. To minimize the risk of accidental movement of the 3D graphical object, some embodiments may require pressing and releasing the mouse button or contacting and disconnecting from the screen near the 3D graphical object, with no significant movement of the mouse or the finger or stylus touching the screen, before transitioning from the second input mode to the first input mode. After transitioning to the first input mode, initiating a mouse or touch drag at a point near the 3D graphical object (user input at 106) may cause movement of the object without any rotation of the 3D representation. Additionally, after entering the first input mode, some embodiments may detect an offset between the starting position of a mouse or touch drag and the 3D graphical object that is in a movable state and maintain that offset between the mouse or touch point and the updated position of the 3D graphical object at step 108 to allow the user to maintain an unobstructed view while the 3D graphical object is being repositioned. Otherwise, it may be obscured by the mouse cursor or finger or stylus being used, making it difficult to accurately position. Thus, in some embodiments, the operational change between the second input mode and the first input mode may be at least partially based on the distance between a first pixel position of a graphical object on the GUI (e.g., a pixel selected at step 106) and the pixel position of a mouse or trackball cursor shown on the display device when a button on the mouse or trackball is pressed, or the distance between pixel positions associated with points on a touch-sensitive panel where contact is initiated on the display device.

[0034] At 108, a first graphical object may be rendered at a first target position in the three-dimensional representation, and a second graphical object may be rendered at a second target position in the two-dimensional image. For example, based on the interaction at step 106 at position 222, a new graphical object (e.g., graphical object 210) may be generated in the 3D representation in the second visual representation 204. Additionally, a second graphical object 230 may be generated at position 232 in the 2D image in the first visual representation 202. The position 210 in the 3D representation and the position 232 in the 2D image may indicate the same position on the target surface.

[0035] In another embodiment, the interaction at step 106 may activate an existing graphical object (e.g., marker 214). Once the marker 214 is activated, it may be manipulated. For example, the marker 214 may be moved in the 3D representation of the second visual representation 204. Figures 5A to 5C An exemplary interaction with the marker 214 is shown. The detection device may receive an input indicating movement of the graphical object (e.g., marker 214) along a first path in the 3D representation. This input may be based on user interaction with the marker 214 (e.g., dragging the marker 214 on the 3D representation). As Figure 5BAs shown, based on this user interaction with marker 214, marker 216 in the 2D image can move along a second path. The second path can be determined by identifying pixel coordinates in the 2D image associated with 3D surface points that traverse along the first path. In some embodiments, the 3D representation can include only 3D surface points that fall within a 3D region of interest, which is determined using separate measurements of features formed on the surface of the observed object using the position of the marker. Since the position of the marker (e.g., 214) is changed in step 108, the region of interest can be changed, resulting in an increase or decrease in the screen space occupied by the 3D surface in the 3D representation. Adjusting the magnification and / or position of the 3D representation during step 108 can make it difficult to control the marker position because the surface can move under the mouse cursor or finger / stylus on the touch screen, and thus change the surface points displayed at the mouse / touch position. In one embodiment, the user will have to manually reposition and / or resize the 3D representation after moving the marker in order to maintain the desired size of the 3D representation on the display. Alternatively, the 3D representation can be automatically resized and recentered in the second visual representation 204, making the operation more efficient. For example, resizing the 3D representation can include reducing or increasing the area (or magnification) of the 3D representation (e.g., by a predetermined ratio), and recentering can include determining the center of the 3D representation and placing the center of the 3D representation at a predetermined center of the second visual representation 204. Automatic resizing and / or recentering can occur when step 108 is completed. For example, if the user input at step 106 is dragging the mouse while holding down the mouse button, then automatic resizing / recentering can occur when the mouse button is released. If the user input for step 106 is the movement of a finger or stylus on the touch screen, then automatic resizing / recentering can occur when contact with the touch screen is broken. If the user input at step 106 is a push of a joystick away from its central position, then automatic resizing / recentering can be performed when the joystick returns to a point near its central position.

[0036] In some specific implementations, the distance between two positions on a target object can be determined based on the interaction with the GUI 200. For example, the user can interact with the 3D representation in the second visual representation 204 (e.g., via a cursor), and select two positions on the 3D representation (e.g., by clicking on two positions, by touching two positions on an interactive touch screen, etc.) (indicating two positions on the surface of the target surface). For example, this can be done by setting the GUI 200 to operate in a measurement mode (e.g., by selecting the measurement mode from the task bar of the GUI). After selecting two positions on the target surface, the detection system can calculate and present the distance between the two positions in the GUI 200. For example, the Euclidean distance can be calculated and presented as the distance between the two positions and / or along various dimensions (e.g., along the x-axis, y-axis, z-axis, etc.).

[0037] In some specific implementations, multiple measurements of a target object can be performed based on the interaction with the GUI 200. The user can switch between various measurements (e.g., measurement values, graphical objects associated with the measurements, etc.). Figure 6 An exemplary switch between various measurements is shown. For example, the user can select a "next measurement" icon in the GUI that allows the user to switch between measurements. In one specific implementation, the method performed by the detection system can include defining a first measurement including a first plurality of measurement cursors and a second measurement including a second plurality of measurement cursors. Each measurement cursor can be associated with a pixel position in the 2D image. The method can further include determining a first 3D region of interest based on the positions of the first plurality of measurement cursors (or markers), and determining a second 3D region of interest based on the positions of the second plurality of measurement cursors (or markers). The first 3D graphical object can be one of the first plurality of 3D graphical objects associated with the first plurality of measurement cursors, and the second plurality of 3D graphical objects can be associated with the second plurality of measurement cursors. The first 3D image can include the first plurality of 3D graphical objects and not include the second plurality of 3D graphical objects, the first representation of the surface of the object excludes 3D surface points outside the first 3D region of interest, and in response to a measurement selection user input action, a new 3D image is displayed. The new 3D image can include the second plurality of 3D graphical indicators and a new representation of the surface of the object excluding 3D surface points outside the second 3D region of interest.

[0038] In some specific implementations, a reference surface may be generated in the 3D representation in the second visual representation 204. This reference surface can be used as a reference relative to which it is possible to measure or identify which position on the target surface. The reference surface can be generated by receiving user input indicating the selection of three positions on the target surface and by generating a reference surface that includes the three selected positions. In some specific implementations, the generated reference surface (e.g., the edges / vertices / markers of the reference surface) may be superimposed on the 3D representation in the second visual representation 204. Additionally or alternatively, portions of the target surface that are located on or near the reference surface (e.g., within a predetermined distance from the reference plane) may be visually indicated in the 3D representation (e.g., by color-coding these portions of the target surface with a predetermined color). In some specific implementations, a graphical object (e.g., a line) may be rendered in the 3D representation, which may indicate a position (e.g., a predetermined target position) on the target surface (e.g., position 222) relative to the reference plane.

[0039] In some specific implementations, a secondary 3D representation may be displayed simultaneously with the first 3D representation in the second visual representation 204. The secondary 3D representation may include a secondary representation of the target surface generated from 3D surface points, with a magnification different from that of the first 3D representation. The first 3D representation and the secondary 3D representation may be rendered from a similar 3D perspective, and the first 3D light punctuation may be displayed in the secondary 3D representation. In some specific implementations, the user input at step 106 may occur within the secondary 3D representation, and the second pixel position on the GUI may be offset from the first pixel position on the GUI. The direction of the offset may be derived from the position of the first user input action on the secondary 3D image. In some specific implementations, the first representation of the target surface includes a different color from that in the 2D image, and the secondary representation of the surface of the object includes the color from the 2D image.

[0040] In some specific implementations, the GUI may include a zoom window that can expand a portion of the 3D representation and / or the 2D image in the GUI. Figure 7An exemplary GUI 300 is shown. The exemplary GUI includes a first visual representation 302 that includes a 2D image and a second visual representation 304 that includes a 3D representation of the 2D image in the first visual representation 302. The second visual representation 304 includes a zoom window 306 that can provide an expanded view of a portion of the 3D representation. For example, the zoom window 306 can expand a portion of the 3D representation located near the cursor on the 3D representation. When the cursor moves to a new position, the zoom window shows the portion of the 3D representation around the new position of the cursor. In some embodiments, the zoom window 306 does not include any graphical objects and provides an unobstructed view of the target surface (or a portion thereof). If the orientation of the 3D representation is changed, then the zoom window 306 shows the target surface (or a portion thereof) from a favorable position relative to the 3D representation in the new orientation. Regardless of the magnification of the second visual representation 304, the zoom window 306 can show the same portion of the 3D representation at the same portion.

[0041] In some embodiments, both the first visual representation 202 and the second visual representation 204 can include 3D representations of different portions of the target surface. This can allow a user to simultaneously access three-dimensional point cloud views of different portions of the target surface. In some embodiments, the GUI 200 can include a third visual representation (not shown) that can include a third 3D representation of a third portion of the target surface. The various 3D representations can be presented from different perspectives. The third visual representation having the third 3D representation of the target surface can be generated based on one or more received user inputs. For example, the one or more user inputs can include the user input received at step 106 in Figure 1 (e.g., based on the movement of a finger or stylus in contact with the touch-sensitive panel) and / or can include a second user input that disconnects the finger or stylus from contact with the touch-sensitive panel. In some embodiments, the first user input can be based on the movement of a mouse or trackball when a button on the mouse or trackball is pressed, and the second user input can be based on the release of the button on the mouse or trackball. In some embodiments, the appearance of a graphical object changes when transitioning between a first input mode and a second user input mode. For example, a first graphical object in the first input mode can include a first set of triangles combined into an approximate ring shape, and the corresponding graphical object in the second input mode can include a change in the size, color, visibility, or shape of the approximate ring.

[0042] In a specific implementation, a user may provide a zoom control input (e.g., at step 106) to change the magnification of a 3D representation (e.g., in the second visual representation 204). The magnification may change the initial 3D representation by a first zoom factor. The sizes of the graphical objects before and after magnification may be different by a second zoom factor. The first zoom factor and the second zoom factor may be different, and the second zoom factor may not be equal to 1. For example, if the 3D representation is magnified by a zoom factor of 2.0, then the size of the 3D graphical object may increase by a zoom factor of 1.4. This may provide a preferred user experience as compared to keeping the size of the graphical object unchanged or increasing the size of the graphical object by the same zoom factor as the 3D representation.

[0043] Figures 2 to 7 A GUI showing two - dimensional and three - dimensional representations of a portion of a target surface is shown. In some specific implementations (not shown), the GUI may include only the three - dimensional representation of a portion of the target surface. As described above, the user performs various interactions with the three - dimensional representation (e.g., selecting multiple locations on the target surface, using a zoom window to provide an expanded view of a portion of the 3D representation, simultaneously accessing 3D point cloud views of different portions of the target surface, etc.).

[0044] The entire contents of U.S. Patent No. 7,170,677 titled "Stereometric Endoscope with 3D Viewing" and U.S. Patent No. 7,821,649 titled "Stripe Projection System and Method for a Probe Suitable for Phase - Shift Analysis" are hereby incorporated by reference.

[0045] Other embodiments are within the scope and spirit of the disclosed subject matter. For example, the inspection system described in this patent application may be used in a facility with complex machines (e.g., power generation turbines) that have multiple operating parameters that need to be modified to change the machine performance. In this application, the use of the words "optimize / optimizing" may imply "improve / improving".

[0046] This document describes certain exemplary embodiments to provide a comprehensive understanding of the structure, function, manufacture, and use principles of the systems, devices, and methods disclosed herein. One or more examples of these embodiments are shown in the accompanying drawings. Those skilled in the art will understand that the systems, devices, and methods specifically described and shown in the drawings are non-limiting exemplary embodiments, and the scope of the present invention is defined only by the claims. Features shown or described in connection with one exemplary embodiment may be combined with the features of other embodiments. Such modifications and variations are intended to be included within the scope of the present invention. Additionally, in this disclosure, components with similar names in the embodiments generally have similar features, so within a specific embodiment, not every feature of each similarly named component necessarily needs to be fully elaborated.

[0047] The subject matter described herein can be implemented in digital electronic circuitry and / or in computer software, firmware, or hardware (including the structural means disclosed in this specification and their structural equivalents) or in a combination thereof. The subject matter described herein can be implemented as one or more computer program products, tangibly embodied in an information carrier (e.g., embodied in a machine-readable storage device) or in a propagated signal, for use by or to control the operation of a data processing apparatus (e.g., a programmable processor, a computer, or multiple computers). A computer program (also referred to as a program, software, software application, or code) can be written in any form of programming language (including a compiled language or an interpreted language), and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file. A program can be stored in a part of a file that holds other programs or data, in a single file dedicated to the program being considered, or in multiple cooperating files (e.g., files that store one or more modules, subroutines, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers, which are located at one site or distributed across multiple sites and interconnected by a communication network.

[0048] The processes and logical flows described in this specification, including the method steps of the subject matter described herein, can be executed by one or more programmable processors executing one or more computer programs to perform the functions of the subject matter described herein by operating on input data and generating output. The processes and logical flows can also be executed by dedicated logic circuitry (such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit)), and the devices of the subject matter described herein can be implemented as dedicated logic circuitry (such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit)).

[0049] By way of example, processors suitable for executing a computer program include both general and special purpose microprocessors, as well as any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include one or more mass storage devices for storing data (e.g., magnetic disks, magneto-optical disks, or optical disks), or be operatively coupled to receive data from and / or transfer data to one or more mass storage devices for storing data (e.g., magnetic disks, magneto-optical disks, or optical disks). Information carriers suitable for embodying computer program instructions and data include all forms of non-volatile memory, including by way of example semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices); magnetic disks (e.g., internal hard disks or removable disks); magneto-optical disks; and optical disks (e.g., CD disks and DVD disks). The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0050] For purposes of providing an interaction with a user, the subject matter described herein may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) or a touch screen by which the user may provide input to the computer. Other kinds of devices may also be used to provide for interaction with the user. For example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input received from the user may be in any form, including acoustic, speech, or tactile input.

[0051] The techniques described herein may be implemented using one or more modules. As used herein, the term "module" refers to computing software, firmware, hardware, and / or various combinations thereof. However, at a minimum, a module should not be construed as software that is not implemented in hardware, firmware, or recorded on a non-transitory processor-readable storage medium (i.e., a module is not software per se). In fact, a "module" will be construed to always include at least some physical non-transitory hardware, such as a processor or a portion of a computer. Two different modules may share the same physical hardware (e.g., two different modules may use the same processor and network interface). The modules described herein may be combined, integrated, separated, and / or replicated to support various applications. Additionally, instead of or in addition to the functions performed at a particular module, functions described herein as being performed at a particular module may be performed at one or more other modules and / or by one or more other devices. Further, modules may be implemented locally or remotely relative to each other across multiple devices and / or other components. Additionally, a module may be moved from one device and added to another device, and / or may be included in both devices.

[0052] The subject matter described herein may be implemented in a computing system that includes backend components (e.g., data servers), middleware components (e.g., application servers), or frontend components (e.g., client computers having a graphical user interface or a web interface through which a user may interact with an implementation of the subject matter described herein), or any combination of such backend components, middleware components, and frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), such as the Internet.

[0053] As used throughout this specification and the claims, approximating language may be used to modify any quantitative representation that could vary but not result in a change in the basic function associated therewith. Accordingly, values modified by one or more terms such as "about" and "substantially" are not to be limited to the exact values specified. In at least some instances, the approximating language may correspond to the precision of the instrument used to measure the value. Herein, as well as throughout the specification and the claims, range limitations may be combined and / or interchanged, unless the context or language indicates otherwise, and such ranges are recognized and include all the subranges contained therein.

Claims

1. A method, the method comprising: Receive data representing a target surface extending in three dimensions; Render a first visual representation and a second visual representation in a graphical user interface display space, the first visual representation including a two-dimensional image of a first portion of the target surface, and the second visual representation including a three-dimensional representation of at least one subset of the first portion of the target surface included in the first visual representation; Based on a first user interaction with the three-dimensional representation, receive a first user input indicating selection of a first position on the target surface; Render a first graphical object at a first target position in the three-dimensional representation and a second graphical object at a second target position in the two-dimensional image, where the first target position and the second target position indicate the first position on the target surface; Based on a second user interaction with the three-dimensional representation, receive a second user input indicating selection of a first position, a second position, and a third position on the target surface; And Indicate a portion of the target surface rendered in the three-dimensional representation that lies on or adjacent to a reference plane determined based on the selected first position, second position, and third position.

2. The method according to claim 1, the method further comprising: Based on a third user interaction with the three-dimensional representation, receive a third user input indicating selection of two positions on the target surface; And Render a first value indicating the distance between the two positions in the graphical user interface.

3. The method according to claim 2, wherein the third user interaction comprises: Select a first position on the three-dimensional representation; And Select a second position on the three-dimensional representation, where the first position and the second position indicate the two positions.

4. The method according to claim 1, wherein the first position, the second position, and the third position are indicated by a first set of three graphical objects having a predetermined color in the two-dimensional image and a second set of three graphical objects having the predetermined color in the three-dimensional representation.

5. The method according to claim 4, the method further comprising: Render a third graphical object indicating the reference plane in the three-dimensional representation; And Render a fourth graphical object indicating the distance between a target position on the three-dimensional representation and the reference plane in the three-dimensional representation.

6. The method according to claim 1, the method further comprising: Based on a fourth user interaction with the first graphical object, receive a fourth user input indicating movement of the first graphical object along a first path in the three-dimensional representation; And In the two-dimensional image, render movement of the second graphical object along a second path, where the second path is determined by identifying pixel coordinates in the two-dimensional image that are associated with surface points on the target surface traversed along the first path.

7. A system, the system comprising: At least one data processor; A memory coupled to the at least one data processor, the memory storing instructions to cause the at least one data processor to perform operations including: Receive data representing a target surface extending in three dimensions; Render a first visual representation and a second visual representation in a graphical user interface display space, the first visual representation including a two-dimensional image of a first portion of the target surface, and the second visual representation including a three-dimensional representation of at least one subset of the first portion of the target surface included in the first visual representation; Based on a first user interaction with the three-dimensional representation, receive a first user input indicating selection of a first position on the target surface; Render a first graphical object at a first target position in the three-dimensional representation and a second graphical object at a second target position in the two-dimensional image, where the first target position and the second target position indicate the first position of the target surface; Based on a second user interaction with the three-dimensional representation, receive a second user input indicating selection of a first position, a second position, and a third position on the target surface; and Indicate a portion of the target surface rendered in the three-dimensional representation, the portion being on or adjacent to a reference plane determined based on the selected first position, second position, and third position.

8. The system according to claim 7, wherein the operation further comprises: Based on a third user interaction with the three-dimensional representation, receive a third user input indicating selection of two positions on the target surface; And Render a first value in the graphical user interface indicating the distance between the two positions.

9. The system according to claim 8, wherein the third user interaction comprises: Select a first position on the three-dimensional representation; And Select a second position on the three-dimensional representation, where the first position and the second position indicate the two positions.

10. The system according to claim 7, wherein the first position, the second position, and the third position are indicated by a first set of three graphical objects having a predetermined color in the two-dimensional image and a second set of three graphical objects having the predetermined color in the three-dimensional representation.

11. The system according to claim 10, wherein the operation further comprises: Render a third graphical object in the three-dimensional representation indicating the reference plane; And Render a fourth graphical object in the three-dimensional representation indicating the distance between a target position on the three-dimensional representation and the reference plane.

12. The system according to claim 7, wherein the operation further comprises: Based on a fourth user interaction with the first graphical object, receive a fourth user input indicating movement of the first graphical object along a first path in the three-dimensional representation; And In the two-dimensional image, render movement of the second graphical object along a second path, where the second path is determined by identifying pixel coordinates in the two-dimensional image that are associated with surface points on the target surface traversed along the first path.

13. A computer program product comprising a machine-readable medium storing instructions that, when executed by at least one programmable processor, cause the at least one programmable processor to perform operations, the operations comprising: Receive data characterizing a target surface that extends in three dimensions; Render a first visual representation and a second visual representation in the display space of a graphical user interface, the first visual representation including a two-dimensional image of a first portion of the target surface, the second visual representation including a three-dimensional representation of at least one subset of the first portion of the target surface included in the first visual representation; Based on a first user interaction with the three-dimensional representation, receive a first user input indicating selection of a first position on the target surface; Render a first graphical object at a first target position in the three-dimensional representation and a second graphical object at a second target position in the two-dimensional image, where the first target position and the second target position indicate the first position of the target surface; Based on a second user interaction with the three-dimensional representation, receive a second user input indicating selection of a first position, a second position, and a third position on the target surface; And Indicate a portion of the target surface rendered in the three-dimensional representation, the portion being on or adjacent to a reference plane determined based on the selected first position, second position, and third position.

14. The computer program product according to claim 13, wherein the operation further comprises: Based on a third user interaction with the three-dimensional representation, receive a third user input indicating selection of two positions on the target surface; And Render a first value indicating the distance between the two positions in the graphical user interface.

15. The computer program product according to claim 14, wherein the third user interaction comprises: Select a first position on the three-dimensional representation; And Select a second position on the three-dimensional representation, where the first position and the second position indicate the two positions.

16. The computer program product according to claim 13, wherein the first position, the second position, and the third position are indicated by a first set of three graphical objects having a predetermined color in the two-dimensional image and a second set of three graphical objects having the predetermined color in the three-dimensional representation.

17. The computer program product according to claim 16, wherein the operation further comprises: Render a third graphical object indicating the reference plane in the three-dimensional representation; And Render a fourth graphical object indicating the distance between the target position on the three-dimensional representation and the reference plane in the three-dimensional representation.

Images