3D modeling and robotic tool systems and methods

Abstract

3D modeling and robotic tool systems and methods are provided that enable more efficient spatial processing of objects and their surroundings. The robotic tool system includes: a scanner configured to capture spatial data of an object and its surroundings; and a processor configured to: assign spatial data to cells of the cellular space, determine one or more characteristics of the cells based on data of adjacent cells, and configure the robotic tool to operate on the object based at least in part on the one or more characteristics.

Description

Technical Field

[0001] The present invention relates to 3D modeling and robotic tools. In particular, although not exclusively, the present invention relates to 3D modeling used when interacting with physical objects with robotic tools (such as robotic welding, spraying, grinding, polishing, deburring and sandblasting tools). Background Art

[0002] The use of robotics in manufacturing, including small-scale manufacturing, is becoming increasingly popular because it enables consistent, high-quality work to be performed at a relatively low cost.

[0003] Examples of such uses of robotics include robotic welding, painting, grinding, polishing, deburring, and sandblasting. However, almost any task that is performed manually can be performed by a robot.

[0004] One problem with using robotics in such tasks is that tasks that are relatively simple for humans may be relatively difficult for robotic systems. As an illustrative example, obstacle avoidance is typically a relatively simple task for humans, but can be complex for robotic systems.

[0005] In many cases, point clouds corresponding to items and obstacles are generated. However, point clouds include many individual points that can be connected to form a mesh. This mesh consists of many small faces and edges, and analyzing these small faces and edges to generate information about items and obstacles is often difficult and time-consuming. In short, processor-intensive and time-consuming edge detection algorithms are often used to detect the edges of items and obstacles to set the operating boundaries.

[0006] Another example of a task that is relatively simple for humans but difficult for robotic systems is path generation (e.g., calculating the best path from a point to a goal while avoiding obstacles). Although path algorithms exist, they are typically processor intensive and must usually be recalculated if the starting point is changed.

[0007] Yet another example of a task that is relatively simple for a human but difficult for a robotic system is being able to accommodate differences between the actual object and the CAD model due to wide tolerances or other reasons.

[0008] Therefore, there is a clear need for improved 3D modeling and robotic tooling methods and systems.

[0009] It should be clearly understood that, if a prior art publication is referred to herein, this reference does not constitute an admission that the publication forms part of the common general knowledge in the art, in Australia or any other country. Summary of the invention

[0010] The present invention is directed to 3D modeling and robotic tool methods and systems that may at least partially overcome at least one of the above-mentioned disadvantages or provide consumers with a useful or commercial choice.

[0011] In view of the above, the present invention broadly resides in a form of a robot tool system, the robot tool system comprising:

[0012] a scanner configured to capture spatial data of the object and its surroundings;

[0013] The processor is configured as:

[0014] Assign spatial data to cells of the formatted space;

[0015] Determining one or more characteristics of a cell of the lattice space based on data of neighboring cells; and configuring the robotic tool to operate on the object based at least in part on the one or more characteristics.

[0016] Advantageously, using a lattice space rather than spatial data (eg, a point cloud) directly enables more efficient spatial processing of an object and its surroundings.

[0017] Preferably, the lattice space includes a plurality of cells of uniform size.

[0018] Preferably, the lattice space is three-dimensional, and each cell corresponds to a position in the three-dimensional space.

[0019] Preferably, the shape of the cell is a cube. Alternatively, the shape of the cube is spherical. Cubic cells are particularly useful for three degrees of freedom (X, Y, Z), while spherical cells are more suitable for six degrees of freedom (X, Y, Z, Roll, Pitch, Yaw).

[0020] Preferably, the scanner is configured to generate spatial data in the form of a point cloud. The scanner may be configured to generate spatial data from a plurality of different angles. The scanner may be coupled to a robotic arm.

[0021] Preferably, a first characteristic of one or more of the cells is initially determined, and a second characteristic of the one or more cells is determined based on the first characteristic. An example of the first characteristic is whether the cell includes an obstacle, and an example of the second characteristic is the distance to the obstacle.

[0022] Preferably, the one or more characteristics of the cell include a proximity pattern, the proximity pattern including a spatial pattern of characteristics of adjacent cells. The spatial pattern may be defined based on a binary characteristic of each of the adjacent cells. An example of a binary characteristic is whether each of the adjacent cells is occupied by an object.

[0023] The neighboring pattern can be represented by a code or sequence that indicates which of the neighboring cells have binary properties.

[0024] The processor may be configured to compare the proximity patterns of adjacent cells to determine their patterns. The proximity patterns may be compared in a binary manner (eg, the proximity patterns are substantially the same or not the same).

[0025] In the case of proximity patterns that define whether each of the neighboring cells is occupied by an object, comparison of proximity patterns can provide an effective means for identifying continuous surfaces and edges. Proximity patterns shared between neighboring cells can provide an effective means for identifying continuous surfaces, and sudden changes in proximity patterns can provide an effective means for identifying edges.

[0026] The neighbor pattern can be three-dimensional. The neighbor pattern may involve a 3×3×3 block of cells centered at the cell.

[0027] Preferably, the one or more characteristics of the cell comprise a fraction.

[0028] Scores can include distance scores. Scores can relate to the distance to the target cell. Scores can relate to the distance to obstacles.

[0029] The distance may include an absolute distance.

[0030] The distance may include a distance measured in one or more steps. The steps may include linear steps.

[0031] The score may include a weighted combination of two or more scores. As an illustrative example, the score may include a combination of the distance to the obstacle and the distance to the target. Such a configuration can prioritize both obstacle avoidance and short paths with appropriate weights.

[0032] In some embodiments, the system may be configured to recognize a marking on an object and configure the robotic tool to operate on the object based at least in part on the recognized marking.

[0033] The marking may include a colored marking. The marking may include a sprayed marking. Alternatively, the marking may include a tape or other type of marking.

[0034] The system can be configured to scan an object, detect landmarks, and generate a point cloud including the detected landmarks.

[0035] The point cloud including the detected markers may be filtered or processed to generate a line (path).The point cloud may be filtered or processed iteratively.

[0036] Lines can be generated based on the average distance to other points.

[0037] Lines can be smoothed, for example, to avoid sharp turns.

[0038] In some embodiments, the system can be configured to receive a model associated with an object. The model can include a CAD model.

[0039] The system may be configured to align the point cloud with the model.

[0040] The system may align features (eg, edges and / or surfaces) of the scan data with features (eg, edges and / or surfaces) of the model.

[0041] The system may use one or more bounding boxes to align features of the scan data with the CAD model.

[0042] The system can measure the deviation between the object and the model.

[0043] The model can be refined based on the deviations.

[0044] The system can determine whether the deviation exceeds a threshold.

[0045] The system can use the partial model data to align the point cloud with the model. This configuration can be particularly useful when only a small portion of the part can be captured by the scan data (e.g., in connection with large objects).

[0046] The system may select a slice or portion of the model and align the point cloud with the slice or portion of the model.

[0047] The system can then align slices or portions of the model with the scan data. This can include pose refinement.

[0048] By using data points that are isolated from data points that are close to the scan data, the system is able to refine a more accurate translation (and rotation) match for the part.

[0049] The system can identify planes in the scanned point cloud data. The system can extrapolate the planes and identify the intersection points between the planes. The route (path) can then be determined based on the intersection of the planes.

[0050] The system can segment the point cloud data into blobs based on the direction of the point normals to identify planes. An average plane can be generated through each of the blobs.

[0051] The system may provide a user interface to the user, thereby enabling a path to be defined thereon. When the path can be defined, the user interface may visualize the object.

[0052] The system may scan an object for quality control after performing one or more tasks on the object.

[0053] The system can determine whether a point is within a predefined threshold. Local averaging can be performed to avoid quality control failures based on outliers in the scan data.

[0054] The system may be configured to visualize the parallax between the scan data and the predefined model. The parallax may be shown as color coding.

[0055] In another form, the invention resides broadly in a robotic tool method comprising:

[0056] Capture spatial data of objects and their surroundings;

[0057] Assign spatial data to cells of the formatted space;

[0058] Determining one or more characteristics of a cell of the lattice space based on data of neighboring cells; and configuring the robotic tool to operate on the object based at least in part on the one or more characteristics.

[0059] In yet another form, the invention resides broadly in a 3D modeling system comprising:

[0060] a scanner configured to capture spatial data of the object and its surroundings; and

[0061] The processor is configured as:

[0062] Assign spatial data to cells of the formatted space;

[0063] One or more characteristics of a cell of the lattice space are determined based on data of neighboring cells.

[0064] In yet another form, the invention resides broadly in a 3D modeling method comprising:

[0065] Capture spatial data of objects and their surroundings;

[0066] Assigning spatial data to cells of the formatted space; and

[0067] One or more characteristics of a cell of the lattice space are determined based on data of neighboring cells.

[0068] Within the scope of the present invention, any feature described herein may be combined with any one or more of the other features described herein, in any combination.

[0069] The reference to any prior art in this specification is not, and should not be taken as, an acknowledgment or any form of suggestion that the prior art forms part of the common general knowledge. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Different embodiments of the present invention will be described with reference to the following drawings, in which:

[0071] Figure 1 A robotic tool system according to an embodiment of the present invention is shown.

[0072] Figure 2 The embodiment of the present invention is shown Figure 1 An exemplary lattice environment for a system of FIG.

[0073] Figure 3 Shows Figure 2 The 3×3×3 cell neighborhood of the gridded environment, including the cell (marked with X) and its neighboring cells.

[0074] Figure 4 Shows Figure 2 Example of a slice of a lattice environment, where occupied cells are shown as square planar surfaces with shading corresponding to items.

[0075] Figure 5 An example of a planar surface including a cell (marked with an X) and its neighboring cells (including their numbers) is shown.

[0076] Figure 6 shows an example of a 2D grid environment, where occupied cells are shaded, the neighbor score is shown in the upper left corner of each cell, and the Figure 5 The nomenclature also stores the details of the neighboring cells through which the scores are provided.

[0077] Figure 7 Another example of a 2D grid environment is shown, where the occupied cells are shaded, the target distance score is shown on top of each cell, and the Figure 5 The nomenclature of , in the base of each cell, stores the details of the neighboring cells through which the score is provided.

[0078] Figure 8 Another example of a 2D lattice environment is shown, where occupied cells are shaded and an obstacle distance score is shown in each cell.

[0079] Fig. 9 A robotic tool method according to an embodiment of the present invention is shown.

[0080] Preferred features, embodiments and variations of the present invention can be discerned from the following detailed description, which provides sufficient information for those skilled in the art to carry out the invention.The detailed description should not be construed as limiting the scope of the foregoing summary of the invention in any way. DETAILED DESCRIPTION

[0081] Embodiments of the invention are disclosed that enable the creation and analysis of 3D models of physical objects and their surroundings in a simple and efficient manner. These embodiments have a wide range of applications, but are particularly suitable for use when interacting with physical objects using robotic tools, such as robotic welding, spraying, grinding, polishing, deburring, and sandblasting tools, among other applications.

[0082] Figure 1 A robotic tool system 100 is shown in accordance with an embodiment of the present invention. Advantageously, the robotic tool system 100 utilizes intelligent 3D modeling to analyze the item 105 and avoid obstacles 110.

[0083] The robotic tool system 100 includes a scanner 115 coupled to a controller (not shown), the scanner 115 being configured to generate a three-dimensional model of the article 105 and its surroundings, including one or more obstacles 110 .

[0084] The scanner 115 may be any suitable type of scanner, and in one embodiment, the scanner 115 is a laser scanner configured to generate a point cloud of the object 105 and the obstacle 110. The scanner is coupled to the robotic arm 120 and may be configured to move around the object 105 and the obstacle 110 to capture data from different angles and / or to avoid occlusions.

[0085] The point cloud generated by the scanner 115 and / or controller may then be pre-processed to remove noise, stitch point cloud data from multiple reference points, or perform any other processing or filtering of the data. In one embodiment, the scanner 115 may combine several scanning technologies (e.g., optical and laser scanning data) to generate a point cloud.

[0086] The controller generates a cellular environment corresponding to the objects 105 and the obstacles 110 (including their surroundings). The cellular environment can be oversized to include the objects 105, the obstacles 110, and the space surrounding them so that the tool 125 coupled to the robotic arm 125 can interact with the objects 105 while remaining within the cellular environment.

[0087] Specifically, the lattice environment includes a plurality of cells, each of which is cubic in shape and uniform in size. Thus, the cells define a coordinate system, wherein each cell is associated with a well-defined space and can be associated with data.

[0088] Figure 2 An exemplary cellular environment 200 is shown in accordance with an embodiment of the present invention. The cellular environment 200 includes a plurality of cells 205, each uniformly sized and cube-shaped.

[0089] Each cell 205 is identifiable by coordinates (x, y, z) that define the relative position of each cell 205 in the lattice environment 200. Each axis of the coordinate system (x, y, and z) is perpendicular to the face of each cell 205. In addition, each cell can be associated with a rotation axis (such as roll, pitch, and yaw) to enable six degrees of freedom to be applied to the tool.

[0090] In the example lattice environment 200, the leftmost, bottommost, and frontmost cell 205 is the source cell (x=1, y=1, z=1), but those skilled in the art will readily appreciate that any source point or reference point may be used.

[0091] In an alternative embodiment, the lattice environment is defined by cells of uniform size but having another shape (e.g., spherical).This configuration is particularly useful when 6 or more degrees of freedom are associated with each cell, as it enables the faces of each cell to be associated with a degree of freedom.

[0092] Those skilled in the art will appreciate that the term spherical does not necessarily strictly mean a spherical shape, but may represent a spherical shape in the context of a cellular environment. As an illustrative example, the shape of a spherical unit cell may actually be a dodecahedron.

[0093] The point cloud created by the scanner is then overlaid on the gridded environment 200, and the cells are classified based on the point cloud. Specifically, each point of the point cloud is assigned to a cell 205, and the cells corresponding to the points of the point cloud are classified as "occupied". However, as outlined below, any classification may be used. In some embodiments, the classification of a cell may include whether the cell is empty or filled, in which case whether it is filled with items or obstacles, details of the material type / texture, cell welding parameters, processing performed on the cell, or any suitable data. In addition, such classification may be provided with respect to cell clusters.

[0094] Similarly, cells may be associated with a strength, like a magnetic field around an object, but related to whether the cell is solid, rather than magnetic. For example, a cell completely filled with solid would have a strength field of 1, and for softer materials, a strength field less than 1 (e.g., 0.9) could be used. Additionally, such strength data could be provided to illustrate how far each cell is from a solid.

[0095] The resolution of the lattice environment 200 may correspond to the size of the scanned data so that adjacent points in the point cloud form a continuous arrangement of cells 205 in the lattice environment 200. In addition, when overlaid onto the lattice environment 200, the point cloud may be filtered, resampled, or otherwise processed.

[0096] Once the cells 205 are initially classified, they may be processed and a score may be generated for each cell based on one or more of its characteristics.

[0097] In one embodiment, a neighborhood pattern is determined for each cell 205 based on characteristics of the immediate neighborhood of the cell 205. The neighborhood pattern may involve binary characteristics, such as whether the neighboring cells are occupied, or any other suitable characteristic of the cell 205.

[0098] Each cell 205 includes 26 neighbors (i.e., a 3×3×3 cube centered at the cell 205), except for cells on the outer edges of the lattice environment 200. A pattern can be calculated for each cell based on the data of each of the 26 surrounding cells, just like a spatial filter (also called a kernel or mask) in image processing.

[0099] The 26 neighbors of a cell with coordinates X,Y,Z are defined by the following coordinates: (X-1,Y-1,Z+1), (X,Y-1,Z+1), (X+1,Y-1,Z+1), (X-1,Y-1,Z), (X,Y-1,Z), (X+1,Y-1,Z), (X-1,Y-1,Z-1), (X,Y-1,Z-1), (X+1,Y-1,Z-1), (X+1,Y-1,Z-1), (X-1,Y,Z+1), (X+1,Y-1,Z-1), (X-1,Y,Z+1), (X+1 The cells are numbered 1-26.

[0100] Figure 3 A 3x3x3 cell neighborhood 300 including cell 205 (marked with an X) and its neighboring cells including their numbers are shown.

[0101] For clarity, the cell neighborhood 300 is shown broken down into three vertical slices 300a, 300b, 300c.

[0102] The numbering enables each cell to be addressed unambiguously and a code or sequence generated. It will be appreciated by those skilled in the art that the numbering is merely a label and any numbering of adjacent cells or any other means of identifying a cell may be used.

[0103] In one embodiment, the neighboring pattern may be defined according to the binary characteristics of each cell, such as the occupancy of each neighboring cell. In particular, each cell may be associated with a pattern corresponding to the occupied surrounding cells.

[0104] As an illustrative example, if the neighboring cell with Y=y is occupied, then use Figure 3 The nomenclature proximity pattern would be: 10, 11, 12, 13, 14, 15, 16, 17 (ie, all cells in slice 300b). This generally corresponds to a vertical surface perpendicular to the y-axis.

[0105] Importantly, neighboring cells 205 that share a common proximity pattern (or substantially the same proximity pattern) typically form part of a common surface. This applies not only to surfaces that are perpendicular to the axis of the coordinate system, but to any suitable surface.

[0106] Similarly, adjacent cells having substantially different neighboring patterns generally form part of an edge or irregularity of a surface.

[0107] Figure 4 An example of a slice 200a of a cellular environment 200 is shown, wherein an occupied cell 205a is shown as having a shading, corresponding to a square planar surface of an item 105 .

[0108] The central occupied cells 205a all have the same neighboring pattern (i.e., 10, 11, 12, 13, 14, 15, 16, 17), while the top occupied cells 205a have a different pattern (i.e., 13, 14, 15, 16, 17). Similarly, the bottom occupied cell 205a has a different pattern from the leftmost and rightmost occupied cells 205a (i.e., 10, 11, 12, 13, 14).

[0109] As a result, comparing the neighboring patterns of occupied cells 205a provides a simple way to identify surfaces (especially uniform surfaces) and edges.

[0110] Although the examples shown include vertical surfaces for clarity, the skilled person will readily appreciate that these surfaces may take any form. As an illustrative example, a rear inclined surface at an angle of about 45 degrees may have a neighboring pattern of (7, 8, 9, 13, 14, 18, 19, 20).

[0111] Any suitable metric may be used to compare proximity patterns, including identical matches, matching proximity patterns above a certain threshold (e.g., above 75%), or a smarter comparison of similarity (e.g., where positions 10 and 18 are considered adjacent to each other and therefore closer to each other than 15 and 18 are).

[0112] While the above example is shown with a neighborhood size of 3x3x3 (26 neighbors plus the cell), in other embodiments, different neighborhood sizes (eg, 5x5x5) and even non-cubic neighborhood shapes may be used.

[0113] Additionally, the system 100 is particularly useful in obstacle avoidance and path generation, particularly when the robotic arm 120 and tool 125 are interacting with the item 105. In this case, a target point (end point) can be identified and the system 100 generates a path for the robotic arm 120 from its current position to the end point.

[0114] To accomplish this or for other types of calculations, in some embodiments, one or more proximity scores are calculated based on characteristics of neighboring cells, including relevance to a target cell.

[0115] An example of such a score is the neighborhood level score, which involves the distance of the cell from the target in terms of linear steps.

[0116] Proximity scoring is performed by starting at the goal cell and setting all non-obstruction cells adjacent to the goal cell to a value of 1, since it is possible to move from each of these neighboring cells to the goal cell in a linear fashion in one step.

[0117] This process is repeated in cells extending linearly outward from the target cell until an obstacle is reached (or the edge of the lattice environment 200 is reached).

[0118] Once all cells are scored in this manner, any unscored neighbors of the scored cell are scored with a value of '2', which is repeated in cells extending linearly outward until an obstacle is reached (or the edge of the cellular environment 200 is reached).

[0119] This process is repeated iteratively, increasing the score by 1 each time, until all cells have been scored.

[0120] In addition to storing the score associated with each cell, details of the neighboring cells through which the score was provided are also stored. This makes it easy to identify not only the number of linear moves each cell made from the target, but also the direction in which these moves were initiated.

[0121] For clarity, examples of proximity scores are described below in a two-dimensional environment. Those skilled in the art will appreciate that the teachings can be equally applied to a three-dimensional environment.

[0122] Each cell 205 on the planar surface has 8 neighbors (ie, a 3×3 square centered at the cell 205 ), which are identified by numbers 1-8, respectively.

[0123] Figure 5 An example of a planar surface 500 is shown including a cell 205 (marked with an X) and its neighboring cells (including their numbers). The numbering enables each of the neighboring cells to be unambiguously addressed relative to the cell 205, and an associated code to be generated.

[0124] As above, it will be appreciated by those skilled in the art that the numbers are merely labels and any number of adjacent cells may be used.

[0125] Figure 6 An example of a 2D gridded environment 200b is shown, in which the occupied cells 205a are shaded, the neighbor scores are shown in the upper left corner of each cell, and the Figure 5 The nomenclature also stores the details of the neighboring cells through which the scores are provided.

[0126] Initially, when cells G1, H1, F2, H2, F3, G3, and H3 are directly close to the target (T), they are given a score of 1, and movement from these cells to the target can be made in a single linear move. Cell F2 gets its score from the target (which is located at position 5 relative to F2).

[0127] Cell E2 (neighbor number 4 of cell F2) also receives a score of 1, as do cells D2 and C2 (all neighbor number 4 in order from the target).

[0128] Level 2 scoring begins once all cells that qualify for score 1 have been assigned. For example, cell D3 gets its score (score 2) from cells C2, D2, and E2 (at positions 1, 2, and 3 for D3).

[0129] The process is then repeated for the remaining cells until all unoccupied cells have been assigned scores.

[0130] In use, the scores and details of the neighboring cells provided with scores can be used to provide an efficient path from any cell to the target T in as few linear steps as possible.

[0131] As an illustrative example, cell B3 has a score of 2, indicating that the target can be reached in two linear moves, and cell B3 has details of adjacent cell '3' (upper right). Thus, the first linear move is diagonally upward until the score changes, which occurs at cell C2. Cell C2 has a score of 1 and details of adjacent cell '5' (right), indicating that the target can be reached in a single linear move to the right.

[0132] Another example of a score is a target distance score, which relates to the distance (ie, absolute distance) of a cell from a target in a plurality of cells.

[0133] This is done by starting at the target cell 205 and setting all non-obstruction cells directly above, below, or next to the target cell 205 to a value of 1 and any cells diagonally adjacent to the target cell to a value of (or its approximation) to perform target distance scoring.

[0134] By scoring each non-obstruction cell directly above, below, or next to cell 205 as value K+1, and scoring any cell diagonally adjacent to the target cell as (or its approximation), and repeat this score in adjacent cells.

[0135] If a cell receives multiple different scores from different neighbors, the minimum score is recorded (ie, the minimum score overrides the larger scores).

[0136] Once all cells have been scored in this manner, the process is repeated for neighboring cells within those cells, and so on, until all cells have been scored.

[0137] Figure 7 An example of a 2D gridded environment 200c is shown, in which the occupied cells 205a are shaded, the target distance score is shown at the top of each cell, and the distance score is shown using Figure 5 Nomenclature, at the bottom of each cell are stored the details of the neighboring cells by the scores being provided.

[0138] Initially, cells G1, F2, H2, and G3 are assigned a score of 1 because they are directly above, below, or next to the target, and cells H1, F3, and H3 are assigned a score of Because they are positioned diagonally relative to the target. Each cell can be given a score for each orientation associated with that cell. The process is then repeated on cells adjacent to the cell just scored, and so on, until all qualifying cells have been scored.

[0139] For example, cell E1 is scored as It is the fraction of F2 (i.e., 1) plus the diagonal distance from F2 (i.e., ). For cell E1, F2 is in position 8, therefore the details of position 8 are stored relative to E1.

[0140] In use, the scores and details of neighboring cells provided with scores can be used to provide the shortest path from any cell to the target.

[0141] As an illustrative example, cell E1 has the fraction Indicate target Steps away, cell E1 has details of neighboring cell '8' (below right). Thus, the first move is diagonally downward (i.e., to cell F2). Cell F2 has a score of 1 and details of neighboring cell '5' (right), indicating that the target can be reached by moving a single step to the right.

[0142] Yet another example of a score is an obstacle distance score, which relates to the distance (ie, absolute distance) of a cell from an obstacle in a plurality of cells.

[0143] Obstacle distance scoring is performed by starting at the obstacle and setting all non-obstacle cells directly adjacent to the obstacle to a value of 1. This process is then repeated for all unscored neighbors, increasing the score by 1 each time.

[0144] Figure 8 An example of a 2D lattice environment 200d is shown, in which occupied cells 205a are shaded and an obstacle distance score is shown in each cell.

[0145] For example, cell B4 is assigned a value of 2, indicating that it is two steps away from the obstacle. At B2 and D4, the distances to the obstacle are equal.

[0146] Since the path data outlined above is generated for each cell, if the starting point changes, the data does not need to be recalculated, which adds flexibility to the system, especially when performing multiple tasks.

[0147] In some embodiments, multiple scores may be calculated for each of the multiple cells and the scores may be combined. As an illustrative example, if a short path is desired, but the distance to obstacles is also maintained, a combination of a target distance score and an obstacle distance score may be used.

[0148] In some embodiments, a combination of the above scores is generated simultaneously. As an illustrative example, an obstacle distance score may be calculated first and considered when calculating the proximity score. In one example, the proximity score may be calculated without considering the additional requirement of cells having a predefined distance (e.g., 1 or 2) or less to an obstacle.

[0149] In one embodiment, cells are classified according to one or more of the scores outlined above, proximity patterns, or derivatives thereof.

[0150] In some embodiments, each cell can be classified in two or more ways.

[0151] The classification and / or score may be used when interacting with item 105. As an illustrative example, tool 125 may include a welding tool, and the score and classification are used to determine a path from the location of tool 125 to item 105 and a welding path along item 105. This may include considering a relevant area and / or other constraints for welding.

[0152] A plurality of different tools 125 may be provided in an interchangeable manner, thereby enabling selection of an appropriate tool. In one embodiment, a welding tool may be used initially, followed by one or more of grinding, deburring, polishing, and spraying.

[0153] Fig. 9 A robotic tool method 900 is shown in accordance with an embodiment of the present invention. The method 900 may be similar or identical to the method used by the system 100.

[0154] At step 905, spatial data of the object and its surroundings is captured, for example, using a laser scanner. The spatial data may include a point cloud.

[0155] At step 910, spatial data is assigned to cells of the lattice space. The lattice space may include a plurality of equally sized cubic cells corresponding to locations in the three-dimensional space.

[0156] At step 915, one or more characteristics of the cell are determined based on the data of the neighboring cells. For example, the characteristics may include whether the cell is occupied by an item, whether the cell includes an obstacle, or the proximity pattern of the cell.

[0157] At step 920, a robotic tool (such as a robotic welder) is configured to operate on an object based at least in part on one or more characteristics. This may include avoiding obstacles, determining an efficient path, or any other characteristics.

[0158] In some embodiments, markings may be placed on the object to help the system identify paths thereon. An example of such a marking is paint.

[0159] In use, a user may intentionally mark an item, for example with coloured paint, to an approximate path, for example for welding. The paint is thus a form of input to the system to define the path to be welded or otherwise processed.

[0160] The system can scan the item to create a point cloud of the item, detect markings (e.g., paint), and identify points in the point cloud associated with the markings.

[0161] Thus, in some embodiments, an item (e.g., a part) can be scanned in a single scan to generate a mesh or point cloud with markers. The system can then run a welding (or other) simulation, followed by a cold run of the welding (or other) tool. Finally, a hot run can be performed on the part. This process can take a very short time (<10 minutes for small parts), which is significantly faster than prior art systems. In addition, once performed once, the scanned part can be used to generate additional items, scan fixtures, and even adjust for obstacles.

[0162] The system can then filter or process these points to generate a line (path). This configuration is necessary because the paint is usually much thicker than the resolution of the scanner, so many points together will correspond to a mark.

[0163] The line can be generated based on the average distance to other points.

[0164] The point cloud may be filtered or processed iteratively. As an illustrative example, the process of selecting a path from the marked points may be performed iteratively. For example, the process may include averaging the points within a certain radius and finding the center of the line using the point as a new starting point.

[0165] The system can also be configured to smooth out the line, for example by penalizing sharp turns.

[0166] The system may be configured to receive a model associated with an object. The model may include a CAD model. As will be readily appreciated, the object may not conform precisely to the object due to wide tolerances or other reasons.

[0167] The system can align the point cloud with the model. This can be achieved by aligning features (e.g., edges and / or surfaces) of the scan data with features (edges and / or surfaces) of the model.

[0168] The system can use one or more bounding boxes to align features of the scan data with the CAD model. The bounding boxes can be used to search for corresponding points in the point cloud.

[0169] The system can measure the deviation between the object and the model and refine the model based on the deviation. The refined model advantageously matches the object better than the original model.

[0170] The system can determine if the deviation is outside of a threshold and issue an alert or warning to the user.

[0171] In many cases, only a portion of an object is described by the scanned point cloud data, such as in connection with large objects. In such cases, matching the point cloud to a model can be problematic using existing methods, especially if many models are not associated with the partial point cloud data.

[0172] In this case, a matching process may be initially performed to provide a rough position of the object with reference to the scan data.

[0173] A comparison is then made between the scan data and the model, and points from the model are selected based on their distance to the scan data.

[0174] This essentially creates a new partial model of the object that corresponds only to the point cloud (and not the areas of the model seen in the point cloud) to simplify and improve alignment. The new partial model may comprise a "slice" of the model.

[0175] The new part model is then referenced to the scan data for matching (alignment), which is simpler because irrelevant points from the model are removed, meaning less data needs to be processed, and more accurate because the effects on the model from potentially irrelevant points are removed.

[0176] It would further be desirable for the system to assist in generating paths, such as for welding, rather than the user having these inputs manually.

[0177] Different types of edge detection can be used to assist in path generation, particularly in relation to welding.

[0178] The system can identify planes in the scanned point cloud data, extrapolate planes, and identify intersections between planes. Then, a line (path) can be determined based on the intersection of the planes. This line is particularly useful when welding objects at their edges.

[0179] In particular, the system can segment the point cloud data into blobs based on the direction of the point normals to identify planes. An average plane can be generated through each of the blobs.

[0180] For each set of planes, the points where the planes intersect can be identified and filtered to create lines (paths).

[0181] This feature can reduce the amount of human interaction required to establish paths for the system.

[0182] In some embodiments, the generation of such lines (corresponding to edges) is used to align the object with the model. In this case, the lines can be regarded as edges and aligned with the edges of the model.

[0183] The system may provide a user interface to the user, thereby enabling a path to be defined thereon. When the path can be defined, the user interface may visualize the object.

[0184] As an illustrative example, a visualization of an object may be provided on which a user can define a path by clicking on points of the object.

[0185] In addition, the teaching disclosed above can be used to generate or define a fixture for welding (or otherwise processing) an article in a repeatable manner. For example, the fixture and the article can be scanned simultaneously, thereby achieving the reconstruction of the fixture and the reconstruction of the article using the fixture.

[0186] In addition to assisting in the processing of an object, the system may scan the object after one or more tasks have been performed on the object for quality control.

[0187] The system can determine whether a point is within a predefined threshold. Local averaging can be performed to avoid quality control failures based on outliers in the scan data.

[0188] The system can be configured to visualize the disparity between the scan data and a predefined model. The disparity can be shown as color coding, like a heat map. This provides a clear, human-readable visualization of the deviations in the object.

[0189] In some embodiments, metrics are provided to allow the robot to automatically perform quality control of its work. For example, along with the CAD model, set tolerances can be provided for the object to determine the accuracy of the artifact relative to the CAD (ideal) model. Based on the tolerances, the final object may pass or fail the final inspection.

[0190] Furthermore, robotic welding machines and the like operate at high levels of precision and, without the need for compensation methods, typically require that any components presented for their work also be at high levels of precision.

[0191] For parts with wide tolerances, it may be beneficial to deform and align the model to the point cloud to essentially generate a new model corresponding to the object using the above method or other methods.

[0192] This matching and deformation can be performed on the surface of the object alone. In this case, for each point on the model surface, the corresponding point in the point cloud is identified and moved. The model is then replaced with the new deviated data points.

[0193] Advantageously, embodiments of the present invention enable the creation and analysis of 3D models of physical objects and their surroundings in a simple and efficient manner, for example to avoid obstacles. These embodiments have a wide range of applications, but are particularly suitable for use when interacting with physical objects using robotic tools (such as robotic welding, spraying, grinding, polishing, deburring, and sandblasting tools). Since path data is generated for each cell, the data does not need to be recalculated if the starting point changes, which adds flexibility to the system, especially when performing multiple tasks.

[0194] In this specification and claims (if any), the word 'comprise' and its derivatives (including 'comprising' and 'comprising') include each of the recited integers but do not exclude the inclusion of one or more additional integers.

[0195] References throughout this specification to 'one embodiment' or 'an embodiment' mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, the phrases "in one embodiment" or "in an embodiment" appearing in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more combinations.

[0196] According to the regulations, the present invention has been described more or less in language specific to structural or methodological features. It should be understood that the present invention is not limited to the specific features shown or described, because the means described herein include preferred forms for making the present invention effective. Therefore, the present invention is protected in any form or modification thereof within the proper scope of the attached claims (if any) appropriately interpreted by those skilled in the art.

Claims

1. A robot tool system, comprising: a scanner configured to capture spatial data of the object and its surroundings; The processor is configured as: assigning the spatial data to cells of the formatted space; determining one or more characteristics of the cell based on data from neighboring cells; as well as The robotic tool is configured to operate on the object based at least in part on the one or more characteristics.

2. The robot tool system according to claim 1, wherein: The lattice space includes a plurality of cells of uniform size.

3. The robotic tool system according to claim 2, wherein: The lattice space is three-dimensional, with each of the cells corresponding to a position and optionally an orientation in the three-dimensional space.

4. The robot tool system according to claim 3, wherein: The unit cell is in the shape of a cube or a sphere.

5. The robotic tool system according to claim 1, wherein: The scanner is configured to generate the spatial data in the form of a point cloud.

6. The robotic tool system according to claim 1, wherein: A first characteristic of one or more of the cells is initially determined, and a second characteristic of the one or more cells is determined based on the first characteristic.

7. The robotic tool system according to claim 6, wherein: The one or more characteristics of the cell include a neighborhood pattern including a spatial pattern of characteristics of the neighboring cells.

8. The robotic tool system according to claim 7, wherein: The proximity pattern may be represented by a code or sequence indicating which of the neighboring cells have a characteristic.

9. The robotic tool system according to claim 7, wherein: The processor is configured to compare the proximity patterns of the neighboring cells to determine their pattern.

10. The robotic tool system according to claim 7, wherein: The proximity model is three-dimensional.

11. The robotic tool system according to claim 1, wherein: The one or more characteristics of the cell include a score, such as a distance score.

12. The robotic tool system according to claim 12, wherein: The score comprises a weighted combination of two or more scores associated with different characteristics.

13. The robotic tool system of claim 1, further configured to recognize a marking on the object and to configure the robotic tool to operate on the object based at least in part on the recognized marking.

14. The robotic tool system according to claim 13, wherein: The markings include colored markings.

15. The robotic tool system of claim 13, configured to scan the object, detect the markers, and generate a point cloud, the point cloud including the detected markers, wherein: The point cloud including the detected markers is filtered or processed to generate a line (path).

16. The robotic tool system of claim 1, further configured to receive a model associated with the object and align the spatial data with the model.

17. The robotic tool system of claim 16, configured to align features (eg, edges and / or surfaces) of the spatial data with features (eg, edges and / or surfaces) of the model.

18. The robotic tool system of claim 16, configured to measure deviations between the object and the model, and to refine the model according to the deviations.

19. The robotic tool system of claim 16, configured to align the spatial data with the model using partial model data in the form of slices or portions of the model.

20. The robotic tool system of claim 1, further configured to identify planes in the scanned spatial data, extrapolate the planes, and identify intersection points between the extrapolated planes as generated lines (paths) based on intersections of the planes.

21. The robotic tool system of claim 1, further configured to scan the object after performing one or more tasks on the object and compare the object to a predefined model for quality control.

22. The robotic tool system of claim 21, further configured to visualize a parallax between the scanned data and a predefined model.

23. A robotic tool method comprising: Capture spatial data of objects and their surroundings; assigning the spatial data to cells of the formatted space; determining one or more characteristics of the cell based on data from neighboring cells; as well as The robotic tool is configured to operate on the object based at least in part on the one or more characteristics.

24. A 3D modeling system comprising: a scanner configured to capture spatial data of the object and its surroundings; as well as The processor is configured as: assigning the spatial data to cells of the formatted space; One or more characteristics of the cell are determined based on data from neighboring cells.

25. A 3D modeling method, comprising: Capture spatial data of objects and their surroundings; assigning the spatial data to cells of the formatted space; as well as One or more characteristics of the cell are determined based on data from neighboring cells.

Images