A trajectory generation method, apparatus, device, and medium
By generating a first mesh model from a local image of the object to be processed and fusing it with a digital model, a three-dimensional image is generated and converted into a second mesh model. This solves the problem of inaccurate trajectory generation in the prior art and achieves higher quality operation trajectory generation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHENGDU CRP ROBOT TECH CO LTD
- Filing Date
- 2022-12-22
- Publication Date
- 2026-05-19
AI Technical Summary
Existing trajectory generation methods cannot accurately obtain the operation trajectory of the object to be processed, which affects the operation quality.
By preprocessing a local image of the object to be processed, a first mesh model is generated and fused with a digital model to obtain the location information of the area to be processed. Then, a three-dimensional image is generated based on the preset path and location information and converted into a second mesh model. Finally, the operation trajectory is obtained based on the features of the theoretical operation trajectory and the second mesh model and optimized.
It enables more accurate operation trajectory generation and improves operation quality.
Smart Images
Figure CN116188526B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of trajectory generation technology, and in particular to trajectory generation methods, apparatus, equipment and media. Background Technology
[0002] In the field of intelligent manufacturing based on industrial robots, there is an increasing demand for high-precision, high-efficiency, high-reliability and highly automated operation trajectory generation methods. For example, industrial manufacturing processes such as welding, painting, gluing, and grinding all have high requirements for robot operation trajectories. In order to generate operation trajectories more efficiently, it is necessary to design specialized trajectory generation methods.
[0003] In the existing technology, the trajectory generation of the object to be processed mainly relies on manual work based on personal experience. As a result, the trajectory generation method in the existing technology cannot accurately obtain the operation trajectory of the object to be processed, thus affecting the operation quality of the object to be processed. Summary of the Invention
[0004] The main purpose of this application is to provide a trajectory generation method, apparatus, device and medium, which aims to solve the technical problem that the existing trajectory generation methods cannot accurately obtain the operation trajectory of the object to be processed, thereby affecting the operation quality of the object to be processed.
[0005] To achieve the above objectives, the first aspect of this application provides a trajectory generation method, the method comprising:
[0006] A local image of the object to be processed is preprocessed to obtain a first mesh model of the local image;
[0007] The first mesh model is fused with the digital model of the object to be processed to obtain the location information of the processing area of the object to be processed; wherein, the digital model includes the processing area of the object to be processed;
[0008] Based on a preset path and the location information of the area to be processed, a three-dimensional image of the area to be processed is obtained; wherein, the preset path is obtained based on the digital model;
[0009] The three-dimensional image of the area to be processed is converted into a second mesh model;
[0010] Based on the characteristics of the theoretical trajectory to be operated on the object to be processed and the processing area of the second mesh model, the operation trajectory of the object to be processed is obtained; wherein, the characteristics of the theoretical trajectory to be operated on the object to be processed are obtained based on the digital model.
[0011] Optionally, before the step of obtaining the operation trajectory of the object to be processed based on the features of the theoretical operation trajectory of the object to be processed and the processing area of the second mesh model, the method further includes:
[0012] Based on the processing area of the second grid model and the processing area of the digital model, determine whether the processing area of the object to be processed meets the processing conditions;
[0013] The process of obtaining the operation trajectory of the object to be processed based on the features of the theoretical trajectory to be processed of the object and the processing area of the second mesh model includes:
[0014] When the processing area of the object to be processed meets the processing conditions, the operation trajectory of the object to be processed is obtained based on the characteristics of the theoretical operation trajectory of the object to be processed and the processing area of the second mesh model.
[0015] Optionally, determining whether the processing area of the object to be processed meets the processing conditions based on the processing area of the second mesh model and the processing area of the digital model includes:
[0016] By comparing the area to be processed in the second mesh model with the area to be processed in the digital model, the error value between the area to be processed in the second mesh model and the area to be processed in the digital model is obtained.
[0017] Based on the relationship between the error threshold and the error value, it is determined whether the processing area of the object to be processed meets the processing conditions.
[0018] Optionally, after determining whether the processing area of the object to be processed meets the processing conditions based on the processing area of the second mesh model and the processing area of the digital model, the method further includes:
[0019] If the processing area of the object to be processed does not meet the processing conditions, a prompt message is sent; wherein, the prompt message includes the positional error information of the processing area of the object to be processed.
[0020] Optionally, obtaining the operation trajectory of the object to be processed based on the features of the theoretical trajectory to be operated on and the processing area of the second mesh model includes:
[0021] Based on the characteristics of the theoretical operation trajectory of the object to be processed, the characteristics of the operation trajectory of the processing area in the second mesh model are obtained.
[0022] Based on the characteristics of the operation trajectory in the area to be processed, the operation trajectory of the object to be processed is obtained.
[0023] Optionally, after the step of obtaining the operation trajectory of the object to be processed based on the features of the theoretical operation trajectory of the object to be processed and the processing area of the second mesh model, the method further includes:
[0024] The second mesh model of the operation trajectory neighborhood of the object to be processed is smoothed and repaired.
[0025] Based on the second mesh model after smoothing and mesh repair, trajectory control points are extracted on the operation trajectory; wherein, the trajectory control points include mesh vertices on the operation trajectory;
[0026] Based on the curve types and trajectory control points in the digital model, the operation trajectory of the object to be processed is optimized; wherein, the curves include Bézier curves and B-spline curves.
[0027] Optionally, before the step of preprocessing the local image of the object to be processed to obtain a first mesh model of the local image, the method further includes:
[0028] Load the digital model of the object to be processed; wherein the digital model includes the geometric and topological information of the object to be processed.
[0029] Adjust the position of the object to be processed on the worktable so that the position of the object to be processed on the worktable is the same as the position of the digital model of the object to be processed.
[0030] Based on the object to be processed after its position has been adjusted, a partial image of the object to be processed is obtained.
[0031] Secondly, this application provides a trajectory generation apparatus, the apparatus comprising:
[0032] The first model is used to preprocess a local image of the object to be processed in order to obtain a first mesh model of the local image.
[0033] The second model is used to fuse the first mesh model with the digital model of the object to be processed to obtain the location information of the processing area of the object to be processed; wherein, the digital model includes the processing area of the object to be processed;
[0034] The third model is used to obtain a three-dimensional image of the area to be processed based on a preset path and the location information of the area to be processed; wherein the preset path is obtained based on the digital model;
[0035] A conversion model is used to convert the three-dimensional image of the area to be processed into a second mesh model;
[0036] The fourth model is used to obtain the operation trajectory of the object to be processed based on the features of the theoretical operation trajectory of the object to be processed and the processing area of the second grid model; wherein the features of the theoretical operation trajectory of the object to be processed are obtained based on the digital model.
[0037] Thirdly, this application provides a computer device including a memory and a processor, wherein the memory stores a computer program and the processor executes the computer program to implement the methods described in the embodiments.
[0038] Fourthly, this application provides a computer-readable storage medium storing a computer program, on which a processor executes the computer program to implement the methods described in the embodiments.
[0039] Through the above technical solution, this application has at least the following beneficial effects:
[0040] This application proposes a trajectory generation method, apparatus, device, and medium, comprising: firstly, preprocessing a local image of an object to be processed to obtain a first mesh model of the local image; then, fusing the first mesh model with a digital model of the object to be processed to obtain position information of a processing area of the object; wherein the digital model includes the processing area of the object; then, obtaining a three-dimensional image of the processing area based on a preset path and the position information of the processing area; wherein the preset path is obtained based on the digital model; then, converting the three-dimensional image of the processing area into a second mesh model; finally, obtaining the operation trajectory of the object to be processed based on the features of the theoretical operation trajectory of the object and the processing area of the second mesh model; wherein the features of the theoretical operation trajectory of the object are obtained based on the digital model.
[0041] That is, when it is necessary to obtain the operation trajectory of the object to be processed, a local image of the object to be processed is first acquired. Then, the local image of the object to be processed is processed and converted into a first mesh model. The first mesh model is then fused with the digital model of the object to be processed. After fusion, the approximate location of the processing area of the object to be processed is predicted. Then, based on the approximate location of the processing area and the preset path from the first location to the second location, a three-dimensional image of the processing area is obtained. Specifically, a local image of the object to be processed can be obtained from the first location, and a three-dimensional image of the object to be processed can be obtained from the second location. Then, the three-dimensional image of the processing area is converted into a second mesh model. Then, based on the characteristics of the theoretical operation trajectory and the processing area of the second mesh model, the preliminary operation trajectory of the object to be processed is obtained. Finally, the preliminary operation trajectory is optimized to obtain the operation trajectory of the object to be processed.
[0042] In other words, this application converts a local image of the object to be processed into a first mesh model, converts a 3D image including the region to be processed into a second mesh model, and then fuses the first mesh model of the local image with the digital model of the object to be processed. Because the mesh model itself is holistic, each triangle of the mesh model can be optimized from the perspective of the whole model, thereby obtaining a globally stable and accurately positioned geometric location. The digital model provides the basic constraints for the optimization of the mesh model, further guiding the specific direction of the optimization. If there is only a digital model and no mesh model, there is obviously only the theoretical position of the trajectory, which cannot achieve accurate trajectory. If there is only a mesh model and no digital model, the constraints of the mesh model optimization are weak, lacking much prior information, and cannot optimize towards the theoretical position of the trajectory, thus failing to obtain an accurate operation trajectory. Therefore, this scheme can obtain the operation trajectory of the object to be processed more accurately, thereby improving the operation quality of the object. Attached Figure Description
[0043] Figure 1 This is a schematic diagram of the computer device structure for the hardware operating environment involved in the embodiments of this application;
[0044] Figure 2 A flowchart illustrating a trajectory generation method provided in this application embodiment;
[0045] Figure 3 This is a flowchart illustrating a specific execution method for step S12 provided in this embodiment;
[0046] Figure 4 This is a flowchart illustrating a specific execution method for step S13 provided in this embodiment;
[0047] Figure 5 This is a schematic diagram of a trajectory generation device provided in an embodiment of this application.
[0048] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0049] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.
[0050] In the field of intelligent manufacturing based on industrial robots, there is an increasing demand for high-precision, high-efficiency, high-reliability, and highly automated target operation trajectory generation methods. For example, industrial manufacturing processes such as welding, painting, gluing, and grinding all have high requirements for the target operation trajectory of robots. Current trajectory generation methods suffer from the following main problems: They require highly skilled and experienced workers; the process is tedious and complex, requiring significant manual intervention; for similar workpieces, the same trajectory generation process must be completed before each welding operation; they demand precise workpiece positioning, requiring fixtures to hold the workpiece in place; they cannot actively identify workpiece errors and structural anomalies, necessitating manual inspection; when using 3D imaging equipment to automatically generate trajectories, the location of the photograph must be manually specified to determine the specific position of the trajectory; trajectory features must be manually specified rather than automatically obtained from the digital model (the local geometric features of the trajectory are automatically generated from the specified region of the operation trajectory in the specified model according to rules); the uneven density of 3D point clouds at varying distances leads to inconsistent trajectory positioning accuracy; filtering methods based on 3D point clouds, while smoothing the data, can also cause edge degradation, resulting in trajectory positioning errors; and they fail to effectively integrate the real and virtual aspects of the workpiece and the 3D digital model, hindering the interactive presentation of the relationship between the actual workpiece information and the 3D digital model.
[0051] Currently, the mainstream methods for generating target operation trajectories include: Manual teaching-based methods, such as in robotic welding, where the start and end points of welding are input into the robot system via human interaction to generate the target welding trajectory. This method requires high professional skills from on-site workers and cannot accurately teach complex target operation trajectories. In scenarios with significant uncertainty in boundary conditions, teaching is required before each operation, resulting in a large amount of repetitive work. Offline programming-based methods, such as in offline programming welding, require programming based on the welding points and trajectories in the digital model of the workpiece to determine the robot's target operation trajectory. This method requires high machining accuracy of the workpiece and precise positioning of the workpiece on the worktable. Point cloud-based 3D vision methods, such as point cloud-based target trajectory generation methods in grinding, utilize... 3D imaging equipment directly obtains 3D point clouds of the welded workpiece surface. The robot's target trajectory is then obtained through algorithmic processing of the point cloud. However, the discrete, disordered, unevenly distributed density, and unstructured nature of point cloud data itself makes the target trajectory generation algorithm highly complex, and optimizing for high-precision, structured trajectories difficult. Interactive trajectory generation methods based on point clouds and digital models, such as directly acquiring 3D point clouds of the welded workpiece surface using 3D imaging equipment during welding, registering the 3D point cloud and digital model to unify and align them in a coordinate system, and then interactively selecting the robot's target trajectory, increase the reliability of the target trajectory by manually eliminating erroneous trajectories through interactive selection. However, these methods still cannot overcome the high algorithmic complexity and difficulty in generating high-precision trajectories caused by the inherent defects of point cloud data. In summary, current trajectory generation methods cannot accurately obtain the operation trajectory of the object being processed, thus affecting the quality of the operation.
[0052] To address the aforementioned technical problems, this application provides a trajectory generation method, apparatus, device, and medium. Before introducing the specific technical solutions of this application, the hardware operating environment involved in the embodiments of this application will be described first.
[0053] Reference Figure 1 , Figure 1 This is a schematic diagram of the computer device structure of the hardware operating environment involved in the embodiments of this application.
[0054] like Figure 1As shown, the computer device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen and an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.
[0055] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the computer device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.
[0056] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a data storage module, a network communication module, a user interface module, and electronic programs.
[0057] exist Figure 1 In the computer device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the computer device of this application can be set in the computer device, and the computer device calls the trajectory generation device stored in the memory 1005 through the processor 1001 and executes the trajectory generation method provided in the embodiment of this application.
[0058] Reference Figure 2 Based on the hardware environment of the foregoing embodiments, embodiments of this application provide a trajectory generation method, the method comprising:
[0059] S10: Preprocess the local image of the object to be processed to obtain the first mesh model of the local image.
[0060] In the specific implementation process, the object to be processed refers to the product awaiting processing, such as parts that need to be welded. A partial image of the object to be processed can be obtained by controlling a 3D camera to be positioned in a first location and capturing the image. This partial image provides 3D information data about the local structure of the object. However, the field of view of a 3D camera is limited; higher-precision 3D cameras have a smaller field of view. Therefore, the area captured is usually only a portion of the surface of the object to be processed. The data obtained by the 3D camera typically includes color and depth information, and generally includes the pose coordinates p(x, y, z) of each pixel p relative to the camera coordinate system. The partial image of the object to be processed obtained by the 3D camera is preprocessed, and a first mesh model of the partial image is obtained through mesh reconstruction technology (i.e., Poisson surface reconstruction). The 3D point cloud data acquired by the 3D camera is used to generate a triangular mesh model of the workpiece surface, where each vertex of the triangle originates from the 3D point cloud.
[0061] S11: The first mesh model is fused with the digital model of the object to be processed to obtain the location information of the processing area of the object to be processed; wherein, the digital model includes the processing area of the object to be processed.
[0062] In the specific implementation process, the digital model of the object to be processed can be a 3D model of the workpiece designed by any type of CAD software, containing all geometric and topological information of the workpiece, such as the coordinates of each point on the workpiece and the start and end points of each line segment. The bottom layer of the generated first mesh model is supported by the point cloud captured by a 3D camera, which is the local surface information of the real workpiece, while the digital model is a theoretical model, a virtual object. In order to establish the fusion of the two, it is necessary to sample the digital model to obtain the corresponding point cloud model. Then, the point cloud of the bottom layer of the first mesh model and the point cloud generated by the digital model are placed in the mesh model coordinate system, and point cloud registration ICP is performed to obtain the transformation T from the point cloud of the digital model to the point cloud of the bottom layer of the first mesh model, thereby aligning and fusing the two point clouds together. Finally, the pose of the digital model is changed using the transformation T, so that the virtual digital model and the first mesh model are also aligned, thereby realizing the spatial fusion of the virtual digital model and the first mesh model obtained by actually photographing the workpiece in the mesh model coordinate system. At this point, the trajectory position of the processing on the virtual model is also transformed T to be unified to the grid model coordinate system. This determines the position of the trajectory to be processed on the virtual digital model in the grid model coordinate system, that is, the position in the real physical coordinate system, so as to obtain the position information of the processing area of the object to be processed.
[0063] S12: Based on the preset path and the location information of the area to be processed, obtain a three-dimensional image of the area to be processed; wherein, the preset path is obtained based on the digital model.
[0064] In the specific implementation process, the 3D image of the area to be processed can be obtained by taking a picture with a 3D camera at the second position. Taking the first position of the local image of the object to be processed obtained in step S10 as the reference, the path of the 3D camera from the first position to the second position is called the preset path. This preset path can be obtained through a digital model. Moving the robotic arm in a straight line from the initial shooting position (the first position in step S10) to the position of the processing trajectory (the second position in this step) may cause a collision hazard, so it is necessary to plan a safe and reliable robotic arm movement trajectory. The physical space accessible to the robotic arm is divided into a 3D cubic grid space according to a certain resolution. Each empty space occupied by the digital model is an impassable position, and the empty space not occupied by other digital models is a passable position. The initial shooting position is the starting empty space, and the shooting position of the processing trajectory is the ending grid. The A* algorithm or Dijkstra's algorithm can be used to find a safe and reliable effective movement trajectory, thereby moving to a reasonable position (the second position) to take a picture of the object to be processed, thereby obtaining a 3D image of the area to be processed.
[0065] S13: Convert the three-dimensional image of the area to be processed into a second mesh model.
[0066] In the specific implementation process, the same Poisson reconstruction method as in step S10 is used to perform mesh reconstruction of the three-dimensional image point cloud of the area to be processed, and obtain the corresponding second mesh model.
[0067] S14: Based on the features of the theoretical trajectory to be operated on the object to be processed and the processing area of the second mesh model, the operation trajectory of the object to be processed is obtained; wherein, the features of the theoretical trajectory to be operated on the object to be processed are obtained based on the digital model.
[0068] In practical implementation, the theoretical trajectory to be operated refers to the operation trajectory given by the digital model of the object to be processed. The characteristics of the theoretical trajectory to be operated refer to certain specific aspects of its operation trajectory, such as size and angle. By using the characteristics of the theoretical trajectory to be operated of the object to be processed and the processing area of the second mesh model, the operation trajectory located within the processing area can be obtained.
[0069] In this embodiment, when it is necessary to obtain the operation trajectory of the object to be processed, a local image of the object is first acquired. This local image is then processed and converted into a first mesh model. The first mesh model is then fused with the digital model of the object to be processed. After fusion, the approximate location of the processing area of the object is predicted. Based on the approximate location of the processing area and a preset path from the first location to the second location, a three-dimensional image of the processing area is obtained. Specifically, a local image of the object to be processed can be obtained from the first location, and a three-dimensional image of the object to be processed can be obtained from the second location. The three-dimensional image of the processing area is then converted into a second mesh model. Based on the characteristics of the theoretical operation trajectory and the processing area of the second mesh model, a preliminary operation trajectory of the object to be processed is obtained. Finally, the preliminary operation trajectory is optimized to obtain the final operation trajectory of the object to be processed. That is, this application converts a local image of the object to be processed into a first mesh model, converts a three-dimensional image including the processing area into a second mesh model, and fuses the first mesh model of the local image with the digital model of the object to be processed. Because the mesh model itself has holistic characteristics, each triangle of the mesh model can be optimized from the perspective of the entire mesh model, thereby obtaining a globally stable and accurately positioned geometric location. Digital models provide fundamental constraints for the optimization of mesh models, further guiding the specific direction of mesh model optimization. Without a mesh model, a digital model only provides the theoretical position of the trajectory, which is insufficient for accurate trajectory determination. Conversely, without a digital model, the constraints on mesh model optimization are weak, lacking sufficient prior information to optimize towards the theoretical trajectory position, thus failing to obtain an accurate operational trajectory. Therefore, this approach enables more accurate acquisition of the operational trajectory of the object to be processed, thereby improving the quality of operation on that object.
[0070] To obtain the operation trajectory of the object to be processed more accurately, in some embodiments, the following technical solution is provided: before the step of obtaining the operation trajectory of the object to be processed based on the features of the theoretical operation trajectory of the object to be processed and the processing area of the second mesh model, the method further includes: judging whether the processing area of the object to be processed meets the processing conditions according to the processing area of the second mesh model and the processing area of the digital model.
[0071] Therefore, the step of obtaining the operation trajectory of the object to be processed based on the features of the theoretical operation trajectory of the object to be processed and the processing area of the second mesh model includes: when the processing area of the object to be processed meets the processing conditions, obtaining the operation trajectory of the object to be processed based on the features of the theoretical operation trajectory of the object to be processed and the processing area of the second mesh model.
[0072] In this embodiment, the characteristics of the theoretical trajectory of the object to be processed given by the digital model are compared and analyzed with the processing area of the second mesh model. The error is calculated and automatic structural defect detection is performed to determine whether the processing area of the object to be processed meets the processing conditions. Taking welding as an example, before welding, the area to be welded is ground to ensure welding quality and avoid the presence of burrs or local protrusions left over from cutting or pre-processing. The mesh model can fit two planes, plane1 and plane2, on both sides of the weld, with normal vectors v1 and v2, and centers c1 and c2 on the two planes. The distance d from point p1 on plane1 of the processing area to plane1 to plane1 is (p1-c1).dot(v1). If d>0, the processing area is protruding at that position; if d<0, the processing area is concave at that position. Welding requires the maximum height of the local protrusion to be H. When d is greater than or equal to H at all points on the two planes on both sides of the weld, the object to be processed meets the prerequisite for welding, that is, it meets the processing conditions for welding.
[0073] If the processing conditions are not met, after the step of determining whether the processing area of the object to be processed meets the processing conditions based on the processing area of the second mesh model and the processing area of the digital model, the process further includes: sending a prompt message if the processing area of the object to be processed does not meet the processing conditions; wherein, the prompt message includes the positional error information of the processing area of the object to be processed. For example, taking welding as an example, if the preconditions for welding are not met, the user is reminded on the visual interactive interface that there is a structural defect in the object to be processed, and the corresponding position of the defect on the virtual digital model is marked with a special color, thereby realizing automatic detection and feedback of structural defects in the object to be processed. The user can obtain defect information in real time from the interactive interface and instruct the robot to skip the current processing trajectory to process the next trajectory or take other actions.
[0074] In some embodiments, the following specific solution is provided: the step of determining whether the processing area of the object to be processed meets the processing conditions based on the processing area of the second mesh model and the processing area of the digital model includes: comparing the processing area of the second mesh model and the processing area of the digital model to obtain the error value between the processing area of the second mesh model and the processing area of the digital model; and determining whether the processing area of the object to be processed meets the processing conditions based on the relationship between the error threshold and the error value.
[0075] In this embodiment, the processing area on the digital model can be understood as a virtual, standard processing area, while the processing area on the second mesh model is the actual processing area obtained from the three-dimensional image of the object to be processed, based on a local image of the object. By comparing the two, the magnitude of the error, i.e., the error value, can be determined. The allowable error magnitude differs for each object to be processed and can be set according to actual needs, i.e., the error threshold. By comparing the relationship between the error threshold and the error value, it can be determined whether the processing area of the object meets the processing conditions. For example, if the error value is less than the error threshold, the processing conditions are met. Thus, by determining whether the processing conditions are met before obtaining the operation trajectory of the object, the operation trajectory of the object can be obtained more accurately.
[0076] In some embodiments, the following technical solution is provided: the step of obtaining the operation trajectory of the object to be processed based on the features of the theoretical operation trajectory of the object to be processed and the processing area of the second mesh model includes: firstly, obtaining the features of the operation trajectory of the processing area in the second mesh model based on the features of the theoretical operation trajectory of the object to be processed; and then obtaining the operation trajectory of the object to be processed based on the features of the operation trajectory in the processing area.
[0077] In this embodiment, based on the digital model of the object to be processed, the features of the theoretical trajectory to be operated on are generated. Then, it is determined whether there are features in the second mesh model that are the same as the features of the theoretical trajectory to be operated on. Based on the features that are the same as the features of the theoretical trajectory to be operated on, the operation trajectory of the object to be processed can be obtained. Specifically, due to the preprocessing of the object to be processed and various practical reasons, there are always some differences between the actual object to be processed and the theoretically designed virtual digital model. They may differ slightly in size or shape, especially when the workpiece has many parts or the preprocessing process is complex. Therefore, it is necessary to find the actual trajectory position to be processed near the theoretical processing position specified after the virtual digital model and the mesh model are merged. More specifically, based on the trajectory to be processed selected on the interactive virtual digital model, the surfaces and planes adjacent to the trajectory to be processed can be found from the topological information of the digital model. The shape, size, and connection angle between each plane are theoretically calculated, i.e., the feature neighborhood association information of the trajectory to be processed. Such patch association information is extracted from the local mesh model, and the intersection line that best matches the theoretical value is found as the actual trajectory position to be processed, i.e., the operation trajectory of the object to be processed.
[0078] In some embodiments, such as Figure 3As shown, after the step of obtaining the operation trajectory of the object to be processed based on the features of the theoretical operation trajectory of the object to be processed and the processing area of the second mesh model, the method further includes:
[0079] S20: Smooth and repair the second mesh model of the operation trajectory neighborhood of the object to be processed.
[0080] In the specific implementation process, the neighborhood of the operation trajectory of the object to be processed refers to the area near the operation trajectory of the object to be processed. Specifically, the second mesh model near the operation trajectory of the object to be processed is smoothed and repaired. This reduces the impact of noise and makes the disordered mesh of the second mesh model orderly. Since the operation trajectory of the object to be processed is on the second mesh model, the global optimization of the operation trajectory of the object to be processed is indirectly achieved.
[0081] S21: Based on the second mesh model after smoothing and mesh repair, extract the trajectory control points on the operation trajectory; wherein, the trajectory control points include the mesh vertices on the operation trajectory.
[0082] In the specific implementation process, based on the second mesh model after smoothing and mesh repair, the mesh vertices on the operation trajectory of the object to be processed can be extracted as trajectory control points.
[0083] S22: Optimize the operation trajectory of the object to be processed based on the curve type and the trajectory control points in the digital model; wherein the curves include Bézier curves and B-spline curves.
[0084] In practice, the corresponding curve is generated from the trajectory control points based on the curve type in the digital model. For example, if the digital model uses a Bézier curve, the corresponding Bézier curve is generated from the trajectory control points; if it uses a B-spline curve, the corresponding B-spline curve is generated. In this way, a globally optimized and highly accurate operation trajectory of the object to be processed can be obtained.
[0085] In this embodiment, by performing global and high-precision optimization on the operation trajectory based on the second grid model, the accuracy of the operation trajectory of the object to be processed can be greatly improved, thereby further improving the operation quality of the object to be processed.
[0086] In some embodiments, such as Figure 4 As shown, the process further includes the following steps before preprocessing the local image of the object to be processed to obtain a first mesh model of the local image:
[0087] S30: Load the digital model of the object to be processed; wherein the digital model includes the geometric and topological information of the object to be processed.
[0088] In the implementation process, the operation trajectory and related operation information are specified in an interactive manner. Specifically, when the program runs, a visual window, such as an interactive trajectory settings window, first opens. In this window, the workpiece to be processed, such as a wheel hub, is selected, and the corresponding 3D structure of the wheel hub is displayed. Users can select the edge corresponding to the grinding trajectory on the wheel hub by clicking with the mouse, clicking, or using the touchscreen. The trajectory can be adjusted flexibly in real time by clicking and dragging. Corresponding expert processing parameters are added in the window that appears on the left, such as grinding angle and intensity, or a pre-set trajectory type is selected. Finally, the workpiece's posture in the interactive window is adjusted to ensure it is similar to the actual posture of the workpiece placed on the worktable relative to the camera.
[0089] S31: Adjust the position of the object to be processed on the workbench so that the position of the object to be processed on the workbench is the same as the position of the digital model of the object to be processed.
[0090] In the specific implementation process, the workpiece to be processed is placed on the workbench, and the posture of the placement is similar to the posture specified in step S30. When the position of the workpiece on the workbench is close to the position specified in the visualization interface in step S30, the similarity between the two is higher, which is more conducive to the fusion of virtual 3D digital model and real imaging data captured by camera.
[0091] S32: Based on the object to be processed after its position is adjusted, obtain a local image of the object to be processed.
[0092] In the specific implementation process, after adjusting the position of the object to be processed on the worktable, a partial image of the object to be processed can be obtained by using a 3D camera in the first position.
[0093] In this embodiment, before obtaining a partial image of the object to be processed, the position of the object on the workbench is adjusted based on the digital model of the object. This makes the actual placement position of the object closer to the theoretical placement position, thereby making it easier and more accurate to obtain the operation trajectory on the object.
[0094] After finding the operation trajectory of the object to be processed using the above method, the operation trajectory is converted to the robot coordinate system and transmitted to the robot control terminal. This application integrates the digital model, the mesh model of the trajectory region, and trajectory-related operation information, and displays the visualization effect in a visual interactive window. Warning information is provided for areas with large errors or anomalies in the trajectory-related areas. The fused digital model and the initial first mesh model are added to the visual interactive window, as are the second mesh model. Simultaneously, the virtual 3D digital model of the trajectory to be processed and the globally and high-precision optimized trajectory are added to the visual interactive window, thus providing the user with a fused interactive window that simultaneously displays the global virtual model, the local actual mesh model, the theoretical trajectory to be processed, and the actual trajectory to be processed. Areas where the difference between the theoretical and actual trajectories is significant are highlighted in the interactive window with prominent colors for real-time user reference. In summary, this application adopts an interactive target trajectory generation method based on a (3D) mesh model. Based on the ordered, uniform, and structured properties of the mesh model, it can effectively and accurately extract the target operation trajectory and perform high-precision algorithm optimization on the trajectory based on the overall mesh structure. The interactive approach allows for appropriate adjustments to the target operation trajectory while ensuring its accuracy and reliability. Once the above process is completed, subsequent workpieces only need to be placed on the worktable in a similar posture to automatically generate the welding trajectory, without the need for manual intervention.
[0095] In another embodiment, such as Figure 5 As shown, based on the same inventive concept as the foregoing embodiments, embodiments of this application also provide a trajectory generation device, which includes:
[0096] The first model is used to preprocess a local image of the object to be processed in order to obtain a first mesh model of the local image.
[0097] The second model is used to fuse the first mesh model with the digital model of the object to be processed to obtain the location information of the processing area of the object to be processed; wherein, the digital model includes the processing area of the object to be processed;
[0098] The third model is used to obtain a three-dimensional image of the area to be processed based on a preset path and the location information of the area to be processed; wherein the preset path is obtained based on the digital model;
[0099] A conversion model is used to convert the three-dimensional image of the area to be processed into a second mesh model;
[0100] The fourth model is used to obtain the operation trajectory of the object to be processed based on the features of the theoretical operation trajectory of the object to be processed and the processing area of the second grid model; wherein the features of the theoretical operation trajectory of the object to be processed are obtained based on the digital model.
[0101] It should be noted that each module in the trajectory generation device in this embodiment corresponds one-to-one with each step in the trajectory generation method in the aforementioned embodiment. Therefore, the specific implementation method and the technical effects achieved in this embodiment can be referred to the implementation method of the aforementioned trajectory generation method, and will not be repeated here.
[0102] Furthermore, in one embodiment, this application also provides a computer device, the computer device including a processor, a memory, and a computer program stored in the memory, the computer program being executed by the processor to implement the methods in the foregoing embodiments.
[0103] In addition, in one embodiment, this application also provides a computer storage medium storing a computer program that is executed by a processor to implement the methods described in the foregoing embodiments.
[0104] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a device including one or any combination of the above-mentioned memories. The computer may be a variety of computing devices, including smart terminals and servers.
[0105] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.
[0106] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).
[0107] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.
[0108] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.
[0109] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0110] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory / random access memory, magnetic disk, optical disk) and includes several instructions to cause a multimedia terminal device (which may be a mobile phone, computer, television receiver, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0111] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.
Claims
1. A trajectory generation method, characterized in that, The method includes: A local image of the object to be processed is preprocessed to obtain a first mesh model of the local image; The first mesh model is fused with the digital model of the object to be processed to obtain the location information of the processing area of the object to be processed; wherein, the digital model includes the processing area of the object to be processed; Based on a preset path and the location information of the area to be processed, a three-dimensional image of the area to be processed is obtained; wherein, the preset path is obtained based on the digital model; The three-dimensional image of the area to be processed is converted into a second mesh model; Based on the characteristics of the theoretical trajectory to be operated on the object to be processed and the processing area of the second mesh model, the operation trajectory of the object to be processed is obtained; wherein, the characteristics of the theoretical trajectory to be operated on the object to be processed are obtained based on the digital model; The process involves fusing the first mesh model with the digital model of the object to be processed to obtain the location information of the processing area of the object, including: The digital model is sampled to obtain the point cloud model corresponding to the digital model; Place the point cloud of the bottom layer of the first mesh model and the point cloud generated by the digital model in the mesh model coordinate system, and perform point cloud registration ICP to obtain the transformation T from the point cloud of the digital model to the point cloud of the bottom layer of the first mesh model. The pose of the digital model is changed based on the transformation T so that the virtual digital model and the first mesh model are aligned. Based on the transformation T, the trajectory position of the processing on the virtual model is unified to the coordinate system of the grid model, and the position of the trajectory to be processed on the virtual digital model in the coordinate system of the grid model is obtained, so as to obtain the position information of the processing area of the object to be processed.
2. The trajectory generation method as described in claim 1, characterized in that, Before the step of obtaining the operation trajectory of the object to be processed based on the features of the theoretical operation trajectory of the object to be processed and the processing area of the second mesh model, the method further includes: Based on the processing area of the second grid model and the processing area of the digital model, determine whether the processing area of the object to be processed meets the processing conditions; The process of obtaining the operation trajectory of the object to be processed based on the features of the theoretical trajectory to be processed of the object and the processing area of the second mesh model includes: When the processing area of the object to be processed meets the processing conditions, the operation trajectory of the object to be processed is obtained based on the characteristics of the theoretical operation trajectory of the object to be processed and the processing area of the second mesh model.
3. The trajectory generation method as described in claim 2, characterized in that, The step of determining whether the processing area of the object to be processed meets the processing conditions based on the processing area of the second mesh model and the processing area of the digital model includes: By comparing the area to be processed in the second mesh model with the area to be processed in the digital model, the error value between the area to be processed in the second mesh model and the area to be processed in the digital model is obtained. Based on the relationship between the error threshold and the error value, it is determined whether the processing area of the object to be processed meets the processing conditions.
4. The trajectory generation method as described in claim 2, characterized in that, After determining whether the processing area of the object to be processed meets the processing conditions based on the processing area of the second mesh model and the processing area of the digital model, the method further includes: If the processing area of the object to be processed does not meet the processing conditions, a prompt message is sent; wherein, the prompt message includes the positional error information of the processing area of the object to be processed.
5. The trajectory generation method as described in claim 1, characterized in that, The process of obtaining the operation trajectory of the object to be processed based on the features of the theoretical trajectory to be processed of the object and the processing area of the second mesh model includes: Based on the characteristics of the theoretical operation trajectory of the object to be processed, the characteristics of the operation trajectory of the processing area in the second mesh model are obtained. Based on the characteristics of the operation trajectory in the area to be processed, the operation trajectory of the object to be processed is obtained.
6. The trajectory generation method as described in claim 1, characterized in that, After the step of obtaining the operation trajectory of the object to be processed based on the features of the theoretical operation trajectory of the object to be processed and the processing area of the second mesh model, the method further includes: The second mesh model of the operation trajectory neighborhood of the object to be processed is smoothed and repaired. Based on the second mesh model after smoothing and mesh repair, trajectory control points are extracted on the operation trajectory; wherein, the trajectory control points include mesh vertices on the operation trajectory; Based on the curve types and trajectory control points in the digital model, the operation trajectory of the object to be processed is optimized; wherein, the curves include Bézier curves and B-spline curves.
7. The trajectory generation method according to any one of claims 1-6, characterized in that, Before the step of preprocessing the local image of the object to be processed to obtain the first mesh model of the local image, the method further includes: Load the digital model of the object to be processed; wherein the digital model includes the geometric and topological information of the object to be processed. Adjust the position of the object to be processed on the worktable so that the position of the object to be processed on the worktable is the same as the position of the digital model of the object to be processed. Based on the object to be processed after its position has been adjusted, a partial image of the object to be processed is obtained.
8. A trajectory generation device, characterized in that, The device includes: The first model is used to preprocess a local image of the object to be processed in order to obtain a first mesh model of the local image. The second model is used to fuse the first mesh model with the digital model of the object to be processed to obtain the location information of the processing area of the object to be processed; wherein, the digital model includes the processing area of the object to be processed; The third model is used to obtain a three-dimensional image of the area to be processed based on a preset path and the location information of the area to be processed; wherein the preset path is obtained based on the digital model; A conversion model is used to convert the three-dimensional image of the area to be processed into a second mesh model; The fourth model is used to obtain the operation trajectory of the object to be processed based on the features of the theoretical operation trajectory of the object to be processed and the processing area of the second mesh model; wherein, the features of the theoretical operation trajectory of the object to be processed are obtained based on the digital model; The process involves fusing the first mesh model with the digital model of the object to be processed to obtain the location information of the processing area of the object, including: The digital model is sampled to obtain the point cloud model corresponding to the digital model; Place the point cloud of the bottom layer of the first mesh model and the point cloud generated by the digital model in the mesh model coordinate system, and perform point cloud registration ICP to obtain the transformation T from the point cloud of the digital model to the point cloud of the bottom layer of the first mesh model. The pose of the digital model is changed based on the transformation T so that the virtual digital model and the first mesh model are aligned. Based on the transformation T, the trajectory position of the processing on the virtual model is unified to the coordinate system of the grid model, and the position of the trajectory to be processed on the virtual digital model in the coordinate system of the grid model is obtained, so as to obtain the position information of the processing area of the object to be processed.
9. A computer device, characterized in that, The computer device includes a memory and a processor, wherein the memory stores a computer program and the processor executes the computer program to implement the method as described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and the processor executes the computer program to implement the method as described in any one of claims 1-7.