An aircraft skin feature recognition and milling path planning method based on semantic segmentation

Abstract

The present application relates to a kind of based on semantic segmentation aircraft skin feature identification and milling edge path planning method.This method includes using line laser scanner to obtain original point cloud image, original point cloud image is preprocessed;Point cloud image is segmented using PointNet model, obtains reference point cloud graph, and the feature extraction is carried out to the point cloud graph;Based on time sequence point cloud posture calculation;Based on the milling edge path planning of feature point.Experiments prove that, compared with traditional method, the limitation of sensor scanning angle is smaller, since the data volume of linear point cloud is smaller than that of surface point cloud, and the geometric profile is simpler than that of surface point cloud, the power cost and the requirement of software and hardware are lower, the high-precision milling edge can be completed while saving cost, and the calculation speed can meet the 12ms communication time interval of KUKA robot RSI, so the method has broad application prospect in the field of milling edge path planning.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of aircraft skin milling edge, and particularly relates to a method and system for aircraft skin feature recognition and milling edge path planning based on semantic segmentation. BACKGROUND

[0002] Common aircraft skin milling edge systems analyze and calculate through mathematical models or actually collected data to generate paths with position and attitude information to guide the robot processing. The invention patent "Aircraft skin automatic high-precision milling edge device and milling edge method" applied by Northwestern Polytechnical University in 2015 proposes a solution for point alignment, normal leveling, flexible clamping and high-precision trajectory planning of the region to be milled, which improves the size precision and normal precision of skin milling edge. The utility model patent "A robot trimming system" applied by Shenyang Aircraft Corporation in 2019 uses a contour scanning device and a depth measuring device to measure and calculate in real time, and improves the robot positioning accuracy through real-time feedback.

[0003] However, at present, in the aircraft assembly industry, although the demand for automatic trimming inside the skin frame is very strong, there is no mature product successfully applied at home and abroad. SUMMARY

[0004] In order to meet the technical needs of automatic trimming inside the skin frame, overcome the defects of low precision, large workload, long time-consuming, low efficiency and high operation hazard of manual trimming, the application proposes a new method for aircraft skin feature recognition and milling edge path planning based on semantic segmentation.

[0005] TERMS EXPLANATION

[0006] Aircraft skin feature recognition and milling edge path planning based on semantic segmentation: semantic segmentation is a deep learning method that can distinguish the category of each point in the aircraft skin point cloud data, so as to accurately distinguish the reference of milling edge, and then calculate the position of feature points on the reference and the inclination angle of the reference surface, and complete the path planning of milling edge according to the calculated position and angle.

[0007] The overall design idea of the method is to use a line laser sensor to obtain point cloud data, then use a PointNet model to perform semantic segmentation on the point cloud data, calculate the milling position and angle based on the segmented reference part point cloud graph, and then complete the path planning of the milling edge. Experiments show that, compared with traditional methods, the method has less restriction on the scanning angle of the sensor. In addition, the data volume of line-shaped point cloud data is smaller than that of surface-shaped point cloud, so the computing power cost of the method is lower, the calculation speed can meet the 12ms communication time interval of KUKA robot RSI, and it is helpful to realize the online milling edge function. Finally, since the geometric profile of the line laser point cloud data is simpler than that of the surface-shaped point cloud, the requirements for software and hardware are lower, the cost can be saved under the condition of completing high-precision milling edge, so the method has a wider application prospect in milling edge path planning and other scenes.

[0008] Overall, the aircraft skin feature recognition and milling edge path planning method based on semantic segmentation includes the following processing processes:

[0009] 1) Point cloud image preprocessing

[0010] Since the surface feature point cloud image obtained by line laser scanning has the characteristics of high resolution, small data volume and low anti-interference, and considering the factors such as small target feature size, complex structure and high real-time requirement of the object to be measured in the machining process, the method (system) needs to preprocess the original point cloud image to reduce the error in the subsequent feature extraction and path planning process. Through the analysis of the point cloud obtained by the simulation machining process experiment, the method uses threshold filtering and Gaussian filtering to process the collected point cloud, and uses the mean filtering function based on time sequence provided by the scanner to optimize the collected point cloud together.

[0011] 2) Point cloud image feature extraction

[0012] According to the machining process, the trajectory path of the trimming (milling) process is mainly based on part of the feature structure of the reference part, such as the chamfer edge in the reference part. Therefore, in the collected point cloud image, the system needs to extract and segment the machining object first to distinguish the skin and the reference part and determine the location of the feature. The method uses a PointNet model to perform semantic segmentation on the preprocessed point cloud image, obtains the reference part point cloud graph and divides it into different semantic sections, obtains the intersection point of the chamfer section and the reference upper plane section in the semantic section as the preliminary feature point, and then uses the straight line fitting intersection point method to extract the feature point accurately to obtain the accurate feature point position.

[0013] 3) Time sequence-based point cloud posture calculation

[0014] In the trimming process, the milling module needs to adjust the posture according to the reference machining surface of the frame. According to the process information, the reference surface has a pitch angle change, so the tool coordinate system established on the machining tool spindle needs to be rotated under the control of the system with the change of the pitch angle. After adjusting the base coordinate system and the tool coordinate system to a reasonable pose, the system will adjust the posture with the main motion direction of the milling process as the axis. In the online trimming mode, since the machining process has a high real-time requirement, the system will directly use the original image obtained by the line laser scanner to perform relative posture calculation based on time sequence.

[0015] 4) Feature point-based path planning

[0016] After obtaining the feature points, the system will plan the path based on the series of feature points. The system re-divides the path points based on the feature points, uses the interpolation method, fully utilizes the change trend of each feature point in the interpolation segment, and calculates to obtain the machining path.

[0017] Specifically, the application provides an aircraft skin feature recognition and milling edge path planning method based on semantic segmentation, which comprises:

[0018] S1. Obtain an original point cloud image by using a line laser scanner, and pre-process the original point cloud image;

[0019] S2. Perform semantic segmentation on the pre-processed point cloud image by using a PointNet model, obtain a reference part point cloud image, and perform feature extraction on the point cloud image;

[0020] S3. Time sequence-based point cloud posture calculation;

[0021] S4. Feature point-based milling edge path planning;

[0022] S5. Machining trajectory fitting and re-division.

[0023] Further, the aircraft skin feature recognition and milling edge path planning method based on semantic segmentation provided by the application comprises the pre-processing of the original point cloud image in step S1, which comprises:

[0024] S11. Analyze the point cloud results obtained by simulating the machining process experiment;

[0025] S12. According to the analysis results of the previous step, the original point cloud image is pre-processed by using threshold filtering and Gaussian filtering;

[0026] S13. The original point cloud image is optimized by using a line laser scanner based on time sequence adjacent interval mean filtering method.

[0027] Furthermore, in step S2 of the semantic segmentation-based aircraft skin feature recognition and milling path planning method of the present invention, the use of the PointNet model to perform semantic segmentation on the preprocessed point cloud image to obtain a reference part point cloud graphic, and the extraction of features from the point cloud graphic, includes:

[0028] S21. Use point cloud data to pre-train the PointNet model using deep learning;

[0029] S22. Use the PointNet model to judge the semantics of each discrete point in the preprocessed point cloud image, complete the semantic segmentation of the overall point cloud, obtain the reference point cloud image and divide it into different semantic segments;

[0030] S23. Obtain the intersection point of the chamfered segment and the reference plane segment in the above semantic segment as the preliminary feature point;

[0031] S24. Use straight line fitting to find intersection points to extract feature points and obtain accurate feature point positions.

[0032] Furthermore, in step S21 of the above-mentioned aircraft skin feature recognition and milling path planning method based on semantic segmentation, the PointNet model is pre-trained using point cloud data. The training parameters of the PointNet model include: batch_size, Epoch, Learning_rate, Decay_rate, Npoint, Step_size, and Lr_decay.

[0033] Furthermore, in step S22 of the above-mentioned semantic segmentation-based aircraft skin feature recognition and milling path planning method, the PointNet model is used to determine the semantics of each discrete point in the preprocessed point cloud image, thereby completing the semantic segmentation of the overall point cloud. The input of the PointNet model is the coordinates of the point cloud data, and the output of the model is m×n scores, corresponding to the scores of n input points relative to m categories; including:

[0034] Estimate an affine transformation matrix (3x3) and multiply the input point cloud by this matrix before feature extraction;

[0035] Each of the n input points has 1024-dimensional features. The maximum value among the n points is selected to obtain a global feature.

[0036] This global feature is copied n times and concatenated with the features obtained from the second layer MLP, that is, the local features and global features of each point are concatenated.

[0037] After two MLPs, the final output is of dimension m x n, i.e., for each point, m classifications are made, outputting its predicted score for each class.

[0038] Further, the aircraft skin feature recognition and milling edge path planning method based on semantic segmentation in step S3 uses the point cloud features of adjacent time points to obtain the relative pose relationship of the tool coordinate system at the corresponding time point by comparing the changes in the poses of the two time points, including:

[0039] S31. The motion direction of the tool coordinate system coincides with the base coordinate system and the main motion direction of the milling process;

[0040] S32. The tool coordinate system on the milling tool spindle rotates with the pitch angle of the reference surface of the reference part, and adjusts the base coordinate system and the tool coordinate system to a reasonable pose;

[0041] S33. Adjust the pose with the main motion direction of the milling process as the axis;

[0042] S34. In the online trimming mode, directly use the original point cloud image obtained by the line laser scanner to perform the relative pose calculation based on time sequence.

[0043] Further, the feature point based path planning in step S4 of the aircraft skin feature recognition and milling edge path planning method based on semantic segmentation includes: after obtaining the feature points, re-dividing the path points based on the feature points, and using the interpolation method to calculate the machining path by using the change trend of each feature point in the interpolation segment.

[0044] On the other hand, the present application also provides an aircraft skin feature recognition and milling edge path planning system based on semantic segmentation. The system is aimed at the skin frame structure of an aircraft, uses a KUKA industrial robot as a carrier, uses a line laser scanner to obtain skin trimming reference information, and plans a machining path in real time according to the machining process to guide the robot to complete the conformal motion task.

[0045] The system provides online or offline data processing modes.

[0046] The system is composed of a software system and a hardware platform, wherein the software system is a data analysis and processing system located on the host computer; the hardware platform includes a host computer, a 3D line laser scanner, a KUKA industrial robot, a milling machining module, a visual calibration board, an end support mechanism, and a system assembly structure.

[0047] The system is implemented and operated according to the above-mentioned aircraft skin feature recognition and milling edge path planning method based on semantic segmentation.

[0048] In addition, the application also relates to application of the aircraft skin feature recognition and milling edge path planning method and system based on semantic segmentation in the aircraft assembly industry.

[0049] To sum up, the aircraft skin feature recognition and milling edge path planning method and system based on semantic segmentation have the following advantages:

[0050] 1) The milling edge system based on the traditional visual algorithm has the disadvantages of poor parameter adaptability / robustness and the need to match different parameters for different working conditions, the method based on artificial intelligence technology can more adaptively complete the milling edge feature recognition and path planning, deep learning can learn deeper workpiece features, and the tolerance to scaling ratio, position and rotation angle is higher than that of the traditional algorithm.

[0051] 2) Experiments prove that the method has smaller limitation on the scanning angle of the sensor compared with the traditional method. In addition, the data amount of linear point cloud data is smaller than that of surface point cloud, so the computing power cost of the method is lower, the calculation speed can meet the 12ms communication time interval of the KUKA robot RSI, which is helpful to realize the online milling edge function. Finally, since the geometric profile of the linear laser point cloud data is simpler than that of the surface point cloud, the method has lower requirements for software and hardware, can save cost under the condition of completing high-precision milling edge, and has a wider application prospect in the milling edge path planning scene.

[0052] 3) In order to ensure the continuity of the processing process and the real-time of the system path planning, the system uses the mean filtering method to uniformly pretreat the feature points, reduces the oscillation error caused by linear laser scanning, avoids excessive jitter of the path, and at the same time ensures the stability of the speed and other parameters in the processing process, improves the smoothness of the processing process. BRIEF DESCRIPTION OF DRAWINGS

[0053] In order to more clearly illustrate the technical scheme of the embodiments of the application, the following briefly introduces the drawings needed to be used in the embodiments of the application, and obviously, the following drawings are only some embodiments described in the application, and other drawings can be obtained by those skilled in the art without paying creative labor on the premise of the drawings.

[0054] Figure 1 The figure is a schematic diagram of the skin and the reference in the workpiece of the embodiments of the application.

[0055] Figure 2 The figure is a schematic diagram of the software system and hardware platform composition structure of the robot intelligent trimming system used in the application.

[0056] Figure 3 The figure is a schematic diagram of the PointNet model structure.

[0057] Figure 4A single-angle experimental point cloud data labeling schematic diagram in an embodiment of the present application.

[0058] Figure 5 A multi-angle mixed experimental point cloud data labeling schematic diagram in an embodiment of the present application.

[0059] Figure 6 A model training detection result schematic diagram in an embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to make the objectives, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described below in detail with specific embodiments and corresponding drawings. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. The present application can also be implemented or applied by using different specific embodiments, and each detail in the specification can be modified or changed based on different views and applications without departing from the spirit of the present application.

[0061] At the same time, it should be understood that the protection scope of the present application is not limited to the following specific specific embodiments; it should also be understood that the terms used in the embodiments of the present application are for describing the specific specific embodiments, but not for limiting the protection scope of the present application.

[0062] Embodiment: A method for aircraft skin feature recognition and milling path planning based on semantic segmentation

[0063] The present embodiment mainly aims at the application scenario of profile milling, and divides the workpiece into two parts of skin and reference. As shown in the figure, Figure 1 the blue part is the skin, and the red part is the reference datum of profile milling. Since the skin has a reserved machining allowance, it will be more prominent than the reference, and the allowances at different positions are different. The application of profile milling of aircraft skin requires that the skin be milled according to the contour shape of the reference part at the corresponding position, so that the milled skin is flush with the reference.

[0064] The hardware platform is the basis for realizing the stable mounting of the end visual sensor and cutting tool of the robot, and for collecting visual information and processing planning. As shown in the figure, Figure 2 the hardware platform mainly consists of one host computer, one KUKA robot, one line laser scanner, one set of cutting tools, one end support mechanism and their respective device structural parts.

[0065] In the research of deep learning based on point cloud data, few networks directly process the point cloud data of the scene. PointNet is a point cloud classification / segmentation deep learning framework proposed by the University of Sydney in 2016, which directly uses the original point cloud data as input, maximizes the preservation of the spatial features of the point cloud, and achieves good results in the final test, laying the foundation for the development of deep learning network structure for processing point cloud data.

[0066] The semantic segmentation effect of the PointNet model is shown in Figure 3 The input of the PointNet model is point cloud data, and the basic description of each point is its coordinates (x, y, z). Color (RGB), normal vector (Normal), and other description information can also be added. The output of the model is m x n scores, corresponding to the scores of n points with respect to m categories.

[0067] PointNet network directly uses multi-layer perceptron (MLP) instead of convolution. Because the traditional convolution framework needs to have a very regular form of input data in order to share weights and optimize, but point cloud is unordered, it generally needs to be converted into a voxel grid or undergo a series of regularization processing, and the use of convolution undoubtedly introduces too many artificial factors, which may damage the data. Fully considering the unordered nature, symmetry and spatial transformation invariance of point cloud, the maximum pooling is selected.

[0068] Since the point cloud label does not change with spatial transformation, an affine transformation matrix (3x3) is estimated, and the input point cloud is multiplied by the estimated matrix before feature extraction (that is, an affine transformation is performed on the input point cloud). It is composed of MLP.

[0069] Each of the n input points has 1024-dimensional features, and then we select the maximum value among the n points (in the dimension of the number of points) to obtain a global feature. For the segmentation task, the global feature is copied n times, and the features obtained by the second layer MLP are spliced together, which is equivalent to splicing the local features and global features of each point. Then, after two MLPs, the output with a dimension of n x m is obtained, that is, for each point, m classifications are performed, and the predicted scores of each class are output.

[0070] The data collection angles of the single-angle experiment are 50°, 55° and 60°, and the mixed angles include collection angles of 40°, 45°, 50°, 55° and 60°. The single-angle experiment only uses 55° point cloud data for model training, and then tests its semantic segmentation accuracy at 50° and 60° respectively to test its transferability (the ability of the model to perform semantic segmentation on point clouds at new collection angles). The multi-angle experiment mixes data at multiple collection angles for training and testing to test the universality of the model (whether a single model can be trained to adapt to semantic segmentation tasks on point clouds at multiple collection angles).

[0071] In addition, due to the fact that a certain amount of milling will be reserved for the workpiece before initial processing in the actual processing process, the height difference between the workpiece and the reference piece in the point cloud data scanned during initial processing will be greater than the height difference during the next processing. With each milling in the processing, the height of the upper surface of the workpiece decreases and approaches the reference piece. Since the reserved milling of the incoming material is not fixed, it will change the shape features exhibited by the point cloud data, thereby affecting the semantic segmentation accuracy of the deep learning model. We conducted supplementary experiments to explore the processing capacity of the deep learning semantic segmentation model for this situation.

[0072] During the experiment, the robot end with a milling cutter and a line laser profile sensor moves, real-time acquires point cloud data of the workpiece to be milled and processes, and manually draws polygons to frame each region of the point cloud for semantic class labeling using cloudcompare software.

[0073] When selecting point cloud data with a collection angle of 55° for labeling and training, the selected point cloud data is a truncated segment (excluding the segment formed by iterative accumulation in the Y-axis direction before the laser scanner moves, and excluding the segment with large fluctuations and incomplete structure at the end of the scan), as shown in Figure 4 As shown, there are a total of 6 semantic classes (label = {0, 1, 2, 3, 4, 5}) in this labeling, which are the left vertical surface of the workpiece (blue, class 0), the upper surface of the workpiece (dark green, class 1), the right vertical surface of the workpiece (light green, class 2), the left vertical surface of the reference piece (yellow, class 3), the chamfer surface of the reference piece (orange, class 4), and the upper surface of the reference piece (red, class 5).

[0074] In addition, the comprehensive labeling and model training with collection angles of 40-60° select full segment point cloud data to maximize the fit with the actual laser scanning situation and test the model's ability to adapt to unexpected situations (such as incomplete structure, large fluctuations and large defects in the workpiece, etc.), as shown in Figure 5As shown, there are 4 semantic categories (label={0, 1, 2, 3}) in total, which are the upper surface of the workpiece (green, category 1), the chamfer surface of the reference part (yellow, category 2), the upper surface of the reference part (red, category 3) and other surfaces (surfaces irrelevant to the tool pose of the trimming system, blue, category 0).

[0075] Following the common data set division ratio in deep learning (training set: validation set: test set = 8:1:1), the data set is divided and the training batch is generated before the model training.

[0076] The training parameters of the model are shown in the following table:

[0077] Table 1 Training parameters of the model

[0078]

[0079]

[0080] The detection results are shown in Figure 6 , which show that the deep learning point cloud semantic segmentation algorithm is suitable for the trimming system, and the accuracy can reach 97-98%, as shown in Table 2. The accuracy calculation rule of this experiment is that the result of each point model semantic segmentation is different from the artificial labeling, so there are situations such as accuracy reduction caused by fuzzy boundaries of each region or inaccurate artificial labeling. After adjusting the accuracy calculation rule and adding a small range of fuzzy boundary points module, the accuracy of this algorithm can be further improved.

[0081] Table 2 Accuracy statistics

[0082]

[0083] In addition to the semantic segmentation accuracy, the running speed is also the focus of our attention. Since the model training can be completed before formal processing and stored, or even directly completed during software development, there is no need for the user to train again, so in this experiment, only the time of model use (semantic segmentation of unlabeled data) is counted.

[0084] Compared with the above experimental results, it can be seen that whether the data collected at an angle of 55° is used for model training and then applied to other collection angle data, or the data collected at multiple angles is used for training and then applied to multiple angle data, the experiments all show high accuracy. The more similar the training set data and the test set data, the higher the accuracy of semantic segmentation (the same collection angle, the point cloud data presents the same form; the more similar the collection angles, the more similar the data forms). The data form mainly shows that the point clouds of the machining part and the reference part on the side surface and the point clouds at the joint are different in completeness. When the collection angle approaches the horizontal direction, the point cloud of one side surface is more complete, and the completeness of the point cloud of the other side surface decreases.

[0085] When the point cloud of the machining part is raised by 1 mm along the Z-axis direction, the semantic segmentation accuracy of the 40-60° model for the 40-60° data decreases by about 2-17 percentage points compared with the data without processing. In addition, when the height difference between the machining part and the reference part increases from 0.4 mm to 0.8 mm, the accuracy of semantic segmentation decreases. The reason is that the data characteristics change after the position of the machining part is raised by data processing, and the difference with the model training set data increases. Similarly, the higher the similarity between the test set data and the training set data, the higher the accuracy of semantic segmentation.

[0086] Therefore, if we want to improve the adaptability of the model and make it applicable to more working conditions and collection angles, we need more training set data, or we can limit the actual working conditions and collection angles within the range where the model semantic segmentation effect is good, or we can improve the model structure to improve the accuracy of semantic segmentation.

[0087] In addition to the semantic segmentation accuracy, the running speed of the model is also the focus of our attention. Compared with the model training speed, we are more concerned about the model test speed which affects the processing speed through data processing speed in actual application. Through testing, the running speeds of different speed-up methods are shown in Table 3.

[0088] Table 3 Running speed test results

[0089] Speed up method Model Single line semantic segmentation speed None 55-degree trained model Approximately 9.5 ms per line Improved data reading approach 40-60-degree trained model Approximately 2.5 ms per line Improved data reading approach + multi-threading 40-60-degree trained model Approximately 0.91 ms per line

[0090] Overall, the method includes the following steps:

[0091] (I) Obtain the original point cloud image by using a line laser scanner, and preprocess the original point cloud image.

[0092] (II) Perform semantic segmentation on the preprocessed point cloud image using a PointNet model to obtain a reference part point cloud image, and perform feature extraction on the point cloud image.

[0093] (III) Time-based point cloud posture calculation.

[0094] (iv) Feature point based edge milling path planning.

[0095] The system measurement reference used in the method is composed of a reference frame (reference part) and a skin, wherein the reference frame and its features are the reference for the conformal machining of the trimming process, and therefore the point cloud obtained needs to be regionally and structurally segmented.

[0096] The skin and frame structure can be obtained after filtering the original point cloud image, and different feature points and feature structures such as chamfers and planes can be distinguished in the frame structure. Based on the imaging characteristics of point cloud, the image is mainly composed of discrete point cloud data, and the single sampling data obtained by the line laser scanner is an ordered point cloud line, so information analysis can be performed on the determined points in the point cloud to determine the corresponding meaning of the points and complete the segmentation of the point cloud structure.

[0097] In the method, the PointNet model is used for semantic segmentation of the preprocessed point cloud image to obtain the reference part point cloud image, and features are extracted from the point cloud image. The PointNet model is pre-trained using point cloud data for deep learning, and then the semantics of each discrete point in the preprocessed point cloud image is judged using the PointNet model to complete the semantic segmentation of the whole point cloud, obtain the reference part point cloud image and divide it into different semantic sections, and then obtain the intersection points of the chamfer section and the reference plane section in the above semantic sections as the preliminary feature points. On this basis, the straight line fitting intersection point method is used for feature point fine extraction to obtain the accurate feature point position.

[0098] In actual machining process, the system needs to adjust the posture according to the inclination characteristics of the reference frame to control the tool coordinate system set in the milling machining module and then drive the cutter to adjust to realize more accurate conformal machining. The traditional pose solving method is to calculate the inclination angle in the visual coordinate system by the line point cloud plane structure obtained by the line laser scanner, and then convert it to the tool coordinate system and the robot base coordinate system, calculate its pose state in space, and make the machining spindle corresponding to the tool coordinate system coincide with it to solve the tool coordinate system at each moment and the corresponding mechanical arm pose state. But this process often accompanies a lot of coordinate transformation process, longer coordinate conversion time and lower precision of the solution. Therefore, under the condition of ensuring the machining process, a relatively efficient and high-precision relative pose calculation method is proposed in combination with the actual machining process and control method.

[0099] The system sets the motion direction of the tool coordinate system to coincide with the base coordinate system and the main motion direction of actual machining, reduces the degrees of freedom of the posture transformation that need to be adjusted as much as possible, and limits the degrees of freedom of the posture transformation to adjustment of the pitch angle. Then, the system uses the consistency and continuity of the machined skin and the reference frame structure in the machining process to convert the complex posture change into a continuous change process based on features, that is, changes the absolute posture calculation at independent time to relative pose calculation in continuous time.

[0100] The time sequence-based point cloud relative pose calculation method mainly uses the point cloud features of adjacent time points, compares the posture changes of the two time points, and obtains the relative pose relationship of the tool coordinate system at the corresponding time point. At adjacent measurement time points, the visual measurement coordinate system will change in pose in the base coordinate system along with the change of the posture of the tool coordinate system at the end of the current mechanical arm, so for the measurement reference at adjacent time points, although the measured reference remains basically unchanged due to the consistency and continuity of its structure, the planar pose of the measured reference will change obviously in the visual measurement coordinate plane. Due to the machining process of the system, the milling tool path is determined by the plane outside the chamfer, so the feature corresponding to the plane structure will directly determine the direction of the tool spindle. In the above method, after feature point extraction and feature structure segmentation, the system performs linear fitting on the feature to obtain the point direction of the straight line where the feature is located, and obtains the unit plane vector of the visual coordinate system in the direction. Similarly, according to the point cloud image at the previous time point, the system can also extract the unit vector of the feature structure at the time point. As shown in the following formula:

[0101]

[0102] wherein, represents the feature vector a of the T i-1 time point, represents the tool coordinate system posture transformation from the T i-1 time point to the T i time point, which can be obtained by calculating the angle of the posture transformation of the end of the mechanical arm.

[0103] From the angle calculation formula, it can be known that:

[0104]

[0105] The rotation direction can be determined by the cross product of the vectors:

[0106]

[0107] After implementing feature point extraction based on the point cloud image and optimizing the feature point extraction, the feature point can more accurately describe the path, but there is still a certain degree of fluctuation. Therefore, the system will optimize and fit the feature points used to generate the path to ensure the accuracy and smoothness of the final path points in the online and offline modes.

[0108] On the basis of the original path point, the system adopts a point cloud smoothing filtering method, iteratively processes the path points using the mean filtering method, and obtains a point cloud path with stronger relevance and consistency, as shown in the following formula:

[0109]

[0110] That is, the system takes the target point as the center and performs mean processing on the points within a certain range in the neighborhood, where n is initially set to 3.

[0111] The above method can greatly optimize the processing of path points and accurately describe the machining path. Since the interval processing method is used during sampling and point cloud processing to increase the adaptability of the system, and the response delay caused by unstable processing time is avoided in a redundant cycle, the path point cannot be directly applied to the machining path guidance, and the system needs to re-segment the path point. Under the condition of known interval period, the system will process the path point using interpolation method, and under the above premise, the following formula is obtained:

[0112]

[0113] Where i = 0, 1, 2... 7, 8

[0114] In offline machining mode, the system will perform spatial high-order curve fitting based on path points on this basis, that is, using the least squares method to solve the overdetermined equation to obtain the final curve parameters. As shown in the following formula, for an N-order curve polynomial:

[0115]

[0116] That is, the above formula can be constructed into an equation group:

[0117] Y = XK;

[0118] Where Y is an [M x 1] matrix composed of parameter points, X is an [m x 6] matrix composed of parameter points, M represents the number of coordinate points participating in fitting, and K is a [6 x 1] coefficient matrix to be solved. Through the least squares solution theorem of overdetermined equations, we can get:

[0119] K = (X T X) -1 X T Y;

[0120] Then the coefficient matrix K is obtained through matrix operation.

[0121] Through the above method, the system will obtain the high-order fitting curve corresponding to the group of feature points, and the obtained path is proportionally divided through the quantitative intercepting method, so as to realize the re-division of the path points and provide accurate machining path guidance for the system.

[0122] The above is only a preferred embodiment of the present application, and does not limit the present application in any form. Any person skilled in the art can make some changes or modifications to the above disclosed technical content without departing from the technical solution of the present application to obtain equivalent embodiments with equivalent changes. However, any simple modification, equivalent change and modification of the above embodiments made according to the technical essence of the present application should be included in the protection scope of the claims of the present application.

Claims

1. An aircraft skin feature recognition and milling path planning method based on semantic segmentation, characterized in that, The method comprises: S1. obtaining an original point cloud image by using a line laser scanner, and preprocessing the original point cloud image; S2. performing semantic segmentation on the preprocessed point cloud image by using a PointNet model, obtaining a reference part point cloud image, and performing feature extraction on the point cloud image, including: S21. performing deep learning training on the PointNet model in advance by using point cloud data; S22. judging the semantics of each discrete point in the preprocessed point cloud image by using the PointNet model, completing semantic segmentation of the overall point cloud, obtaining the reference part point cloud image, and dividing the reference part point cloud image into different semantic sections; S23. obtaining the intersection point of the chamfer section and the reference upper plane section in the semantic sections as a preliminary feature point; S24. performing feature point fine extraction by using a straight line fitting intersection point method to obtain an accurate feature point position; S3. point cloud pose calculation based on time sequence, using point cloud features of adjacent time points, comparing the pose changes of the two to obtain the relative pose relationship of the tool coordinate system at the corresponding time, including: S31. setting the motion direction of the tool coordinate system to coincide with the base coordinate system and the main motion direction of the milling process; S32. establishing a tool coordinate system on the main shaft of the machining tool, which rotates with the pitch angle change of the reference machining surface of the reference part, and adjusting the base coordinate system and the tool coordinate system to a reasonable pose; S33. adjusting the pose with the main motion direction of the milling process as the axis; S34. in the online trimming mode, directly using the original point cloud image obtained by the line laser scanning to perform relative pose calculation based on time sequence; S4. milling edge path planning based on feature points, including: after obtaining the feature points, re-dividing the path points based on the feature points, using an interpolation method, and calculating the machining path by using the change trend of the feature points in the interpolation section; S5. machining trajectory fitting and re-segmentation.

2. The method of claim 1, wherein, The preprocessing of the original point cloud image in step S1 includes: S11. analyzing the point cloud results obtained by simulating the machining process; S12. pre-processing the original point cloud image by using threshold filtering and Gaussian filtering according to the analysis results of the previous step; S13. optimizing the original point cloud image by using a line laser scanner based on time sequence adjacent interval mean filtering method.

3. The method of claim 1, wherein, The deep learning training of the PointNet model in advance by using point cloud data in step S21 includes training parameters of the PointNet model, including: batch_size, Epoch, Learning_rate, Decay_rate, Npoint, Step_size, Lr_decay.

4. The method of claim 3, wherein, The semantic of each discrete point in the preprocessed point cloud image is judged by using the PointNet model in step S22, completing the semantic segmentation of the whole point cloud, the input of the PointNet model is the coordinate of the point cloud data, the output of the model is m×n scores, which respectively correspond to the scores of n input points relative to m categories; including: An affine transformation matrix (3x3) is estimated, which is multiplied by the input point cloud before feature extraction; Each of the n input points has 1024-dimensional features, and a global feature is obtained by selecting the maximum value in the n points; The global feature is copied n times and spliced with the features obtained by the second layer MLP, that is, the local features and global features of each point are spliced; After two MLPs, the final output with a dimension of m×n is obtained, that is, m classifications are performed for each point, and the predicted scores of each class corresponding to the point are output.

5. A semantic segmentation based aircraft skin feature identification and edge milling path planning system, characterized in that, The system realizes the steps of the aircraft skin feature recognition and milling path planning method based on semantic segmentation according to any one of claims 1-4 when running; The system is used for the skin frame structure of the aircraft, uses a line laser scanner to obtain skin trimming reference information, plans a machining path in real time according to a machining process, and guides a robot to complete a conformal motion task, with a KUKA industrial robot as a carrier; The system provides online or offline data processing modes; The system is composed of a software system and a hardware platform, wherein the software system is a data analysis and processing system located on a host computer; the hardware platform includes a host computer, a 3D line laser scanner, a KUKA industrial robot, a milling module, a visual calibration board, an end support mechanism, and a system assembly structure.

Images