Casting riser cleaning expert system design method based on multi-modal characteristics

Abstract

The invention discloses a multi-modal feature-based casting riser cleaning expert system design method, which comprises the following steps of: firstly, training a Faster R-CNN network based on the existing processing experience and data as samples, and storing the sample data into an expert database of an expert system; when a new casting to be cleaned is cleaned, the accurate position of the riser of the casting to be cleaned can be obtained by utilizing the trained Faster R-CNN network, the processing path of a sample with the maximum similarity with the riser of the casting to be cleaned is obtained according to the geometric features in the point cloud data of the riser region, and the final processing path is obtained on the basis of the processing path. And related data of the casting to be cleaned is added to the expert database, and the expert database is updated. According to the method, different types of data can be fused together, comprehensive cognition of the casting riser is formed, meanwhile, an accurate and flexible machining strategy can be provided for a robot, and the intelligent level of the cleaning process is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the fields of foundry cleaning, multimodal features, and artificial intelligence, and particularly to a design method for an expert system for cleaning casting risers based on multimodal features. Background Art

[0002] The cleaning of casting risers is a job with high repetition, high labor intensity, and poor working environment. With the intensification of population aging and low birth rate, using robot technology to achieve automatic cleaning of casting risers has become a trend. Traditional industrial robots rely on manual teaching and complete the processing of castings by manually recording the motion path. However, limited by the complex geometric shape of the castings and the variability of the casting process, the risers of different castings vary in shape and size, resulting in the inapplicability of the original teaching path to new castings. This highly manual teaching-dependent method lacks the adaptive ability in new scenarios and severely restricts the intelligent development of the foundry cleaning process.

[0003] During the automatic cleaning process, industrial robots usually use industrial cameras to obtain three-dimensional point cloud data to identify the parts to be processed on the castings. The point cloud data can provide the three-dimensional shape and spatial depth information of the casting risers, and has a high accuracy in describing complex geometric structures and surface details. However, relying solely on point cloud data for overall shape and pattern analysis has limitations, and it is difficult to effectively identify the overall appearance of the risers at the macroscopic level. At the same time, two-dimensional images have advantages in providing color and texture information, can capture the overall appearance and contour of the risers, but lack the detailed description of spatial depth.

[0004] In the prior art, to solve these problems, an expert system was introduced, but the existing expert systems cannot well combine point cloud data with two-dimensional image information, and the system has weak adaptive path planning ability and low intelligent level. Summary of the Invention

[0005] The present invention discloses a design method for an expert system for cleaning casting risers based on multimodal features to overcome the above technical problems.

[0006] To achieve the above object, the technical solution of the present invention is:

[0007] A design method for an expert system for cleaning casting risers based on multimodal features, comprising the following steps:

[0008] S1: Use manual teaching of an industrial camera to photograph all the risers on the casting according to a pre-set shooting path to obtain a grayscale image of the risers on the casting; mark the positions and riser types of the areas where the risers are located in the grayscale image of the risers, and then use manual teaching of the industrial camera to obtain the point cloud data of the areas where the risers are located;

[0009] S2: Based on the robot performing manual teaching cleaning operations on the riser, obtain the motion path for cleaning the riser; and store the grayscale image of the riser, the point cloud data of the area where the riser is located, the motion path for cleaning the riser, the position of the area where the riser is located in the grayscale image of the riser, and the riser type in the expert database of the expert system.

[0010] S3: Based on the grayscale image of the riser in the expert database, the riser type of the riser in the grayscale image, and the position of the area where the riser is located in the grayscale image, train the Faster R-CNN network to obtain the trained Faster R-CNN network.

[0011] S4: Use the manual teaching industrial camera to take pictures of all the risers on the casting to be cleaned according to the preset shooting path, obtain the grayscale images of the risers on the casting to be cleaned, and based on the trained Faster R-CNN network, obtain the position of the area where the riser is located and the riser type in the grayscale images of the risers on the casting to be cleaned.

[0012] S5: According to the position of the area where the riser is located in the grayscale image of the riser on the casting to be cleaned, obtain the point cloud data of the area where the riser of the casting to be cleaned is located and preprocess it.

[0013] S6: According to the preprocessed point cloud data of the area where the riser of the casting to be cleaned is located, obtain the average normal vector of the points in the point cloud data of the riser area, the variance of the normal vector distribution, the average curvature, and the variance of the curvature distribution.

[0014] S7: According to the average normal vector, the variance of the normal vector distribution, the average curvature, and the variance of the curvature distribution, obtain the similarity between the riser of the casting to be cleaned and the risers of the castings in the expert database; and obtain the motion path for cleaning the riser of the casting in the expert database with the greatest similarity to the riser of the casting to be cleaned, that is, the reference path.

[0015] S8: Obtain the center point of the point cloud data of the area where the riser of the casting to be cleaned is located, and perform translation and rotation on the point cloud data of the riser of the casting in the expert database with the greatest similarity to the riser of the casting to be cleaned, so as to map the ordered sequence P of the path points on the reference path j to the riser area of the casting to be cleaned, and obtain the ordered sequence of the path points on the transformed path.

[0016] S9: According to the ordered sequence of the path points on the transformed path, project the ordered sequence of the path points on the transformed path onto the point cloud surface of the riser of the casting to be cleaned to obtain the ordered sequence of the projected path points.

[0017] S10: According to the ordered sequence of the projected path points, use the cubic spline interpolation method to obtain the final machining path for cleaning the riser of the casting to be cleaned; and store the gray-scale image of the riser on the casting to be cleaned, the position and type of the riser area in the gray-scale image of the riser on the casting to be cleaned, the point cloud data of the riser area on the casting to be cleaned, and the final machining path for cleaning the riser of the casting to be cleaned in the expert database to update the expert database.

[0018] Further, the calculation formulas for the average normal vector, the variance of the normal vector distribution, the average curvature, and the variance of the curvature distribution are as follows:

[0019]

[0020] In the formula: represents the average normal vector of the points in the point cloud data of the riser area; N represents the total number of points in the point cloud data of the riser area; n represents the index of the points in the point cloud data of the riser area; represents the normal vector of the nth point in the point cloud data of the riser area; represents the variance of the surface normal vector distribution of the riser area; ‖·‖ represents the modulus of the vector;

[0021]

[0022]

[0023] In the formula: κ avg represents the average curvature of the points in the point cloud data of the riser area; represents the variance of the curvature distribution of all points in the riser area; κ n represents the curvature of the nth point in the point cloud data of the riser area.

[0024] Further, the formula for obtaining the similarity between the riser of the casting to be cleaned and the riser of the casting in the expert database is as follows:

[0025]

[0026] Among them,

[0027]

[0028] In the formula: S region represents the similarity between the riser of the casting to be cleaned and the riser of the casting in the expert database; exp(·) represents the natural exponential function e x ; is the normalized average normal vector; ∈ is a value used to prevent the denominator from being zero; and is the inverse term of variance; Norm(κ avg ) represents the normalized mean curvature; κ avg，j represents the mean curvature of the j-th riser sample in the expert database; represents the mean normal vector of the j-th riser sample in the expert database; C n is the average value of the normal vector features of the riser samples of the same type in the expert database; C k is the average value of the curvature features of the riser samples of the same type in the expert database.

[0029] Further, in S8, the formula for obtaining the center point of the point cloud data in the riser area of the casting to be cleaned is as follows:

[0030]

[0031] In the formula: C target represents the center point of the point cloud in the riser area of the casting to be cleaned; N represents the total number of points in the point cloud data in the riser area of the casting to be cleaned; n represents the index of the points in the point cloud data in the riser area of the casting to be cleaned; P n represents the coordinates of the n-th point in the point cloud data in the riser area of the casting to be cleaned;

[0032] The rotation axis, rotation angle, and rotation matrix for rotating the point cloud data of the riser of the casting in the expert database with the highest similarity to the riser of the casting to be cleaned are obtained using the following formulas respectively:

[0033]

[0034] where: r represents the unit vector of the rotation axis; θ represents the rotation angle; represents the mean normal vector of the j-th sample in the expert database; represents the mean normal vector of the points in the point cloud data of the riser area; r x represents the component of the rotation axis in the x direction; r y represents the component of the rotation axis in the y direction; r z represents the component of the rotation axis in the z direction;

[0035] R = I + sinθ·K + (1 - cosθ)·K 2

[0036]

[0037] where: R represents the rotation matrix; I represents the identity matrix; K represents the skew-symmetric matrix;

[0038] Then, the method for obtaining an ordered sequence of path points on the transformed path is as follows:

[0039] P′ m = R·(P m - C j ) + C target , m = 1, 2, …, M

[0040] where, P m is the m-th path point in the reference path, P′ m is the m-th path point in the transformed path, C j is the center point of the point cloud data of the riser of the casting in the expert database that has the highest similarity to the riser of the casting to be cleaned; m is the path point index in the sequence of machining path points for cleaning the riser of the casting in the expert database that has the highest similarity to the riser of the casting to be cleaned; M is the total number of path points in the sequence of machining path points.

[0041] Furthermore, the formula for obtaining an ordered sequence of projected path points is as follows:

[0042] P″ m = P′ m - d m ·n m

[0043] where: P″ m represents the m-th path point on the projected path; d m is the perpendicular distance from the m-th transformed path point to the point cloud surface of the riser of the casting to be cleaned, n m is the normal vector direction corresponding to the m-th transformed path point; P′ m is the m-th path point in the transformed path.

[0044] Beneficial effects: In the design method of a riser cleaning expert system for castings based on multi-modal features of the present invention, first, based on existing processing experience and data, grayscale images, point cloud data of the riser, the position and type of the riser in the grayscale image, and the motion path for cleaning the riser are obtained. These data are used as samples to train the Faster R-CNN network, and at the same time, these sample data are stored in the expert database of the expert system. When cleaning a new casting to be cleaned, the trained Faster R-CNN network can be used to obtain the position and type of the riser in the grayscale image of the casting to be cleaned. Further, based on the point cloud data in the riser position area, the average normal vector of the points in the point cloud data of the riser area, the variance of the normal vector distribution, the average curvature, and the variance of the curvature distribution are obtained. The similarity between the riser of the casting to be cleaned and the risers of the castings in the expert database is calculated, and the processing path of the sample in the expert database with the maximum similarity is obtained. Based on this, the final processing path for cleaning the riser of the casting to be cleaned is obtained, and the relevant data of the casting to be cleaned are added to the expert database to update the expert database. During the establishment of the expert database in the expert system of the present invention, different types of data are integrated to form a comprehensive understanding of the riser of the casting. During the use process, the generalization ability of the samples in the database is continuously increased, and at the same time, an accurate and flexible processing strategy can be provided for the robot, significantly improving the intelligent level of the cleaning process. Description of the Drawings

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0046] Figure 1 It is the design method of a riser cleaning expert system for castings based on multi-modal features of the present invention;

[0047] Figure 2 It is the schematic diagram of the design of a riser cleaning expert system for castings based on multi-modal features proposed in the embodiment of the present invention. Detailed Embodiments

[0048] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0049] This embodiment introduces a design method for an expert system for cleaning casting risers based on multimodal features. As Figure 1 and Figure 2 shown, it includes the following steps:

[0050] S1: Use an industrial camera with manual teaching to take pictures of all the risers on the casting according to a pre-set shooting path to obtain the grayscale images of the risers on the casting; manually mark the positions and riser types of the riser areas in the grayscale images of the risers, and then use an industrial camera with manual teaching to obtain the point cloud data of the riser areas.

[0051] Specifically, in this embodiment, an industrial camera with manual teaching is used to take pictures of the positions where each riser of the casting to be cleaned is located according to a pre-set shooting path to obtain the grayscale images of the risers. Further, the obtained grayscale images of the risers are returned to the upper computer, and median filtering is performed on the grayscale images of the casting in the upper computer to remove salt-and-pepper noise, that is, the possible noise points in the images, to improve the image quality. Among them, the filtering formula is: g(x,y) = median{I(x+i,y+j)}, that is, the pixel values corresponding to the pixel points (x+i,y+j) with offsets of i and j respectively relative to the pixel point with coordinates x,y in the grayscale image are arranged in size, and finally the middle value is taken to replace the grayscale value I(x,y) of the original grayscale image. Among them, g(x,y) represents the grayscale value of the pixel point with coordinates (x,y) in the filtered grayscale image of the riser; median{·} represents the operation of taking the median value, that is, the median value of all grayscale values in a specified neighborhood centered on the pixel point with coordinates (x,y), such as a 3×3 or 5×5 window, and this median value is used as the new filtered grayscale value of the pixel point with coordinates (x,y).

[0052] In this embodiment, for the same type of casting, such as a freight locomotive coupler or side frame, the risers that need to be cleaned are determined according to the process requirements, and these different types of risers are artificially named, such as the first type of riser, the second type of riser,..., the nth type of riser. Among them, the risers at the same position in the same type of casting are the same type of risers. Specifically, due to the limitations of the casting process, after the casting comes out, it is necessary to determine which parts of the risers need to be cleaned according to the process design requirements to ensure the smoothness of the casting surface and remove the redundant gating and risers.

[0053] Specifically, before cleaning, the industrial camera is manually taught to take pictures of each riser of each casting to obtain grayscale images of different types of risers; and the positions of the areas where the risers are located in the grayscale images of the risers are manually marked. Then, the industrial camera is used again through manual teaching to obtain the point cloud data (x n , y n , z n ) of the area where the riser is located, and a set of the grayscale images and point cloud data of the riser is established. Among them, I(x, y) represents the grayscale value of the pixel point with coordinates (x, y) in the grayscale image of the riser, x and y respectively represent the horizontal and vertical coordinates of the pixel point, and (x n , y n , z n ) represents the three-dimensional coordinates of the nth point in the point cloud. At the same time, for each riser, based on the manual teaching cleaning operation of the robot, the motion path for cleaning the riser is obtained, and the motion path of the robot is stored in the database together with the image, point cloud data and riser type of the riser in a tagged form.

[0054] S2: Based on the manual teaching cleaning operation of the robot for the riser, the motion path for cleaning the riser is obtained; the grayscale image of the riser, the point cloud data of the area where the riser is located, the motion path for cleaning the riser, the position of the area where the riser is located in the grayscale image of the riser, and the riser type are stored in the expert database;

[0055] S3: Based on the grayscale image of the riser in the expert database, the riser type of the riser in the grayscale image, and the position of the area where the riser is located in the grayscale image, the Faster R-CNN network is trained to obtain the trained Faster R-CNN network;

[0056] Specifically, when the dataset of the grayscale images of the risers reaches a certain scale, the positions of the risers in the grayscale images in the dataset are manually labeled, and a Fast Region-based Convolutional Neural Network (Faster R-CNN) network is trained using this dataset to achieve the detection and recognition of different types of risers in the grayscale images of the castings. The optimization objective of the Faster R-CNN network is L = L cls + λL reg , where L is the optimization objective of the Faster R-CNN network, L cls is the classification loss, and L regis the bounding box regression loss, and λ is the weight coefficient. Among them, the input of the Faster R-CNN network is the grayscale image of the riser, and the output is the type of the riser and the position of the area where the riser is located in the grayscale image. The position of the riser is the coordinate position of the detected riser, represented in the form of a rectangular bounding box. These coordinates are used to calibrate the specific position of the riser in the image, providing a reference for subsequent point cloud scanning and fine recognition.

[0057] S4: Use the artificial teaching industrial camera to take pictures of all the risers on the casting to be cleaned according to the preset shooting path, obtain the grayscale image of the risers on the casting to be cleaned, and obtain the position and type of the area where the risers are located in the grayscale image of the risers on the casting to be cleaned according to the trained Faster R-CNN network;

[0058] Specifically, during the process of cleaning the casting to be cleaned, first use the artificial teaching industrial camera to collect and denoise the grayscale image of the risers on the casting to be cleaned, and then input the grayscale image of the risers on the casting to be cleaned into the trained Faster R-CNN network to determine the position and type of the area where the risers are located in the grayscale image of the risers on the casting to be cleaned, that is, detect the position bounding box (xbbox, y, w, h) of the risers in the grayscale image of the risers on the casting to be cleaned, (x bbox , y bbox , w bbox , h bbox ), and identify the type T of the riser. Among them, x bbox represents the abscissa of the upper left corner of the bounding box; y bbox represents the ordinate of the upper left corner of the bounding box; w bbox represents the width of the bounding box; h bbox represents the height of the bounding box; provide a reference for subsequent point cloud scanning, thus saving scanning and processing time.

[0059] S5: Obtain the point cloud data of the area where the risers of the casting to be cleaned are located according to the position of the area where the risers are located in the grayscale image of the risers on the casting to be cleaned and preprocess it;

[0060] Specifically, at the position of the area where the risers are located in the grayscale image of the risers on the casting to be cleaned, start the artificial teaching industrial camera to perform point cloud scanning on this area, preprocess the collected point cloud, and obtain high-quality point cloud data. Then, based on features such as normal vectors and curvatures, match the point cloud with the existing samples in the expert library, find the most similar sample and extract the motion path as the reference path for the cleaning of the new riser, providing support for the subsequent planning of the cleaning path.

[0061] Specifically, for the point cloud data of the casting to be cleaned, conventional methods used in point cloud processing such as Gaussian filtering and point cloud thinning are first adopted for preprocessing to streamline the data volume of the point cloud and improve the data quality, which will not be described in detail here.

[0062] S6: According to the point cloud data of the riser area of the casting to be cleaned after preprocessing, obtain the average normal vector, variance of the normal vector distribution, average curvature, and variance of the curvature distribution of the points in the riser area point cloud data;

[0063] Preferably, for all the normal vectors of the points in the preprocessed point cloud data of the position of the riser area on the casting to be cleaned perform statistics, and calculate the average normal vector of the riser area on the casting to be cleaned and the variance of the normal vector distribution

[0064]

[0065] In the formula: represents the average normal vector of the points in the riser area point cloud data; N represents the total number of points in the riser area point cloud data; n represents the index of the points in the riser area point cloud data; represents the normal vector of the nth point in the riser area point cloud data; represents the variance of the surface normal vector distribution in the riser area, used to describe the degree of change in the normal distribution; ‖·‖ represents the modulus of the vector;

[0066] Meanwhile, calculate the curvature of all points in this riser area statistical characteristics of, including the average curvature κ avg and the variance of the curvature distribution

[0067]

[0068] In the formula: κ avg represents the average curvature of the points in the riser area point cloud data, used to describe the flatness and concavity of the area, represents the variance of the curvature distribution of all points in the riser area, used to reflect the degree of change in the surface curvature distribution of the area; κ n represents the curvature of the nth point in the riser area point cloud data;

[0069] S7: Obtain the similarity between the riser of the casting to be cleaned and the risers of the castings in the expert database based on the average normal vector, the variance of the normal vector distribution, the average curvature, and the variance of the curvature distribution; and obtain the motion path for cleaning the riser of the casting in the expert database with the highest similarity to the riser of the casting to be cleaned, that is, the reference path. Specifically, take the above statistical characteristics as feature points, and calculate the similarity S one by one with the feature points of j samples of the same type of castings in the expert database region . Finally, select the point with the highest similarity as the optimal matching point, and extract the corresponding robot motion path as the reference. Its calculation formula is:

[0070]

[0071] where

[0072]

[0073] In the formula: S region represents the similarity between the riser of the casting to be cleaned and the risers of the castings in the expert database; exp(·) represents the natural exponential function e x , where e is the natural constant is the normalized average normal vector; ∈ is a value used to prevent the denominator from being zero. Among them and are the inverse terms of the variance, used to dynamically adjust the weights of geometric characteristics in the matching process; Norm(κ avg ) represents the normalized average curvature; κ avg，j represents the average curvature of the j-th riser sample in the expert database represents the average normal vector of the j-th riser sample in the expert database; C n is the average value of the normal vector characteristics of the same type of riser samples in the expert database; C k is the average value of the curvature characteristics of the same type of riser samples in the expert database; among them, C n and C k are used to eliminate the difference in the magnitude of eigenvalue

[0074] Specifically, S region is the similarity score of the point cloud feature, ranging from [0,1]. The larger the value, the more similar the expert database sample and the sample to be matched is the normal vector difference term. If the normal vectors and are closer and the direction difference is smaller, the greater the contribution of this term to the similarity; -|κ avg -κ avg，j | is the curvature difference term, and the curvature κ aveand κ avg，j The closer they are, the smaller the curvature difference, and the greater the contribution of this item to the similarity. Variance can be used to adjust the importance of geometric features in the matching process. If the variance of a region is large (the features change violently and unstably), the weights of the normal vector and curvature can be reduced in the matching to reduce the influence of unreliable features. The normalization term mainly aims to control within a reasonable range when calculating the weights, so as to avoid certain features (such as the normal vector or curvature) having an excessive impact on the similarity calculation due to being too large or too small in magnitude.

[0075] Specifically, after matching the riser sample in the expert database that is most similar to the casting to be cleaned, the corresponding robot motion path is extracted as a reference, and based on the original reference path, the path is further optimized according to the features shown by the current point cloud to better meet the processing requirements of the new riser. After the processing is completed, the host computer will automatically feedback the cleaning path and effect of the new riser to the expert database, thereby increasing the number and quality of samples in the expert database and improving the generalization and robustness of the expert system.

[0076] Specifically, through the similarity S region find the most matching riser sample of the casting to be cleaned in the expert database is the j-th riser sample After successful matching, extract the processing path P of the j-th riser sample in the expert database j as the reference path. Among them, the expert database sample path is usually stored as an ordered sequence of points:

[0077] P j ={P 1 ,P 2 ,…,P m …,P M}

[0078] Among them, P j is the processing path point sequence of the j-th riser sample in the expert database; P m is the m-th path point in the processing path point sequence; m is the path point index in the processing path point sequence; M is the total number of path points in the processing path point sequence; among them, P m =(x m ,y m ,z m ) is a path point in three-dimensional space, and x m ,y m ,z m are the x-axis coordinate, y-axis coordinate, and z-axis coordinate of the m-th path point in the processing path point sequence respectively;

[0079] S8: Obtain the center point of the point cloud data of the riser area of the casting to be cleaned, and translate and rotate the point cloud data of the riser of the casting in the expert database with the highest similarity to the riser of the casting to be cleaned, so as to map the ordered sequence P of the path points on the reference path j to the target area of the casting to be cleaned, and obtain the ordered sequence of the path points on the transformed path;

[0080] Specifically, in order to apply the reference path to the target area, coordinate system alignment and path transformation are required to make the reference path consistent with the feature center and direction of the target area. It mainly includes the following steps:

[0081] Obtain the center point of the casting to be cleaned, and the formula used is as follows:

[0082] Among them, the center point C of the riser of the casting to be cleaned target is the geometric center of the point cloud area to be matched, and is obtained by taking the average of the coordinates of all points in the point cloud data of the riser of the casting to be cleaned:

[0083]

[0084] In the formula: In the formula: C target represents the center point of the point cloud of the riser area of the casting to be cleaned; N represents the total number of points in the point cloud data of the riser area of the casting to be cleaned; n represents the index of the points in the point cloud data of the riser area of the casting to be cleaned; P n represents the coordinate of the nth point in the point cloud data of the riser area of the casting to be cleaned;

[0085] Next, taking C target as the center, rotate the point cloud data of the riser of the casting in the expert database with the highest similarity to the riser of the casting to be cleaned, so as to align the normal vector direction of the reference path and the normal vector direction of the casting to be cleaned;

[0086] The normal vector direction of the reference path and the normal vector direction of the target area may be different, and they need to be aligned by rotation.

[0087] Preferably, the rotation axis for rotating the point cloud data of the riser of the casting in the expert database with the highest similarity to the riser of the casting to be cleaned is: The rotation angle is:

[0088] Among them, r represents the unit vector of the rotation axis; θ represents the rotation angle; represents the average normal vector of the jth sample in the expert database; The average normal vector of the points in the point cloud data representing the riser region;

[0089] The rotation matrix (Rodrigues formula) is: R = I + sinθ·K + (1 - cosθ)·K 2

[0090] Where: r = (r x , r y , r z ) is the unit vector of the rotation axis.

[0091] Where, R represents the rotation matrix; I represents the identity matrix. In three-dimensional space, the identity matrix is a 3×3 matrix with diagonal elements equal to 1 and non-diagonal elements equal to 0; K represents the skew-symmetric matrix; r x represents the component of the rotation axis in the x direction; r y represents the component of the rotation axis in the y direction; r z represents the component of the rotation axis in the z direction;

[0092] Through the rotation matrix R and the translation vector (C target - C j ), the ordered sequence P j of the points on the reference path is mapped to the target region of the casting to be cleaned, and the ordered sequence of the path points on the transformed path is obtained;

[0093] P′ m = R·(P m - C j ) + C target , m = 1, 2, …, M

[0094] Where, P m is the m-th path point in the reference path, P′ m is the m-th path point in the transformed path, C j is the center point of the point cloud data of the riser of the casting in the expert database that has the highest similarity to the riser of the casting to be cleaned; m is the path point index in the machining path point sequence for cleaning the riser of the casting in the expert database that has the highest similarity to the riser of the casting to be cleaned; M is the total number of path points in the machining path point sequence; among them, the total number of path points on the transformed path is the same as the total number of path points on the reference path.

[0095] Furthermore, due to the fact that the geometric characteristics of the target region may have slight differences from those of the reference region, directly using the transformed path may result in a path that is not smooth enough or does not fully conform to the surface of the target region. Therefore, it is necessary to further optimize and smooth the path.

[0096] S9: Obtain an ordered sequence of path points on the transformed path, project the ordered sequence of path points on the transformed path onto the point cloud surface of the riser of the casting to be cleaned, and obtain the ordered sequence of projected path points. Specifically, project the transformed path point cloud {P′ m} onto the point cloud surface of the riser of the casting to be cleaned, ensuring that the machining path points for finally machining the riser of the casting to be cleaned fit the surface of the target area. The projection formula is: P″ m = P′ m - d m ·n m

[0097] where: P″ m represents the m-th path point on the projected path; d m is the vertical distance from the m-th path point after transformation to the point cloud surface of the riser of the casting to be cleaned, and n m is the normal vector direction corresponding to the m-th path point after transformation; P′ m is the m-th path point in the transformed path.

[0098] S10: According to the ordered sequence of the projected path points, use the cubic spline interpolation method to obtain the final machining path for cleaning the riser of the casting to be cleaned; and store the grayscale image of the riser on the casting to be cleaned, the position and riser type of the area where the riser is located in the grayscale image of the riser on the casting to be cleaned, the point cloud data of the area where the riser of the casting to be cleaned is located, and the final machining path for cleaning the riser of the casting to be cleaned in the expert database to update the expert database.

[0099] Next, further, in order to ensure the continuity and stability of the robot movement, it is necessary to smooth the path points by using the cubic spline interpolation method:

[0100] Cubic spline formula: S(t) = at 3 + bt 2 + ct + d, t ∈ [0, 1]

[0101] where S(t) is the interpolation curve, and a, b, c, d are all interpolation coefficients; among them, the interpolation coefficients a, b, c, d are obtained by fitting the path points. t represents the independent variable;

[0102] After the above steps, the final generated machining path sequence for cleaning the riser of the casting to be cleaned is expressed as follows

[0103] P final = {P″ 1 , P″ 2 , …, P″ K′}

[0104] Among them, K′ is the number of optimized path points.

[0105] Finally, after the robot completes the automatic cleaning according to the generated path, the host computer stores the newly generated cleaning path together with the image and point cloud data in the expert database to enrich the number of samples and improve the generalization ability of the expert system.

[0106] In summary, in the embodiments of the present invention, there are mainly the following parts:

[0107] (1) Construction of expert database data: The expert database is the cornerstone and prerequisite for the riser cleaning expert system based on multi-modal features in this embodiment. The expert database contains grayscale images and point cloud data of various types of risers in a large number of castings, as well as the movement paths of the robot during manual teaching for cleaning each riser. During the manual teaching stage, a camera is used to collect two-dimensional grayscale image and point cloud data information of different types of risers, and at the same time, the path used by the robot to clean the corresponding riser is recorded, and then it is stored in the database using a labeled and structured data storage method for subsequent matching and retrieval. Among them, the grayscale image, point cloud, and processing path of the same riser correspond to each other and are stored in the expert database according to the label of the category. At the same time, after accumulating enough data, a Faster R-CNN network is trained using the grayscale image dataset for riser detection and recognition during the subsequent cleaning process. The construction of the expert database effectively transforms human experience into content that the robot can learn and imitate, and as a kind of pre-training, it greatly improves the real-time performance and reliability of the robot in the casting cleaning operation.

[0108] (2) Coarse image positioning: For a brand-new casting to be cleaned, during the coarse image positioning process, the industrial camera captures the original grayscale image of the riser through manual teaching and performs noise reduction processing, and then inputs it into the trained Faster R-CNN network to detect the position and type of the riser, laying a foundation for the next point cloud scanning and matching. This process effectively utilizes the advantages of two-dimensional images in macroscopic performance, which is more efficient than directly using point cloud data and saves the time for scanning and processing a large number of original castings.

[0109] (3) Fine point cloud recognition: After obtaining the position and type information of the riser, the camera re-takes a fine photo of the riser to obtain point cloud data, and combines the normal vector and curvature features of the point cloud at the riser part. Through a similarity formula based on dynamic weight change, it is compared with the samples in the expert database, and the most similar riser point cloud sample is found from the expert database and its processing path is obtained. This process fully utilizes the advantages of point cloud data in describing spatial information. At the same time, the proposed similarity formula based on dynamic weight improves the efficiency and accuracy of point cloud sample matching.

[0110] (4) Path generation: The machining path of the most similar riser obtained from the expert database is used as the reference path. At the same time, the original path is mapped to the new point cloud space by using coordinate system alignment and path transformation. Further, according to the specific characteristics of the new riser, the interpolation algorithm is used to adjust the cleaning path of the new riser, and a new path is adaptively planned on the basis of the reference path to meet the cleaning requirements, ensuring the safety of the robot movement and the reliability of the riser cleaning. Finally, according to the specific effect of the casting cleaning, the newly planned path, the corresponding grayscale image and point cloud data are stored in the expert database together, so as to enrich the data in the expert database and improve the generalization performance of the expert system. Based on the data in the expert database, the adaptive update of the expert system is realized.

[0111] A design method of an expert system for cleaning casting risers based on multi-modal features in this embodiment first establishes an expert database containing casting samples processed by a robot through manual teaching. A corresponding relationship is established for the grayscale image, point cloud data, and robot machining path of each sample riser. When cleaning a new casting, the two-dimensional grayscale image and three-dimensional point cloud data captured by the camera are respectively used to perform rough positioning and fine recognition of the casting riser, so as to obtain the type and spatial geometric information of the riser. Then, these information are used as the basis to compare with the samples in the expert database, find the most similar riser, and generate a new machining path on the basis of its existing path to realize the adaptive cleaning of the robot. Compared with the method of planning the path from scratch in this embodiment, based on the sample data in the expert database in the expert system, the replanning on the original path has a smaller calculation amount and better real-time performance, and at the same time has the advantage of adaptive adjustment. At the same time, the cleaning path and effect of the new riser are fed back to the expert library, enriching the sample size of the expert system and further improving the generalization ability of the expert system and the intelligence level of the robot.

[0112] This embodiment integrates the grayscale image and point cloud features of the riser in the robot casting cleaning operation, improves the recognition efficiency of the riser to be machined, and dynamically generates a new robot motion path, greatly improving the adaptability and autonomy of the robot system.

[0113] A design method of an expert system for cleaning casting risers based on multi-modal features in this embodiment is as follows. First, based on existing processing experience and data, the grayscale image of the riser, point cloud data, the position and type of the riser in the grayscale image, and the motion path for cleaning the riser are obtained. These data are used as samples to train the Faster R-CNN network, and at the same time, these sample data are stored in the expert database of the expert system. When cleaning a new casting to be cleaned, the trained Faster R-CNN network can be used to obtain the position and type of the riser in the grayscale image of the casting to be cleaned. Further, based on the point cloud data in the riser position area, the average normal vector of the points in the point cloud data of the riser area, the variance of the normal vector distribution, the average curvature, and the variance of the curvature distribution are obtained. The similarity between the riser of the casting to be cleaned and the risers of the castings in the expert database is calculated, and the processing path of the sample in the expert database with the maximum similarity is obtained. Based on this, the final processing path for cleaning the riser of the casting to be cleaned is obtained, and the relevant data of the casting to be cleaned are added to the expert database to update the expert database. During the establishment of the expert database in the expert system of the present invention, different types of data are fused together to form a comprehensive understanding of the casting riser. During the use process, the generalization ability of the samples in the database is continuously increased. At the same time, the knowledge base and reasoning mechanism can be used to provide precise and flexible processing strategies for the robot, significantly improving the intelligent level of the cleaning process.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A casting riser cleaning expert system design method based on multi-modal features, characterized in that: The steps include: S1: Using a manual teaching industrial camera, photograph all risers on the casting according to a pre-set shooting path to obtain a grayscale image of the risers on the casting; marking the position and type of the riser area in the grayscale image of the riser, and then using a manual teaching industrial camera to obtain point cloud data of the riser area; S2: Based on the robot, a manual teaching cleaning operation is performed on the riser to obtain a motion path for cleaning the riser; and the grayscale image of the riser, the point cloud data of the area where the riser is located, the motion path for cleaning the riser, the position of the area where the riser is located in the grayscale image of the riser, and the riser type are stored in an expert database of the expert system; S3: Based on the grayscale image of the riser in the expert database, the riser type of the riser in the grayscale image, and the location of the region where the riser is located in the grayscale image, a Faster R-CNN network is trained to obtain a trained Faster R-CNN network; S4: using the artificial teaching industrial camera to shoot all the risers on the casting to be cleaned according to a preset shooting path, obtaining a grayscale image of the risers on the casting to be cleaned, and obtaining the position of the riser area and the riser type in the grayscale image of the risers on the casting to be cleaned according to the trained Faster R-CNN network; S5: acquiring point cloud data of the region where the riser of the casting to be cleaned is located and preprocessing the data according to the position of the region where the riser is located in the grayscale image of the riser on the casting to be cleaned; S6: according to the pre-processed point cloud data of the riser region of the casting to be cleaned, obtaining the average normal vector of the points in the point cloud data of the riser region, the variance of the normal vector distribution, the average curvature, and the variance of the curvature distribution; S7: obtaining the similarity between the riser of the casting to be cleaned and the risers of the castings in the expert database according to the average normal vector, the variance of the normal vector distribution, the average curvature, and the variance of the curvature distribution; and obtaining the motion path for cleaning the riser of the casting in the expert database that has the greatest similarity to the riser of the casting to be cleaned, i.e., the reference path; S8: Obtain the center point of the point cloud data of the region where the riser of the casting to be cleaned is located, and translate and rotate the point cloud data of the riser of the casting in the expert database with the greatest similarity to the riser of the casting to be cleaned, so as to convert the ordered sequence of path points on the reference path P j Mapping to the riser region of the casting to be cleaned, obtaining an ordered sequence of path points on the transformed path; S9: According to the ordered sequence of path points on the transformed path, the ordered sequence of path points on the transformed path is projected onto the point cloud surface of the riser of the casting to be cleaned to obtain the ordered sequence of the projected path points; S10: According to the ordered sequence of the projected path points, a cubic spline interpolation method is used to obtain a final processing path for cleaning the riser of the casting to be cleaned; and the grayscale image of the riser on the casting to be cleaned, the position and type of the riser area in the grayscale image of the riser on the casting to be cleaned, the point cloud data of the riser area of ​​the casting to be cleaned, and the final processing path for cleaning the riser of the casting to be cleaned are stored in an expert database to update the expert database.

2. The method for designing a casting riser cleaning expert system based on multimodal features according to claim 1, characterized in that: The calculation formulas for the average normal vector, the variance of the normal vector distribution, the average curvature, and the variance of the curvature distribution are as follows: Where: represents the average normal vector of the points in the point cloud data of the riser area; N represents the total number of points in the point cloud data of the riser area; n represents the index of the point in the point cloud data of the riser area; Represents the normal vector of the nth point in the point cloud data of the riser area; represents the variance of the surface normal vector distribution in the riser area; ‖·‖ represents the modulus of the vector; Where: κ avg The average curvature of the points in the point cloud data representing the riser area; represents the variance of the curvature distribution of all points in the riser area; κ n Represents the curvature of the nth point in the point cloud data of the riser area.

3. The method for designing a casting riser cleaning expert system based on multimodal features according to claim 1, characterized in that: The formula used to obtain the similarity between the riser of the casting to be cleaned and the riser of the casting in the expert database is as follows: in, Where: S regin represents the similarity between the riser of the casting to be cleaned and the riser of the casting in the expert database; exp(·) represents the natural exponential function e x ; is the normalized average normal vector; ∈ is the value used to prevent the denominator from being zero; and is the inverse term of the variance; Norm(κ avg ) represents the normalized mean curvature; κ avg,j represents the average curvature of the j-th riser sample in the expert database; represents the average normal vector of the j-th riser sample in the expert database; C n is the average value of the normal vector features of the same type of riser samples in the expert database; C k It is the average value of the curvature characteristics of the same type of riser samples in the expert database.

4. The method for designing a casting riser cleaning expert system based on multimodal features according to claim 1, characterized in that: In S8, the formula used to obtain the center point of the point cloud data of the region where the riser of the casting to be cleaned is located is as follows: Where: C target represents the center point of the point cloud of the region where the riser of the casting to be cleaned is located; N represents the total number of points in the point cloud data of the region where the riser of the casting to be cleaned is located; n represents the index of the point in the point cloud data of the region where the riser of the casting to be cleaned is located; P n The coordinates of the nth point in the point cloud data representing the region where the riser of the casting to be cleaned is located; The rotation axis, rotation angle and rotation matrix of the point cloud data of the riser of the casting in the expert database with the greatest similarity to the riser of the casting to be cleaned are respectively obtained by the following formulas: Where: r represents the unit vector of the rotation axis; θ represents the rotation angle; represents the average normal vector of the jth sample in the expert database; The average normal vector of the points in the point cloud data representing the riser area; r x represents the component of the rotation axis in the x direction; r y It represents the component of the rotation axis in the y direction; r z It represents the component of the rotation axis in the z direction; R=I+sinθ·K+(1-cosθ)·K 2 Where: R represents the rotation matrix; I represents the identity matrix; K represents the antisymmetric matrix; Then, the ordered sequence of path points on the transformed path is obtained as follows: P′ m =R·(P m -C j )+C target ,m=1,2,…,M Among them, P m is the mth path point in the reference path, P′ m is the mth path point in the transformed path, C j is the center point of the point cloud data of the riser of the casting in the expert database that has the greatest similarity to the riser of the casting to be cleaned; m is the path point index in the processing path point sequence for cleaning the riser of the casting in the expert database that has the greatest similarity to the riser of the casting to be cleaned; M is the total number of path points in the processing path point sequence.

5. The method for designing a casting riser cleaning expert system based on multi-modal features according to claim 1, characterized in that: The formula used to obtain the ordered sequence of projected path points is as follows: P″ m =P′ m -d m ·n m Where: P″ m represents the mth path point on the projected path; d m is the vertical distance from the transformed mth path point to the point cloud surface of the riser of the casting to be cleaned, n m is the normal vector direction corresponding to the mth path point after transformation; P′ m is the mth path point in the transformed path.

Images