Real-time visual positioning method and system for industrial automation

Abstract

The invention provides a real-time visual positioning method and system for industrial automation, and the method comprises the steps: firstly collecting real-time image streams of a machining object under multiple working conditions, including surface state images at different machining stages, tool interaction images and corresponding working condition identifiers, and collecting time sequence information, then carrying out the working condition adaptability feature analysis of the real-time image streams, and carrying out the real-time visual positioning of the real-time image streams; generating a surface state feature set and a tool interaction feature set, then performing bidirectional verification on the two feature sets based on a machining precision constraint condition, determining an effective positioning feature and outputting a core positioning region description, and constructing a machining positioning reference frame and a dynamic adjustment rule set according to the core positioning region description in combination with machining equipment motion constraint parameters; and finally, related information is fused to generate a positioning guide instruction sequence containing the tool displacement correction and the feeding rhythm adjustment value, the positioning guide instruction sequence is transmitted to a machining control system to execute closed-loop positioning guide action, and efficient and accurate machining positioning guide is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a real-time visual positioning method and system for industrial automation. BACKGROUND

[0002] In the field of automated processing, processing positioning guidance is a key link to ensure processing precision and product quality. Traditional processing positioning guidance methods mostly rely on pre-set fixed reference points and simple sensor feedback. For example, in some mechanical processing, fixed points are marked on the processing object, and light-sensing sensors or contact sensors are used to detect the positions of these points to achieve positioning. However, these methods have many limitations in actual application.

[0003] On the one hand, traditional methods are difficult to adapt to processing needs under multiple working conditions. In actual processing, the processing object will be in different processing stages and will be affected by various working condition factors such as temperature, pressure, and vibration, resulting in complex changes in its surface state and tool interaction. Traditional methods cannot effectively capture these changes and still use fixed reference points and methods for positioning, which can easily cause positioning deviation and affect processing precision.

[0004] On the other hand, traditional positioning guidance lacks dynamic adjustment and bidirectional verification mechanisms. In the processing process, the interaction between the tool and the processing object may produce unexpected situations such as tool wear and processing object deformation, which can affect the accuracy of positioning. However, traditional methods cannot perform bidirectional verification of the surface state and tool interaction in real time, timely detect and solve feature conflict problems, and dynamically adjust positioning parameters according to real-time feedback, making it difficult to ensure the stability and reliability of the processing process. SUMMARY

[0005] In view of the above-mentioned problems, in combination with the first aspect of the present application, the embodiments of the present application provide a real-time visual positioning method for industrial automation, which comprises: Collecting real-time image streams of the processing object under multiple working conditions, the real-time image streams containing surface state images and tool interaction images of the processing object at different processing stages, and carrying corresponding working condition identifiers and collection time sequence information; Performing working condition adaptability feature analysis on the real-time image streams, eliminating the influence of working condition differences through dynamic image comparison, and generating a surface state feature set and a tool interaction feature set of the processing object; Performing bidirectional verification analysis on the surface state feature set and the tool interaction feature set based on processing precision constraint conditions, determining effective positioning features through a feature conflict resolution mechanism, and outputting a core positioning area description of the processing object; constructing a machining positioning reference framework and a dynamic adjustment rule set according to the core positioning area description and in combination with motion constraint parameters of the machining equipment; fusing the machining positioning reference framework, the dynamic adjustment rule set and feedback features of the real-time image stream to generate a positioning guidance instruction sequence containing a tool displacement correction amount and a feed rhythm adjustment value, and transmitting the positioning guidance instruction sequence to a machining control system to perform a closed-loop positioning guidance action.

[0006] In still another aspect, the embodiments of the present application also provide a real-time visual positioning system for industrial automation, which comprises a processor and a machine readable storage medium, the machine readable storage medium is connected with the processor, the machine readable storage medium is used for storing programs, instructions or codes, and the processor is used for executing the programs, instructions or codes in the machine readable storage medium to realize the above-mentioned method.

[0007] Based on the above aspects, the embodiments of the present application can generate accurate surface state feature sets and tool interaction feature sets by collecting real-time image streams of machining objects under multiple working conditions, covering surface state images and tool interaction images in different machining stages, and carrying working condition identification and acquisition timing information, and can effectively overcome the positioning deviation problem caused by working condition changes in traditional methods by performing working condition adaptability feature analysis on the real-time image streams and eliminating the influence of working condition differences by dynamic image comparison. The two feature sets are subjected to bidirectional verification analysis based on machining precision constraint conditions, and effective positioning features are determined through feature conflict resolution mechanism, and a core positioning area description is output, which greatly improves the accuracy and reliability of positioning. The machining positioning reference framework and the dynamic adjustment rule set constructed according to the core positioning area description in combination with the motion constraint parameters of the machining equipment enable the positioning guidance to be flexibly adjusted according to the actual situation. Finally, the positioning guidance instruction sequence is generated by fusing the machining positioning reference framework, the dynamic adjustment rule set and the feedback features of the real-time image stream, and is transmitted to the machining control system to perform a closed-loop positioning guidance action, realizing intelligent, dynamic and accurate machining positioning guidance, effectively improving machining efficiency and product quality, and reducing waste rate and production cost. BRIEF DESCRIPTION OF DRAWINGS

[0008] Figure 1 is an execution flow diagram of the real-time visual positioning method for industrial automation provided by the embodiments of the present application.

[0009] Figure 2 is a schematic diagram of exemplary hardware and software components of the real-time visual positioning system for industrial automation provided by the embodiments of the present application. DETAILED DESCRIPTION

[0010] The present application will be described in detail below with reference to the accompanying drawings, Figure 1is a flowchart of a real-time visual positioning method for industrial automation provided by an embodiment of the present application. The real-time visual positioning method for industrial automation will be described in detail below.

[0011] Step S110: Collecting a real-time image stream of the processing object under multiple working conditions, the real-time image stream containing surface state images and tool interaction images of the processing object at different processing stages, and carrying corresponding working condition identifiers and collection timing information.

[0012] In this embodiment, multiple image collection devices can be selected and installed at different positions of the processing equipment to ensure that the processing area of the processing object can be covered from multiple angles. The lens parameters of the image collection devices are adjusted according to the size of the processing object and the range of the processing area to ensure that the collected images can clearly present the details of the surface of the processing object and the interaction between the tool and the processing object.

[0013] Then, the image collection device is started and works continuously at a set collection frequency. During the collection process, the processing object will go through different processing stages, such as the initial placement stage, the preliminary docking stage, the fine adjustment stage, etc. The image collection device can record the surface state of the processing object at each stage, including whether there are scratches, stains, unevenness on the surface, and texture changes on the surface, etc.

[0014] At the same time, the image collection device can also capture the interaction process between the tool and the processing object, such as the images of the tool approaching the processing object, contacting the processing object, and operating the processing object, etc. Each collected image is assigned a corresponding working condition identifier, which is determined according to the parameters during processing, such as the temperature and humidity of the processing environment, the type of the tool, the pressure during processing, etc. Different parameter combinations correspond to different working condition identifiers.

[0015] In addition, each image can also record the collection timing information, which is accurate to the collection time, so as to facilitate subsequent timing analysis of the image stream. When the processing object surface may contain private information, data encryption technology is used to encrypt the collected image data to prevent information leakage during transmission and storage.

[0016] Step S120: Analyzing the working condition adaptability features of the real-time image stream, eliminating the influence of working condition differences through dynamic image comparison, and generating a surface state feature set and a tool interaction feature set of the processing object.

[0017] In this embodiment, first, the real-time image stream is preprocessed to remove noise in the image. The median filtering method can be used, which traverses each pixel point in the image, sorts the pixel values within a certain range around the pixel point, and takes the middle value as the new value of the pixel point, thereby reducing the salt and pepper noise and other interference in the image.

[0018] Then, the preprocessed real-time image stream is classified and sorted according to the working condition identifier, and images belonging to the same working condition are grouped together. Through the above classification, images under different working conditions can be distinguished, which facilitates feature analysis according to the characteristics of each working condition.

[0019] Step S121: segmenting the real-time image stream according to the working condition identifier to obtain image sub-sequences corresponding to each working condition, and extracting the gray scale distribution features and edge gradient features of each image sub-sequence.

[0020] According to the different working condition identifiers, the real-time image stream is divided into multiple continuous image sub-sequences, each image sub-sequence corresponding to a specific working condition. For example, all images with working condition identifier A form an image sub-sequence, all images with working condition identifier B form another image sub-sequence, and so on.

[0021] For each image sub-sequence, calculate its gray scale distribution feature. Count the gray scale value of each pixel in the image sub-sequence to obtain the range of gray scale value and the frequency of each gray scale value in the image, and then form a gray scale histogram. Through the gray scale histogram, the overall brightness distribution of the image sub-sequence can be intuitively understood.

[0022] When extracting the edge gradient feature, the Sobel operator is used to process each frame of image in the image sub-sequence. The gradient values in the horizontal and vertical directions of the image are calculated respectively, and then the gradient amplitude and gradient direction of each pixel point are calculated based on the gradient values in the two directions. The gradient amplitude reflects the intensity of the edge at the pixel point, and the gradient direction indicates the direction of the edge. The above information collectively constitutes the edge gradient feature.

[0023] For example, step S1211: performing gray scale histogram equalization processing on the image sub-sequence of each working condition to obtain a standardized gray scale histogram, and extracting the peak position and distribution width parameters of the standardized gray scale histogram.

[0024] For each frame of image in the image sub-sequence of each working condition, perform gray scale histogram equalization processing. Calculate the cumulative distribution function of the image, and map the gray scale values of the original image according to the cumulative distribution function, so that the gray scale value distribution of the processed image is more uniform, and the contrast of the image is enhanced.

[0025] After the equalization processing, a standard gray histogram is obtained. The position of the highest frequency of the gray value in the histogram is found, which is the peak position. At the same time, the range of the gray value distribution in the standard gray histogram is calculated. The interval length from the minimum gray value to the maximum gray value is the distribution width parameter, which can reflect the gray distribution characteristics of the equalized image.

[0026] Step S1212: A similarity evaluation function based on Bhattacharyya distance is constructed with the peak position and the distribution width parameter as inputs. The output value of the similarity evaluation function is positively correlated with the gray distribution similarity of the two image subsequences.

[0027] The peak position and the distribution width parameter of the standard gray histograms of the two image subsequences are taken as inputs and substituted into the calculation formula of the Bhattacharyya distance. The calculation of the Bhattacharyya distance is based on the overlap degree between the two probability distributions. The product of the probability values of the corresponding gray levels of the two histograms is taken, and then the sum is taken, and then the negative logarithm is taken to obtain the Bhattacharyya distance.

[0028] The constructed similarity evaluation function is based on the Bhattacharyya distance. The Bhattacharyya distance is converted so that the output value of the function increases as the Bhattacharyya distance decreases. That is, the more similar the gray distributions of the two image subsequences are, the greater the output value of the similarity evaluation function is, so as to realize the positive correlation between the output value and the gray distribution similarity.

[0029] Step S1213: The image subsequence of the reference condition is selected as the reference template, and the similarity values of the image subsequences of other conditions with the reference template are calculated.

[0030] A representative condition is selected from all conditions as the reference condition, and the image subsequence corresponding to the condition is determined as the reference template. Then, the image subsequences of each of the other conditions are sequentially input into the constructed similarity evaluation function together with the reference template.

[0031] The similarity values between each of the image subsequences of the other conditions and the reference template are calculated by the function. These similarity values can quantify the similarity degree of the gray distributions between the image subsequences of different conditions and the image subsequence of the reference condition.

[0032] Step S1214: The image subsequences are sorted in descending order of the similarity values, and the top N image subsequences are selected together with the reference template to form an initial image pair subset.

[0033] All the similarity values of the image subsequences of the other conditions calculated with the reference template are arranged in descending order. A number N is set according to actual needs. The top N image subsequences are selected, and they are respectively combined with the reference template to form image pairs. These image pairs together form an initial image pair subset.

[0034] The initial image pair subset contains image information of multiple working conditions most similar to the reference working condition gray scale distribution.

[0035] Step S1215: The initial image pair subset is subjected to secondary verification, the gray scale difference square sum of corresponding pixel points in the image pair is calculated, the image pair whose gray scale difference square sum exceeds the threshold value is removed, the image pair passing the secondary verification is retained to form the final image pair subset, and the image pairs in the image pair subset have similar gray scale distribution characteristics and edge structure characteristics.

[0036] For each image pair in the initial image pair subset, the gray scale difference of the corresponding position pixels of the two images is calculated pixel by pixel, and then the sum of the squares of these differences is calculated to obtain the gray scale difference square sum. A threshold value is set, which is determined according to the noise level of the image and the requirement for image similarity.

[0037] The gray scale difference square sum of each image pair is compared with the threshold value. If it exceeds the threshold value, it means that the gray scale difference of the image pair at the pixel level is large, and it is removed from the initial image pair subset. The image pair whose gray scale difference square sum does not exceed the threshold value is retained, and the above image pair forms the final image pair subset, which is not only similar in gray scale distribution characteristics, but also has high consistency in edge structure characteristics.

[0038] Step S122: Based on the gray scale distribution characteristics, an image similarity evaluation model between working conditions is constructed, the similarity values of different working condition image subsequences are calculated, and the image pair subset with a similarity value higher than a similarity threshold value is screened.

[0039] Combined with the previously extracted gray scale distribution characteristics such as the peak position of the normalized gray scale histogram and the distribution width parameter, and the constructed similarity evaluation function, a complete image similarity evaluation model between working conditions is formed. The image similarity evaluation model between working conditions can comprehensively consider the gray scale distribution characteristics of the image subsequence, and evaluate the similarity of the image subsequence under different working conditions.

[0040] The model is used to calculate the similarity values between all different working condition image subsequences, and then a similarity threshold value is set. The image pairs with a similarity value higher than the similarity threshold value are screened out to form an image pair subset, and the image pairs in the image pair subset have high similarity in gray scale distribution and are suitable for subsequent dynamic comparison processing.

[0041] Step S123: Dynamic comparison processing is performed on the image pair subset to eliminate geometric deformation caused by working condition difference, and a standard image sequence after alignment is obtained.

[0042] For each image pair in the image pair subset, first detect feature points in the images. The Harris corner detection algorithm can be used to determine the corner points in the image by calculating the autocorrelation matrix of each pixel point in the image, and analyzing the eigenvalues of the matrix. These corner points are the feature points in the image.

[0043] Then, feature point matching is performed in the two images of the image pair to find corresponding feature point pairs. According to the coordinate relationship of these corresponding feature point pairs, a geometric transformation matrix between the two images is calculated, which can describe the geometric deformation of the images due to the difference in working conditions.

[0044] One of the images is transformed according to the calculated geometric transformation matrix, so that the corresponding feature points in the two images can be accurately aligned. The same processing is performed on all image pairs in the image pair subset, and the processed images are arranged in chronological order to obtain a standard image sequence after alignment, which eliminates the influence of geometric deformation caused by the difference in working conditions.

[0045] Step S124: Extract surface state features from the standard image sequence, enhance the surface micro-damage area by Gaussian filtering, mark the damage area boundary using the region growing algorithm, and calculate the damage area area ratio and morphological factor.

[0046] For each image in the standard image sequence, Gaussian filtering is first performed. For example, a suitable Gaussian kernel size and standard deviation are selected, and the Gaussian kernel is convolved with the image to smooth the high-frequency noise in the image while preserving the edge information of the image and highlighting the micro-damage area of the surface.

[0047] In the filtered image, determine the potential micro-damage area as a seed point. The selection of the seed point can be based on the gray value of the image, and the pixel point with a gray value that is significantly different from the surrounding area and may belong to the damage area is selected.

[0048] For example, step S1241: Gaussian filtering is performed on each image in the standard image sequence, and an adaptive threshold segmentation algorithm is used to binarize the filtered image, and the area with a gray value higher than the gray value threshold is marked as a potential damage area.

[0049] For each image in the standard image sequence, Gaussian filtering is applied. According to the size of the micro-damage area in the image, the parameters of the Gaussian kernel are adjusted to ensure that the noise is smoothed while highlighting the micro-damage area.

[0050] After filtering, an adaptive threshold segmentation algorithm is used. The image is divided into multiple small regions, and a suitable threshold is calculated for each small region. The threshold is determined according to the pixel gray scale distribution in the small region. Then, the gray scale value of each pixel is compared with the threshold of the small region where it is located. The regions with gray scale values higher than the threshold are marked as potential damage regions, which may contain micro-damage on the surface.

[0051] Step S1242: Starting from the seed point in the potential damage region, a region growing algorithm is performed to expand the growing region until the boundary is stable by comparing the gray scale difference and gradient direction of adjacent pixels.

[0052] In the marked potential damage region, a seed point is randomly selected or selected according to certain rules. Starting from the seed point, the adjacent pixel points around it are checked, and the gray scale difference between the adjacent pixel and the seed point and the gradient direction of the adjacent pixel are calculated.

[0053] If the gray scale difference of the adjacent pixel is within a certain range and the gradient direction meets the characteristics of the edge of the damage region, the adjacent pixel is included in the growing region. Repeat the process to continuously expand the growing region until no new pixel can meet the inclusion condition. At this time, the region boundary reaches stability, and the grown region is obtained. This region is the more accurate damage region.

[0054] Step S1243: Morphological processing is performed on the grown region. Small holes in the region are removed by erosion operation. After dilation operation to connect the broken damage region boundary, the pixel coordinates of the processed damage region boundary are extracted, the number of pixels in the region surrounded by the boundary is calculated as the damage area, and the ratio of the damage area to the total number of pixels in the image is taken as the area ratio.

[0055] The grown region is subjected to erosion operation, and a suitable structure element is selected. If there is a non-damage region pixel within the structure element range around the pixel point, the pixel point is marked as a non-damage region, and small holes in the region are removed in this way.

[0056] After erosion processing, dilation operation is performed, and the same or different structure element is used to traverse the pixel points on the region boundary. If there is a damage region pixel within the structure element range around the pixel point, the pixel point is marked as a damage region, thereby connecting the broken damage region boundary.

[0057] After processing, the pixel coordinates of the damage region boundary are extracted, the number of pixels in the region surrounded by these boundaries is counted, and it is taken as the damage area. The ratio of the damage area to the total number of pixels in the entire image is calculated to obtain the damage region area ratio, which reflects the proportion of the damage region in the image.

[0058] Step S1244: Calculate the minimum circumscribed rectangle of the damage region, and take the ratio of the long axis to the short axis of the minimum circumscribed rectangle as a morphology factor, the larger the morphology factor is, the more elongated the damage region is.

[0059] For the processed damage region, find the minimum rectangle that can completely surround the region, that is, the minimum circumscribed rectangle. Determine the lengths of the long axis and the short axis of the rectangle, the long axis is the longer side in the rectangle, and the short axis is the shorter side.

[0060] Calculate the ratio of the length of the long axis to the length of the short axis, and take it as the morphology factor of the damage region. The size of the morphology factor can reflect the shape characteristics of the damage region, and when the morphology factor is larger, it means that the damage region presents a more elongated shape.

[0061] Step S125: Analyze the direction distribution and density change of the surface texture in the standard image sequence, extract the energy feature and entropy value feature of the texture through the gray level co-occurrence matrix, and generate a surface texture change trend parameter.

[0062] For each frame of image in the standard image sequence, select multiple different directions, such as 0 degrees, 45 degrees, 90 degrees, 135 degrees, etc., and calculate the texture direction distribution in each direction. The length and number of textures in each direction are counted to determine the main direction and distribution of the surface texture.

[0063] Calculate the density change of the texture, set multiple sampling windows in the image, count the number of texture elements in each window, analyze the difference of texture density in different windows, and get the density change information.

[0064] Construct a gray level co-occurrence matrix, select appropriate distance and angle parameters, and count the probability of occurrence of pixel pairs with different gray values in the image under certain distance and angle. Extract the energy feature and entropy value feature from the gray level co-occurrence matrix, the energy feature reflects the uniformity and regularity of the texture, and the entropy value feature reflects the complexity of the texture.

[0065] Integrate the direction distribution, density change, energy feature and entropy value feature of the texture to generate a surface texture change trend parameter, which can describe the change of the surface texture with time or position.

[0066] Step S126: Identify the tool interaction region in the standard image sequence, use the watershed algorithm to segment the contact boundary between the tool and the machining object, and extract the contour feature and position coordinates of the contact region.

[0067] In the standard image sequence, the regions where tool interaction may exist are preliminarily determined according to the difference in color, gray scale or edge feature between the tool and the machining object. Further analysis and verification are performed on these regions to determine the approximate range of the tool interaction region.

[0068] Step S1261: Perform edge detection processing on each frame of image in the standard image sequence to extract the edge contour line of the tool and the machining object, and obtain an edge image.

[0069] The Canny edge detection algorithm is used to process each frame of image in the standard image sequence. First, Gaussian filtering is performed on the image to reduce noise interference; then, the gradient amplitude and direction of the image are calculated; next, non-maximum suppression is applied to thin the edge; finally, through double-threshold processing and edge connection, the edge contour line of the tool and the machining object is obtained, and an edge image is formed.

[0070] Step S1262: Perform distance transformation processing on the edge image to calculate the distance value of each pixel point to the nearest edge, and generate a distance transformation map.

[0071] For each pixel point in the edge image, the Euclidean distance of the point to the nearest edge contour line is calculated. The distance values of all pixel points form a new image, i.e., a distance transformation map. In the distance transformation map, the greater the distance value, the farther the pixel point is from the edge, and the pixel point with a distance value of zero is located on the edge.

[0072] Step S1263: Mark the seed points of the contact region between the tool and the machining object in the distance transformation map, and the seed points are selected as the pixel points with locally maximum distance transformation value and located between two edges.

[0073] In the distance transformation map, the pixel points with locally maximum distance transformation value are found, which are usually located at the center position of the region. At the same time, it is ensured that these pixel points are located between the two edge contour lines of the tool and the machining object, so as to determine these points as the seed points of the contact region between the tool and the machining object.

[0074] Step S1264: Perform the watershed algorithm starting from the seed points, and expand the region boundary by gradient descent method until the boundaries of adjacent regions meet, to obtain the segmentation result of the contact region.

[0075] The watershed algorithm is applied in the distance transformation map starting from the marked seed points. According to the gradient descent direction, the region boundary is continuously expanded from the seed points, and the pixel points with similar distance transformation characteristics are included in the same region.

[0076] When the extended region boundary meets the boundary of the adjacent region, the extension is stopped, forming a partition line between the regions. In this way, the contact region of the tool and the machining object is segmented from the image, and a segmentation result of the contact region is obtained.

[0077] Step S1265: Extracting contour features of the contact region from the segmentation result, representing the contour shape by a Fourier descriptor of the contour, the contour features including a perimeter of the contour, an area, and a curvature variation rate.

[0078] According to the segmentation result of the contact region, a contour line of the region is extracted. The perimeter of the contour line, i.e., the sum of distances between all pixel points on the contour line, is calculated. The area of the region surrounded by the contour line can be calculated by a polygon area calculation method through the coordinates of the pixel points on the contour line.

[0079] The curvature of the contour line is calculated to analyze the curvature variation at different positions on the contour line, and a curvature variation rate is obtained. Meanwhile, the coordinate information of the contour line is converted into a Fourier descriptor, which can succinctly represent the shape features of the contour and is not affected by translation, rotation, and scaling of the contour.

[0080] Step S1266: Calculating the centroid coordinates of the contact region, taking the centroid coordinates as the position coordinates of the contact region, the centroid coordinates being calculated by the average values of the coordinates of all pixels in the contact region.

[0081] The coordinates of all pixel points in the contact region are obtained, and the average values of the x coordinates and the y coordinates of these pixel points are calculated respectively, and the obtained average values of the x coordinates and the y coordinates are the centroid coordinates of the contact region. The centroid coordinates are taken as the position coordinates of the contact region, which are used to describe the position of the contact region in the image.

[0082] Step S127: Calculating the contact pressure distribution situation of the tool and the machining object based on the contour features of the contact region, and converting it into a pressure gradient parameter of the contact point by a pressure conduction model.

[0083] According to the contour features of the contact region, such as the perimeter, the area, and the curvature variation rate of the contour, and in combination with the physical parameters of the tool and the related parameters in the machining process, such as the hardness of the tool and the driving force during machining, the distribution of the contact pressure of the tool and the machining object is analyzed.

[0084] The pressure conduction model is used to convert the distribution of the contact pressure into a pressure gradient parameter of the contact point. The pressure gradient parameter can reflect the variation rate of the contact pressure in the contact region, and by calculating the pressure values at different positions in the contact region, the variation amount of the pressure per unit distance is obtained to represent the pressure gradient.

[0085] Step S128: integrate the damage area ratio, the morphological factor, and the surface texture change trend parameter into a surface state feature set, and integrate the position coordinates of the contact area and the pressure gradient parameter into a tool interaction feature set.

[0086] The damage area ratio, the morphological factor obtained in step S124, and the surface texture change trend parameter generated in step S125 are summarized and arranged. The above parameters describe the state of the surface of the machining object from different angles. By combining them together, a surface state feature set is formed.

[0087] At the same time, the position coordinates of the contact area obtained in step S126 and the pressure gradient parameter obtained in step S127 are integrated. The position coordinates describe the position of the tool contacting the machining object, and the pressure gradient parameter reflects the change of the contact pressure. Together, they constitute a tool interaction feature set.

[0088] Step S130: based on the machining precision constraint conditions, bidirectional verification analysis is performed on the surface state feature set and the tool interaction feature set, and through a feature conflict resolution mechanism, effective positioning features are determined, and a core positioning area description of the machining object is output.

[0089] First, the machining precision constraint conditions are determined. These conditions are determined according to the quality requirements and subsequent use requirements of the machining object, such as the maximum value of the surface roughness allowed, the range of the contact pressure between the tool and the machining object, etc.

[0090] Each feature in the surface state feature set and the tool interaction feature set is compared with the machining precision constraint conditions to check whether they meet the constraint requirements. Through the above comparison, features that meet the conditions can be screened out.

[0091] Step S131: obtain the machining precision constraint conditions, and match the surface state feature set and the tool interaction feature set with the machining precision constraint conditions respectively, and generate a surface state feature subset and a tool interaction feature subset that match the machining precision constraint conditions.

[0092] Collect the machining precision constraint conditions related to the machining object. These machining precision constraint conditions may include the allowed deviation of the surface flatness, the precision range of the machining size, the upper and lower limits of the pressure of the tool acting on the machining object, etc.

[0093] For each feature in the surface state feature set, it is judged whether it meets the requirements related to the surface state in the machining precision constraint conditions. For example, it is checked whether the damage area ratio is within the allowed range, whether the surface texture change trend meets the specified standard, etc. In this way, features that meet the requirements can be screened out to form a surface state feature subset.

[0094] Similarly, for each feature in the tool interaction feature set, match the part related to tool interaction in the machining precision constraint condition. For example, judge whether the position coordinates of the contact area are within the reasonable machining range, whether the pressure gradient parameter is within the allowed pressure variation range, etc. Thus, the features that meet the conditions can be integrated into a tool interaction feature subset.

[0095] Step S132: Establish the association mapping between the surface state feature subset and the tool interaction feature subset, and determine the correspondence between the surface state feature subset and the tool interaction feature subset through spatial coordinate matching.

[0096] Extract the spatial coordinates of the region corresponding to each feature in the surface state feature subset in the image, and extract the spatial coordinates of the region corresponding to each feature in the tool interaction feature subset.

[0097] By comparing the above spatial coordinates, find the features that correspond to each other in position, and establish the association mapping between the surface state feature subset and the tool interaction feature subset. For example, when a certain damage area in the surface state feature subset overlaps or is adjacent to a certain contact area in the tool interaction feature subset in space, they are associated.

[0098] Step S133: Detect feature conflicts in the association mapping to obtain conflict features, wherein the feature conflicts include the case that the significant damage area indicated by the surface state feature overlaps with the significant pressure area indicated by the tool interaction feature.

[0099] Analyze each pair of associated features in the established association mapping to check whether there is a contradiction or inconsistency between them. For example, focus on the spatial relationship between the area indicated by the surface state feature and the area indicated by the tool interaction feature, and whether the processing state reflected by them is in conflict.

[0100] Step S1331: Normalize the area proportion of the damage area in the surface state feature subset to obtain a damage degree index, and the higher the damage degree index, the more serious the damage of the region.

[0101] Collect the area proportions of all damage areas in the surface state feature subset, and determine the maximum and minimum values of these area proportions. Subtract the minimum value from the area proportion of each damage area, and then divide by the difference between the maximum value and the minimum value to obtain the normalized value, i.e. the damage degree index. The damage degree index has a value range of 0 to 1, and the larger the value, the more serious the damage of the corresponding region.

[0102] Step S1332: Normalize the pressure gradient parameter in the tool interaction feature subset to obtain a pressure intensity index, and the higher the pressure intensity index, the greater the pressure of the region.

[0103] For each pressure gradient parameter in the tool interaction feature subset, find the maximum and minimum values in the parameter. For each pressure gradient parameter, subtract the minimum value from the parameter value, and divide the result by the difference between the maximum and minimum values to obtain a normalized pressure intensity index. The pressure intensity index is also between 0 and 1, and the higher the value, the greater the pressure in the region.

[0104] Step S1333: Set a damage degree threshold and a pressure intensity threshold, and mark a region as a significant damage region if the damage degree index of the region is higher than the damage degree threshold, and mark a region as a significant pressure region if the pressure intensity index of the region is higher than the pressure intensity threshold.

[0105] According to the machining precision requirement and actual machining experience, set the damage degree threshold and the pressure intensity threshold. For example, the damage degree threshold can be set to a value between 0 and 1, and when the damage degree index of a region exceeds the threshold, mark the region as a significant damage region.

[0106] Similarly, when the pressure intensity index of a region is higher than the set pressure intensity threshold, mark the region as a significant pressure region.

[0107] Step S1334: Calculate the spatial overlap degree of the significant damage region and the significant pressure region, which is calculated by the ratio of the intersection area of the significant damage region and the significant pressure region to the union area.

[0108] Determine the boundary coordinates of the significant damage region and the significant pressure region in the image, calculate the intersection area of the two regions, that is, the area of the overlapping part of the two regions. At the same time, calculate the union area of the two regions, that is, the total area covered by the two regions.

[0109] Divide the intersection area by the union area to obtain the spatial overlap degree, which is in the range of 0 to 1, and the higher the value, the higher the overlap degree of the two regions.

[0110] Step S1335: When the spatial overlap degree is higher than a preset conflict threshold, determine that there is a feature conflict between the significant damage region and the significant pressure region, and record the boundary coordinates of the corresponding conflict region and the corresponding damage degree index and pressure intensity index.

[0111] In this embodiment, a conflict threshold can be preset, and when the calculated spatial overlap degree is higher than the conflict threshold, it indicates that the significant damage region and the significant pressure region have a high degree of overlap in space, and there is a feature conflict. At this time, record the boundary coordinates of the conflict region, and the corresponding damage degree index and pressure intensity index of the region, so as to perform conflict processing subsequently.

[0112] Step S1336: Perform conflict detection on all the associated mapped feature regions, aggregate the information of the conflict regions, and generate conflict features, which contain the location, conflict type, and conflict degree parameters of the conflict regions.

[0113] Perform conflict detection on all the associated mapped feature regions according to the method of steps S1331 to S1335. Aggregate the information of all the detected conflict regions, including the location (boundary coordinates), conflict type (such as the overlap of a significant damage region and a significant stress region), and conflict degree parameters (such as the spatial overlap degree, damage degree index, stress intensity index, etc.) of each conflict region, to form conflict features.

[0114] Step S134: Process the conflict features using a feature conflict resolution mechanism, and determine the feature validity of the conflict regions by weighted voting, with the weighted weights being assigned based on the priority of the machining precision constraint conditions.

[0115] According to the importance of the machining precision constraint conditions, assign corresponding weights to different constraint conditions. When processing conflict features, use the above weights to determine which features in the conflict region are valid and which features need to be excluded by weighted voting.

[0116] Step S1341: Assign weight values to the surface roughness requirement and tool contact pressure threshold based on the priority of the machining precision constraint conditions, with higher priority constraint conditions corresponding to higher weight values.

[0117] Prioritize the machining precision constraint conditions, for example, if the surface roughness requirement is more important than the tool contact pressure threshold in machining, the surface roughness requirement has a higher priority.

[0118] According to the priority, assign weight values to the surface roughness requirement and tool contact pressure threshold, with higher priority constraint conditions being assigned higher weight values, and the sum of the weight values can be set to 1.

[0119] Step S1342: Multiply the damage degree index of the conflict region by the weight value of the surface roughness requirement to obtain the voting score of the surface feature.

[0120] For each conflict region, multiply its damage degree index by the weight value corresponding to the surface roughness requirement, and the calculated result is the voting score of the surface feature in that conflict region.

[0121] Step S1343: Multiply the stress intensity index of the conflict region by the weight value of the tool contact pressure threshold to obtain the voting score of the interaction feature.

[0122] Similarly, the pressure intensity index of the conflict region is multiplied by the weight value corresponding to the tool contact pressure threshold to obtain the voting score of the interaction feature.

[0123] Step S1344: Compare the voting score of the surface feature with the voting score of the interaction feature. If the voting score of the surface feature is higher, the region attribute indicated by the surface feature is retained, and the corresponding region attribute in the tool interaction feature is removed. If the voting score of the interaction feature is higher, the region attribute indicated by the tool interaction feature is retained, and the corresponding region attribute in the surface feature is removed.

[0124] The voting score of the surface feature and the voting score of the interaction feature are compared. If the voting score of the surface feature is greater than the voting score of the interaction feature, it indicates that the surface feature meets the priority requirement of the machining precision constraint condition in the conflict region, and therefore the region attribute indicated by the surface feature is retained, and the corresponding region attribute in the tool interaction feature is removed.

[0125] Conversely, if the voting score of the interaction feature is higher, the region attribute indicated by the tool interaction feature is retained, and the corresponding region attribute in the surface feature is removed.

[0126] Step S1345: If the voting score of the surface feature and the voting score of the interaction feature are equal, the feature attribute related to the key requirement of the machining process is preferentially retained.

[0127] When the voting score of the surface feature and the voting score of the interaction feature are equal, it is not possible to directly determine which feature attribute to retain through weighted voting. At this time, the key requirement of the machining process is referred to, for example, if the key requirement of the machining process is to ensure the integrity of the surface, the region attribute indicated by the surface feature is preferentially retained; if the key requirement is to ensure that the contact pressure of the tool and the machining object is appropriate, the region attribute indicated by the tool interaction feature is preferentially retained.

[0128] Step S1346: Record the feature attribute after conflict resolution, and update the set of valid positioning features.

[0129] The feature attribute retained after conflict resolution is recorded, and the set of valid positioning features initially determined is updated according to the retained feature attribute, removing the removed feature attribute, to ensure that the features in the set of valid positioning features are processed through conflict and meet the requirements.

[0130] Step S135: Retain the features that pass the bidirectional check and have no conflict to form a set of valid positioning features, which includes features of smooth region of surface texture change and features of moderate region of tool contact pressure.

[0131] The features in the surface state feature subset and the tool interaction feature subset that pass the bidirectional verification (i.e., simultaneously satisfy the machining precision constraint condition and have no conflicts in the correlation mapping) are screened, and these features are retained.

[0132] The retained features include features of a smooth surface texture change region, i.e., the direction distribution and density of the surface texture change relatively stably without sharp fluctuations, and features of a moderate tool contact pressure region, i.e., the pressure gradient parameter is within a reasonable range and is neither too high nor too low. The above features together constitute an effective positioning feature set.

[0133] Step S136: Extracting a region with the highest positioning accuracy from the effective positioning feature set as a core positioning region, describing the core positioning region by the boundary coordinates of the region and the internal feature parameters, and outputting a core positioning region description.

[0134] The positioning accuracy of each region in the effective positioning feature set is evaluated, and the evaluation basis can include the boundary clarity of the region and the stability of the internal features. The region with the highest positioning accuracy is selected as the core positioning region.

[0135] The core positioning region is described by determining the boundary coordinates of the core positioning region and the internal feature parameters such as the energy features and entropy features of the surface texture and the gradient parameters of the tool contact pressure. After the description information is sorted, a core positioning region description is output.

[0136] Step S140: Constructing a machining positioning reference frame and a dynamic adjustment rule set according to the core positioning region description and in combination with the motion constraint parameters of the machining equipment.

[0137] According to the boundary coordinates and internal feature parameters provided in the core positioning region description, the basic reference point and direction of machining positioning are determined. At the same time, in combination with the motion constraint parameters of the machining equipment itself, such as the motion range and motion speed limit of each axis, a machining positioning reference frame is built.

[0138] According to the features of the core positioning region and the motion characteristics of the machining equipment, a dynamic adjustment rule set is formulated to cope with various changes that may occur during machining.

[0139] Step S141: Analyzing the core positioning region description and extracting the boundary coordinates and the center of mass position of the core positioning region, and taking the center of mass position as an initial positioning reference point.

[0140] The core positioning region description is analyzed in detail, and the boundary coordinates of the core positioning region are extracted from it. These coordinates can be represented by the positions of image pixels.

[0141] The center position of the core positioning area is calculated by averaging the x coordinates of all pixels in the core positioning area to obtain the x coordinate of the center, and by averaging the y coordinates of all pixels to obtain the y coordinate of the center. The center position is determined as the initial positioning reference point.

[0142] In step S142, the motion constraint parameters of the machining device are obtained, including the stroke range, maximum motion speed, acceleration limit and positioning accuracy level of each axis of the device.

[0143] The motion constraint parameters are obtained by communicating with the control system of the machining device or from the technical manual of the device. The stroke range of each axis of the device refers to the distance between the minimum and maximum positions that each axis can move; the maximum motion speed is the highest speed that each axis can reach during motion; the acceleration limit is the maximum acceleration of each axis during acceleration; and the positioning accuracy level describes the error range that the device can produce during positioning.

[0144] In step S143, the machining coordinate system is established with the initial positioning reference point as the origin, and the boundary coordinates of the core positioning area are converted to the machining coordinate system to obtain standardized boundary coordinates.

[0145] The machining coordinate system is established with the initial positioning reference point as the origin, and the directions of the x and y axes are set (for example, the long side of the worktable of the machining device is taken as the x axis and the short side is taken as the y axis).

[0146] According to the relationship between the coordinates of the initial positioning reference point in the image and the origin of the machining coordinate system, the boundary coordinates of the core positioning area are converted from the image coordinate system to the machining coordinate system to obtain standardized boundary coordinates, which facilitates subsequent matching with the motion control of the machining device.

[0147] In step S144, the conversion relationship between the machining coordinate system and the device base coordinate system and the tool coordinate system is determined based on the motion constraint parameters of the machining device, and a multi-coordinate system conversion matrix is constructed to form a machining positioning reference frame.

[0148] In this embodiment, the machining coordinate system is established based on the center of the core positioning area, the device base coordinate system is the basic coordinate system provided by the machining device, and the tool coordinate system takes the end of the tool as the origin. Determining the conversion relationship between the three requires combining the motion axis parameters and motion constraint parameters such as the stroke range of each axis of the machining device, and achieving through coordinate transformation calculation.

[0149] In step S1441, the conversion parameters of the machining device base coordinate system and the axis motion coordinate system are obtained through the kinematic model of the device, including the translation and rotation angles.

[0150] The kinematic model of the machining device contains the relationship between each motion axis and the device base coordinate system. By calling the model, the translation and rotation angles of the device base coordinate system to each motion axis coordinate system can be obtained. For example, for the X-axis motion coordinate system, the translation thereof relative to the device base coordinate system includes offset values in the X, Y and Z directions, and the rotation angles thereof include rotation angles around the X, Y and Z axes; the conversion parameters of the Y-axis and Z-axis motion coordinate systems are also obtained in the same manner. The above parameters quantify the position and posture of each motion axis in the device base coordinate system.

[0151] Step S1442: The coordinates of the origin of the tool coordinate system in the device base coordinate system are measured by the laser tracker to obtain the initial conversion relationship between the tool coordinate system and the device base coordinate system.

[0152] The measurement target ball of the laser tracker is fixed at the end of the tool, i.e. the position of the origin of the tool coordinate system. The laser tracker is started to emit a laser beam and track the target ball. When the tool is in the initial position, the coordinates of the target ball in the device base coordinate system are recorded, which are the initial coordinates of the origin of the tool coordinate system in the device base coordinate system. Thus, the initial translation relationship of the tool coordinate system relative to the device base coordinate system can be determined. If the initial posture of the tool has no rotation, the initial rotation angle is zero, and thus the initial conversion relationship between the two is obtained.

[0153] Step S1443: The coordinates of the centroid position of the core positioning area in the image coordinate system are converted to the device base coordinate system as the origin of the machining coordinate system.

[0154] Firstly, the conversion relationship between the image coordinate system and the device base coordinate system is obtained, which can be obtained by pre-calibration and contains the scaling ratio and offset of the image pixel coordinates to the physical coordinates of the device base coordinate system. The centroid of the core positioning area has corresponding pixel coordinates in the image coordinate system. The pixel coordinates are substituted into the conversion relationship to calculate the physical coordinates of the centroid in the device base coordinate system, which are the origin of the machining coordinate system.

[0155] Step S1444: According to the positional relationship between the origin of the machining coordinate system and the device base coordinate system, the conversion matrix between the machining coordinate system and the device base coordinate system is calculated, which contains translation parameters in three directions and rotation parameters in three directions.

[0156] The coordinates of the origin of the machining coordinate system in the device base coordinate system are known, and thus the translation parameters between the two can be determined, i.e. the offset values of the origin of the machining coordinate system relative to the origin of the device base coordinate system in the X, Y and Z directions. If the axis system direction of the machining coordinate system is consistent with the device base coordinate system, the rotation parameters are zero; if there is an included angle, the rotation angles around the X, Y and Z axes are measured or calculated as the rotation parameters. The translation parameters and the rotation parameters are combined in a specific order to form the conversion matrix between the machining coordinate system and the device base coordinate system.

[0157] Step S1445: Obtain the relative position relationship between the tool coordinate system and the machining coordinate system by the tool calibration method, and calculate the conversion matrix therebetween, the matrix parameters of the conversion matrix being determined based on the measured distance between the tool tip and the origin of the machining coordinate system.

[0158] The tool calibration method, such as the touch calibration method, is adopted to control the tool tip to touch the origin of the machining coordinate system, and the coordinates of the tool coordinate system origin in the machining coordinate system at this time are recorded, which reflect the relative translation relationship between the two. If there is rotation between the tool coordinate system and the machining coordinate system, the rotation parameters are calculated by touching different known points in the machining coordinate system multiple times. According to the relative translation parameters and the rotation parameters, the conversion matrix of the tool coordinate system and the machining coordinate system is constructed, and each parameter in the matrix is determined based on the measured distance and angle data.

[0159] Step S1446: Integrate the conversion matrix of the machining coordinate system and the equipment base coordinate system and the conversion matrix of the tool coordinate system and the machining coordinate system to construct a multi-coordinate system conversion matrix set.

[0160] The conversion matrix of the machining coordinate system and the equipment base coordinate system obtained in step S1444 and the conversion matrix of the tool coordinate system and the machining coordinate system obtained in step S1445 are collected together to form a multi-coordinate system conversion matrix set. The multi-coordinate system conversion matrix set completely records the conversion relationship between the main coordinate systems involved in the machining process.

[0161] Step S1447: Take the multi-coordinate system conversion matrix set as the core, combine the standardized boundary coordinates of the core positioning area and the equipment motion range parameters to construct a machining positioning reference framework, the machining positioning reference framework including the origin positions, axis system directions and effective working ranges of the coordinate systems.

[0162] The multi-coordinate system conversion matrix set clearly defines the conversion rules between the coordinate systems. Taking this as the core, the standardized boundary coordinates of the core positioning area are added, which define the key area range of the machining. At the same time, the equipment motion range parameters, such as the stroke limits of each axis, are included to determine the effective working ranges in each coordinate system. The origin positions and axis system directions of each coordinate system are reflected by the parameters in the conversion matrix. After integrating these information, the machining positioning reference framework is constructed.

[0163] Step S145: Analyze the surface state features of the core positioning area, and when the surface texture change rate exceeds the preset change rate, construct a position correction rule, the position correction rule including a correction amount calculation method along the vertical direction of the texture.

[0164] The surface texture change trend parameter of the core positioning area is analyzed, and the change rate of the surface texture is calculated. If the change rate exceeds the preset change rate, it indicates that the change of the surface texture is relatively severe, which may affect the machining precision. At this time, the position correction rule needs to be constructed.

[0165] The position correction rule specifies the calculation method of the correction amount along the vertical direction of the texture. For example, the size of the correction amount can be determined according to the size of the texture change rate. The larger the change rate, the larger the correction amount.

[0166] Step S146: Analyze the tool interaction feature. When the tool contact pressure gradient is abnormal, construct a posture calibration rule. The posture calibration rule includes the rotation angle adjustment logic of the tool around each axis.

[0167] The pressure gradient parameter in the tool interaction feature is monitored. When the pressure gradient parameter exceeds the normal range, i.e., it is abnormal, it indicates that the contact posture of the tool and the machining object may have a problem, and a posture calibration rule needs to be constructed.

[0168] The posture calibration rule includes the rotation angle adjustment logic of the tool around the x-axis, y-axis, and z-axis. For example, when the pressure gradient is too large in a certain direction, the angle through which the tool needs to be rotated around the corresponding axis is determined by calculation to adjust the posture of the tool and restore the contact pressure gradient to normal.

[0169] Step S147: According to the acceleration limit parameter of the machining equipment, construct a feed speed adjustment rule. The feed speed adjustment rule includes speed switching thresholds for different machining stages.

[0170] Referring to the acceleration limit parameter of the machining equipment, and combining different stages in the machining process (such as the stage of approaching the core positioning area, the stage of machining within the core positioning area, and the stage of leaving the core positioning area), a feed speed adjustment rule is constructed.

[0171] In the rule, the speed switching threshold of each machining stage is set. When the machining proceeds to a certain stage and the speed switching condition is met, the feed speed is adjusted according to the rule to ensure that the motion of the machining equipment does not exceed the acceleration limit, while ensuring the machining efficiency and precision.

[0172] Step S148: Integrate the position correction rule, the posture calibration rule, and the feed speed adjustment rule into a dynamic adjustment rule set. Each dynamic adjustment rule in the dynamic adjustment rule set includes a trigger condition and a corresponding adjustment parameter calculation method.

[0173] The position correction rule constructed in step S145, the posture calibration rule constructed in step S146, and the feed speed adjustment rule constructed in step S147 are summarized to form a dynamic adjustment rule set.

[0174] Each dynamic adjustment rule specifies a trigger condition, i.e. when the rule needs to be applied, and a corresponding adjustment parameter calculation method, i.e. how to calculate the specific value of the adjustment parameter according to the actual situation.

[0175] Step S150: fuse the machining positioning reference frame, the dynamic adjustment rule set and the feedback features in the real-time image stream to generate a positioning guidance instruction sequence containing tool displacement correction amounts and feed rhythm adjustment values, and transmit the positioning guidance instruction sequence to the machining control system to perform closed-loop positioning guidance actions.

[0176] Real-time feedback features are obtained from the image stream, and these features are fused and analyzed with the machining positioning reference frame and the dynamic adjustment rule set. According to the analysis results, the displacement correction amounts and feed rhythm adjustment values required by the tool are calculated, and these parameters are arranged in time sequence to form a positioning guidance instruction sequence.

[0177] The instruction sequence is transmitted to the machining control system, which controls the machining equipment to perform corresponding actions according to the content in the instruction sequence, realizes closed-loop positioning guidance, and ensures the accuracy and stability of the machining process.

[0178] Step S151: extract the latest feedback features from the real-time image stream, which include surface state feedback features and tool interaction feedback features at the current machining position.

[0179] The latest one or more frames of images in the real-time image stream are processed to extract surface state feedback features at the current machining position, such as whether new damage appears on the surface, whether the texture change is within the allowed range, etc.; and tool interaction feedback features, such as the contact position between the current tool and the machining object, the gradient change of the contact pressure, etc.

[0180] Step S152: convert the feedback features to the machining coordinate system of the machining positioning reference frame to obtain standardized feedback features, which contain deviation values between the current position and the core positioning area.

[0181] The spatial coordinates corresponding to the surface state feedback features and the tool interaction feedback features in the feedback features are obtained. These coordinates are initially based on the image coordinate system. According to the conversion relationship between the image coordinate system and the machining coordinate system in the machining positioning reference frame, the coordinates are converted.

[0182] Specifically, the coordinates of the damage area in the surface state feedback feature, the coordinates of the texture change area, and the coordinates of the contact position in the tool interaction feedback feature are converted into the machining coordinate system one by one by using the previously determined conversion matrix. After the conversion is completed, the deviation values of the current machining position from the core positioning area in the machining coordinate system are calculated, including the deviations in the x direction and the y direction. These deviation values and the converted coordinates jointly constitute the standardized feedback feature.

[0183] Step S153: A position correction rule in the dynamic adjustment rule set is called, and the surface texture change rate in the standardized feedback feature is compared with a rule triggering condition. If the rule triggering condition is met, a tool displacement correction amount is calculated.

[0184] The position correction rule is extracted from the dynamic adjustment rule set, and the triggering condition of the rule, that is, the threshold of the surface texture change rate, is determined. The surface texture change rate in the standardized feedback feature is compared with the threshold. If the surface texture change rate exceeds the threshold, it is indicated that the tool position needs to be corrected.

[0185] According to the correction amount calculation method of the position correction rule in the vertical direction of the texture, combined with the specific value of the surface texture change rate, the displacement correction amount of the tool in the x direction and the y direction is calculated. For example, if the texture change rate is large, the corresponding correction amount is calculated according to a preset proportional relationship to compensate for the influence of the texture change on the machining position.

[0186] Step S154: A posture calibration rule in the dynamic adjustment rule set is called, and the pressure gradient parameter in the standardized feedback feature is compared with a rule triggering condition. If the rule triggering condition is met, a tool posture adjustment angle is calculated.

[0187] The posture calibration rule in the dynamic adjustment rule set is extracted, and the triggering condition of the rule, that is, the normal range of the pressure gradient parameter, is determined. The pressure gradient parameter in the standardized feedback feature is compared with the normal range. If the pressure gradient parameter exceeds the range, that is, the triggering condition is met, the tool posture needs to be adjusted.

[0188] According to the rotation angle adjustment logic of the tool around each axis in the posture calibration rule, the rotation angles of the tool around the x axis, the y axis, and the z axis are calculated according to the degree and direction in which the pressure gradient parameter exceeds the normal range. For example, when the pressure gradient is too large in the x direction, the angle by which the tool needs to be rotated around the y axis is calculated to adjust the contact posture of the tool and the machining object, so that the pressure gradient returns to normal.

[0189] Step S155: A feed speed adjustment rule in the dynamic adjustment rule set is called, and a feed rhythm adjustment value is calculated according to the current machining stage and the surface damage degree parameter. The feed rhythm adjustment value is a correction coefficient of the basic feed speed.

[0190] The feed speed adjustment rule is called from the dynamic adjustment rule set, and the division criteria and corresponding speed switching threshold of different machining stages are determined. The current machining stage is determined, such as the stage of approaching the core positioning area, the stage of machining in the core positioning area, etc.

[0191] At the same time, the surface damage degree parameter in the standardized feedback feature is referred to. If the surface damage degree is high, the feed speed needs to be reduced to reduce further damage; if the damage degree is low, the feed speed can be appropriately increased to improve efficiency. According to these factors, combined with the feed speed adjustment rule, the feed rhythm adjustment value is calculated, which is a coefficient used to multiply the basic feed speed to obtain the adjusted feed speed.

[0192] Step S156: The tool displacement correction amount, tool posture adjustment angle, and feed rhythm adjustment value are sorted in time sequence, wherein each time point corresponds to a set of adjustment parameters.

[0193] The generation time of the tool displacement correction amount, tool posture adjustment angle, and feed rhythm adjustment value calculated each time is recorded, and these parameters are arranged in chronological order. Ensure that each time point has a complete set of adjustment parameters, forming a parameter sequence arranged in time sequence, so that the machining control system can execute the adjustment action in time sequence.

[0194] Step S157: Add timestamp information to each set of adjustment parameters to obtain timestamped adjustment parameters, and combine the timestamped adjustment parameters into a positioning guide instruction sequence, wherein each instruction in the positioning guide instruction sequence includes instruction identification, execution time, tool displacement correction amount, posture adjustment angle, and feed rhythm adjustment value.

[0195] For each set of adjustment parameters sorted in time sequence, add corresponding timestamp information, and the timestamp is accurate to the millisecond level to specify the execution time of the parameter set. Assign a unique instruction identification to each timestamped adjustment parameter to distinguish different instructions.

[0196] Combine the information including instruction identification, execution time, tool displacement correction amount, posture adjustment angle, and feed rhythm adjustment value together to form a positioning guide instruction sequence, and each instruction in the sequence is an independent unit that contains all the information required to execute the instruction.

[0197] Step S158: Verify the positioning guide instruction sequence, and transmit the verified positioning guide instruction sequence to the machining control system to enable the machining control system to execute the positioning guide action according to the positioning guide instruction sequence.

[0198] The positioning guide instruction sequence is checked to check whether the parameters in the instruction are within the motion constraint range of the machining equipment, such as whether the tool displacement correction amount exceeds the stroke range of each axis of the equipment, whether the feed pace adjustment value corresponding to the feed speed exceeds the maximum motion speed, and the like. At the same time, it is checked whether the time stamp of the instruction is in conflict or error, to ensure the logicality and rationality of the instruction sequence.

[0199] After the verification confirms that there is no error, the positioning guide instruction sequence is transmitted to the machining control system through the communication interface. After the machining control system receives the instruction sequence, the motion parts of the machining equipment are controlled to perform corresponding displacement adjustment, posture adjustment and feed speed adjustment according to the execution time and parameters in the instruction, to realize closed-loop positioning guidance of the machining process and ensure that the machining precision of the machining object meets the requirements.

[0200] Figure 2 A schematic diagram of exemplary hardware and software components of the industrial automation-oriented real-time visual positioning system 100 that can implement the idea of the present application is shown. For example, the processor 120 can be used in the industrial automation-oriented real-time visual positioning system 100 and used to execute the functions in the present application.

[0201] For example, the industrial automation-oriented real-time visual positioning system 100 can include a network port 110 connected to a network, one or more processors 120 for executing program instructions, a communication bus 130, and different forms of storage media 140, such as a disk, a ROM, or a RAM, or any combination thereof. Exemplarily, the industrial automation-oriented real-time visual positioning system 100 can also include program instructions stored in a ROM, a RAM, or other types of non-transitory storage media, or any combination thereof. The method of the present application can be implemented according to these program instructions. The industrial automation-oriented real-time visual positioning system 100 also includes an input / output (I / O) interface 150 between the computer and other input / output devices.

[0202] In addition, the present application also provides a readable storage medium, in which computer executable instructions are pre-set. When the processor executes the computer executable instructions, the industrial automation-oriented real-time visual positioning method is realized.

[0203] It should be noted that, in order to simplify the description of the present application disclosed and to help the understanding of one or more embodiments of the present application, in the foregoing description of the embodiments of the present application, various features are sometimes incorporated into one embodiment, drawing or description thereof.

Claims

1. A real-time visual positioning method for industrial automation, characterized in that, The method comprises: Collecting real-time image streams of the processing object under multiple working conditions, the real-time image streams containing surface state images and tool interaction images of the processing object at different processing stages, and carrying corresponding working condition identifiers and collection time sequence information; Performing working condition adaptability feature analysis on the real-time image streams, eliminating the influence of working condition differences through dynamic image comparison, generating a surface state feature set and a tool interaction feature set of the processing object; Performing bidirectional verification analysis on the surface state feature set and the tool interaction feature set based on processing precision constraint conditions, determining effective positioning features through a feature conflict resolution mechanism, and outputting a core positioning area description of the processing object; According to the core positioning area description, combining the motion constraint parameters of the processing equipment, constructing a processing positioning reference framework and a dynamic adjustment rule set; Fusing the processing positioning reference framework, the dynamic adjustment rule set, and the feedback features of the real-time image streams, generating a positioning guide instruction sequence containing tool displacement correction amounts and feed rhythm adjustment values, and transmitting the positioning guide instruction sequence to a processing control system to perform closed-loop positioning guide actions.

2. The real-time visual positioning method for industrial automation according to claim 1, characterized in that, The working condition adaptability feature analysis on the real-time image streams, eliminating the influence of working condition differences through dynamic image comparison, generating a surface state feature set and a tool interaction feature set of the processing object, comprises: Segmenting the real-time image streams according to working condition identifiers to obtain image sub-sequences corresponding to each working condition, and extracting gray scale distribution features and edge gradient features of each image sub-sequence; Based on the gray scale distribution features, an image similarity evaluation model between working conditions is constructed, the similarity values of different working condition image sub-sequences are calculated, and a subset of image pairs with similarity values higher than a similarity threshold is screened; Performing dynamic comparison processing on the image pair subset to eliminate geometric deformation caused by working condition differences, and obtaining a standard image sequence after alignment; From the standard image sequence, surface state features are extracted, surface micro-damage areas are enhanced through Gaussian filtering, damage area boundaries are marked using a region growing algorithm, damage area area ratios and morphological factors are calculated; The direction distribution and density change of surface texture in the standard image sequence are analyzed, energy features and entropy value features of the texture are extracted through a gray level co-occurrence matrix, and surface texture change trend parameters are generated; Tool interaction areas in the standard image sequence are identified, a watershed algorithm is used to segment the contact boundary between the tool and the processing object, and contour features and position coordinates of the contact area are extracted; Based on the contour features of the contact area, the contact pressure distribution trend of the tool and the processing object is calculated, and the pressure gradient parameters of the contact points are converted through a pressure conduction model; The damage area area ratio, the morphological factor, and the surface texture change trend parameter are integrated into a surface state feature set, and the position coordinates and the pressure gradient parameter of the contact area are integrated into a tool interaction feature set.

3. The real-time visual positioning method for industrial automation according to claim 2, characterized in that, The tool interaction area in the standard image sequence is identified, the watershed algorithm is used to segment the contact boundary between the tool and the processing object, and the contour features and position coordinates of the contact area are extracted, comprising: Edge detection processing is performed on each frame of image in the standard image sequence, the edge contour line of the tool and the processing object is extracted, and an edge image is obtained; performing distance transform processing on the edge image to calculate distance values of each pixel point to the nearest edge, and generating a distance transform image; marking seed points of a contact area between the tool and the machining object in the distance transform image, the seed points being selected from pixel points with local maximum distance transform values and located between two edges; performing a watershed algorithm with the seed points as starting points, and expanding the boundary of the region by a gradient descent method until the boundaries of adjacent regions meet, to obtain a segmentation result of the contact area; extracting contour features of the contact area from the segmentation result, representing the contour shape by a Fourier descriptor of the contour, and the contour features including a perimeter, an area, and a curvature variation rate of the contour; calculating a centroid coordinate of the contact area, and taking the centroid coordinate as a position coordinate of the contact area, the centroid coordinate being calculated by an average value of coordinates of all pixels in the contact area.

4. The real-time visual positioning method for industrial automation of claim 1, wherein, The bidirectional verification and analysis of the surface state feature set and the tool interaction feature set based on the machining precision constraint condition, determination of effective positioning features through a feature conflict resolution mechanism, and output of a core positioning area description of the machining object include: obtaining a machining precision constraint condition, and matching the surface state feature set and the tool interaction feature set with the machining precision constraint condition respectively, to generate a surface state feature subset and a tool interaction feature subset matched with the machining precision constraint condition respectively; establishing an association mapping of the surface state feature subset and the tool interaction feature subset, and determining a corresponding relationship between the surface state feature subset and the tool interaction feature subset through spatial coordinate matching; detecting feature conflicts in the association mapping to obtain conflict features, the feature conflicts including a case that a significant damage area indicated by a surface state feature overlaps a significant pressure area indicated by a tool interaction feature; processing the conflict features through a feature conflict resolution mechanism, and determining feature effectiveness of a conflict area through a weighted voting method, a weighted weight being distributed based on a priority of the machining precision constraint condition; retaining features without conflicts through bidirectional verification to form an effective positioning feature set, the effective positioning feature set including features of a surface texture variation smooth area and features of a tool contact pressure moderate area; extracting a region with the highest positioning precision from the effective positioning feature set as a core positioning area, and describing the core positioning area through boundary coordinates and internal feature parameters of the region, and outputting a core positioning area description.

5. The real-time visual positioning method for industrial automation according to claim 4, characterized in that, The detection of the feature conflicts in the association mapping to obtain the conflict features includes: normalizing an area proportion of a damage area in the surface state feature subset to obtain a damage degree index, a higher damage degree index indicating a more serious damage of the region; normalizing a pressure gradient parameter in the tool interaction feature subset to obtain a pressure intensity index, a higher pressure intensity index indicating a greater pressure of the region; setting a damage degree threshold and a pressure intensity threshold, marking a region with a damage degree index higher than the damage degree threshold as a significant damage area, and marking a region with a pressure intensity index higher than the pressure intensity threshold as a significant pressure area. calculating a spatial overlap degree of the significant loss region and the significant pressure region, the spatial overlap degree being calculated by a ratio of an intersection area of the significant loss region and the significant pressure region to a union area; when the spatial overlap degree is higher than a preset conflict threshold, determining that the significant loss region and the significant pressure region exist feature conflicts, and recording boundary coordinates of a corresponding conflict region and corresponding damage degree indexes and pressure intensity indexes; performing conflict detection on all associated mapped feature regions, collecting information of the conflict regions, and generating conflict features, the conflict features including positions of the conflict regions, conflict types, and conflict degree parameters.

6. The real-time visual positioning method for industrial automation according to claim 4, characterized in that, the conflict features are processed by using a feature conflict resolution mechanism, feature effectiveness of the conflict regions is determined by using a weighted voting method, and distribution of weighted weights is based on priorities of machining precision constraint conditions, including: weight values are distributed to surface roughness requirements and tool contact pressure threshold values according to the priorities of the machining precision constraint conditions, and the higher the priority of a constraint condition is, the higher the weight value corresponding to the constraint condition is; a damage degree index of the conflict region is multiplied by the weight value of the surface roughness requirement, to obtain a voting score of a surface feature; a pressure intensity index of the conflict region is multiplied by the weight value of the tool contact pressure threshold value, to obtain a voting score of an interactive feature; the voting score of the surface feature is compared with the voting score of the interactive feature, if the voting score of the surface feature is higher, region attributes indicated by the surface feature are retained, and corresponding region attributes in the tool interactive feature are removed, and if the voting score of the interactive feature is higher, region attributes indicated by the tool interactive feature are retained, and corresponding region attributes in the surface feature are removed; if the voting score of the surface feature is equal to the voting score of the interactive feature, feature attributes related to key requirements are preferentially retained by referring to the key requirements; feature attributes after conflict resolution are recorded, and an effective positioning feature set is updated.

7. The real-time visual positioning method for industrial automation of claim 1, wherein, the core positioning region description is analyzed, boundary coordinates and a center position of the core positioning region are extracted, and the center position is taken as an initial positioning reference point; motion constraint parameters of the machining equipment are obtained, the motion constraint parameters including stroke ranges, maximum motion speeds, acceleration limits and positioning accuracy levels of each axis of the equipment; an initial positioning reference point is taken as an origin, a machining coordinate system is established, boundary coordinates of the core positioning region are converted to the machining coordinate system, and standardized boundary coordinates are obtained; conversion relationships among the machining coordinate system, an equipment base coordinate system and a tool coordinate system are determined based on the motion constraint parameters of the machining equipment, a multi-coordinate system conversion matrix is constructed, and a machining positioning reference framework is composed; a surface state feature of the core positioning region is analyzed, and when a surface texture change rate exceeds a preset change rate, a position correction rule is constructed, the position correction rule including a correction amount calculation method along a vertical direction of the texture; a tool interactive feature is analyzed, and when a tool contact pressure gradient is abnormal, a posture calibration rule is constructed, the posture calibration rule including a rotation angle adjustment logic of the tool around each axis. ​ According to the acceleration limit parameter of the machining equipment, a feed speed adjustment rule is constructed, which contains speed switching threshold values of different machining stages; The position correction rule, the posture calibration rule, and the feed speed adjustment rule are integrated into a dynamic adjustment rule set, and each dynamic adjustment rule in the dynamic adjustment rule set contains a trigger condition and a corresponding adjustment parameter calculation method.

8. The real-time visual positioning method for industrial automation according to claim 7, characterized in that, Based on the motion constraint parameter of the machining equipment, the conversion relationship between the machining coordinate system and the equipment base coordinate system and the tool coordinate system is determined, a multi-coordinate system conversion matrix is constructed, and a machining positioning reference frame is formed, which includes: The conversion parameters between the equipment base coordinate system and the motion coordinate system of each axis are obtained through the equipment kinematics model, and the conversion parameters include translation and rotation angles; The coordinates of the origin of the tool coordinate system in the equipment base coordinate system are measured by a laser tracker to obtain the initial conversion relationship between the tool coordinate system and the equipment base coordinate system; The coordinates of the centroid position of the core positioning area in the image coordinate system are converted to the equipment base coordinate system to serve as the origin of the machining coordinate system; According to the positional relationship between the origin of the machining coordinate system and the equipment base coordinate system, the conversion matrix between the machining coordinate system and the equipment base coordinate system is calculated, which contains translation parameters in three directions and rotation parameters in three directions; The relative positional relationship between the tool coordinate system and the machining coordinate system is obtained through a tool calibration method, and the conversion matrix therebetween is calculated, with the matrix parameters of the conversion matrix being determined based on the measured distance between the tool tip and the origin of the machining coordinate system; The conversion matrix between the machining coordinate system and the equipment base coordinate system and the conversion matrix between the tool coordinate system and the machining coordinate system are integrated to construct a multi-coordinate system conversion matrix set; Taking the multi-coordinate system conversion matrix set as the core, the standardized boundary coordinates of the core positioning area and the equipment motion range parameters are combined to construct a machining positioning reference frame, which contains the origin positions, axis system directions, and effective working ranges of each coordinate system.

9. The real-time visual positioning method for industrial automation of claim 1, wherein, The machining positioning reference frame, the dynamic adjustment rule set, and the feedback features of the real-time image stream are fused to generate a positioning guidance instruction sequence containing tool displacement correction amounts and feed rhythm adjustment values, which includes: The latest feedback features are extracted from the real-time image stream, including surface state feedback features and tool interaction feedback features of the current machining position; The feedback features are converted to the machining coordinate system of the machining positioning reference frame to obtain standardized feedback features, which contain deviation values of the current position from the core positioning area; The position correction rule in the dynamic adjustment rule set is called to compare the surface texture change rate in the standardized feedback features with the rule trigger condition, and if the rule trigger condition is met, the tool displacement correction amount is calculated; The posture calibration rule in the dynamic adjustment rule set is called to compare the pressure gradient parameter in the standardized feedback features with the rule trigger condition, and if the rule trigger condition is met, the tool posture adjustment angle is calculated; The feed speed adjustment rule in the dynamic adjustment rule set is called to calculate a feed rhythm adjustment value according to the current machining stage and the surface damage degree parameter, the feed rhythm adjustment value being a correction coefficient of the basic feed speed; The tool displacement correction amount, the tool posture adjustment angle and the feed rhythm adjustment value are sorted in time sequence, wherein each time point corresponds to a group of adjustment parameters; Timestamp information is added to each group of adjustment parameters to obtain timestamped adjustment parameters, and the timestamped adjustment parameters are combined into a positioning guide instruction sequence, each instruction in the positioning guide instruction sequence containing an instruction identifier, an execution time, a tool displacement correction amount, a posture adjustment angle and a feed rhythm adjustment value; The positioning guide instruction sequence is checked, and the checked positioning guide instruction sequence is transmitted to a machining control system to enable the machining control system to perform positioning guide actions according to the positioning guide instruction sequence.

10. A real-time visual positioning system for industrial automation, characterized by The processor and the memory are connected, the memory is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the memory to realize the real-time visual positioning method for industrial automation in any one of claims 1-9.

Images