A visual positioning method and system for manipulator processing

Abstract

The present invention provides a visual positioning method and system for robotic arm processing, which relates to the technical field of visual image processing. The present invention takes two photos of the workpiece before and after the operation by a depth camera to obtain the translation deviation and rotation deviation of the workpiece, trains a deviation model in combination with the positioning information of the robotic arm, provides multiple workpiece position information and attitude information for the robotic arm according to the translation deviation and rotation deviation output by the deviation model, scores the translation deviation and rotation, adopts the positioning information formed by the translation deviation and rotation deviation with the lowest score, and alarms through the position information and attitude information of the workpiece after the operation of the robotic arm, including an information collection module, a model training module, a calculation and statistics module, and an alarm module. The present invention provides more accurate positioning information for the robotic arm and alarms for the failed operations of the robotic arm.

Description

Technical Field

[0001] The present invention relates to the technical field of visual image processing, and specifically provides a visual positioning method and system for robot machining. Background Art

[0002] During the robot machining process, the positioning accuracy is one of the key factors affecting the machining quality and efficiency. Traditional robot positioning methods usually rely on a fixed coordinate system and precise control of the robotic arm. However, due to workpiece positioning errors, attitude changes in the actual machining environment, and the operating errors of the robot itself, the positioning accuracy is often not ideal. Especially in complex machining tasks, the errors will further accumulate, affecting the final machining effect. In addition, most of the existing visual positioning methods lack an effective feedback mechanism and cannot correct errors in real time, resulting in possible position deviations of the robot during the machining process, thus affecting the machining accuracy.

[0003] In the prior art, the publication number CN202410757152.X discloses a visual positioning method and system for robot machining, which obtains an image of the area to be grasped and obtains a to-be-detected image based on the image of the area to be grasped; determines the probability index of the edge pixel points of the connected domain in the to-be-detected image belonging to the workpiece based on the edge features of the workpiece in the to-be-detected image; determines the area where the workpiece is located in the to-be-detected image based on the probability index of the edge pixel points of the connected domain in the to-be-detected image belonging to the workpiece; determines the grasping priority index of the workpiece in the to-be-detected image based on the gray value of the area where the workpiece is located in the to-be-detected image; determines the workpiece with the largest grasping priority index as the current grasping target, and controls the robot to grasp the current grasping target.

[0004] Although the disclosed technical document realizes the positioning and operation of the robot for the workpiece, it cannot avoid the influence of the robot's own errors on the workpiece operation, and there is no determination and warning for the robot's operation mistakes.

[0005] The above information disclosed in the background art section is only used to enhance the understanding of the background of the present disclosure. Therefore, it may include information that does not constitute the prior art known to those of ordinary skill in the art. Summary of the Invention

[0006] The purpose of the present invention is to provide a visual positioning method and system for robot machining to solve the problems raised in the above background art.

[0007] To achieve the above purpose, the present invention provides the following technical solutions:

[0008] A visual positioning method for robot machining, the specific steps include:

[0009] Step 1: Take a photo of the workpiece before the manipulator operates on the workpiece to obtain the first position information and the first attitude information, obtain the positioning information of the manipulator for the workpiece. After the manipulator operation is completed, take a photo of the workpiece again to obtain the second position information and the second attitude information. Obtain the translation deviation and rotation deviation of the workpiece in the two photos through the IPC algorithm, and add the manipulator positioning information to the deviation dataset formed by summarizing the translation deviation and rotation deviation;

[0010] Step 2: Preprocess the deviation dataset and input it into a linear regression model for training. Use the translation deviation and rotation deviation as labels to train the positioning information. Set the loss function as the root mean square of the translation deviation and the root mean square of the rotation deviation to obtain the deviation model;

[0011] Step 3: Deploy the deviation model in the system. Obtain the first position information and the first attitude information of the workpiece through a depth camera, and the manipulator generates positioning information. Input the positioning information generated by the manipulator into the deviation model to obtain the translation deviation and rotation deviation. Correct the first position information and the first attitude information according to the translation deviation and rotation deviation and generate new position information and attitude information, and input them into the deviation model again to obtain a total of five translation deviations and rotation deviations;

[0012] Step 4: Score the five deviations respectively, and perform manipulator operations using the position information and attitude information formed by the translation deviation and rotation deviation with the lowest score;

[0013] Step 5: After the manipulator operation is completed, the depth camera takes a photo of the workpiece for the second time, and forms the final translation deviation, the final attitude deviation and the final score according to the photo taken for the first time. Set a threshold value, and alarm for situations exceeding the threshold value.

[0014] Further, before the manipulator works officially, obtain the position data of the workpiece operated by the manipulator through a depth camera. Set the depth camera at a fixed position beside the workpiece. After the workpiece reaches the specified position, the manipulator positions the workpiece and reaches the position to be operated. At this time, take a photo of the workpiece for the first time through the depth camera to obtain the position information and attitude information of the workpiece;

[0015] The logic for obtaining the position information and attitude information of the workpiece is as follows:

[0016] Smooth the image of the photo taken by the depth camera through a Gaussian filter. Each pixel point of the photo contains a depth value. Convert the pixel points in the picture into three-dimensional coordinates, and the formula is as follows:

[0017]

[0018]

[0019]

[0020] Among them, is the pixel coordinate in the image, is the depth value of the pixel, is the three-dimensional coordinate corresponding to the pixel point, is the horizontal focal length of the depth camera, is the vertical focal length of the depth camera;

[0021] Calibrate the photo obtained from the first photo-taking as the first photo, and calibrate the position information and pose information obtained from the first photo as the first position information and the first pose information. Then, the manipulator starts to operate on the workpiece. At the first moment when the manipulator finishes the operation, take a second photo of the workpiece through the same depth camera. Calibrate the photo obtained from the second photo-taking as the second photo, and calibrate the position information and pose information obtained from the second photo as the second position information and the second pose information, and obtain the second position information and the second pose information in the second photo. Remove the occluded parts of the manipulator in the workpiece picture, and respectively obtain the point clouds of the workpiece in the two photos. The point cloud is a set of three-dimensional coordinates of the workpiece pixel points in each photo. Obtain the translation deviation and rotation deviation of the workpiece in the two photos through the IPC algorithm, and summarize the translation deviation and rotation deviation into a deviation data set;

[0022] When the manipulator positions the workpiece, generate the positioning information of the manipulator according to the manipulator's operation on the workpiece. The positioning information includes the position data of each joint of the manipulator and the load data of the execution part of the manipulator. The load data includes the operation type, weight, and force. Perform a four-dimensional one-hot encoding on the operation type that is different from each other, and map the positioning information of the manipulator in the deviation data set. Each positioning information sample corresponds one-to-one to the translation deviation and rotation deviation caused each time.

[0023] Furthermore, preprocess the deviation data set. Manually screen and remove the operation failure samples, normalize all the sample data, input the deviation data set into a linear regression model for training, use the translation deviation and rotation deviation as labels to train the positioning data, obtain the different translation deviations and rotation deviations caused by the manipulator under the positioning information, set the loss function as the root mean square of the translation deviation and the root mean square of the rotation deviation, set the root mean square threshold of the translation deviation and the root mean square threshold of the rotation deviation. When both the root mean square threshold of the translation deviation and the root mean square threshold of the rotation deviation are satisfied, the model training is completed. Calibrate the trained model as the deviation model, and output the translation deviation and rotation deviation of the manipulator each time it executes an operation.

[0024] Further, deploy the deviation model in the manipulator operating system. After the workpiece reaches the specified position, take the first photo through the depth camera to obtain the first position information and the first attitude information of the workpiece, and calibrate the first position information and the first attitude information as position information I and attitude information I respectively. The manipulator performs positioning according to the position information I and the attitude information I to obtain the manipulator positioning information. At this time, the obtained manipulator positioning information is calibrated as manipulator positioning information I. Map the manipulator positioning information I to the deviation data set and input it into the deviation model to obtain the translation deviation I and the rotation deviation I. Use the translation deviation I and the rotation deviation I to correct the position information I and the attitude information I to obtain the position information II and the attitude information II. The formula for the correction is as follows:

[0025]

[0026] Among them, is the value on the x-axis after the correction of the workpiece, is the value on the x-axis before the correction of the workpiece, is the translation deviation of the x-axis, is the value on the y-axis after the correction of the workpiece, is the value on the y-axis before the correction of the workpiece, is the translation deviation of the y-axis, is the value on the z-axis after the correction of the workpiece, is the value on the z-axis before the correction of the workpiece, is the translation deviation of the z-axis, is the value on the a-axis after the correction of the workpiece, is the value on the a-axis before the correction of the workpiece, is the rotation deviation of the a-axis, is the value on the b-axis after the correction of the workpiece, is the value on the b-axis before the correction of the workpiece, is the rotation deviation of the b-axis, is the value on the c-axis after the correction of the workpiece, is the value on the c-axis before the correction of the workpiece, is the rotation deviation of the c-axis;

[0027] Obtain the manipulator positioning information II again according to the position information II and the attitude information II. Map the manipulator positioning information II to the deviation data set and input it into the deviation model to obtain the translation deviation II and the rotation deviation II. Use the translation deviation II and the rotation deviation II to correct the position information I and the attitude information I again to obtain the position information III and the attitude information III. Repeat this process until five translation deviations and rotation deviations are obtained respectively. The last three are calibrated as the translation deviation III and the rotation deviation III, the translation deviation IV and the rotation deviation IV, and the translation deviation V and the rotation deviation V respectively.

[0028] Further, the translational deviations and rotational deviations obtained five times are respectively scored, and the basis formula is as follows:

[0029]

[0030] Wherein, is the scoring value, is the translational deviation of the x-axis, is the translational deviation of the y-axis, is the translational deviation of the z-axis, is the rotational deviation of the a-axis, is the rotational deviation of the b-axis, is the rotational deviation of the c-axis, and are weight coefficients respectively, , and ;

[0031] Select the position information and attitude information formed by the translational deviation and rotational deviation with the lowest score, and form the manipulator positioning information. According to this positioning information, the manipulator operates on the workpiece.

[0032] Further, when the manipulator completes the operation, the depth camera takes a second photo of the workpiece, and obtains the second position information and the second attitude information. Analyze the translational deviation and attitude deviation of the workpiece in the two photos, calibrate this translational deviation and attitude deviation as the final translational deviation and the final attitude deviation, and score the final translational deviation and the final attitude deviation to form the final score. Respectively set the final translational deviation threshold, the final attitude deviation threshold and the final score threshold. When at least one of the values exceeds the corresponding threshold, an alarm is given and it is determined that the manipulator operation fails.

[0033] The present invention also includes a visual positioning system for manipulator processing. The visual positioning system is used to execute the above-mentioned visual positioning method for manipulator processing, including:

[0034] Information collection module: Obtain the position information and attitude information of the workpiece through the depth camera, obtain the manipulator positioning information, and obtain the translational deviation and rotational deviation when the workpiece is photographed twice;

[0035] Model training module: Summarize the translational deviation, rotational deviation and positioning information into a deviation data set and input it into a linear regression model to obtain a deviation model, and input the positioning information of the manipulator into the deviation model to obtain the translational deviation and rotational deviation;

[0036] Calculation and statistics module: Calculate different position information and attitude information according to the translational deviation and rotational deviation, and count the score, and instruct the manipulator to operate;

[0037] Alarm module: Alarm according to the obtained final translation deviation, final attitude deviation and final score.

[0038] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0039] The present invention trains a deviation model according to the way of obtaining the translation deviation and rotation deviation caused by the manipulator to the workpiece by the depth camera, forms the positioning of the manipulator under different deviation correction conditions, thereby reducing the influence of the manipulator's own error on the workpiece operation, providing more accurate positioning information for the manipulator, and judging whether the operation of the manipulator is successful according to the two photos taken by the depth camera. Description of the Drawings

[0040] Figure 1 It is a schematic diagram of the overall method flow of the present invention;

[0041] Figure 2 It is a schematic diagram of the system composition of the present invention. Detailed Embodiments

[0042] To make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below in conjunction with specific embodiments.

[0043] It should be noted that unless otherwise defined, the technical terms or scientific terms used in the present invention should have the ordinary meanings understood by those with ordinary skills in the field to which the present invention belongs. The "first", "second" and similar terms used in the present invention do not represent any order, quantity or importance, but are only used to distinguish different components. The terms such as "including" or "comprising" mean that the elements or objects appearing before this word cover the elements or objects listed after this word and their equivalents, without excluding other elements or objects. The terms such as "connected" or "coupled" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The terms such as "upper", "lower", "left" and "right" are only used to represent relative positional relationships, and when the absolute position of the object being described changes, the relative positional relationship may also change accordingly.

[0044] Embodiment:

[0045] Please refer to Figure 1 The present invention provides a technical solution:

[0046] A visual positioning method for manipulator processing, the specific steps include:

[0047] Step 1: Take a photo of the workpiece before the manipulator operates on the workpiece to obtain the first position information and the first attitude information, obtain the positioning information of the manipulator with respect to the workpiece, take a photo of the workpiece again after the manipulator finishes the operation, obtain the second position information and the second attitude information, obtain the translational deviation and rotational deviation of the workpiece in the two photos through the IPC algorithm, and add the manipulator positioning information to the deviation dataset formed by summarizing the translational deviation and rotational deviation;

[0048] The content of Step 1 is as follows:

[0049] Before the manipulator starts formal work, obtain the position data of the workpiece operated by the manipulator through a depth camera. Set a depth camera at a fixed position beside the workpiece. After the workpiece reaches the specified position, the manipulator positions the workpiece and reaches the position to be operated. At this time, take the first photo of the workpiece through the depth camera to obtain the position information and attitude information of the workpiece;

[0050] The techniques that can be referred to for the manipulator to position the workpiece are not limited to but include the techniques disclosed in a visual positioning method and system for manipulator processing with the publication number CN118322217A, and other techniques for obtaining the manipulator positioning information by identifying the workpiece position should also be included.

[0051] The logic for obtaining the position information and attitude information of the workpiece is as follows:

[0052] Smooth the image of the photo taken by the depth camera through a Gaussian filter. Each pixel point of the photo contains a depth value. Convert the pixel points in the picture into three-dimensional coordinates according to the following formula:

[0053]

[0054]

[0055]

[0056] Among them, is the pixel coordinate in the image, is the depth value of this pixel, is the three-dimensional coordinate corresponding to this pixel point, is the horizontal focal length of the depth camera, is the vertical focal length of the depth camera;

[0057] Taking the upper left corner of the two-dimensional image as the origin of pixel coordinates, the horizontal right direction as the positive direction of the x-axis of pixel coordinates, the vertical downward direction as the positive direction of the y-axis of pixel coordinates, taking the optical center of the depth camera as the origin of three-dimensional coordinates, the horizontal right as the positive direction of the x-axis of three-dimensional coordinates, the perpendicular upward direction as the positive direction of the y-axis of three-dimensional coordinates, and the depth direction from the camera to the object as the positive direction of the z-axis of three-dimensional coordinates, according to the internal parameters already set by the depth camera, convert the pixel coordinates into three-dimensional coordinates according to the two-dimensional image.

[0058] This method of converting two-dimensional information into three-dimensional coordinates has been maturely applied in this field. This formula is a commonly used formula in this field and will not be elaborated here.

[0059] By converting the two-dimensional information of each pixel point in the photo into three-dimensional coordinates, the three-dimensional information of the workpiece can be constructed, which is convenient for further analyzing the translational deviation and rotational deviation of the workpiece between the two photos.

[0060] Calibrate the photo obtained from the first photographing as the first photo, and calibrate the position information and pose information obtained from the first photo as the first position information and the first pose information. Then, the manipulator starts to operate on the workpiece. At the first moment when the manipulator completes the operation, take a second photograph of the workpiece through the same depth camera. Calibrate the photo obtained from the second photographing as the second photo, and calibrate the position information and pose information obtained from the second photo as the second position information and the second pose information, and obtain the second position information and the second pose information in the second photo. Remove the occluded part of the manipulator in the workpiece picture, and respectively obtain the point clouds of the workpiece in the two photos. The point cloud is a set of three-dimensional coordinates of the workpiece pixel points in each photo. Obtain the translational deviation and rotational deviation of the workpiece in the two photos through the IPC algorithm, and summarize the translational deviation and rotational deviation into a deviation data set;

[0061] As a mature algorithm, the IPC algorithm has been widely used in this field, such as the calibration device and method of the 3D camera and robotic arm coordinate system with the publication number CN111716340A for the determination of target deviation in pictures.

[0062] When the manipulator positions the workpiece, generate the positioning information of the manipulator according to the manipulator's operation on the workpiece. The positioning information includes the position data of each joint of the manipulator and the load data of the execution part of the manipulator. The load data includes the operation type, weight, and force. Perform a four-dimensional one-hot encoding on the operation types that are different from each other, such as [0,0,0,1] for clamping and [0,0,1,0] for lifting, etc. Map the positioning information of the manipulator in the deviation data set, and each positioning information sample corresponds one-to-one to the translational deviation and rotational deviation caused each time.

[0063] Step 2: Preprocess the deviation data set and input it into a linear regression model for training. Use the translational deviation and rotational deviation as labels to train the positioning information. Set the loss function as the root mean square of the translational deviation and the root mean square of the rotational deviation, and obtain the deviation model;

[0064] The above step 2 includes the following contents:

[0065] Preprocess the deviation data set. Manually screen and remove the failed samples, normalize all the sample data, input the deviation data set into the linear regression model for training. Use the translational deviation and rotational deviation as labels to train the positioning data, and obtain the different translational deviations and rotational deviations caused by the manipulator under the positioning information. Set the loss function as the root mean square of the translational deviation and the root mean square of the rotational deviation. Set the root mean square threshold of the translational deviation and the root mean square threshold of the rotational deviation. When both the root mean square threshold of the translational deviation and the root mean square threshold of the rotational deviation are satisfied, the model training is completed. Calibrate the trained model as the deviation model, and output the translational deviation and rotational deviation of the manipulator each time it executes an operation.

[0066] Step 3: Deploy the deviation model in the system. Obtain the first position information and the first attitude information of the workpiece through a depth camera, and the manipulator generates positioning information. Input the positioning information generated by the manipulator into the deviation model to obtain the translational deviation and rotational deviation. Correct the first position information and the first attitude information according to the translational deviation and rotational deviation to generate new position information and attitude information, and input them into the deviation model again to obtain a total of five translational deviations and rotational deviations;

[0067] The above step 3 includes the following contents:

[0068] Deploy the deviation model in the manipulator operating system. After the workpiece reaches the specified position, take the first photo through the depth camera to obtain the first position information and the first attitude information of the workpiece, and label the first position information and the first attitude information as position information Ⅰ and attitude information Ⅰ respectively. The manipulator performs positioning according to the position information Ⅰ and the attitude information Ⅰ to obtain the manipulator positioning information, and at this time, the obtained manipulator positioning information is labeled as manipulator positioning information Ⅰ. Map the manipulator positioning information Ⅰ to the deviation data set and input it into the deviation model to obtain the translational deviation Ⅰ and the rotational deviation Ⅰ. Correct the position information Ⅰ and the attitude information Ⅰ according to the translational deviation Ⅰ and the rotational deviation Ⅰ to obtain the position information Ⅱ and the attitude information Ⅱ. The formula for the correction is as follows:

[0069]

[0070] Where, is the value on the x-axis after the correction of the workpiece, is the value on the x-axis before the correction of the workpiece, is the translation deviation of the x-axis, is the value on the y-axis after workpiece correction, is the value on the y-axis before workpiece correction, is the translation deviation of the y-axis, is the value on the z-axis after workpiece correction, is the value on the z-axis before workpiece correction, is the translation deviation of the z-axis, is the value on the a-axis after workpiece correction, is the value on the a-axis before workpiece correction, is the rotation deviation of the a-axis, is the value on the b-axis after workpiece correction, is the value on the b-axis before workpiece correction, is the rotation deviation of the b-axis, is the value on the c-axis after workpiece correction, is the value on the c-axis before workpiece correction, is the rotation deviation of the c-axis;

[0071] The translation deviation and rotation deviation are caused by the positioning accuracy of the manipulator itself. Before and after the operation, the workpiece is subjected to translational and rotational forces. The manipulator has its own high-precision operation range, and it cannot ensure that each workpiece is located within the high-precision operation range of the manipulator. The translational deviation and rotation deviation before and after the operation are used to evaluate the accuracy of the manipulator for this operation. Moreover, the numerical value of the workpiece deviation itself represents the deviation of the manipulator in this operation. Therefore, correcting the first position information and the first rotation information with the translation deviation and rotation deviation can enable the manipulator to make up for the deviation caused by its own accuracy. When correcting, when calculating the spatial position of the workpiece, add the translation deviation value to the first position information on the x, y, and z axes respectively. When calculating the attitude of the workpiece, add the rotation deviation to the first attitude information on the a, b, and c axes respectively.

[0072] According to the position information II and the attitude information II, obtain the manipulator positioning information II again. Map the manipulator positioning information II to the deviation data set and input it into the deviation model to obtain the translation deviation II and the rotation deviation II. Use the translation deviation II and the rotation deviation II to correct the position information I and the attitude information I again to obtain the position information III and the attitude information III. Repeat this process until five translation deviations and rotation deviations are obtained respectively. The last three are respectively calibrated as the translation deviation III and the rotation deviation III, the translation deviation IV and the rotation deviation IV, and the translation deviation V and the rotation deviation V.

[0073] The workpiece is in different positions. The manipulator can adopt different methods for positioning the workpiece, and different methods will cause different deviations of the manipulator. If the best positioning information is unknown, starting from the first positioning information input into the deviation model to obtain the translational deviation and rotational deviation, a method of finite attempts is used to obtain different translational deviations and rotational deviations, providing reference conditions for selecting the best positioning information in the next step.

[0074] Step 4: Score the five deviations respectively, and use the position information and attitude information formed by the translational deviation and rotational deviation with the lowest score for manipulator operation;

[0075] The said Step 4 includes the following contents:

[0076] Score the translational deviations and rotational deviations obtained five times respectively, and the basis formula is as follows:

[0077]

[0078] Among them, is the score value, is the translational deviation of the x-axis, is the translational deviation of the y-axis, is the translational deviation of the z-axis, is the rotational deviation of the a-axis, is the rotational deviation of the b-axis, is the rotational deviation of the c-axis, and are the weight coefficients respectively, , and ;

[0079] Add the absolute values of the translational deviations of the x, y, and z axes and the absolute values of the rotational deviations of the a, b, and c axes after weighting. The absolute value of the deviation represents the magnitude of the impact of the manipulator on the position and attitude of the workpiece before and after operation. The greater the impact, the greater the absolute value of the translational deviation and rotational deviation, which means that the positioning of the manipulator for the workpiece is less accurate this time, and the score is higher. When the spatial position and attitude of the workpiece deviate within a small range, the influence deviation caused by displacement is greater than that caused by rotation. Therefore, the cumulative coefficient of the absolute value of the translational deviation is greater than the cumulative coefficient of the absolute value of the rotational deviation.

[0080] Select the position information and attitude information formed by the translational deviation and rotational deviation with the lowest score, and form the manipulator positioning information. According to this positioning information, the manipulator operates on the workpiece.

[0081] The lowest score means that the robot has the least impact on the position and posture of the workpiece before and after the operation, which means that the robot is in the most accurate state under this positioning information, making it easier to obtain the best operating effect.

[0082] Step 5: After the robot operation is completed, the depth camera takes a second photo of the workpiece, and forms the final translation deviation, final posture deviation and final score based on the first photo. The threshold is set and an alarm is issued if the threshold is exceeded.

[0083] The step 5 includes the following contents:

[0084] When the robot completes the operation, the depth camera takes a second photo of the workpiece and obtains the second position information and the second posture information. The translation deviation and posture deviation of the workpiece in the two photos are analyzed, and the translation deviation and posture deviation are calibrated as the final translation deviation and final posture deviation. The final translation deviation and final posture deviation are scored to form a final score. The final translation deviation threshold, final posture deviation threshold and final score threshold are set respectively. When at least one value exceeds the corresponding threshold, an alarm is issued and the robot operation is deemed to have failed.

[0085] By taking photos twice, the robot can obtain successful operation results under the most accurate positioning state. For alarms of operation failure, personnel can be reminded to take measures in time.

[0086] like Figure 2 As shown, the present invention also includes a visual positioning system for robot processing, and the visual positioning system is used to execute the above-mentioned visual positioning method for robot processing, including:

[0087] Information collection module: obtains the position and posture information of the workpiece through the depth camera, obtains the positioning information of the robot, and obtains the translation deviation and rotation deviation of the workpiece when taking two photos;

[0088] Model training module: Aggregate the translation deviation, rotation deviation and positioning information into a deviation data set and input it into the linear regression model to obtain the deviation model. Input the positioning information of the manipulator into the deviation model to obtain the translation deviation and rotation deviation.

[0089] Calculation and statistics module: calculates different position information and posture information based on translation deviation and rotation deviation, and statistically scores them to instruct the manipulator to operate;

[0090] Alarm module: issues an alarm based on the final translation deviation, final posture deviation and final score obtained.

[0091] The above formulas are all dimensionless and take their numerical values for calculation. The formulas are obtained by collecting a large amount of data and performing software simulation to get a formula closest to the actual situation. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0092] The above embodiments can be implemented in whole or in part by software, hardware, firmware, or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. Those skilled in the art will realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software methods depends on the specific application and design constraints of the technical solution.

[0093] The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units. They can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0094] As described above, the above are only specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application, and all of them should be covered by the protection scope of the present application.

Claims

1. A visual positioning method for manipulator processing, characterized in that, The specific steps include: Step 1: Take a photo of the workpiece before the manipulator operates on the workpiece to obtain the first position information and the first attitude information, obtain the positioning information of the manipulator for the workpiece, take a photo of the workpiece again after the manipulator operation is completed to obtain the second position information and the second attitude information, obtain the translation deviation and rotation deviation of the workpiece in the two photos through the IPC algorithm, and add the manipulator positioning information to the deviation dataset formed by summarizing the translation deviation and rotation deviation; Step 2: Preprocess the deviation dataset and input it into a linear regression model for training. Use the translation deviation and rotation deviation as labels to train the positioning information, set the loss function as the root mean square of the translation deviation and the root mean square of the rotation deviation, and obtain the deviation model; Step 3: Deploy the deviation model in the system. Obtain the first position information and the first attitude information of the workpiece through a depth camera, and the manipulator generates positioning information. Input the positioning information generated by the manipulator into the deviation model to obtain the translation deviation and rotation deviation. Correct the first position information and the first attitude information according to the translation deviation and rotation deviation and generate new position information and attitude information, and input them into the deviation model again. A total of five translation deviations and rotation deviations are obtained; Step 4: Score the five deviations respectively, and use the position information and attitude information formed by the translation deviation and rotation deviation with the lowest score for the manipulator operation; Score the translation deviations and rotation deviations obtained five times respectively, and the basis formula is as follows: Among them, is the scoring value, is the translation deviation of the x-axis, is the translation deviation of the y-axis, is the translation deviation of the z-axis, is the rotation deviation of the a-axis, is the rotation deviation of the b-axis, is the rotation deviation of the c-axis, and are the weight coefficients respectively, , and ; Select the position information and attitude information formed by the translation deviation and rotation deviation with the lowest score, and form the manipulator positioning information. According to this positioning information, make the manipulator operate on the workpiece; Step 5: After the manipulator operation is completed, the depth camera takes a photo of the workpiece for the second time, and form the final translation deviation, the final attitude deviation and the final score according to the photo taken for the first time. Set a threshold value, and alarm for the situation exceeding the threshold value.

2. The visual positioning method for manipulator processing according to claim 1, characterized in that: Before the manipulator works formally, obtain the position data of the manipulator operating on the workpiece through a depth camera. Set the depth camera at a fixed position beside the workpiece. After the workpiece reaches the specified position, the manipulator positions the workpiece and reaches the position to be operated. At this time, take a photo of the workpiece for the first time through the depth camera to obtain the position information and attitude information of the workpiece; The logic for obtaining the position information and attitude information of the workpiece is as follows: Smooth the image of the photo taken by the depth camera through a Gaussian filter. Each pixel point of the photo contains a depth value. Convert the pixel points in the picture into three-dimensional coordinates, and the basis formula is as follows: wherein, is the pixel coordinate in the image, is the depth value of the pixel, is the three-dimensional coordinate corresponding to the pixel point, is the horizontal focal length of the depth camera, is the vertical focal length of the depth camera; The photo obtained from the first photo shoot is designated as the first photo, and the position information and pose information obtained from the first photo are designated as the first position information and the first pose information. Then, the manipulator starts to operate on the workpiece. At the first moment when the manipulator finishes the operation, the workpiece is photographed for the second time by the same depth camera. The photo obtained from the second photo shoot is designated as the second photo, and the position information and pose information obtained from the second photo are designated as the second position information and the second pose information. Also, the second position information and the second pose information in the second photo are obtained, and the occluded part of the manipulator in the workpiece picture is removed. The point clouds of the workpiece in the two photos are respectively obtained. The point cloud is a set of three-dimensional coordinates of the workpiece pixel points in each photo. The translation deviation and rotation deviation of the workpiece in the two photos are obtained through the IPC algorithm, and the translation deviation and rotation deviation are summarized into a deviation data set. When the manipulator positions the workpiece, the positioning information of the manipulator is generated according to the manipulator's operation on the workpiece. The positioning information includes the position data of each joint of the manipulator and the load data of the execution part of the manipulator. The load data includes the operation type, weight, and force. The operation type is encoded with a four-dimensional one-hot encoding that is mutually different. The positioning information of the manipulator is mapped in the deviation data set, and each positioning information sample corresponds one-to-one to the translation deviation and rotation deviation caused each time.

3. The visual positioning method for manipulator processing according to claim 2, characterized in that: The deviation data set is preprocessed. The operation failure samples are manually screened and removed, and all the sample data are normalized. The deviation data set is input into a linear regression model for training. Using the translation deviation and rotation deviation as labels, the positioning data is trained to obtain the different translation deviations and rotation deviations caused by the manipulator under the positioning information. The loss function is set as the root mean square of the translation deviation and the root mean square of the rotation deviation. The root mean square threshold of the translation deviation and the root mean square threshold of the rotation deviation are set. When both the root mean square threshold of the translation deviation and the root mean square threshold of the rotation deviation are satisfied, the model training is completed. The trained model is designated as the deviation model, and the translation deviation and rotation deviation when the manipulator executes each operation are output.

4. The visual positioning method for manipulator processing according to claim 3, characterized in that: The deviation model is deployed in the manipulator operating system. After the workpiece reaches the specified position, the first photo of the workpiece is taken by the depth camera to obtain the first position information and the first pose information of the workpiece, and the first position information and the first pose information are respectively designated as position information Ⅰ and pose information Ⅰ. The manipulator positions itself according to the position information Ⅰ and the pose information Ⅰ to obtain the manipulator positioning information. The manipulator positioning information obtained at this time is designated as manipulator positioning information Ⅰ. The manipulator positioning information Ⅰ is mapped to the deviation data set and input into the deviation model to obtain the translation deviation Ⅰ and the rotation deviation Ⅰ. The translation deviation Ⅰ and the rotation deviation Ⅰ are used to correct the position information Ⅰ and the pose information Ⅰ to obtain the position information Ⅱ and the pose information Ⅱ. The formula for the correction is as follows: Among them, is the value on the x-axis after workpiece correction, is the value on the x-axis before workpiece correction, is the translation deviation of the x-axis, is the value on the y-axis after workpiece correction, is the value on the y-axis before workpiece correction, is the translation deviation of the y-axis, is the value on the z-axis after workpiece correction, is the value on the z-axis before workpiece correction, is the translation deviation of the z-axis, is the value on the a-axis after workpiece correction, is the value on the a-axis before workpiece correction, is the rotation deviation of the a-axis, is the value on the b-axis after workpiece correction, is the value on the b-axis before workpiece correction, is the rotation deviation of the b-axis, is the value on the c-axis after workpiece correction, is the value on the c-axis before workpiece correction, is the rotation deviation of the c-axis; According to the position information II and the posture information II, the manipulator positioning information II is obtained again, the manipulator positioning information II is mapped to the deviation data set and input into the deviation model to obtain the translation deviation II and the rotation deviation II, and the translation deviation II and the rotation deviation II are used to correct the position information I and the posture information I again to obtain the position information III and the posture information III. This step is repeated until five translation deviations and rotation deviations are obtained respectively. The last three times are calibrated as translation deviation III and rotation deviation III, translation deviation IV and rotation deviation IV, and translation deviation V and rotation deviation V, respectively.

5. A vision positioning method for manipulator machining according to claim 4, characterized in that: When the robot completes the operation, the depth camera takes a second photo of the workpiece and obtains the second position information and the second posture information. The translation deviation and posture deviation of the workpiece in the two photos are analyzed, and the translation deviation and posture deviation are calibrated as the final translation deviation and final posture deviation. The final translation deviation and final posture deviation are scored to form a final score. The final translation deviation threshold, final posture deviation threshold and final score threshold are set respectively. When at least one value exceeds the corresponding threshold, an alarm is issued and the robot operation is deemed to have failed.

6. A visual positioning system for robotic machining, the visual positioning system being used to execute a visual positioning method for robotic machining according to any one of claims 1-5, characterized in that, include: Information collection module: obtains the position and posture information of the workpiece through the depth camera, obtains the positioning information of the robot, and obtains the translation deviation and rotation deviation of the workpiece when taking two photos; Model training module: Aggregate the translation deviation, rotation deviation and positioning information into a deviation data set and input it into the linear regression model to obtain the deviation model. Input the positioning information of the manipulator into the deviation model to obtain the translation deviation and rotation deviation. Calculation and statistics module: calculates different position information and posture information based on translation deviation and rotation deviation, and statistically scores them to instruct the manipulator to operate; Alarm module: issues an alarm based on the final translation deviation, final posture deviation and final score obtained.

Images