Method and equipment for controlling double-arm robot to automatically grind crystal material and medium

Abstract

The invention discloses a method and equipment for controlling a double-arm robot to automatically grind crystal materials and a medium, belongs to the field of experiments for controlling robots to make crystal materials, and is used for a double-arm robot grinding system.The method comprises the steps that 1, the double-arm robot is controlled to move to a grinding platform, and a camera is used for obtaining mortar pose information to locate the position of a mortar; step 2, acquiring a color distribution image of a crystal in the mortar by using a camera to judge subsequent operation, and if the grinding is judged to be stopped, executing step 5; if the operation is determined to be gathering operation, executing the step 3, and if the operation is determined to be grinding operation, executing the step 4; 3, a slave arm of the double-arm robot is controlled to gather the crystals in the mortar, and the step 2 is executed after gathering operation is finished; 4, a slave arm of the double-arm robot is controlled to fix the mortar, a main arm conducts grinding operation on the crystals in the mortar, and the step 2 is executed after grinding operation is finished; and 5, the double-arm robot is controlled to stop the grinding task. According to the method, the crystal grinding efficiency and accuracy can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of controlling robots to conduct crystal material experiments, and particularly to a method for controlling a dual-arm robot to automatically grind crystal materials. Background Art

[0002] In the field of grinding chemical colored crystals, traditional manual operations face challenges such as low efficiency, limited precision, and potential safety risks. In addition, although automated equipment such as ball mills is used to replace manual labor and is applicable to large-scale grinding operations, they often cause a large proportion of material loss when dealing with small batches of colored crystals, and there are limitations in the adaptive conversion between different colored crystals.

[0003] In recent years, some studies have attempted to use multi-degree-of-freedom single-arm robots for grinding operations and use a simple binarization algorithm to detect colored crystals. However, this method has a narrow application range, poor positioning effect, and it is difficult to provide good visual feedback for grinding various types of colored crystals. The visual processing module also does not consider the problem of color gradual change of colored crystals caused by changes in particle size after grinding; it is unable to plan grinding actions and gather colored crystals according to the distribution of chemical colored crystals to achieve subsequent fine grinding operation techniques; for the grinding operation of the robotic arm, a grinding trajectory is generated based on a mortar model and position control is performed. Although this method, combined with a compliant material at the end, reduces the probability of the robotic arm being protected and stopped due to position errors and the distribution of colored crystals, it cannot achieve fine grinding of colored crystals and still cannot ensure long-term safe and reliable grinding operations of the robotic arm; currently, in chemical experiment scenarios, single-arm robots are mostly used to complete operation tasks, and grinding operations also tend to use multi-functional end grippers to achieve grinding and gathering of colored crystals, rarely considering improving efficiency through the collaborative operation of dual-arm robots.

[0004] In view of this, the present invention is specifically proposed. Summary of the Invention

[0005] The purpose of the present invention is to provide a method, device, and medium for controlling a dual-arm robot to automatically grind crystal materials, which can control the dual-arm robot to perform grinding operations on chemical colored crystals, improve the operation efficiency of grinding experiments, and thus solve the above-mentioned technical problems existing in the prior art.

[0006] The purpose of the present invention is achieved through the following technical solutions: A method for controlling a dual-arm robot to automatically grind crystal materials, which is used for a system composed of a dual-arm robot, a grinding platform, a mortar, a crystal collection shovel, a pestle, a six-axis force sensor, a grinding gripper, a gripping tool, and a camera, and includes: Step 1: Control the dual-arm robot to move to the operation position on the grinding platform, and use the camera set on the dual-arm robot to obtain the pose information of the mortar to locate the position of the mortar. Step 2: Obtain the color distribution image of the crystal to be ground in the mortar through the camera, and determine the subsequent operations according to the color distribution image. If it is determined to stop grinding, execute Step 5; if it is determined to perform the gathering operation, execute Step 3; if it is determined to perform the grinding operation, execute Step 4. Step 3: Control the clamping tool of the slave arm of the dual-arm robot through the gathering instruction to clamp the crystal collection shovel to perform the gathering operation on the crystals in the mortar, and execute Step 2 after completion. Step 4: Control the clamping tool of the slave arm of the dual-arm robot through the grinding instruction to clamp and fix the mortar, and the main arm clamps the grinding pestle through the grinding gripper connected by the six-axis force sensor to perform the grinding operation on the crystals in the mortar. After the grinding pestle rotates a predetermined number of circles, end the grinding operation and execute Step 2. Step 5: Control the dual-arm robot to stop grinding and complete the grinding task.

[0007] A processing device, comprising: At least one memory for storing one or more programs; At least one processor capable of executing the one or more programs stored in the memory. When the one or more programs are executed by the processor, the processor can implement the method of the present invention.

[0008] A readable storage medium stores a computer program, and when the computer program is executed by a processor, the method of the present invention can be implemented.

[0009] Compared with the prior art, the method, device and medium for controlling a dual-arm robot to perform automatic grinding of crystal materials provided by the present invention have the following beneficial effects: In a crystal grinding system composed of a dual-arm robot, a grinding platform, a mortar, a crystal collection shovel, a grinding pestle, a six-axis force sensor, a grinding gripper, a clamping tool and a camera, visual feedback is used for task decision-making, and in combination with force-position hybrid control and dual-arm collaborative planning, the control of the dual-arm robot to perform automatic grinding tasks on different colored crystals is realized. This method has strong environmental adaptability and low dependence on tools, and uses force-position hybrid control and dual-arm collaboration to ensure the stability and safety of the grinding operation, and can efficiently complete the fine grinding task of chemical colored crystals. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0011] Figure 1 It is a flowchart of a method for controlling a dual-arm robot to automatically grind crystal materials provided by an embodiment of the present invention.

[0012] Figure 2 It is a schematic diagram of the system composition for representing the grinding operation of a dual-arm robot provided by an embodiment of the present invention.

[0013] Figure 3 It is a schematic diagram of obtaining a grinding trajectory based on the change of the ZYZ Euler angles of the bottom attitude of a mortar provided by an embodiment of the present invention. Detailed implementation manners

[0014] Next, in combination with the specific content of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described; obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments, which does not constitute a limitation to the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the protection scope of the present invention.

[0015] First, the following explanations will be given to the terms that may be used in this article: The term "and / or" means that either or both of the two can be realized. For example, X and / or Y means that it includes both the case of "X" or "Y" and the three cases of "X and Y".

[0016] Descriptions with semantic meanings such as "comprising", "including", "containing", "having" or other similar ones should be interpreted as non-exclusive inclusion. For example: including a certain technical feature element (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, processes, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or articles, etc.) should be interpreted as not only including the explicitly listed certain technical feature element, but also including other well-known technical feature elements in the art that are not explicitly listed.

[0017] The term "consisting of" means excluding any technical feature elements not expressly listed. If this term is used in a claim, it will make the claim a closed type, so that it does not include technical feature elements other than those expressly listed, except for the related conventional impurities. If this term only appears in a certain clause of the claim, then it only limits the elements expressly listed in that clause, and the elements recorded in other clauses are not excluded from the overall claim.

[0018] Unless otherwise expressly stipulated or limited, terms such as "installed", "connected", "joined", "fixed", etc. shall be understood in a broad sense. For example: it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, and it can be the communication inside two components. For those of ordinary skill in the art, the specific meanings of the above terms in this text can be understood according to specific circumstances.

[0019] When concentrations, temperatures, pressures, dimensions or other parameters are expressed in the form of a numerical range, the numerical range should be understood as specifically disclosing all ranges formed by the pairing of any upper limit value, lower limit value, and preferred value within the numerical range, regardless of whether the range is expressly recorded; for example, if the numerical range "2-8" is recorded, then this numerical range should be interpreted as including ranges such as "2-7", "2-6", "5-7", "3-4 and 6-7", "3-5 and 7", "2 and 5-7", etc. Unless otherwise specified, the numerical ranges recorded in this text include both their end values and all integers and fractions within the numerical range.

[0020] The orientation or positional relationship indicated by terms such as "center", "longitudinal", "transverse", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of description and simplification of the description, rather than expressly or implicitly indicating that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation to this text.

[0021] The solutions provided by the present invention will be described in detail below. The content not described in detail in the embodiments of the present invention belongs to the prior art well-known to those of ordinary skill in the art. For those conditions not specified in the embodiments of the present invention, they are carried out according to the conventional conditions in the art or the conditions recommended by the manufacturer. For the reagents or instruments not indicating the manufacturer in the embodiments of the present invention, they are all conventional products that can be obtained through commercial purchase.

[0022] As Figure 1As shown in the figure, an embodiment of the present invention provides a method for controlling a dual-arm robot to perform automatic grinding of crystal materials, which is a method for using a dual-arm robot to achieve fine grinding experimental tasks of different colored crystals. For a system composed of a dual-arm robot, a grinding platform, a mortar, a crystal collection shovel, a grinding pestle, a six-axis force sensor, a grinding gripper, a gripping tool, and a camera, it includes: Step 1: Control the dual-arm robot to move to the operation position of the grinding platform, and use the camera set on the dual-arm robot to obtain the pose information of the mortar to locate the position of the mortar; Step 2: Obtain the color distribution image of the crystal to be ground in the mortar through the camera, and determine the subsequent operations according to the color distribution image. If it is determined to stop grinding, execute Step 5; if it is determined to perform a gathering operation, execute Step 3; if it is determined to perform a grinding operation, execute Step 4; Step 3: Control the gripping tool of the slave arm of the dual-arm robot through a gathering instruction to grip the crystal collection shovel to perform a gathering operation on the crystals in the mortar, and after completion, execute Step 2; Step 4: Control the gripping tool of the slave arm of the dual-arm robot through a grinding instruction to grip and fix the mortar, and the main arm grips the grinding pestle through the grinding gripper connected by the six-axis force sensor to perform a grinding operation on the crystals in the mortar. After the grinding pestle rotates a predetermined number of circles, end the grinding operation and execute Step 2; Step 5: Control the dual-arm robot to stop grinding and complete the grinding task.

[0023] Preferably, in Step 1 of the above method, the dual-arm robot is controlled to move to the operation position of the grinding platform in the following manner, and the camera set on the dual-arm robot is used to obtain the pose information of the mortar to locate the position of the mortar, including: Step 11: Control the dual-arm robot to reach the operation position of the grinding platform, denoted as the waiting pose, and use the camera to capture a color image and a depth image of the grinding platform including the mortar; Step 12: Perform image processing on the color image to obtain an extraction image for extracting the range information of the mortar; Step 13: Use the minimum enclosing circle method through the extraction image to obtain the coordinates of the bottom of the mortar that determines the range of the mortar in the pixel coordinate system and the pixel radius of the target enclosing circle ; Step 14: According to the depth information of the depth image, combined with the pose information of the waiting pose of the dual-arm robot, obtain the conversion relationship between the pixel coordinate system and the base coordinate system of the main arm of the dual-arm robot , combined with the known conversion relationship between the base coordinate system of the main arm and the world coordinate system of the dual-arm robot , according to the calculation formula , calculate the position of the center point of the bottom of the mortar in the world coordinate system of the dual-arm robot .

[0024] Preferably, in the above step 12, for the collected color image , image processing is performed using four steps of Gaussian filtering, grayscale conversion, Otsu binarization, and opening operation to obtain an extraction image for extracting the mortar range information , specifically: ; Among them, the meanings of the parameters are: is the original image captured by the camera; is the image after Gaussian filtering; is the image after grayscale processing; is the binarized image; is the image after opening operation; , , and are Gaussian filtering processing, grayscale processing, binarization processing, and opening operation processing respectively.

[0025] In the above step 13, an extraction image for directly extracting the mortar range information is obtained from the extraction image processed in step 12 . At this time, the background color of the mortar range in the image is black, and the background color of the grinding platform in the image is white. The minimum enclosing circle is used to enclose the black area in the extraction image . Among many enclosing circles, the target enclosing circle with the largest area is judged as the mortar. Due to the influence of noise points in the image after opening operation, there may be small-scale speckles in the image, but there is a large difference in area from the mortar. The formula for determining the mortar range is: , where is the coordinate of the bottom of the mortar in the pixel coordinate system of the extraction image; is the pixel radius of the target enclosing circle; is for obtaining the function of the minimum enclosing circle of the black area in.

[0026] Preferably, in step 2 of the above method, the color distribution image of the crystal being ground in the mortar is obtained by the camera in the following manner, and subsequent operations are determined according to the color distribution image, including: Step 21, use the camera to take a color image of the grinding platform. In the pixel coordinate system of the color image, with the coordinate of the bottom of the mortar obtained in step 1 as the center and the pixel radius of the target enclosing circle as the radius to draw a circle, and the obtained circle is used as the mortar pixel area; Step 22, within the mortar pixel area, with the coordinate Taking the center as the center and n pixel points as the sampling radius, the determined circle is the sampling area R, and the mean and variance of the RGB values of all pixel points within the sampling area R are obtained; Step 23, within the mortar pixel area, according to the mean and variance of the RGB values of all pixel points in the sampling area R, all pixel points whose pixel coordinates satisfy the predetermined constraints are formed into a pixel point set, and the RGB mean and variance of the pixel points within the pixel point set are recalculated; Step 24, the color image in step 21 is grayscale processed into a grayscale image, the grayscale values of the pixel points within the pixel point set are all set to 1, and the remaining areas are all set to 0 to obtain a color distribution image. The minimum bounding rectangle is used for the color distribution image to obtain the bounding rectangle information of the crystal and the mean and variance of the crystal color; Step 25, determine the subsequent operations according to the bounding rectangle information of the crystal and the mean and variance of the crystal color. If the mean and variance of the crystal color meet the expected color of the crystal after grinding and are within the threshold of this color, it is determined to stop grinding;

[0027] If the condition for stopping grinding is not met, it is judged whether the gathering operation is satisfied. According to the conversion relationship between the pixel coordinate system and the base coordinate system of the main arm of the dual-arm robot and the conversion relationship between the main arm and the world coordinate system, the pose of the center pixel point of the bounding rectangle of the crystal in the base coordinate system is obtained, and the distance between this pose and the pose of the center of the bottom of the mortar in the world coordinate system is calculated and an allowable distance is set If then it is determined to perform the gathering operation; If the conditions for stopping grinding and gathering operation are not met, it is determined to perform the grinding operation.

[0028] In the above step 22, since the whole crystal exists at the bottom of the mortar during the grinding process, a sampling area R is first defined within the mortar pixel area. After defining the sampling area, the mean of the RGB values of the pixel points within the sampling area R is calculated according to the following formula and the variance , where , and are the three primary colors at each pixel coordinate respectively, and the calculation formulas are respectively: ; ; The meanings of the parameters in the above formulas are: |R| is the total number of pixel points in the sampling area.

[0029] In the above step 23, the conditions for judging whether the pixel coordinates satisfy the predetermined constraints are as follows: ; Record the pixel coordinates that satisfy the above constraints Together, they form the pixel point set D.

[0030] In the above step 24, the color image is processed using the same grayscale processing as in step 12 to obtain a grayscale image; in this step, for the color distribution image Use the minimum bounding rectangle to obtain the four pixel vertex coordinates of the minimum bounding rectangle: = , where , according to the above pixel vertex coordinates, calculate the center position of the minimum bounding rectangle , and the calculation formula is: ; From step 20 to step 24, the obtained information includes the pixel coordinates of the center of the mortar bottom , the crystal bounding rectangle information 、 and the crystal color mean and variance ; In the above step 25, according to the information obtained in step 24, make a task decision to determine the subsequent operations: If the crystal color mean and variance meet the expected color of the crystal after grinding and are within the threshold of this color, then stop grinding, and the judgment condition is: ; and are the target value and allowable deviation of the expected color respectively, R represents red, B represents blue, and G represents green; If the grinding end condition is not met, determine whether to perform the aggregation operation. According to the conversion relationship between the pixel coordinate system and the base coordinate system of the main arm of the dual-arm robot and the conversion relationship between the main arm and the world coordinate system , obtain the pose of the center pixel point of the crystal bounding rectangle in the base coordinate system, and the calculation formula is: , calculate and the pose of the center of the mortar bottom distance , and set the allowable distance , if , it means that the crystal distribution range is large, and the aggregation operation of aggregating the crystals needs to be performed, and the subsequent step 3 is executed; If the above conditions are not met, continue the grinding operation, and the subsequent step 4 is executed.

[0031] Preferably, in step 3 of the above method, the clamping tool of the slave arm of the dual-arm robot is controlled by a gathering instruction to clamp the crystal collection shovel and perform a gathering operation on the crystals in the mortar, including: Step 31, control the master arm of the dual-arm robot to move to a non-collision area, release the fixation of the mortar by the slave arm and clamp the grinding collection shovel, and operate according to the bounding rectangle information of the crystal. Convert the bounding rectangle information of the crystal into coordinates in the world coordinate system of the dual-arm robot, calculate the centers of two adjacent points on each rectangular side of the bounding rectangle information of the crystal, and calculate the pose of the center point between the center of the bounding rectangle of the crystal and the center pixel point of the crystal bounding rectangle in the base coordinate system The center point distance between , where 、 、 or ; Step 32, set the distance from the center of the bottom of the mortar. If it is determined that the center point distance of a certain rectangular side , then control the clamping tool of the slave arm of the dual-arm robot to clamp the grinding collection shovel to perform a gathering operation on this rectangular side, otherwise do not perform a gathering operation on this rectangular side; Step 33, when it is determined that all rectangular sides of the bounding rectangle of the crystal do not need to be gathered, determine to end the gathering operation.

[0032] Preferably, in step 4 of the above method, the clamping tool of the slave arm of the dual-arm robot is controlled by a grinding instruction to clamp and fix the mortar, and the grinding gripper of the master arm clamps the pestle to perform a grinding operation on the crystals in the mortar, including: Step 41, generate a grinding reference trajectory according to the mathematical model of the mortar, and the pestle in the grinding reference trajectory is perpendicular to the inner surface of the mortar; Step 42, control the slave arm of the dual-arm robot to fix the mortar, and control the master arm of the dual-arm robot to clamp the pestle based on the grinding reference trajectory, and turn on the end force-position hybrid controller to perform a grinding operation on the crystals in the mortar.

[0033] Preferably, in step 4 of the above method, the color of the crystals in the mortar is monitored in the following manner. If the crystal color meets the desired conditions, it is confirmed that the grinding task is completed, including: Obtain the mean and variance of the crystal color from the color distribution image of the crystal being ground captured by the camera. If it is determined that the mean and variance of the crystal color meet the expected color of the ground crystal and are within the threshold of this color, then confirm to stop grinding. The judgment condition is: ; where and The target value and the allowable deviation of the expected color of the crystal, respectively; the mean value of the crystal color is and the variance is .

[0034] In the above step 41, according to the pose of the center of the bottom of the mortar and the set allowable distance design the grinding trajectory. The specific operation is as follows: Let the radius of the inner surface of the mortar be (the inner surface of the used mortar is a part of a sphere with a radius of ), the pose of the center of the bottom of the mortar , where X, Y, Z, θ R , θ P and θ Y are the position of the x-axis, the position of the y-axis, the position of the z-axis and the Roll angle, Pitch angle and Yaw angle of the corresponding Euler angles of the center of the bottom of the mortar in the world coordinate system respectively; the pose points in the grinding feasible region of the inner surface of the mortar obtained by transformation from the center pose are , where x, y, z, r, p and y are the position of the x-axis, the position of the y-axis, the position of the z-axis and the Roll angle, Pitch angle and Yaw angle of the corresponding Euler angles of the pose points in the grinding feasible region of the inner surface of the mortar in the world coordinate system respectively, and the two satisfy the following relationship: ; Among them, is the ZYZ Euler transformation function, and generates a grinding reference trajectory in the form of in the grinding feasible region. This reference trajectory shows that the grinding pestle is perpendicular to the inner surface of the mortar. Define each reference trajectory point as .

[0035] In the above step 42, the model for the main arm to hold the grinding pestle to grind the colored crystal is; ; Among them, M, B, and K are the diagonal matrix of inertia coefficients, the diagonal matrix of damping coefficients, and the diagonal matrix of stiffness coefficients respectively, , , are the actual acceleration, actual velocity, and actual position in the position control subspace, and are the actual force and the desired force in the force control subspace.

[0036] See Figure 2, preferably, in the above method, in the system composed of the dual-arm robot 1, the grinding platform 2, the mortar 3, the crystal collection shovel 4, the grinding pestle 5, the six-axis force sensor, the grinding gripper, the gripping tool, and the camera 6, the mortar is fixed to the grinding platform, and the color of the grinding platform is the opposite color of the mortar; preferably, the color of the grinding platform is white and the color of the mortar is black; One robotic arm of the dual-arm robot 1 is the main arm 11, and a camera 6 is arranged on the main arm. The controller of the dual-arm robot is communicatively connected to the camera. The other robotic arm is the slave arm 12, and a gripping tool for fixing the mortar and gripping the crystal collection shovel is installed at the end of the slave arm.

[0037] An embodiment of the present invention further provides a processing device, including: At least one memory for storing one or more programs; At least one processor capable of executing the one or more programs stored in the memory. When the one or more programs are executed by the processor, the processor can implement the above method.

[0038] An embodiment of the present invention further provides a readable storage medium storing a computer program, which can implement the above method when executed by a processor.

[0039] In summary, the method of the embodiment of the present invention combines visual feedback for task decision-making, and cooperates with force-position hybrid control to achieve the dual-arm collaborative planning of the dual-arm robot, realizing the automatic grinding task of different colored crystals. This method has strong environmental adaptability and low dependence on tools, ensuring the stability and safety of crystal grinding operations, and can efficiently complete the fine grinding task of chemical colored crystals.

[0040] In order to more clearly show the technical solutions provided by the present invention and the technical effects produced, the following uses specific embodiments to describe in detail the solutions provided by the embodiments of the present invention.

[0041] Embodiment 1 This embodiment provides a method for controlling a dual-arm robot to perform automatic grinding of crystal materials, which can control the dual-arm robot to perform chemical colored crystal positioning and grinding operations. As Figure 2 shown, the crystal grinding system specifically used in this method includes: A dual-arm robot, a grinding platform, a mortar, a crystal collection shovel, a grinding pestle, a six-axis force sensor, a grinding gripper, a gripping tool, and a camera. The camera uses an RGBD camera; wherein, The grinding platform, mortar, crystal collection spatula, and grinding pestle form a grinding set of tools. Among them, the mortar is fixed on the grinding platform. The grinding platform and the mortar are in contrasting colors. For example, if the grinding platform is white, the mortar is black. The grinding pestle is installed at the end of the robotic arm through a grinding gripper, and the robotic arm is set as the main arm. Another robotic arm is the slave arm, and a clamping tool is installed at its end position to fix the mortar and use the crystal collection spatula; The six-axis force sensor is installed at the end of the robotic arm and connected to the grinding gripper; The camera is set on the robotic arm of the dual-arm robot. Specifically, the camera is installed at the end of the main arm and can capture and collect the color image and depth image of the grinding platform; Both robotic arms of the dual-arm robot are multi-degree-of-freedom operating arms; The dual-arm robot is communicatively connected to the camera. After receiving the initialization instruction, the robotic arm moves to the operating position of the grinding platform, uses the set camera to collect the color image and depth image of the current grinding platform, and locates the pixel range and the center pose of the mortar based on the color image and depth image. The control instruction is the grinding operation process determined in advance according to the actual operation of the grinding experiment. According to the design of each grinding stage, the dual-arm robot of the system is controlled to execute the control instruction. The control instruction includes: initialization instruction and operation instruction. The operation instruction is divided into: grinding instruction, detection instruction, and gathering instruction. The grinding instruction is to control the slave arm of the dual-arm robot to fix the mortar with a clamping tool, and the main arm uses the grinding pestle to grind the crystals in the mortar; the gathering instruction is to control the slave arm of the dual-arm robot to clamp the crystal collection spatula to gather the crystals in the mortar to facilitate the subsequent grinding. At this time, the main arm is in a waiting state; the detection instruction is to control the dual-arm robot to detect and locate the crystals in the mortar through the camera, obtain the distribution range of the current crystal color, and judge whether the next step is grinding, gathering, or stopping. Control the dual-arm robot to plan the trajectory of the current crystal grinding or gathering operation according to the crystal color distribution in the mortar, and execute the corresponding grinding experiment operation of the current grinding platform according to the planned trajectory until the crystal grinding task of the current grinding platform is completed.

[0042] See Figure 1 , the specific steps of this method include: Step 1, control the camera carried by the dual-arm robot to obtain the pose information of the mortar and transform it into the world coordinate system, including the following steps: Step 11, use the control instruction of the dual-arm robot to reach the operating position of the grinding platform, denoted as the waiting pose, and use the camera to capture the color image and depth image of the current grinding platform including the mortar; Step 12, according to the collected color image , image processing is used, and the image processing is divided into four steps: Gaussian filtering, grayscale conversion, Otsu binarization, and opening operation: ; Among them, is the original image captured by the camera; is the image after Gaussian filtering; is the image after grayscale processing; is the image after binarization; is the image after opening operation processing; , , and are Gaussian filtering processing, grayscale processing, binarization processing, and opening operation processing respectively; Step 13, through the image processing in Step 12, an extraction image for directly extracting the range information of the mortar is obtained , at this time, the background color of the range of the mortar in the image is black and the background color of the grinding platform in the image is white. The minimum bounding circle is used to enclose the black area in the extraction image . Among many bounding circles, the target bounding circle with the largest area is judged as the mortar (due to the influence of noise points in the image after the opening operation, there may be small-scale noise spots in the image, and there is a large difference in area from the mortar). The calculation formula for determining the mortar is: , where is the coordinate of the bottom of the mortar in the pixel coordinate system of the extraction image; is the pixel radius of the target bounding circle; is for obtaining the function of the minimum bounding circle of the black area in; Step 14, according to the depth information of the depth map, combined with the end pose information of the waiting pose of the manipulator of the dual-arm robot, the conversion relationship between the pixel coordinate system and the base coordinate system of the main arm and the conversion relationship between the main arm and the world coordinate system are obtained. According to the calculation formula , the center pose of the bottom of the mortar in the world coordinate system of the dual-arm robot is calculated ; Step 2, the color distribution image of the crystal to be ground is obtained through the camera. According to the color distribution image, the subsequent operations are judged, that is, stopping grinding, gathering operation, or grinding operation, including the following steps: Step 21, the end of the manipulator of the dual-arm robot moves to the waiting pose, and the color image of the grinding platform is captured through the camera. In the pixel coordinate system, with the coordinate of the bottom of the mortar in the pixel coordinate system as the center, Make a circle with the radius, and this circle is the position area of the mortar obtained in step 1 in the pixel coordinate system. Define this area as the mortar pixel area; Step 22, define the sampling area. Since the whole crystal exists at the bottom of the mortar during the grinding process, use the coordinates of the bottom of the mortar in the pixel coordinate system within the mortar pixel area as the center of the circle and n pixel points as the sampling radius to make a circle. The area of this circle is the sampling area, defined as R. Obtain the RGB mean value (i.e., the mean value of the RGB values of the pixel points) and variance of the pixel points within the sampling area R, 、 and are the three primary colors under the pixel coordinate respectively. The RGB mean value and variance of the sampling area R are: ; ; where |R| is the total number of pixel points in the sampling area; Step 23, within the mortar pixel area, according to the RGB mean value and variance of the sampling area, judge whether the pixel coordinate meets the constraint. If the pixel coordinate meets the following constraint: ; then record this pixel coordinate. Finally, obtain the set D of all pixel points within the mortar pixel area that meet this constraint, and recalculate the RGB mean value and variance of the pixel points within the set D to obtain the RGB mean value and variance ; Step 24, use the same grayscale processing as in step 12 to obtain a grayscale image from the color image. Set the grayscale values of the pixel points within the set D to 1 and the rest of the area to 0 to obtain a color distribution image , and use the minimum bounding rectangle for the color distribution image to obtain the four pixel vertices of this minimum bounding rectangle: = , where . According to the above four pixel vertex coordinates, calculate the center position of the minimum bounding rectangle. The calculation formula is: ; Through steps 21 to 24, obtain the center pixel position of the mortar, the crystal bounding rectangle information 、 , as well as the crystal color mean value and variance ; Step 25: Make a task decision based on the information obtained above and determine the subsequent operations: If the average value of the crystal color and the variance meet the expected color of the crystal after grinding and are within the threshold of this color, stop grinding. The judgment condition is: ; where and are respectively the target value and the allowable deviation of the expected color; If the grinding end condition is not met, determine whether to perform the gathering operation. According to the conversion relationship between the pixel coordinate system and the base coordinate system of the main arm of the dual-arm robot and the conversion relationship between the main arm and the world coordinate system , obtain the pose of the center pixel point of the crystal bounding rectangle in the base coordinate system. The calculation formula is: , calculate and the pose of the center of the bottom of the mortar The distance , and set the allowable distance If , it means that the crystal distribution range is large and the crystals need to be gathered. Then perform the gathering operation in step 3 next; If the above conditions are not met, continue the grinding operation and perform the grinding operation in step 4 next.

[0043] The above steps 21 - 24 are implemented through the vision processing module of the controller of the dual-arm robot, and step 25 is implemented through the task decision maker of the controller of the dual-arm robot.<>

[0044] Step 3: According to the color distribution of the crystals, the dual-arm robot performs the gathering operation. The specific operation of this step is: Step 31: The main arm of the dual-arm robot moves to the non-collision area, the slave arm releases the fixation of the mortar, picks up the grinding collection shovel, and operates according to the bounding rectangle information of the crystal; convert the corner points of the minimum bounding rectangle of the crystal in the pixel coordinate system to the coordinates in the world coordinate system of the dual-arm robot. The calculation formula is: ; Calculate the center of two adjacent points on the rectangle side , where , , or , calculate and The distance between ; Step 32: Set the distance from the center of the bottom of the mortar If , use the grinding collection shovel to gather the rectangular edge, otherwise the rectangular edge will not be gathered.

[0045] See also Figure 3 , step 4, according to the color distribution of the crystal, the dual-arm robot performs a grinding operation, the specific operation of this step is; Step 41: According to the mathematical model of the mortar, the center position of the bottom of the mortar And set the allowed distance , design the grinding trajectory. The specific operation is: set the inner surface radius of the mortar to be (The inner surface of the mortar used is the radius part of the sphere), the center of the bottom of the mortar , where X, Y, Z, θ R ,θ P and θ Y are the x-axis position, y-axis position, and z-axis position of the center of the mortar bottom in the world coordinate system and the corresponding Euler angles of Roll, Pitch, and Yaw; the posture point within the feasible area of grinding on the inner surface of the mortar obtained by transforming the center posture is , where x, y, z, r, p, and y are the x-axis position, y-axis position, and z-axis position of the posture point in the feasible area of ​​the inner surface grinding of the mortar in the world coordinate system and the corresponding Euler angle Roll angle, Pitch angle, and Yaw angle, respectively. The two satisfy the following relationship: ; in, is the ZYZ Euler transformation function, and in the grinding feasible area The grinding reference trajectory is generated in the form of the grinding pestle being perpendicular to the inner surface of the mortar. Each reference trajectory point is defined as ; In step 42, the slave arm of the dual-arm robot closes its gripper to secure the mortar. The master arm, following the reference trajectory of step 41, activates a force-position hybrid controller that adjusts and controls the joints of the robot arm based on data from the six-dimensional torque sensor, clamps the grinding gripper, and grinds the colored crystal. The model is as follows: ; Among them, M, B, and K are the diagonal matrix of inertia coefficient, damping coefficient, and stiffness coefficient, respectively. 、 、 is the actual acceleration, actual velocity and actual position in the position control subspace, and is the actual force and desired force in the force control subspace.

[0046] Step 5: Control the dual-arm robot to stop grinding and complete the grinding task.

[0047] Those of ordinary skill in the art can understand that all or part of the processes in the above-described embodiment methods can be completed by instructing relevant hardware through a program. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-described method embodiments. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM), etc.

[0048] As described above, only the preferred specific embodiments of the present invention are provided, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by the present invention should be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims. The information disclosed in the background art part of this article is only intended to deepen the understanding of the overall background art of the present invention, and should not be regarded as an admission or any form of implication that this information constitutes the prior art known to those skilled in the art.

Claims

1. A method for controlling a dual-arm robot to perform automatic grinding of crystal materials, characterized in that, A system consisting of a dual-arm robot, a grinding platform, a mortar, a crystal collection spatula, a grinding pestle, a six-axis force sensor, a grinding gripper, a gripping tool, and a camera, comprising: Step 1: Control the dual-arm robot to move to the operation position of the grinding platform, and use the camera set on the dual-arm robot to obtain the pose information of the mortar to locate the position of the mortar; Step 2: Obtain the color distribution image of the crystal to be ground in the mortar through the camera, and determine the subsequent operations according to the color distribution image. If it is determined to stop grinding, execute Step 5; if it is determined to perform the gathering operation, execute Step 3; if it is determined to perform the grinding operation, execute Step 4; Step 3: Control the gripping tool of the slave arm of the dual-arm robot to grip the crystal collection spatula through the gathering instruction to perform the gathering operation on the crystals in the mortar, and execute Step 2 after completion; Step 4: Control the gripping tool of the slave arm of the dual-arm robot to grip and fix the mortar through the grinding instruction, and the master arm grips the grinding pestle connected by the six-axis force sensor to perform the grinding operation on the crystals in the mortar. After the grinding pestle rotates a predetermined number of turns, end the grinding operation and execute Step 2; Step 5: Control the dual-arm robot to stop grinding to complete the grinding task.

2. The method for controlling a dual-arm robot to automatically grind crystal materials according to claim 1, wherein In the above Step 1, control the dual-arm robot to move to the operation position of the grinding platform in the following manner, and use the camera set on the dual-arm robot to obtain the pose information of the mortar to locate the position of the mortar, including: Step 11: Control the dual-arm robot to reach the operation position of the grinding platform, denoted as the waiting pose, and use the camera to capture the color image and depth image of the grinding platform including the mortar; Step 12: Perform image processing on the color image to obtain the extraction image for extracting the range information of the mortar; Step 13: Obtain the coordinates of the bottom of the mortar that determines the mortar range in the pixel coordinate system by using the minimum circumscribed circle method for the extraction of the image and the pixel radius of the target circumscribed circle ; Step 14: Based on the depth information of the depth map and combined with the pose information of the waiting pose of the dual-arm robot, obtain the conversion relationship between the pixel coordinate system and the base coordinate system of the main arm of the dual-arm robot , combined with the known conversion relationship between the base coordinate system of the main arm and the world coordinate system of the dual-arm robot , according to the calculation formula , calculate the position of the center point at the bottom of the mortar in the world coordinate system of the dual-arm robot .

3. The method for controlling an articulated robot to automatically grind crystal materials according to claim 1 or 2, characterized in that In the above Step 2, obtain the color distribution image of the crystal to be ground in the mortar through the camera in the following manner, and determine the subsequent operations according to the color distribution image, including: Step 21: Take a color image of the grinding platform. In the pixel coordinate system of the color image, use the coordinates of the bottom of the mortar obtained in Step 1 as the center, and the pixel radius of the target circumscribed circle as the radius to draw a circle, and the obtained circle is used as the mortar pixel area; Step 22, within the mortar pixel region, using the coordinates of the bottom of the mortar in the pixel coordinate system as the center, a circle with a sampling radius of n pixel points is defined as the sampling region R, and the mean and variance of the RGB values of all pixel points within the sampling region R are obtained; Step 23: In the pixel area of the mortar, according to the mean and variance of all pixel points RGB in the sampling area R, form a set of pixel points whose pixel coordinates satisfy the predetermined constraints, and recalculate the mean and variance of the pixel points in the set of pixel points; Step 24: Gray-scale the color image in Step 21 into a grayscale image, set the gray values of the pixel points in the set of pixel points to 1, and the rest of the areas to 0 to obtain the color distribution image. Use the minimum bounding rectangle for the color distribution image to obtain the bounding rectangle information of the crystal and the mean and variance of the crystal color; Step 25: Determine the subsequent operations according to the bounding rectangle information of the crystal and the mean and variance of the crystal color. If the mean and variance of the crystal color meet the expected color of the crystal after grinding and are within the threshold of this color, it is determined to stop grinding; If the grinding stop condition is not met, it is determined whether the gathering operation is satisfied. According to the conversion relationship between the pixel coordinate system and the base coordinate system of the main arm of the dual-arm robot and the conversion relationship between the main arm and the world coordinate system, the pose of the center pixel point of the bounding rectangle of the crystal in the base coordinate system is obtained, and the distance between this pose and the pose of the center of the bottom of the mortar in the world coordinate system is calculated , and an allowable distance is set , if , it is determined that the gathering operation is to be performed; If it does not meet the conditions for stopping grinding and gathering operation, it is determined to perform the grinding operation.

4. The method for controlling a dual-arm robot to automatically grind crystal materials according to claim 3, characterized in that, In the above Step 25, the judgment condition for determining to stop grinding is: ； Among them, and are the target value and the allowable deviation of the expected color of the crystal, respectively; the mean value of the crystal color is and the variance is .

5. The method for controlling a dual-arm robot to automatically grind crystal materials according to claim 1 or 2, characterized in that In the above Step 3, control the gripping tool of the slave arm of the dual-arm robot to grip the crystal collection spatula through the gathering instruction to perform the gathering operation on the crystals in the mortar in the following manner, including: Step 31, control the main arm of the dual-arm robot to move to a non-collision area, release the fixing of the mortar by the slave arm and grip the grinding and collecting shovel, operate according to the surrounding rectangle information of the crystal, convert the surrounding rectangle information of the crystal into coordinates in the world coordinate system of the dual-arm robot, calculate the centers of two adjacent points on each rectangular side in the surrounding rectangle information of the crystal, and calculate the pose of the center point between the center of the crystal surrounding rectangle and the pixel point of the center of the crystal surrounding rectangle in the base coordinate system The center point distance between , where , , or ; Step 32, set the maximum dispersion distance of the crystal , if it is determined that the distance between the center point of a certain rectangular side and the center of the bottom of the mortar , then the clamping tool of the slave arm of the dual-arm robot is controlled by the gathering instruction to clamp the grinding and collecting shovel to perform a gathering operation on this rectangular side, otherwise no gathering operation is performed on this rectangular side; Step 33: When it is determined that all the rectangular sides of the bounding rectangle of the crystal do not require the gathering operation, determine to end the gathering operation.

6. The method for controlling a dual-arm robot to perform automatic grinding of crystal materials according to claim 1 or 2, characterized in that, In step 4, the holding tool of the slave arm of the dual-arm robot is controlled by a grinding instruction to hold and fix the mortar, and the grinding gripper of the master arm holds the pestle to grind the crystal in the mortar in the following manner, including: Step 41, generating a grinding reference trajectory according to the mathematical model of the mortar, wherein the pestle is perpendicular to the inner surface of the mortar in the grinding reference trajectory; Step 42, controlling the slave arm of the dual-arm robot to fix the mortar, and controlling the master arm of the dual-arm robot to hold the pestle by turning on the end force-position hybrid controller based on the grinding reference trajectory to grind the crystal in the mortar.

7. The method for controlling a dual-arm robot to automatically grind crystal materials according to claim 1 or 2, characterized in that, In a system composed of a dual-arm robot, a grinding platform, a mortar, a crystal collection shovel, a pestle, a six-axis torque sensor, a grinding gripper, a holding tool and a camera, the mortar is placed on the grinding platform, and the color of the grinding platform and the color of the mortar are contrast colors; One robotic arm of the dual-arm robot is the master arm, and a camera is arranged on the master arm. The controller of the dual-arm robot is communicatively connected to the camera. The other robotic arm is the slave arm, and a holding tool for fixing the mortar and holding the crystal collection shovel is installed at the end of the slave arm.

8. The method for controlling a dual-arm robot to automatically grind crystal materials according to claim 7, characterized in that, The color of the grinding platform is white, and the color of the mortar is black.

9. A processing device, characterized in that, Including: At least one memory for storing one or more programs; At least one processor capable of executing the one or more programs stored in the memory. When the one or more programs are executed by the processor, the processor can implement the method according to any one of claims 1-8.

10. A readable storage medium stores a computer program, characterized in that, When the computer program is executed by the processor, it can implement the method according to any one of claims 1-8.

Images