Frame cutting method, electronic equipment and storage medium

Abstract

The invention is suitable for the technical field of machine vision, and provides a frame cutting method, electronic equipment and a storage medium, the frame cutting method is applied to a robot, the tail end of the robot is provided with a 2D camera, a 3D line scanning camera and a cutting device.The method comprises the steps that a 2D image of a to-be-cut piece collected by the 2D camera, a first conversion relation and a nesting drawing are obtained; determining a first frame point coordinate set; based on the first frame point coordinate set and the second conversion relation, determining a shooting point coordinate set of the 3D line scanning camera, and acquiring a 3D line scanning image set; a cutting point coordinate set is determined based on the 3D line scanning image set, the trepanning drawing and data recording information, and the data recording information is used for indicating the corresponding relation between the coordinates of the feature points in a pixel coordinate system of the 3D line scanning camera and the coordinates of the feature points in a robot base coordinate system; and on the basis of the cutting point coordinate set, the cutting device is controlled to perform frame cutting on the to-be-cut piece, and the frame cutting precision is improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of machine vision, and particularly relates to a frame cutting method, an electronic device and a storage medium. BACKGROUND

[0002] In the manufacturing and processing industry, for example, a steel plate, after cutting the steel plate, the frame often adheres to the part, causing the part to be difficult to automatically sort, seriously affecting the efficiency of part sorting. Therefore, it is an urgent need to cut off the frame.

[0003] Frame cutting is usually directly cutting off the frame during part cutting, but when the frame is not cut or the part still adheres, the frame needs to be further cut off. At present, 3D vision is usually used to identify the frame, and then the coordinates are converted to the robot base coordinate system for cutting. However, this method depends on the calibration results between multiple cameras and robots, and each calibration has an error, and multiple calibration processes further increase the cutting error, resulting in low accuracy of frame cutting.

[0004] Therefore, how to improve the accuracy of frame cutting has become a problem to be solved. SUMMARY

[0005] The embodiments of the present application provide a frame cutting method, an electronic device and a storage medium, which aims to improve the accuracy of frame cutting.

[0006] In a first aspect, the embodiments of the present application provide a frame cutting method applied to a robot, wherein the end of the robot is provided with a 2D camera, a 3D line scanning camera and a cutting device, and the method comprises the following steps: determining a first frame point coordinate set based on a 2D image of a part to be cut collected by the 2D camera, a first conversion relationship and a nesting drawing of the part to be cut, wherein the first conversion relationship is a conversion relationship between a pixel coordinate system of the 2D camera and a robot base coordinate system; determining a shooting point coordinate set of the 3D line scanning camera based on the first frame point coordinate set and a second conversion relationship, wherein the second conversion relationship is a conversion relationship between a camera coordinate system of the 3D line scanning camera and a robot end coordinate system; collecting a 3D line scanning image set based on the 3D line scanning camera and the shooting point coordinate set; determining a cutting point coordinate set based on the 3D line scanning image set, the nesting drawing and data recording information, wherein the data recording information is used to indicate the corresponding relationship between the coordinates of a feature point in a pixel coordinate system of the 3D line scanning camera and the coordinates in the robot base coordinate system, and the feature point is any point in the part to be cut; Based on the cutting point coordinate set, the cutting device is controlled to cut the frame of the to-be-cut part.

[0007] In a possible implementation, the first frame point coordinate set is determined based on the 2D image of the to-be-cut part collected by the 2D camera, the first conversion relationship, and the nesting drawing of the to-be-cut part, and includes the following steps. Based on the first conversion relationship, a target corner point and a target edge point in the 2D image are mapped to the robot base coordinate system respectively to obtain a first coordinate and a second coordinate, the target corner point being any corner point of the to-be-cut part, and the target edge point being any edge point belonging to the same side as the target corner point. The first frame point coordinate set is determined based on the first coordinate, the second coordinate, and the nesting drawing.

[0008] In a possible implementation, the first frame point coordinate set is determined based on the first coordinate, the second coordinate, and the nesting drawing, and includes the following steps. Based on the first coordinate and the second coordinate, a unit vector in the X direction and a unit vector in the Y direction are determined. Based on the coordinates of the frame points in the nesting drawing and the first coordinate, a first coordinate change set is determined, the first coordinate change set including coordinate changes of the frame points in the nesting drawing relative to the target corner point. Based on the first coordinate change set, the first coordinate, the unit vector in the X direction, and the unit vector in the Y direction, all frame points in the nesting drawing are mapped to the robot base coordinate system to obtain the first frame point coordinate set.

[0009] In a possible implementation, the method for determining the data recording information includes the following steps. The feature point is selected in the to-be-cut part, and a coordinate of the feature point in the robot base coordinate system is determined. Based on the 3D line-scan camera and the feature point, a target image of the to-be-cut part is collected, and based on the target image, a coordinate of the feature point in a pixel coordinate system of the 3D line-scan camera is determined. Based on the coordinates of the feature point in the pixel coordinate system of the 3D line-scan camera and in the robot base coordinate system, the data recording information is determined.

[0010] In a possible implementation, the cutting point coordinate set is determined based on the 3D line-scan image set, the nesting drawing, and the data recording information, and includes the following steps. performing frame recognition based on the set of 3D line scan images and the sleeve drawing to determine a second frame point coordinate set, the second frame point coordinate set including coordinates of each frame point in the to-be-cut part in a pixel coordinate system of the 3D line scan camera; determine the cutting point coordinate set based on the second frame point coordinate set and the data recording information.

[0011] In a possible implementation, the performing frame recognition based on the set of 3D line scan images and the sleeve drawing to determine a second frame point coordinate set includes: determine a gap measurement distance based on a gray scale image corresponding to each 3D line scan image in the set of 3D line scan images; determine a gap theoretical distance and an outer frame theoretical distance based on the sleeve drawing; determine a target gap based on the gap measurement distance and the gap theoretical distance, and determine an outer frame measurement distance as a distance between any two target gaps; determine a target outer frame based on the outer frame measurement distance and the outer frame theoretical distance; determine the second frame point coordinate set based on the set of 3D line scan images and the target outer frame.

[0012] In a possible implementation, the performing frame recognition based on the set of 3D line scan images and the sleeve drawing to determine a second frame point coordinate set further includes: determine inner frame theoretical length information based on the sleeve drawing; perform plane extraction on each 3D line scan image in the set of 3D line scan images to determine inner frame measurement length information; determine a target inner frame based on the inner frame theoretical length information and the inner frame measurement length information; determine the second frame point coordinate set based on the set of 3D line scan images and the target inner frame.

[0013] In a possible implementation, the determining the cutting point coordinate set based on the second frame point coordinate set and the data recording information includes: calculate a coordinate change amount between coordinates of each frame point in the second frame point coordinate set and coordinates of the feature point in the data recording information in the pixel coordinate system of the 3D line scan camera to obtain a second coordinate change amount set; determine the cutting point coordinate set based on the second coordinate change amount set and coordinates of the feature point in the data recording information in the robot base coordinate system.

[0014] In a second aspect, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method according to the first aspect or any one of the implementation manners thereof when executing the computer program.

[0015] In a third aspect, a computer readable storage medium is provided, which stores a computer program, and the computer program is executable by a processor to implement the method according to the first aspect or any one of the implementation manners thereof.

[0016] In a fourth aspect, a computer program product is provided, which includes a computer program, and the computer program is executable by a processor to implement the steps of the method according to the first aspect or any one of the implementation manners thereof.

[0017] Compared with the prior art, the embodiments of the present application have the beneficial effects that: based on the 2D image of the to-be-cut part, the first conversion relationship between the pixel coordinate system of the 2D camera and the robot base coordinate system, and the nesting drawing, the first edge frame point coordinate set is determined; based on the first edge frame point coordinate set and the second conversion relationship between the robot end coordinate system and the robot base coordinate system, the shooting point coordinate set of the 3D line scanning camera is determined; based on the shooting point coordinate set, the 3D line scanning camera is guided to shoot to obtain the 3D line scanning image set; based on the 3D line scanning image set, the nesting drawing, and the data recording information, the cutting point coordinate set is determined; and based on the cutting point coordinate set, the cutting device is controlled to perform the edge frame cutting action on the to-be-cut part. The corresponding relationship between the coordinates of any point in the to-be-cut part in the pixel coordinate system of the 3D line scanning camera and the coordinates in the robot base coordinate system is obtained from the data recording information, the actual change amount of the cutting point in the robot base coordinate system corresponding to each edge frame point and the change amount of the point on the actually collected 3D line scanning image are obtained, the conversion between the pixel coordinate system of the 3D line scanning camera and the robot base coordinate system is realized, compared with obtaining the conversion relationship between the two coordinate systems through calibration and then determining the coordinates of the cutting point based on the conversion relationship, the calibration scheme is reduced, thereby avoiding the calibration error caused by calibration, and the accuracy of edge frame cutting is improved.

[0018] It can be understood that the electronic device, the computer readable storage medium, and the computer program product provided by the embodiments of the present application have the same beneficial effects as the above-mentioned edge frame cutting method, which will not be described here. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only some of the embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0020] Figure 1 A flowchart of a frame cutting method provided by an embodiment of the present application; Figure 2 A schematic diagram of a steel plate nesting drawing provided by an embodiment of the present application; Figure 3 A schematic diagram of an outer frame provided by an embodiment of the present application; Figure 4 A schematic diagram of an inner frame provided by an embodiment of the present application; Figure 5 A flowchart of another frame cutting method provided by an embodiment of the present application. DETAILED DESCRIPTION

[0021] In the following description, for the purpose of explanation and not limitation, specific details are set forth, such as particular system configurations, techniques, etc., in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

[0022] It should be understood that the term "comprising" as used in the specification and the appended claims indicates the presence of the recited features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0023] It should also be understood that the term "and / or" as used herein refers to any one of the associated listed items, combinations of one or more of the associated listed items, and all possible combinations thereof.

[0024] As used in the specification and the appended claims, the term "if' can be interpreted as meaning "when" or "once" or "in response to a determination" or "in response to a detection" depending on the context. Similarly, the phrase "if determined" or "if detected [the described condition or event]" can be interpreted to mean "once determined" or "in response to a determination" or "once detected [the described condition or event]" or "in response to a detection [the described condition or event]" depending on the context.

[0025] In addition, in the description of the present application and the appended claims, the terms "first", "second", "third", etc. are used only to distinguish descriptions, and cannot be understood as indicating or implying relative importance.

[0026] In the present application, the reference "one embodiment" or "some embodiments" means that the specific features, structures or characteristics described in connection with the embodiment are included in one or more embodiments of the present application. Therefore, the statements "in one embodiment", "in some embodiments", "in other some embodiments", "in further some embodiments" and the like appearing in different places in the specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized. The terms "include", "contain", "have" and their variants mean "include but not limited to", unless otherwise specifically emphasized.

[0027] In order to facilitate understanding, the technical solutions of the present application will be described in detail below in conjunction with the drawings.

[0028] Figure 1 A flowchart of a bezel cutting method provided by an embodiment of the present application is shown, only the part related to the present embodiment is shown for the purpose of illustration, the method provided by the present embodiment is applied to a robot, the end of the robot is provided with a 2D camera, a 3D line scanning camera and a cutting device, and the method comprises the following steps: S1, determining a first bezel point coordinate set based on a 2D image of a to-be-cut part collected by the 2D camera, a first conversion relationship and a nesting drawing of the to-be-cut part, the first conversion relationship being a conversion relationship between a pixel coordinate system of the 2D camera and a robot base coordinate system.

[0029] Specifically, the first bezel point coordinate set comprises the coordinates of each bezel point in the to-be-cut part in the robot base coordinate system. The bezel points on the to-be-cut part are identified using the 2D image, and the coordinates of each point in the robot base coordinate system are obtained, so that the 3D line scanning camera can be guided to the corresponding shooting point through the external parameters of the 3D line scanning camera and the robot, thereby ensuring that the bezel points are within the field of view of the 3D line scanning camera.

[0030] Optionally, before step S1, a high-precision circular coding calibration plate (Circular Control Points with Arithmetic and Geometric Coding, CCTAG) is used to calibrate the pixel coordinate system of the 2D camera and the robot base coordinate system, determine the first conversion relationship, and ensure that any point on the 2D image has a corresponding coordinate in the robot base coordinate system. The calibration process can include: calculating the position of each circle center in the CCTAG plate on the 2D image and the position of each circle center in the actual robot base coordinate system, and using at least two sets of corresponding points to solve the corresponding relationship between the 2D image and a plane of the robot, which is the first conversion relationship.

[0031] As an example, as shown in the steel plate nesting drawing, the red line on the nesting drawing is the frame to be cut, and the position of the red line in the robot base coordinate system can be known through the 2D image and the first conversion relationship, and the first frame point coordinate set is obtained. Figure 2

[0032] Illustratively, the workpiece to be cut includes but is not limited to steel plate, stainless steel plate, wood plate, plastic plate, glass plate and stone plate.

[0033] In one possible implementation, step S1 can include the following steps: S11, based on the first conversion relationship, mapping the target corner point and the target edge point in the 2D image to the robot base coordinate system respectively to obtain the first coordinate and the second coordinate, the target corner point being any corner point of the workpiece to be cut, and the target edge point being any edge point belonging to the same edge as the target corner point.

[0034] Preferably, the target corner point is the corner point at the top left corner of the workpiece to be cut, and the target edge point is any point on the upper edge of the workpiece to be cut except the corner point.

[0035] As an example, a 2D image of the workpiece to be cut is captured using a 2D camera, which includes at least one corner point and one edge of the workpiece to be cut.

[0036] As another example, a 2D image of the workpiece to be cut is captured using a 2D camera at the head and tail of the workpiece to be cut respectively, the 2D image captured at the head including the target corner point, and the 2D image captured at the tail including the target edge point.

[0037] In a specific implementation, the pixel coordinates of the target corner point and the target edge point in the 2D image are mapped to the robot base coordinate system through the first conversion relationship to obtain the first coordinate and the second coordinate.

[0038] S12, based on the first coordinate, the second coordinate and the nesting drawing, determining the first frame point coordinate set.​

[0039] In a possible implementation, step S12 can optionally include: S121, determining an X-direction unit vector and a Y-direction unit vector based on the first coordinates and the second coordinates.

[0040] As an example, the X-direction unit vector is , and the Y-direction unit vector is .

[0041] S122, determining a first coordinate change set based on the coordinates of the frame points in the nesting drawing and the first coordinates, the first coordinate change set including coordinate changes of the frame points in the nesting drawing relative to the target corner point.

[0042] As an example, the coordinates of the target corner point in the nesting drawing are denoted as (0, 0), the nesting drawing and the 2D image are matched through the target corner point, after the matching is completed, the (0, 0) point in the nesting drawing corresponds to the first coordinates on the 2D image, since the coordinates of the frame points in the nesting drawing are known, the coordinate changes of the frame points relative to the target corner point in the nesting drawing are known, and the coordinate changes of the frame points relative to the target corner point in the nesting drawing are determined as the first coordinate change set.

[0043] As an example, the coordinate change of a frame point in the nesting drawing relative to the target corner point is wherein, is the coordinate of the frame point in the nesting drawing, is the coordinate of the target corner point in the nesting drawing, denotes the coordinate change in the X-direction, denotes the coordinate change in the Y-direction.

[0044] S123, mapping all the frame points in the nesting drawing to the robot base coordinate system based on the first coordinate change set, the first coordinates, the X-direction unit vector, and the Y-direction unit vector, to obtain a first frame point coordinate set.

[0045] Specifically, according to the first coordinate change set, the first coordinates, and the constructed X-direction unit vector and Y-direction unit vector, all the feature points (such as screw holes, notches, and welding lines) on the frame can be mapped from the nesting drawing coordinate system to the robot base coordinate system through vector translation.

[0046] As an example, since the nesting drawing is 1 pixel representing 1 millimeter, the coordinates of any frame point in the robot base coordinate system are . ​

[0047] S2, determine a set of shooting point coordinates of the 3D line-scan camera based on the first set of frame point coordinates and a second conversion relationship, the second conversion relationship being a conversion relationship between a camera coordinate system of the 3D line-scan camera and an end coordinate system of the robot.

[0048] Specifically, after the first set of frame point coordinates is calculated, since the 3D line-scan camera is fixedly installed on the end of the robot, there is a rigid transformation relationship between the end of the robot and the 3D line-scan camera, and thus only the conversion relationship between the end coordinate system of the robot and the camera coordinate system of the 3D line-scan camera (i.e., the second conversion relationship) needs to be known, so as to determine the set of shooting point coordinates of the 3D line-scan camera, including the coordinates of each shooting point of the 3D line-scan camera in the base coordinate system of the robot.

[0049] As an example, the second conversion relationship can be obtained through 3D line-scan ball calibration, i.e., eye-in-hand / eye-to-world calibration.

[0050] Exemplarily, the ground rail of the robot is taken as a Y axis of a truss, the X of the robot body is taken as an X axis, and the ground rail robot is taken as a truss. The position of the frame point in the truss coordinate system is , and the transformation matrix (rigid transformation) of the 3D line-scan camera (sensor) relative to the truss coordinate system is , wherein is a rotation matrix, representing a pose difference; is a translation vector, representing the origin position of the 3D line-scan camera in the truss coordinate system; and is usually obtained through eye-in-hand / eye-to-world calibration.

[0051] S3, based on the 3D line-scan camera and the set of shooting point coordinates, a set of 3D line-scan images is collected.

[0052] Preferably, after the set of shooting point coordinates is determined, the set of shooting point coordinates is returned to the software and the ID thereof is recorded, the software sorts the positions of the shooting points (for example, from left to right, from top to bottom) and then guides the 3D line-scan camera to each corresponding position for shooting, so as to obtain the set of 3D line-scan images.

[0053] S4, based on the set of 3D line-scan images, the nesting drawing, and data recording information, a set of cutting point coordinates is determined, the data recording information being used to indicate the corresponding relationship between the coordinates of a feature point in a pixel coordinate system of the 3D line-scan camera and the coordinates of the feature point in the base coordinate system of the robot, the feature point being any point in the part to be cut.

[0054] In a possible implementation manner, the method for determining the data recording information comprises: selecting a feature point in the part to be cut and determining the coordinates of the feature point in the base coordinate system of the robot. Based on the 3D line-scan camera and the feature points, a target image of the part to be cut is collected, and based on the target image, coordinates of the feature points in a pixel coordinate system of the 3D line-scan camera are determined. Based on the coordinates of the feature points in the pixel coordinate system of the 3D line-scan camera and in a robot base coordinate system, data recording information is determined.

[0055] Preferably, points with distinctive features are selected as the feature points, such as a sharp point or a corner point of a rectangular object, etc.

[0056] As an example, data recording is performed using the 3D line-scan camera. First, the 3D line-scan camera is used to take a photo of the part to be cut and record the photo coordinates, obtaining a target image. Then, feature points are determined in the target image, and the coordinates of the feature points in a pixel coordinate system of the 3D line-scan camera are recorded. The cutting device is actually directed to the feature points, and the coordinates of the feature points in a robot base coordinate system are recorded. Based on the coordinates of the feature points in the pixel coordinate system of the 3D line-scan camera and in the robot base coordinate system, data recording information is determined.

[0057] In a possible implementation, step S4 can optionally include the following steps: S41, based on the 3D line-scan image set and the nesting drawing, edge frame recognition is performed to determine a second edge frame point coordinate set, the second edge frame point coordinate set including coordinates of each edge frame point in the part to be cut in a pixel coordinate system of the 3D line-scan camera.

[0058] Optionally, the edge frame recognition is divided into inner edge frame recognition and outer edge frame recognition, because the inner edge frame and the outer edge frame present different features on the 3D line-scan image. The inner edge frame and the outer edge frame can be recognized by the coordinates of the edge frame points on the nesting drawing. If the start point or the end point is on the nesting drawing Figure Four line, it is an outer edge frame, and vice versa.

[0059] As an example, step S41 can optionally include: Based on the gray-scale images corresponding to each 3D line-scan image in the 3D line-scan image set, a gap measurement distance is determined. Based on the nesting drawing, a gap theoretical distance and an outer edge frame theoretical distance are determined. Based on the gap measurement distance and the gap theoretical distance, target gaps are determined, and the distance between any two target gaps is determined as an outer edge frame measurement distance. Based on the outer edge frame measurement distance and the outer edge frame theoretical distance, target outer edge frames are determined. Based on the 3D line-scan image set and the target outer edge frames, the second edge frame point coordinate set is determined.

[0060] Optionally, as Figure 3As shown, the outer frame is identified using the grayscale image of the 3D line scan image. Gaps appear black in the grayscale image. The width of the black gap is obtained by statistically analyzing the grayscale values, which gives the gap measurement distance. According to the nesting diagram, the theoretical gap distance is about 0.5-2mm. Gaps whose measured distance is within the theoretical gap distance range are identified as target gaps. The distance between gaps is the distance of the frame. The distance between any two target gaps is then identified as the outer frame measurement distance. The theoretical outer frame distance can be obtained from the nesting diagram. The frame whose measured distance is within 5% of the theoretical outer frame distance is identified as the target outer frame. The pixel coordinates of the left and right endpoints of each target outer frame in the 3D line scan image are identified as the second frame point coordinate set. The left and right endpoints of the frame can be selected as the two corresponding endpoints of the frame.

[0061] As another example, step S41 may optionally include: Based on the nesting diagram, determine the theoretical length of the inner border. Planar extraction is performed on each 3D line scan image in the 3D line scan image set to determine the inner border measurement length information; The target inner border is determined based on the theoretical length information and the measured length information of the inner border. Based on the 3D line scan image set and the inner bounding box of the target, the coordinate set of the second bounding box points is determined.

[0062] Optional, such as Figure 4 As shown, the light blue portion in the middle represents the inner border. Inner border identification primarily relies on planar extraction. Since the cut length information of the inner border's nesting diagram is highly reliable, planar extraction provides excellent identification. The theoretical length of the inner border is determined from the nesting diagram. After extracting the planes using the PEAC planar extraction algorithm, the lengths of all planes are statistically analyzed to obtain the measured length of the inner border. If the error between the theoretical length and the corresponding measured length of the inner border is within 5%, and the midpoint of the plane is exactly in the middle of the field of view, then that plane is definitely the target inner border. The pixel coordinates of the left and right endpoints of each target inner border in the 3D line scan image are used to determine the second set of border point coordinates.

[0063] S42, based on the second border point coordinate set and data recording information, determine the cutting point coordinate set.

[0064] In one possible implementation, step S42 may include: Calculate the coordinate change between the coordinates of each border point in the second border point coordinate set and the coordinates of the feature points in the data recording information in the pixel coordinate system of the 3D line scan camera, and obtain the second coordinate change set. Based on the set of second coordinate changes and the coordinates of feature points in the data recording information in the robot's base coordinate system, the set of cutting point coordinates is determined.

[0065] Specifically, after identifying the bounding box, the pixel coordinates of both ends of each bounding box on the 3D line scan image can be determined, resulting in a second set of bounding box point coordinates. Then, based on the coordinates of the feature points in the pixel coordinate system of the 3D line scan camera in the data recording information, the change in pixel coordinates is calculated, resulting in a second set of coordinate change amounts. Finally, by adding the coordinates of the feature points in the robot base coordinate system to the coordinates of each coordinate change amount in the second set of coordinate change amounts, the set of cutting point coordinates is obtained. The cutting point coordinates are the coordinates of the actual bounding box points in the robot base coordinate system.

[0066] As an example, the coordinates of feature points in the data recording information under the pixel coordinates of the 3D line scan camera are: The image pixel coordinates are Depth is The coordinates of the feature point in the robot's base coordinate system are Since the 3D line scan camera maintains a consistent shooting posture for each image capture, with only the x and y coordinates changing, and the camera remaining relatively stationary with respect to the actual border, any movement of the border will inevitably result in a change in the image captured by the camera. Let's assume the coordinates of any border point in the second border point coordinate set are... Image coordinates Depth is The camera pose remains unchanged, but the position (translation) changes, denoted as... The second coordinate change (with true scale information in camera coordinates) can be expressed as: So, the current cutting point coordinates .

[0067] S5, based on the set of cutting point coordinates, controls the cutting device to cut the border of the workpiece to be cut.

[0068] In practice, the cutting device is guided to the corresponding position to perform the border cutting action based on the coordinates of each cutting point in the set of cutting point coordinates.

[0069] The technical solution provided in this application determines a first set of frame point coordinates based on a 2D image of the workpiece to be cut, a first transformation relationship between the pixel coordinate system of the 2D camera and the robot base coordinate system, and a nesting diagram; determines a set of shooting point coordinates for a 3D line scan camera based on the first set of frame point coordinates and a second transformation relationship between the robot end effector coordinate system and the robot base coordinate system; guides the 3D line scan camera to capture a set of 3D line scan images based on the set of shooting point coordinates; determines a set of cutting point coordinates based on the set of 3D line scan images, the nesting diagram, and data recording information; and controls the cutting device to perform a frame cutting action on the workpiece to be cut based on the set of cutting point coordinates. The correspondence between the coordinates of any point in the workpiece to be cut in the pixel coordinate system of the 3D line scan camera and its coordinates in the robot base coordinate system is obtained from the data recording information. By measuring the coordinate changes of each border point on the actual acquired 3D line scan image and the actual changes of the cutting point in the robot base coordinate system, the conversion between the pixel coordinate system of the 3D line scan camera and the robot base coordinate system is realized. Compared with obtaining the conversion relationship between the two coordinate systems through calibration and then determining the coordinates of the cutting point based on the conversion relationship, the calibration scheme is reduced, thereby avoiding the calibration error caused by calibration and improving the accuracy of border cutting.

[0070] Figure 5 This is a flowchart illustrating another border cutting method provided in an embodiment of this application. Based on the above embodiments, this embodiment further explains and optimizes the technical solution. For ease of explanation, only the parts related to this embodiment are shown. Specifically, as... Figure 5 As shown, the method provided in this embodiment includes: First, camera calibration and data recording information are determined, specifically including 2D camera calibration and 3D line scan ball calibration, which are used to determine the first and second transformation relationships, respectively. Next, the shooting point positions are identified and calculated using the 2D image of the part to be cut acquired by the 2D camera, the first transformation relationship, the second transformation relationship, and the nesting diagram of the part to be cut, to obtain a set of shooting point coordinates. Then, based on the coordinates of each shooting point in the set of shooting point coordinates, the 3D line scan camera is guided to take pictures of the part to be cut and identify the border. It is determined whether the border is successfully identified. If so, the cutting point coordinates are recorded; if not, the 3D line scan camera is guided to the next shooting point to take pictures, until pictures are taken at all shooting points. Based on all the recorded cutting point coordinates, the border is cut.

[0071] In addition, this application embodiment also provides a border cutting system, which may include: The image acquisition module includes a 2D camera and a 3D line scan camera, both of which are mounted on the robot's end effector.

[0072] The data processing module, which can be an industrial computer, is responsible for the calculation of the vision algorithm and the overall scheduling of the software. The grasping control module includes an industrial robot with a ground track and a control module, which is responsible for moving to a designated point; The actuator, which can be a cutting gun, is responsible for performing the edge cutting action.

[0073] In summary, the technical solution provided in this application includes the following core inventive points: 1. Automated cutting is performed through visual recognition, achieving sub-millimeter cutting accuracy without relying on calibration. This application avoids calibration errors by reducing calibration methods, and uses visual guidance to cut based on differences in the actual image, thereby making the theoretical accuracy close to the accuracy of the actual image, and the actual cutting accuracy reaches the sub-millimeter level.

[0074] 2. Robust and stable frame gap recognition: 2D vision guides a 3D line scan camera to the frame gap for recognition. The gap includes both inner and outer frame gaps. Both types of gaps need to be recognized before the cutting machine is guided to cut the gap.

[0075] Based on the above-mentioned core inventive points, the technical solution provided in this application has the following beneficial effects: 1. Improve production efficiency (1) The automated cutting system can complete the cutting task quickly and continuously, significantly improving the output per unit time.

[0076] (2) Reduce manual intervention and save auxiliary time such as job change and positioning.

[0077] 2. Improve cutting precision (1) With the help of vision recognition and numerical control system, automation can achieve millimeter-level or even sub-millimeter-level positioning accuracy, ensuring the accuracy and consistency of the dimensions of the frame cutting.

[0078] (2) Reduce human error and improve the finished product qualification rate.

[0079] 3. Reduce labor costs (1) Reduce reliance on skilled workers and lower training and management costs.

[0080] (2) One operator can manage multiple automatic cutting machines.

[0081] 4. Enhance security Cutting work often involves dangerous operations such as high temperature and high speed. Automated equipment can effectively avoid direct human intervention and improve safety.

[0082] On the other hand, this application also provides a computer storage medium storing executable program code; the executable program code is used to execute any of the above-mentioned border cutting methods.

[0083] On the other hand, this application also provides an electronic device, including a memory and a processor; the memory stores program code that can be executed by the processor; the program code is used to execute any of the above-described border cutting methods.

[0084] For example, the program code may be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the program code in an electronic device.

[0085] The electronic device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The electronic device may include, but is not limited to, processors and memory. Those skilled in the art will understand that the electronic device may also include input / output devices, network access devices, buses, etc.

[0086] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0087] The memory can be an internal storage unit of the electronic device, such as a hard drive or RAM. It can also be an external storage device, such as a plug-in hard drive, SmartMedia Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory can include both internal and external storage units. The memory is used to store the program code and other programs and data required by the electronic device. The memory can also be used to temporarily store data that has been output or will be output.

[0088] The computer storage medium and electronic device described above are created based on the above method. Their technical functions and beneficial effects will not be elaborated here. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0089] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A method for cutting borders, characterized in that, Applied to robots, wherein the end effector of the robot is equipped with a 2D camera, a 3D line scan camera, and a cutting device, the method includes: Based on the 2D image of the part to be cut captured by the 2D camera, the first transformation relationship, and the nesting diagram of the part to be cut, a first set of frame point coordinates is determined. The first transformation relationship is the transformation relationship between the pixel coordinate system of the 2D camera and the robot base coordinate system. Based on the first set of bounding box point coordinates and the second transformation relationship, the set of shooting point coordinates of the 3D line scanning camera is determined. The second transformation relationship is the transformation relationship between the camera coordinate system of the 3D line scanning camera and the robot end effector coordinate system. Based on the 3D line scan camera and the set of shooting point coordinates, a set of 3D line scan images is acquired; Based on the 3D line scan image set, the nesting diagram, and the data recording information, a set of cutting point coordinates is determined. The data recording information is used to indicate the correspondence between the coordinates of the feature points in the pixel coordinate system of the 3D line scan camera and the coordinates in the robot base coordinate system. The feature point is any point in the workpiece to be cut. Based on the set of cutting point coordinates, the cutting device is controlled to cut the border of the workpiece to be cut.

2. The method according to claim 1, characterized in that, The determination of the first set of frame point coordinates based on the 2D image of the part to be cut acquired by the 2D camera, the first transformation relationship, and the nesting diagram of the part to be cut includes: Based on the first transformation relationship, the target corner point and target edge point in the 2D image are respectively mapped to the robot base coordinate system to obtain the first coordinate and the second coordinate. The target corner point is any corner point of the part to be cut, and the target edge point is any edge point that belongs to the same side as the target corner point. Based on the first coordinates, the second coordinates, and the nesting diagram, the set of coordinates of the first border points is determined.

3. The method according to claim 2, characterized in that, The step of determining the set of coordinates of the first border points based on the first coordinates, the second coordinates, and the nesting diagram includes: Based on the first coordinate and the second coordinate, determine the unit vector in the X direction and the unit vector in the Y direction; Based on the coordinates of each border point in the nesting diagram and the first coordinate, a first set of coordinate changes is determined. The first set of coordinate changes includes the coordinate changes of each border point in the nesting diagram relative to the target corner point. Based on the first set of coordinate changes, the first coordinate, the unit vector in the X direction, and the unit vector in the Y direction, all border points in the nesting diagram are mapped to the robot base coordinate system to obtain the first set of border point coordinates.

4. The method according to claim 1, characterized in that, The method for determining the data recording information includes: Select the feature point in the workpiece to be cut, and determine the coordinates of the feature point in the robot's base coordinate system; Based on the 3D line scan camera and the feature points, a target image of the workpiece to be cut is acquired, and based on the target image, the coordinates of the feature points in the pixel coordinate system of the 3D line scan camera are determined; The data recording information is determined based on the coordinates of the feature points in the pixel coordinate system of the 3D line scan camera and the coordinates in the robot base coordinate system.

5. The method according to claim 1, characterized in that, The step of determining the set of cutting point coordinates based on the 3D line scan image set, the nesting diagram, and the data recording information includes: Based on the 3D line scan image set and the nesting diagram, border recognition is performed to determine a second border point coordinate set, which includes the coordinates of each border point in the workpiece to be cut in the pixel coordinate system of the 3D line scan camera. Based on the second set of border point coordinates and the data recording information, the set of cutting point coordinates is determined.

6. The method according to claim 5, characterized in that, The step of identifying the border based on the 3D line scan image set and the nesting diagram to determine the second border point coordinate set includes: Based on the grayscale image corresponding to each 3D line scan image in the 3D line scan image set, the gap measurement distance is determined. Based on the nesting diagram, determine the theoretical gap distance and the theoretical outer frame distance; Based on the gap measurement distance and the gap theoretical distance, the target gap is determined, and the distance between any two target gaps is determined as the outer frame measurement distance; The target outer border is determined based on the measured distance and the theoretical distance of the outer border. Based on the 3D line scan image set and the target outer border, the coordinate set of the second border point is determined.

7. The method according to claim 5, characterized in that, The step of performing border recognition based on the 3D line scan image set and the nesting diagram to determine the second border point coordinate set further includes: Based on the nesting diagram, determine the theoretical length information of the inner frame; Planar extraction is performed on each 3D line scan image in the 3D line scan image set to determine the inner border measurement length information; The target inner border is determined based on the theoretical length information and the measured length information of the inner border. Based on the 3D line scan image set and the target inner border, the coordinate set of the second border point is determined.

8. The method according to claim 5, characterized in that, The determination of the cutting point coordinate set based on the second border point coordinate set and the data recording information includes: Calculate the coordinate change between the coordinates of each border point in the second border point coordinate set and the coordinates of the feature points in the data recording information in the pixel coordinate system of the 3D line scan camera to obtain the second coordinate change set; Based on the second set of coordinate changes and the coordinates of the feature points in the data recording information in the robot's base coordinate system, the set of cutting point coordinates is determined.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 8.

Images