A firebrick grabbing detection method and device based on a combined segmentation model

Abstract

The application provides a firebrick grabbing detection method and grabbing device based on a combined segmentation model, and comprises the following steps: S1: configuring a truss robot and a gripper assembly thereof for grabbing firebricks, and an RGB-D camera for collecting firebrick images; the truss robot also has a three-axis movement function and a rotation function relative to the plumb direction; S2: completing RGB-D camera calibration, hand-eye calibration and YOLOv8 instance segmentation network model training; S3: simultaneously collecting and acquiring an RGB image and a depth image through the RGB-D camera, processing and analyzing the RGB image by using a grabbing detection algorithm, and outputting rectangular grabbing frame parameters; then, the mapping relationship between the RGB image and the depth image is obtained by using image registration, three-dimensional coordinates of a rectangular grabbing frame center point in a camera coordinate system are obtained, and the grabbing position of the firebrick in a robot three-dimensional grabbing space coordinate system is obtained by using the result of hand-eye calibration.

Description

Technical Field

[0001] This invention relates to the field of robotic gripping equipment technology, and in particular to a gripping detection method and gripping device for refractory bricks based on a combined segmentation model. Background Technology

[0002] Refractory bricks are a type of brick that is resistant to combustion and high temperatures. The material is pale yellow or brownish and is mainly used for lining smelting furnaces. Manual refractory bricklaying is a highly repetitive and labor-intensive job. Taking coke oven masonry engineering as an example, the coke oven structure is complex, with a wide variety of brick types, a large workload, high quality requirements, and a harsh environment, demanding high skills and physical fitness from the bricklayers. Faced with the industry's predicament of difficulty in recruiting workers and high labor costs, the coke oven masonry construction industry urgently needs to develop intelligent bricklaying robot technology to improve masonry quality and efficiency.

[0003] Grasping is a crucial means for bricklaying robots to interact with their environment and a fundamental skill. Utilizing visual image information for grasping detection can improve the environmental perception and intelligence level of bricklaying robots. The purpose of grasping detection is to obtain the grasping pose of the target in the robot space using image information. The grasping detection task requires not only accurately identifying the type of brick to be laid but also identifying the grasping position of the brick. Inaccurate grasping identification will lead to grasping failure, thus affecting subsequent bricklaying operations. Considering that when a bricklaying robot grasps a brick, the contact point between its front gripper and the brick is located on the boundary, the boundary contour of the refractory brick is an important identification feature in grasping detection. Instance segmentation methods combine the advantages of object detection and semantic segmentation, and can distinguish different instances of the same category in an image. However, due to annotation errors and the influence of deep learning deconvolution operations, the segmentation results inevitably lose some image details, leading to imprecise boundary contour segmentation. Active contour models excel at curve evolution in details, achieving high segmentation accuracy and providing continuous, smooth closed contour segmentation results. However, they cannot learn high-level semantic information and often fail to locate the target in complex background images.

[0004] In summary, it is essential to provide a method for edge detection of refractory bricks that can extract refractory brick edges from image information and a refractory brick grasping device to achieve accurate identification and reliable grasping of refractory bricks. Summary of the Invention

[0005] In view of this, the present invention proposes a refractory brick grasping and detection method and grasping device based on a combined segmentation model, which can improve the fineness of instance segmentation mask contour and improve the robot grasping stability.

[0006] The technical solution of this invention is implemented as follows:

[0007] On the one hand, the present invention provides a method for detecting the grasping of refractory bricks based on a combined segmentation model, comprising the following steps:

[0008] S1: A gantry robot and its gripper assembly are configured to grasp refractory bricks, as well as an RGB-D camera for acquiring images of the refractory bricks; the gantry robot also has three-axis movement and rotation relative to the vertical direction.

[0009] S2: Offline preparation phase, complete RGB-D camera calibration, hand-eye calibration and YOLOv8 instance segmentation network model training;

[0010] S3: In the online grasping and detection stage, RGB images and depth images are acquired simultaneously through an RGB-D camera. The RGB images are processed and analyzed using a grasping and detection algorithm to output the parameters of a rectangular grasping box. Then, the mapping relationship between the RGB images and the depth images is obtained through image registration to obtain the three-dimensional coordinates of the center point of the rectangular grasping box in the camera coordinate system. The grasping position of the refractory brick in the robot's three-dimensional grasping space coordinate system is obtained using the results of hand-eye calibration.

[0011] Based on the above technical solutions, preferably, step S2, which involves completing the RGB-D camera calibration, hand-eye calibration, and YOLOv8 instance segmentation network model training, firstly calibrates the intrinsic parameters of the RGB-D camera to obtain the transformation matrix between the image coordinate system where the RGB image is located and the three-dimensional camera coordinate system.

[0012] Then, hand-eye calibration is to obtain the transformation matrix between the three-dimensional coordinates of the robot's three-dimensional grasping space coordinate system and the three-dimensional coordinates in the camera coordinate system, where the gantry robot and its gripper assembly are located.

[0013] Finally, the YOLOv8 instance segmentation network model is trained by performing instance segmentation on several RGB images acquired by the RGB-D camera to obtain the optimal parameters of the YOLOv8 instance segmentation network model.

[0014] Preferably, in step S3, the simultaneous acquisition of RGB and depth images using an RGB-D camera, the processing and analysis of the RGB images using a grasping detection algorithm, and the output of rectangular grasping box parameters are achieved by using the target object mask obtained from the YOLOv8 instance segmentation network model trained in step S2 as the initial contour. Then, the original RGB image acquired by the RGB-D camera is processed into grayscale, and combined with the initial contour as the input of the Chan-Vese active contour segmentation model, a level set curve evolution is performed to obtain a smooth and continuous closed segmentation boundary contour. The bounding rectangle of this closed segmentation boundary contour is used as the rectangular grasping box g, and the five-dimensional parameters of the rectangular grasping box g are further obtained, mathematically represented as g = {x, y, θ, w, h}, where x and y are the coordinates of the center point of the rectangular grasping box in the planar image, θ is the rotation angle of the rectangular grasping box relative to the x-axis of the planar image, and w and h are the width and height of the rectangular grasping box, respectively.

[0015] In a further preferred embodiment, step S3, which involves obtaining the mapping relationship between the RGB image and the depth image through image registration, is based on the fact that the RGB image and the depth image obtained by the RGB-D camera observe the same point in the robot's three-dimensional grasping space coordinate system. The extrinsic parameters of the coordinate system where the RGB lens in the RGB-D camera is located, i.e., the camera coordinate system and the coordinate system where the depth infrared lens is located, are calibrated respectively to obtain the coordinate transformation relationship between the camera coordinate system and the coordinate system where the depth infrared lens is located, thus obtaining the image-registered RGB image and depth image.

[0016] More preferably, in step S3, obtaining the three-dimensional coordinates of the center point of the rectangular grasping frame in the camera coordinate system and using the hand-eye calibration results to obtain the grasping position of the refractory brick in the robot's three-dimensional grasping space coordinate system involves representing the position G of the refractory brick in the camera coordinate system in the image-registered RGB image as: G = {P, D, R, W}, where P is the three-dimensional coordinate of the grasping center point of the refractory brick, D represents the vertical distance of the gantry robot's gripper assembly toward the grasping center point of the refractory brick, R represents the rotation angle of the gantry robot's gripper assembly relative to the vertical direction, and W is the width of the opening of the gantry robot's gripper assembly; based on the already performed hand-eye calibration, the pose G of the refractory brick in the camera coordinate system is converted into the travel or rotation angle of each axis in the robot's three-dimensional grasping space coordinate system.

[0017] In a further preferred embodiment, the width w of the rectangular gripping frame or the opening width W of the gripper assembly of the gantry robot is not less than the width of the refractory brick.

[0018] On the other hand, the present invention provides a refractory brick gripping device, including

[0019] A gantry robot, fixed relative to the ground;

[0020] A rotating assembly is disposed at the end of the gantry robot away from the ground, the rotating assembly extending along the vertical direction and rotatably connected to the gantry robot;

[0021] A gripper assembly is disposed at the end of the rotating assembly near the ground and is fixedly connected to the rotating assembly.

[0022] An RGB-D camera is positioned at the end of the gantry robot that is furthest from the ground to acquire image information of the refractory bricks directly below the gripper assembly;

[0023] The truss robot and its gripping components use the aforementioned refractory brick detection method based on the combined segmentation model to convert the image information of the refractory brick into the robot's three-dimensional gripping space coordinate system, enabling the gripper components to complete the gripping of the refractory brick.

[0024] Preferably, the gantry robot includes a first body, a first moving mechanism, a second moving mechanism, and a third moving mechanism; the first body is fixed relative to the ground, and a three-dimensional grasping space coordinate system of the robot is constructed with the length direction of the first body as the X-axis direction, the width direction of the first body as the Y-axis direction, the vertical direction as the Z-axis direction, and the vertex of the end of the first body away from the ground as the origin.

[0025] The first moving mechanism is located at the end of the first body away from the ground. The first moving mechanism is slidably connected to the first body. The first moving mechanism moves along the X-axis direction of the robot's three-dimensional grasping space coordinate system.

[0026] The second moving mechanism is mounted on the movable end of the first moving mechanism, and the second moving mechanism moves along the Y-axis of the robot's three-dimensional grasping space coordinate system.

[0027] The third moving mechanism is disposed on the movable end of the second moving mechanism, and the third moving mechanism moves along the Z-axis direction of the robot's three-dimensional grasping space coordinate system;

[0028] Both the rotating component and the RGB-D camera are mounted on the movable end of the third moving mechanism. The rotating component is rotatably connected to the movable end of the third moving mechanism, and the output end of the rotating component extends vertically toward the ground. The RGB-D camera is fixedly connected to the fixed end of the third moving mechanism.

[0029] The gripper assembly is located at the end of the rotating assembly away from the third moving mechanism and is fixedly connected to the rotating assembly.

[0030] Preferably, the gripper assembly includes a mounting plate, a first motor, a second motor, a drive gear, a first transmission shaft, a second transmission shaft, a driven gear, a first clamping plate, and a second clamping plate. The mounting plate has an opening, within which a first transmission shaft and a second transmission shaft are hinged to the inner surface of the mounting plate. One end of the mounting plate away from the opening is fixedly connected to the end of the rotating assembly away from the third moving mechanism. The first motor and the second motor are mounted on the mounting plate, and their output shafts extend into the opening. A drive gear is mounted on the output shaft of the first motor. A driven gear is fixedly mounted at the end of either the first or second transmission shaft, meshing with the drive gear. The surface of the first transmission shaft also meshes with the first clamping plate, and the second transmission shaft also meshes with the second clamping plate. The first clamping plate and the second clamping plate are slidably connected to the mounting plate. The first motor and the second motor correspondingly drive the first and second transmission shafts to rotate synchronously, changing the distance between the first clamping plate and the second clamping plate. The first clamping plate and the second clamping plate also have a position locking function.

[0031] More preferably, a friction layer is provided on the adjacent end faces of the first clamping plate and the second clamping plate, and a pressure sensor is also provided between the surface of the first clamping plate or the second clamping plate and the friction layer, and the pressure sensor is electrically connected to the gantry robot.

[0032] The present invention provides a refractory brick grasping and detection method and grasping device based on a combined segmentation model, which has the following advantages compared with the prior art:

[0033] (1) In view of the problem of plane grasping and detection of refractory bricks, this application designs an image target detection method and its grasping device based on a combined segmentation model. An RGB-D camera that can acquire color images and depth images at the same time is used as an image acquisition device. The five-dimensional rectangular grasping box detected on the image represents the grasping pose, which limits the gripping part and starting position of the grasping device.

[0034] (2) The boundary contour of the upper surface of the refractory brick to be captured is used as the segmentation and detection target. In order to overcome the interference of complex background, the YOLOv8 instance segmentation model is first used to process the color image, segment the target mask, and output the target mask image. In order to obtain a more refined edge contour, the Chan-Vese active contour model is further used. The grayscale processed color image and the target mask image are used as the model input to perform horizontal set curve evolution, thereby obtaining a smooth and closed boundary contour, which improves the fineness and resolution of the contour.

[0035] (3) To detect the rectangular grabbing box of the target object in the image, the center of the area of ​​the segmented closed region is used as the grabbing center point and the direction angle of the smallest bounding rectangle of the closed region is used as the grabbing angle. This method is simple to calculate and has good robustness and anti-interference effect on complex background images.

[0036] (4) The first and second clamping plates with position locking function can cooperate with the friction layer to keep the refractory bricks in the gripping state from falling off. The pressure sensor further provides feedback gripping signals to the control part of the gantry robot. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a flowchart of the detection process of a refractory brick grasping and detection method and grasping device based on a combined segmentation model according to the present invention.

[0039] Figure 2 This is a flowchart of the grabbing and detection algorithm of a refractory brick grabbing and detection method and grabbing device based on a combined segmentation model according to the present invention.

[0040] Figure 3 This is a perspective view of a refractory brick grasping and detection method and grasping device based on a combined segmentation model according to the present invention.

[0041] Figure 4 This is a three-dimensional view of the combined state of the rotating component and the gripper component of the refractory brick grasping and detection method and grasping device based on the combined segmentation model of the present invention.

[0042] Figure 5 This is a schematic diagram of the rectangular grasping frame of a refractory brick according to the present invention, which is a refractory brick grasping and detection method and grasping device based on a combined segmentation model.

[0043] Figure 6 This is a schematic diagram of the gripper assembly of a refractory brick gripping and detection method and gripping device based on a combined segmentation model according to the present invention, gripping a refractory brick.

[0044] Figure 7 This is a comparison of the segmentation effects of an example of a refractory brick grasping and detection method and grasping device based on a combined segmentation model according to the present invention.

[0045] Figure 8 This is a schematic diagram illustrating the image capture and detection effect of a refractory brick grasping and detection method and grasping device based on a combined segmentation model according to the present invention.

[0046] Reference numerals: 1. Truss robot; 2. Rotating component; 3. Gripper component; 4. RGB-D camera; 5. Refractory brick; 6. Base; 31. First motor; 32. Mounting plate; 33. Driven gear; 34. First drive shaft; 35. First clamping plate; 36. Second drive shaft; 37. Second clamping plate; 38. Drive gear; 39. Second motor. Detailed Implementation

[0047] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0048] like Figure 1 Combination Figure 3 and Figure 4 As shown, this application provides a refractory brick grasping and detection method based on a combined segmentation model, including the following steps:

[0049] S1: A gantry robot 1 and its gripper assembly 3 are configured for grasping refractory bricks, along with an RGB-D camera 4 for acquiring images of the refractory bricks. The gantry robot 1 also has three-axis movement and rotational capabilities relative to the vertical direction. To reliably move the gripper assembly 3, the gantry robot 1 possesses four degrees of freedom: translational capability in three axial directions and rotational capability in the vertical direction. The RGB-D camera 4 is a corresponding imaging device set on the same optical axis to acquire RGB color and depth images. The viewing direction is vertically aligned with one or more refractory bricks 5 on the base below the gantry robot 1, thereby acquiring RGB and depth images of the refractory bricks.

[0050] S2: Offline preparation phase, complete RGB-D camera 4 calibration, hand-eye calibration and YOLOv8 instance segmentation network model training;

[0051] Specifically, the intrinsic parameters of the RGB-D camera 4 are first calibrated to obtain the transformation matrix between the image coordinate system of the RGB image and the three-dimensional camera coordinate system. This step is for calibrating the relationship between the image coordinate system / pixel coordinate system and the camera coordinate system of the RGB-D camera 4. The difference is that the axial unit of the pixel coordinate system is pixels, while the axial unit of the image coordinate system is millimeters.

[0052] Then, hand-eye calibration involves obtaining the transformation matrix between the 3D coordinates of the gantry robot 1 and its gripper assembly 3 in the robot's 3D grasping space coordinate system and the 3D coordinates in the camera coordinate system; this can be understood as a 3D-to-3D transformation process that transforms the 3D coordinates of the gantry robot 1 and its gripper assembly 3 in the camera coordinate system to the robot's 3D grasping space coordinate system. Here, the robot's 3D grasping space coordinate system can be understood as the world coordinate system.

[0053] Finally, the YOLOv8 instance segmentation network model is trained by performing instance segmentation on several RGB images acquired by RGB-D camera 4 to obtain the optimal parameters of the YOLOv8 instance segmentation network model. The YOLOv8 model has five models for detection, segmentation, and classification in each category, and it is bundled with several pre-trained models. The code for this model is open source and easily accessible.

[0054] S3: In the online grasping and detection stage, RGB images and depth images are acquired simultaneously through RGB-D camera 4. The RGB images are processed and analyzed using the grasping and detection algorithm to output the parameters of the rectangular grasping box. Then, the mapping relationship between the RGB images and the depth images is obtained by image registration to obtain the three-dimensional coordinates of the center point of the rectangular grasping box in the camera coordinate system. The grasping position of the refractory brick in the robot's three-dimensional grasping space coordinate system is obtained by using the hand-eye calibration results.

[0055] like Figure 2As shown, the specific content of outputting the rectangular grasping box parameters using the grasping detection algorithm mentioned here is as follows: The target object mask is obtained as the initial contour using the YOLOv8 instance segmentation network model trained in step S2; then, the original RGB image acquired by the RGB-D camera 4 is processed into grayscale, and combined with the initial contour as the input of the Chan-Vese active contour segmentation model, a level set curve evolution is performed to obtain a smooth and continuous closed segmentation boundary contour. The bounding rectangle of this closed segmentation boundary contour is used as the rectangular grasping box g, and the five-dimensional parameters of the rectangular grasping box g are further obtained, mathematically represented as g = {x, y, θ, w, h}, where x and y are the coordinates of the center point of the rectangular grasping box in the planar image, θ is the rotation angle of the rectangular grasping box relative to the x-axis of the planar image, which can also be understood as the rotation angle of the rectangular grasping box relative to the initial position of the closed segmentation boundary contour; w and h are the width and height of the rectangular grasping box, respectively. The rectangular grasping frame defines the pre-grabbing pose of the gripper assembly 3. The grasping detection algorithm performs the task of detecting the grasping rectangular frame of the refractory brick in the image and target recognition. The center point (x, y) of the target object mask is used as the center point of the rectangular grasping frame. The height h of the rectangular grasping frame is prior information and is related to the width of the moving part of the gripper assembly 3. The width w of the rectangular grasping frame is determined by the thickness l of the refractory brick along its short side. To prevent interference between the gripper assembly 3 and the refractory brick during movement, the width w of the rectangular grasping frame is extended outward by a distance l1 on both sides. Therefore, the actual width w of the rectangular grasping frame is l + 2l1, meaning the width w of the grasping frame is not less than the width of the refractory brick. The planar image mentioned here can be either a planar image in the image coordinate system of the RGB image acquired by the RGB-D camera 4 or a planar image in the pixel coordinate system.

[0056] like Figure 1 Combination Figure 2 As shown, image registration relies on the mapping relationship between the RGB image and the depth image of the RGB-D camera 4. Since the RGB image and depth image acquired by the RGB-D camera 4 observe the same point in the robot's 3D grasping space coordinate system, extrinsic parameter calibration is performed on the coordinate systems of the RGB lens (i.e., the camera coordinate system) and the depth infrared lens (i.e., the coordinate system of the depth infrared lens). This yields the coordinate transformation relationship between the camera coordinate system and the depth infrared lens coordinate system, resulting in the registered RGB image and depth image. Image registration can be classified in various ways depending on the image acquisition method. This scheme utilizes multimodal registration, i.e., images acquired by different imaging devices in the same scene. This type of registration integrates information from different sources to obtain more detailed and complex data, better restoring the spatial resolution details of the observed target, i.e., the refractory brick, thus facilitating further improvement in the accuracy of the subsequent grasping process.

[0057] like Figure 5 As shown, obtaining the grasping position of the refractory brick in the robot's three-dimensional grasping space coordinate system involves representing the position G of the refractory brick in the camera coordinate system from the image-registered RGB image as: G = {P, D, R, W}, where P is the three-dimensional coordinate of the grasping center point of the refractory brick, i.e. Figure 5 In the diagram, P(X,Y,Z); D represents the vertical distance of the gripper assembly 3 of the gantry robot 1 towards the center point of the refractory brick gripping, R represents the rotation angle of the gripper assembly 3 of the gantry robot 1 relative to the vertical direction, and W is the width of the gripper assembly 3 of the gantry robot 1. Based on the already performed hand-eye calibration, the pose G of the refractory brick in the camera coordinate system is converted into the travel or rotation angle of each axis in the robot's three-dimensional gripping space coordinate system. Similarly, to avoid interference during gripping, the width W of the gripper assembly 3 of the gantry robot 1 is not less than the width of the refractory brick.

[0058] In addition, in order to achieve precise gripping of refractory bricks, such as Figure 3 As shown, the present invention provides a refractory brick gripping device, including...

[0059] The gantry robot 1 is fixed relative to the ground.

[0060] The rotating component 2 is disposed at the end of the gantry robot 1 away from the ground, and the rotating component 2 extends along the vertical direction and is rotatably connected to the gantry robot 1;

[0061] The gripper assembly 3 is disposed at the end of the rotating assembly 2 near the ground and is fixedly connected to the rotating assembly 2;

[0062] An RGB-D camera 4 is positioned at the end of the gantry robot 1 that is away from the ground, and is used to acquire image information of the refractory brick 5 directly below the gripper assembly 3. The RGB-D camera 4 can be installed by extending it horizontally outward a certain distance via a connecting plate.

[0063] The control unit of the gantry robot 1 and its gripping component incorporates a refractory brick detection method based on the aforementioned combined segmentation model. This method converts the image information of the refractory brick into the robot's three-dimensional gripping space coordinate system, enabling the gripper component 3 to grasp the refractory brick. The robot's three-dimensional gripping space coordinate system mentioned in this application is constructed with the length direction of the first body as the X-axis, the width direction of the first body as the Y-axis, the vertical direction as the Z-axis, and the vertex of the end of the first body furthest from the ground as the origin; it can be understood as a world coordinate system.

[0064] like Figure 3As shown, the gantry robot 1 includes a first body, a first moving mechanism, a second moving mechanism, and a third moving mechanism; the first body is fixed relative to the ground, and is hollow inside and spans the ground. A base 6 for placing refractory bricks 5 is provided inside the first body.

[0065] The first moving mechanism is located at the end of the first body away from the ground. The first moving mechanism is slidably connected to the first body. The first moving mechanism moves along the X-axis direction of the robot's three-dimensional grasping space coordinate system. Figure 3 The diagram shows the reciprocating motion direction along the width of the screen. The second moving mechanism is mounted on the movable end of the first moving mechanism, and moves along the Y-axis of the robot's three-dimensional grasping space coordinate system. Figure 3 The direction of reciprocating motion perpendicular to the screen is shown; the third moving mechanism is set on the movable end of the second moving mechanism, and the third moving mechanism moves along the Z-axis of the robot's three-dimensional grasping space coordinate system, that is, along the direction of reciprocating motion along the screen height.

[0066] Both the rotating component 2 and the RGB-D camera 4 are mounted on the movable end of the third moving mechanism. The rotating component 2 is rotatably connected to the movable end of the third moving mechanism. The output end of the rotating component 2 extends vertically towards the ground, driving the gripper assembly 3 to select any angle along the vertical direction for better alignment with the refractory brick 5 below. The gripper assembly 3 is located at the end of the rotating component 2 furthest from the third moving mechanism and is fixedly connected to the rotating component 2. The rotating component 2 can be implemented using a rotary motor or a rotary cylinder mechanism.

[0067] The RGB-D camera 4 is fixedly connected to the fixed end of the third moving mechanism, meaning that no matter how the gripper assembly 3 and the rotating assembly 2 rotate, the position of the RGB-D camera 4 relative to the third moving mechanism of the gantry robot 1 remains unchanged.

[0068] like Figure 4As shown, the gripper assembly 3 includes a mounting plate 32, a first motor 31, a second motor 39, a drive gear 38, a first transmission shaft 34, a second transmission shaft 36, a driven gear 33, a first clamping plate 35, and a second clamping plate 37. The mounting plate 32 has an opening, within which the first transmission shaft 34 and the second transmission shaft 36 are hinged to the inner surface of the mounting plate 32. A guide rod is also fixedly installed within the opening, parallel to and spaced apart from the first transmission shaft 34 and the second transmission shaft 36. The end of the mounting plate 32 furthest from the opening is furthest from the rotating assembly 2. One end of the three moving mechanisms is fixedly connected, and the guide rod is set at the edge of the opening; the first motor 31 and the second motor 39 are set on the mounting plate 32, and the output shafts of the first motor 31 and the second motor 39 extend into the opening. The output shaft of the first motor 31 is provided with a drive gear 38; the end of the first transmission shaft 34 or the second transmission shaft 36 is fixedly provided with a driven gear 33, which meshes with the drive gear 38; as shown in the figure, the first motor 31 drives the first transmission shaft 34 to rotate, and the second motor 39 drives the second transmission shaft 36 to rotate.

[0069] The surface of the first drive shaft 34 is also engaged with the first clamping plate 35, and the second drive shaft 36 is also engaged with the second clamping plate 37; both the first clamping plate and the second clamping plate 37 are slidably connected to the guide rod. The first clamping plate 35 and the second clamping plate 37 are respectively slidably connected to the mounting plate 32; the first motor 31 and the second motor 39 drive the first drive shaft 34 and the second drive shaft 36 to rotate synchronously, changing the distance between the first clamping plate 35 and the second clamping plate 37; the first clamping plate 35 and the second clamping plate 37 also have a position locking function. The first clamping plate 35 and the second clamping plate 37 are the movable parts of the clamping plate assembly 3.

[0070] The position locking function of the first clamping plate 35 and the second clamping plate 37 can be achieved by additionally setting up a tube body, a compression spring, and two pressure plates. For example, along the axial direction of the first drive shaft 34 or the second drive shaft 36, a through window is set on the first clamping plate 35 and the second clamping plate 37. The outline of the window is rectangular. The outer surface of the tube body is hinged to the mounting plate 32, and both ends of the tube body are respectively inserted into the window. Pressure plates that can only be compressed or reset along the axial direction of the tube body are clamped at both ends of the tube body. The outline of the pressure plates is slightly smaller than the outline of the window. A compression spring is set inside the tube body, and both ends of the compression spring are fixedly connected to the ends of the two pressure plates that extend into the tube body. When the tube body and the two pressure plates are in the initial position, the two pressure plates pass through the window on the first clamping plate 35 or the second clamping plate 37 respectively, and the central axis of the two pressure plates is set at an angle to the central axis of the window, so that the pressure plates abut against the mutually distant end faces of the first clamping plate 35 and the second clamping plate 37, and the compression spring is in an uncompressed state. When the first clamping plate 35 and the second clamping plate 37 approach each other to clamp the refractory brick 5, the distance between the two clamping plates and the compression spring between them are compressed, which can prevent the first clamping plate 35 and the second clamping plate 37 from accidentally opening.

[0071] In a preferred embodiment, to improve the tightness of the gripping of the refractory brick, a friction layer is provided on the adjacent end faces of the first clamping plate 35 and the second clamping plate 37. A pressure sensor is also provided between the surface of the first clamping plate 35 or the second clamping plate 37 and the friction layer, and the pressure sensor is electrically connected to the gantry robot 1. The friction layer has a rough surface, which can increase the friction force when in contact with the refractory brick 5, and prevent the refractory brick from falling off during clamping and handling. The further provided pressure sensor can output a detection signal, and based on the presence or absence of the detection signal, it can provide feedback on whether the gripper assembly 3 is in a reliable clamping state.

[0072] Reference Figure 7 , Figure 7 A schematic diagram showing the acquisition of a target object mask for refractory bricks is presented. Figure 7 Figure (a) shows the original refractory brick image; Figure (b) shows the jagged outline obtained by simply using the YOLOv8 instance segmentation mask; Figure (c) shows the result after using the target object mask obtained by the YOLOv8 instance segmentation network model as the initial outline, and then processing the original RGB image into grayscale, combining the initial outline as input to the Chan-Vese active contour segmentation model, and performing level set curve evolution to obtain a smooth and continuous closed segmentation boundary contour; Figure (d) shows the result of manually annotated contours. It is evident that the result of this scheme, as shown in Figure (c), is superior to the result in Figure (b), is closer to and smoother than the result of manually labeled standard contours, significantly eliminates edge jaggedness, improves the accuracy of refractory brick contour segmentation, and provides a better image foundation for the subsequent generation of rectangular grasping boxes.

[0073] like Figure 8 As shown, the instance segmentation effect of target refractory bricks in images acquired under different complex background conditions, including single target objects, multiple target objects, and stacked objects, is demonstrated. The grasping and detection method proposed in this application has high detection accuracy and can meet the stable grasping requirements of gantry robots.

[0074] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A firebrick grabbing detection method based on a combined segmentation model, characterized by, Includes the following steps: S1: A gantry robot (1) for gripping refractory bricks and its gripper assembly (3) are configured, as well as an RGB-D camera (4) for acquiring images of refractory bricks; the gantry robot (1) also has three-axis movement and rotation relative to the vertical direction. S2: Offline preparation stage, complete RGB-D camera (4) calibration, hand-eye calibration and YOLOv8 instance segmentation network model training; Step S2 describes completing the calibration of the RGB-D camera (4), hand-eye calibration, and YOLOv8 instance segmentation network model training. First, the intrinsic parameters of the RGB-D camera (4) are calibrated to obtain the transformation matrix between the image coordinate system where the RGB image is located and the three-dimensional camera coordinate system. Then, hand-eye calibration is to obtain the transformation matrix between the three-dimensional coordinates of the robot's three-dimensional grasping space coordinate system and the three-dimensional coordinates in the camera coordinate system, where the gantry robot (1) and its gripper assembly (3) are located; Finally, the YOLOv8 instance segmentation network model is trained by performing instance segmentation on several RGB images acquired by the RGB-D camera (4) to obtain the optimal parameters of the YOLOv8 instance segmentation network model. S3: In the online grasping and detection stage, RGB images and depth images are acquired simultaneously by an RGB-D camera (4). The RGB images are processed and analyzed using the grasping and detection algorithm to output the parameters of the rectangular grasping box. Then, the mapping relationship between the RGB images and the depth images is obtained by image registration to obtain the three-dimensional coordinates of the center point of the rectangular grasping box in the camera coordinate system. The grasping position of the refractory brick in the robot's three-dimensional grasping space coordinate system is obtained by using the results of hand-eye calibration. Step S3 involves simultaneously acquiring RGB and depth images using an RGB-D camera (4), processing and analyzing the RGB images using a capture detection algorithm, and outputting rectangular capture box parameters. This is achieved by using the target object mask obtained from the YOLOv8 instance segmentation network model trained in step S2 as the initial contour. Then, the original RGB image acquired by the RGB-D camera (4) is processed into grayscale, and combined with the initial contour as the input of the Chan-Vese active contour segmentation model, a level set curve evolution is performed to obtain a smooth and continuous closed segmentation boundary contour. The bounding rectangle of this closed segmentation boundary contour is used as the rectangular capture box. And further obtain the rectangular grab box The five-dimensional parameters are mathematically represented as ,in and The coordinates of the center point of the rectangular capture box in the planar image. For the rectangular capture box relative to the planar image The rotation angle of the shaft; and These are the width and height of the rectangular capture box, respectively. Step S3 involves obtaining the three-dimensional coordinates of the center point of the rectangular grasping frame in the camera coordinate system. Using the hand-eye calibration results, the grasping position of the refractory brick in the robot's three-dimensional grasping space coordinate system is obtained. This involves determining the position of the refractory brick in the camera coordinate system within the RGB image after image registration. Represented as: ,in These are the three-dimensional coordinates of the center point of the refractory brick. This indicates the vertical distance of the gripper assembly (3) of the gantry robot (1) toward the center point of the refractory brick gripping. This indicates the rotation angle of the gripper assembly (3) of the gantry robot (1) relative to the vertical direction. The width of the gripper assembly (3) of the gantry robot (1) is defined; the pose of the refractory brick in the camera coordinate system is determined based on the hand-eye calibration already performed. This is converted into the travel or rotation angle of each axis within the robot's three-dimensional grasping space coordinate system.

2. The firebrick grabbing detection method based on a combined segmentation model according to claim 1, characterized in that, Step S3 describes obtaining the mapping relationship between the RGB image and the depth image by using image registration. This is based on the fact that the RGB image and the depth image obtained by the RGB-D camera (4) observe the same point in the robot's three-dimensional grasping space coordinate system. The external parameters of the coordinate system where the RGB lens in the RGB-D camera (4) is located, namely the camera coordinate system and the coordinate system where the depth infrared lens is located, are calibrated respectively to obtain the coordinate transformation relationship between the camera coordinate system and the coordinate system where the depth infrared lens is located, and thus obtain the RGB image and the depth image after image registration.

3. The firebrick grabbing detection method based on a combined segmentation model according to claim 1, characterized in that, The width of the rectangular gripping frame Or the width of the open jaws of the gripper assembly (3) of the gantry robot (1) Not less than the width of the firebrick.

4. A refractory brick gripping device characterized by, include A gantry robot (1) is fixed relative to the ground; A rotating component (2) is disposed at one end of the gantry robot (1) away from the ground. The rotating component (2) extends along the vertical direction and is rotatably connected to the gantry robot (1). The gripper assembly (3) is disposed at one end of the rotating assembly (2) near the ground and is fixedly connected to the rotating assembly (2); An RGB-D camera (4) is set at one end of the gantry robot (1) away from the ground to acquire image information of the refractory brick (5) directly below the gripper assembly (3); Among them, the truss robot (1) and its gripping component according to any one of claims 1-3, the refractory brick gripping and detection method based on the combined segmentation model, converts the image information of the refractory brick into the robot's three-dimensional gripping space coordinate system, so that the gripper component (3) completes the gripping of the refractory brick.

5. A refractory brick gripping device according to claim 4, wherein The gantry robot (1) includes a first body, a first moving mechanism, a second moving mechanism and a third moving mechanism; the first body is fixed relative to the ground, with the length direction of the first body as the X-axis direction, the width direction of the first body as the Y-axis direction, the vertical direction as the Z-axis direction, and the vertex of the end of the first body away from the ground as the origin to construct the robot's three-dimensional grasping space coordinate system. The first moving mechanism is located at the end of the first body away from the ground. The first moving mechanism is slidably connected to the first body. The first moving mechanism moves along the X-axis direction of the robot's three-dimensional grasping space coordinate system. The second moving mechanism is mounted on the movable end of the first moving mechanism, and the second moving mechanism moves along the Y-axis of the robot's three-dimensional grasping space coordinate system. The third moving mechanism is disposed on the movable end of the second moving mechanism, and the third moving mechanism moves along the Z-axis direction of the robot's three-dimensional grasping space coordinate system; The rotating component (2) and the RGB-D camera (4) are both mounted on the movable end of the third moving mechanism. The rotating component (2) is rotatably connected to the movable end of the third moving mechanism. The output end of the rotating component (2) extends toward the ground along the vertical direction. The RGB-D camera (4) is fixedly connected to the fixed end of the third moving mechanism. The gripper assembly (3) is located at the end of the rotating assembly (2) away from the third moving mechanism and is fixedly connected to the rotating assembly (2).

6. A refractory brick gripping device according to claim 5, wherein The gripper assembly (3) includes a mounting plate (32), a first motor (31), a second motor (39), a drive gear (38), a first transmission shaft (34), a second transmission shaft (36), a driven gear (33), a first clamping plate (35), and a second clamping plate (37). The mounting plate (32) has an opening, in which the first transmission shaft (34) and the second transmission shaft (36) are hinged to the inner surface of the mounting plate (32). One end of the mounting plate (32) away from the opening is fixedly connected to the end of the rotating assembly (2) away from the third moving mechanism. The first motor (31) and the second motor (39) are mounted on the mounting plate (32), and the output shafts of the first motor (31) and the second motor (39) extend into the opening. The output shaft of the first motor (31)... A drive gear (38) is provided on the output shaft; a driven gear (33) is fixedly provided at the end of the first drive shaft (34) or the second drive shaft (36), and the driven gear (33) meshes with the drive gear (38); the surface of the first drive shaft (34) also meshes with the first clamping plate (35), and the second drive shaft (36) also meshes with the second clamping plate (37); the first clamping plate (35) and the second clamping plate (37) are slidably connected to the mounting plate (32); the first motor (31) and the second motor (39) drive the first drive shaft (34) and the second drive shaft (36) to rotate synchronously, changing the size of the distance between the first clamping plate (35) and the second clamping plate (37); the first clamping plate (35) and the second clamping plate (37) also have a position locking function.

7. A refractory brick gripping device according to claim 6, wherein The first clamping plate (35) and the second clamping plate (37) are provided with friction layers on their adjacent end faces. A pressure sensor is also provided between the surface of the first clamping plate (35) or the second clamping plate (37) and the friction layer. The pressure sensor is electrically connected to the gantry robot (1).

Images