Visual guidance methods, devices, and media for fine-tuning of track slabs

Abstract

This disclosure relates to a visual guidance method, apparatus, and medium for fine-tuning of track slabs, and pertains to the field of control technology. The control method includes: acquiring a first target image of a tightening groove mounted on a work tool on the track slab, the tightening groove accommodating and fixing a robot's tightening shaft to achieve a fixed installation between the robot and the work tool; processing the first target image using a deep learning model to determine a first target detection region corresponding to the tightening groove in the first target image; extracting a region of interest (ROI) image corresponding to the tightening groove from the first target image based on the first target detection region; determining target position information of the tightening groove based on the ROI image; and, based on the target position information, aligning the centerline of the tightening shaft with the centerline of the tightening groove and inserting the tightening shaft into the tightening groove. This improves the accuracy of track slab fine-tuning operations.

Description

Technical Field

[0001] This disclosure relates to the field of control technology, and in particular to a visual guidance method, device and medium for fine-tuning of track slabs. Background Technology

[0002] In the field of rail transit construction, track slab fine-tuning is a crucial step in ensuring the safe and stable operation of trains. With the continuous growth of high-speed rail operating mileage and the increasing demands for refined line management, the need for high-precision, automated track slab fine-tuning is becoming increasingly urgent. Summary of the Invention

[0003] The inventors of this disclosure have discovered the following problem in the above-mentioned related technologies: the accuracy of track slab fine-tuning operations is low.

[0004] To address the aforementioned problems, the present disclosure provides the following solutions.

[0005] According to some embodiments of this disclosure, a robot control method is provided, comprising: acquiring a first target image of a tightening groove mounted on a work tool on a track plate, the tightening groove being used to accommodate and fix a tightening shaft of the robot, so that the robot is fixedly installed with the work tool; processing the first target image using a deep learning model to determine a first target detection region corresponding to the tightening groove in the first target image; extracting a region of interest (ROI) image corresponding to the tightening groove from the first target image based on the first target detection region; determining target position information of the tightening groove based on the ROI image; and, based on the target position information, controlling the centerline of the tightening shaft to align with the centerline of the tightening groove, and inserting the tightening shaft into the tightening groove.

[0006] In some embodiments, the target location information includes at least one of the following: the position coordinates of the center point of the tightening groove in the first target image in pixel coordinate system, the size of the tightening groove, and the rotation angle of the center line of the tightening groove relative to the horizontal direction.

[0007] In some embodiments, extracting the ROI image corresponding to the tightening groove from the first target image based on the first target detection region includes: expanding a predetermined number of pixels outward from the boundary of the first target detection region to determine a second target detection region; and extracting the second target detection region as the ROI image from the first target image.

[0008] In some embodiments, determining the target location information of the tightening groove based on the ROI image includes: converting the ROI image into a grayscale image; separating the tightening groove from the background information in the grayscale image using a threshold segmentation algorithm; removing the background information to obtain a second target image of the area where the tightening groove is located; and determining the target location information based on the second target image.

[0009] In some embodiments, removing the background information to obtain a second target image of the area where the tightening groove is located includes: smoothing the edge of the tightening groove in the grayscale image; and removing the background information after the smoothing process to obtain the second target image.

[0010] In some embodiments, determining the target location information based on the second target image includes: performing corner detection on the tightening groove based on the second target image to obtain corner information of the tightening groove; and determining the target location information based on the corner information of the tightening groove.

[0011] In some embodiments, acquiring a first target image of a tightening groove on a working tool mounted on a track plate includes: controlling an image acquisition device mounted on the robot to move to a designated position to acquire the first target image at the designated position, wherein, when the image acquisition device is located at the designated position, the center of the field of view of the image acquisition device coincides with the center of the tightening groove.

[0012] In some embodiments, the target position information includes the position coordinates of the center point of the tightening groove in the first target image in a pixel coordinate system. The step of aligning the centerline of the tightening shaft with the centerline of the tightening groove based on the target position information and inserting the tightening shaft into the tightening groove includes: determining a first offset of the center of the tightening groove relative to the field of view center of the first target image in a pixel coordinate system based on the position coordinates; converting the first offset into a second offset in the robot's manipulator coordinate system; and, based on the second offset, aligning the centerline of the tightening shaft with the centerline of the tightening groove and inserting the tightening shaft into the tightening groove.

[0013] In some embodiments, aligning the centerline of the tightening shaft with the centerline of the tightening groove based on the second offset and inserting the tightening shaft into the tightening groove includes: determining an initial offset of the center of the tightening shaft relative to the field of view center of the first target image in the robotic arm coordinate system; determining a third offset of the center of the tightening shaft relative to the center of the tightening groove in the robotic arm coordinate system based on the initial offset and the second offset; and aligning the centerline of the tightening shaft with the centerline of the tightening groove based on the third offset and inserting the tightening shaft into the tightening groove.

[0014] According to some other embodiments of this disclosure, a robot control device is provided, comprising: an acquisition module configured to acquire a first target image of a tightening groove mounted on a work tool on a track plate, the tightening groove being used to accommodate a tightening shaft of the robot to fix the robot to the work tool; a processing module configured to process the first target image using a deep learning model to determine a first target detection region corresponding to the tightening groove in the first target image; an extraction module configured to extract a region of interest (ROI) image corresponding to the tightening groove from the first target image based on the first target detection region; a determination module configured to determine target position information of the tightening groove based on the ROI image; and a control module configured to, based on the target position information, control the centerline of the tightening shaft to align with the centerline of the tightening groove, and insert the tightening shaft into the tightening groove.

[0015] According to further embodiments of this disclosure, a robot control device is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute the control method of any of the above embodiments based on instructions stored in the memory device.

[0016] According to further embodiments of this disclosure, a robot is provided, including the control device of any of the above embodiments.

[0017] According to further embodiments of the present disclosure, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the control method of any of the above embodiments.

[0018] According to further embodiments of this disclosure, a computer program product is also provided, including instructions that, when executed by a processor, cause the processor to perform the control method according to any of the foregoing embodiments.

[0019] In the above embodiments, the initial position of the tightening groove is quickly captured by the deep learning model, and a visual detection algorithm based on ROI image localization is combined. This eliminates the need for fine-tuning the deep learning model and reduces the interference of background information on the visual detection algorithm for identifying the tightening groove. Thus, the advantages of both methods are combined to achieve the technical effect of quickly and accurately identifying the tightening groove, thereby improving the efficiency and accuracy of track slab fine-tuning operations. Attached Figure Description

[0020] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the specification, serve to explain the principles of this disclosure.

[0021] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein:

[0022] Figure 1 A flowchart illustrating a robot control method according to some embodiments of the present disclosure is shown;

[0023] Figure 2 A perspective view of a working tool according to some embodiments of the present disclosure is shown;

[0024] Figure 3 A top view of a working tool according to some embodiments of the present disclosure is shown;

[0025] Figure 4 A partial structural diagram of a robot according to some embodiments of the present disclosure is shown;

[0026] Figure 5 A flowchart illustrating a robot control method according to other embodiments of the present disclosure is shown;

[0027] Figure 6 A flowchart illustrating some implementation methods of steps 430 and 440;

[0028] Figure 7 A block diagram of a control device for a robot according to some embodiments of the present disclosure is shown;

[0029] Figure 8 A block diagram of a control device for a robot according to other embodiments of the present disclosure is shown;

[0030] Figure 9 A block diagram of a control device for a robot according to some embodiments of the present disclosure is shown. Detailed Implementation

[0031] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0032] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0033] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0034] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0035] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0036] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0037] A key step in track slab fine-tuning is accurately inserting the tightening shaft of the robot performing the fine-tuning (also known as a track fine-tuning robot) into the tightening groove on the working tool (such as a fine-tuning claw) mounted on the track slab, thus securing the robot to the tool. This process is also called fine-tuning guidance (or vision guidance). Afterward, the robot can be controlled to control the working tool for fine-tuning the track slab.

[0038] In related technologies, it is usually necessary to manually align the robot's tightening shaft with the tightening groove of the working tool (such as a fine-tuning claw) to insert the robot's tightening shaft into the tightening groove on the working tool. However, this method relies on manual operation, is susceptible to factors such as fatigue and operating speed, and is difficult to meet the efficiency and accuracy requirements of track slab fine-tuning operations.

[0039] The inventors of this disclosure have discovered through research that the tightening groove of the work tool can be identified by using a deep learning model and a visual detection algorithm, so as to automatically insert the robot's tightening shaft into the tightening groove on the work tool for fixing, thereby performing track plate fine-tuning operations.

[0040] However, on the one hand, if a deep learning model is used alone to detect the tightening grooves of work tools, a large number of labeled data samples need to be collected to train the deep learning model. However, the data collection and labeling process is time-consuming and labor-intensive, making it difficult to meet the needs of rapid deployment and adjustment on the construction site, resulting in low work efficiency. On the other hand, if a visual algorithm is used alone to detect the tightening grooves of work tools, the actual track construction environment has variable lighting conditions, and the tightening grooves of work tools often have problems such as stains, wear, and reflections, resulting in complex background information of the tightening grooves in the acquired images. These interference factors seriously affect the image feature extraction effect of the visual detection algorithm on the tightening grooves, resulting in low recognition accuracy of the tightening grooves, and thus low work accuracy.

[0041] Therefore, relying solely on a single deep learning model or visual inspection algorithm is insufficient to accurately and efficiently identify the tightening grooves of the work tools. In other words, using either a deep learning model or a visual inspection algorithm alone is insufficient to meet the efficiency and accuracy requirements of track slab fine-tuning operations.

[0042] In view of this, this disclosure proposes a robot control method that can accurately and efficiently determine the position of the tightening groove of the working tool by combining a deep learning model and a visual detection algorithm. Then, the center line of the robot's tightening shaft is aligned with the center line of the tightening groove of the working tool to quickly and accurately insert the tightening shaft into the tightening groove, thereby improving the efficiency and accuracy of track slab fine-tuning operations.

[0043] Figure 1 A flowchart illustrating a robot control method according to some embodiments of the present disclosure is shown.

[0044] like Figure 1 As shown, in step 110, a first target image of the tightening groove on the working tool installed on the track plate is obtained.

[0045] Here, the tightening groove is used to accommodate and secure the robot's tightening shaft, so that the robot is fixedly installed with the working tool.

[0046] Figure 2 A perspective view of a working tool according to some embodiments of the present disclosure is shown. Figure 3 A top view of a working tool according to some embodiments of the present disclosure is shown.

[0047] like Figure 2 and Figure 3 As shown, Figure 2 A schematic 3D view of a fine-tuning jaw as the working tool is shown. Figure 3 A schematic top view of the working tool is shown, wherein the tightening groove 201 of the fine adjustment claw can be seen in the rectangular area of ​​the square in the top view.

[0048] Figure 4 A partial structural diagram of a robot according to some embodiments of the present disclosure is shown.

[0049] like Figure 4 As shown, Figure 4 The diagram schematically illustrates the structure of a robot's robotic arm, which has a tightening shaft mounted at its end. For example, to perform a track slab fine-tuning operation, a tightening shaft can be... Figure 4 The tightening shaft 301 shown is inserted. Figure 3 The tightening groove 201 shown is used to securely mount the robot's robotic arm and the fine-tuning claw. The robot's robotic arm can then be controlled to move the fine-tuning claw to fine-tune the track plate.

[0050] In some embodiments, a first target image of the tightening slot of the working tool can be captured by an image acquisition device mounted on the robot. For example, such as Figure 4 As shown, the image acquisition device can be mounted on the robot's robotic arm, with a tightening shaft installed at the end of the robotic arm. The image acquisition device can be a camera 302.

[0051] In some embodiments, an image acquisition device mounted on the robot can be controlled to move to a designated position (also known as the optimal shooting position) to acquire a first target image of the tightening groove of the work tool at the designated position. When the image acquisition device is located at the designated position, the center of the field of view of the image acquisition device coincides with the center of the tightening groove.

[0052] In this method, the field of view of the image acquisition device can fully cover the structural area of ​​the tightening groove, and the tightening groove, which is the object to be detected, is located in the center area of ​​the first target image. This helps to more accurately determine the position of the tightening groove of the working tool, improves the accuracy of visual guidance in the track slab fine-tuning operation, and thus helps to improve the accuracy of the track slab fine-tuning operation.

[0053] In some embodiments, the designated position can be pre-calibrated. For example, the designated position can be calibrated by the following procedure: controlling the image acquisition device to move towards the tightening groove of the work tool until the center of the field of view of the image acquisition device coincides with the center of the tightening groove, and recording the position coordinates (x1, y1, z1) of the image acquisition device in the robot's robotic arm coordinate system at this time, which is the position coordinate of the designated position.

[0054] It should be noted that the robot's coordinate system is a right-hand rule-compliant base coordinate system, which can be used to describe the position and orientation of the robot's joints and links. For example, the positive direction of the X-axis of the robot's coordinate system can be along the robot's direction of travel, the positive direction of the Y-axis can be the direction of rotation 90 degrees counterclockwise from the positive direction of the X-axis, and the positive direction of the Z-axis is perpendicular to the plane containing the X-axis and Y-axis.

[0055] In step 120, a deep learning model is used to process the first target image to determine the first target detection area corresponding to the tightening groove in the first target image.

[0056] In some embodiments, the deep learning model may include an object detection model, such as the YOLOv11 model.

[0057] In some embodiments, images of multiple tightening slots can be acquired using an image acquisition device mounted on the robot as training samples for training a deep learning model.

[0058] For example, the installation position of the image acquisition device can be pre-calibrated (as specified above) to keep its relative position to the tightening shaft fixed. This ensures that the first target image of the tightening groove can be acquired from the same position each time, thereby improving the stability of the data samples and helping to improve the training accuracy of the deep learning model.

[0059] In some embodiments, after processing the first target image using a deep learning model, preliminary position information of the tightening groove can be output, that is, preliminary positioning of the tightening groove is completed. For example, the preliminary position information may include at least one of the following: the position coordinates of the center point of the tightening groove in the first target image in the pixel coordinate system, the size of the tightening groove, and the rotation angle of the centerline of the tightening groove relative to the horizontal direction.

[0060] In some embodiments, the first target detection region can be represented by a detection box located in the image after the first target image has been processed by a deep learning model.

[0061] In step 130, the region of interest (ROI) image corresponding to the tightening groove is extracted from the first target image based on the first target detection region.

[0062] In some embodiments, a second target detection region can be determined by extending a predetermined number of pixels outwards from the boundary of the first target detection region. This second target detection region is then extracted from the first target image as a Region of Interest (ROI). For example, in the first target image, a predetermined number of pixels can be extended outwards from the boundary of the first target detection region, and the region containing this predetermined number of pixels and the first target detection region is determined as the second target detection region. Subsequently, the image corresponding to the second target detection region is cropped from the first target image and retained as a separate image to obtain the ROI image.

[0063] In this approach, the obtained ROI image contains only the tightening groove, which is the object to be detected, and some background information adjacent to the tightening groove. This greatly reduces the interference of complex background information, improves the extraction effect of image features of the tightening groove, thereby improving the recognition accuracy of the tightening groove and thus improving the accuracy of subsequent operations.

[0064] In step 140, the target location information of the tightening groove is determined based on the ROI image.

[0065] In some embodiments, the target location information includes at least one of the following: the position coordinates of the center point of the tightening groove in the first target image in pixel coordinates, the size of the tightening groove, and the rotation angle of the centerline of the tightening groove relative to the horizontal direction. This allows sufficient spatial location information of the tightening groove to be obtained, enabling accurate subsequent insertion of the tightening shaft into the tightening groove.

[0066] It can be understood that the preliminary position information of the tightening groove output after processing the first target image using a deep learning model is consistent with the target position information of the tightening groove determined based on the ROI image. The accuracy of the target position information of the tightening groove determined based on the ROI image is higher than the accuracy of the preliminary position information of the tightening groove output using the deep learning model.

[0067] In step 150, based on the target position information, the center line of the tightening shaft is aligned with the center line of the tightening groove, and the tightening shaft is inserted into the tightening groove.

[0068] In some embodiments, the target position information of the tightening groove determined based on the ROI image may include position information in the pixel coordinate system. According to the coordinate transformation relationship between the pixel coordinate system and the robot's manipulator coordinate system, the target position information can be converted into position information in the manipulator coordinate system. Then, based on the position information in the manipulator coordinate system, the centerline of the tightening shaft is aligned with the centerline of the tightening groove, and the tightening shaft is inserted into the tightening groove.

[0069] In the above embodiment, a first target image of the tightening groove on the working tool installed on the track plate is acquired. A deep learning model is then used to process this first target image to determine the first target detection region corresponding to the tightening groove within the first target image. Then, based on the first target detection region, the ROI image corresponding to the tightening groove is extracted. The target position information determined by the ROI image corresponding to the tightening groove is used to control the alignment of the centerline of the tightening shaft with the centerline of the tightening groove, and the tightening shaft is then inserted into the tightening groove.

[0070] In this approach, a deep learning model is first used to quickly identify the position of the tightening groove. Then, the ROI image corresponding to the tightening groove is extracted based on the first target detection area determined by the deep learning model. This significantly reduces the interference of background information in the ROI image, which is beneficial for accurately locating the position of the tightening groove to meet the efficiency and accuracy requirements of track slab fine-tuning operations.

[0071] In other words, the control method proposed in this embodiment uses the initial position of the tightening groove quickly captured by the deep learning model as a basis, combined with a visual detection algorithm based on ROI image localization. This eliminates the need for refined training of the deep learning model and reduces interference from background information on the visual detection algorithm's identification of the tightening groove. It combines the advantages of both methods, achieving rapid and accurate identification of the tightening groove, improving the efficiency and accuracy of track slab fine-tuning operations, and contributing to the overall quality of the engineering work. Furthermore, this method eliminates the need for manual operation, which helps improve the stability of the tightening groove positioning results, thereby enhancing operational stability.

[0072] The following examples illustrate how to obtain target location information from ROI images based on tightening grooves.

[0073] In some embodiments, the ROI image is converted to a grayscale image, and then a thresholding algorithm is used to separate the tightening groove from the background information in the grayscale image. The background information is then removed to obtain a second target image of the region where the tightening groove is located, and the target location information of the tightening groove is determined based on the second target image. For example, the Otsu thresholding algorithm can be used to separate the tightening groove from the background information in the grayscale image, so as to accurately extract the region where the tightening groove is located subsequently.

[0074] In this way, by performing grayscale processing on the ROI and using a threshold segmentation algorithm to separate the background information from the tightening groove, which is the object to be detected, the background information in the image can be removed more accurately. This further reduces the interference of background information on the detection of the tightening groove, thereby improving the positioning accuracy of the tightening groove and thus helping to improve the accuracy of the track slab fine-tuning operation.

[0075] In some embodiments, the edges of the tightening groove can be smoothed in the grayscale image, and background information can be removed after smoothing to obtain a second target image of the area where the tightening groove is located. For example, only the area where the tightening groove is located can be retained in the second target image.

[0076] In the above embodiments, edge smoothing processing can further eliminate pixel adhesion between the tightening groove and the background that may be caused by segmentation errors. Thus, removing background information after edge smoothing reduces the generation of false edges, making the extraction of the tightening groove area more accurate. This provides a more reliable image basis for subsequent acquisition of target location information, thereby further improving the positioning accuracy of the tightening groove.

[0077] In some embodiments, corner detection can be performed on the tightening groove based on the second target image to obtain corner information of the tightening groove, and then the target position information of the tightening groove can be determined based on the corner information of the tightening groove.

[0078] For example, a corner detection algorithm (such as the ORB algorithm) can be used to detect the corners of the tightening groove to obtain a set of corners. Line fitting is then performed based on the corner set to extract the contour of the tightening groove. Based on the contour of the tightening groove, at least one of the following is calculated in the pixel coordinate system: the position coordinates of the center point of the tightening groove, the size of the tightening groove, and the rotation angle of the centerline of the tightening groove relative to the horizontal direction, as the target position information.

[0079] In the above embodiments, considering that the corner points are located at the turning points of the object's outline, the intersection points of edges, or areas with significant texture changes, the outline of the tightening groove can be accurately determined by corner point detection, thereby accurately detecting the structure of the tightening groove, improving the positioning accuracy of the tightening groove, and further helping to improve the accuracy of the track slab fine-tuning operation.

[0080] The following examples illustrate how to align the centerline of the tightening shaft with the centerline of the tightening groove using the target position information of the tightening groove.

[0081] In some embodiments, the target location information includes the position coordinates of the center point of the tightening groove in the first target image in the pixel coordinate system.

[0082] In some embodiments, a first offset of the center of the tightening groove relative to the field of view center of the first target image in pixel coordinates can be determined based on the position coordinates of the center point of the tightening groove in the first target image in pixel coordinates. The first offset is then converted into a second offset in the robot's manipulator coordinate system. Based on the second offset, the centerline of the tightening shaft is aligned with the centerline of the tightening groove, and the tightening shaft is inserted into the tightening groove.

[0083] For example, in the pixel coordinate system, the position coordinates of the center point of the tightening groove in the first target image are (cx1, cy1), the position coordinates of the field of view center of the first target image are (cx2, cy2), and the first offset between the two is (offsetX1, offsetY1), where offsetX1 represents the offset between the two in the X-axis direction of the pixel coordinate system, offsetY2 represents the offset between the two in the Y-axis direction of the pixel coordinate system, offsetX1 = cx1 - cx2, offsetY1 = cy1 - cy2.

[0084] It is understood that those skilled in the art will recognize that a pixel coordinate system can be used to describe the position of each pixel in an image. For example, the origin of the pixel coordinate system can be located at the top left corner of the image, the horizontal direction extending from the origin to the right is the X-axis, and the vertical direction extending from the origin to the bottom is the Y-axis.

[0085] In some embodiments, the first offset in the pixel coordinate system can be converted into a second offset in the robotic arm coordinate system based on the transformation coefficient between the pixel coordinate system and the robotic arm coordinate system.

[0086] It should be understood that the second offset is the offset of the center of the tightening groove relative to the field of view center of the first target image in the robot arm coordinate system.

[0087] For example, the transformation coefficient between the pixel coordinate system and the robotic arm coordinate system is 'scale'. This transformation coefficient represents the correspondence between pixels and millimeters, that is, how many millimeters one pixel represents. The second offset after transformation is (offsetX2, offsetY2), where offsetX2 represents the offset of the two in the X-axis direction of the robotic arm coordinate system, and offsetY2 represents the offset of the two in the Y-axis direction of the robotic arm coordinate system. offsetX2 = scale * offsetX1, offsetY2 = scale * offsetY1.

[0088] In some embodiments, an initial offset of the center of the tightening shaft relative to the field of view center of the first target image in the robotic arm coordinate system is determined. Based on the initial offset and a second offset, a third offset of the center of the tightening shaft relative to the center of the tightening groove in the robotic arm coordinate system is determined. Then, based on the third offset, the centerline of the tightening shaft is aligned with the centerline of the tightening groove, and the tightening shaft is inserted into the tightening groove.

[0089] It should be noted that when the image acquisition device installed on the robot is moved to a designated position to obtain a first target image of the tightening groove of the working tool at the designated position, the field of view center of the first target image coincides with the field of view center of the image acquisition device.

[0090] Since the field of view center of an image acquisition device is usually located on its optical axis, the offset between the center of the tightening shaft and the field of view center of the image acquisition device can be characterized by the offset between the center line of the tightening shaft and the optical axis of the image acquisition device.

[0091] For example, the initial offset between the centerline of the tightening shaft and the optical axis of the image acquisition device can be predetermined. When the first target image is acquired using the image acquisition device mounted on the robot, this initial offset can be used as the initial offset of the center of the tightening shaft relative to the field of view center of the first target image in the robot arm coordinate system. No additional calculation is required during the control process, which helps to improve the efficiency of aligning the centerline of the tightening shaft with the centerline of the tightening groove, thereby improving the efficiency of the track plate fine-tuning operation.

[0092] In some embodiments, the initial offset of the center of the tightening shaft relative to the field of view center of the image acquisition device in the robot arm coordinate system can be determined and stored during the calibration of the specified position, so that the center line of the tightening shaft can be aligned with the center line of the tightening groove based on the initial offset and the determined target position information of the tightening groove.

[0093] For example, an image acquisition device (e.g., a camera) is controlled to move towards the tightening groove of a work tool until the center of the image acquisition device's field of view coincides with the center of the tightening groove. The position coordinates (x1, y1, z1) of the specified position are recorded. Then, the tightening shaft is controlled to move until it is inserted into the tightening groove. The position coordinates (x2, y2, z2) of the tightening shaft when it is inserted into the tightening groove are recorded.

[0094] The initial offset between the centerline of the tightening shaft and the optical axis of the image acquisition device in the two-dimensional plane is (offsetX3, offsetY3), where offsetX3 represents the offset of the centerline of the tightening shaft relative to the optical axis of the image acquisition device in the X-axis direction of the robot arm coordinate system, and offsetY3 represents the offset of the centerline of the tightening shaft relative to the optical axis of the image acquisition device in the Y-axis direction of the robot arm coordinate system, offsetX3=x1-x2, offsetY3=y1-y2.

[0095] Continuing with the example above, when controlling the image acquisition device mounted on the robot to move to a designated position to acquire a first target image of the tightening groove of the work tool at the designated position, the second offset of the center of the tightening groove relative to the field of view center of the first target image (i.e., the field of view center of the image acquisition device) in the robot arm coordinate system is (offsetX2, offsetY2), and the third offset of the center of the tightening shaft relative to the center of the tightening groove in the robot arm coordinate system is (offsetX4, offsetY4). Here, offsetX4 represents the offset of the center of the tightening shaft relative to the center of the tightening groove in the X-axis direction of the robot arm coordinate system, and offsetY4 represents the offset of the center of the tightening shaft relative to the center of the tightening groove in the Y-axis direction of the robot arm coordinate system. offsetX4 = offsetX2 + offsetX3, offsetY4 = offsetY2 + offsetY3.

[0096] In some embodiments, the tightening shaft is controlled to move a third offset component (e.g., offsetX4) along the extension direction of a first coordinate axis (e.g., the X-axis) of the robotic arm coordinate system, and the tightening shaft is also controlled to move a third offset component (e.g., offsetY4) along the extension direction of a second coordinate axis (e.g., the Y-axis), so that the centerline of the tightening shaft is aligned with the centerline of the tightening groove. After the centerline of the tightening shaft is aligned with the centerline of the tightening groove, the tightening shaft is inserted into the tightening groove.

[0097] In the above embodiment, firstly, the first offset between the center of the tightening groove and the center of the field of view of the first target image is calculated in the pixel coordinate system. Then, based on the transformation relationship between the robotic arm coordinate system and the pixel coordinate system, the first offset is mapped to the robotic arm coordinate system to determine the second offset between the center of the tightening groove and the center of the field of view of the first target image in the robotic arm coordinate system. Subsequently, using this second offset and the initial offset of the center of the tightening shaft relative to the center of the field of view of the first target image in the robotic arm coordinate system, the third offset of the center of the tightening shaft relative to the center of the tightening groove in the robotic arm coordinate system can be accurately determined.

[0098] In this way, the third offset accurately represents the spatial position deviation between the current tightening shaft and the tightening groove. Based on the third offset, the position of the robot's tightening shaft can be precisely adjusted so that the center line of the tightening shaft gradually approaches and eventually aligns with the center line of the tightening groove. This allows the robot's tightening shaft to be accurately inserted into the tightening groove on the working tool mounted on the track plate, which helps to perform track plate fine-tuning operations efficiently and accurately.

[0099] Figure 5 A flowchart illustrating a robot control method according to other embodiments of the present disclosure is shown. For example, Figure 5 The method shown can be used as Figure 1 This is a specific implementation of the method used in the code. Figure 5 In the method shown, the image acquisition device mounted on the robot is schematically an industrial camera.

[0100] like Figure 5 As shown, in step 410, the camera is moved to the photo-taking position. For example, the camera mounted on the robot is controlled to move to a designated position (i.e., the photo-taking position).

[0101] In some embodiments, the camera's focal length, exposure, gain, and light source brightness can be preset to make the captured target image of the tightening groove clearer.

[0102] For example, the camera can be precisely moved to a pre-stored location by acquiring the coordinates of a specified position, driven by the robot's programmable logic controller and through the cooperation of servo motors and guide rails. After the camera has moved, it is initialized according to the pre-set image acquisition parameters and triggered to acquire images.

[0103] In step 420, the camera takes a picture. For example, a first target image of the tightening groove is acquired using the camera. That is, a first target image is obtained by taking a picture using the camera.

[0104] In step 430, the model performs initial localization. For example, a deep learning model is used to perform initial localization of the tightening groove.

[0105] For example, a deep learning model is used to process the first target image to determine the first target detection region corresponding to the tightening groove in the first target image.

[0106] In some embodiments, the deep learning model can be trained before the tightening groove is initially located using the deep learning model.

[0107] As an example, taking the YOLOv11 deep learning model as an example, the training process of this deep learning model can include: acquiring multiple images of tightening grooves as a data sample set; cleaning the dataset and removing samples with poor image quality or damage; labeling the dataset (e.g., using X-Anylabeling to label the dataset and using rectangles with rotated angles to label the tightening grooves), and exporting the label file in YOLO format; dividing the dataset into training sample sets, validation sample sets, and test sample sets (e.g., in an 8:1:1 ratio); adding the dataset to the YOLOv11 model, configuring the dataset path and dataset categories, and training the YOLOv11 model; pixel scaling of the long and short sides of the images in the dataset (e.g., scaling the long side of the image to 640 pixels and padding the short side to 640 pixels), and then normalizing the images in the dataset to obtain the processed dataset as the training dataset. For example, the data normalization method is: X' = X / 255.0, where X is the input image and X' is the output image.

[0108] It is understandable that deep learning models have strong generalization capabilities; they only need to learn a small number of image features of tightening slots to accurately detect their positions in images of various tightening slots. However, the accuracy of the detection results of deep learning models largely depends on the quantity and quality of the training samples used during training. If a deep learning model is used alone for tightening slot localization, in order to improve the localization accuracy, a large number of tightening slot images need to be collected as a dataset, the dataset needs to be cleaned and labeled, and then the deep learning model needs to be trained repeatedly until the detection accuracy of the deep learning model meets the requirements. In this way, the required iteration cycle is long, which cannot meet the needs of rapid on-site adjustment. Therefore, this disclosure proposes to combine deep learning model detection with ROI image-based visual algorithms to improve the overall localization accuracy of tightening slots, thereby meeting the efficiency and accuracy requirements of track slab fine-tuning operations.

[0109] In step 440, a visual inspection algorithm is used for localization. For example, based on the preliminary localization results from a deep learning model, a visual inspection algorithm is used to further locate the tightening groove.

[0110] For example, the ROI image corresponding to the tightening groove is extracted from the first target image based on the first target detection region, and the target position information of the tightening groove is determined based on the ROI image.

[0111] In step 450, the location information is checked for validity. For example, it is determined whether the location is valid. That is, it is determined whether the acquired target location information is valid.

[0112] For example, considering that unforeseen circumstances such as interference from the construction site or errors in the deep learning model may cause garbled or invalid target location information, a validity assessment can be performed after the target location information is acquired to improve the reliability of subsequent control processes.

[0113] If the result of the judgment is that the obtained target location information is invalid, step 460 can be executed.

[0114] If the result of the judgment is that the obtained target location information is valid, step 470 can be executed.

[0115] In step 460, the image capture position is changed. For example, the camera's image capture position is changed. For example, the camera can be moved to another image capture position to re-acquire the first target image, and then the subsequent detection process is performed based on the re-acquired first target image. For example, the other image capture position can be a candidate position pre-calibrated according to the calibration method described above.

[0116] In step 470, coordinate system transformation occurs. For example, based on the target position information and the transformation relationship between the pixel coordinate system and the robotic arm coordinate system, alignment control is performed between the centerline of the tightening axis and the centerline of the tightening groove.

[0117] In some embodiments, the target position information includes the position coordinates of the center point of the tightening groove in the first target image in a pixel coordinate system. Based on the position coordinates of the center point of the tightening groove in the first target image in a pixel coordinate system, a first offset of the center of the tightening groove relative to the field of view center of the first target image in the pixel coordinate system can be determined. According to the transformation relationship between the pixel coordinate system and the robotic arm coordinate system, the first offset is converted into a second offset in the robotic arm coordinate system. Then, based on the second offset, the centerline of the tightening shaft is aligned with the centerline of the tightening groove, and the tightening shaft is inserted into the tightening groove.

[0118] In step 480, the tightening shaft is inserted into the tightening groove. For example, the tightening shaft is inserted into the tightening groove after the center line of the tightening shaft is aligned with the center line of the tightening groove.

[0119] about Figure 5 The steps of the method shown can be referred to in the previous text. Figures 1 to 4 The implementation methods in the relevant embodiments are similar, and for details, please refer to the descriptions in the relevant embodiments above, which will not be repeated here.

[0120] Figure 6 Flowcharts showing some implementations of steps 430 and 440 are provided.

[0121] like Figure 6 As shown, step 440 may include steps 441 to 446.

[0122] In step 430, model detection is performed. That is, a model detection operation is executed. For example, the model detection operation includes: using a deep learning model to perform preliminary positioning of the tightening groove.

[0123] In step 441, the Region of Interest (ROI) is extracted. That is, an ROI extraction operation is performed. For example, the ROI extraction operation includes: extracting the ROI image corresponding to the tightening groove from the first target image based on the first target detection region.

[0124] In step 442, threshold segmentation is performed. That is, a threshold segmentation operation is executed. For example, the threshold segmentation operation includes: converting the ROI image to a grayscale image, and then using a threshold segmentation algorithm to separate the tightening groove from the background information in the grayscale image.

[0125] In step 443, edge smoothing is performed. That is, an edge smoothing operation is performed. For example, the edge smoothing operation includes smoothing the edges of the tightening groove in the grayscale image.

[0126] In step 444, region filtering is performed. That is, a region filtering operation is executed. For example, the region filtering operation includes removing background information after smoothing to obtain a second target image of the region where the tightening groove is located. For example, only the region where the tightening groove is located may be retained in the second target image.

[0127] In step 445, corner detection is performed. That is, a corner detection operation is executed. For example, the corner detection operation includes: based on the second target image, performing corner detection on the tightening groove to obtain corner information of the tightening groove. For example, the corner information includes a set of corners of the tightening groove.

[0128] In step 446, rectangle detection is performed. That is, a rectangle detection operation is executed. For example, the rectangle detection operation includes: performing line fitting based on the set of corner points to extract the contour of the tightening groove, and calculating at least one of the following in the pixel coordinate system as target position information: the position coordinates of the center point of the tightening groove, the size of the tightening groove, and the rotation angle of the center line of the tightening groove relative to the horizontal direction.

[0129] about Figure 6 The steps of the process shown can be implemented similarly to those in the previous embodiments of obtaining target location information based on the ROI image of the tightening groove. For details, please refer to the description in the previous embodiments, which will not be repeated here.

[0130] Figure 7 A block diagram of a robot control device according to some embodiments of the present disclosure is shown.

[0131] like Figure 7As shown, the robot's first control device 600 includes an acquisition module 601, a processing module 602, an extraction module 603, a determination module 604, and a control module 605.

[0132] The acquisition module 601 can be configured to acquire a first target image of a tightening groove on a work tool mounted on a track plate, wherein the tightening groove is used to accommodate and fix the tightening shaft of the robot so as to fix the robot to the work tool.

[0133] The processing module 602 can be configured to process the first target image using a deep learning model to determine the first target detection region corresponding to the tightening groove in the first target image.

[0134] The extraction module 603 can be configured to extract the ROI image corresponding to the tightening groove from the first target image based on the first target detection region.

[0135] The determination module 604 can be configured to determine the target location information of the tightening groove based on the ROI image.

[0136] The control module 605 can be configured to align the centerline of the tightening shaft with the centerline of the tightening groove based on the target position information, and insert the tightening shaft into the tightening groove.

[0137] In some embodiments, the first control device 600 of the robot may further include the execution of the foregoing. Figures 1 to 6 Other modules of other operations in the relevant embodiments.

[0138] Figure 8 A block diagram of a robot control device according to other embodiments of the present disclosure is shown.

[0139] like Figure 8 As shown, the second control device 700 of the robot in this embodiment includes: a first memory 701 and a first processor 702 coupled to the first memory 701. The first processor 702 is configured to execute the robot control method in any embodiment of this disclosure based on the instructions stored in the first memory 701.

[0140] The first memory 701 may include, for example, system memory, fixed non-volatile storage medium, etc. The system memory stores, for example, an operating system, application programs, a boot loader, a database, and other programs.

[0141] Figure 9 A block diagram of a control device for a robot according to some embodiments of the present disclosure is shown.

[0142] like Figure 9As shown, the third control device 800 of the robot in this embodiment includes: a second memory 801 and a second processor 802 coupled to the second memory 801. The second processor 802 is configured to execute the robot control method in any of the foregoing embodiments based on instructions stored in the second memory 801.

[0143] The second memory 801 may include, for example, system memory, fixed non-volatile storage media, etc. The system memory may store, for example, the operating system, application programs, boot loader, and other programs.

[0144] The robot's third control device 800 may further include an input / output interface 803, a network interface 804, and a storage interface 805. These interfaces 803, 804, and 805, as well as the second memory 801 and the second processor 802, can be connected, for example, via a bus 806. The input / output interface 803 provides a connection interface for input / output devices such as displays, mice, keyboards, touchscreens, microphones, and speakers. The network interface 804 provides a connection interface for various networked devices. The storage interface 805 provides a connection interface for external storage devices such as SD cards and USB flash drives.

[0145] This disclosure also provides a robot, including a control device (e.g., a first control device 600 / a second control device 700 / a third control device 800) for the robot in any of the above embodiments.

[0146] This disclosure also provides a computer-readable storage medium including computer program instructions that, when executed by a processor, implement the method of any of the above embodiments.

[0147] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the method of any of the above embodiments.

[0148] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable non-transitory storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0149] The control technology solution for the robot according to this disclosure has now been described in detail. To avoid obscuring the concept of this disclosure, some details known in the art have not been described. Those skilled in the art can fully understand how to implement the technical solution disclosed herein based on the above description.

[0150] The methods and systems of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the specific order described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0151] While specific embodiments of this disclosure have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of this disclosure. Those skilled in the art should understand that modifications can be made to the above embodiments without departing from the scope and spirit of this disclosure. The scope of this disclosure is defined by the appended claims.

Claims

1. A control method of a robot, comprising: obtaining a first target image of a tightening slot of a work tool mounted on a track plate, the tightening slot being configured to accommodate and fix a tightening shaft of the robot so as to fixedly mount the robot and the work tool together; processing the first target image by using a deep learning model to determine a first target detection region corresponding to the tightening slot in the first target image and to determine preliminary position information of the tightening slot; extracting a region of interest (ROI) image corresponding to the tightening slot from the first target image based on the first target detection region by using a visual detection algorithm, and determining target position information of the tightening slot based on the ROI image; controlling a center line of the tightening shaft to align with a center line of the tightening slot based on the target position information, and inserting the tightening shaft into the tightening slot, wherein the extracting the ROI image corresponding to the tightening slot from the first target image based on the first target detection region comprises: extending a boundary of the first target detection region by a preset number of pixels to determine a second target detection region; extracting the second target detection region as the ROI image from the first target image; the determining the target position information of the tightening slot based on the ROI image comprises: converting the ROI image into a grayscale image; segmenting the tightening slot and background information in the grayscale image by using a threshold segmentation algorithm; removing the background information to obtain a second target image of a region where the tightening slot is located; determining the target position information based on the second target image, wherein the target position information comprises position coordinates of a center point of the tightening slot in the first target image in a pixel coordinate system, a size of the tightening slot, and a rotation angle of a center line of the tightening slot relative to a horizontal direction, and a precision of the target position information is higher than a precision of the preliminary position information.

2. The control method according to claim 1, wherein the removing the background information to obtain the second target image of the region where the tightening slot is located comprises: performing smoothing processing on an edge of the tightening slot in the grayscale image; removing the background information after the smoothing processing to obtain the second target image.

3. The control method according to claim 1, wherein the determining the target position information based on the second target image comprises: performing corner point detection on the tightening slot based on the second target image to obtain corner point information of the tightening slot; determining the target position information based on the corner point information of the tightening slot.

4. The control method according to any one of claims 1 to 3, wherein the obtaining the first target image of the tightening slot of the work tool mounted on the track plate comprises: controlling an image acquisition device mounted on the robot to move to a specified position to obtain the first target image at the specified position, wherein a center of a field of view of the image acquisition device coincides with a center of the tightening slot when the image acquisition device is at the specified position.

5. The control method according to any one of claims 1 to 3, wherein the target position information comprises position coordinates of a center point of the tightening slot in the first target image in a pixel coordinate system, The control, based on the target position information, of the center line of the tightening shaft to align with the center line of the tightening slot and the insertion of the tightening shaft into the tightening slot comprises: determining a first offset of the center of the tightening slot relative to the center of the field of view of the first target image in the pixel coordinate system based on the position coordinates; converting the first offset into a second offset in the robot arm coordinate system; controlling, based on the second offset, the center line of the tightening shaft to align with the center line of the tightening slot and inserting the tightening shaft into the tightening slot.

6. The control method according to claim 5, wherein The control, based on the second offset, of the center line of the tightening shaft to align with the center line of the tightening slot and the insertion of the tightening shaft into the tightening slot comprises: determining an initial offset of the center of the tightening shaft relative to the center of the field of view of the first target image in the robot arm coordinate system; determining, based on the initial offset and the second offset, a third offset of the center of the tightening shaft relative to the center of the tightening slot in the robot arm coordinate system; controlling, based on the third offset, the center line of the tightening shaft to align with the center line of the tightening slot and inserting the tightening shaft into the tightening slot.

7. A control device of a robot, comprising: an acquisition module configured to acquire a first target image of a tightening slot installed on a work tool of a track plate, the tightening slot being used to accommodate a tightening shaft of the robot so that the robot is fixed with the work tool; a processing module configured to process the first target image by using a deep learning model to determine a first target detection region corresponding to the tightening slot in the first target image and to determine preliminary position information of the tightening slot; an extraction module configured to extract, based on the first target detection region, a region of interest (ROI) image corresponding to the tightening slot from the first target image by using a visual detection algorithm; a determination module configured to determine target position information of the tightening slot based on the ROI image; a control module configured to control, based on the target position information, the center line of the tightening shaft to align with the center line of the tightening slot and to insert the tightening shaft into the tightening slot, wherein the extraction module is configured to expand a boundary of the first target detection region by a preset number of pixels to determine a second target detection region and to extract the second target detection region as the ROI image from the first target image, the determination module is configured to convert the ROI image into a grayscale image, separate the tightening slot from background information in the grayscale image by a threshold segmentation algorithm, remove the background information to obtain a second target image of a region where the tightening slot is located, and determine the target position information based on the second target image, the target position information comprises position coordinates of a center point of the tightening slot in the first target image in a pixel coordinate system, a size of the tightening slot, and a rotation angle of the center line of the tightening slot relative to a horizontal direction, and the accuracy of the target position information is higher than that of the preliminary position information.

8. A control device of a robot, comprising: a memory; and a processor coupled to the memory, the processor configured to perform the control method of any one of claims 1-6 based on instructions stored in the memory.

9. A robot, comprising: the control device of claim 7 or 8.

10. A computer readable storage medium having stored thereon computer instructions that, when executed by a processor, implement the control method of any one of claims 1-6.

11. A computer program product comprising instructions which, when executed by a processor, cause the processor to perform the control method of any one of claims 1-6.

Images