Visual guidance method and device for fine adjustment of track plate and medium

Abstract

The invention relates to a visual guidance method and device for fine adjustment of a track board and a medium, and relates to the technical field of control. The control method comprises the steps that a first target image of a tightening groove formed in an operation tool installed on a track plate is obtained, and the tightening groove is used for containing and fixing a tightening shaft of a robot so that the robot and the operation tool can be fixedly installed; processing the first target image by using a deep learning model to determine a first target detection area corresponding to the tightening groove in the first target image; extracting a region-of-interest ROI image corresponding to the tightening slot from the first target image based on the first target detection region; determining target position information of the tightening groove based on the ROI image; and based on the target position information, the center line of the tightening shaft is controlled to be aligned with the center line of the tightening groove, and the tightening shaft is inserted into the tightening groove. Therefore, the accuracy of the fine adjustment operation of the track plate is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of control technology, and in particular to a visual guidance method, device and medium for fine-tuning a track plate. Background Art

[0002] In rail transit construction, track plate fine-tuning is a critical step in ensuring safe and smooth train operation. With the continued growth of high-speed rail operating mileage and increasing requirements for refined line management, the demand for high-precision, automated track plate fine-tuning is becoming increasingly urgent. Summary of the Invention

[0003] The inventors of the present disclosure have discovered that the above-mentioned related art has the following problem: the accuracy of the track plate fine adjustment operation is low.

[0004] In order to solve the above problems, the embodiments of the present disclosure provide the following solutions.

[0005] According to some embodiments of the present disclosure, a robot control method is provided, including: acquiring a first target image of a tightening groove on a working tool installed on a track plate, the tightening groove being used to accommodate and fix a tightening shaft of the robot so that the robot and the working tool are fixedly installed; processing the first target image using a deep learning model to determine a first target detection area corresponding to the tightening groove in the first target image; extracting a region of interest (ROI) image corresponding to the tightening groove from the first target image based on the first target detection area; determining target position information of the tightening groove based on the ROI image; and controlling a center line of the tightening shaft to align with a center line of the tightening groove based on the target position information, and inserting the tightening shaft into the tightening groove.

[0006] In some embodiments, the target position information includes at least one of the position coordinates of the center point of the tightening groove in the first target image in a pixel coordinate system, the size of the tightening groove, and the rotation angle of the center line of the tightening groove relative to the horizontal direction.

[0007] In some embodiments, extracting the ROI image corresponding to the tightening groove from the first target image based on the first target detection area includes: expanding a preset number of pixels around the boundary of the first target detection area to determine a second target detection area; and extracting the second target detection area as the ROI image from the first target image.

[0008] In some embodiments, determining the target position information of the tightening groove based on the ROI image includes: converting the ROI image into a grayscale image; separating the tightening groove from the background information in the grayscale image through a threshold segmentation algorithm; removing the background information to obtain a second target image of the area where the tightening groove is located; and determining the target position information based on the second target image.

[0009] In some embodiments, removing the background information to obtain a second target image of the area where the tightening groove is located includes: smoothing the edge of the tightening groove in the grayscale image; and removing the background information after the smoothing to obtain the second target image.

[0010] In some embodiments, determining the target position information based on the second target image includes: performing corner point detection on the tightening groove based on the second target image to obtain corner point information of the tightening groove; and determining the target position information based on the corner point information of the tightening groove.

[0011] In some embodiments, obtaining a first target image of a tightening groove on a working tool installed on a track plate includes: controlling an image acquisition device installed on the robot to move to a specified position to obtain the first target image at the specified position, wherein, when the image acquisition device is located at the specified position, the center of the field of view of the image acquisition device coincides with the center of the tightening groove.

[0012] In some embodiments, the target position information includes the position coordinates of the center point of the tightening groove in the first target image in the pixel coordinate system, and based on the target position information, controlling the center line of the tightening shaft to be aligned with the center line of the tightening groove and inserting the tightening shaft into the tightening groove includes: determining a first offset of the center of the tightening groove relative to the field of view center of the first target image in the pixel coordinate system based on the position coordinates; converting the first offset into a second offset in the robot's robotic arm coordinate system; based on the second offset, controlling the center line of the tightening shaft to be aligned with the center line of the tightening groove, and inserting the tightening shaft into the tightening groove.

[0013] In some embodiments, controlling the center line of the tightening shaft to be aligned with the center line of the tightening slot based on the second offset and inserting the tightening shaft into the tightening slot includes: determining an initial offset of the center of the tightening shaft relative to the center of the field of view of the first target image in the robotic arm coordinate system; determining a third offset of the center of the tightening shaft relative to the center of the tightening slot in the robotic arm coordinate system based on the initial offset and the second offset; controlling the center line of the tightening shaft to be aligned with the center line of the tightening slot based on the third offset, and inserting the tightening shaft into the tightening slot.

[0014] According to other embodiments of the present disclosure, a control device of a robot is provided, including: an acquisition module configured to acquire a first target image of a tightening groove on a working tool mounted on a track plate, the tightening groove being used to accommodate a tightening shaft of the robot so that the robot is fixed to the working tool; a processing module configured to process the first target image using a deep learning model to determine a first target detection area corresponding to the tightening groove in the first target image; an extraction module configured to extract a region of interest (ROI) image corresponding to the tightening groove from the first target image based on the first target detection area; a determination module configured to determine target position information of the tightening groove based on the ROI image; and a control module configured to control the center line of the tightening shaft to be aligned with the center line of the tightening groove based on the target position information, and to insert the tightening shaft into the tightening groove.

[0015] According to some further embodiments of the present disclosure, a control device for a robot is provided, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute the control method in any one of the above embodiments based on instructions stored in the memory device.

[0016] According to some further embodiments of the present disclosure, a robot is provided, comprising the control device according to any one of the above embodiments.

[0017] According to some further embodiments of the present disclosure, a computer-readable storage medium is provided, on which computer instructions are stored. When the instructions are executed by a processor, the control method in any of the above embodiments is implemented.

[0018] According to some further embodiments of the present disclosure, a computer program product is provided, comprising instructions, which, when executed by a processor, enable the processor to perform the control method according to any one of the above embodiments.

[0019] In the above embodiment, based on the preliminary position of the tightening groove quickly captured by the deep learning model and combined with the visual detection algorithm based on ROI image positioning, there is no need to conduct fine-tuning training of the deep learning model, and the interference caused by background information on the use of the visual detection algorithm to identify the tightening groove is reduced, thereby combining the advantages of the two methods to achieve the technical effect of quickly and accurately identifying the tightening groove, and improving the efficiency and accuracy of the track plate fine-tuning operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0021] The present disclosure can be more clearly understood from the following detailed description with reference to the accompanying drawings, in which: Figure 1 A flowchart showing a method for controlling a robot according to some embodiments of the present disclosure is shown; Figure 2 A perspective view showing a work tool according to some embodiments of the present disclosure; Figure 3 shows a top view of a work tool according to some embodiments of the present disclosure; Figure 4 A partial structural diagram of a robot according to some embodiments of the present disclosure is shown; Figure 5 A flowchart showing a method for controlling a robot according to other embodiments of the present disclosure; Figure 6 A flowchart illustrating some implementations of steps 430 and 440; Figure 7 A block diagram illustrating a control device of a robot according to some embodiments of the present disclosure; Figure 8 A block diagram showing a control device of a robot according to other embodiments of the present disclosure; Figure 9 A block diagram illustrating a control device of a robot according to further embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0022] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure.

[0023] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0024] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0025] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0026] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0027] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0028] A key step in track slab fine-tuning is accurately inserting the tightening shaft of the track slab fine-tuning robot (also known as the track fine-tuning robot) into the tightening slot of a working tool (such as a fine-tuning claw) mounted on the track slab, securing the robot and the tool. This process is also known as fine-tuning guidance (or vision-guided) operation. The robot can then be controlled to control the working tool to fine-tune the track slab.

[0029] In the prior art, manual alignment of the robot's tightening shaft with the tightening slot of a work tool (such as a fine-tuning claw) is typically required to insert the robot's tightening shaft into the tightening slot on the work tool. However, this manual approach is susceptible to factors such as fatigue and operating speed, making it difficult to meet the efficiency and accuracy requirements of track slab fine-tuning operations.

[0030] The inventors of the present disclosure have discovered through research that the tightening groove of the working tool can be identified with the help of a deep learning model and a visual detection algorithm, so as to automatically insert the tightening shaft of the robot into the tightening groove on the working tool for fixation, thereby performing the track plate fine-tuning operation.

[0031] However, on the one hand, if a deep learning model is used alone to detect the tightening grooves of work tools, a large number of labeled data samples need to be collected to train the deep learning model. However, the collection and labeling of data samples are time-consuming and labor-intensive, making it difficult to meet the needs of rapid deployment and adjustment at the construction site, resulting in low work efficiency. On the other hand, if a visual algorithm is used alone to detect the tightening grooves of work tools, the background information of the tightening grooves in the collected images is complex due to the variable lighting conditions on site in the actual track construction environment and the frequent stains, wear, and reflections on the surface of the tightening grooves of the work tools. These interference factors seriously affect the visual detection algorithm's ability to extract the image features of the tightening grooves, resulting in low recognition accuracy of the tightening grooves, and thus low work accuracy.

[0032] Therefore, relying solely on a single deep learning model or visual inspection algorithm is insufficient to accurately and efficiently identify the tightening groove of the work tool. In other words, using a single deep learning model or visual inspection algorithm alone is difficult to meet the efficiency and accuracy requirements of track plate fine-tuning operations.

[0033] In view of this, the present disclosure proposes a robot control method, which can combine deep learning models and visual detection algorithms to accurately and efficiently determine the position of the tightening groove of the working tool, and then control the center line of the robot's tightening shaft and the center line of the tightening groove of the working tool to align, so as to quickly and accurately insert the tightening shaft into the tightening groove, thereby improving the efficiency and accuracy of the track plate fine-tuning operation.

[0034] Figure 1 A flowchart illustrating a method for controlling a robot according to some embodiments of the present disclosure is shown.

[0035] like Figure 1 As shown, in step 110 , a first target image of a tightening groove on a working tool installed on a track plate is acquired.

[0036] Here, the tightening groove is used to accommodate and fix the tightening shaft of the robot, so as to achieve a fixed installation between the robot and the working tool.

[0037] Figure 2 A perspective view of a work tool according to some embodiments of the present disclosure is shown. Figure 3 A top view of a work tool according to some embodiments of the present disclosure is shown.

[0038] like Figure 2 and Figure 3 As shown, Figure 2 A perspective view schematically showing a working tool as a fine adjustment claw, Figure 3 The working tool is schematically shown in a top view of a fine adjustment jaw, wherein the tightening groove 201 of the fine adjustment jaw can be seen as a square rectangular area in the top view.

[0039] Figure 4 A partial structural diagram of a robot according to some embodiments of the present disclosure is shown.

[0040] like Figure 4 As shown, Figure 4 The schematic diagram shows the structure of the robot's mechanical arm, which has a tightening shaft installed at the end of the mechanical arm. For example, to perform the track plate fine adjustment operation, the Figure 4 The tightening shaft 301 shown is inserted into Figure 3 The tightening groove 201 is shown so that the robot's mechanical arm and the fine adjustment claw are fixedly mounted. The fine adjustment claw can then be moved by controlling the robot's mechanical arm to fine-tune the track plate.

[0041] In some embodiments, the first target image of the tightening groove of the working tool can be captured by an image acquisition device installed on the robot. Figure 4 As shown, the image acquisition device can be installed on the robot's mechanical arm, and a tightening shaft is installed at the end of the mechanical arm. The image acquisition device can be a camera 302.

[0042] In some embodiments, an image capture device mounted on the robot can be controlled to move to a designated position (also known as a sweet spot) to capture a first target image of a tightening groove of a work tool at the designated position. When the image capture device is at the designated position, the center of the image capture device's field of view coincides with the center of the tightening groove.

[0043] In this way, the shooting field of view of the image acquisition device can fully cover the structural area of ​​the tightening groove, and can make the tightening groove as the object to be detected located in the central area of ​​the first target image acquired, which helps to more accurately determine the position of the tightening groove of the working tool in the subsequent process, thereby improving the accuracy of visual guidance in the track plate fine-tuning operation, thereby helping to improve the accuracy of the track plate fine-tuning operation.

[0044] In some embodiments, the designated position can be pre-calibrated. For example, the designated position can be calibrated according to the following process: the image acquisition device is controlled to move toward the tightening slot of the working tool until the center of the image acquisition device's field of view coincides with the center of the tightening slot, and the position coordinates (x1, y1, z1) of the image acquisition device at this time in the robot's mechanical arm coordinate system are recorded, i.e., the position coordinates of the designated position.

[0045] It should be noted that the robot's arm coordinate system is a base coordinate system that conforms to the right-hand rule and can be used to describe the position and posture of each joint and link of the robot arm. For example, the positive direction of the robot's X-axis can be along the robot's direction of travel, the positive direction of the Y-axis can be 90 degrees counterclockwise from the positive direction of the X-axis, and the positive direction of the Z-axis is perpendicular to the plane containing the X- and Y-axes.

[0046] In step 120 , the first target image is processed using a deep learning model to determine a first target detection area corresponding to the tightening groove in the first target image.

[0047] In some embodiments, the deep learning model may include an object detection model, such as a YOLOV11 model.

[0048] In some embodiments, images of multiple tightening grooves can be obtained by an image acquisition device installed on the robot as training samples for training the deep learning model.

[0049] For example, the image acquisition device's installation position (such as the designated location described above) can be pre-calibrated to maintain a fixed relative position to the tightening shaft. This ensures that the first target image of the tightening slot is captured at the same location each time, improving the stability of the data sample and ultimately boosting the training accuracy of the deep learning model.

[0050] In some embodiments, after processing the first target image using a deep learning model, preliminary position information of the tightening groove can be output, i.e., preliminary positioning of the tightening groove can be completed. For example, the preliminary position information may include at least one of the position coordinates of the center point of the tightening groove in the first target image in a pixel coordinate system, the size of the tightening groove, and the rotation angle of the centerline of the tightening groove relative to the horizontal direction.

[0051] In some embodiments, the first target detection area can be represented by a detection box located in the image after the first target image is processed by a deep learning model.

[0052] In step 130 , a region of interest (ROI) image corresponding to the tightening slot is extracted from the first target image based on the first target detection area.

[0053] In some embodiments, a second target detection region can be determined by expanding the first target detection region by a preset number of pixels in all directions based on the boundary of the first target detection region. The second target detection region can then be extracted from the first target image as an ROI image. For example, in the first target image, a preset number of pixels can be expanded from the boundary of the first target detection region in all directions. The region encompassing the preset number of pixels and the first target detection region is then determined as the second target detection region. Subsequently, the image corresponding to the second target detection region is cropped from the first target image and retained as a separate image to obtain the ROI image.

[0054] In this way, the obtained ROI image only contains the tightening groove as the object to be detected and some background information adjacent to the tightening groove, which greatly reduces the interference of complex background information and improves the extraction effect of the image features of the tightening groove, thereby improving the recognition accuracy of the tightening groove and further improving the accuracy of subsequent operations.

[0055] In step 140 , target position information of the tightening groove is determined based on the ROI image.

[0056] In some embodiments, the target position information includes at least one of the position coordinates of the center point of the tightening slot in the first target image in a pixel coordinate system, the dimensions of the tightening slot, and the rotation angle of the centerline of the tightening slot relative to the horizontal direction. In this way, sufficient spatial position information of the tightening slot can be obtained to facilitate subsequent accurate insertion of the tightening shaft into the tightening slot.

[0057] It can be understood that the preliminary position information of the tightening groove output after processing the first target image using the deep learning model is consistent with the target position information of the tightening groove determined based on the ROI image. The target position information of the tightening groove determined based on the ROI image is more accurate than the preliminary position information of the tightening groove output using the deep learning model.

[0058] In step 150 , based on the target position information, the center line of the tightening shaft is controlled to be aligned with the center line of the tightening groove, and the tightening shaft is inserted into the tightening groove.

[0059] In some embodiments, the target position information of the tightening slot determined based on the ROI image may include position information in a pixel coordinate system. Based on a coordinate conversion relationship between the pixel coordinate system and a robot arm coordinate system, the target position information can be converted into position information in the robot arm coordinate system. The centerline of the tightening shaft is then aligned with the centerline of the tightening slot based on the position information in the robot arm coordinate system, and the tightening shaft is inserted into the tightening slot.

[0060] In the above embodiment, a first target image of a tightening groove on a working tool mounted on a track plate is acquired and processed using a deep learning model to determine a first target detection region corresponding to the tightening groove within the first target image. A ROI image corresponding to the tightening groove is then extracted based on the first target detection region. The centerline of the tightening shaft is aligned with the centerline of the tightening groove using the target position information determined from the ROI image of the tightening groove, and the tightening shaft is then inserted into the tightening groove.

[0061] In this way, the deep learning model is first used to quickly and preliminarily identify the position of the tightening groove, and then the ROI image corresponding to the tightening groove is extracted based on the first target detection area determined by the deep learning model. This significantly reduces the background information interference in the ROI image, which is conducive to accurately locating the position of the tightening groove to meet the efficiency and accuracy requirements of the track plate fine-tuning operation.

[0062] In other words, the control method proposed in the embodiment of the present disclosure uses the initial position of the tightening groove quickly captured by the deep learning model as the basis, combined with the visual detection algorithm based on ROI image positioning. This eliminates the need for detailed training of the deep learning model and reduces the interference caused by background information on the use of the visual detection algorithm to identify the tightening groove. This combines the advantages of both methods, achieving the technical effect of quickly and accurately identifying the tightening groove, improving the efficiency and accuracy of the track plate fine-tuning operation, and contributing to the overall quality of the engineering operation. Moreover, this method does not require manual operation, which helps to improve the stability of the tightening groove positioning results, thereby improving the stability of the operation.

[0063] The following describes, in conjunction with some embodiments, a method of obtaining target position information based on the ROI image of the tightening groove.

[0064] In some embodiments, the ROI image is converted into a grayscale image, and then a threshold segmentation algorithm is used to separate the tightening groove from the background information in the grayscale image. This background information is then removed to obtain a second target image of the tightening groove region. Based on this second target image, the target position information of the tightening groove is determined. For example, the Otsu threshold segmentation algorithm can be used to separate the tightening groove from the background information in the grayscale image, allowing for subsequent accurate extraction of the tightening groove region.

[0065] In this way, by performing grayscale processing on the ROI and using the threshold segmentation algorithm to separate the background information from the tightening groove as the object to be detected, the background information in the image can be removed more accurately, further reducing the interference of the background information on the tightening groove detection, thereby improving the positioning accuracy of the tightening groove, and further helping to improve the accuracy of the track plate fine-tuning operation.

[0066] In some embodiments, the edge of the tightening groove may be smoothed in the grayscale image, and background information may be removed after smoothing to obtain a second target image of the area where the tightening groove is located. For example, only the area where the tightening groove is located may be retained in the second target image.

[0067] In the above-mentioned embodiment, edge smoothing can further eliminate pixel adhesion between the tightening groove and the background, which may be caused by segmentation errors. Thus, removing background information after edge smoothing can reduce the generation of false edges, making the extraction of the tightening groove area more accurate, providing a more reliable image foundation for subsequent acquisition of target position information, and further improving the positioning accuracy of the tightening groove.

[0068] In some embodiments, corner point detection may be performed on the tightening groove based on the second target image to obtain corner point information of the tightening groove, and then target position information of the tightening groove may be determined based on the corner point information of the tightening groove.

[0069] For example, a corner detection algorithm (such as the ORB algorithm) can be used to detect the corners of the tightening groove to obtain a set of corner points of the tightening groove. Linear fitting is performed based on the set of corner points to extract the contour of the tightening groove. Based on the contour of the tightening groove, at least one of the position coordinates of the center point of the tightening groove, the dimensions of the tightening groove, and the rotation angle of the center line of the tightening groove relative to the horizontal direction is calculated in a pixel coordinate system as target position information.

[0070] In the above embodiment, considering that the corner points are located at the turning points of the object contour, the edge intersection points or the areas with significant texture changes, the contour of the tightening groove can be accurately determined through corner point detection, thereby accurately detecting the structure of the tightening groove, improving the positioning accuracy of the tightening groove, and further helping to improve the accuracy of the track plate fine-tuning operation.

[0071] Next, a method of aligning the center line of the tightening shaft with the center line of the tightening groove by using the target position information of the tightening groove is exemplified in conjunction with some embodiments.

[0072] In some embodiments, the target position information includes the position coordinates of the center point of the tightening slot in the first target image in a pixel coordinate system.

[0073] In some embodiments, a first offset of the center of the tightening slot relative to the center of the field of view of the first target image in the pixel coordinate system can be determined based on the position coordinates of the center point of the tightening slot in the first target image in the pixel coordinate system. The first offset is converted into a second offset in the robot's manipulator coordinate system. Based on the second offset, the centerline of the tightening shaft is then aligned with the centerline of the tightening slot, and the tightening shaft is inserted into the tightening slot.

[0074] For example, in the pixel coordinate system, the position coordinates of the center point of the tightening slot in the first target image are (cx1, cy1), the position coordinates of the field of view center of the first target image are (cx2, cy2), and the first offset between the two is (offsetX1, offsetY1), where offsetX1 represents the offset of the two in the X-axis direction of the pixel coordinate system, and offsetY2 represents the offset of the two in the Y-axis direction of the pixel coordinate system, offsetX1 = cx1-cx2, offsetY1 =cy1-cy2.

[0075] It is understood that those skilled in the art are aware that a pixel coordinate system can be used to describe the position of each pixel in an image. For example, the origin of the pixel coordinate system can be located at the upper left corner of the image, with the X-axis extending horizontally from left to right with the origin as the starting point, and the Y-axis extending vertically from top to bottom with the origin as the starting point.

[0076] In some embodiments, the first offset in the pixel coordinate system may be converted into a second offset in the robotic arm coordinate system according to a conversion coefficient between the pixel coordinate system and the robotic arm coordinate system.

[0077] It should be understood that the second offset is the offset of the center of the tightening slot relative to the center of the field of view of the first target image in the robot arm coordinate system.

[0078] For example, the conversion factor between the pixel coordinate system and the robot coordinate system is scale, which represents the correspondence between pixels and millimeters, that is, how many millimeters one pixel represents. The second offset after conversion is (offsetX2, offsetY2), where offsetX2 represents the offset between the two along the X-axis of the robot coordinate system, and offsetY2 represents the offset between the two along the Y-axis of the robot coordinate system. OffsetX2 = scale * offsetX1, and offsetY2 = scale * offsetY1.

[0079] In some embodiments, an initial offset of the center of the tightening shaft relative to the center of the field of view of the first target image in the robot coordinate system is determined. Based on the initial offset and the second offset, a third offset of the center of the tightening shaft relative to the center of the tightening slot in the robot coordinate system is determined. Subsequently, based on the third offset, the centerline of the tightening shaft is aligned with the centerline of the tightening slot, and the tightening shaft is inserted into the tightening slot.

[0080] It should be noted that when controlling the image acquisition device installed on the robot to move to a specified position to obtain a first target image of the tightening groove of the working tool at the specified position, the field of view center of the first target image coincides with the field of view center of the image acquisition device.

[0081] Since the center of the field of view of the image acquisition device is usually located on its optical axis, the offset between the center line of the tightening shaft and the optical axis of the image acquisition device can be used to characterize the offset between the center of the tightening shaft and the center of the field of view of the image acquisition device.

[0082] For example, the initial offset between the center line of the tightening shaft and the optical axis of the image acquisition device can be determined in advance, so that when the first target image is acquired using the image acquisition device installed on the robot, the initial offset can be used as the initial offset of the center of the tightening shaft relative to the center of the field of view of the first target image in the robot arm coordinate system, without the need for additional calculations during the control process, thereby helping to improve the efficiency of controlling the alignment of the center line of the tightening shaft with the center line of the tightening groove, thereby improving the efficiency of the track plate fine-tuning operation.

[0083] In some embodiments, the initial offset of the center of the tightening shaft relative to the center of the field of view of the image acquisition device in the robotic arm coordinate system can be determined and stored during the process of calibrating the specified position, so that the center line of the tightening shaft can be controlled to be aligned with the center line of the tightening groove based on the initial offset and the determined target position information of the tightening groove.

[0084] For example, an image acquisition device (e.g., a camera) is controlled to move toward the tightening slot of a work tool until the center of the image acquisition device's field of view coincides with the center of the tightening slot. The coordinates (x1, y1, z1) of the designated position are recorded. The tightening shaft is then controlled to move until it is inserted into the tightening slot. The position coordinates (x2, y2, z2) of the tightening shaft after the tightening shaft is inserted into the tightening slot are recorded.

[0085] The initial offset between the center line of the tightening shaft and the optical axis of the image acquisition device in the two-dimensional plane is (offsetX3, offsetY3), where offsetX3 represents the offset of the center line of the tightening shaft relative to the optical axis of the image acquisition device in the X-axis direction of the robot arm coordinate system, and offsetY3 represents the offset of the center line of the tightening shaft relative to the optical axis of the image acquisition device in the Y-axis direction of the robot arm coordinate system. OffsetX3 = x1 - x2, offsetY3 = y1 - y2.

[0086] Continuing with the above example, when controlling the image acquisition device mounted on the robot to move to a specified position to obtain a first target image of the tightening groove of the working tool at the specified position, the second offset of the center of the tightening groove relative to the field of view center of the first target image (i.e., the field of view center of the image acquisition device) in the robot coordinate system is (offsetX2, offsetY2), and the third offset of the center of the tightening shaft relative to the center of the tightening groove in the robot coordinate system is (offsetX4, offsetY4), where offsetX4 represents the offset of the center of the tightening shaft relative to the center of the tightening groove in the X-axis direction of the robot coordinate system, and offsetY4 represents the offset of the center of the tightening shaft relative to the center of the tightening groove in the Y-axis direction of the robot coordinate system, offsetX4 = offsetX2 + offsetX3, and offsetY4 = offsetY2+ offsetY3.

[0087] In some embodiments, the tightening shaft is controlled to move along the extension direction of a first coordinate axis (e.g., the X-axis) of the robot arm coordinate system by a component of a third offset in that direction (e.g., offsetX4), and the tightening shaft is controlled to move along the extension direction of a second coordinate axis (e.g., the Y-axis) by a component of the third offset in that direction (e.g., offsetY4) so ​​that the centerline of the tightening shaft is aligned with the centerline of the tightening slot. After the centerlines of the tightening shaft and the tightening slot are aligned, the tightening shaft is inserted into the tightening slot.

[0088] In the above embodiment, a first offset between the center of the tightening slot and the center of the field of view of the first target image is first calculated in the pixel coordinate system. Subsequently, based on the conversion relationship between the robot coordinate system and the pixel coordinate system, the first offset is mapped to the robot coordinate system to determine a second offset between the center of the tightening slot and the center of the field of view of the first target image in the robot coordinate system. Subsequently, using this second offset and the initial offset of the center of the tightening shaft relative to the center of the field of view of the first target image in the robot coordinate system, a third offset of the center of the tightening shaft relative to the center of the tightening slot in the robot coordinate system can be accurately determined.

[0089] In this way, the third offset accurately represents the spatial position deviation between the current tightening shaft and the tightening groove. Based on the third offset, the tightening shaft of the robot can be precisely adjusted so that the center line of the tightening shaft gradually approaches and eventually aligns with the center line of the tightening groove, thereby enabling the tightening shaft of the robot to be accurately inserted into the tightening groove on the working tool installed on the track plate, which helps to perform track plate fine-tuning operations efficiently and accurately.

[0090] Figure 5 A flow chart showing a method for controlling a robot according to some other embodiments of the present disclosure is shown. For example, Figure 5 The method shown can be used as Figure 1 A specific implementation of the method in is executed. Figure 5 In the method shown, it is schematically shown that the image acquisition device installed on the robot is an industrial camera.

[0091] like Figure 5 As shown, in step 410, the robot moves to a photographing position. For example, the robot controls a camera mounted on the robot to move to a designated position (i.e., the photographing position).

[0092] In some embodiments, the focus, exposure, gain, and light source brightness of the camera may be preset so that the captured target image of the tightening slot is clearer.

[0093] For example, the robot can obtain the coordinates of a pre-stored designated location and, driven by the robot's programmable logic controller, precisely move the camera to the preset designated location through the coordination of the servo motor and the guide rail. After the camera moves, it initializes the camera according to the pre-set shooting parameters and triggers the camera to capture images.

[0094] In step 420, a camera takes a picture. For example, a first target image of the tightening groove is obtained by the camera. That is, the first target image is obtained by taking a picture with the camera.

[0095] In step 430, the model is initially positioned. For example, the tightening groove is initially positioned using a deep learning model.

[0096] For example, the first target image is processed using a deep learning model to determine a first target detection area corresponding to the tightening groove in the first target image.

[0097] In some embodiments, before using the deep learning model to preliminarily locate the tightening groove, the deep learning model can be trained first.

[0098] As an example, using the YOLO V11 deep learning model as a training process, the model training process can include: collecting multiple images of tightening slots as a data sample set, cleaning the dataset, and removing samples with poor image quality or damage; annotating the dataset (for example, using the X-Anylabeling annotation tool and annotating the tightening slots with a rotated rectangular box); and exporting the annotation file in YOLO format; partitioning the dataset into a training set, a validation set, and a test set (for example, with an 8:1:1 split); adding the dataset to the YOLO V11 model, configuring the dataset path and dataset category, and training the YOLO V11 model. The images in the dataset can be resized on both the long and short sides (for example, resizing the long side to 640 pixels and padding the short side to 640 pixels); and then normalizing the images to obtain the processed dataset for use as the training dataset. For example, the data normalization method is: X'=X / 255.0, where X is the input image and X' is the output image.

[0099] It can be understood here that the generalization ability of the deep learning model is relatively strong. It only needs to learn a small number of image features of the tightening groove to accurately detect the position of the tightening groove in the images of various tightening grooves. However, the accuracy of the detection results of the deep learning model depends to a large extent on the number and quality of the training samples used in the training process. If the deep learning model is used alone to locate the tightening groove, then in order to improve the positioning accuracy of the tightening groove, it is necessary to collect a large number of images of the tightening groove as a data set, clean and annotate the data set, and then repeatedly train the deep learning model until the detection accuracy of the deep learning model meets the requirements. In this way, the required iteration cycle is relatively long and cannot meet the needs of rapid on-site adjustment. Therefore, the present disclosure proposes to combine the ROI image-based visual algorithm with the detection method based on the deep learning model to improve the positioning accuracy of the tightening groove as a whole, thereby meeting the efficiency and accuracy requirements of the track plate fine-tuning operation.

[0100] In step 440 , the visual inspection algorithm is used for positioning. For example, based on the preliminary positioning results of the deep learning model, the visual inspection algorithm is used to further position the tightening groove.

[0101] For example, a ROI image corresponding to the tightening groove is extracted from the first target image based on the first target detection area, and target position information of the tightening groove is determined based on the ROI image.

[0102] In step 450, whether the positioning information is valid, for example, whether the positioning is valid, that is, whether the acquired target location information is valid.

[0103] For example, considering that the target location information may be garbled or otherwise invalid due to unexpected situations such as interference with the pre-calibrated photo-taking position at the construction site or errors in the operation of the deep learning model, the validity of the target location information can be judged after it is obtained to improve the reliability of the subsequent control process.

[0104] In response to the result of the determination that the acquired target location information is invalid, step 460 may be executed.

[0105] In response to the result of the determination that the acquired target location information is valid, step 470 may be executed.

[0106] In step 460, the photographing position is changed. For example, the photographing position of the camera is changed. For example, the camera can be moved to another photographing position to reacquire the first target image, and then the subsequent detection process is performed based on the reacquired first target image. For example, the alternative photographing position can be an alternative position pre-calibrated according to the calibration method described above.

[0107] In step 470 , the coordinate system is converted. For example, alignment control of the center line of the tightening shaft and the center line of the tightening groove is performed based on the target position information and the conversion relationship between the pixel coordinate system and the robot coordinate system.

[0108] In some embodiments, the target position information includes the position coordinates of the center point of the tightening slot in the first target image in a pixel coordinate system. Based on the position coordinates of the center point of the tightening slot in the first target image in the pixel coordinate system, a first offset of the center of the tightening slot relative to the center of the field of view of the first target image in the pixel coordinate system can be determined. Based on the conversion relationship between the pixel coordinate system and the robotic arm coordinate system, the first offset is converted into a second offset in the robotic arm coordinate system. Based on the second offset, the center line of the tightening shaft is then aligned with the center line of the tightening slot, and the tightening shaft is inserted into the tightening slot.

[0109] In step 480, the tightening shaft is inserted into the tightening groove. For example, after the center line of the tightening shaft is aligned with the center line of the tightening groove, the tightening shaft is inserted into the tightening groove.

[0110] about Figure 5 The steps of the method shown can refer to the previous Figures 1 to 4 The methods in the relevant embodiments are implemented similarly. For specific instructions, please refer to the description in the relevant embodiments above, which will not be repeated here.

[0111] Figure 6 A flow chart illustrating some implementations of steps 430 and 440 is shown.

[0112] like Figure 6 As shown, step 440 may include steps 441 to 446 .

[0113] In step 430, model detection is performed. That is, a model detection operation is performed. For example, the model detection operation includes: using a deep learning model to perform preliminary positioning of the tightening groove.

[0114] In step 441, ROI region extraction is performed. That is, a ROI region extraction operation is performed. For example, the ROI region extraction operation includes: extracting a ROI image corresponding to the tightening groove from the first target image based on the first target detection region.

[0115] In step 442, threshold segmentation is performed. That is, a threshold segmentation operation is performed. For example, the threshold segmentation operation includes: converting the ROI image into a grayscale image, and then using a threshold segmentation algorithm to separate the tightening groove from the background information in the grayscale image.

[0116] In step 443, edge smoothing is performed. That is, an edge smoothing operation is performed. For example, the edge smoothing operation includes: smoothing the edge of the tightening groove in the grayscale image.

[0117] In step 444, region filtering is performed. That is, a region filtering operation is performed. For example, the region filtering operation may include removing background information after smoothing to obtain a second target image of the region where the tightening groove is located. For example, only the region where the tightening groove is located may be retained in the second target image.

[0118] In step 445, corner detection is performed. That is, a corner detection operation is performed. For example, the corner detection operation includes: performing corner detection on the tightening groove based on the second target image to obtain corner point information of the tightening groove. For example, the corner point information includes a set of corner points of the tightening groove.

[0119] In step 446, rectangle detection is performed. That is, a rectangle detection operation is performed. For example, the rectangle detection operation includes: performing straight line fitting based on the set of corner points to extract the outline of the tightening groove; and calculating at least one of the position coordinates of the center point of the tightening groove, the size of the tightening groove, and the rotation angle of the center line of the tightening groove relative to the horizontal direction in a pixel coordinate system based on the outline of the tightening groove as target position information.

[0120] about Figure 6 The steps of the process shown can be implemented similarly to the method described in the above-mentioned related embodiment of obtaining target position information based on the ROI image of the tightening groove. For specific instructions, please refer to the description of the above-mentioned related embodiments, which will not be repeated here.

[0121] Figure 7 A block diagram illustrating a control device of a robot according to some embodiments of the present disclosure is shown.

[0122] like Figure 7As shown, the first control device 600 of the robot includes an acquisition module 601 , a processing module 602 , an extraction module 603 , a determination module 604 and a control module 605 .

[0123] The acquisition module 601 may be configured to acquire a first target image of a tightening groove on a working tool mounted on the track plate, wherein the tightening groove is used to accommodate and fix a tightening shaft of the robot so as to ensure a fixed installation between the robot and the working tool.

[0124] The processing module 602 may be configured to process the first target image using a deep learning model to determine a first target detection area corresponding to the tightening groove in the first target image.

[0125] The extraction module 603 may be configured to extract the ROI image corresponding to the tightening groove from the first target image based on the first target detection area.

[0126] The determination module 604 may be configured to determine target position information of the tightening groove based on the ROI image.

[0127] The control module 605 may be configured to control the center line of the tightening shaft to be aligned with the center line of the tightening groove based on the target position information, and insert the tightening shaft into the tightening groove.

[0128] In some embodiments, the first control device 600 of the robot may also include a controller for executing the above Figures 1 to 6 Other modules for other operations in related embodiments.

[0129] Figure 8 A block diagram illustrating a control device of a robot according to some other embodiments of the present disclosure is shown.

[0130] like Figure 8 As shown, the second control device 700 of the robot of this embodiment includes: a first memory 701 and a first processor 702 coupled to the first memory 701, and the first processor 702 is configured to execute the control method of the robot in any embodiment of the present disclosure based on the instructions stored in the first memory 701.

[0131] The first memory 701 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, a database, and other programs.

[0132] Figure 9 A block diagram illustrating a control device of a robot according to further embodiments of the present disclosure is shown.

[0133] like Figure 9As shown, the third control device 800 of the robot of this embodiment includes: a second memory 801 and a second processor 802 coupled to the second memory 801, and the second processor 802 is configured to execute the control method of the robot in any of the aforementioned embodiments based on the instructions stored in the second memory 801.

[0134] The second memory 801 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, application programs, a boot loader, and other programs.

[0135] The robot's third control device 800 may also include an input / output interface 803, a network interface 804, a storage interface 805, and the like. These interfaces 803, 804, and 805, as well as the second memory 801 and the second processor 802, may be connected, for example, via a bus 806. The input / output interface 803 provides a connection interface for input / output devices such as a display, mouse, keyboard, touch screen, microphone, and speakers. The network interface 804 provides a connection interface for various networked devices. The storage interface 805 provides a connection interface for external storage devices such as SD cards and USB flash drives.

[0136] The embodiments of the present disclosure further provide a robot, comprising the robot control device of any one of the above embodiments (eg, the first control device 600 / the second control device 700 / the third control device 800 ).

[0137] An embodiment of the present disclosure further provides a computer-readable storage medium, comprising computer program instructions, which implement the method of any one of the above embodiments when executed by a processor.

[0138] An embodiment of the present disclosure further provides a computer program product, including a computer program, which implements the method of any one of the above embodiments when executed by a processor.

[0139] Those skilled in the art will appreciate that embodiments of the present disclosure may be provided as methods, systems, or computer program products. Thus, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable, non-transitory storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0140] Thus far, the robot control technology solution according to the present disclosure has been described in detail. To avoid obscuring the concept of the present disclosure, some details known in the art have been omitted. Based on the above description, those skilled in the art will fully understand how to implement the technical solution disclosed herein.

[0141] The methods and systems of the present disclosure may be implemented in many ways. For example, the methods and systems of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above unless otherwise specified. In addition, in some embodiments, the present disclosure may also be implemented as programs recorded in a recording medium, which include machine-readable instructions for implementing the methods according to the present disclosure. Thus, the present disclosure also covers recording media that store programs for executing the methods according to the present disclosure.

[0142] Although some specific embodiments of the present disclosure have been described in detail by way of examples, those skilled in the art will appreciate that the above examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Those skilled in the art will appreciate that modifications may be made to the above embodiments without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. A robot control method, comprising: Acquire a first target image of a tightening groove on a working tool mounted on a track plate, wherein the tightening groove is used to accommodate and fix a tightening shaft of the robot so as to ensure a fixed installation between the robot and the working tool; Processing the first target image using a deep learning model to determine a first target detection area corresponding to the tightening groove in the first target image; Extracting a region of interest (ROI) image corresponding to the tightening groove from the first target image based on the first target detection area; determining target position information of the tightening groove based on the ROI image; Based on the target position information, the center line of the tightening shaft is controlled to be aligned with the center line of the tightening groove, and the tightening shaft is inserted into the tightening groove.

2. The control method according to claim 1, wherein: The target position information includes at least one of the position coordinates of the center point of the tightening groove in the first target image in a pixel coordinate system, the size of the tightening groove, and the rotation angle of the center line of the tightening groove relative to the horizontal direction.

3. The control method according to claim 1, wherein: The extracting the ROI image corresponding to the tightening groove from the first target image based on the first target detection area includes: Expanding a preset number of pixels around the boundary of the first target detection area to determine a second target detection area; The second target detection region is extracted from the first target image as the ROI image.

4. The control method according to claim 3, wherein: Determining the target position information of the tightening groove based on the ROI image includes: Converting the ROI image into a grayscale image; Separating the tightening groove from background information in the grayscale image by a threshold segmentation algorithm; removing the background information to obtain a second target image of the area where the tightening groove is located; The target position information is determined based on the second target image.

5. The control method according to claim 4, wherein: The removing of the background information to obtain a second target image of the area where the tightening groove is located includes: Smoothing the edge of the tightening groove in the grayscale image; The background information is removed after the smoothing process to obtain the second target image.

6. The control method according to claim 4, wherein: The determining the target position information based on the second target image includes: performing corner detection on the tightening groove based on the second target image to obtain corner point information of the tightening groove; The target position information is determined based on the corner point information of the tightening groove.

7. The control method according to any one of claims 1 to 6, wherein: The step of acquiring a first target image of a tightening groove on a working tool installed on a track plate includes: Controlling an image acquisition device installed on the robot to move to a specified position to obtain the first target image at the specified position, wherein when the image acquisition device is located at the specified position, the center of the field of view of the image acquisition device coincides with the center of the tightening slot.

8. The control method according to any one of claims 1 to 6, wherein: The target position information includes the position coordinates of the center point of the tightening groove in the first target image in the pixel coordinate system. The controlling the center line of the tightening shaft to align with the center line of the tightening groove based on the target position information and inserting the tightening shaft into the tightening groove comprises: determining, based on the position coordinates, a first offset of the center of the tightening slot relative to the field of view center of the first target image in the pixel coordinate system; Converting the first offset into a second offset in the robot arm coordinate system; Based on the second offset, the center line of the tightening shaft is controlled to be aligned with the center line of the tightening groove, and the tightening shaft is inserted into the tightening groove.

9. The control method according to claim 8, wherein: The controlling the center line of the tightening shaft to align with the center line of the tightening groove based on the second offset, and inserting the tightening shaft into the tightening groove comprises: Determining an initial offset of the center of the tightening shaft relative to the center of the field of view of the first target image in the robotic arm coordinate system; Determining a third offset of the center of the tightening shaft relative to the center of the tightening slot in the robotic arm coordinate system based on the initial offset and the second offset; Based on the third offset, the center line of the tightening shaft is controlled to be aligned with the center line of the tightening groove, and the tightening shaft is inserted into the tightening groove.

10. A robot control device comprising: an acquisition module configured to acquire a first target image of a tightening groove on a working tool mounted on a track plate, the tightening groove being used to accommodate a tightening shaft of the robot so as to secure the robot to the working tool; a processing module configured to process the first target image using a deep learning model to determine a first target detection area corresponding to the tightening groove in the first target image; An extraction module is configured to extract a region of interest (ROI) image corresponding to the tightening groove from the first target image based on the first target detection area; a determination module, configured to determine target position information of the tightening groove based on the ROI image; The control module is configured to control the center line of the tightening shaft to be aligned with the center line of the tightening groove based on the target position information, and to insert the tightening shaft into the tightening groove.

11. A robot control device, comprising: Memory; and A processor coupled to the memory, wherein the processor is configured to execute the control method according to any one of claims 1 to 9 based on instructions stored in the memory.

12. A robot comprising: The control device according to claim 10 or 11.

13. A computer-readable storage medium having computer instructions stored thereon, wherein when the instructions are executed by a processor, the control method according to any one of claims 1 to 9 is implemented.

14. A computer program product comprising instructions, which, when executed by a processor, cause the processor to perform the control method according to any one of claims 1 to 9.

Images