A method and apparatus for unplugging a wire, an electronic device and a storage medium

Abstract

The application provides a method and device for unplugging a wire, electronic equipment and a storage medium. The method comprises: when it is determined that a target device is in a target area, performing image acquisition on the target device to obtain a first to-be-detected image; the target device comprises a socket; identifying a first position of the socket in the first to-be-detected image; when the first position is identified, adjusting the pose of a to-be-wired mechanical arm based on the first position; and if it is determined that the current to-be-wired mechanical arm can complete the unplugging operation, controlling the to-be-wired mechanical arm to perform the unplugging operation. The method can realize the unplugging operation by identifying the position of the socket and adjusting the pose of the to-be-wired mechanical arm, does not need to modify the production line, reduces the cost, can realize the unplugging operation on different electronic equipment, and improves the flexibility of the unplugging operation.

Description

Technical Field

[0001] This application relates to the field of machine vision positioning, and more specifically, to a method, apparatus, electronic device, and storage medium for plugging and unplugging cables in the field of machine vision positioning. Background Technology

[0002] Industrial production lines have numerous electronic equipment testing stations where wires need to be plugged in and unplugged to test the equipment's performance and functionality. Traditional manual plugging and unplugging methods suffer from high labor costs, high error rates, and low production efficiency.

[0003] To improve production efficiency and reduce human error, there are currently some intelligent automatic insertion and removal solutions. However, implementing intelligent automatic insertion and removal solutions in existing technologies usually requires modifying the production line equipment to meet specific conditions so that the production line equipment can meet the needs of automatic insertion and removal. This often requires more investment in costs and time, and the modified production line is difficult to adapt to different types of electronic equipment, resulting in insufficient flexibility in intelligent automatic insertion and removal operations. Summary of the Invention

[0004] This application provides a method, apparatus, electronic device, and storage medium for plugging and unplugging cables. The method can perform plugging and unplugging operations by identifying the position of the connector and adjusting the posture of the robotic arm to be plugged in. It does not require modification of the production line, reducing costs. It can perform plugging and unplugging operations on electronic devices with different incoming materials, improving the flexibility of plugging and unplugging operations.

[0005] In a first aspect, a method for plugging and unplugging cables is provided. The method includes: acquiring an image of the target device to obtain a first image to be detected, given that the target device is located in a target area; wherein the target device includes a connector; identifying a first position of the connector in the first image to be detected; adjusting the pose of a robotic arm to be plugged in based on the first position after identifying the first position; and controlling the robotic arm to be plugged in to perform the plugging and unplugging operation if it is determined that the current robotic arm can complete the plugging and unplugging operation.

[0006] In the above technical solution, the target device is located in the target area, and an image of the target device is acquired to obtain a first image to be tested. By identifying the first position of the connector in the first image to be tested, the pose of the robotic arm to be inserted is adjusted so that the robotic arm can perform the insertion and removal operation. The robotic arm automatically adjusts its pose based on the connector position to perform the insertion and removal operation, realizing the automation of the insertion and removal operation, reducing the need for human resources, and lowering labor costs. Using image processing technology, the connector position can be accurately identified, avoiding human error and improving the accuracy of test results. The automated operation is faster, improving the efficiency of insertion and removal operations, thereby improving overall production efficiency. Compared with the large-scale production line modification in existing technologies, this solution does not require production line modification, reducing costs. Furthermore, by identifying the first position of the connector, the robotic arm to be inserted can automatically adjust its pose to perform the insertion and removal operation. Therefore, the robotic arm to be inserted can perform insertion and removal operations for different target devices, and the production line does not need to be modified to adapt to a specific target device, improving operational flexibility.

[0007] In conjunction with the first aspect, in some possible implementations, the method further includes: if it is determined that the current robotic arm to be plugged in cannot complete the plugging and unplugging operation, then image acquisition is performed on the target device to obtain a second image to be detected; a second position of the socket is identified in the second image to be detected; if the second position is identified, the pose of the robotic arm to be plugged in is readjusted based on the second position; if it is determined that the robotic arm to be plugged in has completed the readjustment, the robotic arm to be plugged in is controlled to perform the plugging and unplugging operation.

[0008] In the above technical solution, after the first identification and adjustment of the robotic arm to be plugged in, and after determining that the robotic arm cannot complete the plugging and unplugging operation, the image of the target device is acquired again to obtain a second image to be detected. The position of the connector is then identified a second time. Based on the second identified position of the connector, the robotic arm to be plugged in is adjusted again before the plugging and unplugging operation is performed. The second identification and adjustment of the pose can reduce the error that exists in the first identification and adjustment, and ensure that the robotic arm to be plugged in can be more accurately aligned with the connector. This can significantly improve the success rate and reliability of the robotic arm to be plugged in completing the plugging and unplugging operation.

[0009] In combination with the first aspect and the above implementation methods, in some possible implementation methods, identifying the first position of the socket in the first image to be detected includes: acquiring a first template image, the first template image being obtained by processing the depth image of the socket through a first algorithm; identifying the first position of the socket in the first image to be detected based on the first template image; and adjusting the pose of the robotic arm to be connected based on the first position, including: adjusting the pose of the robotic arm to be connected based on the first position and the first template image.

[0010] In the above technical solution, the first template image obtained by depth image processing of the socket is used to identify the first position of the socket. The three-dimensional information in the depth image can be used to more accurately identify the position and posture of the socket, thereby providing reliable support for subsequent adjustment of the posture of the robotic arm to be plugged in in order to realize the plugging and unplugging operation.

[0011] In combination with the first aspect and the above implementation methods, in some possible implementation methods, identifying the first position of the socket in the first image to be detected based on the first template image includes: determining a first window of the same size as the first template image; sliding the first window on the first image to be detected, and calculating the similarity between the first sub-image and the first template image for each position of the first sub-image in the first window during the sliding process; and determining the first position of the socket according to the position of the first sub-image whose similarity with the first template image is greater than a preset similarity.

[0012] In the above technical solution, by sliding a first window on the first image to be detected, the first position of the connector is determined based on the similarity between the first sub-image of the first image to be detected and the first template image within the first window. The sliding window allows for region-by-region comparison between the first image to be detected and the template image, enabling finer matching and improving recognition accuracy. Furthermore, the first template image is a depth image of the connector, including its three-dimensional information. Calculating the similarity between the first sub-image of the first image to be detected and the first template image within the first window using the connector's three-dimensional information to determine the first position of the connector allows for more precise positioning of the connector, enabling the robotic arm to accurately align with the connector and preventing deviations during insertion and removal.

[0013] In combination with the first aspect and the above implementation methods, in some possible implementation methods, adjusting the pose of the robotic arm to be wired based on the first position and the first template image includes: determining the image at the first position in the first image to be detected as the first insertion image; determining multiple first template 3D points included in the first template image; determining a first target 3D point in the first insertion image corresponding to each first template 3D point; determining a first offset pose of the robotic arm to be wired based on each first template 3D point and the first target 3D point corresponding to each first template 3D point; and adjusting the pose of the robotic arm to be wired based on the first offset pose.

[0014] In the above technical solution, by determining the first target 3D point corresponding to each first template 3D point in the first socket image, and based on the one-to-one correspondence between the first template 3D point and the first target 3D point, the first offset pose of the robot arm to be inserted can be accurately determined. Determining the first offset pose and automatically adjusting the pose of the robot arm to be inserted can enable the robot arm to be inserted to adaptively adjust based on the position of the target device socket, so as to realize the insertion and unplugging operation of the target device socket in different postures, thereby improving the flexibility of automatic insertion and unplugging operation.

[0015] In combination with the first aspect and the above implementation methods, in some possible implementation methods, identifying the second position of the socket in the second image to be detected includes: acquiring a second template image, which is a depth image of the socket obtained by processing it through a second algorithm; identifying the second position of the socket in the second image to be detected based on the second template image; and readjusting the pose of the robotic arm to be connected based on the second position, including: readjusting the pose of the robotic arm to be connected based on the second position and the second template image.

[0016] In the above technical solution, the second template image obtained by depth image processing of the socket is used to identify the second position of the socket in the second image to be detected. This can more accurately identify the second position and posture of the socket, which is convenient for subsequent determination of a new offset posture based on the second position to readjust the posture of the robotic arm to be inserted, thus ensuring accurate insertion and removal of the wire.

[0017] In combination with the first aspect and the above implementation methods, in some possible implementation methods, identifying the second position of the socket in the second image to be detected based on the second template image includes: determining a second window of the same size as the second template image; sliding the second window on the second image to be detected, and for each position of the second sub-image in the second window during the sliding process, calculating the normalized correlation coefficient between the second sub-image and the second template image; and determining the second position of the socket based on the position of the second sub-image whose normalized correlation coefficient is greater than a preset coefficient.

[0018] In the above technical solution, by sliding a second window on the second image to be detected, the second position of the connector is determined based on the normalized correlation coefficient between the second sub-image of the second image to be detected and the second template image within the second window. The sliding window allows for region-by-region matching of the second image to be detected and the second template image, achieving a more refined match and thus improving the accuracy of identifying the second position. By calculating and comparing the normalized correlation coefficient between the second sub-image of the second image to be detected and the second template image within the second window with a preset coefficient, the second position of the connector can be determined more accurately. This enables the robotic arm to more accurately align with the connector based on the second position, improving the success rate of the connector insertion and removal operation.

[0019] Combining the first aspect and the above implementation methods, in some possible implementation methods, the pose of the manipulator to be wired is adjusted again based on the second position and the second template image, including: determining the image at the second position in the second image to be detected as the second insertion image; determining multiple second template 3D points included in the second template image; determining a second target 3D point corresponding to each second template 3D point in the second insertion image; determining a second offset pose of the manipulator to be wired based on each second template 3D point and the second target 3D point corresponding to each second template 3D point; and adjusting the pose of the manipulator to be wired again based on the second offset pose.

[0020] In the above technical solution, by determining the second target 3D point corresponding to each second template 3D point in the second socket image, the second offset pose of the robot arm to be inserted can be accurately determined. Based on the second offset pose, the robot arm to be inserted is automatically adjusted a second time. The two adjustments can make the robot arm to be inserted more accurately aligned with the socket, thereby improving the success rate of the robot arm to be inserted automatically.

[0021] Combining the first aspect and the above implementation methods, in some possible implementation methods, the pose of the manipulator to be inserted is adjusted again based on the second offset pose, including: determining the template point cloud corresponding to the second template image; determining the current point cloud corresponding to the second insertion image; registering the template point cloud and the current point cloud using the second offset pose to obtain the third offset pose; and adjusting the pose of the manipulator to be inserted again based on the second offset pose and the third offset pose.

[0022] In the above technical solution, when adjusting the pose of the manipulator to be inserted for the second time, the second offset pose is determined by pre-set 3D points, then the template point cloud corresponding to the second template image and the current point cloud corresponding to the second insertion port image are determined. The third offset pose is determined by registering the point cloud with the second offset pose. The manipulator to be inserted is adjusted based on the second and third offset poses, which realizes a more precise pose adjustment of the manipulator to be inserted, thereby reducing the pose adjustment error and significantly improving the operation accuracy of the manipulator to be inserted and increasing the success rate of the operation.

[0023] In combination with the first aspect and the above implementation methods, in some possible implementation methods, image acquisition of the target device includes: controlling a 3D camera on the robotic arm to be connected to acquire images of the target device.

[0024] In the above technical solution, the target device is imaged by a 3D camera on the robotic arm to be wired. The pose of the 3D camera is consistent with the pose of the robotic arm to be wired, which simplifies the image processing and pose adjustment calculations. The pose of the robotic arm to be wired can be adjusted more directly and accurately based on the image of the target device.

[0025] In summary, this application uses a 3D camera on a robotic arm to capture the first image of the target device's first insertion port and identifies its first position. Based on this first position, the robotic arm's pose is automatically adjusted. Once it is determined that the robotic arm can perform the insertion / removal operation, it is controlled to execute the operation, thus automating the insertion / removal process. This reduces the need for human resources and lowers labor costs. Compared to the large-scale production line modifications required in existing technologies, this solution significantly reduces costs. Because the robotic arm automatically adjusts its pose to perform the insertion / removal operation, it can perform the operation for different target devices without requiring the production line to be modified to adapt to a specific type of target device, thereby improving operational flexibility. After the first identification and adjustment of the robotic arm to be plugged in, if it is determined that the robotic arm cannot complete the plugging and unplugging operation, the target device is imaged again to obtain a second image to be detected, so as to identify the position of the plug for the second time. Based on the second identified position of the plug, the robotic arm to be plugged in is adjusted again, and then the plugging and unplugging operation is performed again. The second identification and adjustment of the posture can reduce the error of the posture adjustment in the first identification, and ensure that the robotic arm to be plugged in can be more accurately aligned with the plug, which can significantly improve the success rate and reliability of the robotic arm to be plugged in to complete the plugging and unplugging operation.

[0026] Secondly, a device for plugging and unplugging cables is provided. The device includes: an imaging module for acquiring an image of the target device when it is determined that the target device is in a target area, thereby obtaining a first image to be detected; wherein the target device includes a connector; an identification module for identifying a first position of the connector in the first image to be detected; an adjustment module for adjusting the pose of a robotic arm to be plugged in based on the first position when the first position is identified; and a control module for controlling the robotic arm to be plugged in to perform the plugging and unplugging operation if it is determined that the current robotic arm to be plugged in can complete the plugging and unplugging operation.

[0027] In conjunction with the second aspect, in some possible implementations, the imaging module is further configured to, if it is determined that the current robotic arm to be plugged in cannot complete the plugging and unplugging operation, acquire an image of the target device to obtain a second image to be detected; the recognition module is further configured to, identify the second position of the connector in the second image to be detected; the adjustment module is further configured to, based on the second position, readjust the pose of the robotic arm to be plugged in when the second position is identified; and the control module is further configured to, when it is determined that the robotic arm to be plugged in has completed the readjustment, control the robotic arm to be plugged in to perform the plugging and unplugging operation.

[0028] In combination with the second aspect and the above implementation methods, in some possible implementation methods, the recognition module is specifically used to: acquire a first template image, which is obtained by processing a depth image of the socket using a first algorithm; identify the first position of the socket in the first image to be detected based on the first template image; and the adjustment module is specifically used to: adjust the pose of the robotic arm to be connected based on the first position and the first template image.

[0029] Combining the second aspect and the above implementation methods, in some possible implementation methods, the recognition module is specifically used to: determine a first window with the same size as the first template image; slide the first window on the first image to be detected; for the first sub-image in the first window at each position during the sliding process, calculate the similarity between the first sub-image and the first template image; and determine the first position of the interposer based on the position of the first sub-image with a similarity greater than a preset similarity with the first template image.

[0030] Combining the second aspect and the above implementation methods, in some possible implementation methods, the adjustment module is specifically used to: determine the image at the first position in the first image to be detected as the first insertion image; determine multiple first template 3D points included in the first template image; determine a first target 3D point corresponding to each first template 3D point in the first insertion image; determine the first offset pose of the manipulator to be inserted based on each first template 3D point and the first target 3D point corresponding to each first template 3D point; and adjust the pose of the manipulator to be inserted based on the first offset pose.

[0031] Combining the second aspect and the above implementation methods, in some possible implementation methods, the recognition module is specifically used to: acquire a second template image, which is a depth image of the socket obtained by processing a second algorithm; based on the second template image, identify the second position of the socket in the second image to be detected; the adjustment module is specifically used to: adjust the pose of the robotic arm to be inserted again based on the second position and the second template image.

[0032] Combining the second aspect and the above implementation methods, in some possible implementation methods, the recognition module is specifically used to: determine a second window of the same size as the second template image; slide the second window on the second image to be detected; for the second sub-image in the second window at each position during the sliding process, calculate the normalized correlation coefficient between the second sub-image and the second template image; and determine the second position of the indenter based on the position of the second sub-image whose normalized correlation coefficient is greater than a preset coefficient.

[0033] Combining the second aspect and the above implementation methods, in some possible implementation methods, the adjustment module is specifically used to: determine the image at the second position in the second image to be detected as the second insertion image; determine multiple second template 3D points included in the second template image; determine the second target 3D point corresponding to each second template 3D point in the second insertion image; determine the second offset pose of the manipulator to be inserted based on each second template 3D point and the second target 3D point corresponding to each second template 3D point; and adjust the pose of the manipulator to be inserted again based on the second offset pose.

[0034] Combining the second aspect and the above implementation methods, in some possible implementation methods, the adjustment module is specifically used to: determine the template point cloud corresponding to the second template image; determine the current point cloud corresponding to the second insertion image; register the template point cloud and the current point cloud using the second offset pose to obtain the third offset pose; and adjust the pose of the robotic arm to be inserted again based on the second offset pose and the third offset pose.

[0035] Combining the second aspect and the above implementation methods, in some possible implementation methods, the shooting module is specifically used to control the 3D camera on the robotic arm to be connected to acquire images of the target device.

[0036] Thirdly, an electronic device is provided, including a memory and a processor. The memory is used to store executable program code, and the processor is used to call and run the executable program code from the memory, causing the electronic device to perform the methods of the first aspect or any possible implementation thereof.

[0037] Fourthly, a computer program product is provided, comprising: computer program code, which, when run on a computer, causes the computer to perform the methods described in the first aspect or any possible implementation thereof.

[0038] Fifthly, a computer-readable storage medium is provided that stores computer program code, which, when executed on a computer, causes the computer to perform the methods described in the first aspect or any possible implementation thereof. Attached Figure Description

[0039] Figure 1 This is a schematic flowchart illustrating a method for plugging and unplugging a cable provided in an embodiment of this application.

[0040] Figure 2 This is a schematic diagram of a first image to be detected provided in an embodiment of this application.

[0041] Figure 3 This is a schematic diagram of the depth image of a socket provided in an embodiment of this application.

[0042] Figure 4 This is a schematic diagram of a second image to be detected provided in an embodiment of this application.

[0043] Figure 5 This is a flowchart illustrating another method for controlling the plug-in / plug-out of the cable provided in this application.

[0044] Figure 6 This is a schematic diagram of the structure of a plug-in / plug-out device provided in an embodiment of this application.

[0045] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0046] The technical solutions in this application will be clearly and thoroughly described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. "And / or" in the text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0047] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0048] On industrial production lines, electronic equipment testing stations require frequent plugging and unplugging of cables to test the equipment's performance and functionality. Traditional manual plugging and unplugging methods have the following problems:

[0049] High labor costs: A large amount of human resources is needed to perform repetitive plugging and unplugging operations, which increases labor costs.

[0050] High error rate: Manual operation is prone to errors, leading to inaccurate test results or equipment damage. Low production efficiency: Manual operation is slow, affecting overall production efficiency.

[0051] To improve production efficiency and reduce human error, some intelligent automatic cable insertion and removal solutions exist. These solutions modify production line equipment to meet specific conditions, enabling automated cable insertion and removal. For example, a production line might be modified to accommodate devices of a certain shape, ensuring a consistent and stable position for the robotic arm to locate the insertion points and facilitate automated cable insertion and removal. However, modifying existing production lines requires significant investment of capital and time, resulting in high costs. Furthermore, once the production line is modified, it struggles to adapt to changes in electronic equipment, making the intelligent automatic cable insertion and removal operation less flexible.

[0052] Based on this, this application provides a method for plugging and unplugging cables, which is applied to the controller of a robotic arm to be plugged in. The method involves capturing an image of the target device using a 3D camera to identify the position of the connector, automatically adjusting the pose of the robotic arm based on the connector position, and controlling the robotic arm to perform the plugging and unplugging operation when it is determined that the robotic arm can complete the operation. By identifying the connector position and adjusting the pose of the robotic arm to achieve the plugging and unplugging operation, no modifications to the production line are required, reducing costs. This method can perform plugging and unplugging operations on electronic devices with different incoming materials, improving the flexibility of the operation.

[0053] Figure 1 This is a flowchart illustrating a method for plugging and unplugging a cable provided in an embodiment of this application.

[0054] For example, such as Figure 1 As shown, the method 100 includes:

[0055] Step 101: After determining that the target device is in the target area, the target device is image acquired to obtain the first image to be detected; wherein, the target device includes a port;

[0056] Step 102: Identify the first position of the socket in the first image to be detected;

[0057] Step 103: If the first position is identified, adjust the pose of the robotic arm to be connected based on the first position;

[0058] Step 104: If it is determined that the robotic arm to be plugged in can complete the plugging and unplugging operation, then control the robotic arm to be plugged in to perform the plugging and unplugging operation.

[0059] exist Figure 1In the illustrated embodiment, when the target device is determined to be in the target area, an image of the target device is acquired to obtain a first image to be detected. By identifying the first position of the connector in the first image to be detected, the pose of the robotic arm to be inserted is adjusted so that the robotic arm can perform the insertion and removal operation. The robotic arm to be inserted automatically adjusts its pose based on the connector position to perform the insertion and removal operation, realizing automation of the insertion and removal operation, reducing the need for human resources, and lowering labor costs. Using image processing technology, the connector position can be accurately identified, avoiding human error and improving the accuracy of test results. The automated operation is faster, improving the efficiency of insertion and removal operations, thereby improving overall production efficiency. Compared with the large-scale modification of the production line in the prior art, this solution does not require modification of the production line, reducing costs. Furthermore, by identifying the first position of the connector, the robotic arm to be inserted can automatically adjust its pose to perform the insertion and removal operation. Therefore, the robotic arm to be inserted can perform insertion and removal operations for target devices with different incoming materials, and the production line does not need to be modified to adapt to a certain type of target device, improving operational flexibility.

[0060] The following is about Figure 1 The specific implementation methods of each step in the illustrated embodiments are explained in detail below:

[0061] In step 101, the target device is the device that needs to be plugged in or unplugged, and the target device includes at least one socket.

[0062] The target device can be, for example, a display screen, circuit board, sensor module, communication equipment (such as routers and switches), medical equipment (such as electrocardiogram monitors and blood pressure monitors), etc. The port can specifically be a power port, network cable port, Type-C port, High-Definition Multimedia Interface (HDMI) port, or Universal Serial Bus (USB) port, etc. This application embodiment does not limit the type of port.

[0063] The target area is typically a designated area on the production line where the robotic arm can reach and perform cable insertion / removal operations. The target device can be transferred to the target area via the production line, and the controller of the robotic arm can confirm that the target device is within the operable target area using sensors (such as infrared sensors) installed on the robotic arm.

[0064] The first image to be detected is a depth image, which can be obtained based on a 3D camera. A depth image is specifically a two-dimensional image, where each pixel contains a depth value, representing the distance (depth) from the camera to the corresponding point in the scene.

[0065] The 3D camera can be a camera that is pre-set in a fixed position to capture the target area. In some embodiments, the 3D camera can also be set on the robotic arm to be connected.

[0066] In one possible implementation, image acquisition of the target device includes controlling a 3D camera on the robotic arm to be connected to acquire images of the target device.

[0067] Specifically, the 3D camera mounted on the robotic arm to be connected can capture a depth image of the target area. After determining that the target device is in the target area, the controller controls the 3D camera on the robotic arm to capture a depth image of the target device, thus obtaining the first image to be detected.

[0068] The robotic arm to be plugged in includes a mechanical gripper that holds a plug. When the mechanical gripper and the corresponding socket of the plug are aligned parallel to each other, the robotic arm can insert the plug into the socket. The plug can be a power plug, network cable plug, Type-C plug, High-Definition Multimedia Interface (HDMI) plug, or Universal Serial Bus (USB) plug, etc. In this embodiment, the type of plug is not limited; however, the plug and socket types must correspond to achieve the plugging and unplugging operation. A 3D camera is specifically mounted on the mechanical gripper of the robotic arm to be plugged in.

[0069] In the above method, the target device is imaged by a 3D camera on the robotic arm to be wired. The pose of the 3D camera is consistent with the pose of the robotic arm to be wired, which simplifies the image processing and pose adjustment calculations. It can more directly and accurately adjust the pose of the robotic arm to be wired based on the image of the target device.

[0070] In step 102, after obtaining the first image to be detected, the position of the target device's port in the first image to be detected can be identified. If the first image to be detected includes the port, the first position of the port in the first image to be detected can be identified.

[0071] It is understandable that the first image to be detected is an image of the target device. Since the target device can be in any orientation when the robotic arm takes the picture, the first image to be detected may or may not include the connector. The first image to be detected that includes the connector is as follows: Figure 2 As shown.

[0072] In some embodiments, when the first image to be detected does not include the connector of the target device, the first position of the connector cannot be identified in the first image to be detected. In this case, the pose of the robotic arm to be connected can be adjusted, or the position of the robotic arm can be moved, and then the target device can be photographed again using the 3D camera on the robotic arm to be connected, to obtain the first image to be detected. The first position of the connector is identified in the first image to be detected, and the above steps are repeated until the first position of the connector is identified in the first image to be detected.

[0073] The first position can be represented based on coordinates, where the coordinate system can be a coordinate system established based on the first detected image. For example, a coordinate system can be established with the top left vertex of the first detected image as the origin, the vertical side from top to bottom as the Y-axis, and the horizontal side from left to right as the X-axis. In this case, the first position can be represented as (x, y).

[0074] It is understandable that a socket usually occupies a certain area, and the position of the top left vertex (x1, y1) and the bottom right vertex (x2, y2) of the circumscribed rectangle of the socket can be recorded as the position of the socket.

[0075] The process of identifying the first position of the socket in the first image to be detected can be achieved by using a deep learning model to identify the first position of the socket of the target device in the first image to be detected.

[0076] In some embodiments, the first position of the target device's port in the first image to be detected can also be identified by template matching. The following embodiments will describe in detail the process of a scheme for identifying the first position of the target device's port in the first image to be detected by template matching:

[0077] In one possible implementation, identifying the first position of the socket in the first image to be detected includes: acquiring a first template image, which is obtained by processing a depth image of the socket using a first algorithm; and identifying the first position of the socket in the first image to be detected based on the first template image.

[0078] The first algorithm can specifically be the Linemod algorithm. Before starting the plugging and unplugging operation on the target device, a 3D camera can be used to photograph the target device's connector to obtain a depth image of the connector. Then, the Linemod algorithm is used to process the depth image of the connector to obtain a first template image, which is stored in the controller of the robotic arm to be plugged in. Assuming the connector is a power connector, the depth image of the connector is as follows: Figure 3 As shown.

[0079] The 3D camera that takes a depth image of the target device's connector in advance can be a 3D camera on the robotic arm to be connected, or another 3D camera. When taking the image of the target device's connector, try to ensure that the plane of the 3D camera and the connector are parallel, so that the position of the connector in the image to be detected can be better identified based on the template image obtained from the depth image.

[0080] Understandably, the linemod algorithm can identify the port by matching visible feature points when partially occluded. It can achieve coarse identification when the target device pose is poor, ensuring the success rate of the identification result and improving the robustness of the identification. When performing the first identification of the port of the target device, the pose of the target device in the first image to be detected may be poor. At this time, a coarse identification can be performed based on the linemod algorithm.

[0081] The process of processing the depth image of the interposer based on the linemod algorithm to obtain the first template image specifically includes the following steps 1 to 4:

[0082] Step 1: Rotate the depth image of the port at preset intervals to obtain multiple depth images at different angles.

[0083] Table 1

[0084] 1 2 3 4 5 6 7 8 45 90 135 180 225 270 315 360

[0085] For example, as shown in Table 1, by taking an angle at 45-degree intervals from 0 degrees to 360 degrees, we obtain eight rotation angles: 45 degrees, 90 degrees, 135 degrees, 180 degrees, 225 degrees, 270 degrees, 315 degrees, and 360 degrees. By rotating the depth image of the socket based on each of the eight rotation angles, we can obtain eight depth images.

[0086] It is understandable that when processing based on the linemod algorithm, rotating an image means rotating the entire image, including its contents, based on a rotation angle.

[0087] Step 2: For each depth image obtained by rotation, calculate the gradient magnitude and gradient direction of each pixel in the depth image.

[0088] Table 2

[0089] pixel Pixel 1 Pixel 2 Pixel 3 Pixel 4 Pixel 5 6 pixels gradient magnitude (10,16) (8,24) (6,32) (6,24) (4,32) (4,24) gradient direction 58 degrees 80 degrees 120 degrees 175 degrees 256 degrees 310 degrees

[0090] For example, Table 2 shows the gradient magnitude and gradient direction calculated for pixels in a depth image rotated by 45 degrees. As shown in Table 2, the depth image rotated by 45 degrees includes: pixel 1, pixel 2, pixel 3, pixel 4, pixel 5, and pixel 6. The gradient magnitude of pixel 1 is (10, 16), and the gradient direction is 58 degrees; the gradient magnitude of pixel 2 is (8, 24), and the gradient direction is 80 degrees; the gradient magnitude of pixel 3 is (6, 32), and the gradient direction is 120 degrees; the gradient magnitude of pixel 4 is (6, 24), and the gradient direction is 175 degrees; the gradient magnitude of pixel 5 is (4, 32), and the gradient direction is 82 degrees; and the gradient magnitude of pixel 6 is (4, 24), and the gradient direction is 42 degrees.

[0091] Step 3: Then, quantize the gradient direction according to different angle ranges.

[0092] Specifically, the range from 0 to 360 degrees can be divided into multiple ranges to determine the range to which the gradient direction of each pixel belongs, thereby quantizing the gradient direction.

[0093] For example, the 360 ​​degrees are divided into four ranges: range 1 is [0, 90) degrees, range 2 is [90, 180) degrees, range 3 is [180, 270) degrees, and range 4 is [270, 360) degrees. The range to which the gradient direction of each pixel belongs is determined, thereby quantizing the gradient direction.

[0094] Table 3

[0095] pixel Pixel 1 Pixel 2 Pixel 3 Pixel 4 Pixel 5 6 pixels gradient direction 58 degrees 80 degrees 120 degrees 175 degrees 256 degrees 310 degrees scope Range 1 Range 1 Scope 2 Scope 2 Range 3 Range 4

[0096] As shown in Table 3, the gradient direction of pixel 1 is 58 degrees and belongs to range 1; the gradient direction of pixel 2 is 80 degrees and belongs to range 1; the gradient direction of pixel 3 is 120 degrees and belongs to range 2; the gradient direction of pixel 4 is 175 degrees and belongs to range 2; the gradient direction of pixel 5 is 256 degrees and belongs to range 3; and the gradient direction of pixel 6 is 310 degrees and belongs to range 4.

[0097] Step 4: For each depth image, the pixels in the depth image whose gradient magnitude is greater than the magnitude threshold are used as the template feature point set of the depth image to obtain the first template image.

[0098] For example, with an amplitude threshold of (3, 21), it can be determined that the gradient amplitudes of pixels 2, 3, 4, 5 and 6 are all greater than the amplitude threshold. Therefore, pixels 2, 3, 4, 5 and 6 can be determined as the template feature point set of the image, and the first template image is obtained.

[0099] It is understandable that when processing the depth image of the interposer based on the linemod algorithm, the first template image obtained is usually multiple images, and each first template image corresponds to its own template feature point set, and the feature points in each feature point set correspond to the quantized gradient direction.

[0100] In some embodiments, when the socket occupies only a part of the depth image of the captured socket, the area of ​​the socket in the depth image can be selected first, and the depth image of the socket area can be processed by the linemod algorithm to obtain the first template image.

[0101] The controller acquires the stored first template image and sequentially matches and identifies the first position of the interposer in the first image to be detected based on the first template image from various angles.

[0102] The following embodiments detail the process of identifying the first position of the insertion port in the first image to be detected based on a first template image from a certain angle:

[0103] In one possible implementation, identifying the first position of the socket in the first image to be detected based on the first template image includes: determining a first window of the same size as the first template image; sliding the first window on the first image to be detected, and calculating the similarity between the first sub-image and the first template image for each position of the first sub-image in the first window during the sliding process; and determining the first position of the socket based on the position of the first sub-image whose similarity with the first template image is greater than a preset similarity.

[0104] Specifically, the first size of the first template image can be obtained first, and a first window of the same size as the first template image can be generated based on the first size of the template image. The first window can completely include the first template image.

[0105] For example, if the first size of the obtained first template image is 2cm long and 1cm wide, then a first window with a length of 2cm and a width of 1cm can be generated based on the first size.

[0106] It's understandable that the size of the first template image obtained by rotating the depth image of the socket during template creation is the size of the smallest bounding rectangle of the rotated depth image of the socket. The smallest bounding rectangle is the smallest rectangle that completely contains the rotated depth image of the socket, and its sides are parallel to the coordinate axes. In other words, the size of the first window may differ depending on the rotation angle of the first template image.

[0107] After generating the first window, the first window can be slid across the first image to be detected at a preset step size. Each time the first window is slid across the first image to be detected, it will have a corresponding position on the first image to be detected, and the first window will select a sub-image of the first image to be detected at that position, which is the first sub-image.

[0108] For example, the first window is 2cm long and 1cm wide. The first window slides starting from the top left corner of the first image to be detected. The coordinate system is established with the top left vertex of the first image to be detected as the origin, as described in the above embodiment. Before sliding, the position of the first window is: top left vertex (0, 0), bottom right vertex (2, 1). Assuming that it slides horizontally from left to right with a preset step size of 0.5cm, the position of the first window after one slide is: top left vertex (0.5, 0), bottom right vertex (2.5, 1), and so on, until it slides horizontally to the right boundary of the first image to be detected. Then it slides vertically downward by one step, and then starts sliding horizontally from left to right again.

[0109] A preset similarity threshold is used to filter out matching results with high similarity. For the first sub-image in the first window at each position during the sliding process, the similarity between the first sub-image and the first template image that generated the first window can be calculated. First sub-images with a similarity greater than the preset similarity are filtered out. When only one first sub-image is filtered out, its position is determined as the first position of the interpolation. When multiple first sub-images are filtered out, the position of the first sub-image with the highest similarity among the filtered first sub-images is determined as the first position of the interpolation. For example, as shown... Figure 2 As shown, the position of the first sub-image in the first window 201 is the first position of the inlet.

[0110] It is understandable that there may be multiple first sub-images with a similarity greater than the preset similarity to the first template image. In this case, the position of the first sub-image with the highest similarity among the first sub-images can be determined as the first position of the socket, which can more accurately determine the first position of the socket.

[0111] The steps for calculating the similarity between the first sub-image and the first template image used to generate the first window specifically include: obtaining the template feature point set of the first template image; calculating the gradient magnitude and gradient direction of each pixel in the first sub-image; quantizing the gradient direction according to different angle ranges, where the angle range for quantizing the gradient direction is the same as the angle range for quantizing the gradient direction of pixels in the first template image when creating the first template image; determining the pixels in the first sub-image with gradient magnitudes greater than a preset magnitude as the target feature point set of the first sub-image; and calculating the similarity based on the target feature point set and the template feature point set.

[0112] For example, the formula for calculating similarity based on the target feature point set and the template feature point set is shown in the following formula (1):

[0113]

[0114] In formula (1), n ​​is the total number of feature points, i is the i-th feature point, and I is any region to be searched. The pixel at position (x, y) in the first image to be tested is offset by a gradient direction of (dx, dy). represents the gradient direction of the pixel at position (x, y) in the first template image, and score represents the similarity, which is also referred to as the matching similarity score in some embodiments.

[0115] Understandably, similarity is specifically used to determine whether the gradient direction of the quantized target feature point set is consistent with the gradient direction of the quantized template feature point set. When the similarity is greater than the preset similarity, it can be determined that the gradient direction of the quantized target feature point set is consistent with the gradient direction of the quantized template feature point set. If the gradient directions are consistent, it is determined that the first sub-image matches the first template image.

[0116] In the above method, by sliding a first window on the first image to be detected, the first position of the connector is determined based on the similarity between the first sub-image of the first image to be detected and the first template image within the first window. The sliding window allows for region-by-region comparison between the first image to be detected and the template image, enabling finer matching and improving recognition accuracy. Furthermore, the first template image is a depth image of the connector, including its three-dimensional information. Using this three-dimensional information to calculate the similarity between the first sub-image of the first image to be detected and the first template image within the first window to determine the first position of the connector allows for more precise positioning. This ensures the robotic arm can accurately align with the connector, preventing deviations during insertion and removal.

[0117] In step 103, the pose of the robot arm to be inserted includes the position coordinates and rotation angle of the robot arm to be inserted. When the first position is identified, the offset pose of the robot arm to be inserted can be determined based on the first position of the insertion port, and the pose of the robot arm to be inserted can be adjusted based on the offset pose.

[0118] Specifically, the angles that need to be adjusted for each joint of the manipulator to be wired can be calculated based on the offset pose. Based on the angles that need to be adjusted for each joint, the rotation of each joint is controlled to achieve the adjustment of the pose of the manipulator to be wired based on the offset pose.

[0119] In one possible implementation, adjusting the pose of the robotic arm to be wired based on the first position includes: adjusting the pose of the robotic arm to be wired based on the first position and the first template image.

[0120] As in the above embodiment, the controller stores a first template image. After determining the first position of the socket based on the first template image, an offset pose can be determined based on the first position and the first template image to adjust the pose of the robotic arm to be plugged in.

[0121] Specifically, the first socket image in the first image to be detected can be determined based on the first position, and the offset pose of the robotic arm to be inserted can be determined based on the first socket image and the first template image. Then, the pose of the robotic arm to be inserted can be adjusted based on the offset pose.

[0122] In the above method, the first template image obtained by processing the depth image of the socket is used to identify the first position of the socket. The three-dimensional information in the depth image can be used to more accurately identify the position and posture of the socket, thereby providing reliable support for subsequent adjustment of the pose of the robotic arm to be plugged in in order to realize the plugging and unplugging operation.

[0123] In one possible implementation, adjusting the pose of the robotic arm to be wired based on a first position and a first template image includes: determining the image at a first position in a first image to be detected as a first insertion port image; determining a plurality of first template 3D points included in the first template image; determining a first target 3D point in the first insertion port image corresponding to each first template 3D point; determining a first offset pose of the robotic arm to be wired based on each first template 3D point and the first target 3D point corresponding to each first template 3D point; and adjusting the pose of the robotic arm to be wired based on the first offset pose.

[0124] Specifically, a local image at a first position in the first image to be detected can be selected and extracted to obtain the first port image of the target device in the first image to be detected.

[0125] The first template 3D points are a set of pre-defined 3D points within the first template image after its creation. These are typically selected from key locations within the first template image, such as the center of the connector or the four vertices. At least three points are generally set to ensure sufficient constraints for pose estimation. Each first template 3D point corresponds to a three-dimensional coordinate system (x, y, z). Here, x represents the point's position in the horizontal direction of the image plane, y represents its position in the vertical direction, and z represents the distance (depth) from the 3D point to the 3D camera.

[0126] After determining the first socket image, the first target 3D point corresponding to each first template 3D point can be determined from the first socket image. Each first target 3D point also corresponds to a three-dimensional coordinate (X, Y, Z). Specifically, based on the relative position of the first template 3D point in the first template image, such as determining the horizontal and vertical distances of the first template 3D point from the center of the socket, the position in the first socket image determined by the same horizontal and vertical distances from the center of the socket can be determined as the first target 3D point corresponding to the first template 3D point.

[0127] As in the above embodiment, if four first template 3D points are preset in the first template image, then four corresponding first target 3D points can be determined in the first insertion image.

[0128] Table 4

[0129] First template 3D points Point 1 Point 2 Point 3 Point 4 First target 3D point 5 points Point 6 7 points 8

[0130] For example, as shown in Table 4, the first template 3D points include four 3D points: point 1, point 2, point 3, and point 4. The first target 3D point corresponding to point 1 is point 5, the first target 3D point corresponding to point 2 is point 6, the first target 3D point corresponding to point 3 is point 7, and the first target 3D point corresponding to point 4 is point 8.

[0131] The first offset pose includes rotation and translation. Each first template 3D point and the corresponding first target 3D point are recorded as a matching pair, which can yield multiple matching pairs: point 1 and point 5, point 2 and point 6, point 3 and point 7, point 4 and point 8. Based on multiple matching pairs, the transformation matrix between the first template 3D point and the first target 3D point can be calculated. The transformation matrix includes rotation and translation components. By extracting the rotation and translation components from the transformation matrix, the first offset pose of the manipulator to be inserted can be determined.

[0132] In the above method, by determining the first target 3D point corresponding to each first template 3D point in the first socket image, and based on the one-to-one correspondence between the first template 3D point and the first target 3D point, the first offset pose of the robot arm to be inserted can be accurately determined. Determining the first offset pose and automatically adjusting the pose of the robot arm to be inserted can enable the robot arm to be inserted to adaptively adjust based on the position of the target device socket, so as to realize the insertion and unplugging operation for target device sockets with different postures, thereby improving the flexibility of automatic insertion and unplugging operation.

[0133] In step 104, after determining that the robotic arm to be plugged in has completed the adjustment of the first offset pose, it can be determined whether the robotic arm to be plugged in can complete the plugging and unplugging operation in the current posture. If it is determined that the robotic arm to be plugged in can complete the plugging and unplugging operation in the current posture, then the robotic arm to be plugged in is controlled to perform the plugging and unplugging operation, that is, the robotic arm to be plugged in is controlled to insert the gripped plug into the socket, and after the test is completed in the socket, the plug is pulled out.

[0134] The step of determining whether the robotic arm to be inserted can complete the insertion / removal operation in its current posture may include: determining whether the first offset pose is less than the preset offset pose; if the first offset pose is determined to be less than the preset offset pose, then the robotic arm to be inserted can complete the insertion / removal operation in its current posture. It can be understood that the first offset pose includes translation and rotation, and the preset offset pose may specifically include preset translation and preset rotation. The preset offset pose is used to determine whether the current adjustment is a fine adjustment or a coarse adjustment. When the first offset pose is determined to be less than the preset offset pose, it can be determined that the current adjustment is a fine adjustment. Fine adjustments typically have smaller errors. In this case, it can be determined that the error between the adjusted pose of the robotic arm to be inserted and the standard pose that can complete the insertion / removal operation is small, and therefore the robotic arm to be inserted can complete the insertion / removal operation in its current posture.

[0135] In some embodiments, after determining that the robotic arm to be connected has completed the adjustment of the first offset pose, the 3D camera on the robotic arm can be controlled to take another new depth image of the target device, extract 3D point cloud data from the depth image, and match the newly extracted 3D point cloud data with the 3D point cloud data of the first template image to find the position and point cloud of the current connector. The ability of the robotic arm to complete the connector connection operation is determined by whether the overlap between the point cloud of the current connector and the point cloud of the connector in the first template image is greater than a preset overlap. Specifically, if the point cloud overlap is greater than the preset overlap, it is determined that the robotic arm to complete the connector connection operation; if the point cloud overlap is less than or equal to the preset overlap, it is determined that the robotic arm to complete the connector connection operation cannot complete the connector connection operation.

[0136] In some embodiments, after taking a new depth image of the target device, the image of the current socket in the new depth image can also be determined. Based on the image of the current socket and the first template image, it is determined whether the current socket and the socket in the first template image are parallel. If it is determined that the current socket and the socket in the first template image are parallel, it is determined that the robotic arm to be plugged in can complete the plugging and unplugging operation.

[0137] In one possible implementation, the method further includes the following steps S11 to S14:

[0138] S11, if it is determined that the current robotic arm to be inserted cannot complete the insertion and removal operation, then the target device is image acquired to obtain a second image to be detected.

[0139] As described in the above embodiment, after the first adjustment of the robot arm's pose, it is determined whether the robot arm can complete the insertion / removal operation based on the first offset pose and the preset offset pose. If the first offset pose is greater than or equal to the preset offset pose, it can be determined that the current adjustment is a coarse adjustment. Coarse adjustments typically have a large error, and in this case, it can be determined that the adjusted pose of the robot arm is significantly different from the standard pose for insertion / removal, thus determining that the robot arm cannot complete the insertion / removal operation in its current pose. Since there is still a certain pose deviation between the current pose of the robot arm and the standard pose that enables the insertion / removal operation, further adjustment of the robot arm's pose is necessary.

[0140] Understandably, the secondary adjustment is based on the current pose of the robotic arm to be connected after the first adjustment. Therefore, the controller uses the current pose of the robotic arm to control the 3D camera to take another picture of the target device, obtaining a new depth image of the target device, which is the second image to be detected. The second image to be detected is shown below. Figure 4 As shown, the port of the target device is located in the second image to be detected, and the position of the port in the second image to be detected is slightly different from that in the first image to be detected.

[0141] S12, identify the second position of the socket in the second image to be detected.

[0142] The second image to be detected usually also includes the port of the target device. After obtaining the second image to be detected, the position of the port of the target device in the second image to be detected can be identified to obtain the second position.

[0143] The second position can also be represented by the position of the top left vertex (x1, y1) and the position of the bottom right vertex (x2, y2) of the rectangle that encloses the slot. In this case, the coordinate system can be established based on the top left vertex of the second image to be detected. The establishment steps are similar to the steps of establishing the coordinate system based on the top left vertex of the first image to be detected, and will not be repeated here.

[0144] After obtaining the second image to be detected, the second position of the target device's port in the second image to be detected can also be identified through a deep learning model. The identification process is similar to the process of identifying the first position in the above embodiment, and will not be described again here.

[0145] In some embodiments, the second position of the target device's port in the second image to be detected can also be identified using template matching. The following embodiments will detail another method for identifying the second position of the target device's port in the second image to be detected using template matching:

[0146] In one possible implementation, identifying the second position of the socket in the second image to be detected includes: acquiring a second template image, which is a depth image of the socket obtained by processing it using a second algorithm; and identifying the second position of the socket in the second image to be detected based on the second template image.

[0147] The depth image of the socket can be the same as the depth image processed by the first algorithm (i.e., the linemod algorithm) in the above embodiments.

[0148] The second algorithm can specifically be a normalized cross-correlation algorithm. The process of processing the depth image of the interposer based on the normalized cross-correlation (NCC) algorithm to obtain the second template image specifically includes: rotating the border of the interposer's depth image around the center of the depth image at preset intervals, for example, rotating it based on multiple rotation angles shown in Table 1 to obtain multiple depth images at different angles. Since only the border of the depth image is rotated, the rotated depth image contains regions of invalid data. Pixels in these regions of invalid data can be identified, and each pixel is filled with a specific value as padding to obtain the complete second template image.

[0149] It is understandable that, based on the NCC algorithm to process the depth image of the interface, the resulting second template image is usually multiple, and the obtained second template image is also multiple, with each second template image corresponding to its own pixels.

[0150] The controller acquires the stored second template image and sequentially matches and identifies the second position of the interposer in the second image to be detected based on the second template image from various angles.

[0151] The following embodiments detail the process of identifying the first position of the insertion port in the first image to be detected based on a first template image from a certain angle:

[0152] In one possible implementation, identifying the second position of the socket in the second image to be detected based on the second template image includes: determining a second window of the same size as the second template image; sliding the second window on the second image to be detected; calculating the normalized correlation coefficient between the second sub-image and the second template image for each position of the second sub-image in the second window during the sliding process; and determining the second position of the socket based on the position of the second sub-image whose normalized correlation coefficient is greater than a preset coefficient.

[0153] Specifically, you can first obtain the second size of the second template image, and then generate a second window with the same size as the second template image based on the second size of the template image. The second window can completely include the second template image.

[0154] For example, if the second dimensions of the obtained second template image are 3cm in length and 2m in width, then a second window with a length of 3cm and a width of 2m can be generated based on the second dimensions.

[0155] It's understandable that the size of the second template image, obtained by rotating the depth image of the interposer during template creation, is the size of the smallest bounding rectangle of the rotated depth image of the interposer. The smallest bounding rectangle is the smallest rectangle that completely contains the rotated depth image of the interposer, and its sides are parallel to the coordinate axes. In other words, the size of the second window may differ depending on the rotation angle of the second template image.

[0156] After generating the second window, it can be slid across the second image to be detected at preset step sizes. Each slide creates a corresponding position on the second image, and the second window selects a sub-image of the second image at that position. The specific process of sliding the second window is similar to that of sliding the first window on the first image to be detected, and will not be repeated here. For example, as shown... Figure 4 As shown, the position of the first sub-image in the second window 401 is the second position of the inlet.

[0157] The preset coefficient is a pre-defined threshold used to filter out matching results with high similarity to the second template. For the second sub-image in the second window at each position during the sliding process, the normalized correlation coefficient between the second sub-image and the second template image that generated the second window can be calculated. Second sub-images with a normalized correlation coefficient greater than the preset coefficient are filtered out. When there is only one filtered second sub-image, the position of the filtered second sub-image is determined as the second position of the interpolation. When there are multiple filtered second sub-images, the position of the second sub-image with the largest normalized correlation coefficient among the filtered second sub-images is determined as the second position of the interpolation.

[0158] It is understandable that there may be multiple second sub-images with a normalized correlation coefficient greater than the preset coefficient. When there are multiple second sub-images with a normalized correlation coefficient greater than the preset coefficient, the position of the second sub-image with the largest normalized correlation coefficient is determined as the second position of the inlet, which can more accurately determine the second position of the inlet.

[0159] The steps for calculating the normalized correlation coefficient between the second sub-image and the second template image specifically include: calculating the normalized correlation coefficient based on the pixel values ​​of each pixel in the second sub-image and the pixel values ​​of each pixel in the second template image.

[0160] For example, the formula for calculating the normalized correlation coefficient is shown in the following formula (2):

[0161]

[0162] In formula (2), T(i,j) is the pixel value of the second template image at position (i,j). Let I(x+i,y+j) be the average pixel value of the second template image, and let I(x+i,y+j) be the pixel value of the detected image at position (x+i,y+j). This represents the average pixel value of the second sub-image in the region at position (x, y).

[0163] In the above method, by sliding a second window on the second image to be detected, the second position of the connector is determined based on the normalized correlation coefficient between the second sub-image of the second image to be detected and the second template image within the second window. The sliding window allows for region-by-region matching of the second image to be detected and the second template image, achieving a more refined match and thus improving the accuracy of identifying the second position. By calculating and comparing the normalized correlation coefficient between the second sub-image of the second image to be detected and the second template image within the second window with a preset coefficient, the second position of the connector can be determined more accurately. This enables the robotic arm to more accurately align with the connector based on the second position, improving the success rate of the connector insertion and removal operation.

[0164] S13, upon recognizing the second position, the pose of the robotic arm to be inserted is adjusted again based on the second position.

[0165] If the second position of the connector is identified in the second image to be detected, a new offset pose of the robotic arm to be connected can be determined based on the second position of the connector, and the pose of the robotic arm to be connected can be adjusted based on the new offset pose.

[0166] In one possible implementation, the pose of the robotic arm to be inserted is adjusted again based on the second position, including: adjusting the pose of the robotic arm to be inserted again based on the second position and the second template image.

[0167] As in the above embodiment, the controller stores a second template image. After determining the second position of the socket based on the second template image, a new offset pose can be determined based on the second position and the second template image to adjust the pose of the robotic arm to be inserted.

[0168] Specifically, the second insertion image of the insertion port in the second image to be detected can be determined based on the second position. Based on the second insertion image and the second template image, a new offset pose of the manipulator to be inserted can be determined. Then, the pose of the manipulator to be inserted can be adjusted based on the new offset pose.

[0169] In the above method, the second template image obtained by depth image processing of the socket is used to identify the second position of the socket in the second image to be detected. This can more accurately identify the second position and posture of the socket, which is convenient for subsequent determination of a new offset posture based on the second position to readjust the posture of the robotic arm to be inserted, thus ensuring accurate insertion and removal of the wire.

[0170] In one possible implementation, the pose of the robotic arm to be wired is readjusted based on the second position and the second template image, including: determining the image at the second position in the second image to be detected as the second insertion image; determining multiple second template 3D points included in the second template image; determining a second target 3D point in the second insertion image corresponding to each second template 3D point; determining a second offset pose of the robotic arm to be wired based on each second template 3D point and the second target 3D point corresponding to each second template 3D point; and readjusting the pose of the robotic arm to be wired based on the second offset pose.

[0171] Specifically, a local image at the second position in the second image to be detected can be selected and extracted to obtain the second port image of the target device in the second image to be detected.

[0172] The second template 3D points are a set of pre-set 3D points within the second template image after it has been created using the NCC algorithm. Similar to the first template 3D points, they are typically selected from key locations in the second template image, such as the center of the slot or the four vertices. At least three points are generally set to ensure sufficient constraints for pose estimation. Each second template 3D point also corresponds to a three-dimensional coordinate system (x, y, z). Here, x represents the horizontal position of the 3D point in the image plane, y represents the vertical position of the 3D point in the image plane, and z represents the distance (depth) from the 3D point to the 3D camera.

[0173] After determining the second interposer image, the second target 3D point corresponding to each second template 3D point can be determined from the second interposer image. Each second target 3D point also corresponds to a three-dimensional coordinate (X, Y, Z). The method for determining the second target 3D point corresponding to each second template 3D point from the second interposer image is similar to the method for determining the first target 3D point, and will not be described in detail here.

[0174] The second offset pose also includes rotation and translation. Each second template 3D point and the corresponding second target 3D point are recorded as a matching pair, which can yield multiple sets of matching pairs. Based on these multiple sets of matching pairs, the transformation matrix between the second template 3D point and the second target 3D point can be calculated. The transformation matrix includes rotation and translation components. By extracting the rotation and translation components from the transformation matrix, the second offset pose of the manipulator to be inserted can be determined.

[0175] In the above method, by determining the second target 3D point corresponding to each second template 3D point in the second socket image, the second offset pose of the robot arm to be inserted can be accurately determined. Based on the second offset pose, the robot arm to be inserted is automatically adjusted a second time. The two adjustments can make the robot arm to be inserted more accurately aligned with the socket, thereby improving the success rate of the robot arm to be inserted automatically.

[0176] In one possible implementation, the pose of the robotic arm to be inserted is adjusted again based on the second offset pose, including: determining the template point cloud corresponding to the second template image; determining the current point cloud corresponding to the second insertion image; registering the template point cloud and the current point cloud using the second offset pose to obtain a third offset pose; and adjusting the pose of the robotic arm to be inserted again based on the second offset pose and the third offset pose.

[0177] After obtaining the second offset pose, the depth data corresponding to each pixel in the second template image and the second interposer image can be extracted respectively. Based on the position information and depth data of each pixel, a point cloud is generated to obtain the template point cloud corresponding to the second template image and the current point cloud corresponding to the second interposer image.

[0178] For example, if the depth data corresponding to pixel 1 is extracted as 8 from the second template image, and the position of pixel 1 is (2, 4), then the three-dimensional coordinates of pixel 1 can be determined as (2, 4, 8). By performing similar processing on all pixels in the second template image as on pixel 1, the template point cloud corresponding to the second template image can be obtained. The method for generating the current point cloud is similar to the method for generating the template point cloud, and will not be described in detail here.

[0179] After obtaining the template point cloud and the current point cloud, the current point cloud is first transformed based on the second offset pose to obtain the transformed current point cloud, which is then aligned with the template point cloud. For the transformed current point cloud and the template point cloud, the Iterative Closest Point (ICP) algorithm is used for registration to obtain point cloud matching pairs between the template point cloud and the transformed current point cloud. Each point cloud matching pair includes a point from the current point cloud and a point from the template point cloud. Based on the point cloud matching pairs, the transformation matrix between the transformed current point cloud and the template point cloud can be calculated. Extracting the rotation and translation components from this transformation matrix allows the determination of the third offset pose of the manipulator to be inserted.

[0180] The specific steps for readjusting the pose of the manipulator to be wired based on the second and third offset poses include: adding the third offset pose to the second offset pose to obtain the final offset pose, and then adjusting the pose of the manipulator to be wired based on the determined final offset pose. Alternatively, in some embodiments, the pose of the manipulator to be wired can be adjusted first based on the second offset pose, and then adjusted again based on the third offset pose to complete the readjustment of the manipulator's pose.

[0181] In the above method, when adjusting the pose of the manipulator to be inserted for the second time, the second offset pose is determined by pre-set 3D points, then the template point cloud corresponding to the second template image and the current point cloud corresponding to the second insertion port image are determined. The third offset pose is determined by registering the point cloud with the second offset pose. The manipulator to be inserted is adjusted based on the second and third offset poses, which realizes a more precise pose adjustment of the manipulator to be inserted, thereby reducing the pose adjustment error and significantly improving the operation accuracy of the manipulator to be inserted and increasing the success rate of the operation.

[0182] In some embodiments, if the initial adjustment of the robot arm's offset pose is too large, the socket position may shift out of the 3D camera's field of view. In this case, the second image to be detected will not include the socket of the target device, making it impossible to identify the second position of the socket in the second image. To resolve this, the pose of the robot arm can be adjusted, or the position of the robot arm can be moved. Then, the target device can be photographed again using the 3D camera on the robot arm to obtain the second image to be detected. The second position of the socket is then identified in the second image, and the above steps are repeated until the second position of the socket is identified in the second image.

[0183] S14, after confirming that the robotic arm to be inserted has completed readjustment, control the robotic arm to be inserted to perform the insertion / removal operation.

[0184] Once it's confirmed that the robotic arm to be connected has completed readjustment, it can be determined that the mechanical gripper of the robotic arm and the connector are precisely aligned. At this point, the connection / removal path can be calculated based on the position of the connector and the current pose of the robotic arm. Each point on the path must not collide with other objects in the environment. The calculated path is then converted into control commands for the robotic arm, which should include the angle changes and movement speeds of each joint. The robotic arm is then controlled to perform the connection / removal operation according to the generated path.

[0185] In some embodiments, after determining that the robotic arm to be inserted has completed readjustment, it is necessary to determine again whether the robotic arm can complete the insertion / removal operation in its current posture. Only after determining that the robotic arm can complete the insertion / removal operation in its current posture is the robotic arm controlled to perform the insertion / removal operation. The determination step may include: determining whether the final offset position obtained from the second and third offset poses is less than a preset offset pose; if the final offset position is determined to be less than the preset offset pose, it is determined that the robotic arm can complete the insertion / removal operation in its current posture.

[0186] In the above method, after the first identification and adjustment of the robotic arm to be plugged in, and after determining that the robotic arm cannot complete the plugging and unplugging operation, the image of the target device is acquired again to obtain a second image to be detected. The position of the socket is then identified a second time. Based on the second identified position of the socket, the robotic arm to be plugged in is adjusted again before the plugging and unplugging operation is performed. The second identification and adjustment of the posture can reduce the error that exists in the first identification and adjustment, and ensure that the robotic arm to be plugged in can be more accurately aligned with the socket. This can significantly improve the success rate and reliability of the robotic arm to be plugged in completing the plugging and unplugging operation.

[0187] Figure 5 This is a flowchart illustrating another method for controlling the plug-in / plug-out of the cable provided in this application.

[0188] For example, such as Figure 5 As shown, the method 500 includes:

[0189] Step 501: Determine that the robotic arm to be connected has moved to the photo-taking position;

[0190] The robotic arm to be connected includes a robotic gripper that holds the plug.

[0191] Understandably, the robotic arm to be plugged in is movable, and the photo is usually taken at any point within the movable area where the robotic arm can perform plugging and unplugging operations.

[0192] Step 502: Determine that the target device is located in the target area;

[0193] The target device is the object to which the robotic arm to plug and unplug the wire is to perform the plug-in / plug-out operation. The target device includes a socket corresponding to the plug.

[0194] It is understandable that the robotic arm to be plugged in usually has a certain length and degree of freedom, and the corresponding robotic arm to be plugged in can have a certain range of motion. The target area is the range of motion in which the robotic arm to be plugged in can perform the plugging operation at the position of the photo.

[0195] As in the above embodiment, the controller of the robotic arm to be connected can determine whether the target device is in the target area through sensors on the robotic arm, such as infrared sensors.

[0196] Step 503: Control the 3D camera on the robotic arm to be connected to take a picture and obtain the first image to be detected;

[0197] Among them, the 3D camera is used to capture the target device in the target area, that is, the first image to be detected is the image of the target device.

[0198] Step 504: Perform the first identification and localization based on the first image to be detected to obtain the first identification result;

[0199] The first identification result includes whether the socket is identified in the first image to be detected.

[0200] It is understandable that when the target device's port is blocked by other obstacles, the port will not be included in the first image to be detected, and the port cannot be identified in the first image to be detected.

[0201] In some embodiments, when the 3D camera cannot capture the port of the target device (e.g., the port is located on the back of the target device facing away from it), the port is not included in the first image to be detected, and in this case, the port cannot be identified in the first image to be detected.

[0202] When a socket is identified in the first image to be detected, the first identification result also includes the first position of the socket in the first image to be detected.

[0203] Step 505: Determine whether the socket has been identified based on the first identification result; if yes, proceed to step 506; otherwise, proceed to step 501.

[0204] Understandably, any position within the moving area where the robotic arm can perform the plugging and unplugging operation can be a photo-taking position. If no plug is detected, the robotic arm can be controlled to move within the moving area and take a photo again at another photo-taking position.

[0205] Step 506: Adjust the posture of the robotic arm to be inserted based on the first recognition result, and control the 3D camera to take another picture to obtain the second image to be detected;

[0206] Specifically, the posture of the robotic arm to be inserted can be adjusted by the first position of the insertion port in the first identification result. The specific adjustment process is as described in the above embodiment and will not be repeated here.

[0207] Step 507: Perform a second identification and localization based on the second image to be detected to obtain a second identification result;

[0208] The second identification result includes whether the socket is identified in the second image to be detected.

[0209] Understandably, after adjusting the posture of the robotic arm to be connected based on the first recognition result, the socket of the target device may be obscured by other obstacles. In this case, the socket will not be included in the second image to be detected, and therefore cannot be identified in the second image. Alternatively, if the 3D camera cannot capture the socket of the target device after adjusting the posture of the robotic arm to be connected based on the first recognition result, the socket will also not be included in the second image to be detected, and therefore cannot be identified in the second image.

[0210] When a socket is identified in the second image to be detected, the second identification result also includes the second position of the socket in the second image to be detected.

[0211] Step 508: Determine whether the socket has been identified based on the second identification result; if yes, proceed to step 509; if no, proceed to step 501.

[0212] Step 509: Adjust the posture of the robotic arm to be connected based on the second recognition result;

[0213] Specifically, the posture of the robotic arm to be inserted can be adjusted by the second position of the insertion port in the second identification result. The specific adjustment process is as described in the above embodiment and will not be repeated here.

[0214] Step 510: Determine whether the plug-in / plug-out operation can be performed; if yes, proceed to step 511; otherwise, proceed to step 501.

[0215] After the robot arm to be plugged in adjusts its posture a second time, the controller can control the 3D camera of the robot arm to take a new picture. Based on the new picture, it is determined whether the mechanical gripper of the robot arm to be plugged in is parallel and aligned with the socket. If the mechanical gripper and the socket are confirmed to be parallel and aligned, it is determined that the plugging and unplugging operation can be performed.

[0216] Step 511: Control the robotic arm to be plugged in to perform the plugging and unplugging operation.

[0217] If it is determined that the robotic arm to be plugged in can perform plugging and unplugging operations, the plugging and unplugging path can be calculated based on the position of the plug and the current pose of the robotic arm to be plugged in, and the robotic arm to be plugged in can be controlled to perform plugging and unplugging operations based on the plugging and unplugging path.

[0218] In summary, this application identifies the first position of the connector in the first image to be detected on the target device, automatically adjusts the pose of the robotic arm to be inserted based on the first position, and controls the robotic arm to perform the insertion / removal operation when it is determined that the robotic arm can complete the insertion / removal operation. This automates the insertion / removal operation, reduces the need for human resources, and lowers labor costs. Compared to the large-scale production line modifications in existing technologies, this solution significantly reduces costs. Since the robotic arm automatically adjusts its pose to perform the insertion / removal operation, it can perform the operation for target devices with different incoming materials, and the production line does not need to be modified to adapt to a specific target device, thus improving operational flexibility. After the first identification and adjustment of the robotic arm to be plugged in, if it is determined that the robotic arm cannot complete the plugging and unplugging operation, the target device is imaged again to obtain a second image to be detected, so as to identify the position of the plug for the second time. Based on the second identified position of the plug, the robotic arm to be plugged in is adjusted again, and then the plugging and unplugging operation is performed again. The second identification and adjustment of the posture can reduce the error of the posture adjustment in the first identification, and ensure that the robotic arm to be plugged in can be more accurately aligned with the plug, which can significantly improve the success rate and reliability of the robotic arm to be plugged in to complete the plugging and unplugging operation.

[0219] Figure 6 This is a schematic diagram of the structure of a plug-in / plug-out device provided in an embodiment of this application.

[0220] For example, such as Figure 6 As shown, the device 600 includes:

[0221] The imaging module 601 is used to acquire an image of the target device when it is determined that the target device is in the target area, and obtain a first image to be detected; wherein the target device includes a port;

[0222] The recognition module 602 is used to identify the first position of the socket in the first image to be detected;

[0223] The adjustment module 603 is used to adjust the pose of the robotic arm to be inserted based on the first position when the first position is detected.

[0224] The control module 604 is used to control the robotic arm to perform the wire insertion / removal operation if it is determined that the robotic arm to be inserted can complete the wire insertion / removal operation.

[0225] In one possible implementation, the imaging module 601 is further configured to, if it is determined that the current robotic arm to be plugged in cannot complete the plugging and unplugging operation, acquire an image of the target device to obtain a second image to be detected; the recognition module 602 is further configured to, identify a second position of the connector in the second image to be detected; the adjustment module 603 is further configured to, based on the second position, readjust the pose of the robotic arm to be plugged in when the second position is identified; and the control module 604 is further configured to, when it is determined that the robotic arm to be plugged in has completed the readjustment, control the robotic arm to be plugged in to perform the plugging and unplugging operation.

[0226] In one possible implementation, the recognition module 602 is specifically used to: acquire a first template image, which is obtained by processing a depth image of the socket using a first algorithm; and identify a first position of the socket in a first image to be detected based on the first template image. The adjustment module 603 is specifically used to: adjust the pose of the robotic arm to be connected based on the first position and the first template image.

[0227] In one possible implementation, the recognition module 602 is specifically used to: determine a first window of the same size as the first template image; slide the first window on the first image to be detected; calculate the similarity between the first sub-image and the first template image for each position of the first sub-image in the first window during the sliding process; and determine the first position of the inlet based on the position of the first sub-image whose similarity with the first template image is greater than a preset similarity.

[0228] In one possible implementation, the adjustment module 603 is specifically used to: determine the image at the first position in the first image to be detected as the first interlock image; determine a plurality of first template 3D points included in the first template image; determine a first target 3D point in the first interlock image corresponding to each first template 3D point; determine a first offset pose of the manipulator to be wired based on each first template 3D point and the first target 3D point corresponding to each first template 3D point; and adjust the pose of the manipulator to be wired based on the first offset pose.

[0229] In one possible implementation, the recognition module 602 is specifically used to: acquire a second template image, which is a depth image of the socket obtained by processing a second algorithm; based on the second template image, identify a second position of the socket in a second image to be detected; and the adjustment module 603 is specifically used to: adjust the pose of the robotic arm to be inserted again based on the second position and the second template image.

[0230] In one possible implementation, the recognition module 602 is specifically used to: determine a second window of the same size as the second template image; slide the second window on the second image to be detected; for the second sub-image in the second window at each position during the sliding process, calculate the normalized correlation coefficient between the second sub-image and the second template image; and determine the second position of the interposer based on the position of the second sub-image whose normalized correlation coefficient is greater than a preset coefficient.

[0231] In one possible implementation, the adjustment module 603 is specifically used to: determine the image at the second position in the second image to be detected as the second interlock image; determine a plurality of second template 3D points included in the second template image; determine a second target 3D point corresponding to each second template 3D point in the second interlock image; determine a second offset pose of the manipulator to be interlocked based on each second template 3D point and the second target 3D point corresponding to each second template 3D point; and adjust the pose of the manipulator to be interlocked again based on the second offset pose.

[0232] In one possible implementation, the adjustment module 603 is specifically used to: determine the template point cloud corresponding to the second template image; determine the current point cloud corresponding to the second insertion image; register the template point cloud and the current point cloud using the second offset pose to obtain the third offset pose; and adjust the pose of the robotic arm to be inserted again based on the second offset pose and the third offset pose.

[0233] In one possible implementation, the imaging module 601 is specifically used to control the 3D camera on the robotic arm to be connected to capture images of the target device.

[0234] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0235] For example, such as Figure 7 As shown, the electronic device 700 includes a memory 701 and a processor 702. The memory 701 stores executable program code 7011, and the processor 702 is used to call and execute the executable program code 7011 to perform a method of plugging and unplugging a cable.

[0236] Furthermore, embodiments of this application also protect an apparatus that may include a memory and a processor, wherein the memory stores executable program code, and the processor is used to call and execute the executable program code to perform a method for plugging and unplugging cables provided in embodiments of this application.

[0237] This embodiment can divide the device into functional modules based on the above method example. For example, each module can correspond to a separate function, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0238] When each functional module is divided according to its corresponding function, the device may also include a shooting module, a recognition module, an adjustment module, and a control module. It should be noted that all relevant content regarding the steps involved in the above method embodiments can be referenced from the functional descriptions of the corresponding functional modules, and will not be repeated here.

[0239] It should be understood that the device provided in this embodiment is used to perform the above-described method of plugging and unplugging a wire, and therefore can achieve the same effect as the above-described implementation method.

[0240] When using integrated units, the device may include a processing module and a storage module. When applied to an electronic device, the processing module can be used to control and manage the operation of the electronic device. The storage module can be used to support the execution of relevant program code by the electronic device.

[0241] The processing module may be a processor or a controller, which can implement or execute various exemplary logic blocks, modules, and circuits shown in conjunction with the disclosure of this application. The processor may also be a combination of functions that implement computing capabilities, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc., and the storage module may be a memory.

[0242] In addition, the device provided in the embodiments of this application may specifically be a chip, component or module. The chip may include a connected processor and a memory. The memory is used to store instructions. When the processor calls and executes the instructions, the chip can execute a plug-in / plug-out method provided in the above embodiments.

[0243] This embodiment also provides a computer-readable storage medium storing computer program code. When the computer program code is run on a computer, the computer executes the above-described related method steps to implement the method for plugging and unplugging cables provided in the above embodiment.

[0244] This embodiment also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the method for plugging and unplugging cables provided in the above embodiment.

[0245] In this embodiment, the device, computer-readable storage medium, computer program product, or chip are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0246] Through the above description of the embodiments, those skilled in the art will understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0247] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0248] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for unplugging and plugging a cable, characterized in that, The method includes: Once it is determined that the target device is located in the target area, an image of the target device is acquired to obtain a first image to be detected; wherein, the target device includes a port; Identify the first position of the socket in the first image to be detected; Upon recognizing the first position, the pose of the robotic arm to be connected is adjusted based on the first position. If it is determined that the robotic arm to be plugged in can complete the plugging and unplugging operation, then the robotic arm to be plugged in is controlled to perform the plugging and unplugging operation.

2. The method according to claim 1, characterized in that, The method further includes: If it is determined that the robotic arm to be inserted cannot complete the insertion and removal operation, then the target device is image acquired to obtain a second image to be detected. Identify the second position of the socket in the second image to be detected; Upon recognizing the second position, the pose of the robotic arm to be inserted is adjusted again based on the second position; Once it is determined that the robotic arm to be inserted has completed readjustment, the robotic arm to be inserted is controlled to perform the insertion / removal operation.

3. The method according to claim 1, characterized in that, The step of identifying the first position of the socket in the first image to be detected includes: A first template image is obtained, which is obtained by processing the depth image of the port using a first algorithm; Based on the first template image, identify the first position of the socket in the first image to be detected; Adjusting the pose of the robotic arm to be wired based on the first position includes: Based on the first position and the first template image, the pose of the robotic arm to be wired is adjusted.

4. The method according to claim 3, characterized in that, The step of identifying the first position of the socket in the first image to be detected based on the first template image includes: Determine a first window that is the same size as the first template image; Slide the first window on the first image to be detected, and for the first sub-image in the first window at each position during the sliding process, calculate the similarity between the first sub-image and the first template image; The first position of the socket is determined based on the position of the first sub-image whose similarity to the first template image is greater than a preset similarity.

5. The method according to claim 3, characterized in that, Adjusting the pose of the robotic arm to be wired based on the first position and the first template image includes: The image at the first position in the first image to be detected is determined as the first port image; Determine multiple first template 3D points included in the first template image; In the first socket image, determine the first target 3D point corresponding to each of the first template 3D points; Based on each of the first template 3D points and the first target 3D point corresponding to each of the first template 3D points, the first offset pose of the robotic arm to be inserted is determined. Based on the first offset pose, the pose of the robotic arm to be wired is adjusted.

6. The method according to claim 2, characterized in that, The step of identifying the second position of the socket in the second image to be detected includes: Obtain a second template image, which is a depth image of the socket obtained by processing it using a second algorithm; Based on the second template image, the second position of the socket is identified in the second image to be detected; The step of readjusting the pose of the robotic arm to be wired based on the second position includes: Based on the second position and the second template image, the pose of the robotic arm to be inserted is adjusted again.

7. The method according to claim 6, characterized in that, The step of identifying the second position of the socket in the second image to be detected based on the second template image includes: Determine a second window of the same size as the second template image; Slide the second window on the second image to be detected, and for the second sub-image in the second window at each position during the sliding process, calculate the normalized correlation coefficient between the second sub-image and the second template image; The second position of the port is determined based on the position of the second sub-image whose normalized correlation coefficient is greater than a preset coefficient.

8. The method according to claim 6, characterized in that, The step of readjusting the pose of the robotic arm to be inserted based on the second position and the second template image includes: The image at the second position in the second image to be detected is determined as the second port image; Determine the multiple second template 3D points included in the second template image; In the second interposer image, determine the second target 3D point corresponding to each of the second template 3D points; Based on each second template 3D point and the second target 3D point corresponding to each second template 3D point, the second offset pose of the robotic arm to be inserted is determined. Based on the second offset pose, the pose of the robotic arm to be wired is adjusted again.

9. The method according to claim 8, characterized in that, The step of readjusting the pose of the robotic arm to be wired based on the second offset pose includes: Determine the template point cloud corresponding to the second template image; Determine the current point cloud corresponding to the second interposer image; The template point cloud and the current point cloud are registered using the second offset pose to obtain the third offset pose; Based on the second offset pose and the third offset pose, the pose of the robotic arm to be inserted is adjusted again.

10. The method according to any one of claims 1 to 9, characterized in that, The image acquisition of the target device includes: Control the 3D camera on the robotic arm to be connected to acquire images of the target device.

11. A device for plugging and unplugging cables, characterized in that, The device includes: A camera module is used to capture an image of the target device when it is determined that the target device is in the target area, thereby obtaining a first image to be detected; wherein the target device includes a port; The identification module is used to identify the first position of the socket in the first image to be detected; An adjustment module is used to adjust the pose of the robotic arm to be inserted based on the first position when the first position is detected. The control module is used to control the robotic arm to perform the wire insertion / removal operation if it is determined that the robotic arm to be inserted can complete the wire insertion / removal operation.

12. An electronic device, characterized in that, The electronic device includes: Memory, used to store executable program code; A processor for calling and running the executable program code from the memory, causing the electronic device to perform the method as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed, implements the method as described in any one of claims 1 to 10.

Images