Elevator assistance robot based on visual target positioning

Abstract

Provided is an elevator auxiliary operation robot based on visual target positioning, comprising a visual sensor, a host computer, a motor driver, and a mechanical arm, wherein: the visual sensor is configured to collect a scene image of a current scene, the current scene comprising an elevator button; the host computer is configured to, based on the scene image of the current scene, use a region-based convolutional neural network to obtain a button state of the elevator button and a spatial coordinate of the elevator button in a visual coordinate system based on the visual sensor, and based on the button state of a target elevator button in the elevator button on which the mechanical arm is to perform an operation and the spatial coordinate of the target elevator button in the visual coordinate system, generate a motion instruction for controlling the mechanical arm to perform the operation on the target elevator button; and the motor driver is configured to drive the mechanical arm to move based on the motion instruction, so that the mechanical arm performs the operation on the target elevator button.

Description

Technical Field

[0001] This invention relates generally to the field of automatic control, and more particularly to an elevator-assisted operation robot based on visual target localization. Background Technology

[0002] Typically, a robot system includes a vision sensor, a robotic arm (including the robotic arm body and the end effector), and a main control computer. Due to its operational flexibility, it has been widely used in fields such as industrial assembly and safety explosion protection. Summary of the Invention

[0003] According to an embodiment of the present invention, an elevator-assisted operation robot based on visual target localization includes a visual sensor, a main control computer, a motor driver, and a robotic arm. The visual sensor is configured to acquire scene images of the current scene, including elevator buttons. The main control computer is configured to, based on the scene images of the current scene, use a region-based convolutional neural network to obtain the button states of the elevator buttons and their spatial coordinates in a visual coordinate system based on the visual sensor. Furthermore, based on the button states of the target elevator button to which the robotic arm will perform operations and the spatial coordinates of the target elevator button in the visual coordinate system, the main control computer generates motion commands to control the robotic arm to perform operations on the target elevator button. The motor driver is configured to drive the robotic arm to move based on the motion commands, thereby enabling the robotic arm to perform operations on the target elevator button. Attached Figure Description

[0004] The invention can be better understood from the following description of specific embodiments of the invention in conjunction with the accompanying drawings, wherein: Figure 1 This is a block diagram illustrating the functional modules of an elevator-assisted operation robot based on visual target localization according to an embodiment of the present invention.

[0005] Figure 2 It is shown Figure 1 The flowchart shown is an example of the control process of the master computer over the robotic arm.

[0006] Figure 3 It is shown Figure 1 The flowchart shown illustrates an example of how the main control computer identifies elevator buttons.

[0007] Figure 4 This is a flowchart illustrating an example calibration process for determining the transformation matrix between the visual coordinate system and the robotic arm coordinate system according to an embodiment of the present invention. Detailed Implementation

[0008] The features and exemplary embodiments of various aspects of the present invention will now be described in detail. Numerous specific details are set forth in the following detailed description to provide a comprehensive understanding of the invention. However, it will be apparent to those skilled in the art that the invention may be practiced without requiring some of these specific details. The following description of embodiments is merely intended to provide a better understanding of the invention by illustrating examples of the invention. The invention is by no means limited to any specific configuration and algorithm presented below, but covers any modifications, substitutions, and improvements to elements, components, and algorithms without departing from the spirit of the invention. Well-known structures and techniques are not shown in the drawings and the following description in order to avoid unnecessarily obscuring the invention. Furthermore, it should be noted that the term "A connected to B" as used herein can mean "A and B are directly connected" or "A and B are indirectly connected via one or more other elements."

[0009] Considering the problems of insufficient accuracy, low efficiency, and excessive human intervention in traditional robot systems when using robotic arms to press elevator buttons, an elevator-assisted operation robot based on visual target localization according to an embodiment of the present invention is proposed. By combining a region-based convolutional neural network (R-CNN) in the control of the robotic arm, accurate positioning, state recognition, and efficient pressing of elevator buttons can be achieved, thereby improving the robot's automation level, operational efficiency, and system reliability.

[0010] Figure 1 This is a block diagram illustrating the functional modules of an elevator-assisted operation robot based on visual target localization according to an embodiment of the present invention. Figure 1 As shown, the elevator-assisted operation robot 100 based on visual target localization includes a visual sensor 102, a main control computer 104, a motor driver 106, and a robotic arm 108. The visual sensor 102 is configured to acquire scene images of the current scene, including elevator buttons. The main control computer 104 is configured to, based on the scene images of the current scene, use R-CNN to obtain the button state (e.g., pressed or unpressed) of the elevator buttons and the spatial coordinates of the elevator buttons in a visual coordinate system based on the visual sensor 102. Based on the button state of the target elevator button to which the robotic arm 108 will perform operations and the spatial coordinates of the target elevator button in the visual coordinate system, it generates motion commands to control the robotic arm 108 to perform operations on the target elevator button. The motor driver 106 is configured to drive the robotic arm to move based on the motion commands, so that the robotic arm performs operations on the target elevator button.

[0011] In some embodiments, the main control computer 104 is further configured to generate motion decisions about the robotic arm 108 moving to the target elevator button and performing an operation on the target elevator button, and motion plans about the manner in which the robotic arm 108 moves to the target elevator button, based on the button state of the target elevator button and the spatial coordinates of the target elevator button in the visual coordinate system, and to generate motion instructions based on at least one of the motion decisions and motion plans.

[0012] In some embodiments, the robotic arm 108 includes a robotic arm body and an end effector. The robotic arm body is used to move the end effector to a target elevator button, and the end effector is used to perform an operation on the target elevator button, such as a pressing operation. In this case, the main control computer 104 is also configured to generate the spatial coordinates of the target elevator button in the robotic arm coordinate system based on a transformation matrix between a visual coordinate system and a robotic arm-based coordinate system, and the spatial coordinates of the target elevator button in the visual coordinate system. Furthermore, based on the spatial coordinates of the target elevator button in the robotic arm coordinate system and the current position of the end effector in the robotic arm coordinate system, the computer calculates the joint angles of one or more joints of the robotic arm body using at least one of a forward kinematics algorithm and an inverse kinematics algorithm, wherein the joint angles of one or more joints of the robotic arm body are included in the motion command.

[0013] In some embodiments, the main control computer 104 is also configured to adjust the joint angles of one or more joints of the robotic arm 108 based on the distance between the end effector of the robotic arm 108 and the target elevator button during the movement of the robotic arm 108 driven by the motor driver 106.

[0014] Figure 2 It is shown Figure 1 The flowchart shown illustrates an example of the control process of the main control computer over the robotic arm. Figure 2As shown, in some embodiments, the control process of the main control computer 104 on the robotic arm 108 includes: S202, generating the spatial coordinates of the target elevator button in the robotic arm coordinate system based on the transformation matrix between the visual coordinate system and the robotic arm coordinate system and the spatial coordinates of the target elevator button in the visual coordinate system; S204, calculating the joint angles of one or more joints of the robotic arm body of the robotic arm 108 using at least one of forward kinematics algorithm and inverse kinematics algorithm based on the spatial coordinates of the target elevator button in the robotic arm coordinate system and the current position of the end effector of the robotic arm 108 in the robotic arm coordinate system; S206, controlling the movement of the robotic arm body of the robotic arm 108 based on the calculated joint angles; S208, adjusting the joint angles of one or more joints of the robotic arm body of the robotic arm 108 using a PID control algorithm based on the distance between the end effector of the robotic arm 108 and the target elevator button; and S210, controlling the end effector of the robotic arm 108 to perform an operation on the target elevator button.

[0015] In some embodiments, the main control computer 104 is further configured to: extract multiple feature maps of different sizes from the scene image of the current scene using the backbone network of R-CNN; identify candidate button regions on the smallest feature map among the multiple feature maps using the Region Proposal Network (RPN) of R-CNN, and predict the bounding box and object score of the candidate button regions, wherein the object score of each candidate button region is related to the probability that the candidate button region corresponds to the real elevator button; extract fixed-size feature blocks from the multiple feature maps using the Context-Aware Feature Enhancement (CAFE) module of R-CNN, and perform feature fusion on the feature blocks, candidate button regions, and bounding boxes and object scores of the candidate button regions using a self-attention mechanism to generate feature-enhanced feature blocks; and identify the button state of the elevator button and the spatial coordinates of the bounding box of the elevator button in the visual coordinate system based on the feature-enhanced feature blocks using the detection and state classification module of R-CNN.

[0016] In some embodiments, the backbone network of R-CNN can employ EfficientNet-B0 to extract multiple feature maps {C1, C2, C3, C4} of different sizes from the scene image of the current scene. The RPN can be used to identify candidate button regions on feature map C4. For example, based on Non-Maximum Suppression (NMS), 200 candidate button regions can be identified, and their bounding boxes and object scores can be predicted. The CAFE module can be used to extract fixed-size feature blocks from feature maps {C1, C2, C3, C4} through Region of Interest Alignment (ROI Align) operations. A self-attention mechanism is then used to fuse the feature blocks, candidate button regions, and their bounding boxes and object scores to enhance the robustness of elevator button localization and state recognition.

[0017] In some embodiments, the detection and state classification module of R-CNN can employ a lightweight model based on depthwise separable convolutions. For example, this lightweight model based on depthwise separable convolutions may include a 3x3 depthwise convolutional layer, a 1x1 convolutional layer based on the softmax activation function, and a 1x1 convolutional layer based on the ReLU activation function. The feature map generated by the 3x3 depthwise convolutional layer after processing the feature enhancement blocks is used in the following two processing branches: a button detection branch, where the 1x1 convolutional layer based on the softmax activation function processes the feature map generated by the 3x3 depthwise convolutional layer to identify elevator buttons, and the 1x1 convolutional layer based on the ReLU activation function processes the feature map generated by the softmax activation function to identify the spatial coordinates of the elevator button's bounding box in the visual coordinate system; and a state classification branch, where the 1x1 convolutional layer based on the softmax activation function processes the feature map generated by the 3x3 depthwise convolutional layer to identify the button state of the elevator buttons.

[0018] In some embodiments, a multi-task loss function (including RPN loss, button detection loss, and state classification loss) can be used to train the lightweight model based on depthwise separable convolutions. The generalization ability of the lightweight model based on depthwise separable convolutions can be improved through data augmentation (e.g., random flipping, scaling, color jitter) and transfer learning. Pruning and quantization of the lightweight model based on depthwise separable convolutions can ensure a latency of less than 60 milliseconds per frame.

[0019] Figure 3 It is shown Figure 1 The flowchart shown illustrates an example of how the main control computer identifies elevator buttons. Figure 3 As shown, in some embodiments, the process of recognizing elevator buttons by the main control computer 104 includes: S302, using EfficientNet-B0 to extract multiple feature maps {C1, C2, C3, C4} of different sizes from the scene image of the current scene; S304, using RPN to find candidate button regions on feature map C4 and predict the bounding box and object score of the candidate button regions; S306, using the CAFE module to extract fixed-size feature blocks from multiple feature maps {C1, C2, C3, C4}, and using a self-attention mechanism to perform feature fusion on the feature blocks, candidate button regions, and the bounding box and object score of the candidate button regions to generate feature-enhanced feature blocks; S308, using the detection and state classification module to recognize the button state of the elevator button and the spatial coordinates of the bounding box of the elevator button in the visual coordinate system based on the feature-enhanced feature blocks.

[0020] In some embodiments, the vision sensor 102 is further configured to acquire images of the calibration board, and the main control computer 104 is further configured to acquire the spatial coordinates of multiple feature points of the calibration board in the visual coordinate system based on the images of the calibration board, and to calculate the transformation matrix between the visual coordinate system and the robotic arm coordinate system based on the spatial coordinates of the multiple feature points in the visual coordinate system and the spatial coordinates of the end effector of the robotic arm 108 at multiple predetermined positions in the robotic arm coordinate system.

[0021] Figure 4 This is a flowchart illustrating an example calibration process for determining the transformation matrix between a vision coordinate system and a robotic arm coordinate system according to an embodiment of the present invention. Figure 4 As shown, the calibration process for determining the transformation matrix between the visual coordinate system and the robotic arm coordinate system includes: S402, acquiring an image of the calibration board using the visual sensor 102; S404, obtaining the spatial coordinates of multiple feature points of the calibration board in the visual coordinate system using the main control computer 104 based on the image of the calibration board; S406, combining the spatial coordinates of multiple feature points of the calibration board in the visual coordinate system and the spatial coordinates of the end effector of the robotic arm 108 at multiple predetermined positions in the robotic arm coordinate system, calculating the transformation matrix between the visual coordinate system and the robotic arm coordinate system using the least squares method; and S408, applying the calculated transformation matrix to the elevator-assisted operation robot 100 based on visual target localization, testing whether the robotic arm 108 can be correctly moved to the corresponding position according to the spatial coordinates of the target elevator button detected by the visual sensor 102, and then adjusting the calibration parameters or adding sampling points according to the error. Repeating the above process can establish the transformation matrix between the visual coordinate system and the robotic arm coordinate system.

[0022] In summary, the elevator-assisted operation robot based on visual target localization according to embodiments of the present invention acquires images of the current scene, including elevator buttons, through visual sensors, and uses a region-based convolutional neural network (R-CNN) to detect, recognize, and spatially locate elevator buttons. Combined with robotic arm control technology, it solves the problem of operating elevator buttons. It is suitable for high-precision and high-efficiency automation scenarios, can reduce manual intervention and operating costs, and provides a practical solution for industrial automation.

[0023] This invention can be implemented in other specific forms without departing from its spirit and essential characteristics. For example, the algorithm described in a particular embodiment can be modified without departing from the basic spirit of the invention. Therefore, the present embodiments are to be regarded as exemplary rather than limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and all changes falling within the meaning and scope of the claims and their equivalents are thus included within the scope of the invention.

Claims

1. An elevator-assisted operation robot based on vision target localization, comprising a vision sensor, a main control computer, a motor driver, and a robotic arm, wherein: The visual sensor is configured to acquire scene images of the current scene, wherein the current scene includes elevator buttons; The main control computer is configured to, based on the scene image of the current scene, use a region-based convolutional neural network to obtain the button state of the elevator button and the spatial coordinates of the elevator button in a visual coordinate system based on the vision sensor. Furthermore, based on the button state of the target elevator button on which the robotic arm will perform an operation and the spatial coordinates of the target elevator button in the visual coordinate system, it generates motion commands to control the robotic arm to perform operations on the target elevator button. The motor driver is configured to drive the robotic arm to move based on the motion command, so that the robotic arm performs an operation on the target elevator button. The main control computer is further configured as follows: Using the backbone network of the region-based convolutional neural network, multiple feature maps of different sizes are extracted from the scene image of the current scene; Using the region proposal network of the region-based convolutional neural network, candidate button regions are found on the feature map with the smallest size among the multiple feature maps, and the bounding box and object score of the candidate button regions are predicted. The object score of each candidate button region is related to the probability that the candidate button region corresponds to the real elevator button. The context-aware feature enhancement module of the region-based convolutional neural network extracts fixed-size feature blocks from the multiple feature maps, and uses a self-attention mechanism to fuse the feature blocks, the candidate button region, the bounding box of the candidate button region, and the object score to generate feature-enhanced feature blocks. Using the detection and state classification module of the region-based convolutional neural network, the button state of the elevator button and the spatial coordinates of the bounding box of the elevator button in the visual coordinate system are identified based on the feature enhancement feature block.

2. The elevator-assisted operation robot based on visual target localization according to claim 1, wherein, The backbone network uses EfficientNet-B0.

3. The elevator-assisted operation robot based on visual target localization according to claim 1, wherein, The main control computer is also configured to extract the feature blocks by using the context-aware feature enhancement module through region of interest alignment operation.

4. The elevator-assisted operation robot based on visual target localization according to claim 1, wherein, The detection and state classification module adopts a lightweight model based on depthwise separable convolution.

5. The elevator-assisted operation robot based on visual target localization according to claim 4, wherein, The lightweight model based on depthwise separable convolution includes a 3x3 depthwise convolutional layer, a 1x1 convolutional layer based on the softmax activation function, and a 1x1 convolutional layer based on the ReLU activation function.

6. The elevator-assisted operation robot based on visual target localization according to claim 1, wherein, The main control computer is also configured to generate motion decisions about the robotic arm moving to the target elevator button and performing an operation on the target elevator button, and motion plans about the manner in which the robotic arm moves to the target elevator button, based on the button state of the target elevator button and the spatial coordinates of the target elevator button in the visual coordinate system, and to generate the motion command based on at least one of the motion decisions and the motion plans.

7. The elevator-assisted operation robot based on visual target localization according to claim 6, wherein, The robotic arm includes a robotic arm body and an end effector. The main control computer is also configured to generate the spatial coordinates of the target elevator button in the robotic arm coordinate system based on the transformation matrix between the visual coordinate system and the robotic arm coordinate system based on the robotic arm, and the spatial coordinates of the target elevator button in the visual coordinate system. Based on the spatial coordinates of the target elevator button in the robotic arm coordinate system and the current position of the end effector in the robotic arm coordinate system, the computer calculates the joint angles of one or more joints of the robotic arm body using at least one of forward kinematics algorithm and inverse kinematics algorithm. The joint angles of one or more joints of the robotic arm body are included in the motion command.

8. The elevator-assisted operation robot based on visual target localization according to claim 7, wherein, The main control computer is also configured to adjust the joint angles of one or more joints of the robotic arm body based on the distance between the end effector and the target elevator button during the process of the motor driver driving the robotic arm to move.

9. The elevator-assisted operation robot based on visual target localization according to claim 7, wherein, The vision sensor is also configured to acquire images of the calibration board, and the main control computer is also configured to acquire the spatial coordinates of multiple feature points of the calibration board in the vision coordinate system based on the images of the calibration board, and to calculate the transformation matrix between the vision coordinate system and the robotic arm coordinate system based on the spatial coordinates of the multiple feature points in the vision coordinate system and the spatial coordinates of multiple predetermined positions of the end effector in the robotic arm coordinate system.

Images