Method and apparatus for controlling a robot arm

Abstract

A method and device for controlling a robot arm are disclosed. The method comprises: acquiring an image of a scene including a target object by a single camera; determining a depth value of the target object based on a depth curve relationship between an image size of the target object and the depth value and the image size of the target object in the acquired image, the depth value of the target object indicating an actual distance between the target object and the camera; determining a relative position of the target object in an image plane coordinate system of the camera based on an image scale and an image position of the target object in the acquired image; and controlling the robot arm to move and perform an operation on the target object based on the depth value of the target object, the relative position of the target object in the image plane coordinate system of the camera, and a relative spatial position of the camera and the robot arm.

Description

Technical Field

[0001] This invention relates to the field of robotic arm control, and more particularly to a method and apparatus for controlling a robotic arm. Background Technology

[0002] When performing operations, robotic arms need to locate target objects. Conventional localization is usually achieved by using images of the target object captured by an RGBD depth camera for target detection and depth acquisition. However, RGBD depth cameras are expensive and have certain limitations in terms of space, and cannot acquire depth information when the distance to the target object is very close (e.g., less than 10cm). Summary of the Invention

[0003] A method for controlling a robotic arm according to an embodiment of the present invention includes: acquiring an image of a scene including a target object using a single camera; determining a depth value of the target object based on a depth curve relationship between the image size and depth value of the target object and the image size of the target object in the acquired image, wherein the depth value of the target object indicates the actual distance between the target object and the camera; determining the relative position of the target object in the image plane coordinate system of the camera based on the image ratio and image position of the target object in the acquired image; and controlling the robotic arm to move and perform an operation on the target object based on the depth value of the target object, the relative position of the target object in the image plane coordinate system of the camera, and the relative spatial position of the camera and the robotic arm.

[0004] An apparatus for controlling a robotic arm according to an embodiment of the present invention includes: a processor; and a memory storing computer-executable instructions thereon, wherein the computer-executable instructions, when executed by the processor, cause the processor to perform the method for controlling the robotic arm described above.

[0005] According to an embodiment of the present invention, a computer-readable storage medium stores computer-executable instructions thereon, wherein, when executed by a processor, these computer-executable instructions cause the processor to perform the above-described method for controlling a robotic arm.

[0006] A computer program product according to an embodiment of the present invention includes computer-executable instructions, wherein, when executed by a processor, these computer-executable instructions cause the processor to perform the above-described method for controlling a robotic arm. Attached Figure Description

[0007] The invention can be better understood from the following description of specific embodiments of the invention in conjunction with the accompanying drawings, wherein:

[0008] Figure 1 A schematic flowchart of a method for controlling a robotic arm according to an embodiment of the present invention is shown.

[0009] Figure 2 Another schematic flowchart of a method for controlling a robotic arm according to an embodiment of the present invention is shown.

[0010] Figure 3 A schematic diagram of a computer system is shown that can implement the method and apparatus for controlling a robotic arm according to embodiments of the present invention. Detailed Implementation

[0011] The features and exemplary embodiments of various aspects of the present invention will now be described in detail. Numerous specific details are set forth in the following detailed description to provide a thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention may be practiced without requiring some of these specific details. The following description of embodiments is merely intended to provide a better understanding of the invention by illustrating examples of the invention. The invention is by no means limited to any specific configurations and algorithms presented below, but covers any modifications, substitutions, and improvements to elements, components, and algorithms without departing from the spirit of the invention. Well-known structures and techniques are not shown in the drawings and the following description in order to avoid unnecessarily obscuring the invention.

[0012] Considering the high cost and spatial limitations of conventional positioning methods using RGBD depth cameras, a method and apparatus for controlling a robotic arm according to embodiments of the present invention are proposed. This method involves acquiring an image of a scene, including a target object, using a single camera. The depth value of the target object is determined based on the depth curve relationship between its image size and depth value, and its image size within the acquired image. The relative position of the target object in the camera's image plane coordinate system is determined based on its image proportion and position within the acquired image. This allows for the positioning of the target object and the control of the robotic arm's movement and the execution of operations on the target object. Since the above technical solution can be achieved using a single conventional RGB camera, the determination of the depth value is not limited by spatial distance, and the cost is significantly reduced.

[0013] Figure 1 A schematic flowchart of a method for controlling a robotic arm according to an embodiment of the present invention is shown. Figure 1As shown, the method 100 for controlling a robotic arm according to an embodiment of the present invention includes: S101: acquiring an image of a scene including a target object through a single camera; S102: determining the depth value of the target object based on the depth curve relationship between the image size and depth value of the target object and the image size of the target object in the acquired image, wherein the depth value of the target object indicates the actual distance between the target object and the camera; S103: determining the relative position of the target object in the image plane coordinate system of the camera based on the image ratio and image position of the target object in the acquired image; and S104: controlling the movement of the robotic arm and performing operations on the target object based on the depth value of the target object, the relative position of the target object in the image plane coordinate system of the camera, and the relative spatial position of the camera and the robotic arm.

[0014] In the method according to embodiments of the present invention, a single camera does not need to have the ability to acquire depth information; the camera can be a conventional RGB camera, which is less expensive than an RGBD camera.

[0015] In the method according to the embodiments of the present invention, the target object can be an object with a defined shape, such as a regular spatial geometric body such as a prism, pyramid, cylinder, cone or sphere, or the visible surface of the object that can be captured by the camera is a plane (e.g., a physical button or operation panel).

[0016] In the method according to an embodiment of the present invention, for a specific target object, different depth values ​​and the image size of the target object in the acquired images corresponding to each depth value can be obtained in advance, and a depth curve relationship can be obtained by curve fitting based on the pre-acquired data. Specifically, the depth curve relationship can be a curve relationship represented by a power function.

[0017] In some embodiments, the process of determining the power function representing the depth curve relationship includes: acquiring multiple sets of depth curve data pairs, each set including a depth value and the image size of the target object in the acquired image corresponding to that depth value; performing logarithmic processing on the multiple sets of depth curve data pairs with a preset number as the base to obtain multiple sets of processed data pairs; determining the slope and intercept of the fitted line corresponding to the multiple sets of processed data pairs using the least squares method; and determining the power function based on the preset number, the slope, and the intercept. Here, the image size can refer to the pixel value of at least one of the long or wide sides of the target object's image, or it can be the width pixel value of the detection box of the target object.

[0018] For example, acquire 30 pairs of depth curve data. The i-th pair (where i is an integer greater than or equal to 1 and less than or equal to 30) includes the depth value `depth_i` and the image size `width_i` of the target object in the acquired image corresponding to that depth. Using a preset natural number `e` as the base, perform logarithmic processing on the multiple pairs of depth curve data to obtain multiple processed data pairs. The i-th processed data pair can be represented as lg(depth_i) and lg(width_i). Fit a straight line using the least squares method and calculate the slope `b` and intercept `c`, using the following formulas:

[0019]

[0020] Where, x i Let lg(depth_i), Let y be the average of all lg(depth_i). i Let lg(width_i) be the width. It is the average of all lg(width_i).

[0021] Based on the preset natural number e, slope b, and intercept c, the power function can be determined as follows:

[0022] depth = e c ×width b ;

[0023] Where depth is the depth value and width is the image size.

[0024] In the method according to embodiments of the present invention, the image proportion of the target object in the acquired image refers to the actual physical length corresponding to a unit length (e.g., a single pixel) of the image size of the target object in the acquired image. For a specific target object, the image proportion can be determined by calculation using the known actual size and the corresponding image size obtained from the acquired image, or it can be determined by a proportional curve relationship similar to the depth curve relationship described above.

[0025] In some embodiments, the method for controlling the robotic arm may further include: determining the image proportion of the target object in the acquired image based on the proportional curve relationship between the depth value of the target object and the image proportion of the target object in the acquired image and the depth value of the target object.

[0026] In some embodiments, the proportional curve relationship can be a curve relationship represented by a power function. Specifically, the process of determining the power function representing the proportional curve relationship includes: acquiring multiple sets of proportional curve data pairs, each set including a depth value and the image proportion of the target object in the acquired image corresponding to that depth value; performing logarithmic processing on the multiple sets of proportional curve data pairs with a preset number as the base to obtain multiple sets of processed data pairs; determining the slope and intercept of the fitted line corresponding to the multiple sets of processed data pairs using the least squares method; and determining the power function based on the preset number, the slope, and the intercept. Here, the power function representing the proportional curve relationship can be expressed as: length = e m ×depth n , where length is the image scale, depth is the depth value, and m and n are the intercept and slope of the fitted line, respectively.

[0027] In some embodiments, the origin of the camera's image plane coordinate system corresponds to the center point of the acquired image. Determining the relative position of the target object in the camera's image plane coordinate system includes: determining the vertical image distance and the horizontal image distance between the image position of the target object in the acquired image and the center point of the acquired image; and determining the vertical actual distance corresponding to the vertical image distance and the horizontal actual distance corresponding to the horizontal image distance based on the image ratio of the target object, wherein the position of the center of the detection box of the target object in the acquired image can be used as the image position of the target object in the acquired image.

[0028] In the method according to this embodiment, knowing the depth value of the target object and its relative position in the image plane coordinate system of the camera, the three-dimensional spatial relationship between the camera and the target object can be determined. Further incorporating the relative spatial relationship between the camera and the robotic arm, the robotic arm can be controlled to move towards the target object and perform operations on it. Specifically, the target object can be a physical button, such as an elevator button. Controlling the robotic arm to perform operations on the target object can include controlling the end effector of the robotic arm to perform a pressing operation on the physical button.

[0029] In some embodiments, the camera may be located at the end of the robotic arm and follow the movement of the end of the robotic arm. In this case, controlling the movement of the robotic arm and performing operations on the target object includes: controlling the movement of the end of the robotic arm and performing operations on the target object based on the depth value of the target object and the relative position of the target object in the image plane coordinate system of the camera.

[0030] In the method according to embodiments of the present invention, controlling the end effector movement of the robotic arm can be achieved through gradual approximation using proportional-integral-derivative (PID) adjustment, or it can be achieved quickly through two positioning steps: first, the end effector of the robotic arm moves to the vicinity of the target object, and second, the end effector of the robotic arm moves to the precise position of the target object, thus improving efficiency while ensuring positioning accuracy. Specifically, controlling the end effector movement of the robotic arm and performing operations on the target object may include: in the case of controlling the end effector movement of the robotic arm for the first time for the target object, recording the difference between the depth value of the target object and a preset value as the updated depth value of the target object; based on the updated depth value of the target object and the relative position of the target object in the image plane coordinate system of the camera, controlling the end effector movement of the robotic arm, and returning to the step of acquiring an image of the scene including the target object; in the case of controlling the end effector movement of the robotic arm for the second time for the target object, based on the depth value of the target object and the relative position of the target object in the image plane coordinate system of the camera, controlling the end effector movement of the robotic arm and performing operations on the target object.

[0031] Figure 2 Another schematic flowchart of a method for controlling a robotic arm according to an embodiment of the present invention is shown. Figure 2 As shown, taking elevator buttons as an example, the camera is located at the end of the robotic arm and moves with the end of the robotic arm. The control of the robotic arm involves a camera thread and a robotic arm thread running in parallel. The camera thread acquires images in real time and uses a target detection algorithm to detect elevator buttons, while the robotic arm thread acquires the position coordinates of the robotic arm and controls the movement of the robotic arm.

[0032] like Figure 2 As shown, the process of controlling the robotic arm is as follows:

[0033] (1) The initial position of the end effector of the robotic arm is set to a location closest to the button panel where all elevator buttons can be seen. Record the coordinates (x0, y0, z0) of the end effector in the robotic arm's coordinate system, where x0 is the coordinate on the horizontal axis, y0 is the coordinate on the horizontal axis, and z0 is the coordinate on the vertical axis. This coordinate value only needs to be recorded once, and all subsequent operations will use this coordinate value as the initial position.

[0034] (2) The robotic arm thread sends the elevator button number to be pressed to the camera thread. After receiving the elevator button number, the camera thread returns the width pixel value w of the detection box of the corresponding elevator button, as well as the horizontal image distance x_diff and the vertical image distance y_diff between the center of the detection box and the center point of the acquired image.

[0035] (3) After receiving the above return value, the robotic arm thread uses the width pixel value w of the detection box corresponding to the elevator button as the image size of the elevator button in the acquired image. Based on the depth curve relationship, it determines the depth value y_arm of the camera's current distance from the elevator button. It then uses x_diff, y_diff, and the image ratio to determine the actual horizontal distance x_arm and the actual vertical distance z_arm between the camera and the elevator button. Here, the horizontal image distance x_diff and the vertical image distance y_diff, as well as the actual horizontal distance x_arm and the actual vertical distance z_arm, are all represented by positive and negative signs.

[0036] (4) When the robotic arm thread first controls the end effector movement of the robotic arm in response to the target elevator button, obtain the current coordinates (x, y, z) of the end effector. Subtract the preset value (e.g., 80 mm) from the depth value y_arm and record the difference as the updated depth value (i.e., y_arm-80). Determine the new coordinates of the end effector after this movement as (x+x_arm, y+y_arm-80, z+z_arm). Control the end effector of the robotic arm to move to this new coordinate. Return to step (2).

[0037] When the robotic arm thread controls the end effector of the robotic arm to move for the second time in response to the target elevator button, it obtains the current coordinates (x, y, z) of the end effector and determines the new coordinates of the end effector after this movement to be (x+x_arm, y+y_arm, z+z_arm). It then controls the end effector of the robotic arm to move to these new coordinates and performs a button press operation on the elevator button.

[0038] (5) After the tap operation is completed, the robotic arm returns to the initial position and waits for the next target button number.

[0039] Figure 3 A schematic diagram of a computer system is shown that can implement the method and apparatus for controlling a robotic arm according to embodiments of the present invention. It should be understood that... Figure 3 The computer system 300 shown is merely an example and should not impose any limitation on the functionality and scope of use of the method and apparatus for controlling a robotic arm according to embodiments of the present invention.

[0040] like Figure 3As shown, the computer system 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the computer system 300. The processing device 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0041] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, cameras, accelerometers, gyroscopes, sensors, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, motors, electronic speed controllers, etc.; storage devices 308 including, for example, flash cards; and communication devices 309. Communication device 309 allows computer system 300 to communicate wirelessly or wiredly with other devices to exchange data. Although... Figure 3 A computer system 300 with various devices is shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have instead. Figure 3 Each box shown can represent a device or multiple devices as needed.

[0042] In particular, according to some embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer programs. For example, a computer-readable medium is provided having a computer program stored thereon, the computer program comprising methods for executing... Figure 1 The program code shown is for a method of controlling a robotic arm. In such an embodiment, the computer program can be downloaded and installed from a network via communication device 309, or installed from storage device 308, or installed from ROM 302. When the computer program is executed by processing device 301, the aforementioned functional units defined in the apparatus for controlling a robotic arm according to an embodiment of the present invention are implemented.

[0043] It should be noted that the computer-readable medium according to embodiments of the present invention may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. The computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. The computer-readable storage medium according to embodiments of the present invention may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. Furthermore, the computer-readable signal medium according to embodiments of the present invention may include a data signal propagated in baseband or as part of a carrier wave, wherein computer-readable program code is carried. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (Radio Frequency), etc., or any suitable combination thereof.

[0044] Computer program code for performing operations according to embodiments of the present invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0045] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0046] This invention can be implemented in other specific forms without departing from its spirit and essential characteristics. For example, the algorithm described in a particular embodiment can be modified without departing from the basic spirit of the invention. Therefore, the present embodiments are to be regarded as exemplary rather than limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and all changes falling within the meaning and scope of the claims and their equivalents are thus included within the scope of the invention.

Claims

1. A method for controlling a robotic arm, comprising: Acquire images of the scene, including the target object, using a single camera; Based on the depth curve relationship between the image size and depth value of the target object and the image size of the target object in the acquired image, the depth value of the target object is determined, and the depth value of the target object indicates the actual distance between the target object and the camera. Based on the image proportion and image position of the target object in the acquired image, the relative position of the target object in the image plane coordinate system of the camera is determined; as well as Based on the depth value of the target object, the relative position of the target object in the image plane coordinate system of the camera, and the relative spatial position of the camera and the robotic arm, the movement of the robotic arm is controlled and operations are performed on the target object. The depth curve relationship is represented by a power function, which is determined through the following steps: Acquire multiple sets of depth curve data pairs, each set of depth curve data pairs including a depth value and the image size of the target object in the acquired image corresponding to the depth value; Using a preset number as the base, logarithmic processing is performed on the multiple sets of depth curve data pairs to obtain multiple sets of processed data pairs; The slope and intercept of the fitted line corresponding to the multiple sets of processed data pairs were determined using the least squares method; and The power function is determined based on the preset number, the slope, and the intercept.

2. The method according to claim 1, wherein, Also includes: Based on the proportional curve relationship between the depth value of the target object and the image proportion of the target object in the acquired image, and the depth value of the target object, the image proportion of the target object in the acquired image is determined.

3. The method according to claim 2, wherein, The proportional curve relationship is a curve relationship represented by a power function.

4. The method according to claim 3, wherein, The power function is determined through the following steps: Acquire multiple sets of scale curve data pairs, each set of scale curve data pairs including a depth value and the image proportion of the target object in the acquired image corresponding to the depth value; Using a preset number as the base, logarithmic processing is performed on the multiple sets of proportional curve data pairs to obtain multiple sets of processed data pairs; The slope and intercept of the fitted line corresponding to the multiple sets of processed data pairs are determined by the least squares method. as well as The power function is determined based on the preset number, the slope, and the intercept.

5. The method according to claim 1, wherein, The origin of the camera's image plane coordinate system corresponds to the center point of the acquired image. Determining the relative position of the target object in the camera's image plane coordinate system includes: Determine the vertical and horizontal image distances between the target object's position in the acquired image and the center point of the acquired image; and Based on the image ratio of the target object, determine the actual vertical distance corresponding to the vertical image distance and the actual horizontal distance corresponding to the horizontal image distance.

6. The method according to claim 1, wherein, The camera is located at the end of the robotic arm and moves with the end of the robotic arm. Controlling the movement of the robotic arm and performing operations on the target object includes: controlling the end of the robotic arm to move and performing operations on the target object based on the depth value of the target object and the relative position of the target object in the image plane coordinate system of the camera.

7. The method according to claim 6, wherein, Controlling the end effector movement of the robotic arm and performing operations on the target object includes: In the first case of controlling the end effector movement of the robotic arm for the target object, the difference between the depth value of the target object and the preset value is recorded as the updated depth value of the target object. Based on the updated depth value of the target object and the relative position of the target object in the image plane coordinate system of the camera, the end effector movement of the robotic arm is controlled, and the step of acquiring an image of the scene including the target object is returned. In the second instance of controlling the end effector movement of the robotic arm for the target object, the end effector movement of the robotic arm is controlled and an operation is performed on the target object based on the depth value of the target object and the relative position of the target object in the image plane coordinate system of the camera.

8. The method according to claim 1, wherein, The target object is a physical button, and controlling the robotic arm to perform operations on the target object includes: Control the end of the robotic arm to perform a press operation on the physical button.

9. The method according to claim 1, wherein, The camera is an RGB camera.

10. A device for controlling a robotic arm, comprising: processor; as well as A memory having stored computer-executable instructions thereon, wherein, when executed by the processor, the computer-executable instructions cause the processor to perform the method of controlling a robotic arm as described in any one of claims 1 to 9.

11. A computer-readable storage medium having stored thereon computer-executable instructions, wherein, When executed by a processor, the computer-executable instructions cause the processor to perform the method of controlling the robotic arm as described in any one of claims 1 to 9.

12. A computer program product comprising computer-executable instructions, wherein, When executed by a processor, the computer-executable instructions cause the processor to perform the method of controlling the robotic arm as described in any one of claims 1 to 9.

Images