Automated robotic arm system

Abstract

This invention provides an automated robotic arm system and a method for coordinating the robotic arm with computer vision. A beam splitter separates visible light and ranging light, guiding them to image acquisition devices and optical ranging devices on different axes. In calibration mode, a transformation relationship is calculated based on multiple calibration postures of the robotic arm and corresponding calibration images. In working mode, the robotic arm's movement is controlled by determining its spatial coordinates based on the captured working images and the transformation relationship. This invention can overlay the imaging optical axis with the robotic arm's flange axis, improving the coordination between the robotic arm and computer vision.

Description

Technical Field

[0001] This invention relates to robotic arms, and more particularly to automated robotic arm systems and methods for coordinating the robotic arm with computer vision. Background Technology

[0002] In existing robotic arm systems, cameras are used to capture images of the work object, the position of the work object is determined through image analysis, and the robotic arm is controlled to move to the determined position to perform actions on the work object.

[0003] However, the drawback of existing robotic arm systems is that if the robotic arm and the camera are configured coaxially, one above the other, it will severely reduce the machining range of the robotic arm and severely limit the upper limit of the camera's size.

[0004] If the robotic arm and camera are set up non-coaxially, there will be an offset between the flange axis of the robotic arm and the optical axis of the camera. This offset will cause random errors between the visual space based on the captured image and the mechanical space of the robotic arm, making it impossible for computer vision to accurately control the robotic arm.

[0005] Therefore, existing robotic arm systems have the aforementioned problems, and there is an urgent need for more effective solutions. Summary of the Invention

[0006] The main objective of this invention is to provide an automated robotic arm system and a method for coordinating the robotic arm with computer vision, which enables the imaging optical axis to overlap with the flange axis, measure the target distance, and coordinate the robotic arm with computer vision.

[0007] In one embodiment, a coordination method between a robotic arm and computer vision includes: in a calibration mode, controlling a robotic arm to move into multiple calibration postures within an effective shooting range of an image acquisition device based on a target distance measured by an optical ranging device, and capturing multiple calibration images in the multiple calibration postures using the image acquisition device, wherein a beam splitter guides visible light to the image acquisition device located outside a flange axis of the robotic arm and guides ranging light to the optical ranging device, the ranging axis of the optical ranging device being parallel to or overlapping the flange axis; calculating a transformation relationship between the visual space of the image acquisition device and the mechanical space of the robotic arm based on the multiple calibration postures and the multiple calibration images; in a working mode, capturing a working image using the image acquisition device, and determining a mechanical space coordinate for performing work based on the working image and the transformation relationship; and controlling the robotic arm to move to the mechanical space coordinate.

[0008] In one embodiment, an automated robotic arm system includes a robotic arm, an image acquisition device, an optical ranging device, an optical path structure, and a control device. The robotic arm is used to move in a three-dimensional space. The image acquisition device is disposed outside the flange axis of the robotic arm and is used to capture images. The optical ranging device is disposed on the robotic arm and is used to measure a target distance; the ranging axis of the optical ranging device is parallel to or overlaps with the flange axis. The optical path structure includes a beam splitter, which guides visible light to the image acquisition device and guides the ranging light to the optical ranging device. The control device connects the robotic arm, the image acquisition device, and the optical ranging device. The control device is configured to, in a calibration mode, control the robotic arm to move into multiple calibration postures within an effective shooting distance of the image acquisition device based on the target distance, and control the image acquisition device to capture multiple calibration images in each of the multiple calibration postures. Based on the multiple calibration postures and the multiple calibration images, a conversion relationship between the visual space of the image acquisition device and the mechanical space of the robotic arm is calculated. The control device is set to, in a working mode, control the image acquisition device to capture a working image, determine a mechanical space coordinate for performing the work based on the working image and the conversion relationship, and control the robotic arm to move to the mechanical space coordinate.

[0009] This invention can superimpose the imaging optical axis and the flange axis of the robotic arm, and improve the coordination between the robotic arm and computer vision. Attached Figure Description

[0010] Figure 1 This is an architectural diagram of an automated robotic arm system according to an embodiment of the present invention;

[0011] Figure 2 This is a partial architecture diagram of an automated robotic arm system according to an embodiment of the present invention;

[0012] Figure 3 This is an architectural diagram of a control device according to an embodiment of the present invention;

[0013] Figure 4 This is a flowchart of a coordination method according to an embodiment of the present invention;

[0014] Figure 5 This is a partial flowchart of a coordination method according to an embodiment of the present invention;

[0015] Figure 6 This is a partial flowchart of a coordination method according to an embodiment of the present invention;

[0016] Figure 7 This is a schematic diagram illustrating the setup of an automated robotic arm system according to an embodiment of the present invention;

[0017] Figure 8This is a first schematic diagram of a correction mode according to an embodiment of the present invention;

[0018] Figure 9 This is a second schematic diagram of a correction mode according to an embodiment of the present invention;

[0019] Figure 10 for Figure 8 A schematic diagram of the first corrected image captured;

[0020] Figure 11 for Figure 9 A schematic diagram of the second corrected image captured;

[0021] Figure 12 A schematic diagram of the setup of an existing robotic arm system;

[0022] Figure 13 This is a schematic diagram showing the field of view of an existing robotic arm system.

[0023] Symbol Explanation

[0024] 1: Robotic arm system

[0025] 10: Robotic Arm

[0026] 11: End effector

[0027] 12: Camera

[0028] 13: Objective

[0029] 140: Flange Shaft

[0030] 141: Shooting the optical axis

[0031] 15: Target Image

[0032] 16: Scope

[0033] 17: Range after rotation

[0034] 18: Axis

[0035] 2: Automated robotic arm system

[0036] 20: Control device

[0037] 21: Image acquisition device

[0038] 210: Photosensitive element

[0039] 211: Lens

[0040] 22: Optical ranging device

[0041] 220: Light emitter

[0042] 221: Optical Receiver

[0043] 23: Robotic Arm

[0044] 230-233: Joints

[0045] 24: Optical Path Structure

[0046] 240: Beam Spectroscope

[0047] 241: Reflector

[0048] 25: Storage device

[0049] 250: Computer program

[0050] 251: Effective shooting distance

[0051] 252: Conversion Relation

[0052] 26: Target

[0053] 30: Controlling the computer

[0054] 31: Robotic Arm Controller

[0055] 32: Peripheral devices

[0056] 40: Shooting Control Module

[0057] 41: Distance Measurement Control Module

[0058] 42: Arm Control Module

[0059] 43: Calibration Control Module

[0060] 44: Work Control Module

[0061] 45: Conversion Processing Module

[0062] 46: Image Analysis Module

[0063] 50: Light source

[0064] 51: Target

[0065] 52: Jig

[0066] 53: Flange Shaft

[0067] 54: Working device

[0068] 55: Mounting base

[0069] 60: Image

[0070] 600-601: Location

[0071] α1: Rotation angle

[0072] d1: Bias

[0073] h1: Target distance

[0074] P1, P2: Posture

[0075] V1: Change

[0076] S10-S16: Coordination Steps

[0077] S20-S25: ​​Calibration Steps

[0078] S30-S33: Working Procedures Detailed Implementation

[0079] The following is a detailed description of a preferred embodiment of the present invention, in conjunction with the accompanying drawings.

[0080] Please see Figure 12 and Figure 13 , Figure 12 This is a schematic diagram of the setup of an existing robotic arm system. Figure 13 This is a schematic diagram showing the field of view of an existing robotic arm system.

[0081] like Figure 12 As shown, the camera 12 and the robotic arm 10 of the robotic arm system 1 are configured with different axes. Due to the different axis configuration, there is an offset d1 between the shooting optical axis 141 of the camera 12 and the flange axis 140 of the robotic arm 10. The offset d1 will cause random errors in the positioning of the visual space and the mechanical space.

[0082] The end effector 11 is directly mounted on the end of the robotic arm 10. When the robotic arm 10 moves the end effector 11 (i.e. changes its posture), the field of view of the camera 12 mounted on the robotic arm also changes accordingly, allowing the target 13 to be photographed from different angles.

[0083] like Figure 13 As shown, when the robotic arm 10 rotates at an angle α1 with the flange shaft 140 as the axis 18, the end effector 11 does not move in the horizontal direction, so it can still process the target 13.

[0084] However, the field of view of the rotated camera 12 will change from range 16 to range 17 after rotation, causing the target image 15 of the target 13 to fall out of the field of view of the camera 12, which will prevent the robotic arm system 1 from visually locating the target 13.

[0085] To address the problems caused by the aforementioned different axis configurations, this invention proposes an automated robotic arm system and a method for coordinating the robotic arm with computer vision. This method utilizes a novel optical path structure (especially a beam splitter) to allow the incident end of the imaging optical axis to align with the flange axis, thereby achieving a hand-eye coaxial effect.

[0086] Furthermore, since the present invention can achieve hand-eye coaxiality, it can eliminate the offset problem caused by different axis configurations, and thus prevent the target object from exceeding the field of vision.

[0087] Furthermore, by employing different axis configurations, this invention can significantly increase the processing range of the robotic arm and greatly increase the upper limit of the camera's volume.

[0088] Furthermore, this invention can also improve the coordination between the robotic arm and computer vision by using optical ranging to assist in positioning correction between visual space and machinery.

[0089] Please see Figure 1 This is an architectural diagram of an automated robotic arm system according to an embodiment of the present invention. The automated robotic arm system 2 of the present invention mainly includes an image acquisition device 21, an optical ranging device 22, a robotic arm 23, a storage device 25, and a control device 20 connected to the above devices.

[0090] Image acquisition device 21, such as a color camera like an RGB camera, is used to capture images of the target in the working area to obtain color images containing the target (such as the correction image and working image described later). The aforementioned color images are mainly used to perform computer vision analysis and provide the calculation results as a motion reference for the robotic arm 23.

[0091] The robotic arm 23 is used to move the mounted device in three-dimensional space to perform tasks such as measurement (mounted optical rangefinder 22), photography (mounted image acquisition device 21), and processing (mounted working device 54) at different positions.

[0092] The end effector of the robotic arm 23 is equipped with a virtual flange axis (e.g., a reference point for the movement of the robotic arm 23), and the spatial position of its end effector can be calculated and determined based on the flange axis. The aforementioned calculation of the flange axis is prior art in the field of robotic arm 23 control and will not be elaborated here.

[0093] In this invention, the image acquisition device 21 is disposed outside the flange shaft of the robotic arm 23, thereby increasing the machinable range of the robotic arm 23 (determined by the movable range of the flange shaft) and raising the upper limit of the allowable volume of the image acquisition device 21, that is, a larger and more powerful camera can be used, and the wiring restrictions are more relaxed.

[0094] In one embodiment, when the end effector 54 of the robotic arm 23 is attached (e.g., a working device 54), Figure 8 and Figure 9 As shown, through the movement of the robotic arm 23, the working device 54 can perform processing at different positions. By equipping different working devices 54, the present invention can achieve different applications.

[0095] In one embodiment, the working device 54 may be connected to and controlled by the control device 20 to perform automated actions.

[0096] For example, when the working device 54 is a clamping end effector, welding heater, marking tool, grinding tool, assembling end effector, gluing tool and / or locking tool, the aforementioned automated actions may be corresponding clamping actions (e.g., clamping or picking up electronic components), welding actions (e.g., controlling the heating of a laser welding head), marking actions (e.g., marking by branding, spraying, etc.), grinding actions (e.g., performing cutting, grinding, etc.), assembly actions (e.g., splicing, stacking, etc. of multiple targets according to a specified assembly method), gluing actions (e.g., applying glue, dispensing glue, etc.) and / or locking actions (e.g., locking screws, nuts).

[0097] An optical ranging device 22, such as an infrared rangefinder, is used to measure the target distance between the optical ranging device 22 and the target by optical means.

[0098] In one embodiment, the aforementioned measurement involves positioning the target on a virtual ranging axis and using triangulation to obtain a measurement in a direction parallel to the ranging axis.

[0099] In one embodiment, the optical ranging device 22 is disposed at the end (or near the end) of the robotic arm 23 and can measure the distance between the end and the target.

[0100] In one embodiment, the ranging axis of the optical ranging device 22 may be parallel to or overlap with the flange axis of the robotic arm 23, so that the measured target distance is the depth value between the end of the robotic arm 23 and the target directly below it in the corresponding flange axis.

[0101] The optical path structure 24 is located at the end (or near the end) of the robotic arm 23 to receive incident light (light emitted or reflected from the target), divides the incident light into visible light and ranging light, and guides them to the image acquisition device 21 and the optical ranging device 22, respectively.

[0102] Specifically, the optical path structure 24 includes a beam splitter 240 (such as...). Figures 7-9For example, an optical prism), the beam splitter 240 can separate incident light into rays of different wavelengths (the principle is that rays of different wavelengths have different refractive indices), for example, separating incident light into visible light and infrared light (range-finding light). After separation, the aforementioned visible light can be guided through a visible light path (which can be provided by a reflector or lens or directly incident) to the lens 211 and photosensitive element 210 of the image acquisition device 21 (e.g., an optical prism). Figure 7 The aforementioned ranging light can be guided (by a reflector or lens or directly incident) to the optical receiver 221 of the optical ranging device 22 via the ranging optical path. In this way, the optical path structure 24 can be coaxially configured at the incident end with the flange axis, ranging axis, and shooting optical axis (e.g., the center point of the shooting field of view or other reference point), and allows the image acquisition device 21 to be located outside the flange axis (and the ranging optical axis).

[0103] Storage device 25, such as hard disk drive, solid-state drive, ROM, RAM, EEPROM, flash memory or any combination of various storage media, is used to store data, such as storing the effective shooting distance 251 and the conversion relationship 252.

[0104] The control device 20 is used to control the automated robotic arm system 2, for example, to control the correction mode and the working mode.

[0105] Please see Figure 2 This is a partial architectural diagram of an automated robotic arm system according to an embodiment of the present invention. In this embodiment, the control device 20 may include a control brain 30 and a robotic arm controller.

[0106] The robotic arm controller 31 is connected to the robotic arm 23 and is used to control the movement of the robotic arm based on the received arm control commands.

[0107] In one embodiment, the robotic arm 23 includes a plurality of joints 230-233 (e.g., ...) for providing multiple degrees of freedom. Figures 8 to 9 Each joint 230-233 is controlled by a servo motor to rotate at an angle, thereby enabling the robotic arm 23 to move in multiple degrees of freedom.

[0108] The arm control command can indicate the destination (mechanical space coordinates) of the robotic arm 23. The robotic arm controller 31 can convert the arm control command into the corresponding posture coordinates (such as the rotation angle of each joint 230-233) and control each joint 230-233 to rotate to pose the posture corresponding to the arm control command.

[0109] A control computer 30, such as an industrial computer or a personal computer, connects (e.g., via an industrial network or other local area network) to a robotic arm controller 31, an image acquisition device 21, an optical rangefinder 22, and a storage device 25, and controls these devices. For example, the control computer 30 can control the robotic arm 23 by issuing the aforementioned arm control commands to the robotic arm controller 31.

[0110] In one embodiment, the control computer 30 is also connected to peripheral devices 32, such as a communication interface (for connecting to a network), a human-machine interface (for interacting with the user), and a power supply device (for providing power).

[0111] Please see Figure 7 This is a schematic diagram of the setup of an automated robotic arm system according to an embodiment of the present invention.

[0112] like Figure 7 As shown, the automated robotic arm system 2 includes a mounting base 55. The mounting base 55 is connected to the end of the robotic arm 23 and can be moved by the robotic arm 23 in a three-dimensional space.

[0113] Furthermore, the image acquisition device 21, the optical ranging device 22, and the optical path structure 24 are all mounted on the mounting base 55.

[0114] In one embodiment, the mounting base 55 may be provided with one or more light sources 50 (such as ring light sources) to illuminate the working area (especially the target 51 and the fixture 52), so that the image acquisition device 21 can obtain a target image with better brightness and greatly reduce the impact of changes in ambient brightness.

[0115] In one embodiment, the optical path structure 24 may include a beam splitter 240 and a reflector 241. The reflector 241 is used to reflect the visible light separated by the beam splitter 240 to the lens 211 and photosensitive element 210 of the image acquisition device 21. Through the beam splitter 240 and the reflector 241, the imaging optical axis of the image acquisition device 21 can be aligned with the flange axis 53 of the robotic arm 23. Furthermore, the ranging optical axis of the optical ranging device 22 can be parallel to or aligned with the flange axis 53.

[0116] In one embodiment, the beam splitter 240 may be a longpass dichroic mirror with a visible light reflectance of 80% or more (e.g., 97%) and an infrared transmittance of 75% or more (e.g., 92%). For example, it allows light with wavelengths above 730 nm (e.g., 750 nm) to pass through and reflects light with wavelengths of 300 nm to 730 nm (e.g., 450 nm to 490 nm).

[0117] In one embodiment, the optical ranging device 22 includes a light emitter 220, a light receiver 221, and a ranging controller (not shown) connected to the above device. The perpendicular line from the midpoint between the light emitter 220 and the light receiver 221 is the ranging optical axis. Figure 7 In the middle, the ranging optical axis is in contact with the flange shaft 53.

[0118] The light emitter 220 emits ranging light (ranging infrared) toward the target 51. After hitting the target 51, the ranging light is reflected to the beam splitter 240 and then reaches the light receiver 221 after passing through the beam splitter 240. The ranging controller (such as a microcontroller or SoC) is configured to perform triangulation to calculate the target distance (i.e., the depth value of the target 51) based on the emission-reception time difference of the ranging light, the speed of light propagation, and the distance between the light emitter 220 and the light receiver 221.

[0119] Please see Figure 3 This is an architectural diagram of a control device according to an embodiment of the present invention. The control device 20 may include modules 40-46. Modules 40-46 are respectively configured to generate different functions to perform the present invention.

[0120] The shooting control module 40 is used to control the image acquisition device 21, such as controlling the shooting action, controlling the focusing action, acquiring image data, and performing the set image processing.

[0121] The ranging control module 41 is used to control the optical ranging device 22, such as controlling the execution of measurement, obtaining measurement data (target distance), and performing measurement correction.

[0122] The arm control module 42 is used to control the posture of the robotic arm 23 by sending arm control commands to the robotic arm controller 31, and can obtain the current position of the robotic arm 23.

[0123] The calibration control module 43 is used to execute the calibration mode.

[0124] Work control module 44 is used to execute work modes.

[0125] The transformation processing module 45 is used to calculate the coordinate transformation from visual space to mechanical space and the coordinate transformation from mechanical space to visual space.

[0126] Image analysis module 46 is used to perform image analysis and processing on the target image.

[0127] The aforementioned modules 40-46 are interconnected (which can be electrical or informational connections) and can be hardware modules (such as electronic circuit modules, integrated circuit modules, SoCs, etc.), software modules (such as firmware, operating systems, or applications), or a combination of hardware and software modules, without limitation.

[0128] Furthermore, when the aforementioned modules 40-46 are software modules (such as firmware, operating system, or application programs), the storage device 25 may include a non-transitory computer-readable recording medium (not shown in the figure). The aforementioned non-transitory computer-readable recording medium stores a computer program 250, which records computer-executable program code. When the control device 20 executes the aforementioned program code, it can implement the functions of the corresponding modules 40-46.

[0129] In one embodiment, the aforementioned modules 40-46 may be located in the control computer 30. For example, the storage device 25 may include the memory of the control computer 30, which stores a computer program 250. The processor of the control computer 30 can execute the computer program 250 to implement the functions of the corresponding modules 40-46.

[0130] Please see Figure 4 The above is a flowchart of a coordination method according to an embodiment of the present invention. The coordination method between the robotic arm and computer vision in this embodiment includes correction steps S10-S12 and working modes S13-S16.

[0131] Step S10: The control computer 30 enters the calibration mode through the calibration control module 43 to perform coordination and calibration between the mechanical space and the visual space.

[0132] For example, the control computer 30 can enter calibration mode when it receives a user's start calibration operation or a calibration command.

[0133] Step S11: The control computer 30 controls the movement of the robotic arm 23 and obtains the current target distance. Based on the current target distance, it determines whether the end of the robotic arm 23 has entered the effective shooting range of the image acquisition device 21.

[0134] If it enters the effective shooting range, the robotic arm 23 is controlled to sequentially pose multiple correction postures within the effective shooting range, and at least one correction image is captured when each correction posture is posed, so as to obtain multiple correction images corresponding to multiple correction postures respectively.

[0135] Step S12: The control computer 30 calculates the conversion relationship between the visual space of the image acquisition device 21 and the mechanical space of the robotic arm 23 based on multiple corrected postures and multiple corrected images through the conversion processing module 45.

[0136] In one embodiment, the control computer 30 can identify the visual space coordinates of the correction target in each correction image, calculate the changes of multiple visual space coordinates of the correction target in multiple correction images, calculate the changes of multiple mechanical space coordinates corresponding to multiple correction postures, and calculate the transformation relationship between visual space and mechanical space based on the changes of the mechanical space coordinates and the changes of the visual space coordinates.

[0137] In one embodiment, visual space Mechanical Space With transformation relationship The mathematical relationship between them is:

[0138]

[0139] In one embodiment, the transformation relationship It can be calculated in the following way.

[0140] The image acquisition device 21 takes multiple images of the feature f (e.g., a checkerboard pattern) of the target to obtain the relationship between the image acquisition device 21 and the feature under multiple different correction postures: Simultaneously obtain the representation of multiple current corrected postures. (W is the mechanical space coordinate, such as world coordinate), because the feature is fixed in the mechanical space coordinate. The relationship between them can be expressed as:

[0141] because Since all data is known, the algorithm can be optimized by obtaining multiple data points to obtain the optimal solution that minimizes the error term. The more correction data there is, the better the transformation relationship becomes. The more accurate.

[0142] Step S13: Control computer 30 enters working mode through working control module 44 to perform work.

[0143] For example, the control computer 30 can enter working mode when it receives a user's start operation or a work command.

[0144] Step S14: The control computer 30 controls the movement of the robotic arm 23 and obtains the current target distance. Based on the current target distance, it determines whether the end of the robotic arm 23 has entered the effective shooting range of the image acquisition device 21.

[0145] If the target enters the effective shooting range, the control computer 30 controls the image acquisition device 21 to capture a working image of the target. The image analysis module 46 then performs work-related image analysis and processing, determining the processing location (visual space coordinates) within the working image. Next, the control computer 30 uses the transformation processing module 45 to convert the visual space coordinates to mechanical space coordinates. Finally, the control computer 30 controls the robotic arm 23 to move to the mechanical space coordinates.

[0146] Step S14: Control computer 30 controls robotic arm 23 to move to the mechanical space coordinates.

[0147] In one embodiment, the control computer 30 can further control the working device 54 to control the robotic arm to perform automated actions in mechanical space coordinates, such as gripping, welding, marking, grinding, assembling, gluing and / or locking actions.

[0148] This invention can correct the robotic arm and computer vision, thereby improving the robot's hand-eye coordination.

[0149] Please also refer to Figure 4 and Figure 5 , Figure 5 This is a partial flowchart of a coordination method according to an embodiment of the present invention. Compared to Figure 4 The coordination method of this embodiment further includes steps S20-S24 in step S11.

[0150] Step S20: Control computer 30 obtains the effective shooting distance of image acquisition device 21 (e.g., ... Figure 1 The effective shooting distance is shown as 251), and the effective shooting range is set based on the effective shooting distance.

[0151] The aforementioned effective shooting distance may be, for example, the maximum or minimum focusing distance of the image acquisition device 21, and the effective shooting range may be, for example, the focusing range of the image acquisition device 21.

[0152] In one embodiment, if the effective shooting distance is 50 cm, the control computer 30 can set 0-50 cm as the effective shooting range, or set 25-50 cm as the effective shooting range, or set 25-75 cm as the effective shooting range, without limitation.

[0153] Furthermore, when the image acquisition device 21 and the target are within the aforementioned effective shooting distance or effective shooting range, the image acquisition device 21 can correctly focus on the target and capture a clear target image; when the image acquisition device 21 and the target are not within the effective shooting range, the image acquisition device 21 cannot focus correctly and will produce a blurry target image.

[0154] Step S21: Control computer 30 controls robotic arm 23 to move and continuously measure target distance until it is determined that the target is within the effective shooting range based on the target distance.

[0155] Step S22: Control computer 30 continuously measures the current target distance, and controls robotic arm 23 to move within the effective shooting range and assume different focusing postures, and captures focusing images of each focusing posture.

[0156] In one embodiment, the aforementioned multiple focusing postures are adopted at different target distances. That is, the control computer 30 continuously changes the height of the robotic arm 23 within the effective shooting range (e.g., from far away from the target to close to the target) to obtain focusing images at different heights.

[0157] Step S23: The control computer 30 performs focus analysis on multiple focus measurement images and corresponding target distances through the image analysis module 46 to determine the reference attitude and reference distance.

[0158] In one embodiment, the aforementioned focus analysis includes selecting one or more focus images (i.e., sharp images) from a plurality of focus images, determining a reference posture (such as the center or centroid of these focus postures) based on the focus posture in which these focus images were captured, and determining a reference distance (such as the average value) based on the target distance in which these focus images were captured.

[0159] In one embodiment, the aforementioned focus analysis can determine the clearest focus image by analyzing the edge features, gradient magnitude, etc. of multiple images, and obtain the focus posture and target distance that can obtain the clearest focus image, and use them as the reference posture and reference distance.

[0160] Step S24: The control computer 30 controls the robotic arm 23 to move to the correction posture based on the reference posture and reference distance, and takes the corresponding correction image under this correction posture.

[0161] In one embodiment, the target distance for each corrected posture is equal to or close to a reference distance, and is based on the reference posture, such as rotating or displacing the end of the robotic arm 23 on the same height plane.

[0162] Step S25: The control computer 30 determines whether the preset stop collection conditions are met through the calibration control module 43, so as to determine whether the collected calibration data is sufficient, such as meeting the preset number of records, such as 10 records, 50 records or 100 records, etc., without limitation.

[0163] If the conditions for stopping data collection are met, the collection of correction data will end; otherwise, step S24 will be executed again to obtain correction images captured under different correction postures.

[0164] In this way, the present invention can continuously change the rotation and displacement of the robotic arm to pose different correction postures and take correction images of each correction posture until enough correction data is collected.

[0165] In one embodiment, among the collected multiple correction postures, at least two correction postures are located in planes parallel to the fixture 52.

[0166] In one embodiment, among the collected multiple corrected postures, at least two corrected postures are at different target distances, i.e., at different heights.

[0167] Because the optical axis of the camera is in contact with the flange axis, the conversion relationship calculated by this invention can be more accurate.

[0168] Please also refer to Figure 4 and Figure 6 , Figure 6 This is a partial flowchart of a coordination method according to an embodiment of the present invention. Compared to Figure 4 The coordination method of this embodiment further includes steps S30-S33 in step S14.

[0169] Step S30: The control computer 30 controls the robotic arm 23 to move (e.g., continuously approaching the work target), continuously obtains the target distance, and determines whether the robotic arm 23 has entered the effective shooting range based on the target distance (e.g., whether the target distance is less than the effective shooting distance 251).

[0170] Step S31: After the robotic arm 23 (including image acquisition device 21) enters the effective shooting range, the control computer 30 takes a picture of the working target to obtain a working image.

[0171] Step S32: Control computer 30 performs image analysis on the working image through image analysis module 46.

[0172] In one embodiment, the aforementioned image analysis may include identifying the target in the working image and performing a work analysis based on the position of the target in visual space to determine the visual space coordinates of the work to be performed.

[0173] In one embodiment, the aforementioned work analysis may be defect detection processing (e.g., detecting component defects), measurement processing (e.g., measuring component area or length), classification and screening processing (e.g., identifying and classifying components), and component positioning processing (e.g., determining the gripping point, assembly point, soldering point, etc. of the component).

[0174] Step S33: Control computer 30 converts the visual space coordinates of the work to the mechanical space coordinates of the work based on the conversion relationship 252.

[0175] In one embodiment, the control computer 30 can further compensate for the mechanical space coordinates based on the positional difference between the working device 54 and the flange shaft to obtain the compensated mechanical space coordinates. Furthermore, the control computer 30 can generate arm control commands based on the compensated mechanical space coordinates and send these commands to the robotic arm controller 31 to control the robotic arm to move the working device 54 to the mechanical space coordinates where the work will be performed.

[0176] Therefore, the present invention can automatically perform processing operations through computer vision.

[0177] Please see Figures 8 to 11 , Figure 8 This is a first schematic diagram of a correction mode according to an embodiment of the present invention. Figure 9 This is a second schematic diagram of the correction mode according to an embodiment of the present invention. Figure 10 for Figure 8 A schematic diagram of the first corrected image captured. Figure 11 for Figure 9 A schematic diagram of the second corrected image captured.

[0178] In this embodiment, the optical path structure only includes a beam splitter 240. The visible light separated by the beam splitter 240 is directly incident on the image acquisition device 21. The lens of the image acquisition device 21 is oriented perpendicular to the flange axis.

[0179] In addition, the working device 54 is located at the bottom of the mounting base 55 and outside the flange shaft to avoid interfering with the entry of incident light.

[0180] Furthermore, in the above configuration, the distance between the working device 54 and the flange shaft is fixed, which allows the control device 20 to quickly and accurately calculate the current spatial position of the working device 54 from the flange shaft.

[0181] After the robotic arm 23 moves to within the effective shooting distance h1 at its end, it can be positioned as follows by adjusting joints 230-233. Figure 8 The first corrected posture P1 is shown, and the image is captured by the image acquisition device 21. Figure 10 The first corrected image shown.

[0182] Next, the robotic arm 23 can adjust joints 232 and 233 to move in a certain position. Figure 9 The different second corrected postures P2 shown are captured by the image acquisition device 21. Figure 11 The second corrected image shown.

[0183] like Figure 10 As shown, under the first correction posture P1, the feature of the target 60 in the first correction image (with the center point here) is located at position 600 in the visual space.

[0184] like Figure 11 As shown, after transforming to the second correction posture P2, the features of the target 60 in the second correction image move to position 601 in the visual space.

[0185] Next, the mechanical space coordinate change between the first correction posture P1 and the second correction posture P2 is calculated, and the visual space change V1 from position 600 to position 601 is calculated. By correlating the two sets of changes, the transformation relationship between visual space and mechanical space can be obtained, thus completing the correction.

[0186] The above description is merely a preferred embodiment of the present invention and is not intended to limit the claims of the present invention. Therefore, all equivalent variations made using the content of the present invention are similarly included within the scope of the present invention and are hereby stated.

Claims

1. A method for coordinating a robotic arm with computer vision, comprising: a) In the correction mode, based on the target distance measured by the optical ranging device, the robotic arm is controlled to move into multiple correction postures within the effective shooting range of the image acquisition device, and multiple correction images are captured by the image acquisition device in the multiple correction postures respectively. The beam splitter guides visible light to the image acquisition device located outside the flange shaft of the robotic arm and guides the ranging light to the optical ranging device. The ranging axis of the optical ranging device is parallel to or overlaps the flange shaft. b) Calculate the transformation relationship between the visual space of the image acquisition device and the mechanical space of the robotic arm based on the multiple corrected postures and the multiple corrected images; c) In working mode, the image acquisition device captures working images, and the mechanical spatial coordinates for performing the work are determined based on the working images and the transformation relationship; and d) Control the robotic arm to move to the mechanical space coordinates.

2. The coordination method between a robotic arm and computer vision according to claim 1, wherein step a) comprises: a1) Obtain the effective shooting distance of the image acquisition device, and set the effective shooting range based on the effective shooting distance; a2) Control the robotic arm to perform multiple focusing postures at different target distances within the effective shooting range, and capture multiple focusing images in the multiple focusing postures respectively; and a3) Perform focus analysis on the multiple focus measurement images and the corresponding multiple target distances to determine the reference attitude and reference distance.

3. The coordination method between the robotic arm and computer vision according to claim 2, wherein the focus analysis includes: e1) Select at least one focus image from the plurality of focus images; and e2) The reference posture and the reference distance are determined based on the focusing posture of the captured clear focusing image and the target distance.

4. The coordination method between a robotic arm and computer vision according to claim 1, wherein step a) comprises: a4) Based on the reference posture and reference distance, continuously control the movement of the robotic arm to different corrected postures, and take multiple corrected images respectively until the collection stop condition is met.

5. The method for coordinating a robotic arm and computer vision according to claim 1, wherein step b) comprises: b1) Identify the visual spatial coordinates of the target to be corrected in each of the corrected images; b2) Calculate the changes in the multiple visual space coordinates of the correction target relative to the multiple correction images; b3) Calculate the changes in the multiple mechanical space coordinates; and b4) Calculate the transformation relationship between visual space and mechanical space based on the changes in the multiple mechanical space coordinates and the multiple visual space coordinates.

6. The method for coordinating a robotic arm and computer vision according to claim 1, wherein step c) comprises: c1) Capture the working image when the robotic arm enters the effective shooting range; c2) Identify the work target in the work image, and perform work analysis based on the position of the work target to determine the visual space coordinates for performing the work; and c3) Based on the transformation relationship, convert the visual space coordinates of the operation to the mechanical space coordinates of the operation.

7. The coordination method between a robotic arm and computer vision according to claim 6, wherein the job analysis includes at least one of defect detection processing, measurement processing, classification and screening processing, and component positioning processing.

8. The coordination method between the robotic arm and computer vision according to claim 1, further comprising: f) Control the robotic arm to perform automated actions in the mechanical space coordinates.

9. The method for coordinating a robotic arm and computer vision according to claim 8, wherein the automated actions include at least one of gripping, welding, marking, grinding, assembling, gluing, and locking actions.

10. The method for coordinating a robotic arm and computer vision according to claim 1, wherein the beam splitter has a visible light reflectance of more than 80% and an infrared transmittance of more than 75%.

11. An automated robotic arm system, comprising: Robotic arms, used for movement in three-dimensional space; An image acquisition device is installed outside the flange shaft of the robotic arm and is used to capture images; An optical ranging device is mounted on the robotic arm to measure the distance to the target. The ranging axis of the optical ranging device is parallel to or overlaps with the flange axis. The optical path structure includes a beam splitter, which is used to guide visible light to the image acquisition device and guide ranging light to the optical ranging device; and A control device is connected to the robotic arm, the image acquisition device, and the optical ranging device; The control device is configured to, in correction mode, control the robotic arm to move into multiple correction postures within the effective shooting distance of the image acquisition device based on the target distance, and control the image acquisition device to capture multiple correction images in the multiple correction postures respectively, and calculate the conversion relationship between the visual space of the image acquisition device and the mechanical space of the robotic arm based on the multiple correction postures and the multiple correction images. The control device is configured to, in working mode, control the image acquisition device to capture working images, determine the mechanical space coordinates for performing the work based on the working images and the transformation relationship, and control the robotic arm to move to the mechanical space coordinates.

12. The automated robotic arm system according to claim 11, wherein the control device comprises: A robotic arm controller, connected to the robotic arm, is used to control the movement of the robotic arm based on received arm control commands; and A control computer is connected to the robotic arm controller, the image acquisition device, and the optical ranging device to issue control commands for the robotic arm. The control computer is configured to obtain the effective shooting distance of the image acquisition device, and set an effective shooting range based on the effective shooting distance. It controls the robotic arm to perform multiple focusing postures for different target distances within the effective shooting range, and captures multiple focusing images in the multiple focusing postures respectively. The control computer is configured to perform focus analysis on the plurality of focus images and the corresponding plurality of target distances to determine the reference attitude and reference distance; The control computer is configured to continuously control the robotic arm to move into different corrected postures based on the reference posture and the reference distance, and to capture multiple corrected images until the conditions for stopping the collection are met.

13. The automated robotic arm system of claim 12, wherein the control computer is configured to select at least one focusing image from the plurality of focusing images, and to determine the reference posture and the reference distance based on the focusing posture of capturing a clear focusing image and the target distance.

14. The automated robotic arm system according to claim 11 further includes a mounting base for mounting the image acquisition device, the optical ranging device, and the optical path structure; in, The end of the robotic arm is connected to the mounting base and is used to move the mounting base in the three-dimensional space; The control device is configured to identify the visual spatial coordinates of the correction target in each of the correction images, calculate the changes in the multiple visual spatial coordinates of the correction target in the multiple correction images, calculate the changes in the multiple mechanical spatial coordinates, and calculate the transformation relationship between visual space and mechanical space based on the changes in the multiple mechanical spatial coordinates and the changes in the multiple visual spatial coordinates.

15. The automated robotic arm system of claim 12, wherein the robotic arm includes a plurality of joints for providing a plurality of degrees of freedom, and the mechanical space coordinates are used to control the robotic arm to move in the plurality of degrees of freedom by adjusting the rotation angles of the plurality of joints based on the mechanical space coordinates; in, The control device is configured to capture the working image when the robotic arm enters the effective shooting range, identify the working target in the working image, perform work analysis based on the position of the working target to determine the visual space coordinates for performing the work, and convert the visual space coordinates for performing the work to the mechanical space coordinates for performing the work based on the transformation relationship. The work analysis includes at least one of defect detection processing, measurement processing, classification and screening processing, and component positioning processing.

16. The automated robotic arm system according to claim 11 further includes a working device connected to the control device; in, The control device is configured to control the working device to perform automated actions when the robotic arm moves to the mechanical space coordinates.

17. The automated robotic arm system of claim 16, wherein the working device includes at least one of a gripper end effector, a welding heater, a marking tool, a grinding tool, an assembly end effector, an adhesive applicator, and a locking tool; in, The automated actions include at least one of the following: clamping action, welding action, marking action, grinding action, assembly action, gluing action, and locking action.

18. The automated robotic arm system of claim 11, wherein the image acquisition device comprises a color camera; in, The optical ranging device includes an infrared rangefinder; The beam splitter is a long-pass dichroic mirror with a visible light reflectance of over 80% and an infrared transmittance of over 75%.

19. The automated robotic arm system according to claim 11, wherein the optical ranging device comprises: A light emitter for emitting the ranging light toward the target; A light receiver for receiving the reflected ranging light; and A ranging controller, connected to the light transmitter and the light receiver, is configured to calculate the target distance based on the emission-reception time difference of the ranging light, the speed of light propagation, and the distance between the light transmitter and the light receiver.

20. The automated robotic arm system of claim 11, wherein the optical path structure further includes a reflector for reflecting visible light reflected by the beam splitter to the image acquisition device.

Images