Robotic in-situ measurement device and method

Abstract

The application relates to the fields of robot technology and computer vision technology, in particular to a robot in-situ measurement device and method, wherein the device comprises: a plurality of acquisition modules arranged on the rack of the robot through a support frame to acquire a work area feature image and a positioning target area image of the robot; a light source module arranged on the rack of the robot through the support frame to provide illumination for the work space area of the robot; a calibration tool module arranged on the end effector of the robot to position the end effector of the robot at a plurality of preset calibration positions; and a measurement module arranged on the base of the robot and electrically connected with the plurality of acquisition modules to obtain the position of the work area feature under the robot work coordinate system according to the work area feature image and the positioning target area image. Therefore, the problems of high cost, low efficiency and great operation difficulty in the related art in-situ measurement which needs to additionally increase a measurement device or manual intervention are solved.

Description

Technical Field

[0001] This invention relates to the fields of robotics and computer vision, and in particular to a robot in-situ measurement device and method. Background Technology

[0002] In-situ machining, characterized by miniaturization, mobility, intelligence, and robotics, is increasingly being applied to the manufacturing, assembly, and repair of large and complex components. This improves operational accessibility, processing flexibility, and efficiency, and is gradually becoming a new trend in high-quality machining of large and complex components. Compared to traditional large-scale machine tool machining, modular machining, and manual manufacturing, in-situ machining simplifies the machining process, automates operations in a flexible and versatile manner, and significantly improves processing efficiency and quality.

[0003] In in-situ machining, whether using mobile robots or stationary machining equipment, precise in-situ measurement is indispensable when processing large and complex structural parts. In-situ measurement ensures the robot's positional accuracy before and after machining, guarantees machining quality, and avoids quality problems and equipment damage caused by positional deviations. This importance is even more pronounced in fields requiring extremely high precision, such as aerospace and precision manufacturing. However, in current technologies, in-situ measurement typically requires additional measuring equipment or manual intervention, which undoubtedly increases machining costs and time. Furthermore, different robot models may require different measuring equipment and methods, further increasing the difficulty and complexity of technical implementation. Therefore, how to simplify and standardize in-situ measurement procedures while maintaining machining efficiency and quality is a pressing technical problem that needs to be solved. Summary of the Invention

[0004] This invention provides a robot in-situ measurement device and method, which solves the problems of high cost, low efficiency and high operation difficulty in in-situ measurement in related technologies, which require additional measurement devices or manual intervention.

[0005] A first aspect of the present invention provides a robot in-situ measurement device, comprising: multiple acquisition modules mounted on the frame of a target robot via a support frame to acquire feature images of the target robot's working area and a positioning target area; a light source module mounted on the frame of the target robot via the support frame to provide illumination for the working space area of ​​the target robot; a calibration fixture module mounted on an end effector connected to the target robot to position the end effector of the target robot at multiple preset calibration positions; and a measurement module mounted on the base of the target robot and electrically connected to the multiple acquisition modules to obtain the position of the working area features in the target robot's working coordinate system based on the feature images of the working area and the positioning target area.

[0006] Optionally, each acquisition module includes a camera lens, an industrial camera, a camera fixture, a camera power cable and a data cable. The camera lens is mounted on the industrial camera, the industrial camera is mounted on the camera fixture, the camera fixture is mounted on the support frame, and the camera power cable and data cable connect the industrial camera and the measurement module.

[0007] The light source module includes an industrial light source, a light source fixture, and a light source controller. The industrial light source is mounted on the light source fixture, which is mounted on the support frame. The light source controller is electrically connected to the industrial light source.

[0008] The calibration fixture module includes a calibration fixture, a visual calibration target ball, and a laser tracker target ball. The calibration fixture is equipped with a magnet and multiple target ball positions to adsorb the visual calibration target ball onto the corresponding target ball position and / or adsorb the laser tracker target ball onto the corresponding target ball position.

[0009] A second aspect of the present invention provides a robot in-situ measurement method, comprising the following steps: calibrating the robot in-situ measurement device to obtain a coordinate system transformation matrix between the target robot and the plurality of acquisition modules;

[0010] The target robot is moved to the target workstation, and the multiple acquisition modules are used to acquire feature images of the target robot's working area and images of the positioning target area;

[0011] Based on the coordinate system transformation matrix, the feature image of the work area and the image of the positioning target area are processed to obtain the position of the work area features in the robot's work coordinate system;

[0012] The position of the work area features in the robot's work coordinate system is transmitted to the target robot, so that the target robot can perform the corresponding work task according to the position.

[0013] Optionally, the calibration of the robot in-situ measurement device to obtain the coordinate system transformation matrix between the target robot and multiple acquisition modules includes:

[0014] Multiple calibration positions are planned according to the workspace of the target robot, and the calibration fixture is set on the end effector of the target robot so that the vision calibration target ball is attached to the target ball position of the calibration fixture.

[0015] The target robot is fed to position the end effector carrying the calibration fixture at each calibration position.

[0016] The calibration image dataset is constructed by capturing calibration images of the calibration fixture at each calibration position using the multiple acquisition modules.

[0017] The measurement module is used to process the constructed calibration image dataset to obtain the coordinate system transformation matrix between the target robot and multiple acquisition modules.

[0018] Optionally, the step of processing the constructed calibration image dataset using the measurement module to obtain the coordinate system transformation matrix between the target robot and multiple acquisition modules includes:

[0019] The pixel information of each calibration image, the calibration information of the multiple acquisition modules, and the position of the end effector of the target robot are obtained.

[0020] Based on the pixel information and the calibration information, the position of the visual calibration target ball in the camera coordinate system is generated;

[0021] Based on the position of the visual calibration target ball in the camera coordinate system and the position of the end effector of the target robot, a coordinate transformation matrix between the target robot and multiple acquisition modules is generated.

[0022] Optionally, each calibration image is corrected based on the calibration information;

[0023] Each calibrated image after correction is input into a preset calibrated target ball recognition model to obtain the position of the target ball in each calibrated image;

[0024] Based on the position information of the target ball in each calibration image, each calibrated image after correction is segmented to obtain a sub-image of the target ball. The sub-image of the target ball is then preprocessed using the LoG edge detection method to obtain a preprocessed sub-image of the target ball.

[0025] Based on the preprocessed sub-image of the target ball, the precise position of the circular bullseye of the target ball in the image is obtained by Hough transform elliptic detection.

[0026] Based on the precise position of the circular bullseye of the target ball in the image and the calibration information, the position of the visual calibration target ball in the camera coordinate system is calculated by triangulation.

[0027] Optionally, the step of processing the work area feature image and the positioning target area image based on the coordinate system transformation matrix to obtain the position of the work area features in the robot's work coordinate system includes:

[0028] Obtain the calibration information of the multiple acquisition modules;

[0029] Based on the image of the target area and the calibration information, the position of the target in the camera coordinate system is generated.

[0030] Based on the coordinate system transformation matrix and the position of the positioning target in the camera coordinate system, the transformation relationship between the positioning target coordinate system and the robot operation coordinate system is obtained;

[0031] Based on the work area feature image and the positioning target area image, the position of the work area feature in the positioning target coordinate system is obtained;

[0032] Based on the transformation relationship between the positioning target coordinate system and the robot operation coordinate system, and the position of the operation area feature in the positioning target coordinate system, the position of the operation area feature in the robot operation coordinate system is obtained.

[0033] A third aspect of the present invention provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the robot in-situ measurement method as described in the above embodiments.

[0034] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described robot in-situ measurement method.

[0035] The robot in-situ measurement device and method of this invention do not require additional measurement devices or manual intervention when performing in-situ measurements. They are applicable to different types of robots, have a high degree of automation and high measurement efficiency, and can simplify and standardize in-situ measurement steps, saving costs for processing and measuring large and complex structural parts.

[0036] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0037] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0038] Figure 1 This is a schematic diagram of a robot in-situ measurement device according to an embodiment of the present invention;

[0039] Figure 2 This is a partial structural schematic diagram of a robot in-situ measurement device according to an embodiment of the present invention;

[0040] Figure 3 This is a schematic diagram of an in-situ measurement calibration fixture provided according to an embodiment of the present invention;

[0041] Figure 4 This is a schematic diagram of another in-situ measurement calibration fixture provided according to an embodiment of the present invention;

[0042] Figure 5 This is a schematic diagram of another robot in-situ measurement device provided according to an embodiment of the present invention;

[0043] Figure 6 This is a partial structural schematic diagram of a robot in-situ measurement device according to another embodiment of the present invention;

[0044] Figure 7 This is a schematic diagram of the structure of a robot in-situ measurement device according to another embodiment of the present invention;

[0045] Figure 8 This is a partial schematic diagram of a robot performing in-situ measurement and operation tasks according to an embodiment of the present invention;

[0046] Figure 9 This is a schematic diagram illustrating a robot performing in-situ measurement and operation tasks according to an embodiment of the present invention;

[0047] Figure 10 A flowchart of a robot in-situ measurement method provided according to an embodiment of the present invention;

[0048] Figure 11 This is a flowchart illustrating the execution of a robot in-situ measurement method according to an embodiment of the present invention.

[0049] Figure 12 This is a flowchart of processing calibration image data according to an embodiment of the present invention;

[0050] Figure 13 This is a flowchart of processing visual target ball image data according to an embodiment of the present invention;

[0051] Figure 14This is a schematic diagram of the result of processing visual target ball image data according to an embodiment of the present invention, wherein (a) is the target ball identification result, (b) is the target ball bullseye sub-image, (c) is the preprocessed sub-image, and (d) is the Hough transform ellipse detection result;

[0052] Figure 15 This is a flowchart illustrating the processing of image data in a work area according to an embodiment of the present invention;

[0053] Figure 16 This is a schematic diagram of coordinate system transformation relationships provided according to an embodiment of the present invention;

[0054] Figure 17 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present invention.

[0055] Explanation of reference numerals in the attached figures:

[0056] 100 - Multiple acquisition modules, 101 - Industrial camera, 102 - Camera lens, 103 - Camera fixture, 200 - Light source module, 201 - Industrial light source, 202 - Light source fixture, 203 - Light source controller, 300 - Calibration fixture module, 301 - Calibration fixture, 302 - Calibration fixture connector, 303 - Vision calibration target ball, 304 - Laser tracker calibration target ball, 400 - Support frame, 500 - Measurement module, 600 - Target robot, 700 - Laser tracker, 800 - Large structural component, 801 - Vision calibration target ball mounted on large structural component, and 802 - Working area features. Detailed Implementation

[0057] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0058] The robot in-situ measurement device and method according to embodiments of the present invention are described below with reference to the accompanying drawings.

[0059] Figure 1 This is a schematic diagram of a robot in-situ measurement device according to an embodiment of the present invention.

[0060] like Figure 1 As shown, the robot in-situ measurement device 10 includes: multiple acquisition modules 100, a light source module 200, a calibration fixture module 300, and a measurement module 500.

[0061] The system includes multiple acquisition modules 100, mounted on the frame of the target robot 600 via a support frame 400, to acquire feature images of the target robot 600's working area and the positioning target area. A light source module 200, also mounted on the frame of the target robot 600 via the support frame 400, provides illumination to the target robot 600's workspace. A calibration fixture module 300 is mounted on the end effector connected to the target robot 600 to position the end effector at multiple preset calibration positions. A measurement module 500 is mounted on the base of the target robot 600 and electrically connected to the acquisition modules 100 to determine the position of the working area features in the target robot's working coordinate system based on the working area feature images and the positioning target area images.

[0062] In some embodiments, such as Figure 2 As shown, each acquisition module 100 may include a camera lens 102, an industrial camera 101, a camera fixture 103, a camera power cable and a data cable. The camera lens 102 is mounted on the industrial camera 101, the industrial camera 101 is mounted on the camera fixture 103, the camera fixture 103 is mounted on the support frame 400, and the camera power cable and data cable connect the industrial camera 101 and the measurement module 500.

[0063] Each acquisition module 100 communicates with the measurement module 500 via the USB protocol, facilitating data communication between the measurement module 500 and multiple acquisition modules 100.

[0064] The light source module 200 may include an industrial light source 201, a light source fixture 202, and a light source controller 203. The industrial light source 201 is mounted on the light source fixture 202, which is mounted on the support frame 400. The light source controller 203 is electrically connected to the industrial light source 201.

[0065] The calibration fixture module 300 may include a calibration fixture 301 and a vision calibration target ball 303. It is connected to the end effector of the target robot 600 via a calibration fixture connector 302. The calibration fixture 301 has multiple target ball positions and a magnet inside the calibration fixture, which can attract the vision calibration target ball 303 to the corresponding target ball position. Thus, the acquisition module 100 mounted on the target robot 600 faces the vision calibration target ball, and hand-eye calibration can be performed.

[0066] For example, such as Figure 3 and 4 As shown, the calibration fixture 301 can have four target ball positions or six target ball positions. In the case of a calibration fixture with six target ball positions, the target ball positions are distributed on both sides, with three target ball positions on each side.

[0067] In some embodiments, such as Figure 5 As shown, the robot in-situ measurement device 10 also includes a laser tracker 700, which is used to assist in calibration. Figure 6 As shown, the current calibration fixture module also includes a laser tracker target ball 304. The calibration fixture 301 is not only equipped with multiple vision calibration target balls 303, but also with a laser tracker target ball 304. The laser tracker target ball 304 is also attached to the corresponding target ball position, so that the laser tracker 700 can simultaneously acquire the end effector position information of the target robot 600 to assist in hand-eye calibration.

[0068] In some embodiments, such as Figure 7 As shown, the target robot 600 can also be connected in series with the in-situ measurement device, and a laser tracker 700 can be used for calibration.

[0069] like Figure 8 and 9 As shown, the working principle of the robot in-situ measurement device in this embodiment of the invention is as follows: The target robot is moved to a working position near a large component. First, based on the required size of the target robot's workspace, the models of the industrial camera 101 and camera lens 102 are determined. The industrial camera 101 and camera lens 102 are then combined and mounted on the camera fixture 103. The support frame 400 is fixed to the frame of the target robot 600. The camera fixture 103 is then mounted on the support frame 403. The installation position of the camera fixture 103 is adjusted so that the camera lens 102 faces the target component. In the workspace area of ​​the target robot 600, an industrial light source 201 is mounted on a light source fixture 202, which is then mounted on a support frame 400. The installation position of the light source fixture 201 is adjusted so that the light emitted by the industrial light source 201 faces the workspace area of ​​the target robot 600. The target robot 600 then performs operations on the stationary large component, calibrating the in-situ measurement device of the robot that needs to perform in-situ measurement and operation tasks. During calibration, the calibration fixture module 300 is fixedly connected to the end effector of the target robot 600. After calibration, the acquisition module 100 captures images of the area of ​​the operation target and the positioning target. The testing module 500 processes the image data of the area to be operated, calculates the operation coordinate system data, and transmits the operation coordinate system data to the CNC system of the target robot 600. The target robot 600 then begins to execute the operation task based on the coordinate information.

[0070] Next, the robot in-situ measurement method according to an embodiment of the present invention is described with reference to the accompanying drawings.

[0071] Figure 10 This is a flowchart illustrating a robot in-situ measurement method provided in an embodiment of the present invention.

[0072] like Figure 10 As shown, the robot in-situ measurement method includes the following steps:

[0073] In step S1001, the robot in-situ measurement device is calibrated to obtain the coordinate system transformation matrix between the target robot and multiple acquisition modules.

[0074] Furthermore, in one embodiment of the present invention, the robot in-situ measurement device is calibrated to obtain the coordinate system transformation matrix between the target robot and multiple acquisition modules, including:

[0075] Multiple calibration positions are planned according to the workspace of the target robot, and the calibration fixture is set on the end effector of the target robot so that the vision calibration target ball is attached to the target ball position of the calibration fixture.

[0076] The target robot is fed to position the end effector carrying the calibration fixture at each calibration position.

[0077] Multiple acquisition modules were used to capture calibration images of the calibration fixture at each calibration position, and a calibration image dataset was constructed.

[0078] The measurement module is used to process and construct a calibration image dataset, and the coordinate transformation matrix between the target robot and multiple acquisition modules is obtained.

[0079] Specifically, such as Figure 11 As shown, the in-situ testing device is installed on the target robot. The installation process is as follows: Based on the size requirements of the target robot's workspace, determine the model of the industrial camera and camera lens. Combine the industrial camera and camera lens and mount them on the camera fixture. Fix the support frame to the robot frame. Mount the camera fixture on the support frame. Adjust the installation position of the camera fixture so that the camera lens faces the robot's workspace area. Mount the industrial light source on the light source fixture. Mount the light source fixture on the support frame. Adjust the installation position of the light source fixture so that the light emitted by the industrial light source faces the target robot's workspace area. The in-situ measurement device of the robot that needs to perform in-situ measurement and operation tasks is calibrated.

[0080] Furthermore, the calibration process is as follows:

[0081] Multiple calibration positions were planned according to the size of the target robot's workspace. The calibration fixture was installed on the robot's end effector, and the vision calibration target ball was installed on the target ball position of the calibration fixture. The in-situ measurement device entered the calibration working state.

[0082] The target robot is fed to position the end effector carrying the calibration fixture at any calibration location.

[0083] The acquisition module captures calibration images of the area where the calibration fixture on the end effector of the target robot is located. The image data is transmitted to the test module for storage, and it is determined whether the calibration images have been captured at all calibration positions. If the images have been captured at all calibration positions, the test module processes all the calibration image data and calculates the coordinate transformation matrix between the target robot and the camera. Otherwise, the capturing continues until the images have been captured at all calibration positions.

[0084] Finally, the calibration target ball and calibration fixture are removed from the end effector of the target robot, restoring it to its working state and completing the calibration of the in-situ measuring device.

[0085] In some embodiments, a laser tracker can be additionally installed on the target ball position of the calibration fixture to calibrate the target ball, and the coordinates of the end effector of the target robot can be obtained more accurately by measuring with the laser tracker.

[0086] In some embodiments, images of the calibration board may be taken additionally for camera parameter calibration.

[0087] Furthermore, in one embodiment of the present invention, a calibration image dataset is constructed using a measurement module to obtain a coordinate system transformation matrix between the target robot and multiple acquisition modules, including:

[0088] Acquire pixel information for each calibration image, calibration information from multiple acquisition modules, and the position of the end effector of the target robot;

[0089] Based on pixel information and calibration information, the position of the visual calibration target ball in the camera coordinate system is generated;

[0090] Based on the position of the visually calibrated target ball in the camera coordinate system and the position of the end effector of the target robot, a coordinate transformation matrix between the target robot and multiple acquisition modules is generated.

[0091] Specifically, such as Figure 12 As shown, pixel information of the calibration image is obtained, calibration information of the camera module is obtained, and position information of the robot end effector is obtained.

[0092] like Figure 13 and 14 As shown, based on pixel information and calibration information, each calibration image is corrected using the calibration information. The corrected calibration image is then input into a preset calibration target ball recognition model to obtain the position information of the target ball in each calibration image. The preset calibration target ball recognition model is obtained by training an initial model using sample images labeled with target ball recognition results. The initial model uses the YOLO v5 model.

[0093] Based on the position information of the target ball in each calibration image, each calibration image after correction is segmented to obtain a sub-image of the target ball, and the sub-image of the target ball is preprocessed using the LoG edge detection method:

[0094]

[0095] In the formula, L(x,y) is the Laplacian transform result of the image, also known as the LoG response, which represents the second derivative of the image at position (x,y) and is used to detect edges. G(x,y) is the Gaussian function, which is used to smooth the image. It is obtained by convolving the image I(x,y) with the Gaussian kernel H(x,y). I(x,y) is the gray value of the input image at position (x,y), H(x,y) is the Gaussian kernel function used to smooth the image, σ is the standard deviation of the Gaussian kernel, and (x,y) is the pixel position in the image, i.e., the row and column index of the image.

[0096] Based on the preprocessed sub-image of the target sphere, the precise position of the circular bullseye on the image is obtained through Hough transform ellipse detection. The Hough transform ellipse detection algorithm filters point pairs based on ellipse constraints. For each hypothetical ellipse: the center and major axis of the ellipse are calculated; for all other points, the minor axis corresponding to the major axis is calculated; the value of the minor axis is put into an accumulator; the accumulator is smoothed, and the maximum weight result is found; if the score of the current ellipse is greater than the existing best ellipse, it is replaced; finally, the parameters of the ellipse with the highest score are returned.

[0097] Based on the precise position and calibration information of the circular bullseye on the image, the position of the visually calibrated target ball in the camera coordinate system is calculated through triangulation.

[0098] Based on the target ball position information and the end effector position information, represent the target ball position information and the end effector position information as two three-dimensional point sets, and calculate the centroids of the two three-dimensional point sets A and B:

[0099]

[0100]

[0101] In the formula, Let n be the centroid of the target ball's position information, n be the number of points in the point set, and p be the three-dimensional coordinates of the points in the point set. The centroid of the end effector position information;

[0102] Move two 3D point sets A and B to the origin along their centroids:

[0103]

[0104]

[0105] Calculate the covariance matrix H of two 3D point sets:

[0106] H = A *T B *

[0107] Calculate the singular value decomposition of the covariance matrix H:

[0108] H=UΣV T

[0109] Calculate the rotation matrix R using V and U from the singular value decomposition:

[0110]

[0111] Calculate the translation vector t using the rotation matrix R:

[0112]

[0113] Obtain the coordinate transformation matrix between the target robot and the camera. b M c :

[0114]

[0115] The coordinate transformation matrix between the target robot and multiple acquisition modules is then saved to the test module.

[0116] In step S1002, the target robot is moved to the target workstation, and multiple acquisition modules are used to acquire feature images of the target robot's working area and images of the positioning target area.

[0117] Specifically, after the target robot completes calibration, it is moved to the designated workstation to enable the in-situ measurement device to enter the working state. Multiple acquisition modules are used to collect feature images of the target robot's working area and images of the positioning target area.

[0118] In step S1003, based on the coordinate system transformation matrix, the feature image of the work area and the image of the positioning target area are processed to obtain the position of the feature of the work area in the robot's work coordinate system.

[0119] Furthermore, based on the coordinate system transformation matrix, the feature image of the work area and the image of the target area are processed to obtain the position of the work area features in the robot's work coordinate system, including:

[0120] Obtain calibration information from multiple acquisition modules;

[0121] Based on the target area image and calibration information, the position of the target in the camera coordinate system is generated.

[0122] Based on the coordinate system transformation matrix and the position of the positioning target in the camera coordinate system, the transformation relationship between the positioning target coordinate system and the robot operation coordinate system is obtained;

[0123] Based on the feature image of the work area and the image of the positioning target area, the position of the feature of the work area in the coordinate system of the positioning target is obtained;

[0124] Based on the transformation relationship between the positioning target coordinate system and the robot operation coordinate system, and the position of the operation area features in the positioning target coordinate system, the position of the operation area features in the robot operation coordinate system is obtained.

[0125] Specifically, such as Figure 15 As shown, the system acquires the image information of the target to be located, the calibration information of the camera module, the coordinate system transformation matrix between the robot and the camera, and the position of the working area features in the target coordinate system. Based on the target image information and the camera module calibration information, the position information of the target in the camera coordinate system is generated. Based on the target position information and the coordinate system transformation matrix between the robot and the camera, the transformation relationship between the target coordinate system and the robot working coordinate system is obtained. Based on the transformation relationship between the target coordinate system and the robot working coordinate system and the position of the working area features in the target coordinate system, the position of the working area features in the robot working coordinate system is obtained.

[0126] In addition, outside of the process, the transformation relationship between the positioning target coordinate system and the global coordinate system of the large component can be measured by using a laser tracker; based on the digital model of the large component, the position of the work area features in the positioning target coordinate system can be calculated.

[0127] In step S1004, the position of the work area features in the robot's work coordinate system is transmitted to the target robot, so that the target robot can perform the corresponding work task according to the position.

[0128] like Figure 16 As shown, in this embodiment of the invention, the transformation relationship between the robot coordinate system and the camera coordinate system is obtained through hand-eye calibration; the transformation relationship between the camera coordinate system and the positioning target coordinate system is obtained through photogrammetry; the transformation relationship between the positioning target coordinate system and the global coordinate system of the large component is obtained through laser tracker measurement; and the position of the working area features in the global coordinate system of the large component is obtained through digital modeling.

[0129] It should be noted that the foregoing explanation of the robot in-situ measurement method embodiment also applies to the robot in-situ measurement device of this embodiment, and will not be repeated here.

[0130] The robot in-situ measurement method proposed in this embodiment of the invention does not require additional measuring devices or manual intervention during in-situ measurement, and is applicable to different types of robots. It has a high degree of automation and high measurement efficiency, and can simplify and standardize in-situ measurement steps, saving costs for processing and measuring large and complex structural parts.

[0131] Figure 17 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. The electronic device may include:

[0132] The memory 1701, the processor 1702, and the computer program stored on the memory 1701 and executable on the processor 1702.

[0133] When the processor 1702 executes the program, it implements the robot in-situ measurement method provided in the above embodiments.

[0134] Furthermore, electronic devices also include:

[0135] Communication interface 1703 is used for communication between memory 1701 and processor 1702.

[0136] Memory 1701 is used to store computer programs that can run on processor 1702.

[0137] The memory 1701 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0138] If the memory 1701, processor 1702, and communication interface 1703 are implemented independently, then the communication interface 1703, memory 1701, and processor 1702 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 17 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0139] Optionally, in a specific implementation, if the memory 1701, processor 1702, and communication interface 1703 are integrated on a single chip, then the memory 1701, processor 1702, and communication interface 1703 can communicate with each other through an internal interface.

[0140] Processor 1702 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0141] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described robot in-situ measurement method.

[0142] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0143] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0144] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0145] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0146] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0147] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0148] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0149] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. A robot in-situ measurement device, characterized in that, include: Multiple acquisition modules are mounted on the frame of the target robot via a support frame to acquire feature images of the target robot's working area and images of the positioning target area; A light source module is mounted on the frame of the target robot via the support frame to provide illumination for the workspace area of ​​the target robot. A calibration fixture module is provided on the end effector of the target robot to position the end effector of the target robot at multiple preset calibration positions. A measurement module is mounted on the base of the target robot and electrically connected to the multiple acquisition modules to obtain calibration information from the multiple acquisition modules; based on the target area image and calibration information, the position of the target in the camera coordinate system is generated. Based on the coordinate system transformation matrix and the position of the positioning target in the camera coordinate system, the transformation relationship between the positioning target coordinate system and the robot operation coordinate system is obtained; based on the operation area feature image and the positioning target area image, the position of the operation area feature in the positioning target coordinate system is obtained; based on the transformation relationship between the positioning target coordinate system and the robot operation coordinate system and the position of the operation area feature in the positioning target coordinate system, the position of the operation area feature in the target robot operation coordinate system is obtained.

2. The robot in-situ measurement device according to claim 1, characterized in that, Each acquisition module includes a camera lens, an industrial camera, a camera fixture, a camera power cable, and a data cable. The camera lens is mounted on the industrial camera, the industrial camera is mounted on the camera fixture, the camera fixture is mounted on the support frame, and the camera power cable and data cable connect the industrial camera and the measurement module. The light source module includes an industrial light source, a light source fixture, and a light source controller. The industrial light source is mounted on the light source fixture, which is mounted on the support frame. The light source controller is electrically connected to the industrial light source.

3. The robot in-situ measurement device according to claim 1, characterized in that, The calibration fixture module includes a calibration fixture, a visual calibration target ball, and a laser tracker target ball. The calibration fixture is equipped with a magnet and multiple target ball positions to adsorb the visual calibration target ball onto the corresponding target ball position and / or adsorb the laser tracker target ball onto the corresponding target ball position.

4. A robot in-situ measurement method, characterized in that, The robot in-situ measurement device according to any one of claims 1-3 includes the following steps: The robot in-situ measurement device is calibrated to obtain the coordinate system transformation matrix between the target robot and the multiple acquisition modules; The target robot is moved to the target workstation, and the multiple acquisition modules are used to acquire feature images of the target robot's working area and images of the positioning target area; Based on the coordinate system transformation matrix, the feature image of the work area and the image of the positioning target area are processed to obtain the position of the work area feature in the robot work coordinate system, including: obtaining the calibration information of the multiple acquisition modules; Based on the target region image and the calibration information, the position of the target in the camera coordinate system is generated; based on the coordinate system transformation matrix and the position of the target in the camera coordinate system, the transformation relationship between the target coordinate system and the robot operation coordinate system is obtained; based on the operation area feature image and the target region image, the position of the operation area feature in the target coordinate system is obtained; based on the transformation relationship between the target coordinate system and the robot operation coordinate system and the position of the operation area feature in the target coordinate system, the position of the operation area feature in the robot operation coordinate system is obtained. The position of the work area features in the robot's work coordinate system is transmitted to the target robot, so that the target robot can perform the corresponding work task according to the position.

5. The robot in-situ measurement method according to claim 4, characterized in that, The calibration of the robot's in-situ measurement device to obtain the coordinate system transformation matrix between the target robot and multiple acquisition modules includes: Multiple calibration positions are planned according to the workspace of the target robot, and the calibration fixture is set on the end effector of the target robot so that the vision calibration target ball is attached to the target ball position of the calibration fixture. The target robot is fed to position the end effector carrying the calibration fixture at each calibration position. The calibration image dataset is constructed by capturing calibration images of the calibration fixture at each calibration position using the multiple acquisition modules. The measurement module is used to process the constructed calibration image dataset to obtain the coordinate system transformation matrix between the target robot and multiple acquisition modules.

6. The robot in-situ measurement method according to claim 5, characterized in that, The process of using the measurement module to process the constructed calibration image dataset to obtain the coordinate system transformation matrix between the target robot and multiple acquisition modules includes: The pixel information of each calibration image, the calibration information of the multiple acquisition modules, and the position of the end effector of the target robot are obtained. Based on the pixel information and the calibration information, the position of the visual calibration target ball in the camera coordinate system is generated; Based on the position of the visual calibration target ball in the camera coordinate system and the position of the end effector of the target robot, a coordinate transformation matrix between the target robot and multiple acquisition modules is generated; This includes representing the position of the target ball in the camera coordinate system and the position of the end effector in the robot base coordinate system as two sets of three-dimensional points. A and B Calculate the coordinate system transformation matrix using the following steps: Calculate the covariance matrix of the two three-dimensional point sets. ,in , These are the decentralized point sets, , Let be the centroid of the corresponding point set; Singular value decomposition is performed on the covariance matrix H. ; Through singular value decomposition V and U Calculate the rotation matrix ;pass Calculate the translation vector t to obtain the coordinate transformation matrix between the target robot and the camera. .

7. The robot in-situ measurement method according to claim 6, characterized in that, The step of generating the position of the visual calibration target ball in the camera coordinate system based on the pixel information and the calibration information includes: Each calibration image is corrected based on the calibration information; Each calibrated image after correction is input into a preset calibrated target ball recognition model to obtain the position of the target ball in each calibrated image; Based on the position information of the target ball in each calibration image, each calibrated image after correction is segmented to obtain a sub-image of the target ball. The sub-images of the target ball are then preprocessed using the LoG edge detection method to obtain preprocessed sub-images of the target ball. This preprocessing includes: transforming the sub-images of the target ball. ,in This is a Gaussian function used to smooth images. For the input image at position grayscale value at that location Let σ be the Gaussian kernel function, and σ be the standard deviation of the Gaussian kernel. These are the pixel positions in the image, i.e., the row and column indices of the image; Based on the preprocessed sub-image of the target ball, the precise position of the circular bullseye of the target ball in the image is obtained by Hough transform elliptic detection. Based on the precise position of the circular bullseye of the target ball in the image and the calibration information, the position of the visual calibration target ball in the camera coordinate system is calculated by triangulation.

8. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the robot in-situ measurement method as described in any one of claims 4-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the robot in-situ measurement method as described in any one of claims 4-7.

Images