Camera calibration method, robot control method, device, equipment and medium

Abstract

The invention provides a camera calibration method, a robot control method, a device, equipment and a medium, which can be applied to the field of computer vision and the field of intelligent robots. The camera calibration method comprises the following steps: acquiring a mark image, and acquiring a mark point arranged on a carrier to obtain the mark image when a reference camera is static relative to a working area in which a robot equipped with the carrier executes operation; detecting the mark image to obtain a reference coordinate of the mark point in a reference camera coordinate system of the reference camera; according to the reference coordinate and a preset carrier coordinate, a movable camera arranged on the robot is calibrated, a target transformation matrix is obtained, the preset carrier coordinate represents the position of the marking point in a carrier coordinate system of the carrier, and the target transformation matrix represents the space transformation relation between the carrier coordinate system and an area coordinate system of the working area; the target conversion matrix is used for controlling the robot to execute operation.

Description

Technical Field

[0001] The present disclosure relates to the fields of computer vision and intelligent robots, and specifically to a camera calibration method, a robot control method, an apparatus, a device, a medium, and a program product. Background Art

[0002] With the rapid development of science and technology, in operation scenarios such as freight logistics and product manufacturing, intelligent robots can be used to perform specified operations to improve the efficiency of related operation scenarios. However, the low accuracy of intelligent robots in performing operations can easily lead to problems such as reduced efficiency. Summary of the invention

[0003] In view of the above problems, the present disclosure provides a camera calibration method, a robot control method, an apparatus, a device, a medium and a program product.

[0004] According to the first aspect of the present disclosure, a camera calibration method is provided, comprising: acquiring a marker image, when a reference camera is stationary relative to a working area in which a robot equipped with a carrier performs an operation, collecting the marker point arranged on the carrier to obtain the marker image; detecting the marker image to obtain the reference coordinates of the marker point in the reference camera coordinate system of the reference camera; calibrating a movable camera arranged on the robot according to the reference coordinates and preset carrier coordinates to obtain a target transformation matrix, wherein the preset carrier coordinates represent the position of the marker point in the carrier coordinate system of the carrier, the target transformation matrix represents the spatial transformation relationship between the carrier coordinate system and the regional coordinate system of the working area, and the target transformation matrix is ​​used to control the robot to perform the operation.

[0005] According to an embodiment of the present disclosure, the movable camera arranged on the above-mentioned robot is calibrated according to the above-mentioned reference coordinates and the preset vehicle coordinates to obtain a target transformation matrix, including: spatially transforming the above-mentioned reference coordinates based on a first transformation matrix that characterizes the spatial transformation relationship between the above-mentioned reference camera coordinate system and the above-mentioned regional coordinate system to obtain the reference regional coordinates of the above-mentioned marking points in the above-mentioned regional coordinate system; and determining the above-mentioned target transformation matrix based on the reference regional coordinates and the preset vehicle coordinates of each of the above-mentioned multiple marking points.

[0006] According to an embodiment of the present disclosure, the above-mentioned vehicle includes a base and a vehicle body, the above-mentioned marking points are arranged on the above-mentioned vehicle body, and the above-mentioned robot includes a mechanical arm installed on the above-mentioned vehicle body through the above-mentioned base; wherein, based on the reference area coordinates and preset vehicle coordinates of each of the above-mentioned multiple marking points, determining the above-mentioned target transformation matrix includes: calibrating according to the reference area coordinates and vehicle coordinates of each of the above-mentioned multiple marking points to obtain a first intermediate transformation matrix, and the above-mentioned first intermediate transformation matrix represents the spatial transformation relationship between the above-mentioned regional coordinate system and the above-mentioned vehicle body coordinate system; fusing the above-mentioned first intermediate transformation matrix and the preset second intermediate transformation matrix to obtain the above-mentioned target transformation matrix, wherein the above-mentioned second intermediate transformation matrix represents the spatial transformation relationship between the above-mentioned vehicle body coordinate system and the base coordinate system of the above-mentioned base, and the above-mentioned target transformation matrix represents the spatial transformation relationship between the above-mentioned base coordinate system and the above-mentioned regional coordinate system.

[0007] According to an embodiment of the present disclosure, the above-mentioned vehicle includes a base and a vehicle body, the above-mentioned marking points are arranged on the above-mentioned vehicle body, and the above-mentioned robot includes a mechanical arm installed on the above-mentioned vehicle body through the above-mentioned base; the movable camera arranged on the robot is calibrated according to the above-mentioned reference coordinates and the preset vehicle coordinates, and the target transformation matrix obtained includes: calibration based on the above-mentioned reference coordinates and the preset vehicle coordinates to obtain a second transformation matrix characterizing the spatial transformation relationship between the above-mentioned reference camera coordinate system and the above-mentioned vehicle coordinate system; and based on the above-mentioned second transformation matrix, the preset second intermediate transformation matrix and the first transformation matrix, the above-mentioned target transformation matrix is ​​obtained, wherein the above-mentioned first transformation matrix characterizes the spatial transformation relationship between the above-mentioned reference camera coordinate system and the above-mentioned regional coordinate system, and the above-mentioned second intermediate transformation matrix characterizes the spatial transformation relationship between the vehicle body coordinate system of the above-mentioned vehicle body and the base coordinate system of the above-mentioned base.

[0008] According to an embodiment of the present disclosure, the above-mentioned marked image includes a first marked image and a second marked image respectively captured by a first reference camera and a second reference camera; wherein, detecting the above-mentioned marked image to obtain the reference coordinates of the above-mentioned marked point in the reference camera coordinate system of the above-mentioned reference camera includes: detecting the above-mentioned first marked image and the above-mentioned second marked image to obtain the first image point coordinates of the above-mentioned marked point in the first image coordinate system, and the second image point coordinates of the above-mentioned marked point in the second image coordinate system; and performing spatial transformation on the above-mentioned first image point coordinates and the above-mentioned second image point coordinates to obtain the above-mentioned reference coordinates, wherein the above-mentioned reference coordinates are related to the first reference camera coordinate system of the above-mentioned first reference camera, or the above-mentioned reference coordinates are related to the second reference coordinate system of the above-mentioned second reference camera.

[0009] According to an embodiment of the present disclosure, spatial transformation is performed on the first image point coordinates and the second image point coordinates to obtain the reference coordinates, including: processing the first image point coordinates and the second image point coordinates using a coordinate conversion function determined based on reference camera parameters of the first reference camera and the second reference camera to obtain initial reference coordinates of the marking point; processing the first image point coordinates and the second image point coordinates using a differential function corresponding to the coordinate conversion function to obtain reference coordinate errors of the initial reference coordinates; and updating the initial reference coordinates based on the reference coordinate errors to obtain the reference coordinates.

[0010] According to an embodiment of the present disclosure, spatial transformation is performed on the first image point coordinates and the second image point coordinates to obtain the reference coordinates, including: processing the first image point coordinates and the second image point coordinates using an error distribution function to obtain an augmented image point coordinate error, wherein the error distribution function characterizes that the coordinate error distribution of the first image point coordinates or the second image point coordinates in the image coordinate system satisfies a preset distribution morphology condition; processing the augmented image point coordinate error using a differential function corresponding to the coordinate transformation function to obtain a plurality of augmented point coordinate errors of the marking point in the reference camera coordinate system, wherein the coordinate transformation function is determined based on reference camera parameters of the first reference camera and the second reference camera; determining a plurality of augmented point coordinates of the marking point based on the plurality of augmented point coordinate errors; and determining the reference coordinates of the marking point from the plurality of augmented point coordinates.

[0011] According to an embodiment of the present disclosure, determining the reference coordinates of the above-mentioned marking point from the above-mentioned multiple amplification point coordinates includes: performing plane fitting based on the above-mentioned multiple amplification point coordinates to obtain a fitting plane; and determining the reference coordinates of the above-mentioned marking point from the above-mentioned multiple amplification point coordinates of the above-mentioned marking point based on the positional relationship between the above-mentioned amplification point coordinates and the above-mentioned fitting plane.

[0012] Another aspect of the present disclosure further provides a robot control method, wherein the robot includes a vehicle and a movable camera, and marking points are set on the vehicle. The method includes: controlling the robot to perform operations based on the target transformation matrix and the target image captured by the movable camera, wherein the target transformation matrix is ​​determined according to the camera calibration method provided in an embodiment of the present disclosure.

[0013] According to an embodiment of the present disclosure, controlling the robot to perform operations based on the target transformation matrix and the target image captured by the movable camera includes: performing target object detection based on the target image to obtain the initial object coordinates of the target object in the vehicle coordinate system; performing spatial transformation on the initial object coordinates based on the target transformation matrix to obtain the target object coordinates of the target object in the regional coordinate system; and controlling the robot to perform operations based on the target object coordinates.

[0014] Another aspect of the present disclosure provides a camera calibration device, including: an acquisition module, used to acquire a marking image, when a reference camera is stationary relative to a working area where a robot equipped with a carrier performs an operation, the marking point set on the carrier is collected to obtain the marking image; a detection module, used to detect the marking image to obtain the reference coordinates of the marking point in the reference camera coordinate system of the reference camera; a calibration module, used to calibrate a movable camera set on the robot according to the reference coordinates and preset carrier coordinates to obtain a target transformation matrix, wherein the preset carrier coordinates represent the position of the marking point in the carrier coordinate system, the target transformation matrix represents the spatial transformation relationship between the carrier coordinate system and the regional coordinate system of the working area, and the target transformation matrix is ​​used to control the robot to perform operations.

[0015] Another aspect of the present disclosure further provides a robot control device, wherein the robot includes a vehicle and a movable camera, and marking points are set on the vehicle. The device includes: a control module, used to control the robot to perform operations based on the target transformation matrix and the target image captured by the movable camera, wherein the target transformation matrix is ​​determined according to the camera calibration method provided in an embodiment of the present disclosure.

[0016] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0017] The fourth aspect of the present disclosure further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the above computer program or instructions are executed by a processor.

[0018] The fifth aspect of the present disclosure further provides a computer program product, including a computer program or instructions, which implement the steps of the above method when the above computer program or instructions are executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0020] Figure 1 A schematic diagram showing an application scenario of a camera calibration method, a robot control method, and a device according to an embodiment of the present disclosure;

[0021] Figure 2 A flowchart of a camera calibration method according to an embodiment of the present disclosure is schematically shown;

[0022] Figure 3 A schematic diagram schematically shows a marking point according to an embodiment of the present disclosure;

[0023] Figure 4 The following schematically shows an application scenario of the camera calibration method according to an embodiment of the present disclosure;

[0024] Figure 5 A diagram schematically shows an application scenario of a camera calibration method according to another embodiment of the present disclosure;

[0025] Figure 6 A flowchart of a robot control method according to an embodiment of the present disclosure is schematically shown;

[0026] Figure 7 The structure block diagram of the camera calibration device according to the embodiment of the present disclosure is schematically shown;

[0027] Figure 8 A schematic diagram of a robot control device according to an embodiment of the present disclosure is shown; and

[0028] Fig. 9 A block diagram of an electronic device suitable for implementing a camera calibration method and a robot control method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0029] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0030] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0031] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0032] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0033] In the technical solution of the present disclosure, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0034] The inventors found that intelligent robots can be used to perform operations in the work area. For example, an intelligent robot equipped with a vehicle such as an AGV (Automated Guided Vehicle) can travel to the work area to assemble or weld the parts to be assembled, so as to realize automated intelligent equipment manufacturing. The inventors found that the intelligent robot with automatic movement function has low accuracy in performing operations and is difficult to meet the actual requirements of intelligent equipment manufacturing.

[0035] The embodiments of the present disclosure provide a camera calibration method, a robot control method, an apparatus, a device and a medium. The camera calibration method includes: acquiring a marker image, when a reference camera is stationary relative to a working area where a robot equipped with a carrier performs an operation, collecting a marker point set on the carrier to obtain a marker image; detecting the marker image to obtain the reference coordinates of the marker point in the reference camera coordinate system of the reference camera; calibrating a movable camera set on the robot according to the reference coordinates and preset carrier coordinates to obtain a target transformation matrix, wherein the preset carrier coordinates represent the position of the marker point in the carrier coordinate system of the carrier, the target transformation matrix represents the spatial transformation relationship between the carrier coordinate system and the regional coordinate system of the working area, and the target transformation matrix is ​​used to control the robot to perform the operation.

[0036] According to an embodiment of the present disclosure, by acquiring a marker image captured of the robot's vehicle when the reference camera and the working area are relatively still, and detecting the reference coordinates of the marker points in the reference camera coordinate system based on the marker image captured by the reference camera for the marker points of the vehicle, it is possible to spatially transform the position of the moving vehicle, and characterize the position of the robot's vehicle in the reference camera coordinate system based on the reference coordinates. The robot's movable camera is calibrated by the reference coordinates and the vehicle coordinates of the marker points in the preset vehicle coordinate system, so that the target transformation matrix obtained represents the spatial transformation relationship between the regional coordinate system and the vehicle coordinate system, and then the target image captured by the movable camera can be spatially transformed based on the target transformation matrix to indicate the accurate position of the target object in the working area where the robot performs the operation, thereby improving the accuracy of the robot's operation in the working area, and then improving the robot's operating efficiency.

[0037] An embodiment of the present disclosure also provides a robot control method, which includes: controlling the robot to perform operations based on a target transformation matrix and a target image captured by a movable camera, wherein the target transformation matrix is ​​determined based on a camera calibration method provided by an embodiment of the present disclosure.

[0038] Figure 1 The application scenario diagram of the camera calibration method, robot control method and device according to the embodiments of the present disclosure is schematically shown.

[0039] like Figure 1As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, a server 105, a reference camera 111, and a robot 112. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, the server 105, the reference camera 111, and the robot 112. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0040] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0041] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0042] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0043] The reference camera 111 may include any type of image acquisition device such as a still camera, a video camera, etc. The reference camera 111 may be disposed near a working area Z111 where the robot 112 performs operations.

[0044] The robot 112 may be any type of robot such as a transport robot, a grasping robot, or an inspection robot.

[0045] In this embodiment, the robot 112 may be provided with a carrier and a movable camera, and the movable camera may be provided on a robotic arm of the robot 112 .

[0046] It should be noted that the camera calibration method and robot control method provided in the embodiments of the present disclosure can be executed by the server 105. Accordingly, the camera calibration device and robot control device provided in the embodiments of the present disclosure can be set in the server 105. The camera calibration method and robot control method provided in the embodiments of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103, the robot 112 and / or the server 105. Accordingly, the camera calibration device and robot control device provided in the embodiments of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0047] Alternatively, the camera calibration method and robot control method provided in the embodiments of the present disclosure may also be executed by any one or more of the first terminal device 101, the second terminal device 102, the third terminal device 103, and the robot 112. The camera calibration device and robot control device provided in the embodiments of the present disclosure may be set in any one or more of the first terminal device 101, the second terminal device 102, the third terminal device 103, and the robot 112.

[0048] It should be understood that Figure 1 The number of terminal devices, cameras, robots, networks, and servers in the embodiment is only for illustration. Any number of terminal devices, cameras, robots, networks, and servers may be provided as required.

[0049] Figure 2 The flowchart of the camera calibration method according to an embodiment of the present disclosure is schematically shown.

[0050] like Figure 2 As shown, the camera calibration method of this embodiment includes operations S210 to S230.

[0051] In operation S210, a marker image is acquired.

[0052] In operation S220 , the marker image is detected to obtain reference coordinates of the marker point in a reference camera coordinate system of the reference camera.

[0053] In operation S230 , a movable camera disposed on the robot is calibrated according to the reference coordinates and the preset vehicle coordinates to obtain a target transformation matrix.

[0054] According to an embodiment of the present disclosure, a marking image is acquired by capturing a marking point set on a carrier when a reference camera is stationary relative to a working area where a robot equipped with a carrier performs an operation. The robot may include a carrier and a movable camera, and the movable camera may be movable relative to the working area. The marking point is set on the carrier, for example, the marking point may be set on the plane of the carrier body. The reference camera may be relatively stationary with respect to the working area during the robot performing the operation. The reference camera may be set in the working area or near the working area.

[0055] According to an embodiment of the present disclosure, the working area may be an area where the robot performs operations, for example, it may be an area where a transport robot performs a transport task, or it may also be an assembly area where an assembly robot performs component assembly.

[0056] According to the embodiments of the present disclosure, the movable camera or the reference camera may be any type of image acquisition device, for example, a still camera, a video camera, a depth camera, etc. The embodiments of the present disclosure do not limit the number or type of the movable camera or the reference camera.

[0057] According to an embodiment of the present disclosure, the marking point may include a graphic with a significant appearance feature and a clear size. The marking point may include one or more, the spacing between the multiple marking points may be a preset fixed spacing, and the size parameters of the marking points may be obtained through precise measurement.

[0058] According to an embodiment of the present disclosure, the marking points set on the vehicle can be recorded in the marking image, the marking image can be detected based on the target detection algorithm to obtain the reference coordinates of the marking points in the reference camera coordinate system, and the position of the marking points in the vehicle in the reference camera coordinate system can be represented based on the reference coordinates.

[0059] In one example, the marked image can be detected based on the target detection algorithm to obtain the image coordinates (x 1 ,y 1 ), based on the reference camera parameters of the reference camera, the image coordinates (x 1 ,y 1 ) to transform the coordinate system and obtain the reference coordinates.

[0060] According to an embodiment of the present disclosure, the preset vehicle coordinate characterization mark point is located in the vehicle coordinate system of the vehicle. The vehicle coordinate system may be a spatial coordinate system corresponding to the vehicle, for example, the vehicle coordinate system may be a vehicle body coordinate system corresponding to the vehicle body. Alternatively, the vehicle coordinate system may also be a coordinate system corresponding to other structural components of the vehicle, for example, the vehicle coordinate system may also be a base coordinate system corresponding to the base of the vehicle. The base may be used to install the operating components of the robot, for example, the base may be used to install a robotic arm for performing a grasping operation. By determining a target transformation matrix representing the spatial transformation relationship between the regional coordinate system and the base coordinate system, the robotic arm of the robot can achieve rapid calibration of the movable camera while freely rotating around the base, thereby achieving precise positioning of the robot and improving the operation execution accuracy and efficiency of the robot.

[0061] According to an embodiment of the present disclosure, the target transformation matrix represents the spatial transformation relationship between the carrier coordinate system and the regional coordinate system of the working area, and the target transformation matrix is ​​used to control the robot to perform operations. Based on the target transformation matrix, the target image captured by the movable camera for the working area is detected and spatially transformed, so as to obtain the more accurate coordinate positions of the target objects such as assembly parts, objects to be transported, obstacles, etc. in the target image in the regional coordinate system, so as to improve the operation accuracy of the robot in performing operations in the working area.

[0062] Figure 3 A schematic diagram of marking points according to an embodiment of the present disclosure is schematically shown.

[0063] like Figure 3 As shown, the marking points may be two-dimensional marking points (or 2D Mark points). The marking points may be arranged in a fixed equidistant array, and the number of marking points may be 20. Each marking point may be a black circle with a fixed diameter, so as to more accurately identify the image coordinates of the marking point in the image coordinate system from the marking image.

[0064] According to an embodiment of the present disclosure, the carrier includes a base and a carrier body, the marking point is set on the carrier body, and the robot includes a robotic arm installed on the carrier body through the base.

[0065] In one example, the vehicle body may be represented as a shell of the vehicle, and the marking points are arranged on the surface of the vehicle body.

[0066] In one example, the base may be a rotatable base of the robot arm, and the base may be rotated relative to the carrier body to control the freedom of rotation of the robot arm on the surface of the carrier body.

[0067] According to an embodiment of the present disclosure, calibrating a movable camera according to reference coordinates and preset vehicle coordinates to obtain a target transformation matrix may include: calibrating based on the reference coordinates and the preset vehicle coordinates to obtain a second transformation matrix that characterizes the spatial transformation relationship between the reference camera coordinate system and the vehicle coordinate system; and obtaining the target transformation matrix based on the second transformation matrix, a preset second intermediate transformation matrix and the first transformation matrix.

[0068] According to an embodiment of the present disclosure, the preset vehicle coordinates may represent the coordinates of the marking point in the vehicle body coordinate system O p The image coordinates of each of the multiple markers in the preset image captured by the preset camera can be determined based on the target detection algorithm, and the image coordinates in the preset image can be converted into the preset vehicle coordinates in the vehicle body coordinate system based on the camera intrinsic parameters and camera extrinsic parameters of the preset camera.

[0069] According to the embodiments of the present disclosure, when there are multiple marking points, the reference coordinates and the preset vehicle coordinates corresponding to the multiple marking points can be processed by any calibration algorithm such as the Zhang Zhengyou calibration method to obtain the reference camera coordinate system. c1 and the vehicle body coordinate system O p The second transformation matrix of the spatial transformation relationship between .

[0070] According to an embodiment of the present disclosure, the first conversion matrix Characterize the reference camera coordinate system O c1 With the regional coordinate system O d The spatial transformation relationship between them, the second intermediate transformation matrix The vehicle body coordinate system O representing the vehicle body p The base coordinate system O of the base b The spatial transformation relationship between them.

[0071] In one example, a transformation matrix representing the spatial transformation relationship between two different coordinate systems may be determined based on the following operations.

[0072] For example, the transformation matrix representing the homogeneous coordinate transformation from the E coordinate system to the F coordinate system can be expressed as formula (1).

[0073] (1)

[0074] Among them, R represents the rotation matrix around the three coordinate values, and V represents the translation vector along the coordinate axis of the three coordinate values. , then R can be obtained by multiplying the rotation matrices of the three coordinates. The rotation matrix R is expressed as formula (2).

[0075] (2)

[0076] Among them, r i is the coordinate system rotation parameter, α, β and θ represent the angles between the corresponding two coordinate axes in the two coordinate systems, i=1,2,…,9. Respectively represent the translation distance along the three coordinate axes. To make the calculation more convenient, the transformation matrix can be expressed in the form of homogeneous coordinates as shown in formula (3).

[0077] (3)

[0078] It should be understood that the conversion matrix involved in the embodiment of the present disclosure can be determined based on the above formulas (1) to (3). The conversion matrix involved in the embodiment of the present disclosure includes but is not limited to the first conversion matrix, the first intermediate conversion matrix, the second conversion matrix, the target conversion matrix, etc. The embodiment of the present disclosure will not repeat the process of determining the conversion matrix.

[0079] Figure 4 The application scenario diagram of the camera calibration method according to an embodiment of the present disclosure is schematically shown.

[0080] like Figure 4 As shown, the application scenario may include a reference camera 411, a computing unit 412, a workbench 413 and a movable robot. The movable robot includes a carrier having a carrier body 4211 and a base 4212, a mechanical arm 423 mounted on the base 4212 and a movable camera 422 mounted on the mechanical arm 423. The reference camera 411 may be a monocular camera, and the reference camera 411 and the computing unit 412 are communicatively connected via a wired communication link or a wireless communication link. The reference camera 411, the computing unit 412 and the robot may be communicatively connected via a wired communication link or a wireless communication link. The workbench 413 may be a working area where the robot's mechanical arm 423 performs component assembly operations.

[0081] Reference camera coordinate system of reference camera 411 The coordinate system of the workbench 413 includes the coordinate axes x1, y1 and z1. The carrier body coordinate system of the carrier body 4211 includes the coordinate axes xd, yd and zd. The base coordinate system of the base 4212 includes the coordinate axes xp, yp and zp. The movable camera coordinate system of the movable camera 422 includes coordinate axes xb, yb and zb. Includes coordinate axes x3, y3 and z3.

[0082] A plurality of marking points are set on the carrier body platform of the carrier body 4211, and the reference camera 411 can acquire images of the plurality of marking points on the carrier body platform to obtain a marking image, which can be transmitted to the calculation unit 412 for processing.

[0083] In one example, the computing unit 412 may obtain the preset vehicle coordinates of each of the plurality of marking points from the storage device of the robot to execute the camera calibration method provided in the embodiment of the present disclosure to obtain the target transformation matrix.

[0084] In one example, the computing unit 412 may transmit the marking image to the edge processor of the robot, and the edge processor of the robot may calibrate the movable camera 422 according to the preset vehicle coordinates of each of the plurality of marking points to obtain a target transformation matrix.

[0085] In one example, the camera calibration method provided by the embodiment of the present disclosure can be performed based on the computing unit 412 and the edge processor of the robot. For example, the computing unit 412 can detect the marked image to obtain the respective reference coordinates of the multiple marked points, and the computing unit 412 transmits the respective reference coordinates of the multiple marked points to the storage device of the robot. The edge processor of the robot calls the respective reference coordinates of the multiple marked points and the preset vehicle coordinates from the storage device to calibrate the movable camera to obtain the target transformation matrix.

[0086] The camera calibration method provided by the embodiment of the present disclosure realizes automatic calibration of the movable camera of the robot by adding an external reference camera with a fixed viewing angle, and can realize automatic positioning of the robot, thereby improving the positioning accuracy, accelerating the positioning process, and improving the degree of automation.

[0087] In one example, based on the second conversion matrix, the preset second intermediate conversion matrix and the first conversion matrix, a target conversion matrix is ​​obtained, which can be determined based on the following operations.

[0088] To obtain the base coordinate system of the robot To the local coordinate system of the desktop workbench The target transformation matrix of the spatial transformation relationship between , the coordinate transformation relationship in different coordinate systems can be expressed as .in, is the second transformation matrix, is the first transformation matrix, is the second intermediate transformation matrix. In order to determine the second transformation matrix , the first transformation matrix , the second intermediate transformation matrix The specific parameters of each rotation transformation matrix in can be determined according to the robot system structure, sensor mathematical model, and robot arm kinematics equation. For example, the target transformation matrix can be determined based on the following steps S11 to S17 .

[0089] S11, determine the fixture coordinate system of the end fixture of the robot arm to the base coordinate system of the robot arm through the robot arm system structure and kinematic model The fixture-base transformation matrix between , step S11 can be calibrated only once during system deployment, and the fixture base conversion matrix is ​​determined Then, the camera calibration method provided by the embodiment of the present disclosure is executed. For example, a Denavit-Hartenberg model of the robot arm can be established for the base coordinate system and the fixture coordinate system, and the fixture base transformation matrix can be determined through the robot arm kinematic model. .

[0090] S12, determine the vehicle body coordinate system based on Zhang Zhengyou calibration algorithm To the movable camera coordinate system The area between the camera transformation matrix can be moved An imaging model expressed in formula (4) can be established for the movable marker image captured by the movable camera.

[0091] (4);

[0092] and represents the imaging point of the target marker in the movable camera, represents the focal length of the movable camera, , and Represents the coordinates of the target marker in the world coordinate system. All parameters in formula (4) can be obtained through Zhang Zhengyou calibration algorithm.

[0093] S13, calibrate the movable camera through the robot arm hand-eye calibration algorithm to obtain the movable camera coordinate system Transformation matrix of the movable camera fixture to the end-of-arm fixture coordinate system Step S13 only needs to be calibrated once during the system deployment process, and does not need to be calibrated again when executing the camera calibration method of the embodiment of the present disclosure.

[0094] S14, by Get the vehicle body coordinate system With base coordinate system The second intermediate transformation matrix between , S14 performs a calibration of the second intermediate conversion matrix during system deployment The data may be stored in a storage device used to execute the camera calibration method according to an embodiment of the present disclosure.

[0095] S15, determine the regional coordinate system through Zhang Zhengyou calibration algorithm To the reference camera coordinate system The first transformation matrix between Step S15 can be executed based on a processing procedure similar to step S12, and the embodiments of the present disclosure will not be repeated here.

[0096] S16, use Zhang Zhengyou calibration algorithm to process the reference coordinates of multiple markers and the preset vehicle coordinates to obtain the reference camera coordinate system To regional coordinate system The second transformation matrix between .

[0097] S17, according to To process the second transformation matrix , the preset second intermediate conversion matrix and the first transformation matrix , get the base coordinate system To regional coordinate system The target transformation matrix between .

[0098] It should be noted that the above steps S11 to S15 can be performed before executing the camera calibration method provided in the embodiment of the present disclosure to obtain , , , , The system operation phase can be understood as executing the camera calibration method provided by the embodiment of the present disclosure, and the target transformation matrix can be obtained by executing steps S16 and S17.

[0099] In one example, the robot may initiate a positioning request to the computing unit 412 after controlling the vehicle to stop moving. The computing unit 412 obtains the marker image captured by the reference camera and starts the calibration process. Then, step S17 is executed to determine the target transformation matrix The computing unit 412 may execute the camera calibration method provided by the embodiment of the present disclosure again after the robot sends a positioning request next time.

[0100] According to an embodiment of the present disclosure, the marker image includes a first marker image and a second marker image acquired by a first reference camera and a second reference camera, respectively.

[0101] In one example, the first reference camera and the second reference camera may be fixedly installed as binocular cameras at a location near the working area, so as to facilitate movable camera calibration for the robot to perform operations on the working area.

[0102] According to an embodiment of the present disclosure, detecting a marked image to obtain the reference coordinates of the marked point in the reference camera coordinate system of the reference camera may include: detecting a first marked image and a second marked image to obtain the first image point coordinates of the marked point in the first image coordinate system, and the second image point coordinates of the marked point in the second image coordinate system; and performing spatial transformation on the first image point coordinates and the second image point coordinates to obtain the reference coordinates.

[0103] According to an embodiment of the present disclosure, the reference coordinates are related to a first reference camera coordinate system of a first reference camera, or the reference coordinates are related to a second reference coordinate system of a second reference camera.

[0104] In one example, the reference coordinates are coordinates of the marking point in the first reference coordinate system.

[0105] In one example, the reference coordinates are coordinates of the marking point in the second reference coordinate system.

[0106] It should be noted that, for the convenience of explaining the camera calibration method provided in the embodiment of the present disclosure, it can be set that when the marked image includes the first marked image and the second marked image, the reference coordinates are set as the coordinates of the marked point in the first reference coordinate system, and it is not used to limit the coordinate system type corresponding to the reference coordinates.

[0107] According to an embodiment of the present disclosure, the first marked image and the second marked image can be processed based on the target detection algorithm to obtain the respective marked point regions of the marked point in the first marked image and the second marked image. The coordinates of the first image point (x 1 ,y 1 ) and the second image point coordinates (x 2 ,y 2 ).

[0108] In one example, the marking point is a circular point with significant visual features, which can be conveniently extracted and segmented in the first marking image or the second marking image. The first marking image or the second marking image is processed by an image processing algorithm to obtain the center coordinates of all marking points, which can be determined based on the following operations. Perform black and white threshold segmentation on the first marking image or the second marking image to obtain a black and white image; perform circular contour detection in the black and white image to obtain a circular circumscribed contour area of ​​the marking point; average all pixel points in the circular circumscribed contour area to obtain the center coordinates. The coordinate value of the center coordinate is used as the coordinate value of the first image point (x 1,y 1 ) and the second image point coordinates (x 2 ,y 2 ).

[0109] According to an embodiment of the present disclosure, spatially transforming the first image point coordinates and the second image point coordinates to obtain the reference coordinates may include processing the first image point coordinates and the second image point coordinates to obtain the reference coordinates based on the camera intrinsic parameters and camera extrinsic parameters of each of the first reference camera and the second reference camera.

[0110] In one example, the first image point coordinate (x) may be determined based on the following formula (5): 1 ,y 1 ) and the coordinates in the first reference coordinate system, formula (5) represents the imaging model of the marker point in the first reference coordinate system.

[0111] (5);

[0112] Among them, z 1 represents the coordinate value of the first image point on the z-axis, f 1 Indicates the focal length of the first reference camera.

[0113] The coordinates of the second image point (x 2 ,y 2 ) is the spatial transformation relationship between the coordinates in the second reference coordinate system, and formula (6) represents the imaging model of the marker point in the second reference coordinate system.

[0114] (6);

[0115] Among them, z 2 represents the coordinate value of the second image point on the z-axis, f 2 Indicates the focal length of the second reference camera.

[0116] Determine the reference space transformation matrix between the first reference coordinate system and the second reference coordinate system In the case of, the reference coordinates can be determined based on formula (7) [ , , ].

[0117] (7);

[0118] Combining formulas (5) to (7), we can obtain the following formula (8).

[0119] (8);

[0120] In formula (8), when determining the coordinates of the first image point (x 1 ,y 1 ) and the second image point coordinates (x 2 ,y 2 ) is a known quantity, the reference coordinates can be determined according to formula (9): , , ].

[0121] (9)

[0122] According to an embodiment of the present disclosure, by determining the reference coordinates of the marking point in the reference coordinate system through the first marking image and the second marking image captured by different reference cameras, it is possible to more accurately detect the coordinate position of the marking point in the reference coordinate system through the coordinate position of the marking point in different image coordinate systems, so as to improve the coordinate accuracy represented by the reference coordinates. Thus, by calibrating the movable camera according to the reference coordinates and the preset vehicle coordinates, the obtained target transformation matrix can more accurately represent the spatial transformation relationship between the vehicle coordinate system of the moving robot and the regional coordinate system of the working area, so as to improve the calibration accuracy for the movable camera, and then the target image captured by the movable camera for the working area can be processed by the target transformation matrix to obtain the precise coordinates of the target object in the regional coordinate system, so as to improve the operation accuracy of the robot for the working area.

[0123] In one example, the first reference camera and the second reference camera can perform periodic image acquisition of the marking points set on the carrier during the movement based on a preset calibration cycle to obtain the first marking image and the second marking image of each calibration cycle. The camera calibration method provided by the embodiment of the present disclosure is performed for the first marking image and the second marking image of each calibration cycle, so that the movable camera of the robot during the movement can be periodically calibrated to obtain the target conversion matrix of each calibration cycle. In this way, when the robot moves through the carrier, the target conversion matrix of each calibration cycle can be used to control the robot during the movement to accurately perform operations on the working area, so that the robot during the movement can be controlled to accurately perform operations on the working area by setting a shorter calibration cycle or a calibration cycle that is compatible with the movement mode of the carrier, avoiding the need for the robot to stop the movement of the carrier to calibrate the movable camera of the robot, thereby improving the operation accuracy and work efficiency of the robot.

[0124] According to an embodiment of the present disclosure, performing spatial transformation on the first image point coordinates and the second image point coordinates to obtain the reference coordinates may include: processing the first image point coordinates and the second image point coordinates using a coordinate transformation function determined based on reference camera parameters of the first reference camera and the second reference camera to obtain initial reference coordinates of the marking point; processing the first image point coordinates and the second image point coordinates using a differential function corresponding to the coordinate transformation function to obtain a reference coordinate error of the initial reference coordinates; and updating the initial reference coordinates based on the reference coordinate error to obtain the reference coordinates.

[0125] According to an embodiment of the present disclosure, the reference coordinates of the marker point in the reference coordinate system can be obtained by spatially transforming the image coordinates of the marker point in different first marker images and second marker images. When the reference coordinates are used as the coordinate position of the marker point in the world coordinate system, due to the influence of factors such as pixel loss and affine transformation in the processing of the first marker image and the second marker image, the coordinate values ​​of the first image point coordinates and the second image point coordinates are different. and , and There will be a certain error, and based on the error propagation, coordinate positioning errors will occur when the reference coordinates are used as world coordinates.

[0126] According to an embodiment of the present disclosure, the coordinate conversion function may be a function constructed based on the reference camera parameters for converting the first image point coordinates and the second image point coordinates into the reference coordinates. For example, the coordinate conversion function may be expressed based on formula (7). The reference camera parameters may include the camera intrinsic parameters and camera extrinsic parameters of the first reference camera and the second reference camera, respectively.

[0127] In one example, the initial reference coordinates may be obtained by performing a calculation operation based on formula (7) or formula (9).

[0128] In one example, the differential function corresponding to the coordinate conversion function may be a Jacobian matrix obtained by performing a total differential operation on the coordinate conversion function (eg, formula (7)). For example, the differential function may be expressed as formula (10).

[0129] (10);

[0130] Formula (10) is calculated based on the contents shown in Table 1 below.

[0131] Table 1

[0132] #timg# [1] #timg# [2] #timg# [3] #timg# [4] #timg# [5] #timg# [6]

[0133] Based on formula (10), the source and propagation characteristics of the coordinate error of the reference coordinates can be expressed. The coordinate solution accuracy of the reference coordinate point is not linearly related, and contains a more complex coupling relationship. It is impossible to simply use the pixel error to substitute into formula (9) to solve the error range. And observing formula (10), it can be seen that is a multivariable equation system. The reference coordinate error of the initial reference coordinate can be obtained by solving the Hessian matrix and the Newton-Raphson iterative algorithm to solve formula (10): Based on the reference coordinate errors of the initial reference coordinates of the multiple markers, the error correction calculation is performed on the initial reference coordinates of the multiple markers, and the corrected reference coordinates of the multiple markers can be obtained. In this way, the reference coordinate positions of the multiple markers in the reference coordinate system can be accurately detected, and the detection accuracy of the target transformation matrix can be improved, thereby improving the calibration accuracy for the movable camera.

[0134] According to an embodiment of the present disclosure, performing spatial transformation on the first image point coordinates and the second image point coordinates to obtain the reference coordinates may also include: processing the first image point coordinates and the second image point coordinates using an error distribution function to obtain an augmented image point coordinate error; processing the augmented image point coordinate error using a differential function corresponding to the coordinate transformation function to obtain multiple augmented point coordinate errors of the marked point in the reference camera coordinate system; determining multiple augmented point coordinates of the marked point based on the multiple augmented point coordinate errors; and determining the reference coordinates of the marked point from the multiple augmented point coordinates.

[0135] According to an embodiment of the present disclosure, the error distribution function characterizes that the coordinate error distribution of the first image point coordinates or the second image point coordinates in the image coordinate system satisfies a preset distribution morphology condition; the coordinate error distribution of the first image point coordinates or the second image point coordinates in the image coordinate system can determine the error distribution morphology of the image coordinates based on multi-sample detection, and determine the error distribution function according to the preset distribution morphology.

[0136] In one example, the error distribution function may be determined to be a Gaussian function, and the Gaussian function may indicate that the coordinate error distribution of the first image point coordinate or the second image point coordinate in the image coordinate system satisfies a normal distribution condition.

[0137] According to an embodiment of the present disclosure, the coordinate conversion function is determined based on the reference camera parameters of the first reference camera and the second reference camera. For example, the coordinate conversion function can be expressed based on formula (7) or (9) in the above example, and the embodiments of the present disclosure will not be repeated.

[0138] According to an embodiment of the present disclosure, processing the first image point coordinate and the second image point coordinate using the error distribution function may include: To process the first image point coordinates and the second image point coordinates, . Get the coordinates of the first image point (x 1 ,y 1 ) and the second image point coordinates (x 2 ,y 2 ) The coordinate error of the amplified image points corresponding to each of the amplified image points. The coordinate error of the amplified image points can represent the coordinate error corresponding to each of the preset number N amplified points, where N is an integer greater than 1. Based on the coordinate error of the amplified image points, the error limit corresponding to each of the N amplified points can be determined. ,in , the error limit can be understood as the coordinate error of multiple image coordinate values ​​of the amplified point. Substitute the generated coordinate error of each amplified image point into the differential function (such as formula (10)) to obtain the coordinate error of multiple amplified points in the reference camera coordinate system. ,in Therefore, the coordinate error range of the marking point in the reference coordinate system can be expressed based on the coordinate error of the amplified point. The coordinates of the amplification points in the reference coordinate system can be determined , for example, the coordinates of the amplification points may include K×N, where , is the number of marker points. Each marker point corresponds to N amplification points.

[0139] According to an embodiment of the present disclosure, determining the reference coordinates of a marking point from multiple amplification point coordinates can include calculating the distance between each of the N amplification points and a preset plane for each marking point, and determining the amplification point closest to the preset plane as the reference coordinate of the marking point in the reference coordinate system, thereby completing error correction for the reference coordinates and improving the position representation accuracy of the reference coordinates.

[0140] In one example, the preset plane may be determined based on a carrier plane on which the marking points are set. For example, a spatial transformation may be performed based on the carrier plane on which the marking points are set to obtain the preset plane in the reference coordinate system.

[0141] According to the embodiments of the present disclosure, the first image coordinates and the second image coordinates are processed based on the error distribution function, and then the coordinate error of the amplified point in the reference coordinate system is determined by determining the extended image point coordinate error, so that the distribution form of the coordinate error of the image coordinates of the marked point can be quantified more accurately. Processing the amplified image point coordinate error based on the differential function can make the obtained amplified point coordinate error more accurately reflect the propagation characteristics of the image coordinate error propagating into the reference coordinate system, and then determine the multiple amplified point coordinates of each marked point through the amplified point coordinate error, and determine the reference coordinates of the marked point from the multiple expanded store coordinates based on the preset plane, so that the coordinate errors existing in the detected image coordinates can be corrected more accurately, and the representation accuracy of the marked point in the reference coordinate system can be improved, and then the calibration accuracy for the movable camera can be improved, while avoiding the large computational overhead and computational efficiency problems caused by solving the Newton-Raphson iterative algorithm, and further improving the execution efficiency of the robot operation.

[0142] According to an embodiment of the present disclosure, determining the reference coordinates of a marking point from a plurality of amplification point coordinates includes: performing plane fitting based on the plurality of amplification point coordinates to obtain a fitting plane; and determining the reference coordinates of the marking point from the plurality of amplification point coordinates of the marking point based on the positional relationship between the amplification point coordinates and the fitting plane.

[0143] In one example, performing plane fitting based on multiple amplification point coordinates may include performing plane fitting based on the full amount of K×N amplification point coordinates to obtain a fitting plane. The distance between the multiple amplification point coordinates and the fitting plane is calculated to obtain the positional relationship between the amplification point coordinates and the fitting plane. Among the N amplification point coordinates corresponding to each marking point, the amplification point whose positional relationship characterizes the shortest distance between the amplification point coordinate and the fitting plane can be determined as the marking point.

[0144] In one example, performing plane fitting based on multiple amplification point coordinates may include performing plane fitting based on N amplification point coordinates of each marker point to obtain fitting planes corresponding to each of the N marker points. Calculating the distances between the multiple amplification point coordinates and the fitting planes to obtain the positional relationship between the amplification point coordinates and the fitting planes. Among the N amplification point coordinates corresponding to each marker point, the amplification point whose positional relationship characterizes that the amplification point coordinates are the shortest from the fitting plane can be determined as the marker point.

[0145] In one example, a plurality of amplification point coordinates may be processed based on a plane fitting algorithm such as the least square method to obtain a fitting plane.

[0146] For example, the coordinates of multiple amplification points can be processed based on the following formula (11):

[0147]

[0148] (11);

[0149] Formula (11) has a closed-form solution, which can be expressed as . Among them, the fitting plane normal vector n is expressed as formula (12), and the fitting plane distance constant is expressed as formula (13).

[0150] (12);

[0151] (13);

[0152] Amplification point coordinates p k The distance D between the coordinate and the fitting plane k It can be expressed as formula (14).

[0153] (14);

[0154] in represents the magnitude of the normal vector, . Traverse the coordinates of the N amplification points corresponding to each marked point, so that The coordinates of the amplification point with the minimum value As the reference coordinate of the kth marking point, the reference coordinates corresponding to all the K marking points can be converted.

[0155] According to an embodiment of the present disclosure, calibrating a movable camera arranged on a robot based on reference coordinates and preset vehicle coordinates to obtain a target transformation matrix may include: spatially transforming the reference coordinates based on a first transformation matrix that characterizes the spatial transformation relationship between a reference camera coordinate system and a regional coordinate system to obtain reference regional coordinates of the marking point in the regional coordinate system; and determining the target transformation matrix based on the reference regional coordinates and preset vehicle coordinates of each of a plurality of marking points.

[0156] According to an embodiment of the present disclosure, the first conversion matrix Characterize the reference camera coordinate system O c1 With the regional coordinate system O d The spatial conversion relationship between them can be determined by a related calibration algorithm before executing the camera calibration method provided by the embodiment of the present disclosure.

[0157] According to an embodiment of the present disclosure, based on the first conversion matrix Process the reference coordinates of the i-th marker point , get the i-th marked point in the regional coordinate system O d The coordinates of the base area in .

[0158] According to an embodiment of the present disclosure, determining a target transformation matrix based on the reference region coordinates and preset vehicle coordinates of each of the multiple marking points may include establishing K sets of equations with solvers based on the reference region coordinates and preset vehicle coordinates of each of the K marking points. The target transformation matrix is ​​obtained by solving the set of equations.

[0159] In one example, the carrier includes a base and a carrier body, the marking point is set on the carrier body, and the robot includes a robotic arm mounted on the carrier body through the base.

[0160] Based on the reference area coordinates and preset vehicle coordinates of each of the multiple marking points, determining the target transformation matrix may include: calibrating according to the reference area coordinates and vehicle coordinates of each of the multiple marking points to obtain a first intermediate transformation matrix; fusing the first intermediate transformation matrix and the preset second intermediate transformation matrix to obtain the target transformation matrix.

[0161] According to an embodiment of the present disclosure, the first intermediate conversion matrix Representation area coordinate system With the vehicle body coordinate system The reference region coordinates and the vehicle coordinates of the multiple marking points can be processed based on the calibration algorithm to obtain the first intermediate conversion matrix.

[0162] According to an embodiment of the present disclosure, the second intermediate conversion matrix Characterizing the vehicle body coordinate system Base coordinate system with base The spatial transformation relationship between them, the target transformation matrix Characterizing the base coordinate system With regional coordinate system The spatial transformation relationship between them.

[0163] According to an embodiment of the present disclosure, fusing the first intermediate conversion matrix and the preset second intermediate conversion matrix may include: With the second intermediate transformation matrix The product between them is used to determine the target transformation matrix .

[0164] In one example, the target transformation matrix may be determined based on the following operations SA1 to SA2.

[0165] Operation SA1 determines the transformation matrix between various components in the robot based on the following steps.

[0166] Step 1: Determine the vehicle body coordinate system To the movable camera coordinate system The transformation matrix The matrix Contains 11 unknowns (internal parameter matrix and external parameter matrix).

[0167] Step 2: Determine the movable camera coordinate system Transformation matrix to the end-of-arm fixture coordinate system , the matrix Contains 6 unknown quantities (three-axis rotation parameters and three-axis translation parameters).

[0168] Step 3: Determine the coordinate system of the end fixture of the robot arm To base coordinate system The transformation matrix , the transformation matrix It can be obtained from the structure and kinematics model of the robot system. is a known quantity.

[0169] Operate SA2 to realize the coordinate system of AGV platform To the robot base coordinate system The second intermediate transformation matrix .in and All of them are parameter matrices to be determined, which can be obtained by calibrating the movable camera with a monocular camera. , based on the hand-eye calibration algorithm, it can be determined This allows us to determine the second intermediate transformation matrix The second intermediate transformation matrix The calculation may be performed in advance and stored in a storage device associated with the work area, or in a storage device associated with the robot.

[0170] Operation SA3, next determine the regional coordinate system based on the product of multiple transformation matrices determined in the following steps: To the vehicle body coordinate system The first intermediate transformation matrix between .

[0171] Step 4: Determine the regional coordinate system To the first reference camera coordinate system of the first reference camera Conversion matrix , the transformation matrix Contains 11 unknowns (internal and external parameter matrices)

[0172] Step 5: Determine the regional coordinate system To the second reference camera coordinate system of the second reference camera The transformation matrix between , the matrix contains 11 unknowns (internal parameter matrix and external parameter matrix);

[0173] Step 6: First reference camera coordinate system of the first reference camera To the second reference camera coordinate system The transformation matrix , the transformation matrix is a transformation matrix containing 6 unknown quantities (three-axis rotation and three-axis translation).

[0174] Step 7: First reference camera coordinate system To the vehicle body coordinate system The transformation matrix between , the transformation matrix Contains 11 unknowns (intrinsic parameter matrix and extrinsic parameter matrix).

[0175] Step 8: Second reference camera coordinate system To the vehicle body coordinate system The transformation matrix between , the transformation matrix Contains 11 unknowns.

[0176] It should be noted that the conversion matrix in the above steps 4 to 8 may be determined based on K preset sample marking points, and K may be an integer greater than the number of unknown quantities.

[0177] In Example 1, the coordinate system of the end fixture of the robot arm to the base coordinate system is obtained through the robot arm system structure and kinematic model. The transformation matrix between .

[0178] In Example 2, the vehicle body coordinate system is obtained by Zhang Zhengyou calibration method To the movable camera coordinate system The transformation matrix between .

[0179] In Example 3, the movable camera coordinate system is obtained by the robot hand-eye calibration method The transformation matrix between the coordinate system of the end fixture of the robot arm .

[0180] In Example 4, by Get the vehicle body coordinate system To the base coordinate system of the robot The transformation matrix between .

[0181] In Example 5, the desktop workbench coordinate system is obtained by Zhang Zhengyou calibration method To the first reference coordinate system The transformation matrix between .

[0182] In Example 6, the regional coordinate system is obtained by Zhang Zhengyou calibration method To the second reference coordinate system The transformation matrix between .

[0183] In Example 7, the first reference coordinate system is obtained by using the binocular vision calibration method. To the second reference coordinate system The transformation matrix between .

[0184] In Example 8, the first reference coordinate system is obtained by Zhang Zhengyou calibration method To the vehicle body coordinate system The transformation matrix between .

[0185] In Example 9, the second reference coordinate system is obtained by Zhang Zhengyou calibration method To the vehicle body coordinate system The transformation matrix between .

[0186] The above examples 1 to 9 can be the preparatory work for system deployment, and only one calibration is required, and no further execution is required after the system is running normally. After the preparatory work for system deployment is performed to obtain multiple transformation matrices, the camera calibration method of the embodiment of the present disclosure can be executed.

[0187] In Example 10, based on the aforementioned transformation matrix and the marker point detection algorithm, the first marker image and the second marker image captured by the first reference camera and the second reference camera can be processed to obtain the coordinates of any marker point in the first reference coordinate system. The reference coordinates under the world coordinate system are the coordinates of the marker point. The regional coordinate system is obtained by the least squares method and SVD decomposition method. To the vehicle body coordinate system The transformation matrix between ;pass To determine the base coordinate system To regional coordinate system The target transformation matrix between .

[0188] Figure 5 The following schematically shows an application scenario of a camera calibration method according to another embodiment of the present disclosure.

[0189] like Figure 5As shown, the application scenario may include a first reference camera 5111 and a second reference camera 5112. The application scenario also includes: a computing unit 412, a workbench 413, and a movable robot. The movable robot includes a carrier having a carrier body 4211 and a base 4212, a mechanical arm 423 mounted on the base 4212, and a movable camera 422 mounted on the mechanical arm 423. The first reference camera 5111 and the second reference camera 5112 can be used as fixed-viewing angle binocular cameras fixedly mounted on the workbench 413.

[0190] Reference camera coordinate system of the first reference camera 5111 The reference camera coordinate system of the second reference camera 5112 includes the coordinate axes x1, y1 and z1. Includes coordinate axes x2, y2 and z2.

[0191] A plurality of marking points are set on the carrier body platform of the carrier body 4211, and the reference camera 411 can acquire images of the plurality of marking points on the carrier body platform to obtain a marking image, which can be transmitted to the calculation unit 412 for processing.

[0192] In one example, the computing unit 412 may obtain the preset vehicle coordinates of each of the plurality of marking points from the storage device of the robot to execute the camera calibration method provided in the embodiment of the present disclosure to obtain the target transformation matrix.

[0193] In one example, the computing unit 412 may transmit the marking image to the edge processor of the robot, and the edge processor of the robot may calibrate the movable camera 422 according to the preset vehicle coordinates of each of the plurality of marking points to obtain a target transformation matrix.

[0194] In one example, the camera calibration method provided by the embodiment of the present disclosure can be performed based on the computing unit 412 and the edge processor of the robot. For example, the computing unit 412 can detect the marked image to obtain the respective reference coordinates of the multiple marked points, and the computing unit 412 transmits the respective reference coordinates of the multiple marked points to the storage device of the robot. The edge processor of the robot calls the respective reference coordinates of the multiple marked points and the preset vehicle coordinates from the storage device to calibrate the movable camera to obtain the target transformation matrix.

[0195] According to the camera calibration method provided in the embodiment of the present disclosure, the movable camera is calibrated by using a binocular camera composed of a first reference camera and a second reference camera. This allows the robot to achieve real-time calibration during the movement of the vehicle, so as to quickly determine the target transformation matrix of the robot at each moving position. In this way, the robot can be continuously and accurately controlled based on the target transformation matrix to perform operations on the working area during the movement of the robot, thereby improving the robot's operating accuracy and operation execution efficiency.

[0196] Figure 6 The flowchart of the robot control method according to the embodiment of the present disclosure is schematically shown.

[0197] like Figure 6 As shown, the robot control method of this embodiment includes operation S610.

[0198] In operation S610, the robot is controlled to perform an operation based on the target conversion matrix and the target image captured by the movable camera.

[0199] According to an embodiment of the present disclosure, a robot includes a carrier and a movable camera. The movable camera can be set on a mechanical arm or a carrier of the robot. The carrier is provided with a marking point for executing the camera calibration method provided in the embodiment of the present disclosure.

[0200] According to an embodiment of the present disclosure, the target transformation matrix is ​​determined based on the camera calibration method provided according to an embodiment of the present disclosure.

[0201] According to an embodiment of the present disclosure, controlling a robot to perform an operation based on a target transformation matrix and a target image captured by a movable camera may include: performing target object detection based on the target image to obtain the initial object coordinates of the target object in a vehicle coordinate system; performing spatial transformation on the initial object coordinates based on the target transformation matrix to obtain the target object coordinates of the target object in a regional coordinate system; and controlling the robot to perform the operation based on the target object coordinates.

[0202] According to an embodiment of the present disclosure, the target image may be an image captured by a movable camera of a working area, and the target image may record target objects such as assembly parts and items to be transported.

[0203] According to an embodiment of the present disclosure, target object detection based on a target image may include processing the target image based on a target detection algorithm to obtain the image coordinates of the target object in the image coordinate system of the movable camera. The image coordinates of the target object in the image coordinate system of the movable camera are processed by using the robot motion equation and transformation matrix that have been determined by the robot, so as to transform the image coordinates of the target object and obtain the initial object coordinates of the target object in the vehicle coordinate system.

[0204] According to an embodiment of the present disclosure, the carrier may include a carrier body and a base, and the base is used to install a mechanical arm of the robot. The initial object coordinates may include coordinates in a carrier body coordinate system, or the initial object coordinates may also be coordinates in a base coordinate system.

[0205] According to the embodiments of the present disclosure, the target transformation matrix represents the spatial transformation relationship between the carrier coordinate system and the regional coordinate system of the working area where the robot performs the operation. The initial object coordinates are spatially transformed based on the target transformation matrix to obtain the target object coordinates in the regional coordinate system. Therefore, the robot can be more accurately controlled to perform any type of operation such as installation operation and handling operation on the target object in the working area according to the target object coordinates.

[0206] It should be noted that the technical terms involved in the robot control method provided in the embodiment of the present disclosure are the same as the technical terms involved in the camera calibration method provided in the embodiment of the present disclosure, and the embodiments of the present disclosure will not be repeated here.

[0207] Based on the above camera calibration method, the present disclosure also provides a camera calibration device. Figure 7 The device is described in detail.

[0208] Figure 7 The structural block diagram of a camera calibration device according to an embodiment of the present disclosure is schematically shown.

[0209] like Figure 7 As shown, the camera calibration device 700 of this embodiment includes an acquisition module 710 , a detection module 720 and a calibration module 730 .

[0210] The acquisition module 710 is used to acquire a marked image. When the reference camera is stationary relative to a working area where a robot equipped with a carrier performs an operation, the marked image is acquired by capturing the marked points set on the carrier.

[0211] The detection module 720 is used to detect the marked image and obtain the reference coordinates of the marked point in the reference camera coordinate system of the reference camera.

[0212] The calibration module 730 is used to calibrate the movable camera set on the robot according to the reference coordinates and the preset vehicle coordinates to obtain a target transformation matrix, wherein the preset vehicle coordinates represent the position of the marking point in the vehicle coordinate system of the vehicle, and the target transformation matrix represents the spatial transformation relationship between the vehicle coordinate system and the regional coordinate system of the working area, and the target transformation matrix is ​​used to control the robot to perform operations.

[0213] According to an embodiment of the present disclosure, the calibration module 730 includes: a first obtaining submodule and a first determining submodule.

[0214] The first acquisition submodule is used to perform spatial transformation on the reference coordinates based on a first transformation matrix that represents the spatial transformation relationship between the reference camera coordinate system and the regional coordinate system to obtain the reference regional coordinates of the marking point in the regional coordinate system.

[0215] The first determination submodule is used to determine the target transformation matrix based on the reference area coordinates and the preset vehicle coordinates of each of the plurality of marking points.

[0216] According to an embodiment of the present disclosure, the carrier includes a base and a carrier body, the marking point is set on the carrier body, and the robot includes a robotic arm installed on the carrier body through the base.

[0217] According to an embodiment of the present disclosure, the first determining submodule includes: a first obtaining unit and a fusion unit.

[0218] The first obtaining unit is used to calibrate according to the reference area coordinates and the vehicle coordinates of each of the multiple marking points to obtain a first intermediate conversion matrix, and the first intermediate conversion matrix represents the spatial conversion relationship between the area coordinate system and the vehicle body coordinate system.

[0219] A fusion unit is used to fuse the first intermediate transformation matrix and the preset second intermediate transformation matrix to obtain a target transformation matrix, wherein the second intermediate transformation matrix represents the spatial transformation relationship between the vehicle body coordinate system and the base coordinate system of the base, and the target transformation matrix represents the spatial transformation relationship between the base coordinate system and the regional coordinate system.

[0220] According to an embodiment of the present disclosure, the carrier includes a base and a carrier body, the marking point is set on the carrier body, and the robot includes a robotic arm installed on the carrier body through the base.

[0221] According to an embodiment of the present disclosure, the calibration module 730 includes: a calibration submodule and a second acquisition submodule.

[0222] The calibration submodule is used to perform calibration based on the reference coordinates and the preset vehicle coordinates to obtain a second transformation matrix that characterizes the spatial transformation relationship between the reference camera coordinate system and the vehicle coordinate system.

[0223] The second acquisition submodule is used to obtain a target transformation matrix based on a second transformation matrix, a preset second intermediate transformation matrix and the first transformation matrix, wherein the first transformation matrix represents the spatial transformation relationship between the reference camera coordinate system and the regional coordinate system, and the second intermediate transformation matrix represents the spatial transformation relationship between the vehicle body coordinate system of the vehicle body and the base coordinate system of the base.

[0224] According to an embodiment of the present disclosure, the marker image includes a first marker image and a second marker image acquired by a first reference camera and a second reference camera, respectively.

[0225] According to an embodiment of the present disclosure, the detection module 720 includes: a detection submodule and a third acquisition submodule.

[0226] The detection submodule is used to detect the first marked image and the second marked image to obtain the first image point coordinates of the marked point in the first image coordinate system and the second image point coordinates of the marked point in the second image coordinate system.

[0227] The third acquisition submodule is used to perform spatial transformation on the first image point coordinates and the second image point coordinates to obtain reference coordinates, wherein the reference coordinates are related to the first reference camera coordinate system of the first reference camera, or the reference coordinates are related to the second reference coordinate system of the second reference camera.

[0228] According to an embodiment of the present disclosure, the third obtaining submodule includes: a second obtaining unit, a third obtaining unit and a fourth obtaining unit.

[0229] The second obtaining unit is used to process the first image point coordinates and the second image point coordinates by using a coordinate conversion function determined based on reference camera parameters of the first reference camera and the second reference camera to obtain initial reference coordinates of the marking point.

[0230] The third obtaining unit is used to process the first image point coordinates and the second image point coordinates by using a differential function corresponding to the coordinate conversion function to obtain a reference coordinate error of the initial reference coordinates.

[0231] The fourth obtaining unit is used to update the initial reference coordinates based on the reference coordinate error to obtain the reference coordinates.

[0232] According to an embodiment of the present disclosure, the third obtaining submodule includes: a fifth obtaining unit, a sixth obtaining unit, an amplification point coordinate determining unit, and a reference coordinate determining unit.

[0233] A fifth obtaining unit is used to process the first image point coordinates and the second image point coordinates by using an error distribution function to obtain an amplified image point coordinate error, wherein the error distribution function indicates that the coordinate error distribution of the first image point coordinates or the second image point coordinates in the image coordinate system satisfies a preset distribution morphology condition;

[0234] a sixth obtaining unit, configured to process the coordinate errors of the augmented image points by using a differential function corresponding to a coordinate conversion function to obtain coordinate errors of multiple augmented points of the marking point in a reference camera coordinate system, wherein the coordinate conversion function is determined based on reference camera parameters of the first reference camera and the second reference camera;

[0235] an amplification point coordinate determining unit, configured to determine a plurality of amplification point coordinates of a marking point based on a plurality of amplification point coordinate errors; and

[0236] The reference coordinate determining unit is used to determine the reference coordinates of the marking point from the coordinates of the multiple amplification points.

[0237] According to an embodiment of the present disclosure, the reference coordinate determining unit includes: a fitting plane obtaining subunit and a reference coordinate determining subunit.

[0238] The fitting plane obtaining subunit is used to perform plane fitting based on the coordinates of multiple amplification points to obtain a fitting plane.

[0239] The reference coordinate determination subunit is used to determine the reference coordinate of the marking point from the multiple amplification point coordinates of the marking point based on the positional relationship between the amplification point coordinates and the fitting plane.

[0240] Based on the above robot control method, the present disclosure also provides a robot control device. Figure 8 The device is described in detail.

[0241] Figure 8 The structural block diagram of the robot control device according to the embodiment of the present disclosure is schematically shown.

[0242] like Figure 8 As shown, the robot control device 800 of this embodiment includes a control module 810 .

[0243] The control module 810 is used to control the robot to perform operations based on the target conversion matrix and the target image captured by the movable camera, wherein the target conversion matrix is ​​determined based on the camera calibration method provided in the embodiment of the present disclosure.

[0244] According to an embodiment of the present disclosure, the control module 810 includes: an initial object coordinate obtaining submodule, a target object coordinate obtaining submodule and a control submodule.

[0245] The initial object coordinate acquisition submodule is used to detect the target object based on the target image and obtain the initial object coordinates of the target object in the vehicle coordinate system.

[0246] The target object coordinate acquisition submodule is used to perform spatial transformation on the initial object coordinates based on the target transformation matrix to obtain the target object coordinates of the target object in the regional coordinate system.

[0247] The control submodule is used to control the robot to perform operations based on the target object coordinates.

[0248] According to an embodiment of the present disclosure, any multiple modules of the acquisition module 710, the detection module 720 and the calibration module 730 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the acquisition module 710, the detection module 720 and the calibration module 730 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, at least one of the acquisition module 710, the detection module 720 and the calibration module 730 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be performed.

[0249] Fig. 9 A block diagram of an electronic device suitable for implementing a camera calibration method and a robot control method according to an embodiment of the present disclosure is schematically shown.

[0250] like Fig. 9 As shown, the electronic device 900 according to an embodiment of the present disclosure includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage part 908 to a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include an onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0251] In RAM 903, various programs and data required for the operation of electronic device 900 are stored. Processor 901, ROM 902 and RAM 903 are connected to each other via bus 904. Processor 901 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 902 and / or RAM 903. It should be noted that the program can also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0252] According to an embodiment of the present disclosure, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to the bus 904. The electronic device 900 may further include one or more of the following components connected to the input / output (I / O) interface 905: an input portion 906 including a keyboard, a mouse, etc.; an output portion 907 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 908 including a hard disk, etc.; and a communication portion 909 including a network interface card such as a LAN card, a modem, etc. The communication portion 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the input / output (I / O) interface 905 as needed. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 910 as needed, so that a computer program read therefrom is installed into the storage portion 908 as needed.

[0253] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0254] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 902 and / or RAM 903 described above and / or one or more memories other than ROM 902 and RAM 903.

[0255] The embodiments of the present disclosure also include a computer program product, which includes a computer program, and the computer program includes a program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the camera calibration method and robot control method provided by the embodiments of the present disclosure.

[0256] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 901. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0257] In one embodiment, the computer program may be based on a tangible storage medium such as an optical storage device, a magnetic storage device, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of a signal on a network medium, and downloaded and installed through the communication part 909, and / or installed from a removable medium 911. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0258] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 909, and / or installed from the removable medium 911. When the computer program is executed by the processor 901, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0259] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0260] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0261] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations or combinations are not explicitly described in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0262] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A camera calibration method, characterized in that: The method comprises: Acquire a marking image, when the reference camera is stationary relative to a working area where a robot equipped with a carrier performs an operation, collect the marking point set on the carrier to obtain the marking image; Detecting the marked image to obtain reference coordinates of the marked point in a reference camera coordinate system of the reference camera; The movable camera arranged on the robot is calibrated according to the reference coordinates and the preset vehicle coordinates to obtain a target transformation matrix, wherein the preset vehicle coordinates represent the position of the marking point in the vehicle coordinate system of the vehicle, and the target transformation matrix represents the spatial transformation relationship between the vehicle coordinate system and the regional coordinate system of the working area, and the target transformation matrix is ​​used to control the robot to perform operations.

2. The method according to claim 1, characterized in that The movable camera disposed on the robot is calibrated according to the reference coordinates and the preset vehicle coordinates to obtain a target transformation matrix including: Performing spatial transformation on the reference coordinates based on a first transformation matrix representing a spatial transformation relationship between the reference camera coordinate system and the regional coordinate system to obtain reference regional coordinates of the marking point in the regional coordinate system; and The target transformation matrix is ​​determined based on the reference area coordinates and the preset vehicle coordinates of each of the plurality of marking points.

3. The method according to claim 2, characterized in that The carrier includes a base and a carrier body, the marking point is arranged on the carrier body, and the robot includes a mechanical arm mounted on the carrier body through the base; Wherein, based on the reference area coordinates and the preset vehicle coordinates of each of the plurality of marking points, determining the target transformation matrix comprises: Calibrate the reference area coordinates and the vehicle coordinates of each of the plurality of marking points to obtain a first intermediate conversion matrix, wherein the first intermediate conversion matrix represents a spatial conversion relationship between the area coordinate system and the vehicle body coordinate system; The target transformation matrix is ​​obtained by fusing the first intermediate transformation matrix and the preset second intermediate transformation matrix, wherein the second intermediate transformation matrix represents the spatial transformation relationship between the vehicle body coordinate system and the base coordinate system of the base, and the target transformation matrix represents the spatial transformation relationship between the base coordinate system and the regional coordinate system.

4. The method according to claim 1, characterized in that: The marked image includes a first marked image and a second marked image captured by a first reference camera and a second reference camera respectively; The step of detecting the marked image to obtain the reference coordinates of the marked point in the reference camera coordinate system of the reference camera includes: Detecting the first marked image and the second marked image to obtain first image point coordinates of the marked point in a first image coordinate system and second image point coordinates of the marked point in a second image coordinate system; and The first image point coordinates and the second image point coordinates are spatially transformed to obtain the reference coordinates, wherein the reference coordinates are related to a first reference camera coordinate system of the first reference camera, or the reference coordinates are related to a second reference coordinate system of the second reference camera.

5. The method according to claim 4, characterized in that Performing spatial transformation on the first image point coordinates and the second image point coordinates to obtain the reference coordinates includes: Processing the first image point coordinates and the second image point coordinates by using a coordinate conversion function determined based on reference camera parameters of the first reference camera and the second reference camera to obtain initial reference coordinates of the marking point; Processing the first image point coordinates and the second image point coordinates using a differential function corresponding to the coordinate conversion function to obtain a reference coordinate error of the initial reference coordinates; and The initial reference coordinates are updated based on the reference coordinate errors to obtain the reference coordinates.

6. The method according to claim 4, characterized in that Performing spatial transformation on the first image point coordinates and the second image point coordinates to obtain the reference coordinates includes: Processing the first image point coordinates and the second image point coordinates using an error distribution function to obtain an amplified image point coordinate error, wherein the error distribution function indicates that a coordinate error distribution of the first image point coordinates or the second image point coordinates in an image coordinate system satisfies a preset distribution morphology condition; Processing the coordinate errors of the augmented image points by using a differential function corresponding to a coordinate conversion function to obtain a plurality of augmented point coordinate errors of the marking point in the reference camera coordinate system, wherein the coordinate conversion function is determined based on reference camera parameters of the first reference camera and the second reference camera; Determining a plurality of amplification point coordinates of the marking point based on a plurality of amplification point coordinate errors; and The reference coordinates of the marking point are determined from the plurality of amplification point coordinates.

7. A robot control method, characterized in that: The robot comprises a carrier and a movable camera, a marking point is set on the carrier, and the method comprises: The robot is controlled to perform an operation based on the target conversion matrix and the target image captured by the movable camera, wherein the target conversion matrix is ​​determined based on the method according to any one of claims 1 to 6.

8. A camera calibration device, characterized in that: The device comprises: An acquisition module, used for acquiring a marking image, wherein when a reference camera is stationary relative to a working area where a robot equipped with a carrier performs an operation, the marking image is acquired by acquiring a marking point set on the carrier; A detection module, used to detect the marked image and obtain the reference coordinates of the marked point in the reference camera coordinate system of the reference camera; A calibration module is used to calibrate the movable camera set on the robot according to the reference coordinates and the preset vehicle coordinates to obtain a target transformation matrix, wherein the preset vehicle coordinates represent the position of the marking point in the vehicle coordinate system of the vehicle, and the target transformation matrix represents the spatial transformation relationship between the vehicle coordinate system and the regional coordinate system of the working area, and the target transformation matrix is ​​used to control the robot to perform operations.

9. A robot control device, characterized in that: The robot comprises a carrier and a movable camera, a marking point is set on the carrier, and the device comprises: A control module, used to control the robot to perform operations based on the target transformation matrix and the target image captured by the movable camera, wherein the target transformation matrix is ​​determined based on the method described in any one of claims 1 to 6.

10. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 7.

Images