Robot camera calibration method, device, computer equipment and storage medium

Abstract

The present application relates to a camera calibration method, device, computer equipment, storage medium and computer program product for a robot. The method includes: in response to a calibration instruction, controlling the robot to enter and exit a base station at least once, and when the robot enters and exits the base station, controlling the camera on the robot to shoot the identification of the base station, and obtaining a base station image including the identification taken each time entering and exiting the base station; identifying the coordinates of the identification in the base station image, and converting the coordinates of the identification in the base station image to obtain the actual position information of the current camera; when it is determined that the calibration conditions are met according to the actual position information of the current camera, the compensation information of the camera is obtained based on the difference between the actual position information of the current camera and the preset position information; and calibrating the camera according to the compensation information of the camera. The method uses image recognition to calculate the compensation information, and the calibration can be completed based on the original hardware design, reducing the calibration cost.

Description

Technical Field

[0001] The present application relates to the technical field of intelligent robots, and in particular to a robot camera calibration method, device, computer equipment, storage medium and computer program product. Background Art

[0002] As people's living standards continue to improve, the utilization rate of intelligent robots in life is gradually increasing.

[0003] Intelligent robots use visual cameras to identify obstacles, and the recognition accuracy of visual cameras depends on the installation accuracy of the cameras. The camera position is calibrated at the factory. With the use of intelligent robots and due to bumps and other reasons, the camera position is no longer the factory calibrated position.

[0004] There is an urgent need for a robot camera calibration method. Summary of the invention

[0005] Based on this, it is necessary to provide a robot camera calibration method, device, computer equipment, computer-readable storage medium and computer program product that can improve recognition accuracy in response to the above technical problems.

[0006] In a first aspect, the present application provides a robot camera calibration method, the method comprising:

[0007] In response to the calibration instruction, the robot is controlled to enter and exit the base station at least once, and when the robot enters and exits the base station, the camera on the robot is controlled to shoot the identification of the base station, and the base station image including the identification is obtained each time the robot enters and exits the base station;

[0008] Identify the coordinates of the marker in the base station image, and transform the coordinates of the marker in the base station image to obtain actual position information of the current camera;

[0009] When it is determined that the calibration condition is met according to the actual position information of the current camera, compensation information of the camera is obtained based on the difference between the actual position information of the current camera and the preset position information;

[0010] The camera is calibrated according to the compensation information of the camera.

[0011] In one embodiment, when it is determined that the calibration condition is met according to the actual position information of the current camera, compensation information of the camera is obtained based on the difference between the actual position information of the current camera and the preset position information, including:

[0012] Calculating the position error of the current camera according to the difference between the actual position information of the current camera and the preset position information;

[0013] If the position errors of the current camera calculated based on at least three of the base station images are all greater than a preset value, it is determined that the calibration condition is met;

[0014] Calculate compensation information corresponding to each current camera according to the position error of the current camera in each base station image;

[0015] An average value and / or variance is taken for each compensation information to obtain final compensation information of the camera.

[0016] In one embodiment, the method further comprises:

[0017] When the number of base station images collected reaches a set value, it is determined that the calibration condition is met.

[0018] In one embodiment, when the number of base station images collected reaches a set value, taking an average value and / or a variance of each compensation information to obtain the final compensation information of the camera includes:

[0019] Sorting all the compensation information in descending order according to the absolute values ​​of the compensation information;

[0020] Remove the first N pieces of compensation information that are sorted in front;

[0021] An average value and / or variance is taken for the retained compensation information to obtain final compensation information of the camera.

[0022] In one embodiment, the actual position information includes actual height information and actual angle information; the preset position information includes preset height information and preset angle information; the position error includes height error and angle error;

[0023] The calculating the position error of the camera according to the difference between the preset position information of the camera and the actual position information includes:

[0024] The height error of the camera is calculated based on the difference between the actual height information of the current camera and the preset height information, and the angle error of the camera is calculated based on the difference between the actual angle information of the current camera and the preset angle information;

[0025] If the position errors of the camera calculated based on at least three of the base station images are greater than preset values, it is determined that the calibration conditions are met, including: if the height errors and / or angle errors of the camera calculated based on at least three of the base station images are greater than preset values, it is determined that the calibration conditions are met.

[0026] In one embodiment, a method of calculating compensation information corresponding to the current camera according to a position error of the current camera in the base station image includes:

[0027] According to the preset horizontal position information of the identifier and the camera, the actual height information of the current camera, the actual angle information of the current camera, the preset height information of the camera and the preset angle information of the camera, compensation information corresponding to the current camera is obtained by trigonometric calculation.

[0028] In a second aspect, the present application provides a camera calibration device for a robot, the device comprising:

[0029] a shooting module, configured to control the robot to enter and exit the base station at least once in response to a calibration instruction, and to control a camera on the robot to shoot an identification of the base station when the robot enters and exits the base station, so as to obtain an image of the base station including the identification taken each time the robot enters and exits the base station;

[0030] an identification module, used for identifying the coordinates of the marker in the base station image, converting the coordinates of the marker in the base station image, and obtaining the actual position information of the current camera;

[0031] A compensation module, configured to obtain compensation information of the camera based on a difference between the actual position information of the current camera and preset position information when it is determined that a calibration condition is met according to the actual position information of the current camera;

[0032] A calibration module is used to calibrate the camera according to the compensation information of the camera.

[0033] In a third aspect, the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the following method steps when executing the computer program: in response to a calibration instruction, controlling the robot to enter and exit a base station at least once, and when the robot enters and exits the base station, controlling a camera on the robot to photograph an identification of the base station, and obtaining an image of the base station including the identification taken each time the robot enters and exits the base station;

[0034] Identify the coordinates of the marker in the base station image, and transform the coordinates of the marker in the base station image to obtain actual position information of the current camera;

[0035] When it is determined that the calibration condition is met according to the actual position information of the current camera, compensation information of the camera is obtained based on the difference between the actual position information of the current camera and the preset position information;

[0036] The camera is calibrated according to the compensation information of the camera.

[0037] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following method steps are implemented: in response to a calibration instruction, controlling the robot to enter and exit a base station at least once, and when the robot enters and exits the base station, controlling a camera on the robot to photograph an identification of the base station, and obtaining an image of the base station including the identification taken each time the robot enters and exits the base station;

[0038] Identify the coordinates of the marker in the base station image, and transform the coordinates of the marker in the base station image to obtain actual position information of the current camera;

[0039] When it is determined that the calibration condition is met according to the actual position information of the current camera, compensation information of the camera is obtained based on the difference between the actual position information of the current camera and the preset position information;

[0040] The camera is calibrated according to the compensation information of the camera.

[0041] In a fifth aspect, the present application provides a computer program product, including a computer program, which implements the following method steps when executed by a processor: in response to a calibration instruction, controlling the robot to enter and exit a base station at least once, and when the robot enters and exits the base station, controlling a camera on the robot to photograph an identification of the base station, and obtaining an image of the base station including the identification taken each time the robot enters and exits the base station;

[0042] Identify the coordinates of the marker in the base station image, and transform the coordinates of the marker in the base station image to obtain actual position information of the current camera;

[0043] When it is determined that the calibration condition is met according to the actual position information of the current camera, compensation information of the camera is obtained based on the difference between the actual position information of the current camera and the preset position information;

[0044] The camera is calibrated according to the compensation information of the camera.

[0045] The camera calibration method, device, computer equipment, storage medium and computer program product of the robot described above control the robot to enter and exit the base station multiple times to shoot multiple groups of base station images containing logos, identify the logos in the base station images, and perform coordinate conversion on the coordinates of the logos in the base station images to obtain the actual position information of the current camera. When the actual position of the current camera determines that the calibration conditions are met, the compensation information of the camera is obtained based on the difference between the preset position information and the actual position information of the camera, thereby realizing the position calibration of the camera. On the one hand, since the position of the camera can be calibrated, the recognition accuracy of the robot is guaranteed. On the other hand, the calibration method uses image recognition to calculate the compensation information, and the camera calibration can be completed based on the original hardware design, without replacing the camera, thereby reducing the calibration cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A diagram showing an application environment of a robot camera calibration method in one embodiment;

[0047] Figure 2 is a schematic flow chart of a camera calibration method for a robot in one embodiment;

[0048] Figure 3 is a schematic diagram of a marker on a base station in one embodiment;

[0049] Figure 4 is a flowchart of a process for determining compensation information of a camera in one embodiment;

[0050] Figure 5 is a flowchart of a process for determining compensation information of a camera in another embodiment;

[0051] Figure 6 A schematic diagram of the relationship between the camera and the position of the display in one embodiment;

[0052] Figure 7 A schematic diagram of a flow chart of a method for a user to select automatic calibration of a camera in one embodiment;

[0053] Figure 8 is a structural block diagram of a camera calibration device for a robot in one embodiment;

[0054] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] Visual ranging depends on the installation accuracy of the camera. When the accuracy is low, the ranging error will increase, which will affect the obstacle avoidance function. The current method is to ensure the tolerance and installation accuracy of the parts during production, and then calibrate the camera to eliminate the ranging error as much as possible. However, after the product is shipped, there are often problems such as changes in the position of the camera due to bumps, structural wear and other reasons during use. There is no way to deal with it, resulting in an increase in the ranging error and a significant reduction in the obstacle avoidance effect. At present, there is no good solution for the ranging error of the product's camera. The user can only hope that the structure will be kept as stable as possible before the product is shipped. If the product has a ranging error, it can only be returned to the factory for after-sales processing.

[0057] In view of this, the robot camera calibration method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The server 104 communicates with the robot 106. The terminal 102 and the robot 106 can also communicate directly, and can communicate through wireless transmission methods such as Bluetooth, NFC, ANT+, etc.

[0058] The robot 106 responds to the calibration instruction and controls the robot 106 to enter and exit the base station at least once. When the robot 106 enters and exits the base station, the robot 106 controls the camera on the robot 106 to shoot the logo of the base station, and obtains the base station image including the logo captured each time entering and exiting the base station; the robot 106 identifies the coordinates of the logo in the base station image, transforms the coordinates of the logo in the base station image, and obtains the actual position information of the current camera; when the robot 106 determines that the calibration conditions are met according to the actual position information of the current camera, the compensation information of the camera is obtained based on the difference between the actual position information of the current camera and the preset position information; the server 104 calibrates the camera according to the compensation information of the camera.

[0059] Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented as an independent server or a server cluster composed of multiple servers. The robot 106 can be an intelligent sweeping robot, an intelligent lawn mowing robot, or other intelligent robots with certain image acquisition and recognition capabilities.

[0060] In one embodiment, Figure 2 As shown, a robot camera calibration method is provided, and the method is applied to Figure 1 The robot in the example is used to illustrate the following steps:

[0061] S202, in response to the calibration instruction, control the robot to enter and exit the base station at least once, and when the robot enters and exits the base station, control the camera on the robot to shoot the base station logo, and obtain the base station image including the logo taken each time the robot enters and exits the base station.

[0062] Among them, the calibration instruction is a type of robot control instruction. The control instruction is used to control the robot to move according to the preset teaching trajectory and execute the corresponding program, such as controlling the robot to perform shooting actions, controlling the robot to perform obstacle avoidance actions, controlling the robot to perform recharging actions, and controlling the robot to perform camera calibration actions. Specifically, the calibration instruction can be issued by the user through the terminal, or a control button can be set on the robot, and the user operates the control button to issue the calibration instruction.

[0063] It should be noted that the calibration instruction can also be issued by the robot's own judgment. For example, if the robot detects through the collision sensor that the number of collisions of the robot is greater than the preset number within a certain period of time, the robot will automatically issue a calibration instruction. The preset number can be flexibly selected according to the changes in the robot model and the robot's usage scenario, and is not limited here.

[0064] The base station can be a fixed place for charging, storage, calibration, etc. for the robot. Specifically, the base station can be a robot recharging base station, a temporary storage point for the robot, or a station with identification information on the robot's route. For a single robot, the number of base stations can be one or more, depending on the complexity of the robot's work, the robot's working time, and the robot's endurance.

[0065] Among them, the camera is a camera installed at the recognition end of the robot, and the camera can be used to shoot the surrounding environment of the robot.

[0066] The camera captures the surrounding environment at a certain refresh rate to form a continuous video stream. Ordinary cameras capture images at a rate of 30 images per second. In addition to capturing color images, depth cameras can also read the distance of each pixel from the camera, and the corresponding material cost is also higher.

[0067] Among them, cameras can be divided into monocular cameras, binocular cameras and depth cameras according to their working mode.

[0068] Monocular cameras have a simple structure and low material cost. Essentially, they shoot scenes and leave a projection of the scene on the camera's imaging plane, reflecting the three-dimensional world in a two-dimensional form.

[0069] A binocular camera is generally composed of two monocular cameras. The distance between the two monocular cameras is a fixed value, which is also called the baseline. The baseline is used to estimate the spatial position of each pixel. The larger the baseline distance, the farther the distance that can be measured. The binocular camera obtains the depth information of the image through calculation. After being accelerated by GPU and FPGA devices, the depth information of the image can be calculated in real time.

[0070] Depth cameras, also known as RGB-D cameras, use the principles of infrared structured light or TOF sensors to actively emit infrared structured light to objects and receive the returned light, and then measure the distance of objects from the camera based on the principle of triangulation. Different types of cameras are used according to the accuracy of the measurement. TOF cameras can measure distance at the centimeter level, RGB binocular cameras can measure distance at the millimeter level, and infrared structured light cameras can measure distance at close range with an accuracy of 0.01mm to 1mm.

[0071] Specifically, the camera installed at the robot recognition end may be a monocular camera. By photographing the base station logo through the monocular camera, an image of the base station including the logo can be obtained each time the robot enters and exits the base station.

[0072] The logo refers to an identification object of a specific shape that has been set when the robot leaves the factory. Specifically, the logo can be a two-dimensional picture, such as Figure 3 A schematic diagram of the identification on the base station is shown, where a identification includes several identification objects of specific shapes. The identification object can be a rectangle or polygon whose side length is parallel to the boundary of the two-dimensional image. The color of the two-dimensional image in the identification object can be selected according to the actual test effect, for example, white.

[0073] The position of the marker may be set at a position that is a preset distance from the limiting position of the base station. For example, the marker may be set at a position 10 cm away from the limiting position of the base station.

[0074] It should be noted that if the robot's camera is offset, the actual position of the camera at this time can be determined by identifying a marker of a specific shape, thereby achieving the purpose of calibration.

[0075] The moment of shooting may be when the robot reaches the limit position of the base station. The limit position is used to fix the robot so that the relative position of the robot and the marker remains fixed. The limit position may be a position inside the base station or outside the base station.

[0076] Specifically, in response to the calibration instruction, the robot immediately enters and exits the base station on its own, simulating the robot's normal movements of returning to the base station to charge and leaving the base station to perform specific work. During this action, the camera on the robot is controlled to capture the base station logo, and an image of the base station including the logo is obtained each time the robot enters and exits the base station.

[0077] Among them, when the robot is at the limit position of the base station, the base station logo can be photographed to obtain the base station image when the robot returns from outside the base station or when the robot goes from inside the base station to outside the base station.

[0078] Specifically, in some cases, when the robot enters and exits the base station once, the robot is at the limit position of the base station. In some cases, due to errors in the robot's recharging action, the robot does not accurately reach the limit position, and the robot needs to be controlled to enter and exit the base station at least once to make the robot accurately reach the limit position of the base station. The more times the robot enters and exits the base station, the more base station images the robot takes, and the higher the recognition accuracy of the base station logo.

[0079] It should be noted that after the robot finishes its work, it often needs to return to the base station for charging. That is, recharging can be done by photographing the base station logo when the robot returns to the base station or leaves the base station, and saving the base station image when the robot enters and leaves the base station to provide a basis for subsequent identification and calibration.

[0080] S204, identifying the coordinates marked in the base station image in the base station image, and transforming the coordinates marked in the base station image to obtain actual position information of the current camera.

[0081] The coordinates of the marker in the base station image may be pixel coordinates, which are obtained by photographing the marker with a camera on the robot. In the technical field of image processing, the pixel coordinates of the marker in the pixel coordinate system may be converted into coordinates in different coordinate systems by using the conversion relationship between coordinate systems, such as coordinates in the image coordinate system, coordinates in the camera coordinate system, and world coordinates in the world coordinate system.

[0082] Specifically, the pixel coordinates of the marker A in the pixel coordinate system uv are (u, v), the pixel coordinates (u, v) can be converted to the image coordinates (x, y) in the image coordinate system O-xy, and the image coordinates (x, y) can be converted to the camera coordinate system O c -X c Y c Z c The camera coordinates (X c ,Y c ,Z c ), identifies the camera coordinates (X c ,Y c ,Z c ) can be transformed into the world coordinate system O w -X w Y w Z w The world coordinates (X w ,Y w ,Z w ).

[0083] Among them, the origin of the camera coordinate system is O c It can be the optical center of the camera, the origin of the world coordinate system O w It can be the optical center of the camera when the camera is in a preset position.

[0084] It should be noted that the world coordinates are real-world coordinates, which can be measured manually. Pixel coordinates, image coordinates, and camera coordinates are all camera internal coordinate types, which cannot be obtained through manual measurement and are virtual.

[0085] Specifically, according to the marked world coordinates (X w ,Y w ,Z w ) can be inferred to get the world coordinates (X wc ,Y wc ,Z wc ).

[0086] S206, when it is determined that the calibration condition is met according to the actual position information of the current camera, compensation information of the camera is obtained based on the difference between the actual position information of the current camera and the preset position information.

[0087] The preset position information of the camera can be the world coordinates (X w1 ,Y w1 ,Z w1 ).

[0088] Specifically, calculate the world coordinates (X w1 ,Yw1 ,Z w1 ) and the world coordinates (X wc ,Y wc ,Z wc ), the distance between the calculated world coordinates exceeds the preset threshold, which is considered to meet the calibration condition. The preset threshold is flexibly selected according to the actual situation and is not limited here.

[0089] The compensation information of the camera may be the difference between the preset position information and the actual position information of the camera.

[0090] Specifically, by calculating the world coordinates (X w1 ,Y w1 ,Z w1 ) and the world coordinates (X wc ,Y wc ,Z wc ), the position information difference is obtained, and according to the position information difference, the position difference of the coordinate is reversed back to the camera coordinate system according to the coordinate system conversion formula. c -X c Y c Z c The virtual camera coordinates (X c ,Y c ,Z c ), the virtual image coordinates (x, y) in the image coordinate system O-xy and the virtual pixel coordinates in the pixel coordinate system uv are (u, v).

[0091] S208: Calibrate the camera according to the compensation information of the camera.

[0092] Specifically, according to the virtual camera coordinates (X c ,Y c ,Z c ), virtual image coordinates (x, y) and virtual pixel coordinates (u, v) are used to calibrate the camera.

[0093] In the camera calibration method of the robot described above, the robot is controlled to enter and exit the base station multiple times to capture multiple sets of base station images containing logos, the logos in the base station images are identified, and the coordinates of the logos in the base station images are converted to obtain the actual position information of the current camera. When the calibration conditions are determined to be met based on the actual position of the current camera, the compensation information of the camera is obtained based on the difference between the preset position information and the actual position information of the camera, thereby achieving the position calibration of the camera. On the one hand, since the position of the camera can be calibrated, the recognition accuracy of the robot is guaranteed. On the other hand, this calibration method uses image recognition to calculate the compensation information, and the camera calibration can be completed based on the original hardware design, without the need to replace the camera, thereby reducing the calibration cost.

[0094] When the robot is recharging, it may not be in place, that is, the robot is not close to the limit position of the base station. At this time, although the robot can capture the base station logo and charge, the position error of the camera calculated based on the difference between the preset position information and the actual position information of the camera is large and has a small reference value. Therefore, it is necessary to avoid unnecessary errors caused by the robot not reaching the recharging position or not being close to the limit position of the base station. In view of this, in one embodiment, when it is determined that the calibration conditions are met based on the actual position information of the current camera, compensation information of the camera is obtained based on the difference between the actual position information of the current camera and the preset position information, such as Figure 4 The flowchart of the method for determining the compensation information of the camera shown includes:

[0095] S402, calculating a position error of the current camera according to a difference between the actual position information of the current camera and the preset position information.

[0096] The camera position error calculated by the base station image can be obtained by comparing the difference between the preset position information of the camera and the actual position information of the camera.

[0097] Specifically, the camera position error calculated by the base station image can be obtained by comparing the preset position information (X w1 ,Y w1 ,Z w1 ) and the actual location information of the camera (X wc ,Y wc ,Z wc ), in cm. For example, the preset position information of the camera is (0,0,0), and the actual position information of the camera is (0,0,2). The distance between the two is 2 cm, which means that the actual position of the camera has shifted upward by 2 cm from the preset position when it left the factory.

[0098] The preset value is a judgment threshold of the position error, and the size of the preset value can be flexibly selected according to actual conditions, for example, any value between 1 cm and 5 cm.

[0099] Taking the preset value of 1cm as an example, specifically, when the position errors of the camera calculated based on at least three base station images are greater than 1cm, it is determined that the calibration conditions are met, and it is considered that in the current situation, the degree of camera offset has affected the normal working process of the robot.

[0100] S404: If the position errors of the current camera calculated based on at least three base station images are all greater than a preset value, it is determined that the calibration condition is met.

[0101] If the position error of the current camera calculated based on at least three base station images is not greater than a preset value, the base station images continue to be collected.

[0102] The at least three base station images may be three consecutive base station images taken by a camera when the robot performs a recharging action, or may be three discontinuous base station images.

[0103] S406, calculating compensation information corresponding to each current camera according to the position error of the current camera in each base station image.

[0104] Among them, the average value or variance of the camera position error of at least three base station images can be taken to obtain a processed camera position error, and the processed camera position error can be used as a reference for compensation information, and the compensation information can be calculated based on the reference of the compensation information.

[0105] S408, taking an average value and / or a variance of each compensation information to obtain final compensation information of the camera.

[0106] In this embodiment, by setting a pre-calibration step, when the position errors of the camera calculated by identifying three base station images are greater than the preset values, it is determined that the calibration conditions are met. On the basis of avoiding miscalibration due to accidental factors, the image acquisition time can be reduced, thereby improving the robot's movement continuity and work efficiency.

[0107] In one embodiment, when the number of base station images collected reaches a set value, it is determined that the calibration condition is met.

[0108] Among them, Figure 5 The flowchart of the process of determining the compensation information of the camera shown includes:

[0109] S502, calculating a position error of the current camera according to a difference between the actual position information of the current camera and the preset position information;

[0110] S504, if the position errors of the current camera calculated based on at least three base station images are all greater than a preset value, it is determined that the calibration condition is met;

[0111] If the position error of the current camera calculated based on at least three base station images is not greater than a preset value, S506 is executed.

[0112] S506, continue to collect base station images, and when the number of collected base station images reaches a set value, it is determined that the calibration condition is met.

[0113] The setting value may be determined according to the processing capability of the server, and a setting value equal to 10 times is taken as an example for explanation.

[0114] S508, calculating compensation information corresponding to each current camera according to the position error of the current camera in each base station image.

[0115] Among them, the average value or variance of the camera position errors of the 10 base station images can be taken to obtain a processed current camera position error to replace the current camera position error in each base station image, and the compensation information corresponding to each current camera is calculated based on the processed current camera position error.

[0116] S510, taking an average value and / or a variance of each compensation information to obtain final compensation information of the camera.

[0117] It should be noted that the 10 base station images contain 10 groups of camera position errors. The value of the camera position error can be positive or negative. Before taking the average or variance of the camera position errors of the 10 base station images, the two groups of camera position errors with the largest and smallest position errors among the 10 groups of cameras can be screened out to obtain the remaining 8 groups of camera position errors. Then, the average or variance of the remaining 8 groups of camera position errors can be taken to obtain a processed camera position error, and the processed camera position error can be used as a reference for compensation information.

[0118] In this embodiment, the position errors of multiple groups of cameras are obtained by collecting multiple base station images, and the two end values ​​of the position errors of the multiple groups of cameras are screened out. This can reduce the situation where the position errors of the cameras are too large due to the robot's inadequate recharging or other reasons, thereby improving the calculation accuracy of the position errors.

[0119] In one embodiment, when the number of base station images collected reaches a set value, the average value and / or variance of each compensation information is taken to obtain the final compensation information of the camera, including: sorting all compensation information in descending order according to the absolute value of the compensation information; removing the first N compensation information in the front sorting; taking the average value and / or variance of the retained compensation information to obtain the final compensation information of the camera.

[0120] Among them, the absolute values ​​of the compensation information of multiple groups of cameras calculated from all the base station images are calculated. The larger the absolute value of the compensation information is, the greater the deviation between the actual position of the camera and the preset position is, that is, the greater the probability that the robot has not been recharged into place.

[0121] Specifically, the value of N is at least 1, and the specific value of N can be flexibly selected according to actual conditions.

[0122] The absolute value of the retained compensation information is averaged or its variance is taken to obtain the final compensation information of the camera.

[0123] In this embodiment, by screening out the top N compensation information with the largest absolute values ​​of compensation information of multiple groups of cameras, the situation in which the position error is falsely high or low due to the robot failing to reach the recharging limit position during shooting or other reasons is avoided, thereby improving the calculation accuracy of the compensation information.

[0124] In one embodiment, the actual position information includes actual height information and actual angle information; the preset position information includes preset height information and preset angle information; the position error includes height error and angle error; the position error of the camera is calculated based on the difference between the preset position information and the actual position information of the camera, including: the height error of the camera is calculated based on the difference between the actual height information of the current camera and the preset height information, and the angle error of the camera is calculated based on the difference between the actual angle information of the current camera and the preset angle information; if the position error of the camera calculated based on at least three base station images is greater than a preset value, it is determined that the calibration condition is met, including: if the height error and / or angle error of the camera calculated based on at least three base station images is greater than a preset value, it is determined that the calibration condition is met.

[0125] Among them, when the robot is recharging, the robot reaches the limit position of the base station, and the horizontal distance between the camera and the base station mark is relatively fixed. Therefore, the position information can be converted into height information, angle information, and the horizontal distance between the camera and the base station mark.

[0126] Specifically, the actual height information may be the height of the current camera from the ground, and the actual angle information may be the angle between the current camera and the ground.

[0127] Among them, if the height error and / or angle error of the camera calculated based on at least three base station images are greater than the preset value, it is determined that the calibration conditions are met, including: the difference between the current camera's height above the ground and the preset camera's height above the ground is greater than the preset height, the difference between the current camera's angle to the ground and the preset camera's angle to the ground is greater than the preset angle, and the difference between the current camera's height above the ground and the preset camera's height above the ground is greater than the preset height, and the difference between the current camera's angle to the ground and the preset camera's angle to the ground is greater than the preset angle.

[0128] Specifically, when the above calibration conditions are met, the compensation information of the camera is obtained according to the difference between the height of the camera above the ground and the preset height of the camera above the ground, and the difference between the angle between the camera and the ground and the preset angle between the camera and the ground.

[0129] In this embodiment, the calculation process is simplified by converting position information into height information and angle information. The phenomenon of camera offset due to structural wear or other conditions after the robot leaves the factory is taken into consideration, and different compensation information generation methods are set for different offset phenomena, thereby improving the accuracy of camera calibration.

[0130] For the camera's offset in the height direction or the angle direction, different calculation methods need to be determined according to the different types of camera offset to obtain camera compensation. In one embodiment, the method for calculating the compensation information corresponding to the current camera is based on the position error of the current camera in the base station image, including: using trigonometric functions to calculate the compensation information corresponding to the current camera based on the identifier and the preset horizontal position information of the camera, the actual height information of the current camera, the actual angle information of the current camera, the preset height information of the camera, and the preset angle information of the camera.

[0131] The preset horizontal position information of the marker and the camera indicates the horizontal distance between the camera and the marker, and the horizontal distance is generally a fixed value.

[0132] Specifically, Figure 6 The schematic diagram of the relationship between the camera and the position shown includes: the preset horizontal position information X of the identification and the camera, the actual height information H of the current camera, the actual angle information α of the current camera, the preset height information h of the camera and the preset angle information β of the camera, wherein A is the preset camera position, B is the current camera position, and C is the identification position of the base station.

[0133] In this case, the camera's angle offset (α to β) and the camera's height offset (h to H) are both generated. Take the compensation information corresponding to the preset camera height and the preset camera angle as 0 as an example to illustrate: the calculation formula of the camera's compensation information at the current moment is: compensation information = H-h + X (tanα-tanβ).

[0134] In this embodiment, the compensation information is calculated according to the trigonometric function formula, which can provide a basis for subsequent camera calibration.

[0135] In one embodiment, Figure 7 As shown, a method for a user to select automatic calibration of a camera is provided, comprising:

[0136] S702, in response to the calibration instruction, control the robot to enter and exit the base station at least once, and when the robot enters and exits the base station, control the camera on the robot to shoot the base station logo, and obtain the base station image including the logo taken each time the robot enters and exits the base station.

[0137] S704, determine whether there are 10 base station images. If so, execute S708. If not, execute S706.

[0138] S706, controlling the robot to enter idle mode.

[0139] The idle mode refers to the robot's camera entering sleep mode, the camera does not collect images, and the robot does not process images. In this mode, the robot can be charged.

[0140] S708, identifying the coordinates of the base station image marked in the base station image and writing the coordinates into coordinate.txt, and deleting all images at the same time.

[0141] Among them, coordinate.txt refers to a coordinate file, which is used to temporarily store the coordinates marked in the base station image.

[0142] S710, after the recharging is completed, check whether the coordinate.txt file exists. If it exists, execute S712. If it does not exist, no calibration is performed and the process ends.

[0143] S712, transform the coordinates of the marker on the base station image to obtain actual position information of the current camera.

[0144] S714, calculating the position error of the camera according to the difference between the preset position information and the actual position information of the camera.

[0145] S716: If the position errors of the camera calculated based on at least three base station images are all greater than a preset value, it is determined that the calibration condition is met.

[0146] When the number of base station images collected reaches a set value, it is determined that the calibration condition is met.

[0147] Among them, when the number of base station images collected reaches the set value, the average value and / or variance of each compensation information is taken to obtain the final compensation information of the camera, including: sorting all compensation information in descending order according to the absolute value of the compensation information; removing the first N compensation information in the front sorting; taking the average value and / or variance of the retained compensation information to obtain the final compensation information of the camera.

[0148] The actual position information includes actual height information and actual angle information; the preset position information includes preset height information and preset angle information; the position error includes height error and angle error;

[0149] According to the difference between the preset position information and the actual position information of the camera, the position error of the camera is calculated, including:

[0150] The height error of the camera is calculated based on the difference between the actual height information of the current camera and the preset height information, and the angle error of the camera is calculated based on the difference between the actual angle information of the current camera and the preset angle information;

[0151] If the position errors of the camera calculated based on at least three base station images are greater than the preset values, it is determined that the calibration conditions are met, including: if the height errors and / or angle errors of the camera calculated based on at least three base station images are greater than the preset values, it is determined that the calibration conditions are met.

[0152] The method of calculating the compensation information corresponding to the current camera according to the position error of the current camera in the base station image includes:

[0153] According to the preset horizontal position information of the marker and the camera, the actual height information of the current camera, the actual angle information of the current camera, the preset height information of the camera and the preset angle information of the camera, the compensation information corresponding to the current camera is obtained by using trigonometric function calculation.

[0154] S718, calculating compensation information corresponding to each current camera according to the position error of the current camera in each base station image.

[0155] S720, taking an average value and / or a variance of each compensation information to obtain final compensation information of the camera.

[0156] S722, calibrate the camera according to the compensation information of the camera.

[0157] S724, save the calibration parameters to calibration.txt and delete the coordinate.txt file.

[0158] Among them, calibration.txt refers to a calibration file, which is used to save the changes of the internal parameters of the camera during the camera calibration process according to the camera compensation information.

[0159] In this embodiment, the robot is controlled to enter and exit the base station multiple times to shoot multiple groups of base station images containing logos, the logos in the base station images are identified, and the coordinates of the logos in the base station images are converted to obtain the actual position information of the current camera. When the calibration conditions are determined to be met based on the actual position of the current camera, the compensation information of the camera is obtained based on the difference between the preset position information and the actual position information of the camera, and the position calibration of the camera is realized. On the one hand, since the position of the camera can be calibrated, the recognition accuracy of the robot is guaranteed. On the other hand, this calibration method uses image recognition to calculate the compensation information, and the camera calibration can be completed based on the original hardware design, without replacing the camera, which reduces the calibration cost. .

[0160] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0161] Based on the same inventive concept, the embodiment of the present application also provides a robot camera calibration device for implementing the robot camera calibration method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of the camera calibration device for one or more robots provided below can refer to the limitations of the robot camera calibration method above, and will not be repeated here.

[0162] In one embodiment, Figure 8 As shown, a camera calibration device for a robot is provided, comprising: a shooting module 802, a recognition module 804, a compensation module 806 and a calibration module 808, wherein:

[0163] The shooting module 802 is used to control the robot to enter and exit the base station at least once in response to the calibration instruction, and control the camera on the robot to shoot the identification of the base station when the robot enters and exits the base station, so as to obtain the base station image including the identification taken each time when the robot enters and exits the base station;

[0164] The recognition module 804 is used to recognize the coordinates marked in the base station image in the base station image, and transform the coordinates marked in the base station image to obtain the actual position information of the current camera;

[0165] The compensation module 806 is used to obtain compensation information of the camera based on the difference between the actual position information of the current camera and the preset position information when it is determined that the calibration condition is met according to the actual position information of the current camera;

[0166] The calibration module 808 is used to calibrate the camera according to the compensation information of the camera.

[0167] In one embodiment, the compensation module 806 is also used to calculate the position error of the current camera based on the difference between the actual position information of the current camera and the preset position information; if the position errors of the current camera calculated based on at least three base station images are greater than the preset values, it is determined that the calibration conditions are met; according to the position errors of the current camera in each base station image, the compensation information corresponding to each current camera is calculated; and the average value and / or variance of each compensation information is taken to obtain the final compensation information of the camera.

[0168] In one embodiment, the camera calibration device of the robot further includes a collection module, which is used to determine that the calibration condition is satisfied when the number of base station images collected reaches a set value.

[0169] In one embodiment, the acquisition module is also used to sort all compensation information in descending order according to the absolute value of the compensation information; remove the first N compensation information in the front sorting; and take the average value and / or variance of the retained compensation information to obtain the final camera compensation information.

[0170] In one embodiment, the actual position information includes actual height information and actual angle information; the preset position information includes preset height information and preset angle information; the position error includes height error and angle error, and the camera calibration device of the robot also includes a calculation module for calculating the height error of the camera based on the difference between the actual height information of the current camera and the preset height information, and calculating the angle error of the camera based on the difference between the actual angle information of the current camera and the preset angle information; if the height error and / or angle error of the camera calculated based on at least three base station images are greater than the preset value, it is determined that the calibration conditions are met.

[0171] In one embodiment, the calculation module is also used to obtain compensation information corresponding to the current camera using trigonometric function calculation based on the preset horizontal position information of the identifier and the camera, the actual height information of the current camera, the actual angle information of the current camera, the preset height information of the camera and the preset angle information of the camera.

[0172] Each module in the camera calibration device of the robot can be implemented in whole or in part by software, hardware and a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0173] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store base station image data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a camera calibration method for a robot is implemented.

[0174] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0175] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0176] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0177] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0178] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0179] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0180] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0181] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A robot camera calibration method, It is characterized in that The method comprises: In response to the calibration instruction, the robot is controlled to enter and exit the base station at least once, and when the robot enters and exits the base station, the camera on the robot is controlled to shoot the identification of the base station, and the base station image including the identification is obtained each time the robot enters and exits the base station; Identify the coordinates of the marker in the base station image in a pixel coordinate system, transform the coordinates of the marker in the pixel coordinate system, obtain the coordinates of the marker in an image coordinate system, transform the coordinates of the marker in the image coordinate system, obtain the coordinates of the marker in a camera coordinate system, transform the coordinates of the marker in the camera coordinate system, obtain the coordinates of the marker in a world coordinate system, and based on the coordinates of the marker in the world coordinate system, obtain actual position information of the current camera in the world coordinate system; the actual position information includes actual height information and actual angle information, the actual height information refers to the height of the current camera from the ground, and the actual angle information refers to the angle between the current camera and the ground; The height error of the current camera is calculated based on the difference between the actual height information of the current camera and the preset height information, and the angle error of the current camera is calculated based on the difference between the actual angle information of the current camera and the preset angle information; If the height error and / or angle error of the current camera calculated based on at least three of the base station images are greater than a preset value, it is determined that the calibration condition is met; For each of the base station images, according to the identifier in the base station image and the preset horizontal position information of the current camera, the actual height information of the current camera, the actual angle information of the current camera, the preset height information of the current camera and the preset angle information of the current camera, the compensation information corresponding to the current camera is obtained by using trigonometric function calculation; wherein the compensation information corresponding to the current camera=H-h+X(tanα-tanβ), H represents the actual height information, h represents the preset height information, X represents the horizontal position information, α represents the actual angle information, and β represents the preset angle information; Taking an average value of each compensation information to obtain final compensation information of the camera; The camera is calibrated according to the final compensation information of the camera.

2. The method according to claim 1, It is characterized in that The method further comprises: If the position error of the current camera calculated based on at least three base station images is not greater than a preset value, continue to collect base station images; When the number of base station images collected reaches a set value, it is determined that the calibration condition is met.

3. The method according to claim 2, It is characterized in that When the number of base station images collected reaches a set value, taking an average value of each compensation information to obtain the final compensation information of the camera includes: Sorting all the compensation information in descending order according to the absolute values ​​of the compensation information; Remove the first N pieces of compensation information that are sorted in front; An average value is taken for the retained compensation information to obtain final compensation information of the camera.

4. The method according to claim 1, It is characterized in that The method further comprises: Obtaining the number of collisions of the robot detected by a collision sensor within a certain period of time; If the number of collisions is greater than a preset number, a calibration instruction is issued.

5. The method according to claim 1, It is characterized in that The method further comprises: When the robot reaches the limit position of the base station, the camera on the robot is controlled to shoot the logo of the base station.

6. A camera calibration device for a robot, It is characterized in that The device comprises: a shooting module, configured to control the robot to enter and exit the base station at least once in response to a calibration instruction, and to control a camera on the robot to shoot an identification of the base station when the robot enters and exits the base station, so as to obtain an image of the base station including the identification taken each time the robot enters and exits the base station; an identification module, used to identify the coordinates of the identifier in the base station image in a pixel coordinate system, transform the coordinates of the identifier in the pixel coordinate system, obtain the coordinates of the identifier in the image coordinate system, transform the coordinates of the identifier in the image coordinate system, obtain the coordinates of the identifier in the camera coordinate system, transform the coordinates of the identifier in the camera coordinate system, obtain the coordinates of the identifier in the world coordinate system, and based on the coordinates of the identifier in the world coordinate system, obtain the actual position information of the current camera in the world coordinate system; the actual position information includes actual height information and actual angle information, the actual height information refers to the height of the current camera from the ground, and the actual angle information refers to the angle between the current camera and the ground; A compensation module, used to calculate the height error of the current camera according to the difference between the actual height information and the preset height information of the current camera, and to calculate the angle error of the current camera according to the difference between the actual angle information and the preset angle information of the current camera; if the height error and / or angle error of the current camera calculated according to at least three of the base station images are greater than the preset value, it is determined that the calibration condition is met; for each of the base station images, according to the identifier in the base station image and the preset horizontal position information of the current camera, the actual height information of the current camera, the actual angle information of the current camera, the preset height information of the current camera and the preset angle information of the current camera, the compensation information corresponding to the current camera is calculated using trigonometric functions; wherein the compensation information corresponding to the current camera == H-h+X(tanα-tanβ), H represents the actual height information, h represents the preset height information, X represents the horizontal position information, α represents the actual angle information, and β represents the preset angle information; average the compensation information to obtain the final compensation information of the camera; A calibration module is used to calibrate the camera according to the final compensation information of the camera.

7. The device according to claim 6, It is characterized in that The camera calibration device of the robot also includes: a collection module: The acquisition module is used to continue acquiring base station images if the position error of the current camera calculated based on at least three base station images is not greater than a preset value; when the number of base station images acquired reaches a set value, determine that the calibration condition is met.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program. It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images