Robot sensor calibration method and device, electronic equipment and storage medium
By identifying the target mark and calculating the yaw angle, the problem of return charging yaw angle caused by position changes of the sweeping robot camera is solved, the success rate of return charging is improved, and convenient and efficient yaw angle calibration is achieved.
Patent Information
- Application Number
- CN202411123285.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-14
- Publication Date
- 2024-11-29
AI Technical Summary
During the use of existing sweeping robots, due to changes in the camera position, the yaw angle problem during recharge is affected, which affects the success rate of recharge. The existing technology is difficult to effectively solve the yaw angle problem of the camera.
By obtaining the position where the robot recognizes the target mark at different positions, calculates the yaw angle of the environmental sensor, and uses linear motion and image processing technology to determine the yaw angle of the camera, providing a robot sensor calibration method and device to achieve bias against the camera Automatic calibration of the azimuth angle.
It improves the success rate of recharge of the sweeping robot and simplifies the calibration process. Users do not need to return to the factory for repair or use professional equipment, achieving convenient and efficient yaw angle calibration.
Smart Images

Figure CN119045483A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of intelligent robots, and in particular to a robot sensor calibration method, device, electronic equipment and storage medium. Background Art
[0002] The existing sweeping robot can automatically return to the base station for charging (i.e., recharging) when completing the cleaning task or when the power is low. However, in order for the sweeping robot to accurately achieve recharging, the position and posture of the camera carried by the sweeping robot must be clearly understood.
[0003] After a sweeping robot is put into use, the position of the camera often changes due to collisions, structural wear and other reasons. Therefore, the position of the camera carried by the sweeping robot may change after a period of use, resulting in the inability of the sweeping robot to accurately recharge. In particular, when the position of the camera carried by the sweeping robot has a yaw angle relative to the theoretical position, it will seriously affect the success rate of the sweeping robot's recharging. Therefore, how to determine the yaw angle of the camera carried by the sweeping robot is an issue that needs to be solved urgently. Summary of the invention
[0004] The purpose of the embodiments of the present application is to provide a robot sensor calibration method, device, electronic device and storage medium to solve the problem of how to determine the yaw angle of the robot environment sensor. The specific technical solution is as follows:
[0005] In a first aspect, an embodiment of the present application provides a robot sensor calibration method, the method comprising:
[0006] Acquire the position of the target identifier in the environmental sensor coordinate system obtained by the environmental sensor recognizing the target identifier when the robot is in the first position, and obtain the first identifier position; wherein the environmental sensor is arranged on the robot;
[0007] Acquire the position of the target identifier in the environmental sensor coordinate system obtained by the environmental sensor identifying the target identifier when the robot is in the second position, and obtain a second identifier position; the second position is obtained when the robot moves in a straight line along the first position with the orientation unchanged;
[0008] A first yaw angle of the environmental sensor is obtained according to an angle between a straight line formed by the first marking position and the second marking position and an axis of the environmental sensor coordinate system.
[0009] In some embodiments, the method further comprises:
[0010] Obtain a first identification posture according to the posture of the target identifier in the environmental sensor coordinate system obtained by the environmental sensor identifying the target identifier when the robot is in the first posture; and determine a first theoretical position of the robot in the world coordinate system according to the first identification posture;
[0011] According to the pose of the target identifier in the environmental sensor coordinate system obtained by the environmental sensor identifying the target identifier when the robot is in the second pose, a second identification pose is obtained; and according to the second identification pose, a second theoretical position of the robot in the world coordinate system is determined; wherein the second pose is obtained after the robot rotates in the first pose;
[0012] A first offset of the environmental sensor is obtained according to a distance between the first theoretical position and the second theoretical position and a rotation angle from the first posture to the second posture.
[0013] In some embodiments, the second posture is obtained by rotating the robot 60 degrees from the first posture, and obtaining the first offset of the environmental sensor according to the distance between the first theoretical position and the second theoretical position and the angle of rotation from the first posture to the second posture includes:
[0014] The distance between the first theoretical position and the second theoretical position is calculated to obtain a first offset of the environmental sensor.
[0015] In some embodiments, determining a first theoretical position of the robot in a world coordinate system according to the first identified posture includes:
[0016] Determine a first environmental sensor posture of the environmental sensor in a world coordinate system according to the first identification posture; and obtain the first theoretical position according to a conversion relationship between the environmental sensor coordinate system and the robot coordinate system, and the first environmental sensor posture;
[0017] Determining a second theoretical position of the robot in a world coordinate system according to the second identification posture includes:
[0018] Determine a second environmental sensor posture of the environmental sensor in the world coordinate system according to the second identification posture; and obtain the second theoretical position according to the conversion relationship between the environmental sensor coordinate system and the robot coordinate system, and the second environmental sensor posture.
[0019] In some embodiments, the method further comprises:
[0020] Repeat the steps of obtaining the first identification position, obtaining the second identification position, and obtaining the first yaw angle until a first preset number of times is reached, thereby obtaining a plurality of first yaw angles;
[0021] An average value of the plurality of first yaw angles is calculated as a second yaw angle of the environmental sensor.
[0022] In some embodiments, the method further comprises:
[0023] Repeat the steps of determining the first theoretical position, determining the second theoretical position, and obtaining the first offset of the environmental sensor for a second preset number of times to obtain a plurality of first offsets;
[0024] An average of the plurality of first offsets is calculated as a second offset of the environment sensor.
[0025] In some embodiments, the robot is a sweeping robot, the environmental sensor is a camera, and the target identifier is a QR code set on a base station; wherein the identifier position is obtained by:
[0026] Determine the positions of the four corner points of the two-dimensional code from the image obtained by the camera shooting the target identifier;
[0027] Determine the position of the two-dimensional code in the image according to the positions of the four corner points;
[0028] Projecting the position of the two-dimensional code in the image into the camera coordinate system to obtain the identification position;
[0029] or,
[0030] The robot is a sweeping robot, the environmental sensor is a radar, and the target identifier is a reflective sticker arranged on a base station; wherein the reflective sticker can reflect the radar signal emitted by the radar;
[0031] In a second aspect, an embodiment of the present application further provides a robot sensor calibration device, the device comprising:
[0032] The identification position determination module is used to obtain the position of the target identification obtained by the environmental sensor identifying the target identification when the robot is in the first position in the environmental sensor coordinate system, and obtain the first identification position; wherein the environmental sensor is arranged on the robot; obtain the position of the target identification obtained by the environmental sensor identifying the target identification when the robot is in the second position in the environmental sensor coordinate system, and obtain the second identification position; the second position is obtained by the robot moving in a straight line along the first position without changing its orientation;
[0033] The yaw angle determination module is used to obtain a first yaw angle of the environmental sensor according to an angle between a straight line formed by the first marker position and the second marker position and an axis of the environmental sensor coordinate system.
[0034] In some embodiments, the apparatus further comprises:
[0035] An identification posture determination module is used to obtain a first identification posture according to the posture of the target identifier in the environmental sensor coordinate system obtained by the environmental sensor identifying the target identifier when the robot is in a first posture; and determine the first theoretical position of the robot in the world coordinate system according to the first identification posture; obtain a second identification posture according to the posture of the target identifier in the environmental sensor coordinate system obtained by the environmental sensor identifying the target identifier when the robot is in a second posture; and determine the second theoretical position of the robot in the world coordinate system according to the second identification posture; wherein the second posture is obtained after the robot rotates in the first posture;
[0036] The offset determination module is used to obtain a first offset of the environmental sensor according to the distance between the first theoretical position and the second theoretical position and the rotation angle from the first posture to the second posture.
[0037] In some embodiments, the second posture is obtained by rotating the robot 60 degrees from the first posture, and obtaining the first offset of the environmental sensor according to the distance between the first theoretical position and the second theoretical position and the angle of rotation from the first posture to the second posture includes:
[0038] The distance between the first theoretical position and the second theoretical position is calculated to obtain a first offset of the environmental sensor.
[0039] In some embodiments, determining a first theoretical position of the robot in a world coordinate system according to the first identification posture includes:
[0040] Determine a first environmental sensor posture of the environmental sensor in a world coordinate system according to the first identification posture; and obtain the first theoretical position according to a conversion relationship between the environmental sensor coordinate system and the robot coordinate system, and the first environmental sensor posture;
[0041] Determining a second theoretical position of the robot in a world coordinate system according to the second identification posture includes:
[0042] Determine a second environmental sensor posture of the environmental sensor in the world coordinate system according to the second identification posture; and obtain the second theoretical position according to the conversion relationship between the environmental sensor coordinate system and the robot coordinate system, and the second environmental sensor posture.
[0043] In some embodiments, the yaw angle determination module is further used to:
[0044] Repeat the steps of obtaining the first identification position, obtaining the second identification position, and obtaining the first yaw angle until a first preset number of times is reached, thereby obtaining a plurality of first yaw angles;
[0045] An average value of the plurality of first yaw angles is calculated as a second yaw angle of the environmental sensor.
[0046] In some embodiments, the offset determination module is further used to:
[0047] Repeat the steps of determining the first theoretical position, determining the second theoretical position, and obtaining the first offset of the environmental sensor for a second preset number of times to obtain a plurality of first offsets;
[0048] An average of the plurality of first offsets is calculated as a second offset of the environment sensor.
[0049] In some embodiments, the robot is a sweeping robot, the environmental sensor is a camera, and the target identifier is a QR code set on a base station; wherein the identifier position is obtained by:
[0050] Determine the positions of the four corner points of the two-dimensional code from the image obtained by the camera shooting the target identifier;
[0051] Determine the position of the two-dimensional code in the image according to the positions of the four corner points;
[0052] Projecting the position of the two-dimensional code in the image into the camera coordinate system to obtain the identification position;
[0053] or,
[0054] The robot is a sweeping robot, the environmental sensor is a radar, and the target identifier is a reflective sticker arranged on a base station; wherein the reflective sticker can reflect the radar signal emitted by the radar.
[0055] In a third aspect, an embodiment of the present application further provides an electronic device, including:
[0056] Memory, used to store computer programs;
[0057] The processor is used to implement any of the above-mentioned robot sensor calibration methods when executing the program stored in the memory.
[0058] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the robot sensor calibration method described above is implemented.
[0059] Beneficial effects of the embodiments of the present application:
[0060] In the technical solution provided in the embodiment of the present application, it can be understood that when the robot keeps its orientation unchanged and moves in a straight line, the straight line formed by the first identification position and the second identification position should be parallel to the movement direction of the robot. Based on this, when the environmental sensor has a yaw angle (that is, there is an angle between the orientation of the environmental sensor and the orientation of the robot), if the robot moves in a straight line along its orientation, the straight line formed by the first identification position and the second identification position is parallel to the orientation of the robot. At this time, the angle between the straight line and the axis of the environmental sensor coordinate system is calculated, such as the angle between the straight line and the orientation of the environmental sensor, and the angle between the robot orientation and the orientation of the environmental sensor, that is, the yaw angle of the environmental sensor, can be obtained; if the robot keeps its orientation unchanged and moves in a straight line in other directions, the straight line formed by the first identification position and the second identification position is parallel to its movement direction. According to the angle between the straight line (that is, its movement direction) and the orientation of the robot, and the angle between the straight line and the axis of the environmental sensor coordinate system (such as the angle between the straight line and the orientation of the environmental sensor), the angle between the orientation of the environmental sensor and the orientation of the robot, that is, the yaw angle of the environmental sensor, can also be calculated.
[0061] Based on this, in an embodiment of the present application, when the robot is in the first position, the environmental sensor identifies the target identifier and obtains the position of the target identifier in the environmental sensor coordinate system, thereby obtaining the first identifier position; the robot is controlled to move in a straight line along the first position while keeping the orientation unchanged, thereby obtaining the second position; when the robot is in the second position, the environmental sensor identifies the target identifier and obtains the position of the target identifier in the environmental sensor coordinate system, thereby obtaining the second identifier position; the first yaw angle of the environmental sensor can be obtained based on the angle between the straight line formed by the first identifier position and the second identifier position and the axis of the environmental sensor coordinate system.
[0062] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS
[0063] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.
[0064] Figure 1 A schematic diagram of a robot coordinate system and a camera coordinate system under theoretical conditions provided in an embodiment of the present application;
[0065] Figure 2 A schematic diagram of a flow chart of a robot sensor calibration method provided in an embodiment of the present application;
[0066] Figure 3 A schematic diagram of a robot provided in an embodiment of the present application moving from a first position to a second position;
[0067] Figure 4 Another schematic diagram of a robot provided in an embodiment of the present application moving from a first position to a second position;
[0068] Figure 5 for Figure 3 A schematic diagram of calculating the yaw angle corresponding to the example shown;
[0069] Figure 6 for Figure 3 Another schematic diagram of calculating the yaw angle corresponding to the example shown;
[0070] Figure 7 for Figure 4 A schematic diagram of calculating the yaw angle corresponding to the example shown;
[0071] Figure 8 A schematic diagram of a flow chart of a robot camera calibration method provided in an embodiment of the present application;
[0072] Fig. 9 A schematic diagram of determining the camera posture according to an embodiment of the present application;
[0073] Fig.10 A schematic diagram of a robot provided in an embodiment of the present application moving from a first posture to a second posture;
[0074] Fig.11 A schematic diagram of determining an offset provided in an embodiment of the present application;
[0075] Fig.12 Another schematic diagram of a flow chart of a robot camera calibration method provided in an embodiment of the present application;
[0076] Fig.13A schematic diagram of the structure of a robot camera calibration device provided in an embodiment of the present application;
[0077] Fig.14 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0078] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.
[0079] Whether the robot vacuum cleaner can be accurately recharged depends on whether the position of the camera carried by the robot vacuum cleaner is clear. In theory, the camera is installed at the center line of the robot vacuum cleaner body and in front of the body, and the direction of the camera coordinate system is consistent with the direction of the robot vacuum cleaner coordinate system. For details, see Figure 1 The example shown, Figure 1 The middle gray coordinate system C1 represents the coordinate system of the camera, the x'-axis represents the direction of the camera, and the y'-axis represents the axis perpendicular to the x'-axis; the black coordinate system R1 represents the coordinate system of the sweeping robot, the x-axis represents the direction of the sweeping robot, and the y-axis represents the axis perpendicular to the x-axis; the origin of the coordinate system C1 represents the position of the camera, and the origin of the coordinate system R1 represents the position of the sweeping robot; Figure 1 As shown, the coordinate system C1 is located at the center line of the robot body and in front of the cleaning robot body, and the x' axis of the coordinate system C1 coincides with the x axis of the coordinate system R1, that is, the direction of the camera coordinate system is consistent with the direction of the cleaning robot.
[0080] Usually, in order to clarify the camera's posture, the tolerance and installation accuracy of the parts are guaranteed as much as possible when producing the sweeping robot, so that the camera is in the theoretical position as much as possible, and then the camera is calibrated. However, after the sweeping robot is put into use, the camera's posture often changes due to bumps, structural wear and other reasons. Therefore, after a period of use, the camera's posture may no longer be the posture of the camera specified at the factory, which will seriously affect the accuracy and success rate of the sweeping robot's recharging.
[0081] In the related art, there is a robot camera calibration method, device, computer equipment and storage medium, through which the camera can be calibrated to clarify the camera's position and posture. However, this method requires the robot to enter and exit the base station multiple times to make corrections, and when the camera has a yaw angle, the robot may not be able to successfully enter the base station. In addition, this solution only corrects the offset and pitch angle of the camera, but in actual applications, especially recharging, the yaw angle of the camera is the key factor that determines whether the robot can recharge successfully.
[0082] Therefore, the embodiment of the present application provides a robot sensor calibration method. The application of the technical solution provided in the embodiment of the present application can clarify the yaw angle of the robot environment sensor, and then when the posture of the robot environment sensor deviates from the theoretical posture, the success rate of the robot recharging can be improved. Below, the robot sensor calibration method provided in the embodiment of the present application is described in detail through specific examples.
[0083] The robot sensor calibration method provided in the embodiment of the present application can be applied to electronic devices. Specifically, the electronic device can be a desktop computer, a portable computer, a smart mobile terminal, etc.
[0084] See also Figure 2 , Figure 2 A schematic flow chart of a robot sensor calibration method provided in an embodiment of the present application includes the following steps:
[0085] Step S21: obtaining the position of the target identifier obtained by the environmental sensor recognizing the target identifier when the robot is in the first position in the environmental sensor coordinate system, and obtaining the first identifier position; wherein the environmental sensor is arranged on the robot;
[0086] Step S22: acquiring the position of the target identifier obtained by the environmental sensor recognizing the target identifier when the robot is at the second position in the environmental sensor coordinate system, and obtaining a second identifier position;
[0087] Step S23: Obtaining a first yaw angle of the environmental sensor according to an angle between a straight line formed by the first marking position and the second marking position and an axis of the environmental sensor coordinate system.
[0088] In the technical solution provided in the embodiment of the present application, it can be understood that when the robot keeps its orientation unchanged and moves in a straight line, the straight line formed by the first identification position and the second identification position should be parallel to the movement direction of the robot. Based on this, when the environmental sensor has a yaw angle (that is, there is an angle between the orientation of the environmental sensor and the orientation of the robot), if the robot moves in a straight line along its orientation, the straight line formed by the first identification position and the second identification position is parallel to the orientation of the robot. At this time, the angle between the straight line and the axis of the environmental sensor coordinate system is calculated, such as the angle between the straight line and the orientation of the environmental sensor, and the angle between the robot orientation and the orientation of the environmental sensor, that is, the yaw angle of the environmental sensor, can be obtained; if the robot keeps its orientation unchanged and moves in a straight line in other directions, the straight line formed by the first identification position and the second identification position is parallel to its movement direction. According to the angle between the straight line (that is, its movement direction) and the orientation of the robot, and the angle between the straight line and the axis of the environmental sensor coordinate system (such as the angle between the straight line and the orientation of the environmental sensor), the angle between the orientation of the environmental sensor and the orientation of the robot, that is, the yaw angle of the environmental sensor, can also be calculated.
[0089] Based on this, in an embodiment of the present application, when the robot is in the first position, the environmental sensor identifies the target identifier and obtains the position of the target identifier in the environmental sensor coordinate system, thereby obtaining the first identifier position; the robot is controlled to move in a straight line along the first position while keeping the orientation unchanged, thereby obtaining the second position; when the robot is in the second position, the environmental sensor is identified by the target identifier and obtains the position of the target identifier in the environmental sensor coordinate system, thereby obtaining the second identifier position; the first yaw angle of the environmental sensor can be obtained according to the angle between the straight line formed by the first identifier position and the second identifier position and the axis of the environmental sensor coordinate system.
[0090] In addition, according to the technical solution provided in the embodiment of the present application, it is only necessary to set the target mark and then control the robot to perform a simple translational action to calibrate the yaw angle of the environmental sensor. Especially for a sweeping robot, the user only needs to issue a command to make the sweeping robot execute the robot sensor calibration method provided in the embodiment of the present application. For example, the user's mobile phone is installed with an application for controlling the sweeping robot. The user only needs to click a button in the application to instruct the sweeping robot to automatically execute the environmental sensor calibration method provided in the embodiment of the present application. There is no need to return the robot to the factory for repair or to use professional tooling calibration equipment for processing. The robot can autonomously complete the robot sensor calibration method provided in the embodiment of the present application. It is simple and convenient, and the reliability of the yaw angle of the calibrated environmental sensor is high.
[0091] In the above step S21, the robot can be a sweeping robot, an automated guided vehicle (AGV), or other robots with certain image acquisition and recognition capabilities. The environmental sensor can be a camera set on the robot, or a radar set on the robot. When the environmental sensor is a camera set on the robot, the target identifier can be an identifier of a specific shape that has been set when the robot leaves the factory. In order to facilitate the robot to easily identify the target identifier and efficiently determine the position of the target identifier in the camera coordinate system, the target identifier can be an identifier that is easy to be recognized by the robot and easy to determine the feature point, such as a QR code; when the environmental sensor is a radar set on the robot, the target identifier can be a feature that can reflect the lightning signal emitted by the radar, such as a reflective sticker. In addition, in order to facilitate the robot to quickly determine the orientation of the target identifier in a large scene, the target identifier can be placed on a device that communicates with the robot. For example, when the robot is a sweeping robot, the target identifier can be placed on a base station for the sweeping robot to recharge, such as a QR code attached to the base station for the sweeping robot to recharge.
[0092] The principle of implementing the present application when the environmental sensor is a radar is the same as the principle of implementing the present application when the environmental sensor is a camera. Therefore, for the convenience of description, the camera environmental sensor calibration method provided in the present application is explained below only by taking the environmental sensor as a camera as an example.
[0093] The electronic device can control the movement of the robot and determine whether there is a target marker in the camera's field of view through the image captured by the camera. When the target marker is detected in the camera's field of view, the current position of the robot can be regarded as the first position, and the image obtained by the camera capturing the target marker at the first position is the first image.
[0094] In some embodiments, the electronic device can determine the positions of multiple feature points of the target identifier from a first image obtained by photographing the target identifier, determine the position of the target identifier in the first image based on the positions of the multiple feature points of the target identifier, and project the position of the target identifier in the first image into the camera coordinate system to obtain the position of the target identifier in the camera coordinate system, i.e., the first identifier position.
[0095] For example, when the robot is a sweeping robot and the target identifier is a QR code set on a base station, the electronic device can determine the positions of the four corner points of the QR code from the first image; then, determine the position of the QR code in the first image based on the positions of the four corner points, and then project the position of the QR code in the first image into the camera coordinate system, thereby obtaining the position of the QR code in the camera coordinate system, that is, the first identifier position.
[0096] In the above step S22, the electronic device can control the robot to move linearly from the first position to the second position while keeping the orientation unchanged, such as controlling the robot to move linearly from the first position to the second position along its orientation (as described below). Figure 3 Alternatively, the robot is controlled to move linearly to the left or right along the first position to the second position (as shown below) while maintaining the same orientation. Figure 4 shown).
[0097] Among them, the distance for linear motion can be a preset distance, such as the distance for linear motion can be preset to 10cm, 20cm, etc.; or, the distance for linear motion can be controlled by the user according to actual conditions. For example, the user can set the distance for linear motion of the sweeping robot according to the working scene of the sweeping robot at home. When the working scene of the sweeping robot at home is small, the user can set the distance for linear motion of the sweeping robot to be smaller, such as 10cm; when the working scene of the sweeping robot at home is large, the user can set the distance for linear motion of the sweeping robot to be larger, such as 20cm.
[0098] The second image is obtained by photographing the target mark when the robot is in the second position; the method of obtaining the second mark position according to the second image is the same as the method of obtaining the first mark position according to the first image in the above step S21, and please refer to the description in the above step S21.
[0099] For example, the robot is a sweeping robot, the target identifier is a QR code located on the base station, and the preset distance is 20 cm. Figure 3 and Figure 4 The method of the robot moving from the first position to the second position is described below. The principles of other robots are the same and will not be described here.
[0100] Figure 3 In the figure, the QR code 302 is attached to the base station 301, position A is the first position shown in the above step S21, R2 represents the coordinate system of the sweeping robot 303 when it is at position A, and the x-axis represents the orientation of the sweeping robot 303; C2 represents the coordinate system of the camera when the sweeping robot 303 is at position A, and the x'-axis represents the orientation of the camera; the sweeping robot 303 is controlled to move in a straight line along the direction indicated by the x-axis for 20 cm to position B, and position B can be regarded as the second position in the above step S22, R3 represents the coordinate system of the sweeping robot 303 when it is at position B, and C3 represents the coordinate system of the camera when the sweeping robot 303 is at position B.
[0101] Figure 4In the figure, the QR code 302 is attached to the base station 301, position C is the first position shown in the above step S21, R4 represents the coordinate system of the sweeping robot 303 when it is at position C, and the x-axis represents the orientation of the sweeping robot 303; C4 represents the coordinate system of the camera when the sweeping robot 303 is at position C, and the x'-axis represents the orientation of the camera; the sweeping robot 303 is controlled to move 20 cm to the left front in a straight line while keeping the orientation unchanged to position D, and position D can be regarded as the second position in the above step S22, R5 represents the coordinate system of the sweeping robot 303 when it is at position D, and C5 represents the coordinate system of the camera when the sweeping robot 303 is at position D.
[0102] It can be understood that in order to ensure that the robot can still keep the target mark within the field of view of the camera after moving left or right to obtain the second position, the relative position relationship between the first position and the target mark can be referred to when controlling the robot to move left or right, for example, Figure 4 The first position is far away from the base station 301 and is located on the right side of the QR code 302. From the position of the QR code 302 in the camera image, it can be seen that if the sweeping robot 303 moves to the right, the QR code 302 may not be within the field of view of the camera. Therefore, the sweeping robot 303 can be controlled to move to the left.
[0103] In the above step S23, the yaw angle is the angle between the actual orientation of the camera and the theoretical orientation. Since the orientation of the camera is consistent with the orientation of the robot in theory, the yaw angle can also be the angle between the actual orientation of the camera and the orientation of the robot. When the camera has a yaw angle, the first yaw angle of the camera can be obtained according to the angle between the straight line formed by the first mark position and the second mark position as two points and the axis of the camera coordinate system.
[0104] Next, through Figure 5 and Figure 7 The principle of obtaining the first yaw angle based on the angle between the straight line formed by the first marking position and the second marking position and the axis of the camera coordinate system in the embodiment of the present application is explained.
[0105] With the above Figure 3 Take the movement shown as an example, see Figure 5 , Figure 5 The angle θ1 between the x-axis of the robot coordinate system and the x'-axis of the camera coordinate system is the yaw angle of the camera. The robot moves linearly from position A to position B along the x-axis direction, and the QR code moves relative to the movement direction of the sweeping robot body. Therefore, the straight line formed by the first identification position and the second identification position as two points should be parallel to the straight line where the x-axis is located. For ease of description, Figure 5The middle gray straight line represents the straight line formed by the first identification position and the second identification position. The gray straight line is parallel to the straight line where the x-axis is located. Therefore, the electronic device can calculate the angle between the gray straight line and the x' axis of the camera coordinate system to obtain the first yaw angle of the camera. For example, by calculating θ2, θ1 can be obtained, that is, the first yaw angle of the camera. Alternatively, in some embodiments, the electronic device can also first calculate other angles that can reflect the size of θ1, and then determine the size of θ1 based on these angles. For example, the electronic device can calculate the angle between the gray straight line and the y' axis of the camera coordinate system. For example, by calculating θ3, θ1 can also be calculated based on θ3, that is, the first yaw angle of the camera.
[0106] After converting the camera coordinate system and the determined first marker position and second marker position to the world coordinate system, we can get Figure 6 The schematic diagram of calculating the first yaw angle is shown in FIG. Figure 6 The middle coordinate system is the camera coordinate system, Ta represents the first marking position, Tb represents the second marking position, and the angle θ1 between the straight line formed by the first marking position and the second marking position and the x' axis is the first yaw angle of the camera.
[0107] With the above Figure 4 Take the movement shown as an example, see Figure 7 , Figure 7 The angle θ4 between the x-axis of the robot coordinate system and the x'-axis of the camera coordinate system is the first yaw angle of the camera. The robot moves straight to the left front to position D while keeping the orientation unchanged. The QR code moves relative to the movement direction of the robot body. Therefore, the straight line formed by the first identification position and the second identification position should be parallel to the movement direction of the robot. For the convenience of description, Figure 7 The medium gray straight line represents the straight line formed by the first mark position and the second mark position. Figure 7 It can be known that the electronic device can obtain the first yaw angle of the camera by calculating the angle between the gray straight line and the x' axis of the camera coordinate system. For example, if θ5 is calculated, θ4 (i.e., the first yaw angle) can be calculated based on θ5. Specifically, the electronic device can determine the angle between the robot's movement direction and the robot's orientation based on devices such as motion sensors, that is, the angle between the gray straight line and the x-axis of the robot coordinate system (i.e., the sum of θ4 and θ5). In addition, the electronic device can calculate the angle between the gray straight line and the x' axis of the camera coordinate system (i.e., θ5). Based on this, the electronic device can obtain θ4 (i.e., the first yaw angle).
[0108] Alternatively, in some embodiments, the electronic device may first calculate other angles that can reflect the size of θ4, and then determine the size of θ4 based on these angles. For example, the electronic device may also calculate the angle between the gray straight line and the y' axis of the camera coordinate system. If θ6 is calculated, θ4 (i.e., the first yaw angle) can also be calculated based on θ6. Specifically, based on θ6, the electronic device can determine θ5, and then based on the angle between the robot's movement direction and the robot's orientation (i.e., the sum of θ4 and θ5), θ4 (i.e., the first yaw angle) can be obtained.
[0109] In some embodiments, the electronic device may directly use the obtained first yaw angle as the yaw angle of the camera, thereby reducing the complexity of calibration and improving the efficiency of camera calibration.
[0110] In some other embodiments, since the first yaw angle is obtained only after one calibration, directly using the first yaw angle as the yaw angle of the camera may not be accurate enough. Based on this, the electronic device can repeatedly perform the above steps S21 to S23 until a first preset number of times is reached to obtain multiple first yaw angles, and calculate the average of the multiple first yaw angles to obtain the second yaw angle of the camera, and use the second yaw angle as the yaw angle of the camera.
[0111] For example, the electronic device may control the robot to repeat the process of moving from the first position shown in step S21 to the second position shown in step S22 and calculating the first yaw angle, for example, as shown in step S21. Figure 3 As shown, after the electronic device controls the robot to move from position A to position B and calculates a first yaw angle, the robot can be controlled to return to position A again, and then move from position A to position B again. Figure 3 Move to position B in the manner shown, and then calculate a yaw angle. Repeat this rule until the number of repetitions reaches a first preset number of times to obtain multiple first yaw angles, and then use the average of the multiple first yaw angles as the second yaw angle, and use the second yaw angle as the yaw angle of the camera. In this way, the accuracy of the calibrated yaw angle can be improved.
[0112] Alternatively, during the repeated execution, the electronic device may only maintain the mode of moving from the first position to the second position, while the specific first position and the second position may change, for example, as described above. Figure 3 As shown, after the electronic device controls the robot to move from position A to position B and calculates a first yaw angle, the robot can be controlled to rotate, translate, etc. to obtain a new first position, and then the new first position is adjusted according to the Figure 3Move to a new second position in the manner shown, and obtain a first yaw angle by calculation. Repeat this rule until the number of repetitions reaches a first preset number of times to obtain multiple first yaw angles, and then use the average of the multiple first yaw angles as the yaw angle of the camera. In this way, the accuracy of the calibrated yaw angle can also be improved.
[0113] The second yaw angle is actually a value used to represent the concentration trend of multiple first yaw angles. Therefore, in addition to using the mean of multiple first yaw angles as the second yaw angle, the second yaw angle can also be calculated in other ways. For example, after obtaining multiple first yaw angles, the median or mode of the multiple first yaw angles can also be used as the second yaw angle. Alternatively, multiple first yaw angles can be clustered to determine isolated points, and then the mean or median of the first yaw angles other than the isolated points can be used as the second yaw angle. The first preset number of times is the number of times that steps S21-S23 are pre-set to be repeated. The size of the first preset number of times can be set according to user needs, such as the size of the first preset number of times can be set to 10 times, 20 times, 30 times, etc.
[0114] The above has explained how to calibrate the yaw angle. It is understandable that in addition to the yaw angle, the camera may also have an offset during use. Based on this, the present application also provides a robot camera calibration method capable of calibrating the offset. The method can be found in Figure 8 , Figure 8 A schematic flow chart of a robot camera calibration method provided in an embodiment of the present application may include the following steps:
[0115] Step S81: acquiring a first image obtained by photographing a target mark when the robot is at the first position, determining the position of the target mark in the camera coordinate system according to the first image, and obtaining a first mark position;
[0116] Step S82: acquiring a second image obtained by photographing the target mark when the robot is at the second position, determining the position of the target mark in the camera coordinate system according to the second image, and obtaining the second mark position;
[0117] Step S83: obtaining a first yaw angle of the camera according to an angle between a straight line formed by the first marking position and the second marking position and an axis of the camera coordinate system;
[0118] Step S84: determining the position and posture of the target marker in the camera coordinate system according to the third image obtained by photographing the target marker when the robot is in the first posture, and obtaining the first marker position and posture; and determining the first theoretical position of the robot in the world coordinate system according to the first marker position and posture;
[0119] Step S85: determining the position and posture of the target mark in the camera coordinate system according to the fourth image obtained by photographing the target mark when the robot is in the second position and posture, and obtaining the second mark position and posture; and determining the second theoretical position of the robot in the world coordinate system according to the second mark position and posture; wherein the second position and posture is obtained after the robot is rotated in the first position and posture;
[0120] Step S86: Obtain a first offset of the camera according to the distance between the first theoretical position and the second theoretical position and the rotation angle from the first posture to the second posture.
[0121] In the technical solution provided by the embodiment of the present application, the electronic device can determine the pose of the target marker in the camera coordinate system according to the third image obtained by photographing the target marker when the robot is in the first pose, and obtain the first marker pose; and determine the first theoretical position of the robot in the world coordinate system according to the first marker pose; determine the pose of the target marker in the camera coordinate system according to the fourth image obtained by photographing the target marker when the robot is in the second pose, and obtain the second marker pose; and determine the second theoretical position of the robot in the world coordinate system according to the second marker pose, wherein the second pose is obtained after the robot rotates in the first pose; the first offset of the camera can be obtained according to the distance between the first theoretical position and the second theoretical position, and the rotation angle from the first pose to the second pose.
[0122] In addition, it can be understood that when the camera does not have a yaw angle but only an offset, the aforementioned steps S81 to S83 may not be performed, and only steps S84 to S86 may be performed to obtain the first offset. In the case where the camera has a yaw angle, if steps S84 to S86 are required to determine the first offset, the yaw angle of the camera may be first determined according to steps S81 to S83, and then the first offset may be determined using steps S84 to S86; or, the electronic device may first determine the yaw angle of the camera using other methods in the relevant technology, and then determine the first offset using steps S84 to S86. In this way, when there is a yaw angle, even if steps S81 to S83 are not used, the first offset may be determined based on steps S84 to S86.
[0123] Based on this, according to the technical solution provided in the embodiment of the present application, it is only necessary to set the target mark and then control the robot to perform a simple translation or rotation action to complete the calibration of the camera yaw angle and offset. Especially for a sweeping robot, the user only needs to issue a command to make the sweeping robot execute Figure 8 The robot camera calibration method shown in the figure can be used. For example, if a user's mobile phone has an application for controlling the sweeping robot installed, the user only needs to click a button in the application to instruct the sweeping robot to automatically execute Figure 8The camera calibration method shown in the figure can be used. There is no need to return the camera to the factory for repair or use professional tooling calibration equipment for processing. The robot can complete this task autonomously. Figure 8 The robot camera calibration method shown is simple and convenient, and the calibration results (i.e., yaw angle and offset) obtained by calibration are highly reliable.
[0124] The above steps S81 to S83 are the same as the above steps Figure 2 Steps S21 to S23 are the same and will not be repeated here.
[0125] In the above step S84, the posture represents the position and angle. The first posture can be any posture of the robot that can capture the target mark, and the electronic device can determine the first posture in the manner of determining the first position in the above step S21. In some embodiments, if the yaw angle of the camera is calibrated first, then after the calibration of the yaw angle is completed, the posture corresponding to the second position shown in the above step S22 can be directly used as the first posture of the robot, so that the steps of moving the robot again and determining whether the target mark is within the field of view can be avoided, thereby improving execution efficiency and reducing resource consumption. Alternatively, after the calibration of the yaw angle is completed, the robot can be controlled to return from the second position in the above step S22 to the first position in the above step S21. At the first position, the target mark must be within the field of view of the camera. Therefore, the posture corresponding to the first position can be used as the first posture of the robot, which can also reduce time consumption and improve execution efficiency.
[0126] The third image is an image obtained by shooting the target mark when the robot is in the first posture; the theoretical position (i.e., the first theoretical position and the second theoretical position) is the position of the robot calculated according to the position and posture of the camera when the offset and yaw angle are ignored. For details, please refer to the following Fig.11 Description.
[0127] The electronic device can determine the position and posture of the target identification in the camera coordinate system according to the third image, and obtain the first identification posture. Specifically, referring to the description in the above step S21, it can be known that the electronic device can determine the identification position based on the positions of multiple feature points of the target identification, and thus, the electronic device can also determine the orientation of the target identification based on the positions of multiple feature points of the target identification, and based on the determined identification position and orientation of the target identification, the position and posture of the target identification in the camera coordinate system can be obtained, and the first identification posture can be obtained.
[0128] After determining the first identification posture, the electronic device can determine the first theoretical position of the robot in the world coordinate system according to the first identification posture. Specifically, the electronic device can determine the first camera posture of the camera in the world coordinate system according to the first identification posture, and determine the first theoretical position according to the first camera posture and the conversion relationship between the camera coordinate system and the robot coordinate system. Among them, when there is a yaw angle, the yaw angle of the camera is also considered in the conversion relationship between the camera coordinate system and the robot coordinate system. Specifically, when there is a yaw angle, the role of the conversion relationship between the camera coordinate system and the robot coordinate system in determining the theoretical position can be understood as: first determine the posture of the camera without a yaw angle according to the yaw angle, and then determine the theoretical position of the robot corresponding to the camera posture without a yaw angle according to the conversion relationship between the camera coordinate system and the robot coordinate system under theoretical conditions. When there is no yaw angle, the value of the yaw angle can be regarded as 0 for processing.
[0129] It can be understood that the conversion relationship between the world coordinate system and the camera coordinate system can be determined based on the position and posture of the target identifier in the world coordinate system and the obtained position and posture of the target identifier in the camera coordinate system (i.e., the first identifier position and posture), and then the position and posture of the camera in the world coordinate system (i.e., the first camera position and posture) can be determined. Furthermore, the first theoretical position can be determined based on the position and posture of the camera in the world coordinate system and the conversion relationship between the camera coordinate system and the robot coordinate system.
[0130] like Fig. 9 As shown, w represents the world coordinate system, Tw represents the position and posture of the target marker in the world coordinate system; c represents the camera coordinate system, Tw represents the position and posture of the target marker in the world coordinate system; Tc represents the position and posture of the target marker in the camera coordinate system (i.e., the first marker posture); based on the corresponding relationship between Tw and Tc, the conversion relationship between w and c can be obtained, and then the position and posture of the camera in the world coordinate system (i.e., the first camera posture) can be obtained, and the first theoretical position can be determined based on the position and posture of the camera in the world coordinate system and the conversion relationship between the camera coordinate system and the robot coordinate system.
[0131] The following is a specific example to illustrate the method of determining the first theoretical position according to the first identification posture.
[0132] For example, taking w as the world coordinate system and taking the position of the target marker as the origin of the world coordinate system, the position of the camera in the world coordinate system (i.e., the first camera position) can be determined by the following formula (1), and the first theoretical position can be determined by the following formula (2):
[0133] T cw =(T wc ) -1 (1)
[0134] Among them, T wc is the position and posture of the target marker in the camera coordinate system determined according to the third image (ie, the first marker position and posture); T cw is the first camera pose of the camera in the world coordinate system; () -1 Indicates inversion.
[0135] T mw =T rc T cw (2)
[0136] Among them, T rc is the transformation relationship between the camera coordinate system and the robot coordinate system; T cw is the first camera position; T mw is the posture of the robot at the first theoretical position, and can also represent the first theoretical position of the robot.
[0137] In the above step S85, the second posture is obtained after the robot rotates in the first posture. Here, the second posture is obtained by rotating directly on the basis of the first posture. In the process of obtaining the second posture, the robot does not perform any movement other than rotation, that is, does not perform translational movement. Fig.10 The example shown illustrates how to obtain the second pose. Fig.10 The black solid line coordinate system Ra in the figure represents the coordinate system of the robot when it is in the first posture, and the gray solid line coordinate system Ca represents the coordinate system of the camera when the robot is in the first posture. The second posture is obtained by controlling the robot to rotate right, as shown in Fig.10 As shown, the black dotted coordinate system Rb represents the coordinate system of the robot when the robot is in the second posture, and the gray dotted coordinate system Cb represents the coordinate system of the camera when the robot is in the second posture.
[0138] In some embodiments, in order to ensure that the target mark is still within the field of view of the camera after rotation as much as possible, the direction of rotation of the robot can be determined according to the position of the target mark in the third image taken at the first posture, such as Fig.10 As shown, the target mark should be located on the right side in the third image. It can be seen that when rotating, it is more likely that the target mark can be recognized by controlling the robot to rotate to the right. Therefore, the robot can be controlled to rotate to the right.
[0139] The rotation angle of the robot can be a preset angle, such as 60°; or, the rotation angle of the robot can be determined according to actual conditions, for example, based on the position of the target mark in the image taken by the camera, with the goal of the target mark still being within the field of view of the camera after rotation, the direction and angle of the robot's rotation are determined; after the robot rotates, the rotation angle of the robot is determined in real time by a motion sensor provided in the robot; or, in some embodiments, the electronic device can control the robot to rotate 60° first, and determine whether the target mark is within the field of view of the camera after rotating 60°. If the target mark is still within the field of view of the camera after rotating 60°, If the target mark is within the field of view of the camera, the position of the robot after rotating 60° can be directly used as the second position; if the target mark is not within the field of view of the camera after rotating 60°, the robot can be appropriately controlled to rotate according to the position of the target mark in the third image taken in the first position, so that the target mark is located within the field of view of the camera again. For example, if the target mark is located in the middle right position in the third image, after the robot rotates 60° to the right, if the target mark is not within the field of view of the camera, the camera can be controlled to rotate to the left, and continuously determine whether the target mark is within the field of view of the camera until the target mark is within the field of view of the camera, and the current position can be regarded as the second position.
[0140] The method of determining the second theoretical position according to the second identification posture is the same as the principle of determining the first theoretical position according to the first identification posture in the above step S84. For details, please refer to the description in the above step S84.
[0141] For example, continuing to use w to represent the world coordinate system and taking the location of the target marker as the origin of the world coordinate system, the position of the camera in the world coordinate system (i.e., the second camera position) can be determined by the following formula (3), and the second theoretical position can be determined by the following formula (4):
[0142] T cw ’ =(T wc ’ ) -1 (3)
[0143] Among them, T wc ’ is the position and posture of the target marker in the camera coordinate system determined according to the fourth image (ie, the second marker position and posture); T cw ’ is the second camera pose in the world coordinate system, () -1 Indicates inversion.
[0144] T nw =T rc Tcw ’ (4)
[0145] Among them, T nw is the robot’s position at the second theoretical position, and can also represent the robot’s second theoretical position; rc is the transformation relationship between the camera coordinate system and the robot coordinate system.
[0146] In the above step S86, the electronic device can obtain the first offset of the camera according to the distance between the first theoretical position and the second theoretical position and the rotation angle from the first posture to the second posture. Fig.11 The principle of obtaining the first offset according to the first theoretical position, the second theoretical position, and the rotation angle from the first posture to the second posture is explained. For the convenience of description, this article takes the offset as the distance between the actual position of the camera and the theoretical position in the horizontal direction as an example for explanation, where the horizontal direction refers to the direction perpendicular to the robot's orientation, as mentioned above Figure 7 The direction pointed by the y-axis. For different robots, the definition rules of their camera offsets may be different, but the calculation method of their offsets is the same as the principle of this application.
[0147] Understandably, Fig.11 The camera coordinate system in is the coordinate system of the camera when there is no yaw angle and only offset, or, Fig.11 The camera coordinate system in can also be understood as the coordinate system of the camera when there is a yaw angle, but it has been determined based on the yaw angle. Fig.11 , Fig.11 In the figure, the coordinate system shown by the black solid line is the coordinate system of the robot when it is in the first posture, x1 indicates the orientation of the robot when it is in the first posture, and y1 is perpendicular to x1; the coordinate system shown by the gray solid line is the coordinate system of the camera when the robot is in the first posture, x'1 indicates the orientation of the camera when the robot is in the first posture, and y'1 is perpendicular to x'1. Control the robot to rotate at the first posture to obtain the second posture, Fig.11 In the figure, the coordinate system shown by the black dotted line is the coordinate system of the robot when it is in the second posture, x2 represents the orientation of the robot when it is in the second posture, and y2 is perpendicular to x2; the coordinate system shown by the gray dotted line is the coordinate system of the camera when the robot is in the second posture, x'2 represents the orientation of the camera when the robot is in the second posture, and y'2 is perpendicular to x'2.
[0148] It can be understood that since the robot only performs rotational motion and no translational motion when moving from the first posture to the second posture, the actual position of the robot in the first posture and the second posture is the same. Taking the position of the robot in the first posture as Ro as an example, we construct Ro as the origin to get Fig.11 The black solid line coordinate system in the figure; Rm is the intersection of the reverse extension line of the x'1 axis and the reverse extension line of the y1 axis, that is, the position of the robot in the theoretical case determined by the camera coordinate system corresponding to the first posture (that is, the first theoretical position); Rn is the intersection of the reverse extension line of the x'2 axis and the reverse extension line of the y2 axis, that is, the position of the robot in the theoretical case determined by the camera coordinate system corresponding to the second posture (that is, the second theoretical position).
[0149] Based on the above Fig.11 It can be seen that the three points Ro, Rm and Rn can form a triangle. The electronic device determines the rotation angle of the robot. For example, after determining ∠x1Rox2, since ∠x1RoRm is a right angle, ∠x2RoRm can be obtained based on ∠x1Rox2 and ∠x1RoRm, and then ∠RmRoRn can be obtained based on ∠x2RoRn being a right angle.
[0150] In addition, since the second pose is directly obtained by rotating on the first pose, and the x1 axis is parallel to the x'1 axis, and Rm is located on the reverse extension line of the y1 axis, the distance between Ro and Rm is the offset of the camera; similarly, the x2 axis is parallel to the x'2 axis, and Rn is located on the reverse extension line of the y2 axis, so the distance between Ro and Rn is also the offset of the camera, so the distance between Ro and Rm is equal to the distance between Ro and Rn, that is, RoRm = RoRn. Therefore, the triangle RoRmRn is an isosceles triangle.
[0151] Based on this, when the triangle RoRmRn is an isosceles triangle with known ∠RmRoRn, and when the coordinates of Rm (i.e., the first theoretical position) and the coordinates of Rn (i.e., the second theoretical position) are clear, the distance between Rm and Rn can be calculated. Therefore, based on the coordinates of Rm (i.e., the first theoretical position), the coordinates of Rn (i.e., the second theoretical position), and the rotation angle of the robot, the distance between Ro and Rn, or the distance between Ro and Rm, can be calculated, both of which can be used as the offset of the camera.
[0152] In some embodiments, the angle at which the robot rotates can be set to a preset angle in advance, so that when calculating the offset, ∠RmRoRn is always a fixed known quantity, and there is no need to use additional motion sensors to monitor the rotation angle of the robot in real time, which can reduce hardware resources and calculations. Alternatively, in some embodiments, the angle at which the robot rotates can be directly set to 60°, so that the triangle RoRmRn will be an equilateral triangle, and the distance between Rm and Rn can be directly calculated as RmRo or RoRn, without further calculating RmRo or RoRn based on RmRn, which reduces the complexity of the calculation and improves the efficiency of determining the offset.
[0153] In some embodiments, since the first offset is obtained by only one calibration, directly using the first offset as the offset of the camera may not be accurate enough. Therefore, the electronic device can repeatedly perform the above steps S81-S83 until a second preset number of times is reached to obtain multiple first offsets, and obtain the second offset of the camera by calculating the average of the multiple first offsets, and use the second offset as the offset of the camera.
[0154] For example, the electronic device can control the robot to repeat the process of moving from the first posture shown in step S84 to the second posture shown in step S85 and calculating the first offset, for example, as shown in step S84. Fig.11 As shown, the electronic device controls the robot to move from the posture shown in the black solid line coordinate system to the posture shown in the black dotted line coordinate system and calculates a first offset, then controls the robot to return to the posture shown in the black solid line coordinate system and move from the black solid line coordinate system to the posture shown in the black dotted line coordinate system. Fig.11 Move to the black dotted coordinate system in the manner shown, and then calculate a first offset. Repeat this rule until a second preset number of times is reached to obtain multiple first offsets, and then use the average of the multiple first offsets as the offset of the camera. In this way, the accuracy of the calibrated offset can be improved.
[0155] Alternatively, during repeated execution, the electronic device may only maintain the movement from the posture indicated by the black solid line to the posture indicated by the black dotted line, while the specific first posture and the second posture may change, for example, as described above. Fig.11 As shown, the electronic device controls the robot to move from the posture shown by the black solid line to the posture shown by the black dotted line and calculates a first yaw angle, and then controls the robot to rotate, translate, etc. to obtain a new first posture, and then the new first posture is adjusted according to Fig.11 Move to a new second position in the manner shown, and then calculate a first offset. Repeat this rule until a second preset number of times is reached to obtain multiple first offsets, and then use the average of the multiple first offsets as the offset of the camera. In this way, the accuracy of the calibrated offset can also be improved.
[0156] Among them, the second offset is actually a value used to represent the central tendency of multiple first offsets. Therefore, in addition to using the mean of multiple first offsets as the second offset, the second offset can also be calculated in other ways. For example, after obtaining multiple first offsets, the median or mode of these multiple first offsets can also be used as the second offset, or multiple first offsets can be clustered to determine isolated points, and then the mean or median of the first offsets other than the isolated points can be used as the second offset. The second preset number of times is the number of times that steps S84-S86 are pre-set to be repeated. The size of the second preset number of times can be set according to user needs, such as the size of the second preset number of times can be set to 10 times, 20 times, 30 times, etc.
[0157] It can be understood that, in the case of determining the yaw angle, the offset can be determined only according to steps S84 to S86, so the electronic device can first determine the yaw angle using steps S81 to S83, and then determine the offset based on the determined yaw angle using steps S84 to S86. That is, if multiple first offsets need to be calculated, it is not necessary to execute steps S81 to S83 before executing steps S84 to S86 each time. When the yaw angle has been determined according to steps S81 to S83, steps S84 to S86 can be directly repeated.
[0158] Next, through Fig.12 The robot camera calibration method provided in the embodiment of the present application is described again. It can be understood that: Fig.12 Only the process of determining the first yaw angle and the first offset is reflected, and the process of determining the second yaw angle and the second offset is not reflected. Fig.12 The following steps may be included:
[0159] Step S121: the robot moves to a first position;
[0160] The electronic device controls the movement of the robot and continuously determines whether the target mark is within the field of view of the camera. When it is determined that the target mark is within the field of view of the camera, the current position can be regarded as the first position. For details, please refer to the description in the above step S21.
[0161] Step S122: the camera captures a first image;
[0162] When the robot is located at the first position, the camera can capture an image with the target mark as the first image. For details, please refer to the description in the above step S21.
[0163] Step S123: the electronic device obtains a first identification position;
[0164] Based on the first image captured by the camera, the electronic device can determine the position of the target mark in the camera coordinate system as the first mark position. For details, please refer to the description in the above step S21.
[0165] Step S124: the robot moves to a second position;
[0166] The electronic device can control the robot to move linearly from the first position to the second position while keeping the orientation unchanged. For details, please refer to the description in the above step S22.
[0167] Step S125: the camera captures a second image;
[0168] When the robot is located at the second position, the camera can capture an image with the target mark as the second image. For details, please refer to the description in the above step S22.
[0169] Step S126: the electronic device obtains a second identification position;
[0170] Based on the second image captured by the camera, the electronic device can determine the position of the target mark in the camera coordinate system as the second mark position. For details, please refer to the description in the above step S22.
[0171] Step S127: the electronic device obtains a first yaw angle;
[0172] Based on the first identification position and the second identification position, the electronic device can obtain the first yaw angle of the camera. For details, refer to the description in the above step S23.
[0173] Step S128: the robot moves to the first posture;
[0174] The electronic device controls the movement of the robot and continuously determines whether the target mark is within the field of view of the camera. When it is determined that the target mark is within the field of view of the camera, the current posture can be regarded as the first posture. For details, please refer to the description in the above step S84.
[0175] Step S129: the camera captures a third image;
[0176] When the robot is in the first posture, the camera can capture an image with the target mark as the third image. For details, please refer to the description in the above step S84.
[0177] Step S1210: the electronic device obtains a first theoretical position;
[0178] Based on the third image captured by the camera, the electronic device can obtain the first theoretical position of the robot in the world coordinate system. For details, please refer to the description in the above step S84.
[0179] Step S1211: the robot moves to a second posture;
[0180] The electronic device can control the robot to perform rotational movement in the first posture without performing translational movement, and rotate to the second posture. For details, please refer to the description in the above step S85.
[0181] Step S1212: the camera captures a fourth image;
[0182] When the robot is in the second position, the camera can capture an image with the target mark as the fourth image. For details, please refer to the description in the above step S85.
[0183] Step S1213: the electronic device obtains a second theoretical position;
[0184] Based on the fourth image captured by the camera, the electronic device can determine the second theoretical position of the robot in the world coordinate system. For details, please refer to the description in the above step S85.
[0185] Step S1214: The electronic device obtains a first offset.
[0186] Based on the first theoretical position and the second theoretical position, the electronic device can determine a first offset of the camera, for details, refer to the description in the above step S86.
[0187] It can be understood that the robot sensor calibration method provided by the present application is described above by taking the camera as an example of an environmental sensor. The principle is the same as that of the case where the environmental sensor is a radar. The only difference is the target identification, the method for determining the identification position, and the method for determining the identification posture. For example, referring to the description in the above step S21, when the environmental sensor is a radar, the target identification is a feature that can reflect the radar signal emitted by the radar; for another example, when the environmental sensor is a radar, in the above step S21, the method for determining the identification position can be that the electronic device determines the distance between the radar and the target identification according to the time difference between the radar transmitting the signal wave and the receiving the reflected radar signal, and the position of the target identification can be obtained according to the distance between the radar and the target identification, and the direction and elevation angle of the radar wave emitted by the radar, and the position of the target identification can be determined based on the position of the target identification in the radar coordinate system to obtain the identification position. After determining the identification position, the electronic device can also determine the orientation of the target identification according to the strength of the reflected radar signal, and based on the determined identification position and target identification orientation, the posture of the target identification in the radar coordinate system can be obtained to obtain the identification posture.
[0188] Corresponding to the above robot sensor calibration method, the present application embodiment also provides a robot sensor calibration device, such as Fig.13 As shown, the device comprises:
[0189] The identification position determination module 131 is used to obtain the position of the target identification obtained by the environmental sensor identifying the target identification when the robot is in the first position in the environmental sensor coordinate system, and obtain the first identification position; obtain the position of the target identification obtained by the environmental sensor identifying the target identification when the robot is in the second position in the environmental sensor coordinate system, and obtain the second identification position; the second position is obtained by the robot moving in a straight line along the first position with the orientation unchanged; wherein the environmental sensor is arranged on the robot;
[0190] The yaw angle determination module 132 is used to obtain a first yaw angle of the environmental sensor according to the angle between the straight line formed by the first marker position and the second marker position and the axis of the environmental sensor coordinate system.
[0191] In the technical solution provided in the embodiment of the present application, it can be understood that when the robot keeps its orientation unchanged and moves in a straight line, the straight line formed by the first identification position and the second identification position should be parallel to the movement direction of the robot. Based on this, when the environmental sensor has a yaw angle (that is, there is an angle between the orientation of the environmental sensor and the orientation of the robot), if the robot moves in a straight line along its orientation, the straight line formed by the first identification position and the second identification position is parallel to the orientation of the robot. At this time, the angle between the straight line and the axis of the environmental sensor coordinate system is calculated, such as the angle between the straight line and the orientation of the environmental sensor, and the angle between the robot orientation and the orientation of the environmental sensor, that is, the yaw angle of the environmental sensor, can be obtained; if the robot keeps its orientation unchanged and moves in a straight line in other directions, the straight line formed by the first identification position and the second identification position is parallel to its movement direction. According to the angle between the straight line (that is, its movement direction) and the orientation of the robot, and the angle between the straight line and the axis of the environmental sensor coordinate system (such as the angle between the straight line and the orientation of the environmental sensor), the angle between the orientation of the environmental sensor and the orientation of the robot, that is, the yaw angle of the environmental sensor, can also be calculated.
[0192] Based on this, in an embodiment of the present application, when the robot is in the first position, the environmental sensor identifies the target identifier and obtains the position of the target identifier in the environmental sensor coordinate system, thereby obtaining the first identifier position; the robot is controlled to move in a straight line along the first position while keeping the orientation unchanged, thereby obtaining the second position; when the robot is in the second position, the environmental sensor identifies the target identifier and obtains the position of the target identifier in the environmental sensor coordinate system, thereby obtaining the second identifier position; the first yaw angle of the environmental sensor can be obtained based on the angle between the straight line formed by the first identifier position and the second identifier position and the axis of the environmental sensor coordinate system.
[0193] In some embodiments, the apparatus further comprises:
[0194] The identification posture determination module is used to obtain a first identification posture according to the posture of the target identification obtained by the environmental sensor identifying the target identification when the robot is in the first posture in the environmental sensor coordinate system; and determine the first theoretical position of the robot in the world coordinate system according to the first identification posture; obtain a second identification posture according to the posture of the target identification obtained by the environmental sensor identifying the target identification when the robot is in the second posture in the environmental sensor coordinate system; and determine the second theoretical position of the robot in the world coordinate system according to the second identification posture; wherein the second posture is obtained after the robot rotates in the first posture;
[0195] The offset determination module is used to obtain the first offset of the environmental sensor according to the distance between the first theoretical position and the second theoretical position and the rotation angle from the first posture to the second posture.
[0196] In some embodiments, the second posture is obtained by rotating the robot 60 degrees from the first posture, and the first offset of the environmental sensor is obtained according to the distance between the first theoretical position and the second theoretical position, and the angle of rotation from the first posture to the second posture, including:
[0197] The distance between the first theoretical position and the second theoretical position is calculated to obtain a first offset of the environmental sensor.
[0198] In some embodiments, determining a first theoretical position of the robot in a world coordinate system according to the first identified posture includes:
[0199] Determine a first environmental sensor posture of the environmental sensor in the world coordinate system according to the first identification posture; and obtain a first theoretical position according to a conversion relationship between the environmental sensor coordinate system and the robot coordinate system, and the first environmental sensor posture;
[0200] Determining a second theoretical position of the robot in the world coordinate system according to the second identification posture includes:
[0201] The second environment sensor posture of the environment sensor in the world coordinate system is determined according to the second identification posture; and the second theoretical position is obtained according to the conversion relationship between the environment sensor coordinate system and the robot coordinate system, and the second environment sensor posture.
[0202] In some embodiments, the yaw angle determination module 132 is further configured to:
[0203] Repeat the steps of obtaining the first identification position, obtaining the second identification position, and obtaining the first yaw angle until a first preset number of times is reached, thereby obtaining a plurality of first yaw angles;
[0204] An average value of the multiple first yaw angles is calculated as the second yaw angle of the environmental sensor.
[0205] In some embodiments, the offset determination module is further used to:
[0206] Repeat the steps of determining the first theoretical position, determining the second theoretical position, and obtaining the first offset of the environmental sensor for a second preset number of times to obtain a plurality of first offsets;
[0207] An average of the plurality of first offsets is calculated as a second offset of the environment sensor.
[0208] In some embodiments, the robot is a sweeping robot, and the target identifier is set as a QR code on the base station; wherein the identifier position is obtained by:
[0209] Determine the positions of the four corner points of the QR code from the image obtained by the camera shooting the target mark;
[0210] Determine the position of the QR code in the image based on the positions of the four corner points;
[0211] Project the position of the QR code in the image into the camera coordinate system to obtain the identification position;
[0212] or,
[0213] The robot is a sweeping robot, the environmental sensor is a radar, and the target identifier is a reflective sticker set on a base station; wherein the reflective sticker can reflect the radar signal emitted by the radar.
[0214] Corresponding to the above robot sensor calibration method, the embodiment of the present application also provides an electronic device, such as Fig.14 As shown, including:
[0215] A memory 141, for storing computer programs;
[0216] The processor 142 is used to implement any of the above-mentioned robot sensor calibration methods when executing the program stored in the memory 141.
[0217] In addition, the electronic device may further include a communication bus and / or a communication interface, and the processor 142, the communication interface, and the memory 141 communicate with each other via the communication bus.
[0218] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
[0219] The communication interface is used for communication between the above electronic device and other devices.
[0220] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.
[0221] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0222] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned robot sensor calibration methods are implemented.
[0223] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the robot sensor calibration method described in any one of the above embodiments.
[0224] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.
[0225] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0226] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for devices, electronic devices, storage media, and computer program products, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.
[0227] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.
Claims
1. A robot sensor calibration method, It is characterized in that The method comprises: Acquire the position of the target identifier in the environmental sensor coordinate system obtained by the environmental sensor recognizing the target identifier when the robot is in the first position, and obtain the first identifier position; wherein the environmental sensor is arranged on the robot; Acquire the position of the target identifier in the environmental sensor coordinate system obtained by the environmental sensor identifying the target identifier when the robot is in the second position, and obtain a second identifier position; the second position is obtained when the robot moves in a straight line along the first position with the orientation unchanged; A first yaw angle of the environmental sensor is obtained according to an angle between a straight line formed by the first marking position and the second marking position and an axis of the environmental sensor coordinate system.
2. The method according to claim 1, It is characterized in that The method further comprises: Obtain a first identification posture according to the posture of the target identifier in the environmental sensor coordinate system obtained by the environmental sensor identifying the target identifier when the robot is in the first posture; and determine a first theoretical position of the robot in the world coordinate system according to the first identification posture; According to the pose of the target identifier in the environmental sensor coordinate system obtained by the environmental sensor identifying the target identifier when the robot is in the second pose, a second identification pose is obtained; and according to the second identification pose, a second theoretical position of the robot in the world coordinate system is determined; wherein the second pose is obtained after the robot rotates in the first pose; A first offset of the environmental sensor is obtained according to a distance between the first theoretical position and the second theoretical position and a rotation angle from the first posture to the second posture.
3. The method according to claim 2, It is characterized in that The second posture is obtained by rotating the robot 60 degrees from the first posture, and the first offset of the environmental sensor is obtained according to the distance between the first theoretical position and the second theoretical position and the angle of rotation from the first posture to the second posture, including: The distance between the first theoretical position and the second theoretical position is calculated to obtain a first offset of the environmental sensor.
4. The method according to claim 2, It is characterized in that Determining a first theoretical position of the robot in a world coordinate system according to the first identification posture includes: Determine a first environmental sensor posture of the environmental sensor in a world coordinate system according to the first identification posture; and obtain the first theoretical position according to a conversion relationship between the environmental sensor coordinate system and the robot coordinate system, and the first environmental sensor posture; Determining a second theoretical position of the robot in a world coordinate system according to the second identification posture includes: Determine a second environmental sensor posture of the environmental sensor in the world coordinate system according to the second identification posture; and obtain the second theoretical position according to the conversion relationship between the environmental sensor coordinate system and the robot coordinate system, and the second environmental sensor posture.
5. The method according to claim 1, It is characterized in that The method further comprises: Repeat the steps of obtaining the first identification position, obtaining the second identification position, and obtaining the first yaw angle until a first preset number of times is reached, thereby obtaining a plurality of first yaw angles; An average value of the plurality of first yaw angles is calculated as a second yaw angle of the environmental sensor.
6. The method according to claim 2, It is characterized in that The method further comprises: Repeat the steps of determining the first theoretical position, determining the second theoretical position, and obtaining the first offset of the environmental sensor for a second preset number of times to obtain a plurality of first offsets; An average of the plurality of first offsets is calculated as a second offset of the environment sensor.
7. The method according to claim 1, It is characterized in that The robot is a sweeping robot, the environmental sensor is a camera, and the target identifier is a QR code set on a base station; wherein the identifier position is obtained by: Determine the positions of the four corner points of the two-dimensional code from the image obtained by the camera shooting the target identifier; Determine the position of the two-dimensional code in the image according to the positions of the four corner points; Projecting the position of the two-dimensional code in the image into the camera coordinate system to obtain the identification position; or, The robot is a sweeping robot, the environmental sensor is a radar, and the target identifier is a reflective sticker arranged on a base station; wherein the reflective sticker can reflect the radar signal emitted by the radar.
8. A robot sensor calibration device, It is characterized in that The device comprises: The identification position determination module is used to obtain the position of the target identification obtained by the environmental sensor of the robot identifying the target identification in the environmental sensor coordinate system at the first position, and obtain the first identification position; obtain the position of the target identification obtained by the environmental sensor identifying the target identification when the robot is at the second position, and obtain the second identification position; the second position is obtained by the robot moving in a straight line along the first position without changing its orientation; Yaw angle determination module: used to obtain a first yaw angle of the environmental sensor according to an angle between a straight line formed by the first marker position and the second marker position and an axis of the environmental sensor coordinate system.
9. An electronic device, It is characterized in that include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-7 when executing a program stored in a memory.
10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.
