Method and device for determining change of external parameters of camera and storage medium

Abstract

The invention discloses a method and device for determining external parameter changes of a camera and a storage medium, and relates to the technical field of visual inspection, and the method for determining the external parameter changes of the camera comprises the steps: collecting a preset calibration region through a depth camera of a robot, so as to obtain first point cloud data related to the preset calibration region, after a coordinate system of the first point cloud data is converted to a base coordinate system of the robot, a target plane is generated through fitting; and according to the target plane and a reference plane, external parameter changes between the robot and the depth camera are determined, and the reference plane is a plane obtained by collecting second point cloud data of the preset calibration area through the depth camera installed on the robot in a standard mode and fitting the second point cloud data under the base coordinate system. The technical problem of how to improve the accuracy and efficiency of determining the change of the external parameters of the camera is solved.

Description

Technical Field

[0001] The present application relates to the field of visual detection technology, and in particular to a method, device and storage medium for determining changes in camera extrinsic parameters. Background Art

[0002] With the rapid development of mobile robot technology, depth cameras, as important sensing devices for mobile robots, are widely used in environmental perception, target recognition, spatial positioning and other fields. However, in practical applications, due to the complexity of the environment in which the mobile robot is located and the errors of the depth camera itself, the extrinsic parameters of the depth camera will change. The extrinsic parameters are the parameters that describe the position and posture of the camera in the world coordinate system. Therefore, the accuracy of the camera's extrinsic parameters is crucial to the performance of the entire system. However, during long-term operation, due to mechanical shaking, wear, impact and other reasons, the installation position of the camera may change slightly, which will affect the change of the camera's extrinsic parameters and further affect the robot's operating accuracy.

[0003] At present, the external parameters of the depth camera can be determined by scanning a customized calibration plate to see if the camera's external parameters have changed. However, this method is limited to calibration before leaving the factory. After the machine enters hotels, restaurants and other places to work, asking customers to use the calibration plate regularly will greatly affect the user experience and is costly. Summary of the invention

[0004] The main purpose of this application is to provide a method, device and storage medium for determining changes in camera extrinsic parameters, aiming to solve the technical problem of how to improve the accuracy and efficiency of determining changes in camera extrinsic parameters.

[0005] To achieve the above object, the present application provides a method for determining a change in camera extrinsic parameters, the method comprising:

[0006] Capturing a preset calibration area through a depth camera of the robot to obtain first point cloud data related to the preset calibration area;

[0007] After converting the coordinate system of the first point cloud data into the base coordinate system of the robot, fitting and generating a target plane;

[0008] According to the target plane and the reference plane, the external parameter change between the robot and the depth camera is determined, wherein the reference plane is a plane fitted in a base coordinate system by collecting second point cloud data of a preset calibration area through a depth camera standardly installed on the robot.

[0009] In one embodiment, before the step of collecting a preset calibration area by a depth camera of the robot to obtain first point cloud data related to the preset calibration area, the step further includes:

[0010] Install the depth camera standard on the robot;

[0011] Capturing a preset calibration area through the depth camera to obtain second point cloud data related to the preset calibration area;

[0012] After the coordinate system of the second point cloud data is converted to the base coordinate system of the robot, the reference plane is generated by fitting.

[0013] In one embodiment, the step of fitting and generating a target plane after converting the coordinate system of the first point cloud data to the base coordinate system of the robot comprises:

[0014] Obtain the baseline external parameters of the depth camera standard installed on the robot;

[0015] According to the reference external parameters, the coordinate system of the first point cloud data is converted into a base coordinate system of the robot;

[0016] The plane fitting algorithm is used to convert the coordinate system into the third point cloud data of the base coordinate system, and the target plane is generated by fitting.

[0017] In one embodiment, the step of determining the change of external parameters between the robot and the depth camera according to the target plane and the reference plane includes:

[0018] Obtaining a first normal vector of the target plane and a second normal vector of the reference plane;

[0019] According to the difference between the first normal vector and the second normal vector, an extrinsic parameter change between the robot and the depth camera is determined.

[0020] In one embodiment, the method further comprises:

[0021] According to the time dimension, counting the long-term external parameter changes between the robot and the depth camera, and counting the short-term external parameter changes between the robot and the depth camera;

[0022] When the long-term extrinsic parameter change is greater than or equal to a first threshold, and / or when the short-term extrinsic parameter change is greater than or equal to a second threshold, it is determined that an abnormality exists in the robot and the depth camera.

[0023] In one embodiment, the method further comprises:

[0024] When it is determined that there is an abnormality in the robot and the depth camera, an abnormality alarm is issued.

[0025] In one embodiment, the method further comprises:

[0026] The external parameters between the robot and the depth camera are adjusted according to the long-term external parameter change amount and / or the short-term external parameter change amount.

[0027] In one embodiment, the method further comprises:

[0028] When it is determined that there is an abnormality in the robot and the depth camera, an abnormality alarm is issued in a first alarm mode at a preset time interval;

[0029] If no confirmation instruction based on the abnormal alarm is received, and when the long-term external parameter change is greater than or equal to the third threshold, and / or when the short-term external parameter change is greater than or equal to the fourth threshold, the operation is stopped and maintenance is performed.

[0030] In addition, to achieve the above-mentioned purpose, the present application also provides a device for determining changes in camera extrinsic parameters, wherein the device for determining changes in camera extrinsic parameters comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the device for determining changes in camera extrinsic parameters is used to execute the steps of the method for determining changes in camera extrinsic parameters as described above.

[0031] In addition, to achieve the above-mentioned purpose, the present application also provides a storage medium, which is a computer-readable storage medium, and a computer program is stored on the computer-readable storage medium. The computer program is executed by a processor to implement the steps of the method for determining the change of camera extrinsic parameters as described above.

[0032] The present application provides a method for determining changes in camera extrinsic parameters. The present application pre-stores a preset calibration area for determining extrinsic parameters in the robot. During the operation of the robot, the preset calibration area can be used to confirm whether the extrinsic parameters of the depth camera have changed. The depth camera of the robot is controlled to collect first point cloud data of the preset calibration area, and the coordinate system of the depth camera of the first point cloud data is converted to the base coordinate system of the robot. A plane is fitted based on the converted data to obtain a target plane, and then the target plane and the reference plane are combined to determine whether the extrinsic parameters between the robot and the depth camera have changed. The reference plane is the plane formed by fitting the second point cloud data of the preset calibration area collected when the depth camera on the robot is in the standard installation position in the base coordinate system of the robot, so as to improve the accuracy and efficiency of determining changes in camera extrinsic parameters.

[0033] In summary, the present application utilizes the point cloud data of the preset calibration area collected by the robot's depth camera to perform plane fitting, and then combines the reference plane to solve the external parameters of the target plane, that is, to determine whether the external parameters of the camera have changed. This overcomes the problem that during the use of the robot, it is necessary to regularly scan the customized calibration plate to detect the external parameters of the depth camera, which affects the user experience and is costly. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0035] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0036] Figure 1 A flowchart diagram of an embodiment of a method for determining camera extrinsic parameter changes provided by the present application;

[0037] Figure 2 Schematic diagram of the structure of the device for determining the change of camera extrinsic parameters in an embodiment of the present application.

[0038] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0039] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0040] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0041] The main solution of the present application is: a method for determining the change of camera extrinsic parameters, a robot is equipped with a depth camera, the method comprising: collecting a preset calibration area through the robot's depth camera to obtain first point cloud data related to the preset calibration area; converting the coordinate system of the first point cloud data to the robot's base coordinate system, and fitting to generate a target plane; determining the change of extrinsic parameters between the robot and the depth camera based on the target plane and the reference plane, wherein the reference plane is a plane fitted in the base coordinate system by collecting second point cloud data of the preset calibration area through the depth camera standardly installed on the robot.

[0042] This application uses the point cloud data of the preset calibration area collected by the robot's depth camera to perform plane fitting, and then combines the reference plane to solve the external parameters of the target plane. It can determine whether the external parameters of the camera have changed. This overcomes the problem of needing to regularly scan customized calibration plates to detect the external parameters of the robot and depth camera during the use of the robot, which affects the user experience and is costly.

[0043] It should be noted that the execution subject of this embodiment may be a robot control system, or a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a device that can achieve the above functions and determine the change of camera external parameters, etc., and this embodiment does not specifically limit this. The following takes the robot control system as the execution subject as an example to illustrate this embodiment and the following embodiments.

[0044] Based on this, the present application proposes a method for determining camera extrinsic parameter changes in the first embodiment, please refer to Figure 1 The method for determining the change of camera extrinsic parameters includes steps S10 to S30:

[0045] Step S10: collecting a preset calibration area through the robot's depth camera to obtain first point cloud data related to the preset calibration area.

[0046] It should be noted that during the use of the robot, a preset calibration area for external parameter calibration is determined in advance, and a mark is set for the calibration area. The mark is pre-stored in the robot. When the robot moves to the calibration area, it determines whether the calibration requirements are met based on the data collected by the depth camera and the lidar. After determining that the calibration requirements are met, the calibration begins. During calibration, the robot's depth camera collects an image of the preset calibration area, and the image is analyzed to obtain the first point cloud data of the preset calibration area.

[0047] In this embodiment, before the robot starts to be used, an image corresponding to a reference plane for extrinsic calibration of the depth camera is first acquired, an area corresponding to the image is determined as a preset calibration area, and the preset calibration area is marked. The marking confirmation method includes but is not limited to marking using a feature pattern in the reference plane, marking using a feature point in the reference plane, marking using an identifier in the reference plane, marking using a depth value of the reference plane, and the like.

[0048] The reference plane is a plane formed by collecting second point cloud data of a preset calibration area through a depth camera installed in a standard manner on the robot and fitting it in the base coordinate system of the robot.

[0049] In the hotel scenario, a more common solution is to calibrate at the door of the room. The high-frequency task is to deliver items. After the customer picks up the items, this area can be verified. The ground in front of the room is usually flat enough, and combined with the shape features of the door frame, it can also be used to calibrate all external parameters. This scenario also does not require additional settings for the calibration area and its identifier features. When the machine is stationary, the radar detects that the room door is closed, and then calibration can be performed.

[0050] Among them, the characteristic pattern in the reference plane can be a checkerboard, a dot array, etc. When confirming whether the image captured by the robot's depth camera is a preset calibration area, it can be determined by judging whether the image contains characteristic patterns such as a checkerboard, a dot array, etc.; the extraction of feature points in the reference plane can be realized by using image processing algorithms such as corner detection and edge detection, so as to complete the identification of the reference plane, and the feature points can be extracted by performing corner detection, edge detection, etc. on the image captured by the robot's depth camera, and the preset calibration area can be determined by judging whether there are feature points in the image with the reference plane, or by comparing the extracted feature point positions with the known feature point positions in the reference plane. If the two are consistent or nearly consistent, it can be determined that the area corresponding to the image captured by the depth camera is the preset calibration area; the identifier in the reference plane can be a QR code, a barcode, etc., and the image captured by the depth camera can be analyzed to determine whether there are identifiers such as QR codes and barcodes in the image to determine whether the area corresponding to the current image is the preset calibration area; it is also possible to obtain the depth information of the scene and determine whether the area corresponding to the current image is the preset calibration area by the depth values ​​of different areas in the image captured by the depth camera.

[0051] It should be noted that after confirming that the image currently captured by the depth camera is an image of the preset calibration area, a corresponding depth image is generated, and then the depth map is converted into point cloud data using the camera intrinsic parameters that are calibrated when the depth camera leaves the factory and built into the camera driver. That is, each pixel in the depth image is converted into a point in three-dimensional space, and its position in space is determined according to the depth value to obtain complete point cloud data.

[0052] It is easy to imagine that the process of acquiring point cloud data for the preset calibration area may be affected by factors such as viewing angle, lighting conditions, and reflective characteristics of the object surface. The acquired point cloud data can be filtered and processed to remove noise, smooth the surface, remove dynamic objects, etc., so that the collected data is more accurate.

[0053] It should be noted that point cloud data is a collection of a series of discrete points measured by a depth camera, and each point contains its coordinate information (X, Y, Z) in three-dimensional space.

[0054] Step S20, converting the coordinate system of the first point cloud data into the base coordinate system of the robot, and then fitting and generating a target plane;

[0055] It should be noted that since the first point cloud data collected by the depth camera is based on the camera's own coordinate system, and the operation and control of the robot is usually based on its base coordinate system. Therefore, it is necessary to convert the coordinate system of the first point cloud data collected from the camera coordinate system to the robot's base coordinate system. The plane fitting algorithm is used to perform plane fitting on the point cloud data collected by the depth camera to extract a target plane that best represents the distribution of these points.

[0056] After the target plane is fitted and generated, it needs to be combined with the reference plane to determine the changes in external parameters between the robot and the depth camera. The first point cloud data is the data collected by the depth camera based on the coordinate system of the depth camera itself. Therefore, in the process of fitting the first point cloud data collected by the depth camera to form the target plane, it is necessary to first convert the first point cloud data collected by the depth camera from the camera coordinate system to the base coordinate system of the robot to improve the positioning accuracy and data processing speed.

[0057] Among them, the plane fitting algorithm is used in the process of plane fitting according to the point cloud data. The plane fitting algorithm may include but is not limited to the least squares method, the random sampling consensus algorithm (RANSAC algorithm) and the Hough transform (Hough Transform). The least squares method for plane fitting is to minimize the square sum of the errors of the vertical distance from the point in the point cloud data to the target plane, so that the fitting object is closer and closer to the final object, thereby determining the target plane; RANSAC algorithm: The algorithm randomly selects three points in the point cloud, calculates the plane they form, and calculates the distance from all points to the plane to determine which points are inliers (i.e., points with a distance less than a certain threshold). Through multiple iterations, the algorithm selects the plane with the most inliers as the best target plane; Hough transform: Hough transform is a technique for mapping point cloud data to parameter space (such as plane normal vector and intercept). By converting each point in the point cloud to the parameter space, a cumulative vote of the parameter space can be formed to determine the most likely plane parameters. Generally, the RANSAC algorithm performs well for most scenarios. In practice, these methods can be combined to obtain a parameter solution, and some methods and indicators can be used to determine whether to accept the fitting result.

[0058] In one embodiment, the step S20, after converting the coordinate system of the first point cloud data to the base coordinate system of the robot, fitting and generating the target plane includes: step S201, obtaining the benchmark external parameters of the depth camera standard installed on the robot; step S202, according to the benchmark external parameters, converting the coordinate system of the first point cloud data to the base coordinate system of the robot; step S203, using a plane fitting algorithm, converting the coordinate system of the third point cloud data after converting it to the base coordinate system, and fitting and generating the target plane.

[0059] It should be noted that when the depth camera is installed on the robot in a standard manner, the relative position of the depth camera and the robot base is determined. Therefore, the reference extrinsic parameters between the coordinate system of the depth camera and the robot coordinate system can be calculated through the installation position of the depth camera on the robot. The reference extrinsic parameters include a rotation matrix and a translation matrix, that is, the reference extrinsic parameters represent the conversion between the camera coordinate system of the depth camera and the base coordinate system of the robot. Therefore, the coordinate system of the first point cloud data collected by the depth camera is converted to the base coordinate system of the robot using the reference extrinsic parameters, and then the point cloud data after conversion to the base coordinate system is fitted using a plane fitting algorithm to form a fitting plane.

[0060] The preset calibration area may be a wall or a ground. Assuming that the preset calibration area includes the wall and the ground, in the process of fitting the target plane to the first point cloud data collected by the depth camera, in order to ensure the accuracy of the data in the process of fitting the plane, the collection range of the first point cloud data may be increased to increase the amount of data, thereby avoiding the point cloud data collected not being in the same plane or too little data to cause the inability to calibrate when there are many debris. The point cloud data within the preset range is collected with the preset calibration plane as the dividing line, that is, the collected point cloud data includes the data of the preset calibration plane. After the point cloud data is collected, the collected point cloud data is segmented, and the collected point cloud data are all used as candidate points of the ground, and the area above this range is used as candidate points of the wall. In order to avoid the influence of the transition area between the wall and the ground, the points in the central area of ​​the candidate points on the ground are first selected as target candidate points, and then the point cloud data of the target candidate points are plane fitted using the plane fitting algorithm.

[0061] In a feasible implementation, the process of determining the target plane includes: step S2031, using a plane fitting algorithm to perform plane fitting on the point cloud data of the target candidate point to determine the parameters of the target plane; step S2032, calculating the distance between each point cloud data in the target candidate point and the target plane based on the parameters, and determining the corresponding average distance; step S2033, and calculating the standard deviation between each point cloud data in the target candidate point and the target plane based on the parameters; step S2034, taking the parameter corresponding to the average value being less than the distance threshold and the standard deviation being less than the standard deviation threshold as the target parameter, and taking the plane corresponding to the target parameter as the target plane.

[0062] The plane fitting algorithm is used to fit the point cloud data of the target candidate points to determine the parameters of the target plane including the normal vector and the intercept. Then, the target plane is expressed as ax+by+cz+d=0 using the parameters of the target plane. Each data point in the point cloud data is represented by three-dimensional data (x i ,y i , z i) is represented by, and the distance (i.e., error) and the corresponding average distance (i.e., average error) between each data point and the target plane are calculated, as well as the standard deviation between each data point and the target plane. The average distance and standard deviation can reflect the distance and discrete relationship between each data point and the target plane. Whether the target plane is flat is determined based on whether the average distance of each data point is less than the distance threshold and whether the standard deviation of each data point is less than the standard deviation threshold. The parameters of the plane corresponding to the minimum average distance in the average distance less than the distance threshold and the minimum standard deviation in the standard deviation less than the standard deviation threshold are taken as target parameters, and the plane corresponding to the target parameters is taken as the target plane. It should be noted that if both the average distance and the standard deviation are less than the preset threshold, it means that the planes corresponding to these target candidate points have good flatness, and the plane corresponding to the target parameters is taken as the target plane. Otherwise, this calibration is invalid and waits for the next calibration.

[0063] One feasible way is to first use the ground plane corresponding to the design value as the reference plane, select a small range up and down, and segment the point cloud collected by the depth camera. The points within this range are used as ground candidate points, and the points above this range are used as wall candidate points. For the ground candidate points, first select the points in the central area as target candidate points to avoid interference from the transition area between the ground and the wall. Specifically, it can be assumed that there is a plane, which is generally expressed as ax+by+cz+d=0, and the plane is fitted to obtain the parameters a, b, c, and d of plane A. For the points in plane A, due to measurement errors and calculation errors, these target candidate points will not be exactly in the plane, but the distance from plane A is very small. Therefore, for each point in the point cloud data, calculate its distance to the target plane A (that is, the error), which can be achieved through the point-to-plane distance formula.

[0064] Before determining the target plane, it is necessary to determine whether the number of candidate points of the plane reaches the threshold. In the process of determining the candidate points, the point cloud data corresponding to the debris in the calibration area collected may be excluded by calculating the average distance and standard deviation. Therefore, the point cloud data collected in the calibration area may not reach the threshold. Because there are too few candidate points in the target plane, the external parameters of the depth camera may have errors or calibration failures during the calibration process.

[0065] Step S30, determining the external parameter change between the robot and the depth camera according to the target plane and the reference plane, wherein the reference plane is a plane fitted in the base coordinate system by collecting the second point cloud data of the preset calibration area with the robot's depth camera through standard installation.

[0066] It should be noted that the reference plane is determined before the robot is used, and the reference plane is a plane formed by plane fitting by collecting the second point cloud data of the preset calibration area by the depth camera. After the reference plane is determined, the external parameter changes between the robot and the depth camera are determined according to the parameters of the target plane and the parameters of the reference plane.

[0067] In a feasible implementation, the method for determining the reference plane includes: installing a depth camera standard on the robot; capturing a preset calibration area through the depth camera to obtain second point cloud data related to the preset calibration area; and converting the coordinate system of the second point cloud data into the polar coordinates of the robot, and then fitting to generate the reference plane.

[0068] It should be noted that when the depth camera is installed in the standard installation position of the robot, the second point cloud data of the preset calibration area is collected by controlling the depth camera, and the second point cloud data is converted into the base coordinate system using the reference external parameters between the depth camera and the robot. Then, the second point cloud data is fitted with a plane fitting algorithm to form a reference plane.

[0069] The preset calibration areas include but are not limited to walls, floors, hotel doors, etc.

[0070] It should be noted that the method of converting the second point cloud data into the base coordinate system and fitting it to form a reference plane in the embodiment of the present application is the same as the method of converting the first point cloud data into the base coordinate system and fitting it to form a target plane, and will not be repeated here.

[0071] In a feasible implementation, step S30, determining the change of external parameters between the robot and the depth camera according to the target plane and the reference plane includes: step S301, obtaining the first normal vector of the target plane, and the second normal vector of the reference plane; step S302, determining the change of external parameters between the robot and the depth camera according to the first normal vector and the second normal vector.

[0072] It should be noted that after the target plane and the reference plane are determined, the target plane and the reference plane are expressed using plane equations, and the first normal vector of the target plane and the second normal vector of the reference plane are determined according to the plane equations, respectively. According to the change between the first normal vector and the second normal vector, the change of the external parameters between the robot and the depth camera is determined. It can be understood that if the position between the robot and the depth camera has not changed, the target plane formed by fitting the first point cloud data collected by the depth camera and the reference plane formed by fitting the second point cloud data should coincide, that is, the corresponding first normal vector and the second normal vector are equal. If the position between the robot and the depth camera changes, the first normal vector corresponding to the target plane will change. Therefore, the first normal vector and the second normal vector are used to determine whether the external parameters between the robot and the depth camera have changed.

[0073] The external parameters of the depth camera are the camera's position and posture, including the camera's rotation matrix and translation, including Euler angles and deviations from the x, y, and z axes of the machine center. The Euler angles include pitch angle, roll angle, and yaw angle. The normal vector of the target plane is determined based on the target plane determined by the fitting algorithm and its corresponding parameters. The normal vector of the reference plane is a unit vector. The rotation matrix and translation of the target plane relative to the reference plane are calculated by the first normal vector of the reference plane and the second normal vector of the target plane, thereby determining the external parameters of the depth camera.

[0074] For the first normal vector v1 (0, 0, 1) of the reference plane and the second normal vector v2 (a, b, c) of the target plane, as well as the rotation matrix between the first normal vector and the second normal vector, the corresponding quaternion can be calculated by the following method:

[0075] Normalize vectors: First make sure that both vectors v1 and v2 are unit vectors. If not, they need to be normalized first:

[0076]

[0077] Calculate the rotation axis: The rotation axis n can be calculated by the cross product of two vectors:

[0078] n=v′1×v′2

[0079] Calculate the rotation angle: The rotation angle θ can be calculated by taking the modulus of the dot product and cross product of two vectors:

[0080] cosθ=v′1*v′2

[0081] sinθ=||v′1×v′2||

[0082] Normalize the rotation axis: Make sure the rotation axis is a unit vector. If not, normalize it:

[0083]

[0084] Calculate quaternion: Based on the rotation axis and rotation angle, the quaternion q can be calculated by the following formula:

[0085]

[0086] in:

[0087] n′=(n′ x ,n′ y ,n′ z )

[0088] Through the above steps, we can calculate the quaternion representation of rotating vector v1 to vector v2. This quaternion represents a rotation with the rotation axis n' and the rotation angle θ.

[0089] According to the rotation order of zyx, solve the yaw angle, pitch angle and roll angle, and set the yaw angle to zero, because the yaw angle cannot be calibrated based on the ground, and the rotation matrix between the two normal vectors has no unique solution. The rotation matrix with the smallest rotation angle is calculated previously, but generally speaking, the roll angle and pitch angle are not 0, resulting in a, b, and c are not 0. The rotation vector is neither perpendicular nor parallel to the x-axis. If it is directly used as a calibration value, a yaw angle error will be introduced. Recalculate the quaternion after the yaw angle is set to zero.

[0090] The rotation matrix can be constructed by the yaw angle, pitch angle and roll angle. The installation height, that is, the z-axis translation relative to the center of the machine, can be calculated by h = d / sqrt(a^2+b^2+c^2).

[0091] With the roll angle roll, pitch angle pitch and installation height (i.e. z-axis translation), the point cloud can be intercepted according to the radar height and further calibrated with the radar. Since the radar is installed on the chassis, the installation parameters can be basically trusted, or no adjustment will be made after calibration at the factory, unlike the depth camera installed at a high place, which is easy to loosen and tilt. Because the hotel room door has obvious signs and fixed positions, the environment will not have too many dynamic changes, and will not cause too much interference to the calibration. Therefore, the hotel room door is a scene suitable for calibrating the remaining external parameters (yaw angle, x and y axis translation). The machine usually stays still here waiting for the user to open the door to pick up the object. The depth camera and radar have a common field of view and can be used for calibration. The depth camera point cloud intercepted at the radar height can force the z value (z-axis translation) to 0 and align with the radar on the xOy plane. Usually, the design values ​​of these three values ​​can be given according to the installation design values ​​of the machine, and then the iterative closest point (ICP) algorithm is used for iterative optimization to obtain the final result. If the mean square error is less than the error threshold, the depth camera is calibrated according to the calculated quaternion and the translation corresponding to the x, y, and z axes respectively.

[0092] Of course, during calibration, the machine's imu, odometer and other data must be used to ensure that the machine is stationary. Bumps during movement can easily lead to calibration errors.

[0093] It is easy to understand that in the process of calibrating the rotation matrix and translation of the depth camera, the external parameter error of the depth camera is monitored to determine whether the current robot can operate normally. The monitoring method may include step S401, according to the time dimension, counting the long-term external parameter changes between the robot and the depth camera, and counting the short-term external parameter changes between the robot and the depth camera; step S402, when the long-term external parameter change is greater than or equal to a first threshold, and / or when the short-term external parameter change is greater than or equal to a second threshold, it is determined that the robot and the depth camera are abnormal.

[0094] It should be noted that the external parameter changes between the robot and the depth camera are monitored, and the long-term external parameter changes and the short-term external parameter changes are counted, and a second threshold is set for the short-term external parameter changes. The short-term external parameter changes are monitored by the second threshold. When the external parameter changes are greater than or equal to the second threshold, the mounting bracket of the depth camera may be loose or deformed. A first threshold is set for the long-term external parameter changes, and whether the external parameter changes are greater than or equal to the first threshold is determined by long-term monitoring to determine whether there are abnormalities between the current robot and the depth camera. If the long-term external parameter changes are greater than or equal to the first threshold, the mounting bracket of the depth camera may be loose or deformed.

[0095] The short-term extrinsic parameter change is the monitoring of the change between the current extrinsic parameter calibration of the depth camera and the last extrinsic parameter calibration; the long-term extrinsic parameter change is affected by the short-term extrinsic parameter change. When the short-term extrinsic parameter change is less than the second threshold, the long-term extrinsic parameter change is monitored. If any external parameter change is greater than or equal to the second threshold during the monitoring process, it is determined that the current robot and depth camera have abnormalities. If there is no external parameter change greater than or equal to the second threshold, the external parameter change is continuously monitored to determine whether there is a situation where the external parameter change is greater than or equal to the first threshold. When the long-term external parameter change is greater than or equal to the first threshold, it is determined that the current robot and depth camera have abnormalities such as deformation or looseness.

[0096] According to the long-term external parameter changes and the short-term external parameter changes, when it is determined that the current robot and the depth camera have abnormalities, an abnormal alarm is issued to remind the management personnel to repair the robot and the depth camera.

[0097] A preset time period can be used for long-term external parameter changes. By monitoring and counting the external parameter changes within the preset time period, it can be determined whether there are any abnormalities in the robot and the depth camera. The preset monitoring time period may include but is not limited to 5 days, 10 days, 15 days, etc.

[0098] The first threshold value and the second threshold value described as an example in the embodiment of the present application are only for illustrating the principle, and the specific values ​​may be adjusted according to the actual situation.

[0099] It should be noted that when it is necessary to remind the management personnel, the alarm method can be determined based on the comparison results between the long-term external parameter change and the short-term external parameter change and the threshold, including the following steps: step S501, when it is determined that the robot and the depth camera have an abnormality, an abnormality alarm is issued at a preset time interval in the first alarm mode; step S502, when no confirmation instruction based on the abnormal alarm is received, and when the long-term external parameter change is greater than or equal to the third threshold, and / or when the short-term external parameter change is greater than or equal to the fourth threshold, the operation is stopped.

[0100] It should be noted that when the long-term external parameter change is greater than or equal to the first threshold or the short-term external parameter change is greater than or equal to the second threshold, it is determined that the current robot may have an abnormality, and an abnormal alarm is issued in accordance with the first alarm method at a preset time interval, and it is determined whether a confirmation instruction from the management personnel to respond to the abnormal alarm of the first alarm method is received within the preset time interval. If the confirmation instruction from the management personnel is not received, and when it is monitored that the long-term external parameter change is greater than or equal to the third threshold, and or the short-term external parameter change is greater than or equal to the fourth threshold, the robot stops running.

[0101] In a feasible implementation, the response time of the management personnel according to the abnormal alarm can be set. If the confirmation instruction of the response is not made within the response time, if the abnormal alarm is caused by the long-term external parameter change being greater than or equal to the first threshold, the difference between the long-term external parameter change and the first threshold is calculated. If the abnormal alarm is caused by the short-term external parameter change being greater than or equal to the second threshold, the difference between the short-term external parameter change and the second threshold is calculated. According to the size of the difference, the second alarm mode can be selected for abnormal alarm. The alarm intensity of the second alarm mode is greater than the alarm intensity of the first alarm mode. In a feasible implementation, after calculating the difference between the external parameter change and the corresponding threshold, different processing methods can also be selected according to the size of the difference. The processing methods include but are not limited to continuing to make abnormal alarms in the first alarm mode, making abnormal alarms in the second alarm mode, directly stopping operation, etc.

[0102] Among them, the first alarm method can be a message, vibration, ringing, etc. on the administrator's terminal device, and the second alarm method can be a text message prompt, ringing vibration, accelerated vibration, shortened interval ringing, etc. on the administrator's terminal device. The first alarm method and the second alarm method can both use one or more of the above methods to alarm during the alarm process.

[0103] For example, the preset time interval is 1 minute. After the alarm is issued to the management personnel in the first alarm mode, no instructions from the management personnel are received within one minute. At this time, the management personnel may not be around or the equipment used for the first alarm mode may fail. In this case, the second alarm mode is confirmed or the operation is stopped directly according to the difference between the external parameter change and the corresponding threshold. The larger the difference between the external parameter change and the corresponding threshold, the stronger the alarm level corresponding to the second alarm mode, and vice versa.

[0104] In this embodiment, a preset calibration area is formed by setting marks on the working area, and the point cloud data of the preset calibration area is captured by the depth camera periodically or when passing through the preset calibration area. The wall and the ground are identified by plane fitting, and the external parameters are solved, so that the system can monitor and adaptively adjust the external parameters in real time. The system tracks the accumulated errors of long-term external parameter changes and the jitter of short-term external parameter changes, and intelligently determines whether the external parameter changes exceed the corresponding threshold value, which is used to determine whether the mounting bracket of the depth camera is deformed or loose. Through real-time monitoring and adaptive adjustment, it is ensured that the external parameters are always kept in the best state, thereby improving the accuracy and stability of the robot operation, reducing the need for manual calibration, and reducing maintenance costs and operating costs. The system can respond and adjust immediately when changes occur, reducing operational errors and downtime caused by inaccurate external parameters.

[0105] It should be noted that the above examples are only used to assist in understanding the present application and do not constitute a limitation on the method of determining changes in camera extrinsic parameters in the present application. More forms of simple transformations based on this technical concept are all within the scope of protection of the present application.

[0106] The present application provides a device for determining changes in camera extrinsic parameters, and the device for determining changes in camera extrinsic parameters includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for determining changes in camera extrinsic parameters in the above-mentioned embodiment one.

[0107] Reference below Figure 2, which shows a schematic diagram of the structure of a device suitable for implementing the embodiment of the present application to determine the change of camera extrinsic parameters. The device for determining the change of camera extrinsic parameters in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 2 The device for determining the change of camera extrinsic parameters shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0108] like Figure 2 As shown, the device for determining the change of camera extrinsic parameters may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the device for determining the change of camera extrinsic parameters are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems may be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 may allow the device for determining changes in camera extrinsic parameters to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a device for determining changes in camera extrinsic parameters with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have instead.

[0109] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0110] The present application provides a device for determining changes in camera extrinsic parameters, which uses a method for determining changes in camera extrinsic parameters in the above embodiment to solve the technical problem of how to improve the accuracy and efficiency of robot work. Compared with the prior art, the beneficial effects of the device for determining changes in camera extrinsic parameters provided by the present application are the same as the beneficial effects of the method for determining changes in camera extrinsic parameters provided by the above embodiment, and the other technical features of the device for determining changes in camera extrinsic parameters are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0111] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0112] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0113] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, the computer-readable program instructions being used to execute a method for determining changes in camera extrinsic parameters in the above-mentioned embodiment.

[0114] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0115] The computer-readable storage medium may be included in a device for determining changes in camera extrinsic parameters; or may exist independently without being assembled into a device for determining changes in camera extrinsic parameters.

[0116] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0117] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned method for determining changes in camera extrinsic parameters, and can solve the technical problem of how to improve the accuracy and efficiency of robot work. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the method for determining changes in camera extrinsic parameters provided by the above-mentioned embodiment, and will not be repeated here.

[0118] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent processing scope of the present application.

Claims

1. A method for determining changes in camera extrinsic parameters, wherein a robot is equipped with a depth camera, characterized in that: The method comprises: Capturing a preset calibration area through a depth camera of the robot to obtain first point cloud data related to the preset calibration area; After converting the coordinate system of the first point cloud data into the base coordinate system of the robot, fitting and generating a target plane; According to the target plane and the reference plane, the external parameter change between the robot and the depth camera is determined, wherein the reference plane is a plane fitted in a base coordinate system by collecting second point cloud data of a preset calibration area through a depth camera standardly installed on the robot.

2. The method according to claim 1, characterized in that Before the step of collecting the preset calibration area by the robot's depth camera to obtain the first point cloud data related to the preset calibration area, the method further includes: Install the depth camera standard on the robot; Capturing a preset calibration area through the depth camera to obtain second point cloud data related to the preset calibration area; After the coordinate system of the second point cloud data is converted to the base coordinate system of the robot, the reference plane is generated by fitting.

3. The method according to claim 1, characterized in that The step of converting the coordinate system of the first point cloud data into the base coordinate system of the robot and then fitting and generating the target plane comprises: Obtain the baseline external parameters of the depth camera standard installed on the robot; According to the reference external parameters, the coordinate system of the first point cloud data is converted into a base coordinate system of the robot; The plane fitting algorithm is used to convert the coordinate system into the third point cloud data of the base coordinate system, and the target plane is generated by fitting.

4. The method according to claim 1, characterized in that The step of determining the change of external parameters between the robot and the depth camera according to the target plane and the reference plane comprises: Obtaining a first normal vector of the target plane and a second normal vector of the reference plane; An extrinsic parameter change between the robot and the depth camera is determined according to the first normal vector and the second normal vector.

5. The method according to claim 1, characterized in that The method further comprises: According to the time dimension, counting the long-term external parameter changes between the robot and the depth camera, and counting the short-term external parameter changes between the robot and the depth camera; When the long-term extrinsic parameter change is greater than or equal to a first threshold, and / or when the short-term extrinsic parameter change is greater than or equal to a second threshold, it is determined that an abnormality exists in the robot and the depth camera.

6. The method according to claim 5, characterized in that The method further comprises: When it is determined that there is an abnormality in the robot and the depth camera, an abnormality alarm is issued.

7. The method according to claim 5, characterized in that The method further comprises: The external parameters between the robot and the depth camera are adjusted according to the long-term external parameter change amount and / or the short-term external parameter change amount.

8. The method according to claim 5, characterized in that The method further comprises: When it is determined that there is an abnormality in the robot and the depth camera, an abnormality alarm is issued in a first alarm mode at a preset time interval; If no confirmation instruction based on the abnormal alarm is received, and when the long-term external parameter change is greater than or equal to the third threshold, and / or when the short-term external parameter change is greater than or equal to the fourth threshold, the operation is stopped and maintenance is performed.

9. A device for determining a change in camera extrinsic parameters, characterized in that: The device for determining changes in camera extrinsic parameters comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the method for determining changes in camera extrinsic parameters as described in any one of claims 1 to 8.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, the steps of the method for determining the change of camera extrinsic parameters as described in any one of claims 1 to 8 are implemented.

Images