Camera external parameter calibration method, self-moving device and program product

Abstract

The embodiment of the invention provides a camera external parameter calibration method, self-moving equipment and a program product, and the method comprises the steps: controlling the self-moving equipment to move according to preset movement track information, and determining the corresponding real-time poses of the self-moving equipment in a local coordinate system and a camera coordinate system in the movement process of the self-moving equipment, thereby achieving the calibration of the external parameters of a camera. And further generating corresponding moving tracks under the two coordinate systems. And on the basis, point cloud registration operation is carried out on the moving tracks under the two coordinate systems, so that the camera external parameters are determined according to the transformation matrix corresponding to the registration process, and then the camera external parameters are calibrated. Through the mode, automatic calibration of the camera external parameters of the self-moving equipment can be realized without the help of external auxiliary tools or environmental characteristics, manual intervention is not needed in the whole calibration process, and the calibration result is higher in accuracy.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a camera extrinsic parameter calibration method, a self-contained device, and a program product. Background Art

[0002] When a mobile device is moving, it usually needs to use a camera or other collection device to collect environmental image information for navigation and obstacle avoidance. In order to achieve precise navigation and obstacle avoidance, there are very high requirements for the accuracy of environmental information collected by the camera. Therefore, it is extremely important to ensure the accuracy of camera parameters.

[0003] At present, when calibrating the camera extrinsic parameters of a self-moving device, it is usually necessary to rely on auxiliary tools, specific environments, or calibrate each other's camera extrinsic parameters by cooperating with multiple self-moving devices. The entire calibration process is cumbersome, and the accuracy of the calibration parameter results is difficult to guarantee. Summary of the invention

[0004] Multiple aspects of the present application provide a camera extrinsic parameter calibration method, a self-mobile device, and a program product to improve the accuracy of the self-mobile device camera extrinsic parameter calibration and help the self-mobile device achieve precise navigation and obstacle avoidance.

[0005] An embodiment of the present application provides a camera extrinsic parameter calibration method for a self-moving device, comprising: obtaining preset trajectory information, and controlling the movement of the self-moving device according to the preset trajectory information; during the movement process, generating a corresponding movement trajectory in the local coordinate system of the self-moving device and a corresponding movement trajectory in the camera coordinate system; performing a registration operation on the two movement trajectories to obtain the camera extrinsic parameters of the self-moving device; and calibrating the camera parameters of the self-moving device according to the camera extrinsic parameters.

[0006] In an optional embodiment, obtaining preset trajectory information includes: reading a preset configuration file, and obtaining a plurality of preset feature point information from the preset configuration file; or obtaining an environment map, and identifying a plurality of preset feature point information from the environment map.

[0007] In an optional embodiment, an environmental map is obtained, and multiple preset feature point information is identified from the environmental map, including: obtaining and displaying the environmental map; and in response to a user's marking operation on the environmental map, obtaining multiple feature point information marked on the environmental map.

[0008] In an optional embodiment, before generating the corresponding movement trajectory in the local coordinate system of the self-moving device and the corresponding movement trajectory in the camera coordinate system, it also includes: acquiring the real-time positioning information of the self-moving device and the image information collected in real time by the camera; determining the posture information of the self-moving device in the local coordinate system based on the real-time positioning information; and determining the posture information of the self-moving device in the camera coordinate system based on the real-time collected image information.

[0009] In an optional embodiment, a movement trajectory corresponding to the local coordinate system of the self-moving device and a movement trajectory corresponding to the camera coordinate system are generated, including: generating a movement trajectory corresponding to the local coordinate system of the self-moving device and a movement trajectory corresponding to the camera coordinate system according to the posture information of the self-moving device respectively corresponding to the two coordinate systems.

[0010] In an optional embodiment, a registration operation is performed on the two moving trajectories to obtain the camera extrinsic parameters of the self-moving device, including: obtaining point cloud data corresponding to the two moving trajectories based on the real-time positioning information and the real-time collected image information; and performing a point cloud registration operation on the two moving trajectories based on the point cloud data to obtain a rotation matrix corresponding to the registration process.

[0011] In an optional embodiment, a point cloud registration operation is performed on the two moving trajectories according to the point cloud data, including: using a first registration algorithm and a second registration algorithm to perform a coarse registration operation and a fine registration point operation on the two moving trajectories respectively.

[0012] In an optional embodiment, the camera parameters of the self-moving device are calibrated according to the camera extrinsic parameters, including: converting the rotation matrix into Euler angles or four elements, and calibrating the camera parameters of the self-moving device according to the Euler angles or four elements.

[0013] An embodiment of the present application also provides a self-moving device, including a camera, a positioning system and a processor, wherein the camera is used to collect environmental images during the movement of the self-moving device, and the processor is used to: obtain preset trajectory information, and control the movement of the self-moving device according to the preset trajectory information; during the movement, generate a corresponding movement trajectory in the local coordinate system of the self-moving device and a corresponding movement trajectory in the camera coordinate system; perform a registration operation on the two movement trajectories to obtain the camera extrinsic parameters of the self-moving device; and calibrate the camera parameters of the self-moving device according to the camera extrinsic parameters.

[0014] In an optional embodiment, the processor is also used to: obtain real-time positioning information of the self-mobile device and image information collected by the camera in real time; determine the posture information of the self-mobile device in the camera coordinate system based on the real-time collected image information; and determine the posture information of the self-mobile device in the local coordinate system based on the real-time positioning information.

[0015] In an optional embodiment, when the processor generates the movement trajectory corresponding to the local coordinate system of the self-moving device and the movement trajectory corresponding to the camera coordinate system, it is used to: generate the movement trajectory corresponding to the local coordinate system of the self-moving device and the movement trajectory corresponding to the camera coordinate system according to the posture information corresponding to the self-moving device in the two coordinate systems respectively.

[0016] In an optional embodiment, when performing a registration operation on two moving trajectories, the processor is used to: obtain point cloud data corresponding to the two moving trajectories respectively according to the real-time positioning and real-time collected image information; and perform a point cloud registration operation on the two moving trajectories according to the point cloud data.

[0017] An embodiment of the present application also provides a computer program product, including a computer program / instruction. When the computer program / instruction is executed by a processor, the processor is enabled to implement each step in the above method embodiment.

[0018] In an embodiment of the present application, the self-moving device is controlled to move according to the preset moving trajectory information, and the real-time posture of the self-moving device in the local coordinate system and the camera coordinate system is determined during the movement of the self-moving device, thereby generating the corresponding moving trajectories in the two coordinate systems. Based on this, the moving trajectories in the two coordinate systems are subjected to point cloud registration operations to determine the camera extrinsic parameters according to the transformation matrix corresponding to the registration process, and then the camera extrinsic parameters are calibrated. In this way, the camera extrinsic parameters of the self-moving device can be automatically calibrated without the help of external auxiliary tools or environmental features. The entire calibration process does not require manual intervention, and the calibration result is more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0020] Figure 1 A flowchart of a method for calibrating camera extrinsic parameters of a mobile device provided in an embodiment of the present application;

[0021] Figure 2aA schematic diagram of point cloud data corresponding to a movement trajectory generated by a mobile device in a local coordinate system and a camera coordinate system respectively provided in an embodiment of the present application;

[0022] Figure 2b A schematic diagram corresponding to a certain moment in the process of performing point cloud registration on two moving trajectories of a mobile device in a local coordinate system and a camera coordinate system provided in an embodiment of the present application;

[0023] Figure 2c A schematic diagram of the corresponding registration results of two moving trajectories of a mobile device in a local coordinate system and a camera coordinate system after coarse registration of point clouds provided in an embodiment of the present application;

[0024] Figure 2d A schematic diagram of the corresponding registration results of two moving trajectories of a mobile device in a local coordinate system and a camera coordinate system after point cloud precise registration provided by an embodiment of the present application;

[0025] Figure 3 A structural diagram of the connection relationship between the functional components of the self-equipped device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0027] Usually, the camera external parameters of the self-moving device are calibrated before the self-moving device is put into use to improve the accuracy of the environment information collected by the self-moving device during subsequent operations. Several commonly used calibration methods are exemplified below.

[0028] In one calibration method, the self-mobile device is positioned by placing an auxiliary tool in the working environment of the self-mobile device, and then the camera extrinsic parameters are calibrated online based on the located position information. For example, a QR code or a specific target is placed in the working environment of the self-mobile device for the self-mobile device to scan during movement to obtain the corresponding position information, and then the camera extrinsic parameters of the self-mobile device are calculated based on the position information of the QR code or the specific target and the current position of the self-mobile device. By placing a QR code or a specific target in the working scene of the QR code or the specific target in advance, the camera extrinsic parameters of the self-mobile device can be calculated through the position of the pre-placed target, thereby realizing online calibration.

[0029] In another calibration method, by acquiring environmental information collected by the mobile device during movement, specific texture features are extracted from the acquired environmental information and matched with preset texture features to calculate the camera external parameters of the mobile device, thereby realizing online calibration.

[0030] In another calibration method, multiple self-moving devices obtain environmental information collected from an area, wherein the area contains specific feature points. Each self-moving device can extract feature information corresponding to the same feature point from the collected environmental information. Based on this, the feature information extracted by the multiple self-moving devices is matched, and the relative position relationship between each other can be calculated, thereby calculating the camera extrinsic parameters of the multiple self-moving devices.

[0031] It can be seen that the first calibration method requires the use of external auxiliary tools and pre-processing of the position information of the QR code or specific target, and the operation process is cumbersome. Although the second calibration method does not require the use of external auxiliary tools, it has high requirements for the operating environment of the self-moving device. For an operating environment with simple environmental information and no special texture, it is difficult to meet the camera external parameter calibration requirements. Moreover, for an operating environment that meets the calibration requirements, it is also necessary to pre-store specific texture information in the operating environment. For the third calibration method, not only multiple self-moving devices are required to cooperate with each other, but also certain requirements for the operating environment. It can be seen that no matter which method is used, the entire calibration process is relatively complicated, and it requires the use of external environmental information, and it is difficult to ensure the accuracy of the calibration results.

[0032] Therefore, in order to solve the problem that the calibration process of the camera extrinsic parameters of the self-mobile device is complicated and the calibration result is not accurate enough, the embodiment of the present application provides a camera extrinsic parameter calibration method for the self-mobile device, which can realize the automatic calibration of the camera extrinsic parameters of the self-mobile device without the help of external auxiliary tools or environmental features. The entire calibration process does not require manual intervention, and the calibration result is more accurate. The camera extrinsic parameter calibration method provided by the embodiment of the present application is described in detail below in conjunction with specific drawings.

[0033] Figure 1 A flowchart of a camera extrinsic parameter calibration method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method includes:

[0034] S1. Obtain preset trajectory information, and control the movement of the mobile device according to the preset trajectory information;

[0035] S2. During the movement, a movement trajectory corresponding to the local coordinate system of the mobile device and a movement trajectory corresponding to the camera coordinate system are generated;

[0036] S3, performing a registration operation on the two moving trajectories to obtain the camera extrinsic parameters of the mobile device;

[0037] S4. Calibrate the camera parameters of the mobile device according to the camera external parameters.

[0038] In an embodiment of the present application, the self-moving device can move according to preset trajectory information, and during the movement, generate a corresponding movement trajectory in the local coordinate system of the self-moving device and a corresponding movement trajectory in the camera coordinate system, and then perform a registration operation on the two movement trajectories to obtain the camera extrinsic parameters of the self-moving device, so as to calibrate the camera parameters of the self-moving device according to the obtained camera extrinsic parameters. Since the local coordinate system and the camera coordinate system of the self-moving device are not affected by the movement process of the self-moving device and the external environmental information, the movement trajectories generated in the two coordinate systems are accurate, and the camera extrinsic parameters obtained based on the registration of the two movement trajectories are more accurate. In addition, the entire camera extrinsic parameter calculation and calibration process is completed by the self-moving device itself, without the need for external auxiliary tools or environmental feature information, and the operation is simpler.

[0039] In the embodiments of the present application, the specific method of obtaining the preset trajectory information from the mobile device is not limited. According to the different methods of providing the preset trajectory information, the method of obtaining the preset trajectory information from the mobile device may also be different. Further, the specific form of the preset trajectory information is not limited. Optionally, the preset trajectory information may be a complete moving route, or feature point information corresponding to multiple preset feature points, wherein the continuous distribution trend between multiple preset feature points may constitute the moving route of the mobile device. Optionally, in order to improve the accuracy of the registration result, when setting multiple preset feature points corresponding to the preset trajectory information, the continuous distribution trend of the multiple preset feature points may be set to an asymmetric shape such as "S", "Z", "L", etc., so that when performing point cloud registration on the moving trajectory in the two coordinate systems, the effect is more intuitive and accurate. This example is explained by taking a similar "S" type as an example. Of course, it is not limited to this in actual applications.

[0040] In an optional embodiment, a corresponding preset configuration file can be set for the self-mobile device control program, and multiple preset feature point information can be preset in the preset configuration file. When the self-mobile device represents the external parameters of the camera, the preset configuration file can be read, and multiple preset feature point information can be obtained from the preset configuration file. According to the multiple preset feature point information, the preset trajectory information is determined. In another optional embodiment, multiple feature point information can also be pre-set in the environment map corresponding to the self-mobile device. When the self-mobile device represents the external parameters of the camera, the environment map can be obtained, and multiple preset feature point information can be identified from the environment map, and then the preset trajectory information can be determined according to the multiple preset feature point information.

[0041] It should be noted that the embodiments of the present application do not limit the storage location of the preset configuration file and the environment map, which can be stored locally on the mobile device or on an external server or cloud server that is connected to the mobile device, depending on the actual situation. It should be further noted that in the case of obtaining multiple preset feature point information from the environment map, the embodiments of the present application do not limit the method for determining the multiple preset feature point information.

[0042] In an optional embodiment, when generating an environmental map, multiple preset feature points can be directly determined in the environmental map and corresponding feature point information can be generated. In another optional embodiment, an editing operation can be provided for the environmental map. For example, the self-mobile device is provided with a display screen, which supports editing operations on the environmental map, and the user can trigger the editing operation on the display screen. Based on this, when the self-mobile device responds to the editing operation, the environmental map can be obtained from the local or cloud server and displayed on the display screen. Further, the user can select multiple preset feature points and perform marking operations for the displayed environmental map. Based on this, when the self-mobile device responds to the marking operation performed by the user on the environmental map, it can obtain multiple feature point information marked on the environmental map.

[0043] In an embodiment of the present application, in order to generate a moving trajectory corresponding to the local coordinate system of the mobile device and a moving trajectory corresponding to the camera coordinate system, it is necessary to determine the corresponding posture information of the mobile device in the local coordinate system and the corresponding posture information in the camera coordinate system during the movement. Optionally, when determining the posture information of the mobile device in its own coordinate system, the real-time positioning information of the mobile device can be obtained through the visual odometer system (Visual Odometry, VO) of the mobile device, and then, according to the real-time positioning information and the preset height corresponding to the center point of the mobile device, the posture information of the mobile device in the local coordinate system can be determined. When determining the posture information of the mobile device in the camera coordinate system, the image information corresponding to the surrounding environment during the movement can be collected in real time by the camera of the mobile device, and further, the posture information of the mobile device in the camera coordinate system can be determined according to the image information through the positioning and mapping system (Simultaneous Localization And Mapping, SLAM).

[0044] It should be noted that the embodiments of the present application do not limit the type of SLAM system of the self-mobile device. According to the different positioning and map construction requirements of the self-mobile device, the type of corresponding SLAM system may also be different. For example, if the self-mobile device has map reuse, loopback and relocation requirements during the positioning and map construction process, an ORB-SLAM2 type SLAM system can be used. For another example, if the self-mobile device needs to use a pinhole or fisheye lens model to implement functions such as vision, visual inertia or multi-map construction during the positioning and map construction process, an ORB-SLAM3 type SLAM system can be used. Accordingly, the embodiments of the present application do not limit the type of camera used by the self-mobile device. According to the specific use requirements and application scenarios, an adapted type of camera can be used. For example, the type of camera can be a monocular camera, a binocular camera, a depth camera, etc.

[0045] Based on the above, when the pose information in the local coordinate system and the camera coordinate system during the movement of the self-moving device is determined respectively, the corresponding movement trajectory in the local coordinate system of the self-moving device and the corresponding movement trajectory in the camera coordinate system can be generated according to the pose information corresponding to the self-moving device in the two coordinate systems. Further, the two movement trajectories can be registered to obtain the camera extrinsic parameters of the self-moving device. Optionally, when performing the registration operation on the two movement trajectories, the point cloud data corresponding to the two movement trajectories can be obtained according to the collected image information, and then the point cloud registration operation can be performed on the two movement trajectories according to the obtained point cloud data to obtain the rotation matrix corresponding to the registration process as the camera extrinsic parameters of the self-moving device.

[0046] It should be noted that the embodiment of the present application does not limit the specific method of performing the registration operation on the two moving trajectories. Optionally, the two moving trajectories can be registered only once, or the two moving trajectories can be coarsely registered first, and then finely registered to obtain a more accurate rotation matrix. Furthermore, the embodiment of the present application does not limit the specific algorithm type used for the registration operation on the two moving trajectories. According to specific actual needs, an adaptive registration algorithm can be used; accordingly, for the method of using a two-step registration operation, the relationship between the registration algorithm types used in the two registration operations is not limited. The two can be the same registration algorithm or different types of registration algorithms. For example, taking the example of using different types of registration algorithms for two registration operations, when performing a coarse registration operation on two moving trajectories, a first registration algorithm and a second registration algorithm can be used to perform coarse registration operations and fine registration point operations on the two moving trajectories respectively. Optionally, the first registration algorithm can be a fast point feature histogram algorithm (Fast Point Feature Histogram, FPFH), and the second registration algorithm can be an iterative closest point algorithm (Iterative Closest Point, ICP), but of course it is not limited to this.

[0047] Furthermore, when the rotation matrix corresponding to the registration process is obtained, the obtained rotation matrix can also be converted into Euler angles or four elements, so as to calibrate the external parameters of the camera of the self-moving device according to the converted Euler angles or four elements. Optionally, when the self-moving device is put into use for the first time, the camera does not have corresponding external parameters, then the converted Euler angles or four elements can be saved in the parameter calibration configuration file of the self-moving device to perform the first external parameter calibration of the camera of the self-moving device; or, when the corresponding default external parameters are available when the self-moving device leaves the factory, the converted Euler angles or four elements can be used to update the default external parameters.

[0048] Further optionally, when the self-moving device has been used for a period of time, there may be a deviation between the actual extrinsic parameters and the initial extrinsic parameters due to reasons such as equipment wear. Therefore, the self-moving device may also periodically perform the above operations to obtain the converted Euler angles or four elements according to the transformation matrix corresponding to the alignment process, and then use the converted Euler angles or four elements to update the historical extrinsic parameters to ensure accurate operation.

[0049] The following is an exemplary description of the execution process of the camera extrinsic parameter calibration method provided in the embodiment of the present application with reference to the accompanying drawings.

[0050] In the embodiment of the present application, after setting a plurality of preset feature points corresponding to the preset trajectory information, the self-moving device can be placed in a relatively spacious working area and started, and the camera external parameter calibration method is executed to obtain the preset trajectory information and control the self-moving device to move according to the preset trajectory information, and generate the movement trajectory information corresponding to the local coordinate system and the movement trajectory information corresponding to the camera coordinate system of the self-moving device. The specific method of how to set the plurality of preset feature points corresponding to the preset trajectory information can be referred to the description of the above embodiment, which will not be repeated here.

[0051] Figure 2a It is a schematic diagram of the point cloud data corresponding to the moving trajectory generated by the mobile device in the local coordinate system and the point cloud data corresponding to the moving trajectory generated in the camera coordinate system, such as Figure 2a As shown in the figure, since the moving trajectory of the mobile device in the local coordinate system and the moving trajectory in the camera coordinate system have different directions, in order to determine the camera extrinsic parameters, it is necessary to align the two moving trajectories, transform the two moving trajectories into the same coordinate system and overlap them, so as to obtain the corresponding rotation matrix in the alignment process as the camera extrinsic parameters.

[0052] Optionally, in order to distinguish the point cloud data corresponding to the two moving trajectories, the point cloud data corresponding to the moving trajectory of the self-moving device in the local coordinate system can be recorded as point cloud P, and the point cloud data corresponding to the moving trajectory in the camera coordinate system can be recorded as point cloud Q. Based on this, when performing point cloud registration operations on point cloud P and point cloud Q, n sampling points can be selected from point cloud P, and the FPFH algorithm can be used to find one or more similar points in point cloud Q that have similar FPFH features to the n sampling points in point cloud P. Further, for one or more similar points in point cloud Q corresponding to the n sampling points in point cloud P, a similar point is randomly selected as the corresponding point of point cloud P in point cloud Q. Further, the rigid body transformation matrix between the n sampling points in point cloud P and the corresponding points in point cloud Q is calculated respectively to complete the point cloud coarse registration operation between the two moving trajectories.

[0053] Optionally, in order to ensure that the n sampling points sampled in the point cloud P have different FPFH features as much as possible, the minimum distance threshold d between the sampling points can be set in advance, and when the n sampling points are selected, the distance between each two of the n sampling points is made greater than the minimum distance threshold d, wherein the specific value of the minimum distance threshold d is not limited in this example and can be determined according to actual needs.

[0054] Figure 2b In the process of point cloud registration of the two moving trajectories of the mobile device in the local coordinate system and the camera coordinate system, the two moving trajectories are rotated to the corresponding schematic diagram at a certain moment in the same coordinate system, as shown in FIG. Figure 2bAs shown in the figure, the relative position relationship between the two moving trajectories of the mobile device in the local coordinate system and the camera coordinate system is compared with Figure 2a The relative position relationship shown has changed to a certain extent. Figure 2c This is a schematic diagram of the corresponding registration results of the two moving trajectories of the mobile device in the local coordinate system and the camera coordinate system after the point cloud is roughly registered, as shown in Figure 2c As shown in the figure, after the rough registration of the point cloud, the two moving trajectories basically coincide in the same coordinate system. The rotation matrix obtained at this time can basically determine the camera external parameters of the self-moving device. However, in order to obtain more accurate results, it is necessary to further perform point cloud fine registration operations on the two moving trajectories based on the rough registration results.

[0055] Optionally, when performing the point cloud fine registration operation, the point cloud P after the point cloud coarse registration transformation can be recorded as point cloud P′, and point cloud P′ and point cloud Q can be used as the initial point set for point cloud fine registration. Based on this, the ICP algorithm can be used to find the nearest corresponding point Qi in point cloud Q for each point Pi in point cloud P′, which is used as the corresponding point of point Pi in point cloud Q to form an initial corresponding point pair. Since the point cloud P′ and point cloud Q after the above-mentioned point cloud coarse registration operation are not completely consistent, the corresponding relationship in each initial corresponding point pair is not correct, and the wrong corresponding relationship will affect the final registration result. Therefore, the wrong corresponding point pairs can be eliminated before the subsequent point cloud registration operation.

[0056] Optionally, a direction vector threshold can be used to eliminate erroneous corresponding point pairs. First, the normal vector of each point Pi and the corresponding point Qi and the normal vector angle between the two are calculated respectively, and then the corresponding point pairs with erroneous corresponding relationships are determined and eliminated based on the size of the normal vector angle between each corresponding point pair. Optionally, a preset angle threshold t can be set. If the normal vector angle between each corresponding point pair is less than the preset angle threshold t, the corresponding relationship between the corresponding point pairs is considered to be correct. Otherwise, it is considered to be wrong, and the corresponding point pairs with the wrong corresponding relationship are eliminated. The specific value of the preset threshold t is not limited in this example and can be determined according to actual needs.

[0057] Furthermore, when the corresponding point pairs with the correct corresponding relationship are obtained, the rotation matrix R and the translation vector T can be calculated according to the corresponding relationship between each corresponding point pair, and the corresponding point pair with the smallest mean square error can be determined, and the corresponding rotation matrix R and translation vector T can be used as the result of point cloud precise registration. Optionally, when determining the corresponding point pair with the smallest mean square error, an error threshold ε and a maximum number of iterations N can be set. First, the rigid body transformation matrix obtained in the previous point cloud rough registration process is applied to the point cloud P′ to obtain a new point cloud P″, and then the distance error between P″ and Q is calculated, and so on, until it is determined that the distance error is less than the threshold ε or the current number of iterations is greater than N, then the iteration ends.

[0058] Figure 2d is a schematic diagram of the relative position relationship between the two moving trajectories of the mobile device in the local coordinate system and the camera coordinate system after the above point cloud precise registration, as shown in Figure 2d As shown, compared to Figure 2c After the point cloud rough registration, the two moving trajectories basically overlap after the point cloud fine registration. Based on this, a more accurate rotation matrix between the two can be obtained as the camera extrinsic parameters that need to be determined in the end. Based on this, the final rotation matrix can be converted into Euler angles or four elements, and the camera extrinsic parameters of the self-moving device can be calibrated according to the converted Euler angles or four elements.

[0059] In an embodiment of the present application, the self-moving device is controlled to move according to the preset moving trajectory information, and the real-time posture of the self-moving device in the local coordinate system and the camera coordinate system is determined during the movement of the self-moving device, thereby generating the corresponding moving trajectories in the two coordinate systems. Based on this, the moving trajectories in the two coordinate systems are subjected to point cloud registration operations to determine the camera extrinsic parameters according to the transformation matrix corresponding to the registration process, and then the camera extrinsic parameters are calibrated. In this way, the camera extrinsic parameters of the self-moving device can be automatically calibrated without the help of external auxiliary tools or environmental features. The entire calibration process does not require manual intervention, and the calibration result is more accurate.

[0060] It should be noted that in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel, and the sequence numbers of the operations, such as S1, S2, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0061] In addition, the execution subject of each step of the method provided in the above embodiment may be the same device, or the method may be executed by different devices. For example, the execution subject of step S1 may be device A; for another example, the execution subject of step S1 may be device A, and the execution subject of steps S2 to S4 may be device B.

[0062] Based on the above, an embodiment of the present application also provides a self-moving device that can execute the above-mentioned camera external parameter calibration method, including a camera and a positioning system, wherein the camera is used to collect environmental images during the movement of the self-moving device, so as to determine the position and posture of the self-moving device in the camera coordinate system according to the collected environmental information; the positioning system is used to locate the position information of the self-moving device in real time, so as to determine the position and posture of the self-moving device in the local coordinate system.

[0063] In addition, self-mobile devices also include Figure 3 The main functional components shown, Figure 3 This is a schematic diagram of the structure of the connection relationship between the main functional components of the mobile device provided in the embodiment of the present application. Figure 3 As shown, the mobile device further includes: a processor 31 and a memory 32 storing a computer program; wherein the processor 31 and the memory 32 may be one or more.

[0064] The memory 32 is mainly used to store computer programs, which can be executed by the processor 31, so that the processor 31 controls the self-mobile device to realize corresponding functions, complete corresponding actions or tasks. In addition to storing computer programs, the memory 32 can also be configured to store various other data to support operations on the self-mobile device. Examples of such data include instructions for any application or method operating on the self-mobile device.

[0065] In the embodiment of the present application, the implementation form of the processor 31 is not limited, for example, it can be but not limited to a CPU, a GPU or an MCU. The processor 31 can be regarded as a control system of the self-mobile device, which can be used to execute the computer program stored in the memory 32 to control the self-mobile device to realize corresponding functions and complete corresponding actions or tasks. It is worth noting that, depending on the implementation form of the self-mobile device and the different scenarios in which it is located, the functions to be realized, the actions or tasks to be completed will be different; accordingly, the computer programs stored in the memory 32 will also be different, and the processor 31 can control the self-mobile device to realize different functions and complete different actions or tasks by executing different computer programs.

[0066] In some optional embodiments, such as Figure 3 As shown, the self-moving device may also include other components such as a display 33, a power supply component 34 and a communication component 35. The display 33 is used to display an environmental map corresponding to the working environment of the self-moving device. Figure 3 Only some components are shown schematically, which does not mean that the vehicle only includes Figure 3 The components shown in the figure can also include other components for different application requirements. For example, in the case of voice interaction requirements, Figure 3As shown, the self-moving device may further include an audio component 36. The components that the self-moving device may include may depend on the product form of the self-moving device, and are not limited here.

[0067] In the embodiment of the present application, when the processor 31 executes the computer program in the memory 32, it is used to: obtain preset trajectory information, and control the movement of the self-moving device according to the preset trajectory information; during the movement process, generate a corresponding movement trajectory in the local coordinate system of the self-moving device and a corresponding movement trajectory in the camera coordinate system; perform registration operations on the two movement trajectories to obtain the camera extrinsic parameters of the self-moving device; and calibrate the camera parameters of the self-moving device according to the camera extrinsic parameters.

[0068] In an optional embodiment, when acquiring preset trajectory information, the processor 31 is used to: read a preset configuration file, and acquire a plurality of preset feature point information from the preset configuration file; or acquire an environment map, and identify a plurality of preset feature point information from the environment map.

[0069] In an optional embodiment, when the processor 31 obtains the environment map and identifies multiple preset feature point information from the environment map, it is used to: obtain and display the environment map; and obtain multiple feature point information marked on the environment map in response to the user's marking operation on the environment map.

[0070] In an optional embodiment, before generating the corresponding movement trajectory in the local coordinate system of the self-mobile device and the corresponding movement trajectory in the camera coordinate system, the processor 31 is also used to: obtain the real-time positioning information of the self-mobile device and the image information collected by the camera in real time; determine the posture information of the self-mobile device in the local coordinate system based on the real-time positioning information; and determine the posture information of the self-mobile device in the camera coordinate system based on the image information collected in real time.

[0071] In an optional embodiment, when the processor 31 generates the movement trajectory corresponding to the local coordinate system of the self-mobile device and the movement trajectory corresponding to the camera coordinate system, it is used to: generate the movement trajectory corresponding to the local coordinate system of the self-mobile device and the movement trajectory corresponding to the camera coordinate system according to the posture information corresponding to the self-mobile device in the two coordinate systems respectively.

[0072] In an optional embodiment, when the processor 31 performs a registration operation on the two moving trajectories to obtain the camera external parameters of the self-moving device, it is used to: obtain point cloud data corresponding to the two moving trajectories respectively according to the image information; and perform a point cloud registration operation on the two moving trajectories according to the point cloud data to obtain a rotation matrix corresponding to the registration process.

[0073] In an optional embodiment, when the processor 31 performs a point cloud registration operation on two moving trajectories according to the point cloud data, it is used to: use a first registration algorithm and a second configuration algorithm to perform a coarse registration operation and a fine registration point operation on the two moving trajectories respectively.

[0074] In an optional embodiment, when calibrating the camera parameters of the self-mobile device according to the camera extrinsic parameters, the processor 31 is used to: convert the rotation matrix into Euler angles or four elements, and calibrate the camera parameters of the self-mobile device according to the Euler angles or four elements.

[0075] Accordingly, an embodiment of the present application further provides a computer program product, including a computer program / instruction. When the computer program / instruction is executed by a processor, each method step in the above method embodiment can be executed by a mobile device.

[0076] The memory in the above embodiments can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0077] The communication component in the above-mentioned embodiment is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0078] The display in the above-mentioned embodiment includes a screen, and the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0079] The power supply assembly in the above embodiments provides power to various components of the device where the power supply assembly is located. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply assembly is located.

[0080] The audio component in the above embodiments may be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal may be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal.

[0081] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0082] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0083] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0084] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0085] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0086] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0087] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0088] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0089] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A camera extrinsic calibration method for a mobile device, characterized in that: include: Acquire preset trajectory information, and control the movement of the mobile device according to the preset trajectory information; During the movement, a movement trajectory corresponding to the local coordinate system of the self-mobile device and a movement trajectory corresponding to the camera coordinate system are generated; Performing a registration operation on the two moving trajectories to obtain the camera extrinsic parameters of the self-moving device; The camera parameters of the mobile device are calibrated according to the camera external parameters.

2. The method according to claim 1, characterized in that: Get preset track information, including: Reading a preset configuration file, and obtaining a plurality of preset feature point information from the preset configuration file; Alternatively, an environment map is obtained, and a plurality of preset feature point information is identified from the environment map.

3. The method according to claim 2, characterized in that Acquiring an environment map, and identifying a plurality of preset feature point information from the environment map, including: Obtain and display environmental maps; In response to a marking operation of the user on the environment map, information of a plurality of feature points marked on the environment map is acquired.

4. The method according to any one of claims 1 to 3, characterized in that: Before generating the movement trajectory corresponding to the local coordinate system of the mobile device and the movement trajectory corresponding to the camera coordinate system, the method further includes: Acquire the real-time positioning information of the mobile device and the image information collected by the camera in real time; Determining the position information of the mobile device in the local coordinate system according to the real-time positioning information; and The position and posture information of the mobile device in the camera coordinate system is determined based on the real-time collected image information.

5. The method according to claim 4, characterized in that Generating a movement trajectory corresponding to the local coordinate system of the self-mobile device and a movement trajectory corresponding to the camera coordinate system, including: According to the position information corresponding to the self-moving device in the two coordinate systems, a movement trajectory corresponding to the local coordinate system of the self-moving device and a movement trajectory corresponding to the camera coordinate system are generated.

6. The method according to claim 5, characterized in that Performing a registration operation on the two moving trajectories to obtain the camera extrinsic parameters of the self-moving device includes: According to the real-time positioning information and the real-time collected image information, point cloud data corresponding to the two movement trajectories are respectively obtained; According to the point cloud data, a point cloud registration operation is performed on the two movement trajectories to obtain a rotation matrix corresponding to the registration process.

7. The method according to claim 6, characterized in that According to the point cloud data, a point cloud registration operation is performed on the two movement trajectories, including: The first registration algorithm and the second registration algorithm are used to perform a coarse registration operation and a fine registration point operation on the two movement trajectories respectively.

8. The method according to claim 6 or 7, characterized in that: Calibrate the camera parameters of the mobile device according to the camera external parameters, including: The rotation matrix is ​​converted into Euler angles or four elements, and the camera parameters of the self-moving device are calibrated according to the Euler angles or four elements.

9. A self-propelled device, characterized in that: The device comprises a camera, a positioning system and a processor, wherein the camera is used to collect environmental images during the movement of the mobile device, and the processor is used to: Acquire preset trajectory information, and control the movement of the mobile device according to the preset trajectory information; During the movement, a movement trajectory corresponding to the local coordinate system of the self-mobile device and a movement trajectory corresponding to the camera coordinate system are generated; Performing a registration operation on the two moving trajectories to obtain the camera extrinsic parameters of the self-moving device; The camera parameters of the mobile device are calibrated according to the camera external parameters.

10. The device according to claim 9, characterized in that The processor is further configured to: Acquire the real-time positioning information of the mobile device and the image information collected by the camera in real time; Determine the position and posture information of the mobile device in the camera coordinate system according to the real-time collected image information; According to the real-time positioning information, the position and posture information of the mobile device in the local coordinate system is determined.

11. The device according to claim 10, characterized in that When generating the movement trajectory corresponding to the local coordinate system of the mobile device and the movement trajectory corresponding to the camera coordinate system, the processor is used to: According to the posture information corresponding to the self-moving device in the two coordinate systems, a movement trajectory corresponding to the local coordinate system of the self-moving device and a movement trajectory corresponding to the camera coordinate system are generated.

12. The device according to claim 11, characterized in that When performing the registration operation on two movement trajectories, the processor is used to: According to the real-time positioning information and the real-time collected image information, point cloud data corresponding to the two movement trajectories are respectively obtained; A point cloud registration operation is performed on the two movement trajectories according to the point cloud data.

13. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images