Method for calibrating external parameters between sensors and related device

Abstract

The invention provides an external parameter calibration method between sensors and a related device, which are used for improving the accuracy of external parameter calibration, the method can be applied to the field of automatic driving, and the method comprises the following steps: in response to a target object input coordinate range, presenting target point cloud data of a target calibration object collected by a laser radar in a point cloud rendering interface; in response to a point selection operation triggered by the target object for the target point cloud data, constructing a calibration object coordinate system based on the selected reference points; obtaining a first position of each feature point in the target calibration object in a calibration object coordinate system based on the image collected by the camera; on the basis of the first positions, in combination with the relative poses between the calibration object coordinate system and the laser radar coordinate system, second positions of the feature points in the laser radar coordinate system are obtained; and based on each second position, obtaining a relative pose between the laser radar coordinate system and the camera coordinate system.

Description

Background Art

[0002] The extrinsic calibration between sensors refers to obtaining the relative pose between the coordinate systems corresponding to two sensors respectively, and the relative pose includes the relative position information and relative attitude information between the two coordinate systems.

[0003] In the related art, the relative pose between two coordinate systems is determined by matching the feature points of the target calibration object under the two coordinate systems. However, in the case where the sensors are lidar and camera, due to the problem of sparse data in the point cloud data collected by the lidar, it is difficult to accurately match the feature points in the camera coordinate system to the feature points in the reference coordinate system, resulting in low calibration accuracy. Summary of the Invention

[0004] The embodiments of the present application provide an extrinsic calibration method between sensors and related devices to improve the accuracy of the extrinsic calibration between lidar and camera.

[0005] In a first aspect, the embodiments of the present application provide an extrinsic calibration method between sensors, including:

[0006] In response to the coordinate range in the lidar coordinate system input by the target object, present the target point cloud data corresponding to the coordinate range in the point cloud rendering interface; wherein, the coordinate range represents the position information of the target calibration object, and the target calibration object includes each feature point;

[0007] In response to the point selection operation triggered by the target object for the target point cloud data, construct a calibration object coordinate system in the lidar coordinate system based on each selected reference point;

[0008] Based on the image of the target calibration object collected by the camera, obtain the first position of each feature point in the calibration object coordinate system;

[0009] Based on each first position, and in combination with the relative pose between the calibration object coordinate system and the lidar coordinate system, obtain the second position of each feature point in the lidar coordinate system;

[0010] Based on each second position, obtain the relative pose between the lidar coordinate system and the camera coordinate system.

[0011] In a second aspect, the embodiments of the present application provide an extrinsic calibration device between sensors, including:

[0012] A range selection unit, configured to present the target point cloud data corresponding to the coordinate range in the point cloud rendering interface in response to the coordinate range in the lidar coordinate system input by the target object; wherein, the coordinate range represents the position information of the target calibration object, and the target calibration object includes each feature point;

[0013] A point cloud selection unit, configured to, in response to a point selection operation triggered by the target object on the target point cloud data, construct a calibration object coordinate system in the lidar coordinate system based on each selected reference point.

[0014] A first conversion unit, configured to obtain a first position of each feature point in the calibration object coordinate system based on an image of the target calibration object collected by a camera.

[0015] A second conversion unit, configured to obtain a second position of each feature point in the lidar coordinate system based on each first position and in combination with a relative pose between the calibration object coordinate system and the lidar coordinate system.

[0016] A third conversion unit, configured to obtain a relative pose between the lidar coordinate system and the camera coordinate system based on each second position.

[0017] As a possible implementation manner, when constructing the calibration object coordinate system in the lidar coordinate system in response to the point selection operation triggered by the target object on the target point cloud data and based on each selected reference point, the point cloud selection unit is specifically configured to:

[0018] In response to an origin selection operation triggered by the target object on the target point cloud data, use a selected reference point as the origin of the calibration object coordinate system.

[0019] In response to an axis selection operation triggered by the target object on the target point cloud data, obtain relative pose information of the target calibration board relative to the origin based on two selected reference points, and construct the calibration object coordinate system in combination with the origin based on the relative pose information.

[0020] As a possible implementation manner, if the target calibration object is a rectangle, the one reference point is one of the four corner points of the rectangle, and the two reference points are located on two sides adjacent to the one corner point in the rectangle.

[0021] As a possible implementation manner, when obtaining the first position of each feature point in the calibration object coordinate system based on the image of the target calibration object collected by the camera, the first conversion unit is specifically configured to:

[0022] Obtain the image of the target calibration object collected by the camera, perform feature point detection on the image, and obtain the image position of each feature point in the image.

[0023] Based on the respective image positions of the feature points, and in combination with the image positions of the reference points in the image, obtain the relative position relationships between the feature points and the reference points respectively;

[0024] Based on the relative position relationships between the feature points and the reference points respectively, obtain the first positions of the feature points in the calibration object coordinate system.

[0025] As a possible implementation, among the reference points, there is included: a reference point for indicating the origin of the calibration object coordinate system;

[0026] When obtaining the relative position relationships between the feature points and the reference points respectively based on the respective image positions of the feature points and in combination with the image positions of the reference points in the image, the first conversion unit is specifically configured to:

[0027] Based on the respective image positions of the feature points, and in combination with the image position of the one reference point in the image, obtain the relative position relationships between the feature points and the one reference point respectively;

[0028] When obtaining the first positions of the feature points in the calibration object coordinate system based on the relative position relationships between the feature points and the reference points respectively, the first conversion unit is specifically configured to:

[0029] Based on the relative position relationships between the feature points and the one reference point respectively, obtain the first positions of the feature points in the calibration object coordinate system.

[0030] As a possible implementation, based on the respective second positions, obtain the relative pose between the lidar coordinate system and the camera coordinate system. The third conversion unit is specifically configured to:

[0031] Based on the image of the target calibration object collected by the camera, obtain the third positions of the feature points in the camera coordinate system respectively;

[0032] Based on the respective second positions and in combination with the respective third positions, obtain the relative pose between the lidar coordinate system and the camera coordinate system.

[0033] As a possible implementation, before presenting the target point cloud data corresponding to the coordinate range in the point cloud rendering interface in response to the coordinate range in the lidar coordinate system input by the target object, the range selection unit is further configured to:

[0034] Present a point cloud rendering interface, where the point cloud rendering interface includes: initial point cloud data of the target acquisition scene collected by the lidar, and the target acquisition scene includes a target calibration object;

[0035] When presenting the target point cloud data corresponding to the coordinate range in the lidar coordinate system in response to the input of the target object in the point cloud rendering interface, the range selection unit is specifically configured to:

[0036] In response to the coordinate range in the lidar coordinate system input by the target object, present the target point cloud data corresponding to the coordinate range in the initial point cloud data in the point cloud rendering interface.

[0037] As a possible implementation manner, the third conversion unit is further configured to:

[0038] Based on the obtained relative pose between the lidar coordinate system and the camera coordinate system, and combining the second positions of the respective feature points in the lidar coordinate system, obtain the fourth positions of the respective feature points in the camera coordinate system;

[0039] Based on the third positions of the respective feature points in the camera coordinate system, and combining the fourth positions of the respective feature points in the camera coordinate system, obtain the reprojection error;

[0040] Based on the reprojection error, obtain the new relative pose between the lidar coordinate system and the camera coordinate system.

[0041] In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory. Among them, the memory stores a computer program, and when the computer program is executed by the processor, the processor is caused to execute the steps of the above method.

[0042] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which includes a computer program. When the computer program runs on an electronic device, the computer program is used to cause the electronic device to execute the steps of the method in any of the above aspects.

[0043] In a fifth aspect, an embodiment of the present application provides a computer program product. The program product includes a computer program, the computer program is stored in a computer-readable storage medium, and a processor of an electronic device reads and executes the computer program from the computer-readable storage medium, so that the electronic device executes the steps of the method in any of the above aspects.

[0044] In the embodiments of the present application, in the point cloud rendering interface, the target point cloud data corresponding to the position information of the target calibration object input by the target object is presented. Through the position information of the target calibration object, the point cloud data corresponding to the target calibration object can be accurately located, and the noise points existing at the edge of the target calibration object can be reduced, thereby improving the calibration accuracy. Then, in response to the point selection operation triggered by the target object for the target point cloud data, based on the selected reference points, a calibration object coordinate system is constructed in the lidar coordinate system. Compared with the feature point matching of the point cloud data and the image by the target object, the number of reference points selected for constructing the calibration object coordinate system is less than the number of feature points. Therefore, the dependence on the target object can be reduced to a certain extent, thereby improving the stability and accuracy of camera calibration. Further, based on the image of the target calibration object collected by the camera, the first position of each feature point in the calibration object coordinate system is obtained, and based on each first position, combined with the relative pose between the calibration object coordinate system and the lidar coordinate system, the second position of each feature point in the lidar coordinate system is obtained, and based on each second position, the relative pose between the lidar coordinate system and the camera coordinate system is obtained. By converting each feature point from the calibration object coordinate system to the lidar coordinate system and using the positions of the feature points in the lidar coordinate system for the external parameter calibration between sensors, the calibration accuracy is improved.

[0045] Other features and advantages of the present application will be described in the following specification, and some of them will become obvious from the specification or be understood by implementing the present application. The objectives and other advantages of the present application can be achieved and obtained by the structures specifically pointed out in the written specification, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The drawings described herein are used to provide a further understanding of the present application and form a part of the present application. The schematic embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation to the present application. In the drawings:

[0047] Figure 1 It is a schematic diagram of an application scenario provided in the embodiments of the present application;

[0048] Figure 2 It is a schematic flowchart of a method for external parameter calibration between sensors provided in the embodiments of the present application;

[0049] Figure 3 It is a schematic diagram of a target calibration object provided in the embodiments of the present application;

[0050] Figure 4 It is a schematic diagram of a target acquisition scenario provided in the embodiments of the present application;

[0051] Figure 5Schematic diagram of an initial point cloud data provided in an embodiment of the present application;

[0052] Figure 6 Schematic diagram of a target point cloud data provided in an embodiment of the present application;

[0053] Figure 7 Schematic diagram of a target point cloud data provided in an embodiment of the present application;

[0054] Figure 8 Schematic diagram of a reference point provided in an embodiment of the present application;

[0055] Figure 9 Schematic diagram of a feature point in a lidar coordinate system provided in an embodiment of the present application;

[0056] Figure 10 Schematic diagram of an external parameter calibration process between sensors in an AR scenario provided in an embodiment of the present application;

[0057] Figure 11 Schematic diagram of the structure of an external parameter calibration device between sensors provided in an embodiment of the present application;

[0058] Figure 12 Schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Detailed implementation manners

[0059] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, rather than all, of the embodiments of the technical solutions of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments recorded in this application document without creative efforts belong to the scope protected by the technical solutions of the present application.

[0060] In the embodiments of the present application, the term "module" or "unit" refers to a computer program with a predetermined function or a part of a computer program, which works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as a processing circuit or a memory), or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of the overall module or unit that includes the function of that module or unit.

[0061] It is understandable that in the specific embodiments of the present application, relevant data such as point cloud data and images are involved. When the above embodiments of the present application are applied to specific products or technologies, user permission or consent needs to be obtained, and the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of relevant countries and regions.

[0062] The Intelligent Traffic System (ITS), also known as the Intelligent Transportation System, effectively integrates advanced scientific and technological means (information technology, computer technology, data communication technology, sensor technology, electronic control technology, automatic control theory, operations research, artificial intelligence, etc.) into transportation, service control, and vehicle manufacturing, strengthening the connection among vehicles, roads, and users, thereby forming an integrated transportation system that ensures safety, improves efficiency, enhances the environment, and saves energy.

[0063] The Intelligent Vehicle Infrastructure Cooperative Systems (IVICS, vehicle-road cooperation system) is a development direction of the Intelligent Traffic System (ITS). The vehicle-road cooperation system uses advanced wireless communication and new-generation Internet technologies to comprehensively implement dynamic real-time information interaction between vehicles and vehicles, and between vehicles and roads. Based on the collection and fusion of all-time and all-space dynamic traffic information, it conducts active vehicle safety control and road cooperation management, fully realizing the effective cooperation among people, vehicles, and roads, ensuring traffic safety, improving traffic efficiency, and thus forming a safe, efficient, and environmentally friendly road traffic system.

[0064] In related technologies, the extrinsic calibration between a lidar and a camera is to determine the relative pose between the camera coordinate system and the lidar coordinate system by matching the feature points of the target calibration object in the camera coordinate system and the lidar coordinate system. However, due to the problem of sparse data in the point cloud data collected by the lidar, it is difficult to accurately match the feature points in the camera coordinate system to the feature points in the reference coordinate system, resulting in low calibration accuracy.

[0065] In the embodiment of the present application, in the point cloud rendering interface, the target point cloud data corresponding to the position information of the target calibration object input by the target object is presented. Through the position information of the target calibration object, the point cloud data corresponding to the target calibration object can be accurately located, and the noise points existing at the edge of the target calibration object are reduced, thereby improving the calibration accuracy. Then, in response to the point selection operation triggered by the target object for the target point cloud data, based on the selected reference points, a calibration object coordinate system is constructed in the lidar coordinate system. Compared with the feature point matching of the point cloud data and the image by the target object, the number of reference points selected for constructing the calibration object coordinate system is less than the number of feature points. Therefore, the dependence on the target object can be reduced to a certain extent, thereby improving the stability and accuracy of camera calibration. Further, based on the image of the target calibration object collected by the camera, the first position of each feature point in the calibration object coordinate system is obtained, and based on each first position, combined with the relative pose between the calibration object coordinate system and the lidar coordinate system, the second position of each feature point in the lidar coordinate system is obtained, and based on each second position, the relative pose between the lidar coordinate system and the camera coordinate system is obtained. By converting each feature point from the calibration object coordinate system to the lidar coordinate system and using the positions of the feature points in the lidar coordinate system for external parameter calibration between sensors, the calibration accuracy is improved.

[0066] Refer to Figure 1 As shown, it is a schematic diagram of an application scenario provided in the embodiment of the present application. This application scenario includes: a device to be calibrated 110 and a calibration processing device 120. The number of devices to be calibrated 110 can be one or more. The number of calibration processing devices 120 can also be one or more. The present application does not make a specific limitation on the number of devices to be calibrated 110 and calibration processing devices 120. The device to be calibrated 110 and the calibration processing device 120 can be directly or indirectly connected through wired or wireless communication methods, and the present application does not limit this here.

[0067] In the embodiment of the present application, a camera and a lidar are deployed in the device to be calibrated 110. Among them, the camera is used to collect images of the target acquisition scene, and the lidar is used to collect point cloud data of the target acquisition scene. The device to be calibrated 110 can be a vehicle, a virtual reality (VR) device, an augmented reality (AR) device, a mixed reality (MR) device, or an electronic device integrating the three technologies of VR, AR, and MR, but is not limited thereto.

[0068] The calibration processing device 120 is a device for performing external parameter calibration between sensors. In the embodiment of the present application, the calibration processing device 120 also has a human-computer interaction function.

[0069] As a possible implementation, the calibration processing device 120 may be, but is not limited to, terminal devices such as smart phones, tablet computers, laptop computers, desktop computers, smart speakers, smart watches, Internet of Things devices, smart home appliances, in-vehicle terminals, etc. A client related to point cloud rendering may be installed on the terminal device, and the client may be software (such as a browser, a map software, etc.), or a web page, a small program, etc.

[0070] As another possible implementation, the calibration processing device 120 may also be a server corresponding to the client. The server may be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers. It may also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms.

[0071] The terminal device and the server may be directly or indirectly connected through wired or wireless communication methods, and this application does not make any restrictions here.

[0072] In the embodiments of this application, the method for calibrating the external parameters between sensors is implemented independently by the terminal device or the server, or can be jointly implemented by the terminal device and the server. As an example, in response to the coordinate range in the lidar coordinate system input by the target object, the terminal device presents the target point cloud data corresponding to the coordinate range in the point cloud rendering interface; wherein, the coordinate range represents the position information of the target calibration object, and the target calibration object includes each feature point; in response to the point selection operation triggered by the target object for the target point cloud data, a calibration object coordinate system is constructed in the lidar coordinate system based on the selected reference points; based on the image of the target calibration object collected by the camera, the first position of each feature point in the calibration object coordinate system is obtained; based on each first position, combined with the relative pose between the calibration object coordinate system and the lidar coordinate system, the second position of each feature point in the lidar coordinate system is obtained; based on each second position, the relative pose between the lidar coordinate system and the camera coordinate system is obtained.

[0073] As another example, in response to a coordinate range in the lidar coordinate system input by a target object, the terminal device presents target point cloud data corresponding to the coordinate range in the point cloud rendering interface, where the coordinate range represents the position information of a target calibration object, and the target calibration object includes each feature point. Then, in response to a point selection operation triggered by the target object for the target point cloud data, the terminal device obtains each reference point selected by the target object. After that, the terminal device sends each reference point selected by the target object to the server. Based on each selected reference point, the server constructs a calibration object coordinate system in the lidar coordinate system, and based on an image of the target calibration object collected by a camera, obtains the first position of each feature point in the calibration object coordinate system, and based on each first position, combines the relative pose between the calibration object coordinate system and the lidar coordinate system to obtain the second position of each feature point in the lidar coordinate system. Furthermore, based on each second position, the relative pose between the lidar coordinate system and the camera coordinate system is obtained.

[0074] The embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, intelligent transportation, assisted driving, autonomous driving, AR, VR, etc.

[0075] Refer to Figure 2 As shown in the figure, it is a schematic flowchart of a method for calibrating the external parameters between sensors provided in the embodiments of the present application. This process can be applied to a calibration processing device, and the specific process is as follows:

[0076] S201. In response to a coordinate range in the lidar coordinate system input by a target object, present target point cloud data corresponding to the coordinate range in the point cloud rendering interface; where the coordinate range represents the position information of a target calibration object, and the target calibration object includes each feature point.

[0077] In the embodiments of the present application, the target calibration object can be a calibration object distributed at preset positions in the target acquisition scene. The calibration object includes a QR code pattern or a checkerboard, and the checkerboard is usually composed of black and white squares. The feature point can refer to the key point of the target calibration object. For example, the feature point can be the corner point of the target calibration object, and the corner point refers to the intersection point formed by two sides of a corner of a square on the checkerboard.

[0078] Refer to Figure 3 As shown in the figure, it is a schematic diagram of a target calibration object provided in the embodiments of the present application. The target calibration object is composed of four QR code patterns (abbreviated as QR codes) and five circular holes staggered. Each QR code has four corner points of 0, 1, 2, and 3. In this article, the four corner points of the four QR codes are all used as feature points. That is to say, the target calibration object includes 16 feature points.

[0079] As a possible implementation, a target calibration object is set in the target acquisition scenario. The device to be calibrated equipped with a lidar and a camera performs data acquisition in the target acquisition scenario. The acquired data includes: the initial point cloud data of the target acquisition scenario collected by the lidar, and the images of the target acquisition scenario collected by the camera. After the data acquisition is completed, the vehicle to be calibrated can upload the acquired data to the server or store it locally. In this way, when it is necessary to perform external parameter calibration between the sensors on the device to be calibrated, the calibration processing device obtains the initial point cloud data of the target acquisition scenario collected by the lidar and the images of the target acquisition scenario collected by the camera through the server or the device to be calibrated, and then performs external parameter calibration between the sensors based on the initial point cloud data and the images.

[0080] Taking the autonomous driving scenario as an example, refer to Figure 4 As shown, it is a schematic diagram of a target acquisition scenario provided in an embodiment of the present application. The device to be calibrated is a vehicle, and the vehicle travels in the target acquisition scenario to perform data acquisition. A Figure 3 As shown target calibration object is set in the target acquisition scenario. Through acquisition, the following can be obtained: the initial point cloud data of the target acquisition scenario collected by the lidar, and the images of the target acquisition scenario collected by the camera. Refer to Figure 5 As shown, black dots are used to represent the point cloud data. The initial point cloud data contains Figure 4 As shown, the point cloud data corresponding to the target calibration object. The point cloud data corresponding to the target calibration object is the point cloud in the areas corresponding to two nearly circular holes. Obviously, due to the problem of sparse data in the point cloud, the shape of the point cloud data corresponding to the target calibration object is close to the true shape of the target calibration object, but there are certain differences.

[0081] As a possible implementation, before executing S201, the calibration processing device can also present a point cloud rendering interface. The point cloud rendering interface includes: the initial point cloud data of the target acquisition scenario collected by the lidar, and the target acquisition scenario includes a target calibration object. Correspondingly, in response to the coordinate range in the lidar coordinate system input by the target object, the target point cloud data corresponding to the coordinate range is presented in the point cloud rendering interface, specifically including:

[0082] The calibration processing device presents the target point cloud data corresponding to the coordinate range in the initial point cloud data in the point cloud rendering interface in response to the coordinate range in the lidar coordinate system input by the target object.

[0083] In the embodiment of the present application, after the calibration processing device obtains the initial point cloud data of the target acquisition scenario collected by the lidar, it renders the initial point cloud data, thereby presenting the initial point cloud data to the target object. In this way, the target object can select the target point cloud data for viewing in the initial point cloud data.

[0084] As a possible implementation, in the point cloud rendering interface, there is also a range input component for inputting the coordinate range of the target calibration object. When the target object triggers the range input component, the calibration processing device responds to the coordinate range in the lidar coordinate system input by the target object through the range input component and presents the target point cloud data corresponding to the coordinate range in the point cloud rendering interface.

[0085] Considering that in three-dimensional space, the target calibration object has three-dimensional coordinates, in order to facilitate the input of the coordinate range of the target calibration object, the range input component may specifically include range input components corresponding to each of the three axes: the range input component specifically includes an x-axis range input component, a y-axis range input component, and a z-axis range input component. Among them, the x-axis range input component is used to input the coordinate range of the target calibration object on the x-axis, the y-axis range input component is used to input the coordinate range of the target calibration object on the y-axis, and the z-axis range input component is used to input the coordinate range of the target calibration object on the z-axis.

[0086] Refer to Figure 6 as shown, refer to Figure 6 as shown, which is a schematic diagram of the initial point cloud data in a point cloud rendering interface provided in an embodiment of the present application. The initial point cloud data is composed of each three-dimensional point. The initial point cloud data includes the point cloud data of the target calibration object and also includes the point cloud data of objects such as houses, trees, and fences involved in the target acquisition scene.

[0087] In the left area of the point cloud rendering interface, there is a setting component, and the setting component includes a range input component. The range input component specifically includes an x-axis range input component, a y-axis range input component, and a z-axis range input component. The Xrange parameter in the x-axis range input component is used to configure the range threshold value along the positive x-axis direction, and the -Xrange parameter in the x-axis range input component is used to configure the range threshold value along the negative x-axis direction. The Yrange parameter in the y-axis range input component is used to configure the range threshold value along the positive y-axis direction, and the -Yrange parameter in the y-axis range input component is used to configure the range threshold value along the negative y-axis direction. The Zrange parameter in the z-axis range input component is used to configure the range threshold value along the positive y-axis direction, and the -Zrange parameter in the z-axis range input component is used to configure the range threshold value along the negative y-axis direction. When the target object configures the Xrange parameter, -Xrange parameter, Yrange parameter, -Yrange parameter, Zrange parameter, and -Zrange parameter to 15, -15, 15, -15, 1.5, and -1.5 respectively, it means that the coordinate range of the target calibration object on the x-axis is: (-15, 15), the coordinate range on the y-axis is: (-15, 15), and the coordinate range on the z-axis is: (-1.5, 1.5).

[0088] Refer to Figure 7 As shown, it is a schematic diagram of a target point cloud data provided in an embodiment of the present application. The calibration processing device responds to the coordinate range in the lidar coordinate system input by the target object through the range input component: the coordinate range on the x-axis is: (-15, 15), the coordinate range on the y-axis is: (-15, 15), and the coordinate range on the z-axis is: (-1.5, 1.5). In the point cloud rendering interface, the target point cloud data corresponding to this coordinate range is presented.

[0089] By continuously adjusting the coordinate range in the range input component, the point cloud data of the target calibration object can be accurately presented in the point cloud rendering interface, reducing the influence of noise points on the calibration result and improving the calibration accuracy.

[0090] To facilitate the target object to adjust the coordinate range, Figure 6 The point cloud rendering interface shown may also include a reset component (Reset). When the target object triggers the reset component, the values of the Xrange parameter, -Xrange parameter, Yrange parameter, -Yrange parameter, Zrange parameter, and -Zrange parameter can be reset.

[0091] S202. In response to the point selection operation triggered by the target object for the target point cloud data, based on the selected reference points, a calibration object coordinate system is constructed in the lidar coordinate system.

[0092] As a possible implementation manner, in the embodiment of the present application, when executing S202, the following manner can be adopted but is not limited to:

[0093] The calibration processing device responds to the origin selection operation triggered by the target object for the target point cloud data, and takes a selected reference point as the origin of the calibration object coordinate system;

[0094] The calibration processing device responds to the axial selection operation triggered by the target object for the target point cloud data, obtains the relative pose information of the target calibration board relative to the origin based on the selected two reference points, and constructs the calibration object coordinate system based on the relative pose information and in combination with the origin.

[0095] In the embodiment of the present application, the relative relationship between two coordinate systems (such as the lidar coordinate system and the camera coordinate system, the calibration object coordinate system and the lidar coordinate system) can be called the relative pose, and the relative pose is composed of relative position information and relative pose information. In this article, the relative position information can be represented by a translation vector (t), the relative pose information can be represented by a rotation matrix (R), and the relative pose can be represented by T.

[0096] As an example, the point cloud rendering interface further includes an origin input component. When the target object selects a point in the target point cloud data and triggers the origin input component, the calibration processing device determines that the target object triggers an origin selection operation for the target point cloud data, and uses a reference point selected by the target object as the origin of the calibration object coordinate system.

[0097] Use PO to represent the origin of the calibration object coordinate system selected by the target object. Refer to Figure 8 As shown, the setting component of the point cloud rendering interface further includes a GetZeroPoint button, which is the origin input component. When the target object selects a reference point in the target point cloud data and triggers the GetZeroPoint button, the calibration processing device determines that the target object triggers an origin selection operation for the target point cloud data, and uses a reference point selected by the target object as the origin PO of the calibration object coordinate system.

[0098] As another example, when the target object selects a reference point in the target point cloud data and triggers a setting operation (such as double-clicking, long-pressing, etc., but not limited to this), the calibration processing device determines that the target object triggers an origin selection operation for the target point cloud data, and uses a reference point selected by the target object as the origin of the calibration object coordinate system.

[0099] In the embodiments of the present application, two reference points obtained by the target object through the axial selection operation can also be referred to as axial points. The two axial points are respectively used to indicate the positions of the coordinate axes of the calibration coordinate system. Use PX to represent the axial point for indicating the x-axis position, and use PY to represent the axial point for indicating the y-axis position. The line connecting PX and PO is the x-axis, and the line connecting PY and PO is the y-axis.

[0100] As a possible implementation, in order to further improve the calibration accuracy, in the embodiments of the present application, a rectangular calibration plate can be used as the target calibration plate, and data association can be performed using the edge intersection points of the calibration plate. Specifically, if the target calibration position is rectangular, one reference point is one of the four corner points of the rectangle, and the two reference points are located on two sides adjacent to one corner point in the rectangle. It should be noted that the two sides adjacent to one corner point can also be understood as the two sides connected to this corner point.

[0101] As an example, the point cloud rendering interface further includes two axial input components, which are respectively used to select PX and PY. When the target object selects a point in the target point cloud data and triggers the axial input component corresponding to the x-axis, the calibration processing device determines that the target object triggers an axial point selection operation for the target point cloud data, and uses the reference point selected by the target object as PX in the calibration object coordinate system. When the target object selects another point in the target point cloud data and triggers the axial input component corresponding to the y-axis, the calibration processing device determines that the target object triggers an axial point selection operation for the target point cloud data, and uses the reference point selected by the target object as PY in the calibration object coordinate system.

[0102] For example, still referring to Figure 8 As shown, the setting component of the point cloud rendering interface further includes a GetXPoint button, and the GetXPoint button is the axial input component corresponding to the x-axis. When the target object selects a reference point in the target point cloud data and triggers the GetXPoint button, the calibration processing device determines that the target object triggers an axial point selection operation for the target point cloud data, and uses the reference point selected by the target object as PX. The setting component of the point cloud rendering interface further includes a GetYPoint button, and the GetYPoint button is the axial input component corresponding to the y-axis. When the target object selects a reference point in the target point cloud data and triggers the GetYPoint button, the calibration processing device determines that the target object triggers an axial point selection operation for the target point cloud data, and uses the point selected by the target object as PY.

[0103] As another example, when the target object selects a reference point in the target point cloud data and triggers a setting operation (such as double-clicking, long-pressing, etc., but not limited to this), the calibration processing device determines that the target object triggers an axial point selection operation for the target point cloud data, and uses the two reference points selected by the target object as the axial points in the calibration object coordinate system.

[0104] S203. Based on the image of the target calibration object collected by the camera, obtain the first position of each feature point in the calibration object coordinate system.

[0105] Specifically, when executing S203, the calibration processing device may adopt but is not limited to the following methods:

[0106] The calibration processing device acquires the image of the target calibration object collected by the camera, and performs feature point detection on the image to obtain the image positions of the feature points in the image;

[0107] The calibration processing device obtains the relative position relationships between each feature point and each reference point based on the image positions of the feature points and in combination with the image positions of the reference points in the image;

[0108] Based on the relative position relationships between each feature point and each reference point, the calibration processing device obtains the first position of each feature point in the calibration object coordinate system.

[0109] It should be noted that in the embodiments of the present application, the method for detecting feature points is not limited, and thus will not be elaborated herein.

[0110] As a possible implementation manner, among the reference points, there is a reference point for indicating the origin of the calibration object coordinate system. The calibration processing device obtains the relative position relationships between each feature point and each reference point based on the image positions of each feature point and the image positions of each reference point in the image, which specifically includes: obtaining the relative position relationships between each feature point and the origin based on the image positions of each feature point and the image position of the origin in the image; obtaining the first position of each feature point in the calibration object coordinate system based on the relative position relationships between each feature point and each reference point, which specifically includes: obtaining the first position of each feature point in the calibration object coordinate system based on the relative position relationships between each feature point and the origin.

[0111] That is to say, in the embodiments of the present application, the relative position relationships between each feature point in the image and the origin of the coordinate axis are used as the relative position relationships between each feature point in the calibration object coordinate system and the origin of the coordinate axis. Then, according to the relative position relationships between each feature point in the calibration object coordinate system and the origin of the coordinate axis, each feature point in the image is converted into the calibration object coordinate system.

[0112] For example, the calibration processing device acquires an image of a target calibration object collected by a camera, detects feature points in the image, and obtains the image positions of 16 feature points in the image. Assume that the upper left corner of the target calibration object is used as the origin PO of the target calibration object. Based on the image positions of the 16 feature points and the image position of PO in the image, the relative position relationships between each feature point and the origin are obtained. Then, based on the relative position relationships between the 16 feature points and PO, the first positions of the 16 feature points in the calibration object coordinate system are obtained. Taking one of the 16 feature points as an example, in the image of the target calibration object collected by the camera, the relative position relationship between the feature point and the target calibration object indicates that the feature point is located at the lower right side of the target calibration object. Based on the relative position relationship between the feature point and the target calibration object, the first position of the feature point in the calibration object coordinate system is obtained.

[0113] S204. Based on each first position and in combination with the relative pose between the calibration object coordinate system and the lidar coordinate system, obtain the second position of each feature point in the lidar coordinate system.

[0114] In the embodiments of the present application, the following information can be obtained by selecting PX, PY, and PO:

[0115] (1) The attitude of the calibration board relative to the origin of the coordinate system, that is, the relative attitude information of the target calibration board relative to the origin.

[0116] (2) The origin PO of the target calibration board.

[0117] Based on the relative attitude information of the target calibration board relative to the origin and the origin of the target calibration board, the calibration processing device can obtain the relative pose between the calibration object coordinate system and the lidar coordinate system.

[0118] Define the relative attitude information of the target calibration board relative to the origin as a rotation matrix R, and assign the first column of R to V X , assign the second column of R to V Y , and assign the third column of R to V Z . Exemplarily, V X can be represented by formula (1), V Y can be represented by formula (2), and V Z can be represented by formula (3):

[0119] V X = V 1 Formula (1)

[0120] V Y = V X ×V X Formula (2)

[0121] V Z = V 1 ×V 2 Formula (3)

[0122] Among them, V 1 can be represented by formula (4), and V 2 can be represented by formula (5):

[0123] V 1 = PX - PO Formula (4)

[0124] V 2 = PY - PO Formula (5)

[0125] Define the transformation matrix of the calibration board in the lidar coordinate system (i.e., the relative pose between the calibration object coordinate system and the lidar coordinate system) as T. The rotation part of T is R, and the translation part is PO.

[0126] As a possible implementation, the calibration processing device obtains the second position of each feature point in the lidar coordinate system based on the first position of each feature point in the calibration object coordinate system and by using the above transformation matrix T.

[0127] That is to say, based on the first position of each feature point in the calibration object coordinate system and combining the relative pose between the calibration object coordinate system and the lidar coordinate system, the calibration processing device can transform each feature point from the calibration object coordinate system to the lidar coordinate system to obtain the second position of each feature point in the lidar coordinate system.

[0128] For example, as shown in Figure 9 Using a five-pointed star to represent the feature points, based on the first position of each of the 16 feature points in the calibration object coordinate system and combining the relative pose between the calibration object coordinate system and the lidar coordinate system, the calibration processing device can transform the 16 feature points from the calibration object coordinate system to the lidar coordinate system to obtain the second position of each of the 16 feature points in the lidar coordinate system.

[0129] S205. Obtain the relative pose between the lidar coordinate system and the camera coordinate system based on each second position.

[0130] Specifically, when executing S205, the calibration processing device can adopt but is not limited to the following methods:

[0131] The calibration processing device obtains the third position of each feature point in the camera coordinate system based on the image of the target calibration object collected by the camera;

[0132] The calibration processing device obtains the relative pose between the lidar coordinate system and the camera coordinate system based on each second position and in combination with each third position.

[0133] It should be noted that in the embodiments of the present application, whether it is the first position, the second position, or the third position, it can be represented by coordinates.

[0134] As a possible implementation, the calibration processing device obtains the relative pose between the lidar coordinate system and the camera coordinate system by using the Perspective-n-Point (PnP) algorithm based on the second position of each feature point in the lidar coordinate system and in combination with the third position of each feature point in the camera coordinate system.

[0135] PnP is a method for solving the 3D-to-2D point pair motion, aiming to solve the pose of the camera coordinate system relative to the reference coordinate system. PnP describes how to estimate the pose of the camera when the coordinates of n 3D points and the pixel coordinates of these points are known, that is, to solve the rotation matrix and translation vector of the camera coordinate system relative to the reference coordinate system. In the embodiments of the present application, the reference coordinate system is the lidar coordinate system. Therefore, the PnP algorithm is used to solve the pose of the camera coordinate system relative to the lidar coordinate system, and the pose includes a rotation matrix and a translation vector.

[0136] The PnP algorithm includes but is not limited to: Direct Linear Transform (DLT), Perspective-3-Point (P3P), EPnP, Bundle Adjustment (BA), etc.

[0137] For example, the calibration processing device uses the PoPs algorithm to obtain the relative pose between the lidar coordinate system and the camera coordinate system based on the second positions of 16 feature points in the lidar coordinate system and the third positions of the 16 feature points in the camera coordinate system.

[0138] As a possible implementation, considering that there is a certain error in the relative pose between the obtained lidar coordinate system and the camera coordinate system, in the embodiments of the present application, error estimation is performed through the reprojection error. Specifically, the calibration processing device can also obtain the fourth positions of the feature points in the camera coordinate system based on the relative pose between the obtained lidar coordinate system and the camera coordinate system and the second positions of the feature points in the lidar coordinate system. Then, based on the third positions of the feature points in the camera coordinate system and the fourth positions of the feature points in the camera coordinate system, the reprojection error is obtained. After that, based on the reprojection error, a new relative pose between the lidar coordinate system and the camera coordinate system is obtained.

[0139] That is to say, the calibration processing device uses the obtained relative pose between the lidar coordinate system and the camera coordinate system to convert each feature point from the lidar coordinate system to the camera coordinate system. Then, according to the converted position and the real position of each feature point in the camera coordinate system, the reprojection error is determined. Furthermore, the relative pose is optimized according to the reprojection error, so as to improve the accuracy of the external parameters of the camera.

[0140] Among them, when obtaining the new relative pose between the lidar coordinate system and the camera coordinate system based on the reprojection error, the new relative pose can be obtained with the goal of minimizing the sum of the sub-errors corresponding to each feature point. Here, the sub-error corresponding to each feature point refers to the difference between the third position and the fourth position of the feature point.

[0141] It should be noted that in the embodiments of the present application, only one reprojection process is described. In actual application, reprojection can also be performed multiple times to calibrate the external parameters of the camera by minimizing the reprojection error.

[0142] For example, the calibration processing device can also convert the 16 feature points to the camera coordinate system based on the relative pose between the lidar coordinate system and the camera coordinate system obtained, and combine the second positions of the 16 feature points in the lidar coordinate system respectively, to obtain the fourth positions of the 16 feature points in the camera coordinate system respectively. The fourth positions are denoted as 0', 1', 2', 3'. Then, based on the third positions of the 16 feature points in the camera coordinate system respectively, and combining the fourth positions of the respective feature points in the camera coordinate system, the reprojection error is obtained. Furthermore, based on the reprojection error, a new relative pose between the lidar coordinate system and the camera coordinate system is obtained.

[0143] Next, the present application will be described in conjunction with the calibration process in the AR scenario.

[0144] See Figure 10 As shown, the interaction object wears an AR headset device. A lidar and a camera are deployed in the AR headset device. The target acquisition scene contains a target calibration object, which is composed of three QR codes and three circular holes staggered. Each QR code has four corner points, and the target calibration object contains 12 feature points.

[0145] After the interaction object wears the AR headset device, data acquisition is performed in the target acquisition scene. The acquired data includes: the initial point cloud data of the target acquisition scene acquired by the lidar, and the image of the target acquisition scene acquired by the camera. After the data acquisition is completed, the interaction object uploads the data acquired through the VR headset device to the server, and the server stores the acquired data in the data storage system.

[0146] When the target object opens the client, the terminal device obtains the initial point cloud data of the target acquisition scene acquired by the lidar through the server.

[0147] Furthermore, the terminal device presents a point cloud rendering interface, which includes the initial point cloud data. The point cloud rendering interface includes a range input component, an origin input component (GetZeroPoint button), and two axial input components (GetXPoint button and GetYPoint button).

[0148] Further, the target object inputs a coordinate range through the range input component: the coordinate range on the x-axis is (-10, 10), the coordinate range on the y-axis is (-10, 10), and the coordinate range on the z-axis is (-1, 1). In response to the coordinate range in the lidar coordinate system input by the target object through the range input component, the calibration processing device presents the target point cloud data corresponding to the coordinate range in the point cloud rendering interface.

[0149] Further, when the target object selects a reference point in the target point cloud data and triggers the GetZeroPoint button, the terminal device determines that the target object triggers an origin selection operation for the target point cloud data, and uses a reference point selected by the target object as the origin PO of the calibration object coordinate system.

[0150] Further, when the target object selects a reference point in the target point cloud data and triggers the GetXPoint button, the calibration processing device determines that the target object triggers an axial point selection operation for the target point cloud data, and uses the reference point selected by the target object as PX. When the target object selects a reference point in the target point cloud data and triggers the GetYPoint button, the calibration processing device determines that the target object triggers an axial point selection operation for the target point cloud data, and uses the point selected by the target object as PY.

[0151] Further, the terminal device sends PO, PX, and PY to the server. Among them, PO is located at the upper left corner point of the target calibration object, and PX and PY are respectively located on two adjacent sides of the target calibration object to the upper left corner.

[0152] Further, the server constructs a calibration object coordinate system in the lidar coordinate system based on PO, PX, and PX.

[0153] Further, the server performs feature point detection on the image of the target calibration object collected by the camera, obtains the image positions of 12 feature points in the image, and based on the respective image positions of the 12 feature points, combines the image position of the upper left corner point of the target calibration object in the image to obtain the relative position relationships between the 12 feature points and the upper left corner point of the target calibration object, and based on the relative position relationships between the respective feature points and the upper left corner point of the target calibration object, obtains the first positions of the 12 feature points in the calibration object coordinate system.

[0154] Further, the server converts the 12 feature points from the calibration object coordinate system to the lidar coordinate system based on the first positions of the 12 feature points in the calibration object coordinate system and the relative pose between the calibration object coordinate system and the lidar coordinate system, and obtains the second positions of the 12 feature points in the lidar coordinate system.

[0155] Further, the server obtains the third positions of the 12 feature points in the camera coordinate system based on the images of the target calibration object collected by the camera, and then, based on the third positions of the 12 feature points in the camera coordinate system and in combination with the second positions of the 12 feature points in the lidar coordinate system, uses the PnP algorithm to obtain the relative pose between the lidar coordinate system and the camera coordinate system, and the relative pose between the lidar coordinate system and the camera coordinate system is the external parameters of the camera.

[0156] Based on the same inventive concept, an embodiment of the present application provides an external parameter calibration device between sensors.

[0157] As Figure 11 shown, it is a schematic structural diagram of an external parameter calibration device 1100 between sensors, which may include:

[0158] A range selection unit 1101, configured to present the target point cloud data corresponding to the coordinate range in the point cloud rendering interface in response to the coordinate range in the lidar coordinate system input by the target object; wherein, the coordinate range represents the position information of the target calibration object, and the target calibration object includes each feature point;

[0159] A point cloud selection unit 1102, configured to construct a calibration object coordinate system in the lidar coordinate system based on the selected reference points in response to the point selection operation triggered by the target object for the target point cloud data;

[0160] A first conversion unit 1103, configured to obtain the first positions of the respective feature points in the calibration object coordinate system based on the images of the target calibration object collected by the camera;

[0161] A second conversion unit 1104, configured to obtain the second positions of the respective feature points in the lidar coordinate system based on the respective first positions and in combination with the relative pose between the calibration object coordinate system and the lidar coordinate system;

[0162] A third conversion unit 1105, configured to obtain the relative pose between the lidar coordinate system and the camera coordinate system based on the respective second positions.

[0163] As a possible implementation manner, when constructing the calibration object coordinate system in the lidar coordinate system based on the point selection operation triggered by the target object for the target point cloud data and based on the selected reference points, the point cloud selection unit 1102 is specifically configured to:

[0164] In response to the origin selection operation triggered by the target object for the target point cloud data, use one of the selected reference points as the origin of the calibration object coordinate system;

[0165] In response to the axial selection operation triggered by the target object for the target point cloud data, based on two selected reference points, obtain the relative pose information of the target calibration board with respect to the origin, and based on the relative pose information, combine with the origin to construct the calibration object coordinate system.

[0166] As a possible implementation, if the target calibration is a rectangle, one reference point is one of the four corner points of the rectangle, and the two reference points are located on two sides adjacent to the one corner point in the rectangle.

[0167] As a possible implementation, when obtaining the first position of each feature point in the calibration object coordinate system based on the image of the target calibration object collected by the camera, the first conversion unit 1103 is specifically configured to:

[0168] Obtain the image of the target calibration object collected by the camera, and perform feature point detection on the image to obtain the image positions of the respective feature points in the image;

[0169] Based on the respective image positions of the feature points, in combination with the image positions of the respective reference points in the image, obtain the relative position relationships between the respective feature points and the respective reference points;

[0170] Based on the relative position relationships between the respective feature points and the respective reference points, obtain the first positions of the respective feature points in the calibration object coordinate system.

[0171] As a possible implementation, among the respective reference points, there is one reference point for indicating the origin of the calibration object coordinate system;

[0172] When obtaining the relative position relationships between the respective feature points and the respective reference points based on the respective image positions of the feature points, in combination with the image positions of the respective reference points in the image, the first conversion unit 1103 is specifically configured to:

[0173] Based on the respective image positions of the feature points, in combination with the image position of the one reference point in the image, obtain the relative position relationships between the respective feature points and the one reference point;

[0174] When obtaining the first positions of the respective feature points in the calibration object coordinate system based on the relative position relationships between the respective feature points and the respective reference points, the first conversion unit 1103 is specifically configured to:

[0175] Based on the relative position relationships between the respective feature points and the one reference point, obtain the first positions of the respective feature points in the calibration object coordinate system.

[0176] As a possible implementation, based on each second position, the relative pose between the lidar coordinate system and the camera coordinate system is obtained. Specifically, the third conversion unit 1105 is configured to:

[0177] Based on the image of the target calibration object collected by the camera, obtain the third position of each feature point in the camera coordinate system;

[0178] Based on each second position and in combination with each third position, obtain the relative pose between the lidar coordinate system and the camera coordinate system.

[0179] As a possible implementation, before presenting the target point cloud data corresponding to the coordinate range in the point cloud rendering interface in response to the coordinate range in the lidar coordinate system input by the target object, the range selection unit 1101 is further configured to:

[0180] Present a point cloud rendering interface, where the point cloud rendering interface includes: the initial point cloud data of the target acquisition scene collected by the lidar, and the target acquisition scene includes a target calibration object;

[0181] When presenting the target point cloud data corresponding to the coordinate range in the point cloud rendering interface in response to the coordinate range in the lidar coordinate system input by the target object, the range selection unit 1101 is specifically configured to:

[0182] In response to the coordinate range in the lidar coordinate system input by the target object, present the target point cloud data corresponding to the coordinate range in the initial point cloud data in the point cloud rendering interface.

[0183] As a possible implementation, the third conversion unit 1105 is further configured to:

[0184] Based on the obtained relative pose between the lidar coordinate system and the camera coordinate system, and in combination with the second position of each feature point in the lidar coordinate system, obtain the fourth position of each feature point in the camera coordinate system;

[0185] Based on the third position of each feature point in the camera coordinate system and in combination with the fourth position of each feature point in the camera coordinate system, obtain the reprojection error;

[0186] Based on the reprojection error, obtain the new relative pose between the lidar coordinate system and the camera coordinate system.

[0187] For the convenience of description, the above - mentioned parts are divided into various modules (or units) according to their functions and described separately. Of course, when implementing this application, the functions of the various modules (or units) can be implemented in the same or multiple software or hardware.

[0188] Regarding the device in the above - mentioned embodiments, the specific manner in which each unit executes the request has been described in detail in the embodiments related to the method, and will not be elaborated here.

[0189] In the embodiments of the present application, by using the candidate feature representations of adjacent candidate modality nodes to update the candidate feature representations, the candidate modality nodes corresponding to each modality information can learn the information contained in the candidate modality nodes corresponding to other modality information, improving the accuracy of the extrinsic parameter calibration between sensors. In addition, through the bidirectional iterative update method, the interaction between multiple modality information can be enhanced, strengthening the information flow, so that the candidate modality nodes corresponding to each modality information can better learn the information contained in the candidate modality nodes corresponding to other modality information, bridging the semantic gap between different modality information, thereby learning better feature representations and improving the accuracy of feature representations, and further improving the accuracy of abstract generation.

[0190] Those skilled in the art can understand that various aspects of the present application can be implemented as a system, a method, or a program product. Therefore, various aspects of the present application can be specifically implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation combining hardware and software aspects, which can be collectively referred to as "circuit", "module", or "system" here.

[0191] Based on the same inventive concept, the embodiments of the present application also provide an electronic device. In one embodiment, the electronic device can be a server or a terminal device. Refer to Figure 12 As shown, it is a schematic structural diagram of a possible electronic device provided in the embodiments of the present application. Figure 12 In it, the electronic device 1200 includes: a processor 1210 and a memory 1220.

[0192] Among them, the memory 1220 stores a computer program that can be executed by the processor 1210. By executing the instructions stored in the memory 1220, the processor 1210 can execute the steps of the above - mentioned extrinsic parameter calibration method between sensors.

[0193] The memory 1220 can be a volatile memory, such as a random-access memory (RAM); the memory 1220 can also be a non-volatile memory, such as a Read-Only Memory (ROM), a flash memory, a hard disk drive (HDD), or a solid-state drive (SSD); or the memory 1220 is any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 1220 can also be a combination of the above memories.

[0194] The processor 1210 can include one or more central processing units (CPUs) or be a digital processing unit, etc. The processor 1210 is used to implement the above-mentioned external parameter calibration method between sensors when executing the computer program stored in the memory 1220.

[0195] In some embodiments, the processor 1210 and the memory 1220 can be implemented on the same chip. In some embodiments, they can also be separately implemented on independent chips.

[0196] In the embodiments of the present application, the specific connection medium between the above-mentioned processor 1210 and the memory 1220 is not limited. In the embodiments of the present application, taking the connection between the processor 1210 and the memory 1220 through a bus as an example, the bus is Figure 12 described by a thick line in []. The connection manners between other components are only for illustrative purposes and are not to be construed as limiting. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of description, Figure 12 only a thick line is used to describe it in [], but it does not describe that there is only one bus or one type of bus.

[0197] Based on the same inventive concept, the embodiments of the present application provide a computer-readable storage medium, which includes a computer program. When the computer program runs on an electronic device, the computer program is used to cause the electronic device to execute the steps of the above-mentioned external parameter calibration method between sensors. In some possible implementation manners, each aspect of the external parameter calibration method between sensors provided in the present application can also be implemented in the form of a program product, which includes a computer program. When the program product runs on an electronic device, the computer program is used to cause the electronic device to execute the steps in the above-mentioned external parameter calibration method between sensors. For example, the electronic device can execute as Figure 2 the steps shown in [].

[0198] The program product may adopt any combination of one or more readable media. The readable media may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the readable storage medium include: an electrical connection with one or more wires, a portable disk, a hard disk, a RAM, a ROM, an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (Compact Disk Read Only Memory, CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0199] The program product of the embodiments of the present application may adopt a CD-ROM and include a computer program, and may run on an electronic device. However, the program product of the present application is not limited thereto. In this document, the readable storage medium may be any tangible medium that contains or stores a computer program, and this computer program may be used by or in combination with a command execution system, apparatus, or device.

[0200] The readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a readable computer program is carried. Such a propagated data signal may take various forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable signal medium may also be any readable medium other than the readable storage medium, and this readable medium may send, propagate, or transmit a computer program for use by or in combination with a command execution system, apparatus, or device.

[0201] Although the preferred embodiments of the present application have been described, those skilled in the art can make additional changes and modifications to these embodiments once they learn the basic creative concepts. Therefore, the appended claims are intended to be construed to include the preferred embodiments as well as all changes and modifications falling within the scope of the present application.

[0202] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application is also intended to include these modifications and variations.

Claims

1. A method for calibrating external parameters between sensors, characterized in that: include: In response to the coordinate range of the target object in the laser radar coordinate system input, the target point cloud data corresponding to the coordinate range is presented in the point cloud rendering interface; wherein the coordinate range represents the position information of the target calibration object, and the target calibration object includes each feature point; In response to a point selection operation triggered by the target object on the target point cloud data, a calibration object coordinate system is constructed in the laser radar coordinate system based on the selected reference points; Based on the image of the target calibration object captured by the camera, obtaining the first position of each of the feature points in the calibration object coordinate system; Based on each first position, in combination with the relative position between the calibration object coordinate system and the laser radar coordinate system, obtain the second position of each feature point in the laser radar coordinate system; Based on each second position, a relative pose between the laser radar coordinate system and the camera coordinate system is obtained.

2. The method according to claim 1, characterized in that The step of constructing a calibration object coordinate system in the laser radar coordinate system based on each selected reference point in response to the point selection operation triggered by the target object on the target point cloud data includes: In response to an origin selection operation triggered by the target object on the target point cloud data, a selected reference point is used as the origin of the calibration object coordinate system; In response to an axial selection operation triggered by the target object on the target point cloud data, relative posture information of the target calibration plate relative to the origin is obtained based on the two selected reference points, and the calibration object coordinate system is constructed based on the relative posture information and combined with the origin.

3. The method according to claim 2, characterized in that If the target mark is positioned as a rectangle, the one reference point is one of the four corner points of the rectangle, and the two reference points are located on two sides of the rectangle adjacent to the one corner point.

4. The method according to claim 1, 2 or 3, characterized in that: The obtaining, based on the image of the target calibration object acquired by the camera, the first position of each feature point in the calibration object coordinate system comprises: Acquire an image of the target calibration object captured by a camera, and perform feature point detection on the image to obtain the image position of each feature point in the image; Based on the image positions of the feature points, and in combination with the image positions of the reference points in the image, obtaining the relative positional relationships between the feature points and the reference points; Based on the relative positional relationship between each of the feature points and each of the reference points, a first position of each of the feature points in the calibration object coordinate system is obtained.

5. The method according to claim 4, characterized in that The reference points include: a reference point for indicating the origin of the calibration object coordinate system; The obtaining the relative positional relationship between each feature point and each reference point based on the image position of each feature point and the image position of each reference point in the image includes: The step of obtaining the relative positional relationship between each of the feature points and the one reference point based on the image position of each of the feature points and the image position of the one reference point in the image; The obtaining, based on the relative positional relationship between each feature point and each reference point, a first position of each feature point in the calibration object coordinate system includes: Based on the relative positional relationship between each of the feature points and the one reference point, a first position of each of the feature points in the calibration object coordinate system is obtained.

6. The method according to claim 1, 2 or 3, characterized in that: Based on each second position, obtaining a relative pose between the laser radar coordinate system and the camera coordinate system, including: Based on the image of the target calibration object captured by the camera, obtaining a third position of each of the feature points in the camera coordinate system; Based on each second position and in combination with each third position, a relative position and posture between the laser radar coordinate system and the camera coordinate system is obtained.

7. The method according to claim 1, 2 or 3, characterized in that: In response to the coordinate range of the laser radar coordinate system input by the target object, before presenting the target point cloud data corresponding to the coordinate range in the point cloud rendering interface, the method further includes: Presenting a point cloud rendering interface, the point cloud rendering interface comprising: initial point cloud data of a target acquisition scene acquired by a laser radar, the target acquisition scene comprising a target calibration object; In response to the coordinate range of the target object in the laser radar coordinate system input, the target point cloud data corresponding to the coordinate range is presented in the point cloud rendering interface, including: In response to the coordinate range of the target object input in the laser radar coordinate system, the target point cloud data corresponding to the coordinate range in the initial point cloud data is presented in the point cloud rendering interface.

8. The method according to claim 1, 2 or 3, characterized in that: Also includes: Based on the obtained relative position between the laser radar coordinate system and the camera coordinate system, and in combination with the second position of each feature point in the laser radar coordinate system, obtain the fourth position of each feature point in the camera coordinate system; Obtaining a reprojection error based on the third position of each feature point in the camera coordinate system and in combination with the fourth position of each feature point in the camera coordinate system; Based on the reprojection error, a new relative pose between the lidar coordinate system and the camera coordinate system is obtained.

9. A device for calibrating external parameters between sensors, characterized in that: include: A range selection unit, for presenting target point cloud data corresponding to the coordinate range in the laser radar coordinate system input by the target object in a point cloud rendering interface; wherein the coordinate range represents the position information of the target calibration object, and the target calibration object includes each feature point; A point cloud selection unit, configured to construct a calibration object coordinate system in the laser radar coordinate system based on each selected reference point in response to a point selection operation triggered by the target object on the target point cloud data; A first conversion unit, configured to obtain a first position of each of the feature points in a coordinate system of the calibration object based on an image of the target calibration object acquired by a camera; A second conversion unit is used to obtain the second position of each feature point in the laser radar coordinate system based on each first position and in combination with the relative position between the calibration object coordinate system and the laser radar coordinate system; The third conversion unit is used to obtain the relative posture between the laser radar coordinate system and the camera coordinate system based on each second position.

10. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of any one of the methods of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: It includes a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of any method described in claims 1 to 8.

12. A computer program product, characterized in that It includes a computer program, which is stored in a computer-readable storage medium. The processor of the electronic device reads and executes the computer program from the computer-readable storage medium, so that the electronic device executes the steps of any method described in claims 1 to 8.

Images