External parameter calibration method, aircraft monitoring equipment, computer equipment and medium

Abstract

The present invention discloses an external parameter calibration method, an aircraft monitoring device, a computer device and a medium, which relate to the field of external parameter calibration and include the following steps: obtaining multiple data groups of preset targets, wherein each of the data groups includes the positioning coordinates of the target at the same time, a target detection frame collected by at least one first sensor and target point cloud data collected by at least one second sensor; determining the error weight of each target detection frame based on the target point cloud data in the data group and the internal parameters of each of the first sensors; constructing a first error function using the external parameters to be calibrated, the positioning coordinates in each of the data groups, the target detection frame and the corresponding error weight, and constructing a second error function using the external parameters to be calibrated, the positioning coordinates in each of the data groups and the target point cloud data; and calibrating the external parameters to be calibrated according to the first error function and the second error function.

Description

Technical Field

[0001] The present invention relates to the field of external parameter calibration, and in particular to an external parameter calibration method, aircraft monitoring equipment, computer equipment and medium. Background Art

[0002] In order to improve the accuracy and position precision of target detection, multi-source heterogeneous data is often used, such as image plus radar point cloud, image plus millimeter wave point cloud, so the integrated machine often has more than two sensors inside, and these sensors all need to be calibrated with external parameters. When there is a fixed reference object in the field of view of the sensor, the fixed reference object can be combined with map collection to complete the calibration. For example, in the vehicle-road collaboration industry, sensors installed on the roadside often illuminate ground objects, so the pre-made map can be used to calibrate the sensor to obtain the position and rotation angle of the sensor in the map.

[0003] However, if there are no fixed reference objects within the field of view of the sensor, the above method cannot complete the calibration. For example, for an all-in-one device for monitoring low-altitude aircraft, the sensor inside is often irradiated toward the sky, and there are no fixed reference objects within the irradiated area. Therefore, it is impossible to use pre-made maps and fixed reference objects to complete the calibration of the sensor. Summary of the invention

[0004] In view of this, in order to overcome at least one aspect of the above problems, an embodiment of the present invention proposes an external parameter calibration method for a fusion sensing device, comprising the following steps:

[0005] Acquire multiple data groups of preset targets, wherein each of the data groups includes the positioning coordinates of the target, the target detection frame collected by at least one first sensor, and the target point cloud data collected by at least one second sensor at the same time;

[0006] Determine an error weight of each of the target detection boxes based on the target point cloud data in the data group and an internal parameter of each of the first sensors;

[0007] Constructing a first error function using the extrinsic parameters to be calibrated, the positioning coordinates in each of the data groups, the target detection frame, and the corresponding error weights, and constructing a second error function using the extrinsic parameters to be calibrated, the positioning coordinates in each of the data groups, and the target point cloud data;

[0008] The extrinsic parameter to be calibrated is calibrated according to the first error function and the second error function.

[0009] In some embodiments, obtaining multiple data groups of a preset target, wherein each of the data groups includes the positioning coordinates of the target, the target detection frame collected by at least one first sensor, and the target point cloud data collected by at least one second sensor at the same time, further includes:

[0010] Receiving the positioning coordinates sent by the target, collecting the target image using at least one of the first sensors, and collecting the target point cloud data using at least one of the second sensors;

[0011] Performing image detection and recognition on each target image captured by at least one of the first sensors to obtain a corresponding target detection frame;

[0012] The data group is constructed using the positioning coordinates received at the same time, the target detection frame corresponding to at least one of the target images collected, and at least one of the target point cloud data collected.

[0013] In some embodiments, constructing the data group using the positioning coordinates received at the same time, the target detection frame corresponding to at least one of the target images acquired, and at least one of the target point cloud data acquired further includes:

[0014] Acquire a collection grid and a regional grid corresponding to each of the first sensors, wherein each second grid in the regional grid is larger than each first grid in the collection grid;

[0015] Determine in sequence the first position of the center of each target detection frame in the corresponding acquisition grid and the second position in the corresponding regional grid;

[0016] Determine whether there is a mark in the first grid corresponding to the first position;

[0017] In response to the absence of a mark, retaining the corresponding target detection frame and marking the corresponding first grid, and marking in the corresponding second grid according to the second position;

[0018] In response to the existence of the mark, deleting the corresponding object detection box;

[0019] In response to the proportion of the marked second grids in each of the regional grids reaching a threshold, the construction of the data set is stopped.

[0020] In some embodiments, determining the error weight of each of the target detection boxes based on the target point cloud data in the data group and the intrinsic parameters of each of the first sensors further includes:

[0021] Determine the distance between the target and each first sensor according to the target point cloud data in the data group;

[0022] An error weight of each of the target detection frames is determined using the first focal length and the second focal length of the first sensor and the distance.

[0023] In some embodiments, the first error function is constructed using the extrinsic parameters to be calibrated, the positioning coordinates in each of the data groups, the target detection frame, and the corresponding error weights, and the second error function is constructed using the extrinsic parameters to be calibrated, the positioning coordinates in each of the data groups, and the target point cloud data, further comprising:

[0024] Acquire a relative extrinsic parameter of each of the first sensors and a relative extrinsic parameter of each of the second sensors;

[0025] Using the extrinsic parameter to be calibrated and the relative extrinsic parameter of each of the first sensors, the positioning coordinates are converted into a coordinate system of each of the first sensors to obtain a first projection coordinate;

[0026] Constructing a first error function based on the first projection coordinates, the corresponding target detection frame, and the error weight;

[0027] Using the extrinsic parameter to be calibrated and the relative extrinsic parameter of each second sensor, the positioning coordinate is converted into the coordinate system of each second sensor to obtain a second projection coordinate;

[0028] A second error function is constructed based on the second projection coordinates and the corresponding target point cloud data.

[0029] In some embodiments, obtaining a relative extrinsic parameter of each of the first sensors and a relative extrinsic parameter of each of the second sensors further includes:

[0030] Using each of the first sensors to collect an image of a preset marker and extract a binary image of the preset marker, and using each of the second sensors to collect point cloud data of the preset marker;

[0031] Randomly select a coordinate system of a second sensor as the base coordinate system;

[0032] Obtaining a relative extrinsic parameter of each second sensor based on the point cloud data collected by the randomly selected second sensor and the point cloud data collected by the remaining second sensors;

[0033] constructing a third error function using an intrinsic parameter of each of the first sensors, point cloud data collected by a randomly selected second sensor, and a relative extrinsic parameter of each of the first sensors;

[0034] A relative extrinsic parameter of each of the first sensors is obtained based on the third error function.

[0035] In some embodiments, calibrating the extrinsic parameter to be calibrated according to the first error function and the second error function further includes:

[0036] The external parameter to be calibrated is adjusted until the sum of the first error function and the second error function is minimized, so as to complete the calibration of the external parameter to be calibrated.

[0037] Based on the same inventive concept, according to another aspect of the present invention, an embodiment of the present invention further provides an aircraft monitoring device, comprising:

[0038] at least one first sensor, each of the first sensors being used to collect an object detection frame of an object;

[0039] at least one second sensor, each of the first sensors being used to collect target point cloud data of a target;

[0040] The aircraft monitoring device performs external parameter calibration based on the target detection frame and the target point cloud data using a method as described in any of the above embodiments.

[0041] Based on the same inventive concept, according to another aspect of the present invention, an embodiment of the present invention further provides a computer device, including:

[0042] at least one processor; and

[0043] A memory, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the steps of any one of the external parameter calibration methods of the fusion perception device as described above are performed.

[0044] Based on the same inventive concept, according to another aspect of the present invention, an embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it performs the steps of the external parameter calibration method of any one of the fusion perception devices described above.

[0045] The present invention has one of the following beneficial technical effects: the solution proposed by the present invention combines the positioning coordinates of the target and the data collected by multiple sensors to perform external parameter calibration of the fusion perception device (all-in-one device) when there is no fixed reference object within the target movement area and no need to collect maps. At the same time, the data collected by multiple sensors are fused through error weights to associate the two-dimensional data with the three-dimensional data, and then the first error function and the second error function are associated, which reduces error accumulation and solves the problem of error accumulation in the calibration of multi-sensor equipment, thereby making the external parameter calibration more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can be obtained based on these drawings without paying creative work.

[0047] Figure 1 A schematic diagram of a flow chart of an external parameter calibration method for a fusion sensing device provided in an embodiment of the present invention;

[0048] Figure 2 A schematic diagram of a collection grid provided by an embodiment of the present invention;

[0049] Figure 3 A schematic diagram of a regional grid provided for an embodiment of the present invention;

[0050] Figure 4 A schematic diagram of the mapping relationship between pixel distance and physical distance provided by an embodiment of the present invention;

[0051] Figure 5 A schematic diagram of the marker production and extraction effect provided by an embodiment of the present invention;

[0052] Figure 6 A schematic diagram of comparison before and after relative external parameter optimization provided for an embodiment of the present invention;

[0053] Figure 7 A schematic diagram of the structure of an external parameter calibration system provided by an embodiment of the present invention;

[0054] Figure 8 A schematic diagram of the structure of a computer device provided by an embodiment of the present invention;

[0055] Fig. 9 A schematic diagram of the structure of a computer-readable storage medium provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the embodiments of the present invention are further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0057] It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are for distinguishing two non-identical entities with the same name or non-identical parameters. It can be seen that "first" and "second" are only for the convenience of expression and should not be understood as limitations on the embodiments of the present invention. The subsequent embodiments will not explain this one by one.

[0058] In an embodiment of the present invention, a target may refer to a low-altitude aircraft such as a drone or an object located in a scene without a fixed reference object. A fusion sensing device (all-in-one device) refers to a monitoring device for monitoring the target. The device generally includes at least one camera (first sensor), at least one laser radar or at least one millimeter-wave radar (second sensor). The external parameter calibration method of a fusion sensing device proposed in the present invention is applied to the external parameter calibration of a fusion sensing device (all-in-one device).

[0059] According to one aspect of the present invention, an embodiment of the present invention provides an external parameter calibration method for a fusion sensing device, such as Figure 1 As shown, it may include the steps of:

[0060] S1, acquiring multiple data groups of preset targets, wherein each of the data groups includes the positioning coordinates of the target, a target detection frame acquired by at least one first sensor, and target point cloud data acquired by at least one second sensor at the same time;

[0061] S2, determining an error weight of each of the target detection boxes based on the target point cloud data in the data group and the internal parameters of each of the first sensors;

[0062] S3, constructing a first error function using the extrinsic parameters to be calibrated, the positioning coordinates in each of the data groups, the target detection frame, and the corresponding error weights, and constructing a second error function using the extrinsic parameters to be calibrated, the positioning coordinates in each of the data groups, and the target point cloud data;

[0063] S4: Calibrate the extrinsic parameter to be calibrated according to the first error function and the second error function.

[0064] The solution proposed in the present invention combines the positioning coordinates of the target and the data collected by multiple sensors to perform external parameter calibration of the all-in-one device when there is no fixed reference object within the target movement area and no need to collect maps. At the same time, the data collected by multiple sensors are fused through error weights to associate the two-dimensional data with the three-dimensional data, and then the first error function and the second error function are associated, which reduces error accumulation and solves the problem of error accumulation in the calibration of multi-sensor equipment, thereby making the external parameter calibration more accurate.

[0065] In some embodiments, obtaining multiple data groups of a preset target, wherein each of the data groups includes the positioning coordinates of the target, the target detection frame collected by at least one first sensor, and the target point cloud data collected by at least one second sensor at the same time, further includes:

[0066] Receiving the positioning coordinates sent by the target, collecting the target image using at least one of the first sensors, and collecting the target point cloud data using at least one of the second sensors;

[0067] Performing image detection and recognition on each target image captured by at least one of the first sensors to obtain a corresponding target detection frame;

[0068] The data group is constructed using the positioning coordinates received at the same time, the target detection frame corresponding to at least one of the target images collected, and at least one of the target point cloud data collected.

[0069] Specifically, after the all-in-one device is installed and fixed outdoors, the drone is controlled to fly within the monitoring range of the all-in-one device and sends high-precision positioning coordinates in the world coordinate system at all times. At this time, at least one first sensor (such as a camera) of the all-in-one device is used to collect image data, and at least one second sensor (such as a laser radar or a millimeter wave radar) is used to collect point cloud data.

[0070] Since the drone equipment contains more information in the image, mature image detection models such as R-CNN, YOLO, DETR, etc. can be used to detect the drone in the image and obtain the target detection frame. At the same time, the relative external parameters between sensors are used to project the collected point cloud data onto the image, and the point cloud is filtered according to whether it falls within the detection frame, and finally the target point cloud data reflected by the drone is obtained.

[0071] Since the speed of the drone is relatively high during flight, in order to better collect data, it is necessary to ensure that the time interval for collecting data is as small as possible, and align the data based on the collection time, that is, to construct the data group with the target positioning coordinates received at the same time, the target detection frame corresponding to at least one of the target images collected, and the at least one target point cloud data collected. In this way, by controlling the flight of the drone, M groups of data of the drone at different positions can be obtained: , where the mth group of data Includes: UAV positioning coordinates in the world coordinate system , at least one target detection frame detected by a first sensor, at least one target point cloud data detected by a second sensor .

[0072] It should be noted that the number of target detection frames and target point cloud data in different groups of data may be different. For example, when multiple first sensors are set at different positions and angles in the integrated device, only a certain number of first sensors may collect target image data at the same time. The data may include the positioning data acquired at the same time, the target detection frame A1 corresponding to the target image acquired by the first sensor A, the target detection frame B corresponding to the target image acquired by the first sensor B, the target point cloud data acquired by the second sensor a, and the target point cloud data acquired by the second sensor b; and the second set of data It may include positioning data acquired at the same time, target detection frame A2 corresponding to the target image acquired by the first sensor A, target detection frame C corresponding to the target image acquired by the first sensor C, target point cloud data acquired by the second sensor a, and target point cloud data acquired by the second sensor c.

[0073] In some embodiments, constructing the data group using the positioning coordinates received at the same time, the target detection frame corresponding to at least one of the target images acquired, and at least one of the target point cloud data acquired further includes:

[0074] Acquire a collection grid and a regional grid corresponding to each of the first sensors, wherein each second grid in the regional grid is larger than each first grid in the collection grid;

[0075] Determine in sequence the first position of the center of each target detection frame in the corresponding acquisition grid and the second position in the corresponding regional grid;

[0076] Determine whether there is a mark in the first grid corresponding to the first position;

[0077] In response to the absence of a mark, retaining the corresponding target detection frame and marking the corresponding first grid, and marking in the corresponding second grid according to the second position;

[0078] In response to the existence of the mark, deleting the corresponding object detection box;

[0079] In response to the proportion of the marked second grids in each of the regional grids reaching a threshold, the construction of the data set is stopped.

[0080] Specifically, when the time interval for collecting data is short, although it ensures that the collected data is sufficient, sufficient data will affect the amount of calculation and further affect the calibration speed. Therefore, the collected data can be screened.

[0081] First, it is necessary to set an acquisition grid of m rows and n columns and a regional grid of p rows and q columns according to the size of the image collected by each first sensor, and each second grid in the regional grid is larger than each first grid in the acquisition grid. Then, according to the center position of each target detection frame, the corresponding first grid and the corresponding second grid are determined. If there is no mark in the first grid, the target detection frame data is retained and marked in the corresponding first grid and second grid. If there is no mark in the first grid, the target detection frame is deleted. Finally, based on the proportion of the second grid with marks in the regional grid, it is determined whether to stop the construction of the data group.

[0082] Taking the first sensor A as an example, according to the size of its collected image, a collection grid of 6 rows and 8 columns and a regional grid of 3 rows and 4 columns can be set, and then the first grid and the second grid corresponding to the center of the target detection frame corresponding to the first sensor A in each data group are determined in turn. If there is no mark in the first grid, the target detection frame is retained and a mark is added to the first grid and the second grid. If a mark already exists, the target detection frame is deleted. For example, in the first data set in the above example, And the second set of data , if the first set of data The first grid corresponding to the target detection box A1 in the middle does not have a mark, while the second set of data If the first grid corresponding to the target detection frame A2 in the image has a mark, the target detection frame A1 is retained and the target detection frame A2 is deleted. Finally, we can get the following Figure 2 The acquisition grid shown and Figure 3 The regional grid shown in Figure 1. When the proportion of the second grid with marks in the regional grid is greater than α (0<α<1), the data set construction will stop. Figure 3 In the 4*3 area grid shown, there are 9 areas with data records, accounting for 3 / 4. If the threshold α is set to 2 / 3, the construction of the data group will be stopped at this time, and the subsequent calibration process will be automatically triggered.

[0083] In some embodiments, determining the error weight of each of the target detection boxes based on the target point cloud data in the data group and the intrinsic parameters of each of the first sensors further includes:

[0084] Determine the distance between the target and each first sensor according to the target point cloud data in the data group;

[0085] An error weight of each of the target detection frames is determined using the first focal length and the second focal length of the first sensor and the distance.

[0086] Specifically, the first error function is constructed based on the target detection box, which is associated with the two-dimensional image. The second error function is constructed based on the target point cloud data, which is associated with the three-dimensional space. This results in different scale units for the two. By setting an error weight for the first error function, the pixel distance of the two-dimensional image is associated with the physical distance in the three-dimensional space, and then the first error function is associated with the second error function to prevent the phenomenon of error transmission and accumulation, thereby making the calibration more accurate.

[0087] like Figure 4 As shown, according to the pinhole imaging principle, although The distance between The distance between them is the same, but the pixel distance on the imaging plane is and However, they are not the same. At the same time, according to the relationship between similar triangles, the following equation holds:

[0088]

[0089] in, is the physical focal length of the camera, which is a fixed value. If you want to calculate the physical distance corresponding to a single pixel distance, you can Set to 1 pixel, then the right side of the equation is the focal length in the camera intrinsic parameters and Since the size of the photosensitive element in the xy axis direction is roughly the same, the xy axis can be ignored. Approximately equal to , thus obtaining:

[0090]

[0091] Error weight :

[0092]

[0093] in Represents the physical distance between the object and the camera. This distance can be calculated using the target point cloud data, i.e., point cloud The average distance between each point in and the origin:

[0094]

[0095] in, Represents any point cloud in the xth group of data The coordinates of the nth point in the cloud, where N is the number of points. Alternatively, the average distance between each point in the point cloud and the origin can be calculated, and then the average can be obtained. .

[0096] In some embodiments, the first error function is constructed using the extrinsic parameters to be calibrated, the positioning coordinates in each of the data groups, the target detection frame, and the corresponding error weights, and the second error function is constructed using the extrinsic parameters to be calibrated, the positioning coordinates in each of the data groups, and the target point cloud data, further comprising:

[0097] Acquire a relative extrinsic parameter of each of the first sensors and a relative extrinsic parameter of each of the second sensors;

[0098] Using the extrinsic parameter to be calibrated and the relative extrinsic parameter of each of the first sensors, the positioning coordinates are converted into a coordinate system of each of the first sensors to obtain a first projection coordinate;

[0099] Constructing a first error function based on the first projection coordinates, the corresponding target detection frame, and the error weight;

[0100] Using the extrinsic parameter to be calibrated and the relative extrinsic parameter of each second sensor, the positioning coordinate is converted into the coordinate system of each second sensor to obtain a second projection coordinate;

[0101] A second error function is constructed based on the second projection coordinates and the corresponding target point cloud data.

[0102] Specifically, in the integrated machine external parameter E B2W When the target positioning coordinates are projected to the coordinate system of the first sensor, the projection point Should be close to the center of the 2D detection box , where the projection formula is:

[0103]

[0104] in, Represents the intrinsic parameter matrix of the i-th first sensor with a dimension of 3*3, including the focal length , and optical center position , ; yes The inverse matrix of is the relative external parameter matrix of the i-th first sensor; ( , , ) is the positioning coordinates of the drone; Represents the three-dimensional point converted from the positioning coordinates to the camera coordinate system; Represents the two-dimensional point mapped to the image coordinate system, that is, the projection point position.

[0105] External reference in the integrated machine E B2W If the (extrinsic parameter to be optimized) is inaccurate, errors will occur when projecting the target's positioning coordinates onto the camera image. In this way, the errors of each target detection box in each set of data are accumulated to obtain the first error function:

[0106]

[0107] Wherein, m represents the mth data group, M is the number of data groups; i represents the i-th target detection frame, and I represents the number of target detection frames in the Mth data group; E m i is the error weight corresponding to the i-th target detection box in the m-th group of data, is the projection point of the positioning coordinates in the mth group of data onto the i-th target detection box in the mth group of data, is the center position of the i-th target detection box in the m-th group of data.

[0108] External reference in the integrated machine E B2W If the (external parameters to be optimized) are accurate, the target positioning coordinates ( , , ) is projected into the coordinate system of the second sensor, the projection point Should be located in the point cloud Near the middle, where the projection formula is:

[0109]

[0110] in is the inverse matrix of the relative extrinsic parameter of the second sensor.

[0111] External reference in the integrated machine E B2W If the (extrinsic parameter to be optimized) is inaccurate, errors will occur when projecting the target's positioning coordinates into the coordinate system of the second sensor. In this way, the errors of each target point cloud data in each set of data are accumulated to obtain the second error function:

[0112]

[0113] m represents the mth data group, M is the number of data groups, x Indicates x target point cloud data, X represents the number of target point cloud data in the Mth data group; The projection point of the positioning coordinates in the mth group of data to the xth target point cloud in the mth group of data, is the coordinate of the nth point in the xth target point cloud in the mth group of data.

[0114] In some embodiments, calibrating the extrinsic parameter to be calibrated according to the first error function and the second error function further includes:

[0115] The external parameter to be calibrated is adjusted until the sum of the first error function and the second error function is minimized, so as to complete the calibration of the external parameter to be calibrated.

[0116] Specifically, by optimizing the external parameters of the integrated machine E B2W , thereby changing the projection point of the target in the coordinate system of the first sensor and the projection point in the coordinate system of the second sensor , so that the sum of the first error function and the second error function is minimized, thereby achieving the purpose of joint optimization. In some embodiments, a nonlinear optimization method can be used to optimize the external parameters. E B2W Make optimizations.

[0117] In some embodiments, obtaining a relative extrinsic parameter of each of the first sensors and a relative extrinsic parameter of each of the second sensors further includes:

[0118] Using each of the first sensors to collect an image of a preset marker and extract a binary image of the preset marker, and using each of the second sensors to collect point cloud data of the preset marker;

[0119] Randomly select a coordinate system of a second sensor as the base coordinate system;

[0120] Obtaining a relative extrinsic parameter of each second sensor based on the point cloud data collected by the randomly selected second sensor, the point cloud data collected by the remaining second sensors, and a first preset algorithm;

[0121] constructing a third error function using an intrinsic parameter of each of the first sensors, point cloud data collected by a randomly selected second sensor, and a relative extrinsic parameter of each of the first sensors;

[0122] A relative extrinsic parameter of each of the first sensors is obtained based on the third error function.

[0123] Specifically, the relative external parameters between multiple sensors in the all-in-one device can be calibrated indoors. After the all-in-one device is installed outdoors, only the external parameters of the all-in-one device need to be calibrated. The sensor external parameters can be obtained through the relative external parameters and the external parameters of the all-in-one device, thereby reducing the workload of external parameter calibration by several times.

[0124] Since it is relatively complicated to calculate the relative external parameters between two images, a second sensor can be randomly selected as the reference sensor. The coordinate system of this sensor is used as the coordinate system of the integrated device, that is, the reference coordinate system. The relative external parameters of this sensor are equal to the unit matrix. Since the relative positions of multiple sensors in the same type of integrated device are basically close, the initial external parameters of other sensors converted to the reference coordinate system can be measured, marked as , , , respectively represent the relative external parameter matrices of the transformation of the i-th camera, j-th lidar and k-th radar in the all-in-one machine to the basic coordinate system, and the dimension of the external parameter matrix is ​​4*4.

[0125] When calibrating relative to the outside, when the second sensor is a laser radar, since the laser point cloud is a reflection of the object surface, the reflection intensity is related to the object material. The corner reflection material is different from the outer contour material, and can be distinguished by the reflection intensity; at the same time, the distance range between the marker and the all-in-one machine can be easily determined indoors, so the marker in the laser point cloud can be extracted by the reflection intensity and point cloud distance to obtain the point cloud. When the second sensor is a millimeter-wave radar, it will only respond to objects of specific materials, but cannot accurately reflect the surface and position information of the object. However, it will produce high-intensity reflections at diagonal angles. Therefore, the markers in the laser point cloud can also be extracted by the reflection intensity and point cloud distance to obtain the point cloud. .

[0126] Therefore, if Figure 5 As shown, in order to facilitate the extraction of the marker in the second sensor, the corner inversion can be selected as the marker. At the same time, in order to facilitate the extraction of the pixel area of ​​the corner inversion in the image, a baffle of a specific color can be installed on the outer edge of the corner inversion to distinguish the background. In this way, the first sensor can also use a threshold of a specific color to filter the baffles in the image to obtain a binary image with multiple baffles. Since the inside of the baffle is the corner inversion, the internal contour is extracted from the binary image. The area inside the contour is the position of the corner inversion in the image, and finally a binary image of the corner inversion is obtained, which is recorded as .

[0127] It should be noted that, while ensuring that the marker can be sensed, the marker needs to be as far away from the all-in-one machine as possible, the number of markers should be at least three, and multiple markers should be evenly distributed in the overlapping area of ​​the sensing range of multiple sensors.

[0128] After data collection of the marker, since the data returned by lidar (laser radar) and radar (millimeter wave radar) are point clouds, they are often used in conjunction with cameras, that is, lidar+camera or radar+camera. For the relative external parameter calibration between each lidar or each radar and the selected second sensor, all point cloud data can be transformed into the basic coordinate system (the coordinate system of the selected second sensor). The point of the jth lidar Transform to the base coordinate system , the transformation formula is as follows:

[0129]

[0130] The kth radar point Transform to the base coordinate system , the transformation formula is as follows:

[0131]

[0132] Then, based on the converted coordinates and the point cloud data of the marker collected by the selected second sensor, the relative external parameters are directly solved using existing algorithms, such as the ICP (iterative nearest neighbor algorithm) provided in the PCL library. and .

[0133] For the calibration of the relative external parameters between each first sensor and the selected second sensor, it is necessary to project the point cloud data of the selected second sensor onto the binary image corresponding to the first sensor so as to associate the point cloud with the pixel. The specific projection formula is as follows:

[0134] (The selected second sensor is LiDAR)

[0135] (The selected second sensor is millimeter wave radar)

[0136] in, Represents the i-th camera intrinsic parameter matrix with a dimension of 3*3, including the focal length , and optical center position , ; yes The inverse matrix of is the relative external parameter matrix of the i-th first sensor; Represents the three-dimensional point transformed from the point cloud to the camera coordinate system; Represents a two-dimensional point mapped to the image coordinate system, that is, the pixel position, where the upper left corner of the image is the origin, the right is the positive direction of the x-axis, and the downward is the positive direction of the y-axis. The initial external parameter projection effect is as follows Figure 6 As shown in the figure on the left, the millimeter wave point cloud or laser point cloud Projected onto the image, at this time in the binary image Select and project points The nearest pixel As matching points. Since the projection points of the point cloud on the image are accurate relative to the external parameters, will be evenly distributed around the corner, so the accumulated pixel error between all projection points and matching points in the point cloud should have a minimum value. Optimize to minimize the following error and obtain the optimal relative external parameter. The effect after optimization is as follows Figure 6 As shown in the figure on the right:

[0137]

[0138] Wherein N represents the total number of points in the point cloud of the marker collected by the selected second sensor.

[0139] The solution proposed in the present invention combines the positioning coordinates of the target and the data collected by multiple sensors to perform external parameter calibration of the all-in-one device when there is no fixed reference object within the target movement area and no need to collect maps. At the same time, the data collected by multiple sensors are fused during the external parameter calibration, which solves the problem of calibration error accumulation of multi-sensor equipment, thereby making the external parameter calibration more accurate.

[0140] Based on the same inventive concept, according to another aspect of the present invention, an embodiment of the present invention further provides an aircraft monitoring device, comprising:

[0141] at least one first sensor, each of the first sensors being used to collect an object detection frame of an object;

[0142] at least one second sensor, each of the first sensors being used to collect target point cloud data of a target;

[0143] The aircraft monitoring device performs external parameter calibration based on the target detection frame and the target point cloud data using a method as described in any of the above embodiments.

[0144] Based on the same inventive concept, according to another aspect of the present invention, an embodiment of the present invention further provides an external parameter calibration system 400, such as Figure 7 As shown, including:

[0145] The acquisition module 401 is configured to acquire multiple data groups of a preset target, wherein each of the data groups includes the positioning coordinates of the target, the target detection frame acquired by at least one first sensor, and the target point cloud data acquired by at least one second sensor at the same time;

[0146] A calculation module 402 is configured to determine an error weight of each of the target detection boxes based on the target point cloud data in the data group and the internal parameters of each of the first sensors;

[0147] The processing module 403 is configured to construct a first error function using the extrinsic parameters to be calibrated, the positioning coordinates in each of the data groups, the target detection box and the corresponding error weights, and to construct a second error function using the extrinsic parameters to be calibrated, the positioning coordinates in each of the data groups and the target point cloud data;

[0148] The optimization module 404 is configured to calibrate the extrinsic parameter to be calibrated according to the first error function and the second error function.

[0149] In some embodiments, the acquisition module 401 is further configured to:

[0150] Receiving the positioning coordinates sent by the target, collecting the target image using at least one of the first sensors, and collecting the target point cloud data using at least one of the second sensors;

[0151] Performing image detection and recognition on each target image captured by at least one of the first sensors to obtain a corresponding target detection frame;

[0152] The data group is constructed using the positioning coordinates received at the same time, the target detection frame corresponding to at least one of the target images collected, and at least one of the target point cloud data collected.

[0153] In some embodiments, the acquisition module 401 is further configured to:

[0154] Acquire a collection grid and a regional grid corresponding to each of the first sensors, wherein each second grid in the regional grid is larger than each first grid in the collection grid;

[0155] Determine in sequence the first position of the center of each target detection frame in the corresponding acquisition grid and the second position in the corresponding regional grid;

[0156] Determine whether there is a mark in the first grid corresponding to the first position;

[0157] In response to the absence of a mark, retaining the corresponding target detection frame and marking the corresponding first grid, and marking in the corresponding second grid according to the second position;

[0158] In response to the existence of the mark, deleting the corresponding object detection box;

[0159] In response to the proportion of the marked second grids in each of the regional grids reaching a threshold, the construction of the data set is stopped.

[0160] In some embodiments, determining the error weight of each of the target detection boxes based on the target point cloud data in the data group and the intrinsic parameters of each of the first sensors further includes:

[0161] Determine the distance between the target and each first sensor according to the target point cloud data in the data group;

[0162] An error weight of each of the target detection frames is determined using the first focal length and the second focal length of the first sensor and the distance.

[0163] In some embodiments, the processing module 403 is further configured to:

[0164] Acquire a relative extrinsic parameter of each of the first sensors and a relative extrinsic parameter of each of the second sensors;

[0165] Using the extrinsic parameter to be calibrated and the relative extrinsic parameter of each of the first sensors, the positioning coordinates are converted into a coordinate system of each of the first sensors to obtain a first projection coordinate;

[0166] Constructing a first error function based on the first projection coordinates, the corresponding target detection frame, and the error weight;

[0167] Using the extrinsic parameter to be calibrated and the relative extrinsic parameter of each second sensor, the positioning coordinate is converted into the coordinate system of each second sensor to obtain a second projection coordinate;

[0168] A second error function is constructed based on the second projection coordinates and the corresponding target point cloud data.

[0169] In some embodiments, the processing module 403 is configured to:

[0170] Using each of the first sensors to collect an image of a preset marker and extract a binary image of the preset marker, and using each of the second sensors to collect point cloud data of the preset marker;

[0171] Randomly select a coordinate system of a second sensor as the base coordinate system;

[0172] Obtaining a relative extrinsic parameter of each second sensor based on the point cloud data collected by the randomly selected second sensor, the point cloud data collected by the remaining second sensors, and a first preset algorithm;

[0173] constructing a third error function using an intrinsic parameter of each of the first sensors, point cloud data collected by a randomly selected second sensor, and a relative extrinsic parameter of each of the first sensors;

[0174] A relative extrinsic parameter of each of the first sensors is obtained based on the third error function.

[0175] In some embodiments, the optimization module 404 is further configured to:

[0176] The external parameter to be calibrated is adjusted until the sum of the first error function and the second error function is minimized, so as to complete the calibration of the external parameter to be calibrated.

[0177] Based on the same inventive concept, according to another aspect of the present invention, Figure 8 As shown, an embodiment of the present invention further provides a computer device 501, including:

[0178] at least one processor 520; and

[0179] The memory 510 stores a computer program 511 that can be run on the processor. When the processor 520 executes the program, the steps of any of the above-mentioned external parameter calibration methods for the fusion perception device are performed.

[0180] Based on the same inventive concept, according to another aspect of the present invention, Fig. 9 As shown, an embodiment of the present invention further provides a computer-readable storage medium 601, which stores a computer program 610. When the computer program 610 is executed by a processor, the steps of the external parameter calibration method of any of the above fusion perception devices are performed.

[0181] Finally, it should be noted that a person skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods.

[0182] Furthermore, it should be appreciated that the computer-readable storage medium (eg, memory) herein may be either volatile memory or nonvolatile memory, or may include both volatile memory and nonvolatile memory.

[0183] It will also be appreciated by those skilled in the art that various exemplary logic blocks, modules, circuits and algorithm steps described in conjunction with the disclosure herein can be implemented as electronic hardware, computer software or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, a general description has been given to the functions of various schematic components, blocks, modules, circuits and steps. Whether this function is implemented as software or hardware depends on specific applications and the design constraints imposed on the entire system. Those skilled in the art can implement the function in various ways for each specific application, but this implementation decision should not be interpreted as causing a departure from the disclosed scope of the embodiments of the present invention.

[0184] The above are exemplary embodiments disclosed in the present invention, but it should be noted that various changes and modifications may be made without departing from the scope disclosed in the embodiments of the present invention as defined in the claims. The functions, steps and / or actions of the method claims according to the disclosed embodiments described herein do not need to be performed in any particular order. In addition, although the elements disclosed in the embodiments of the present invention may be described or required in individual form, they may also be understood as multiple unless explicitly limited to the singular.

[0185] It should be understood that, as used herein, the singular forms "a", "an" are intended to include the plural forms as well, unless the context clearly supports an exception. It should also be understood that, as used herein, "and / or" refers to any and all possible combinations including one or more of the associated listed items.

[0186] The serial numbers of the embodiments disclosed in the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0187] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0188] A person skilled in the art should understand that the discussion of any of the above embodiments is only exemplary and is not intended to imply that the scope of the disclosure of the embodiments of the present invention (including the claims) is limited to these examples; under the concept of the embodiments of the present invention, the technical features in the above embodiments or different embodiments can also be combined, and there are many other changes in different aspects of the above embodiments of the present invention, which are not provided in detail for the sake of simplicity. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present invention should be included in the protection scope of the embodiments of the present invention.

Claims

1. A method for calibrating external parameters of a fusion sensing device, characterized in that: The following steps are involved: Acquire multiple data groups of preset targets, wherein each of the data groups includes the positioning coordinates of the target, the target detection frame collected by at least one first sensor, and the target point cloud data collected by at least one second sensor at the same time; Determine an error weight of each of the target detection boxes based on the target point cloud data in the data group and an internal parameter of each of the first sensors; Constructing a first error function using the extrinsic parameters to be calibrated, the positioning coordinates in each of the data groups, the target detection frame, and the corresponding error weights, and constructing a second error function using the extrinsic parameters to be calibrated, the positioning coordinates in each of the data groups, and the target point cloud data; Calibrate the extrinsic parameter to be calibrated according to the first error function and the second error function; Wherein, determining the error weight of each of the target detection frames based on the target point cloud data in the data group and the internal parameters of each of the first sensors further includes: Determine the distance between the target and each first sensor according to the target point cloud data in the data group; An error weight of each of the target detection frames is determined using the first focal length and the second focal length of the first sensor and the distance.

2. The method according to claim 1, characterized in that Acquiring multiple data groups of preset targets, wherein each of the data groups includes the positioning coordinates of the target, the target detection frame collected by at least one first sensor, and the target point cloud data collected by at least one second sensor at the same time, further comprising: Receiving the positioning coordinates sent by the target, collecting the target image using at least one of the first sensors, and collecting the target point cloud data using at least one of the second sensors; Performing image detection and recognition on each target image captured by at least one of the first sensors to obtain a corresponding target detection frame; The data group is constructed using the positioning coordinates received at the same time, the target detection frame corresponding to at least one of the target images collected, and at least one of the target point cloud data collected.

3. The method according to claim 2, characterized in that Constructing the data group using the positioning coordinates received at the same time, the target detection frame corresponding to at least one of the target images acquired, and at least one of the target point cloud data acquired, further includes: Acquire a collection grid and a regional grid corresponding to each of the first sensors, wherein each second grid in the regional grid is larger than each first grid in the collection grid; Determine in sequence the first position of the center of each target detection frame in the corresponding acquisition grid and the second position in the corresponding regional grid; Determine whether there is a mark in the first grid corresponding to the first position; In response to the absence of a mark, retaining the corresponding target detection frame and marking the corresponding first grid, and marking in the corresponding second grid according to the second position; In response to the existence of the mark, deleting the corresponding object detection box; In response to the proportion of the marked second grids in each of the regional grids reaching a threshold, the construction of the data set is stopped.

4. The method according to claim 1, characterized in that The method further comprises: constructing a first error function using the extrinsic parameters to be calibrated, the positioning coordinates in each of the data groups, the target detection frame and the corresponding error weights, and constructing a second error function using the extrinsic parameters to be calibrated, the positioning coordinates in each of the data groups and the target point cloud data. Acquire a relative extrinsic parameter of each of the first sensors and a relative extrinsic parameter of each of the second sensors; Using the extrinsic parameter to be calibrated and the relative extrinsic parameter of each of the first sensors, the positioning coordinates are converted into a coordinate system of each of the first sensors to obtain a first projection coordinate; Constructing a first error function based on the first projection coordinates, the corresponding target detection frame, and the error weight; Using the extrinsic parameter to be calibrated and the relative extrinsic parameter of each second sensor, the positioning coordinate is converted into the coordinate system of each second sensor to obtain a second projection coordinate; A second error function is constructed based on the second projection coordinates and the corresponding target point cloud data.

5. The method according to claim 4, characterized in that Acquiring a relative extrinsic parameter of each of the first sensors and a relative extrinsic parameter of each of the second sensors further includes: Using each of the first sensors to collect an image of a preset marker and extract a binary image of the preset marker, and using each of the second sensors to collect point cloud data of the preset marker; Randomly select a coordinate system of a second sensor as the base coordinate system; Obtaining a relative extrinsic parameter of each second sensor based on the point cloud data collected by the randomly selected second sensor and the point cloud data collected by the remaining second sensors; constructing a third error function using an intrinsic parameter of each of the first sensors, point cloud data collected by a randomly selected second sensor, and a relative extrinsic parameter of each of the first sensors; A relative extrinsic parameter of each of the first sensors is obtained based on the third error function.

6. The method according to claim 1, characterized in that Calibration of the to-be-calibrated external parameter according to the first error function and the second error function further includes: The external parameter to be calibrated is adjusted until the sum of the first error function and the second error function is minimized, so as to complete the calibration of the external parameter to be calibrated.

7. An aircraft monitoring device, characterized in that: include: at least one first sensor, each of the first sensors being used to collect an object detection frame of an object; at least one second sensor, each of the first sensors being used to collect target point cloud data of a target; Wherein, the aircraft monitoring device performs external parameter calibration based on the target detection frame and the target point cloud data using the method described in any one of claims 1 to 6.

8. A computer device comprising: at least one processor; as well as A memory storing a computer program executable on the processor, wherein the processor executes the steps of the method according to any one of claims 1 to 6 when executing the program.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are performed.

Images