Multi-aperture array imaging system calibration method and target distance measurement method

Abstract

The present invention relates to a target detection method, and more specifically to a multi-aperture array imaging system calibration method and a target distance measurement method. To address the shortcomings of prior art binocular ranging methods, such as complex procedures, high feature point matching accuracy requirements, and large random errors, the multi-aperture array imaging system calibration method of the present invention includes internal parameter separation calibration and external parameter cyclic calibration, separating the internal and external parameters and calibrating them separately. Furthermore, a target distance measurement method is provided based on the multi-aperture array imaging system calibration method, combining a target detection model with the sub-aperture imaging principle for ranging, thereby calculating the final target distance.

Description

Technical Field

[0001] The present invention relates to a target detection method, and in particular to a multi-aperture array imaging system calibration method and a target distance measurement method. Background Art

[0002] The visual system is a crucial organ for all organisms to acquire information about their surroundings. Depending on their living environments and lifestyles, organisms have evolved two main types of visual systems: single-aperture eyes and compound eyes. Single-aperture eyes, which are mostly binocular, have a larger aperture and higher spatial resolution. They are common in most organisms, including vertebrates and birds, and require head rotation to observe their surroundings. Compound eyes, commonly found in insects and arthropods, offer a larger field of view, higher temporal resolution, and are sensitive to moving targets.

[0003] Different types of imaging systems derived from different visual systems also inherit these advantages. To meet the needs of wide-area imaging monitoring, single-aperture fisheye cameras can achieve a very large imaging field of view, but they suffer from severe peripheral distortion and complex post-processing algorithms, limiting their application. Therefore, compound eye imaging systems have become the current research frontier for large-field-of-view detection, and their multi-dimensional detection advantages are also attracting considerable attention.

[0004] Because there is a certain distance between human eyes, parallax exists in the imaging process. Combining human experience and the simple concept of "near is big and far is small", the distance of targets within a certain range can be judged, but there is a large error. The binocular ranging method derived from this requires stereo correction, feature point matching, parallax calculation and other steps for each distance measurement. Among them, the stereo correction process is complex, the feature point matching accuracy requirements are high, and the random error is relatively large. Summary of the Invention

[0005] The purpose of the present invention is to solve the shortcomings of the existing binocular ranging method, such as complex process, high feature point matching accuracy requirements, and large random errors, and to provide a multi-aperture array imaging system calibration method and a target distance measurement method to achieve wide-area distance measurement.

[0006] To achieve the above objectives, the present invention provides the following technical solutions:

[0007] A multi-aperture array imaging system calibration method is characterized in that it includes the following steps:

[0008] Step 1. Separation and calibration of internal parameters;

[0009] Each sub-aperture of the multi-aperture array imaging system is calibrated separately to obtain its internal parameters;

[0010] The multi-aperture array imaging system includes a plurality of sub-apertures for imaging, the plurality of sub-apertures forming a multi-aperture array, and each sub-aperture and a plurality of sub-apertures in its neighborhood forming a cluster eye;

[0011] Step 2. External parameter cyclic calibration;

[0012] Based on the internal parameters of each sub-aperture, the sub-apertures except the outermost layer of the multi-aperture array are cyclically calibrated with cluster eyes as units to obtain their external parameters.

[0013] Furthermore, step 1 is specifically as follows:

[0014] Step 1.1. Use a multi-aperture array imaging system to image the self-identification calibration plate or the Zhang Zhengyou checkerboard calibration plate. Adjust the imaging system angle so that each sub-aperture can capture multiple calibration information images of the self-identification calibration plate or the Zhang Zhengyou checkerboard calibration plate.

[0015] Step 1.2. Calculate the homography matrix of each calibration information image corresponding to the sub-aperture;

[0016] Step 1.3. Solve the simultaneous equations of the homography matrices of the calibration information images of the same sub-aperture to obtain the internal parameters of the sub-aperture;

[0017] Step 1.4. Repeat step 1.3 until the internal parameters of each sub-aperture are obtained.

[0018] Furthermore, step 2 is specifically as follows:

[0019] Step 2.1. Based on the intrinsic parameters of each sub-aperture, calibrate the cluster eye at the center of the multi-aperture array to obtain the extrinsic parameters of each sub-aperture;

[0020] Step 2.2. Use the edge subapertures of the previous cluster as the edge subapertures of the next cluster to be calibrated. Use the next subaperture on the extension line between the center of the subaperture and the center of the multi-aperture array as the center of the next cluster to be calibrated. Calibrate the next cluster to be calibrated and obtain the external parameters of each subaperture in the cluster.

[0021] Step 2.3. Repeat step 2.2 and perform multi-target calibration on a cluster basis to obtain the external parameters of all sub-apertures except the outermost layer of the multi-aperture array.

[0022] Step 2.4. Unify the sub-aperture external parameters based on the center of the multi-aperture array.

[0023] Furthermore, step 2.1 includes the following steps:

[0024] Step 2.1.1. Use a multi-aperture array imaging system to simultaneously image the calibration plate through each sub-aperture of the cluster eye located at the center of the multi-aperture array. Rotate and tilt the calibration plate to obtain multiple images of the calibration plate at different angles for each sub-aperture.

[0025] The positioning calibration plate includes a calibration plate body, and a direction positioning point, a center positioning point, and a main pattern located on the calibration plate body; the direction positioning point includes a plurality of black and white grid points arranged along the periphery of the calibration plate body, which are used to locate the rotation direction of the calibration plate; the center positioning point is set as a black grid point located at the center of the calibration plate body, which is used to locate the center position of the calibration plate; the main pattern includes black and white grid points located between the direction positioning point and the center positioning point, and does not have rotational symmetry, serving as image detection points that can be directly identified;

[0026] Step 2.1.2. Calculate the homography matrix for each sub-aperture corresponding to the positioning calibration plate image, and solve the simultaneous equations for the homography matrices of the positioning calibration plate images corresponding to the same sub-aperture;

[0027] Step 2.1.3. Based on the internal parameters of each sub-aperture, solve the set of equations in 2.1.2 to obtain the external parameters of each sub-aperture in the cluster eye.

[0028] Furthermore, in step 1.1, each sub-aperture collects at least 5 calibration information images; the multiple sub-apertures are arranged in a regular hexagonal honeycomb structure, and all sub-apertures form a regular hexagon;

[0029] Step 2.3 also includes selecting the subaperture at the vertex of the outermost layer of the multi-aperture array or at least four subapertures set at equal intervals in the outermost layer, performing binocular calibration, comparing the calibration results with the empirical values ​​or design values ​​of the corresponding subapertures, and verifying the calibration accuracy of the remaining external parameters in cluster eyes. If the deviation is too large, return to step 2.1 and re-calibrate the external parameters.

[0030] Furthermore, in step 1.1, the self-identification calibration plate is a CALTag self-identification calibration plate; each sub-aperture collects more than 25 calibration information images;

[0031] Step 1.2 is as follows: input a single calibration information image of the sub-aperture into the CALTag parsing algorithm, obtain the corner position parameters of each coding mark in the calibration information image based on corner point detection, index the coding mark of the CALTag self-identification calibration plate, obtain the coding sequence and grid point size corresponding to each coding mark in the calibration information image, and calculate the homography matrix of the sub-aperture corresponding to the calibration information image through the actual size of the grid point and the imaging size of the corresponding grid point in the calibration information image.

[0032] At the same time, the present invention also provides a target distance measurement method, which is special in that it includes the following steps:

[0033] S1. The multi-aperture array imaging system is calibrated by the multi-aperture array imaging system calibration method to obtain the internal and external parameters of each sub-aperture;

[0034] A multi-aperture array imaging system is used to acquire the original compound eye image, where the target to be measured is simultaneously captured by at least four sub-apertures located in the same cluster eye; multiple sub-aperture images of the target to be measured are obtained by preprocessing;

[0035] S2. Use the target detection model to identify multiple sub-aperture images and output the detection parameters corresponding to each sub-aperture of the same cluster eye in step S1. The detection parameters include category, confidence, x-coordinate of the prediction box center, y-coordinate of the prediction box center, width of the prediction box ImgWidth, and height of the prediction box ImgHeight;

[0036] S3. Based on the detection parameters and internal parameters corresponding to each sub-aperture of the same cluster of eyes, the ranging results of each sub-aperture are calculated;

[0037] S4. Calculate the average of the sub-aperture ranging results to obtain the final target distance;

[0038] S5. Output the final target distance.

[0039] Furthermore, step S5 is specifically as follows:

[0040] S5.1. Reconstruct the original compound eye image into a large field of view image.

[0041] S5.2. Select the set of detection parameters with the highest confidence in step S2;

[0042] S5.3. Draw a prediction box on the large field of view image based on the detection parameters selected in step S5.2, and mark the target category, confidence level, and final target distance in the detection parameters selected in step S5.2 around the prediction box.

[0043] Furthermore, step S3 is specifically as follows: the target distance of the target to be measured in the sub-aperture of the corresponding cluster eye is calculated by the following formula: ;

[0044]

[0045]

[0046]

[0047] in, is the estimated width of the target to be measured, is the estimated height of the target to be measured, is the target distance of the target to be measured in the x-axis direction of the sub-aperture of the corresponding cluster eye, is the target distance of the target to be measured in the y-axis direction of the sub-aperture of the corresponding cluster eye, k is the prediction box scaling factor of the target to be measured in the sub-aperture of the corresponding cluster eye, x is the focal length of the subaperture in the x-axis direction, f y is the focal length of the subaperture in the y-axis direction;

[0048] Step S4 specifically comprises: selecting the ranging results corresponding to the first n sub-aperture images arranged in descending order of confidence from the multiple sub-aperture images in step S2, where n≥4, and calculating their average value to obtain the final target distance.

[0049] Furthermore, in step S1, the multi-aperture array imaging system is a divided-aperture compound-eye imaging system or a multi-camera array compound-eye imaging system;

[0050] In step S2, the target detection model is a cross-aperture target detection network model, which is trained by the following process: obtaining multiple target images of the same type as the target to be detected, manually annotating them and dividing them into a training set and a validation set, training and validating the target detection network to obtain a cross-aperture target detection network model; the target detection network is a YOLOv8 model; the multiple target images respectively include targets under different lighting conditions and different backgrounds, and the different lighting conditions include overexposure, underexposure, and normal exposure;

[0051] The target image is acquired by a multi-aperture array imaging system, specifically by imaging the target through the multi-aperture array imaging system to obtain a compound eye original image, pre-processing the compound eye original image to obtain the center coordinate position and sub-aperture radius of each sub-aperture, and obtaining multiple sub-aperture images, and performing distortion correction on the sub-aperture images to obtain the target image;

[0052] In step S4, n=5.

[0053] Beneficial effects of the present invention:

[0054] 1. In the multi-aperture array imaging system calibration method of the present invention, an innovative distributed cyclic calibration method is set up for the multi-aperture array imaging system, each sub-aperture is treated as an independent camera for distributed calibration, and the internal parameters and external parameters are separated and calibrated separately, thereby improving the calibration accuracy of the internal and external parameters.

[0055] 2. In the multi-aperture array imaging system calibration method of the present invention, the internal parameters of each sub-aperture are calibrated with high precision using a self-identifying calibration plate, while the external parameters are cyclically calibrated and optimized from the inside out using "cluster eyes" as units using a self-made positioning calibration plate. This solves the problem of reduced internal parameter accuracy caused by the unified optimization of internal and external parameter results in traditional multi-camera calibration methods.

[0056] 3. The target distance measurement method of the present invention adopts a multi-aperture array imaging system to obtain the original image of the compound eye. The multi-aperture array imaging system is designed and manufactured to simulate the biological compound eye, and can achieve large field of view imaging. There is a certain field of view overlap between different sub-apertures, so that the same target can be captured by multiple sub-apertures, realizing reliable detection of the target. At the same time, based on the detection parameters of the target, the ranging result can be calculated according to the imaging principle, without the need for complex steps such as stereo correction.

[0057] 4. The target distance measurement method of the present invention measures the target distance based on the prediction frame output after target detection and the internal parameters of the sub-aperture. Compared with the binocular ranging method and depth estimation network that calculates "parallax", the distance measurement calculation is simpler, the number of parameters is smaller, and it is more suitable for deployment and integration in a multi-aperture array imaging system.

[0058] 5. The target distance measurement method of the present invention utilizes a cross-aperture target detection network model, trained using a training set and validation set constructed from compound eye raw images, effectively incorporating the unique characteristics of the compound eye imaging system. The multi-aperture array imaging system can capture the same target multiple times using a single frame of the compound eye raw image. The target in each of the multiple sub-aperture images is consistent, but the relative position and corresponding background differ. This dataset inherently implements hardware-level data enhancement, ensuring that all target information is real. This significantly improves the reliability and accuracy of data enhancement compared to traditional methods such as translation, rotation, and cropping.

[0059] 6. The target distance measurement method of the present invention uses deep learning network training on the target detection network to obtain a cross-aperture target detection network model. During the training process, target images are acquired under different lighting conditions and backgrounds, effectively improving the robustness of the detection network. Furthermore, the multi-aperture array imaging system can capture and detect the same target multiple times, greatly improving detection reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 is a schematic diagram of a flow chart of an embodiment of the present invention;

[0061] Figure 2 Schematic diagram of the CALTag self-identification calibration board in step 1.1 of an embodiment of the present invention;

[0062] Figure 3is a schematic diagram of the positioning calibration plate in step 1.2 of an embodiment of the present invention;

[0063] Figure 4 is a schematic diagram of subapertures of a multi-aperture array imaging system according to an embodiment of the present invention;

[0064] Figure 5 is a schematic diagram of step 1.2 of an embodiment of the present invention;

[0065] Figure 6 Schematic diagram of the imaging principle of a multi-aperture array imaging system according to an embodiment of the present invention;

[0066] Figure 7 3 is a schematic diagram of a sub-aperture image of the target to be measured in step 3.1 of an embodiment of the present invention; wherein A is the original image of the compound eye of the target to be measured, and B is an enlarged image of the sub-aperture image of the eye located in the center of A;

[0067] Figure 8 Schematic diagram of the imaging principle of the subaperture in step 3.1 of an embodiment of the present invention; wherein A is the imaging principle in the x-direction, and B is the imaging principle in the y-direction;

[0068] Figure 9 3. It is the large field of view image reconstructed in step 3.2 of the embodiment of the present invention, wherein A is the reconstructed large field of view image, and B is a partial enlarged schematic diagram of A. DETAILED DESCRIPTION

[0069] The target distance measurement method of the present invention uses a multi-aperture array imaging system to acquire a target image. The multi-aperture array imaging system includes multiple sub-apertures for imaging. The multiple sub-apertures form a multi-aperture array. The center of the multi-aperture array is the center of the multi-aperture array imaging system. The sub-apertures do not crosstalk with each other and are each imaged in an independent area of ​​the same detector. There is a large overlap in the field of view between the sub-apertures, allowing the same target to be captured by multiple sub-apertures. That is, a single-frame compound eye original image can contain multiple sub-aperture images of the same target. The multi-aperture array imaging system of the present invention can be configured as a sub-aperture compound eye imaging system or a multi-camera array compound eye imaging system, capable of achieving low-distortion and large-field-of-view imaging.

[0070] In this embodiment, a sub-aperture compound eye imaging system is used, specifically a biomimetic curved compound eye imaging system, wherein multiple sub-apertures are arranged in a regular hexagonal honeycomb structure to form a multi-aperture array. All sub-apertures form a regular hexagon, and each sub-aperture and multiple sub-apertures within its neighborhood constitute a cluster eye. Specifically, a cluster eye includes its central sub-aperture and six surrounding sub-apertures located at the vertices of the honeycomb structure. The cluster eye can capture the same target during imaging. In other embodiments of the present invention, the sub-apertures of each cluster eye can be arranged in other ways, such as circular, square, regular octagonal, etc. Each cluster eye includes at least four sub-apertures, and at least four sub-apertures can capture the same target. Other imaging systems having all of the above features can also be used.

[0071] like Figure 1 As shown in the figure, the target distance measurement method of the present invention realizes target detection through a deep learning network and calculates the target distance based on sub-aperture detection parameters and camera internal and external parameters. It mainly includes three parts:

[0072] 1. Calibration of distributed circular cameras in multi-aperture array imaging systems;

[0073] like Figure 1 As shown in the figure, the distributed cyclic camera calibration method for multi-aperture array imaging system mainly includes two stages: internal parameter separation calibration based on sub-apertures and external parameter cyclic calibration based on cluster eyes.

[0074] The internal parameters include the focal length of the subaperture in the x-axis direction ( f x ), the focal length of the sub-aperture in the y-axis direction ( f y ), principal point coordinate position ( u 0, v 0 ), distortion parameters. The external parameters of each sub-aperture include its rotation matrix and translation matrix relative to the sub-aperture at the center of the multi-aperture array.

[0075] In the distributed cyclic camera calibration method for multi-aperture array imaging systems, the distributed method unifies the internal and external parameters in the traditional camera calibration method for calibration and optimization, so that the internal and external parameters no longer restrict each other during the calibration and optimization process. The internal parameters with higher accuracy are first calibrated, and then the external parameters with equally high accuracy are directly calculated by combining the internal parameters, thereby improving the calibration accuracy.

[0076] 1.1. Separation and calibration of internal parameters based on sub-aperture;

[0077] Sub-aperture internal parameter separation calibration refers to separating each sub-aperture in a multi-aperture array imaging system and calibrating it as a camera to obtain the internal parameters of each sub-aperture. The specific steps are as follows:

[0078] 1) If Figure 2 As shown in the figure, a multi-aperture array imaging system is used to image the CALTag (CALibration Tag) self-identification calibration plate to obtain a compound eye raw image. An edge detection algorithm is used to detect the image edges corresponding to each sub-aperture in the compound eye raw image. The obtained edges are used to obtain the center coordinate position and sub-aperture radius of each sub-aperture. The compound eye raw image is then segmented based on the sub-aperture center coordinate position and sub-aperture radius to obtain the calibration information image for each sub-aperture. The imaging system angle is adjusted so that each sub-aperture can capture multiple calibration information images of the CALTag self-identification calibration plate.

[0079] In this embodiment, each sub-aperture collects more than 25 calibration information images to ensure the accuracy of internal parameter calibration; in other embodiments of the present invention, at least 5 calibration information images may be collected.

[0080] The CALTag self-identification calibration plate consists of black and white grid points of known size, such as Figure 2 As shown in the figure, black and white grids form a coding mark that looks like a QR code. The coding mark of each area is automatically generated by the CALTag generation algorithm, and the coding mark information such as the corresponding grid point size, corner point position parameters, and coding sequence are recorded in the coding mark.

[0081] In this embodiment, the CALTag self-identification calibration plate is used. In other embodiments of the present invention, the Zhang Zhengyou checkerboard calibration plate or other calibration plates may be used.

[0082] 2) Each calibration image is fed into the CALTag parsing algorithm. Corner point detection is used to obtain the corner position parameters of each coded marker in the calibration image. The coded markers on the CALTag self-identification calibration plate are indexed to obtain the corresponding code sequence and grid size for each coded marker in the calibration image. The homography matrix for the subaperture corresponding to the calibration image is calculated by comparing the actual grid size with the imaged size of the corresponding grid point in the calibration image.

[0083] When each of the above sub-apertures obtains the calibration information image, the coding mark on the image is only a part of the coding mark of the actual CALTag self-identification calibration plate, so an index is required to determine the corresponding coding sequence.

[0084] 3) Solve the simultaneous equations of the homography matrices of the calibration information images of the same sub-aperture. Since the rotation matrices in the external parameters are mutually orthogonal at this time, this equation group is solved to obtain the internal parameters of the sub-aperture.

[0085] 4) Repeat step 3) until the internal parameters of each sub-aperture are obtained.

[0086] 1.2. External parameter cyclic calibration based on cluster eyes;

[0087] Cluster-eye-based external parameter cyclic calibration refers to the unified calibration of external parameters based on cluster eyes to reduce their impact on internal parameters during the optimization process. The specific steps are as follows:

[0088] 1) Perform multi-target positioning on the cluster eye at the center of the multi-aperture array to obtain the external parameters of each sub-aperture. Specifically:

[0089] A multi-aperture array imaging system is used to image the positioning calibration plate to obtain the original compound eye image, which is then segmented to obtain the positioning calibration plate image corresponding to each sub-aperture.

[0090] Because the multi-aperture array imaging system has a large field of view, when a sub-aperture clearly images the calibration plate, only the surrounding sub-apertures can capture the calibration plate image, and not all sub-apertures can be covered. Therefore, during imaging, it is necessary to control the sub-apertures of the cluster so that they simultaneously image the calibration plate.

[0091] The positioning calibration plate can be customized according to actual needs. Figure 3 As shown in the figure, the requirements for positioning the calibration plate are as follows: positioning points composed of different black and white grids are set on the periphery to locate the rotation direction of the calibration plate; a central positioning point represented by a black grid in the center is used to locate the center position of the calibration plate; the main pattern of the calibration plate is composed of black and white grids and does not have rotational symmetry, which is used to provide sufficient image detection points that can be directly identified.

[0092] Rotate and tilt the calibration plate to change its angle, and capture several images of the calibration plate at different angles. Using the sub-aperture image coordinates and the known distances between the corner points of the calibration plate, calculate the homography matrix for the calibration plate image corresponding to the sub-aperture. Then, use the homography matrices of the calibration plate images corresponding to the same sub-aperture to solve a simultaneous equation system. In this embodiment, at least 100 calibration plate images are captured for each eye cluster to ensure the accuracy of extrinsic parameter calibration.

[0093] Using the internal parameters of the corresponding sub-aperture obtained in step 1.1, solve the set of simultaneous equations to obtain the external parameters of the corresponding sub-aperture, so as to solve the problem of accuracy degradation caused by the optimization of the external parameters of the multi-camera in the traditional method.

[0094] 2) To avoid data redundancy, the edge sub-aperture of the previous cluster eye is used as the edge sub-aperture of the next cluster eye to be calibrated. The next sub-aperture on the extension line of the sub-aperture center and the center of the multi-aperture array is used as the center of the next cluster eye to be calibrated. The corresponding cluster eye is calibrated to obtain the external parameters of each sub-aperture in the cluster eye.

[0095] like Figure 4 and Figure 5As shown in the figure, the sub-apertures are divided into layers according to the Euclidean distance between the center coordinates of the sub-aperture and the center coordinates of the multi-aperture array, and the sub-apertures of each layer are numbered in a counterclockwise direction. Figure 4 Taking subaperture 2 as an example, when it is used as the edge subaperture of the next cluster eye to be calibrated, the next subaperture on the extension line of the multi-aperture array center (the center of subaperture 1), that is, subaperture 9, is used as the center of the next cluster eye to be calibrated. The corresponding cluster eye consisting of subapertures 2, 8, 9, 10, 20, 21, and 22 is calibrated to obtain the external parameters of each subaperture in this cluster eye. The image sequence number corresponding to each subaperture in the original compound eye image is the same as the sequence number of the corresponding subaperture.

[0096] 3) Repeat step 2) and perform external parameter calibration on all sub-apertures except the outermost sub-aperture of the multi-aperture array, taking the cluster eye as the unit, to achieve cyclic calibration of the sub-aperture external parameters.

[0097] 4) After cluster eye calibration, the six outermost subapertures of the multi-aperture array, located at the vertices of the regular hexagon, are calibrated for extrinsic parameters using binocular calibration. Binary calibration uses the vertex subaperture and its adjacent subapertures, which have undergone extrinsic parameter calibration in the above steps, as units.

[0098] Because the distortion of the outermost subaperture is too large, affecting the accuracy of the ranging results, imaging is primarily performed using subapertures other than the outermost one, and external parameter calibration can be omitted for the outermost subaperture. Multi-target timing performed on a cluster-eye basis inevitably reduces the calibration accuracy of some subapertures due to the global optimization of the external parameters of each subaperture. Compared to multi-target calibration, dual-target calibration is more reliable but less efficient. To further improve the accuracy of external parameter calibration, this embodiment performs dual-target calibration on the subaperture at the outermost vertex. The dual-target calibration results are compared with the empirical or designed values ​​of the corresponding subaperture to reversely verify the accuracy of the external parameter calibration on a cluster-eye basis. If the deviation is too large, recalibration can be performed to reduce the error of the external parameter cyclic calibration.

[0099] 5) Taking the center of the multi-aperture array as the reference and the edge sub-apertures calibrated in adjacent clusters as the intermediary, the external parameters of all calibrated sub-apertures are unified through matrix calculation to obtain the external parameters of each sub-aperture based on the central sub-aperture.

[0100] 2. Train the cross-aperture target detection network model;

[0101] like Figure 1 As shown in the figure, cross-aperture target detection is mainly divided into two stages: dataset construction based on sub-aperture images and target detection network training based on deep learning.

[0102] 2.1. Construction of a dataset based on sub-aperture images;

[0103] The construction of a dataset based on sub-aperture images refers to collecting target images of the same type as the target to be measured, annotating the categories and boundaries, and then inputting the sub-aperture image set into the deep learning network for training and verification. The specific steps are:

[0104] 1) Detection training image capture;

[0105] The multi-aperture array imaging system is used to image targets under different lighting conditions and backgrounds, generating raw compound eye images. These images can be over-exposed, under-exposed, or normally exposed.

[0106] In this embodiment, a multi-aperture array imaging system is used to image a human target. In other embodiments of the present invention, other types of targets such as vehicles, riders, bicycles, and trees may be imaged.

[0107] 2) Preprocessing of compound eye raw images;

[0108] The edge detection algorithm is used to detect the edge of each sub-aperture image on the original compound eye image, and the sub-aperture center coordinate position and the corresponding sub-aperture radius are calculated, and the original compound eye image is divided into multiple sub-aperture images.

[0109] Then, according to the distortion parameters in the camera calibration internal parameters, the sub-aperture image is distorted and the target image is obtained as the detection training image.

[0110] 3) Dataset annotation;

[0111] All targets contained in the target image are selected to complete the annotation. The annotated image is set as the ground truth to require the annotation box to fit closely to the edge of the target, and the annotation label is exported in text form.

[0112] In this embodiment, the annotation label includes the category, the center position of the annotation box, the width of the annotation box, and the height of the annotation box, and is saved in the form of a ".txt" file.

[0113] 4) Dataset partitioning;

[0114] Redundant images that do not contain any targets are removed, the labeled target images are uniformly numbered, and divided into training sets and validation sets according to a certain ratio.

[0115] In this embodiment, the ratio of the training set to the validation set is set to 8:2. In other embodiments of the present invention, the ratio can be adjusted according to one's own needs.

[0116] 2.2. Object detection network training based on deep learning;

[0117] Deep learning-based target detection network training refers to the detection of targets through deep learning, combined with the characteristics of multi-aperture array imaging systems, target categories, etc. The specific steps are:

[0118] 1) Select the network architecture of the target detection network. In this embodiment, the network architecture of the target detection network is the existing YOLOv8 model, which consists of three parts: the backbone network (Backbone), the neck network (Neck), and the detection head network (Head).

[0119] 2) The training set is used as input to the backbone network, neck network, and detection head network to perform deep learning network training on the object detection network. The validation set is then input for verification to obtain the trained object detection network, which serves as the cross-aperture object detection network model.

[0120] 3. Multi-aperture target detection distance measurement;

[0121] like Figure 1 As shown in the figure, the distance measurement of multi-aperture target detection is mainly divided into two stages: distance calculation based on cross-aperture target detection results and result output based on large field of view reconstruction image. Specifically, it includes the following steps:

[0122] 3.1. Ranging calculation based on cross-aperture target detection results;

[0123] The distance calculation based on the cross-aperture target detection result is to obtain the target prediction frame through target detection, and calculate the target distance based on the prediction frame width, prediction frame height, actual target size and sub-aperture calibration internal parameters. The specific steps are:

[0124] 1) If Figure 6 As shown in FIG, a multi-aperture array imaging system is used to image the target to be measured, and the compound eye original image is obtained as the target image to be measured. Since there is a certain overlap in the field of view between the sub-apertures of the multi-aperture array imaging system, the same target to be measured is captured by multiple sub-apertures at the same time. Figure 7 As shown, multiple sub-aperture images of the target to be measured are obtained by pre-processing the original compound eye image in step 2.1. In this embodiment, the same target to be measured is captured simultaneously by 7 sub-apertures located in the same cluster eye.

[0125] 2) Using the seven sub-apertures that capture the same target as the basic unit, input the seven sub-aperture images into the cross-aperture target detection network model trained and derived in step 2.2 to perform target detection and output seven sets of detection parameters corresponding to the target. Each set of detection parameters includes the category (Class), confidence (Confidence), the x-coordinate of the center of the prediction box (x0), the y-coordinate of the center of the prediction box (y0), the width of the prediction box (ImgWidth), and the height of the prediction box (ImgHeight). The width of the prediction box (ImgWidth) and the height of the prediction box (ImgHeight) are calculated according to the number of pixels detected using the following formula:

[0126]

[0127]

[0128] in, is the detector pixel size, Indicates the number of pixels occupied by the predicted frame width of the target to be measured in the sub-aperture of the corresponding cluster eye, It represents the number of pixels occupied by the predicted box height of the target in the sub-aperture of the corresponding cluster eye.

[0129] 3) If Figure 8 As shown, according to the imaging principle of each sub-aperture in the multi-aperture array imaging system, each sub-aperture can be regarded as a pinhole imaging model. Combined with the internal parameters obtained in step 1.1, the ranging calculation is performed in the following way to obtain the ranging results of each sub-aperture in the cluster eye corresponding to the target to be measured, that is, the target distance of the target to be measured in the sub-aperture of the corresponding cluster eye :

[0130]

[0131]

[0132]

[0133]

[0134] in, is the estimated width of the target to be measured, is the estimated height of the target to be measured, is the target distance of the target to be measured in the x-axis direction of the sub-aperture of the corresponding cluster eye, is the target distance of the target to be measured in the y-axis direction of the sub-aperture of the corresponding cluster eye, k is the prediction box scaling factor of the target to be measured in the sub-aperture of the corresponding cluster eye, is the target distance of the target to be measured in the sub-aperture of the corresponding cluster eye.

[0135] During the dataset construction process, in order to ensure that the annotation box contains all the features of the target to be detected, when manually annotating the target, it is inevitable that the annotation box size will be slightly larger than the actual size of the target. This also leads to the prediction box being too large during the target detection network training process. Therefore, a prediction box scaling factor is required. k Offset the effect of the larger prediction box.

[0136] In this embodiment, humans are used as the target for training and detection. For example, an adult male is used. The number of pixels occupied by the real target at 100m is 22 × 8 pixels, and the corresponding human body shape is 1.7 × 0.5 m. Therefore, the scaling factor in this embodiment is k Since the present invention is used for long-distance ranging over 100 meters, the width and height of the target to be measured are estimated according to its type in the calculation, and the error in the final result can be ignored.

[0137] 4) Obtain seven sets of detection parameters and ranging results. Since they all correspond to the ranging results of the same target to be measured, their distances relative to the multi-aperture array imaging system are approximately consistent. To eliminate random errors in the target detection and ranging calculation process, the ranging results corresponding to the n sub-aperture images with the highest confidence among the seven sets of detection parameters are taken. In this embodiment, n=5, and the average is calculated as the final target distance. .

[0138]

[0139] in, is the target distance in the x-axis direction of the i-th sub-aperture in the n sub-aperture images with the largest confidence, is the target distance in the y-axis direction of the i-th sub-aperture among the n sub-aperture images with the largest confidence.

[0140] In other embodiments of the present invention, to ensure the accuracy of the final target distance, preferably n≥4.

[0141] 3.2. Output of image reconstruction results based on large field of view;

[0142] The result output based on the large field of view reconstructed image means that the detection parameters of each sub-aperture and the final target distance in step 3.1 are output and displayed in the large field of view reconstructed image. The specific steps are:

[0143] 1) According to the imaging principle of the multi-aperture array imaging system, through the spatial mapping relationship between the corresponding points in the image space and the object space, the original compound eye images that are separated and have a certain field of view overlap are reconstructed into a large field of view image suitable for human eyes, such as Figure 9 shown.

[0144] 2) Among the seven sets of detection parameters corresponding to the same target in step 3.1, select the set of detection parameters with the highest confidence.

[0145] 3) Draw the prediction box in the reconstructed large field of view image, and output the target category, confidence and final target distance around the prediction box, such as Figure 9 As shown, the final target distance of this embodiment is 101.88m, the target category is Person, and the confidence level is 0.88.

Claims

1. A multi-aperture array imaging system calibration method, characterized in that: The following steps are involved: Step 1. Separation and calibration of internal parameters; Step 1.

1. Use a multi-aperture array imaging system to image the self-identification calibration plate or the Zhang Zhengyou checkerboard calibration plate. Adjust the imaging system angle so that each sub-aperture can capture multiple calibration information images of the self-identification calibration plate or the Zhang Zhengyou checkerboard calibration plate. The multi-aperture array imaging system includes a plurality of sub-apertures for imaging, the plurality of sub-apertures forming a multi-aperture array, and each sub-aperture and a plurality of sub-apertures in its neighborhood forming a cluster eye; Step 1.

2. Calculate the homography matrix of each calibration information image corresponding to the sub-aperture; Step 1.

3. Solve the simultaneous equations of the homography matrices of the calibration information images of the same sub-aperture to obtain the internal parameters of the sub-aperture; Step 1.

4. Repeat step 1.3 until the internal parameters of each sub-aperture are obtained; Step 2. External parameter cyclic calibration; Step 2.

1. Obtain the external parameters of each sub-aperture in the cluster eye located at the center of the multi-aperture array; specifically: Step 2.1.

1. Use a multi-aperture array imaging system to simultaneously image the calibration plate through each sub-aperture of the cluster eye located at the center of the multi-aperture array. Rotate and tilt the calibration plate to obtain multiple images of the calibration plate at different angles for each sub-aperture. The positioning calibration plate includes a calibration plate body, and a direction positioning point, a center positioning point, and a main pattern located on the calibration plate body; the direction positioning point includes a plurality of black and white grid points arranged along the periphery of the calibration plate body, which are used to locate the rotation direction of the calibration plate; the center positioning point is set as a black grid point located at the center of the calibration plate body, which is used to locate the center position of the calibration plate; the main pattern includes black and white grid points located between the direction positioning point and the center positioning point, and does not have rotational symmetry, serving as image detection points that can be directly identified; Step 2.1.

2. Calculate the homography matrix for each sub-aperture corresponding to the positioning calibration plate image, and solve the simultaneous equations for the homography matrices of the positioning calibration plate images corresponding to the same sub-aperture; Step 2.1.

3. Based on the internal parameters of each sub-aperture, solve the system of equations in 2.1.2 to obtain the external parameters of each sub-aperture in the cluster eye; Step 2.

2. Use the edge subaperture of each cluster eye as the edge subaperture of the next cluster eye to be calibrated. Use the subaperture located on the extension of the line connecting the center of the edge subaperture and the center of the multi-aperture array and close to the edge subaperture as the center of the next cluster eye to be calibrated. Calibrate the next cluster eye to obtain the external parameters of each subaperture in the cluster eye. Step 2.

3. Repeat step 2.2 and perform multi-target calibration on a cluster basis to obtain the external parameters of all sub-apertures except the outermost layer of the multi-aperture array. Step 2.

4. Unify the sub-aperture external parameters based on the center of the multi-aperture array.

2. The multi-aperture array imaging system calibration method according to claim 1, characterized in that: In step 1.1, each sub-aperture collects at least 5 calibration information images; multiple sub-apertures are arranged in a regular hexagonal honeycomb structure, and all sub-apertures form a regular hexagon; Step 2.3 also includes selecting the subaperture at the vertex of the outermost layer of the multi-aperture array or at least four subapertures set at equal intervals in the outermost layer, performing binocular calibration, comparing the calibration results with the empirical values ​​or design values ​​of the corresponding subapertures, and verifying the calibration accuracy of the remaining external parameters in cluster eyes. If the deviation is too large, return to step 2.1 and re-calibrate the external parameters.

3. The multi-aperture array imaging system calibration method according to claim 2, wherein: In step 1.1, the self-identification calibration plate is a CALTag self-identification calibration plate; each sub-aperture collects more than 25 calibration information images; Step 1.2 is as follows: input a single calibration information image of the sub-aperture into the CALTag parsing algorithm, obtain the corner position parameters of each coding mark in the calibration information image based on corner point detection, index the coding mark of the CALTag self-identification calibration plate, obtain the coding sequence and grid point size corresponding to each coding mark in the calibration information image, and calculate the homography matrix of the sub-aperture corresponding to the calibration information image through the actual size of the grid point and the imaging size of the corresponding grid point in the calibration information image.

4. A target distance measurement method, characterized in that: The following steps are involved: S1. Calibrate the multi-aperture array imaging system by the multi-aperture array imaging system calibration method according to any one of claims 1-3 to obtain the internal and external parameters of each sub-aperture; A multi-aperture array imaging system is used to acquire the original compound eye image, where the target to be measured is simultaneously captured by at least four sub-apertures located in the same cluster eye; multiple sub-aperture images of the target to be measured are obtained by preprocessing; S2. Use the target detection model to identify multiple sub-aperture images and output the detection parameters corresponding to each sub-aperture of the same cluster eye in step S1. The detection parameters include category, confidence, x-coordinate of the prediction box center, y-coordinate of the prediction box center, width of the prediction box ImgWidth, and height of the prediction box ImgHeight; S3. Based on the detection parameters and internal parameters corresponding to each sub-aperture of the same cluster of eyes, the ranging results of each sub-aperture are calculated; S4. Calculate the average of the sub-aperture ranging results to obtain the final target distance; S5. Output the final target distance.

5. The target distance measurement method according to claim 4, characterized in that: Step S5 is specifically as follows: S5.

1. Reconstruct the original compound eye image into a large field of view image. S5.

2. Select the set of detection parameters with the highest confidence in step S2; S5.

3. Draw a prediction box on the large field of view image based on the detection parameters selected in step S5.2, and mark the target category, confidence level, and final target distance in the detection parameters selected in step S5.2 around the prediction box.

6. The target distance measurement method according to claim 4 or 5, characterized in that: Step S3 is specifically as follows: the target distance L1 of the target to be measured at the sub-aperture of the corresponding cluster eye is calculated by the following formula; Among them, Width is the estimated width of the target to be measured, Height is the estimated height of the target to be measured, L x is the target distance of the target to be measured in the x-axis direction of the sub-aperture of the corresponding cluster eye, L y is the target distance of the target to be measured in the y-axis direction of the sub-aperture of the corresponding cluster eye, k is the prediction box scaling factor of the target to be measured in the sub-aperture of the corresponding cluster eye, f x is the focal length of the subaperture in the x-axis direction, f y is the focal length of the subaperture in the y-axis direction; Step S4 specifically comprises: selecting the ranging results corresponding to the first n sub-aperture images arranged in descending order of confidence from the multiple sub-aperture images in step S2, where n≥4, and calculating their average value to obtain the final target distance.

7. The target distance measurement method according to claim 6, characterized in that: In step S1, the multi-aperture array imaging system is a divided-aperture compound-eye imaging system or a multi-camera array compound-eye imaging system; In step S2, the target detection model is a cross-aperture target detection network model, which is trained by the following process: obtaining multiple target images of the same type as the target to be detected, manually annotating them and dividing them into a training set and a validation set, training and validating the target detection network to obtain a cross-aperture target detection network model; the target detection network is a YOLOv8 model; the multiple target images respectively include targets under different lighting conditions and different backgrounds, and the different lighting conditions include overexposure, underexposure, and normal exposure; The target image is acquired by a multi-aperture array imaging system, specifically by imaging the target through the multi-aperture array imaging system to obtain a compound eye original image, pre-processing the compound eye original image to obtain the center coordinate position and sub-aperture radius of each sub-aperture, and obtaining multiple sub-aperture images, and performing distortion correction on the sub-aperture images to obtain the target image; In step S4, n=5.

Images