Multi-camera cooperative state estimation method based on active infrared visual features

Abstract

The invention discloses a multi-camera cooperative state estimation method based on active infrared visual features, and belongs to the technical field of image processing and computer vision. The existing visual marking technology depends on ambient light and is poor in performance under dark light or complex illumination, the observation visual field of a single camera or a fixed camera is limited, distortion exists, and consequently the pose resolving precision and robustness are insufficient. According to the method, active infrared visual feature marking is adopted, dependence on ambient light is avoided, and feature point information is stably provided. And efficient and accurate feature point identification and matching are realized through multi-camera collaborative observation in combination with binarization processing of dynamic threshold adjustment, geometric feature screening and a matching algorithm. And carrying out pose calculation by using a PnP algorithm, and checking the goodness of fit between the axis and the gravity direction. Meanwhile, multiple cameras are dynamically scheduled to rotate cooperatively according to the observation quality, and the observation view is expanded. The method is mainly used for accurate positioning of targets such as an unmanned aerial vehicle, significantly improves the environmental adaptability and observation stability of the system, reduces the deployment cost, and improves the pose calculation precision and robustness.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing and computer vision, and particularly relates to a multi-camera cooperative state estimation method based on active infrared visual features. BACKGROUND

[0002] In the rapid development of robot technology and unmanned aerial vehicle cooperative systems, accurate identification and real-time positioning of moving targets have become a key technical bottleneck for further development in this field. Visual feature observation, as a core means of obtaining target pose information, captures visual features on the target through a camera, and accurately determines the relative position relationship between the robot and the target through feature recognition and pose solving, providing a solid technical support for the autonomous navigation, obstacle avoidance and cooperative work of unmanned aerial vehicles. However, the existing visual feature observation technology still faces many challenges in dealing with complex dynamic environments.

[0003] Currently, visual marker features are mainly divided into passive and active types according to their working principles, and have been widely explored in multi-camera cooperative technology. Passive visual markers, such as AprilTag and its improved version AprilTag2 proposed by Olson et al., achieve high pose estimation accuracy by extracting edge black and white corner positions and combining PnP algorithm, but they are easily affected by environmental light intensity, and the feature recognition stability decreases significantly in dark or complex light source conditions, making it difficult to meet the use requirements in harsh environments. This limitation directly restricts the application effect of passive markers in complex environments.

[0004] Active visual marker technology, such as the arrangement of active infrared light points on unmanned aerial vehicles by Wenzel et al., and the dual-source state estimation method combining YOLOv4 network and active infrared light points by Wang et al., has improved the environmental adaptability to some extent, but still has significant defects. The observation scheme of a single fixed camera is limited by the observation field of view and camera distortion, making it difficult to ensure the stability and robustness of the state estimation results, especially when dealing with high-speed moving targets. In addition, multi-camera co-observation schemes, such as motion capture systems Kinect, Vicon, etc., achieve millimeter-level state estimation accuracy by arranging multiple fixed cameras, but the high cost, reduced environmental adaptability and poor dynamic performance limit their flexible deployment in dynamic scenes.

[0005] In comparison with existing patent technologies, we found that CN115223023B proposes a human contour estimation method based on stereo vision and deep neural networks. Although this method improves the accuracy of human contour estimation, its camera layout relies on a pre-set fixed layout and lacks dynamic adjustment capability. In complex or dynamic environments, a fixed layout may not meet the changing observation needs, leading to a decrease in estimation accuracy. For example, in the cooperative operation of a drone swarm, a fixed camera layout may not adapt to the rapid movement of targets or changes in complex environments, thereby affecting the accuracy of state estimation. In addition, although this method uses deep neural networks for feature extraction and pose estimation, real-time performance may be affected when dealing with high-speed moving targets, and the delay in data synchronization and processing may become a problem.

[0006] On the other hand, CN118570312B proposes a multi-camera cooperative calibration method suitable for dynamic vision sensors, which improves calibration accuracy but has a relatively complex calibration process that requires high-precision hardware support and complex calculation processes, increasing the difficulty and cost of practical applications. Meanwhile, under extreme or rapidly changing environmental conditions, maintaining the sustained effectiveness and accuracy of calibration remains a challenge. Dynamic vision sensors are sensitive to environmental changes, and calibration results may quickly become invalid due to environmental factors, which is a problem that cannot be ignored in practical applications.

[0007] To overcome the shortcomings of existing technologies, the present invention proposes a multi-camera cooperative estimation method based on active infrared visual features. This method, through innovative marker design and multi-camera scheduling strategy, aims to achieve accurate identification and positioning of moving targets in complex environments, further improving the accuracy and stability of state estimation in harsh environments, and making up for the shortcomings of existing technologies in dynamic adaptability and environmental compatibility. SUMMARY

[0008] The technical problem to be solved by the present invention is to provide a multi-camera cooperative state estimation method based on active infrared visual features, to solve the problem of insufficient accuracy and robustness in the field of moving target identification, real-time positioning and attitude calculation between cooperative systems, especially between unmanned aerial vehicles and ground observation platforms. Specifically, passive visual markers in existing technologies are easily affected by environmental light, and their performance decreases in dark or complex lighting conditions. Single fixed camera solutions are limited by observation fields of view and camera distortion, making it difficult to ensure the stability and robustness of state estimation results. Multi-fixed camera solutions, although improving accuracy, have problems such as high cost, reduced environmental adaptability and poor dynamic performance. Therefore, through the method of the present invention, the above-mentioned specific limitations in existing technologies are overcome, achieving high-precision and high-robustness target identification and pose calculation.

[0009] To achieve the above technical objectives, the present invention adopts the following technical solutions: The multi-camera cooperative state estimation method based on active infrared visual features of the application realizes high-precision and high-robustness state estimation of a moving target through the whole-process technical innovation of "active infrared marker design-image feature processing-pose solution-multi-camera cooperative scheduling", and specifically includes the following key links: 1. Active infrared visual marker design A specific geometric shape "T" shaped feature marker combination composed of infrared lamp beads is designed, each "T" shaped marker including four infrared lamp beads arranged at the four corner points of the "T" shape. In actual application, multiple "T" shaped marker combinations can be arranged on a moving target (such as a UAV) to provide sufficient feature points for subsequent PnP pose solution, thereby improving the solution accuracy and robustness. At the same time, in cooperation with the infrared filter carried by the camera, most of the light noise in the environment can be filtered out in real time to ensure the clarity of the infrared feature points in the image.

[0010] 2. Image feature extraction and matching Dynamic threshold binarization processing: Since the brightness of infrared lamp beads in the image varies greatly at different distances and directions, and environmental light noise can interfere with feature recognition, dynamic threshold adjustment is needed for the "region where feature points may exist a priori" in the image to complete black and white image binarization. Set the initial threshold value, adjust the threshold value through a preset formula, and ensure the effective separation of feature points in different environments.

[0011] Feature point contour extraction: use the find Contours function of the Open CV open source library to extract the feature point contour of the binarized image, and obtain the coordinates of the projection points of the "T" shaped marker in the camera image. To be compatible with a small amount of noise interference, set the rule that "feature points within the preset number of accepted set multiple of preset marker points enter the next step"; at the same time, the single feature point extraction loop is repeated several times until the feature points meeting the quantity requirement are obtained.

[0012] Feature point screening and matching: based on the geometric features of the "T" shaped marker, the extracted feature points are screened and matched, and the specific steps are as follows: (1) Identify three collinear feature points, calculate their parallelism, and preliminarily screen the feature point groups meeting the "T" shape structure; (2) Evaluate the distance between feature points, calculate their equidistance, and further verify the geometric rationality of the feature point groups; (3) Record the midpoint of the three collinear points as point A, and the remaining one feature point as point D. Rotate the AD line counterclockwise with point A as the axis, and determine the other two feature points (B and C) according to the rotation order to clearly define the correspondence of the four feature points; (4) Calculate the comprehensive score S for all possible feature point combinations, and select the feature point group with the highest score and exceeding the preset threshold for subsequent pose solution.

[0013] 3. Calculation of pose of moving target Based on the coordinates of four selected and matched feature points, the PnP algorithm is used to estimate the pose of a moving target (such as a drone), and the estimation results are verified. The alignment between the axis and the direction of gravity ensures the physical rationality of the pose calculation results and avoids calculation deviations caused by mismatched feature points.

[0014] 4. Multi-camera collaborative scheduling Observation quality assessment: The pose estimation results are reprojected onto the pixel coordinate system of each camera to obtain the coordinates of the projected points, and the offset from the camera optical center coordinates is calculated. Considering the nonlinear influence of different regions of the image on the estimation results, an arctangent function is designed as the loss function to calculate the offset loss, thereby quantifying the observation quality of a single camera.

[0015] Dynamic weight allocation and phased control: Based on the observation quality of a single camera and the total observation quality of all cameras, the weight of the control variable "camera gimbal rotation angle" in the linear control model is dynamically allocated, using a phased scheduling strategy. (1) When the moving target moves quickly, the overall observation quality is poor. Select the 2-3 cameras with the worst observation quality and control the pan-tilt rotation to adjust the observation direction. Keep the other cameras still to quickly improve the overall observation quality. (2) When the moving target moves slowly or hovers, the overall observation quality is good. Only the camera with the worst observation quality is selected for gimbal control, while the other cameras remain stationary to reduce system fluctuations and avoid target loss.

[0016] Optimization of control variables: A linear control model is constructed with "observation quality" as the state variable and "gimbal rotation angle" as the control variable. A controller and cost function are designed for the set time. The minimum control variable is solved by minimizing the cost function, so as to achieve accurate and efficient scheduling of the camera gimbal.

[0017] The multi-camera collaborative state estimation method based on active infrared visual features provided by this invention has the following beneficial effects: 1. This invention addresses the issues of insufficient accuracy and robustness in collaborative systems, particularly in the areas of moving target recognition, real-time positioning, and attitude calculation between UAVs and ground observation platforms. Specifically, it overcomes the limitations of existing passive visual markers, which are susceptible to ambient light and experience performance degradation under low light or complex lighting conditions. It also solves the problems of single-fixed-camera schemes being limited by the observation field of view and camera distortion, making it difficult to guarantee the stability and robustness of state estimation results, as well as the high cost, reduced environmental adaptability, and poor dynamic performance of multi-fixed-camera schemes. This invention achieves high-precision, highly robust target recognition and pose calculation.

[0018] 2、The application adopts active infrared visual markers to replace traditional passive visual markers, effectively resists harsh environmental interference such as dark light, complex illumination, weak texture and GNSS denial, ensures the stability of feature recognition and pose estimation, is not limited by environmental light, provides stable and reliable feature point information, and greatly improves the adaptability and reliability of the system.

[0019] 3、The application realizes that the state estimation RMSE (root mean square error) is less than 0.1 m within the flight height range of the moving target within 4 m by designing the geometric characteristics of the T-shaped marker, combining multi-camera cooperative scheduling and PnP algorithm optimization, meeting the high-precision application demand, and significantly improving the positioning accuracy.

[0020] 4、The dynamic threshold binarization processing method and the geometric feature screening and matching algorithm of the application effectively improve the feature point recognition accuracy and matching success rate in complex illumination and noise environment, reduce the interference of environmental noise on feature point recognition, and enhance the robustness of feature extraction.

[0021] 5、The multi-camera cooperative observation system of the application expands the observation field of view through the cooperative work of multiple cameras, can observe the target from different angles, solves the problems of limited observation field of view and camera distortion of the single fixed camera scheme, improves the accuracy and robustness of pose solution, and realizes omnidirectional and multi-angle target tracking.

[0022] 6、The multi-camera cooperative scheduling algorithm of the application can dynamically adjust the rotation direction and speed of the camera holder according to the observation quality of different cameras, realize continuous and stable observation of the target, while the single fixed camera scheme cannot realize this function, improve the stability and efficiency of target tracking, and ensure the continuity of observation.

[0023] 7、Compared with the traditional multi-fixed camera scheme, the application reduces the number of camera deployment through active camera dynamic scheduling, reduces the hardware cost, and at the same time supports flexible deployment, can flexibly adjust the number and layout of cameras according to actual needs, improves the environmental adaptability and deployment convenience, and meets the application needs in different scenes.

[0024] 8、In the feature point recognition and matching process, the application adjusts the dynamic threshold, effectively adapts to the feature change under different illumination conditions, improves the stability and accuracy of feature point recognition, reduces the recognition error caused by illumination change, and enhances the environmental adaptability of the system.

[0025] 9、The system of the application realizes real-time monitoring and accurate judgment of the motion state of the target through the cooperative work of multiple cameras, can adjust the observation strategy according to the different motion states (such as fast motion, slow motion, hovering, etc.) of the target, reduces the risk of target loss, and improves the overall robustness of the system.

[0026] 10、The application adopts the ideas of integration and modularization in hardware design, simplifies the system structure, reduces the maintenance cost, and improves the scalability and upgradeability of the system, providing convenience for future function expansion and technology upgrade.

[0027] 11、The application is verified by experiments, and the performance of the scheme is superior in harsh environments (such as GNSS denial, dim light or complex light, weak texture, etc.), which proves its effectiveness and practicality in complex environments, provides a new technical path for target recognition and positioning in dynamic scenes, and has wide application prospects.

[0028] 12、The system also has powerful data processing and analysis capabilities, can process and analyze a large amount of observation data in real time, extract valuable information, and provide support for decision-making, while supporting data storage and playback, facilitating subsequent analysis and research, and improving the intelligent level of the system. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 is the active infrared visual feature marker physical map of the application; Figure 2 is a schematic diagram of the visual feature "T" shape marker of the application; Figure 3 is a schematic diagram of the ground observation platform structure of the application; Figure 4 is a detail drawing of the active vision mechanism (camera + gimbal) of the application; Figure 5 is a schematic diagram of the feature points extracted by the infrared camera of the application; Figure 6 is an effect diagram of the feature point matching of the infrared camera of the application; Figure 7 is a harsh environment experimental verification diagram of the application. DETAILED DESCRIPTION

[0030] The technical solutions in the application will be further described below in combination with the drawings and examples: Example 1 This embodiment provides a multi-camera cooperative state estimation method based on active infrared visual features, which is the application of a multi-camera cooperative state estimation system based on active infrared visual features in unmanned aerial vehicle positioning, and the specific steps are as follows: 1、System hardware construction Unmanned aerial vehicle side: such as Figure 1 and Figure 2As shown, an active infrared visual feature marker combination with a specific "T" shape geometry is installed on the UAV. The marker combination is composed of multiple infrared light beads, and the layout of the infrared light beads strictly follows the "T" shape structure to ensure that it can be clearly captured by the camera at different angles. For example, infrared light beads are evenly arranged at the intersection of the horizontal and vertical of the "T" shape and at appropriate positions at both ends of the horizontal and vertical to provide rich feature point information.

[0031] Ground observation platform end: as shown in Figure 3 and Figure 4 , multiple cameras are deployed, which are evenly distributed around the ground observation platform to form a multi-camera observation array. The model selection of the cameras needs to consider factors such as resolution, frame rate, and sensitivity to infrared light to ensure that the infrared visual feature markers on the UAV can be accurately captured. At the same time, each camera is equipped with a pan-tilt head with adjustable angle to adjust the observation direction of the camera according to actual needs.

[0032] 2、Feature point recognition and matching algorithm implementation Image acquisition: multiple cameras collect active infrared visual feature marker images on the UAV in real time, and transmit the images to the central processing unit of the ground observation platform.

[0033] Image preprocessing: as shown in Figure 5 and Figure 6 , the central processing unit performs dynamic threshold adjustment binarization processing on the collected images. Specifically, according to the gray scale distribution of the image, a suitable threshold is automatically calculated, and the pixel points in the image with a gray scale value higher than the threshold are set to white, and the pixel points with a gray scale value lower than the threshold are set to black, thereby effectively filtering out environmental noise and highlighting the infrared visual feature markers.

[0034] Feature point contour extraction: use the find Contours function in the Open CV open source library to extract the feature point contour of the binarized image. This function can accurately identify the contour information in the image, and then obtain the coordinates of the projection points of the visual feature markers in the camera image.

[0035] Feature point matching: use geometric feature screening and matching method to match the extracted feature points. First, evaluate the collinearity, parallelism and equidistance between the feature points. For example, by calculating the slope of the line connecting the feature points, it is judged whether they are collinear; calculate the distance between adjacent feature points to judge whether they are equidistant. Then, combined with rotation matching and other operations, ensure that the corresponding feature point pairs can be accurately matched. For example, according to the feature point distribution law of the UAV at different attitudes, the feature points are matched by rotation to eliminate the influence of the change of feature point position caused by the rotation of the UAV.

[0036] 3、Pose solution The PnP (Perspective-n-Point) algorithm is used to accurately estimate the pose information of the UAV based on well-matched feature point pairs. The PnP algorithm solves for the UAV's position and attitude parameters by establishing the projection relationship between feature points in the camera coordinate system and the world coordinate system.

[0037] For the estimation results The alignment between the axis and the direction of gravity is rigorously verified. This is achieved by analyzing the estimated UAV pose information. The axial direction is compared with the pre-defined gravity direction, and the angle between the two is calculated. If the angle exceeds a certain threshold, the estimation result is considered to have an error, and feature point matching and pose calculation need to be repeated to ensure the accuracy of the estimation result.

[0038] 4. Multi-camera collaborative scheduling The weights of control variables are dynamically and intelligently allocated based on the observation quality of individual cameras and all cameras. Specifically, an observation quality evaluation index is set for each camera, which can be calculated based on factors such as the camera's image sharpness and feature point matching success rate. For example, image sharpness can be evaluated by calculating parameters such as image contrast and sharpness; the feature point matching success rate can be calculated by statistically analyzing the ratio of the number of successfully matched feature point pairs to the total number of feature points.

[0039] Based on the observation quality evaluation index, corresponding control variable weights are assigned to each camera. Cameras with higher observation quality are assigned larger control variable weights; conversely, cameras with lower observation quality are assigned smaller control variable weights.

[0040] Based on the pre-assigned control weights, the rotation direction and speed of the camera gimbal are adjusted to achieve coordinated rotation among multiple cameras. For example, if a camera has high observation quality and the target object is moving to the right within that camera's field of view, the rotation speed of that camera's gimbal is appropriately increased. At the same time, based on the observations of other cameras, the rotation of the other camera gimbals is coordinated to ensure continuous and stable observation of the target.

[0041] Example 2 In another preferred embodiment, based on the above embodiment 1, such as Figure 7 As shown, this embodiment provides a multi-camera cooperative state estimation method based on active infrared visual features, mainly describing the system's adaptability adjustment under different environmental conditions, as detailed below: 1. Low-light environment In a dark environment, the active infrared visual feature marker has more advantages. Since the infrared light bead can actively emit infrared light, it is not limited by the ambient light, so the camera can still clearly capture the feature marker. At this time, the exposure time and gain parameters of the camera can be adjusted appropriately to improve the brightness of the image, but attention should be paid to avoid overexposure leading to image distortion. For example, the exposure time of the camera is appropriately prolonged, and the gain is appropriately increased, so that the feature marker in the image is more clearly visible.

[0042] For feature point recognition and matching algorithm, the parameters of dynamic threshold adjustment can be further optimized. Since the gray scale distribution of the image in the dark environment is different from that in the normal lighting environment, the threshold calculation method needs to be adjusted according to the actual situation to ensure accurate binary processing and feature point contour extraction.

[0043] 2. Complex lighting environment In a complex lighting environment, such as strong light direct or shadow interference, the system may be affected. At this time, image enhancement techniques such as histogram equalization, adaptive contrast enhancement, etc. can be used to process the collected images to improve the quality of the images and improve the recognition rate of the feature points.

[0044] For multi-camera cooperative scheduling algorithm, the observation strategy of the camera can be dynamically adjusted according to the observation situation of different cameras. For example, if a camera is affected by strong light direct, which leads to a decrease in imaging quality, the control weight of this camera can be appropriately reduced, while the control weight of other cameras not affected by strong light can be appropriately increased to ensure stable observation of the target.

[0045] Embodiment 3 In another preferred embodiment, based on the above embodiments 1 and 2, this embodiment provides a multi-camera cooperative state estimation method based on active infrared visual features, which is an application of indoor unmanned aerial vehicle cluster cooperative pose estimation, and the specific steps are as follows: 1. Application scenario This embodiment is aimed at the environment of indoor GNSS (Global Navigation Satellite System) denial, weak texture (wall is single white paint) and complex lighting (including top LED (light emitting diode) lamp, side window natural light), realizes the real-time pose estimation and cooperative tracking of all unmanned aerial vehicles by the ground observation platform in the cluster cooperative system composed of 3 unmanned aerial vehicles, and supports the formation flight and task cooperation of unmanned aerial vehicle cluster (such as indoor material transfer and equipment inspection).

[0046] 2. System building 2.1. Moving target end (unmanned aerial vehicle) Each drone (model: DJI Mavic Mini 2 modified) has two sets of "T" shaped active infrared visual markers symmetrically arranged on the bottom of the fuselage, each set of markers consists of four high-brightness infrared light beads with a wavelength of 850nm, the light bead spacing is set to 8cm (adapted to the size of the drone fuselage), the light beads adopt a 120° divergence angle design to ensure that the camera can capture clear light spots under different observation angles; The infrared light beads are powered by the built-in lithium battery of the drone and are equipped with an independent control module, which can automatically adjust the brightness of the light beads according to the intensity of the ambient light (brightness adjusted to 80% in dark light environment and 100% in strong light environment), avoiding feature point loss caused by overexposure or insufficient brightness.

[0047] 2.2, ground observation end (observation platform) Four active visual cameras (model: Basler acA1920-40gc, equipped with 850nm narrowband infrared filter) are deployed at the four corners of the indoor space (10m long x 8m wide x 5m high), the cameras are fixed at a height of 2.5m through a bracket, ensuring that the observation field covers the entire indoor space; Each camera is equipped with a two-axis gimbal driven by two high-precision servos (model: MG996R), which can achieve horizontal 0°~360° and vertical -30°~60° rotation, and the gimbal control signal communicates with the ground control unit through RS485 bus; The ground control unit is equipped with a multi-camera synchronous triggering module, which sends a 5V TTL (Transistor-Transistor Logic) synchronous signal to control the four cameras to synchronously capture images at a frame rate of 25fps, with a synchronization error of ≤1ms, avoiding position deviation of the drone caused by time difference in capturing.

[0048] 3, implementation process 3.1, image acquisition and preprocessing After starting the system, the infrared light beads are powered on and emit light, and the four cameras synchronously capture images containing the infrared markers of the drones, the infrared filter filters out the indoor visible light noise, and only the infrared light spot is retained; Preprocess each frame of image: first, based on the initial formation position of the drone (preset as the center area of the indoor space), define a "prior feature area" of ±250 pixels from the center of the image; then perform dynamic threshold binaryzation, set the initial threshold , through the formula: (1) Adjust the threshold, for example, when the window side camera is disturbed by natural light, the threshold is automatically adjusted to 85, ensuring that the light spot is separated from the background; In formula (1), This represents the dynamic threshold used for image binarization. Its value is a non-negative integer (pixel gray value range is 0~255). It directly determines the separation effect of infrared feature points and background in the image and needs to be dynamically adapted according to ambient light noise and infrared lamp brightness. This represents the initial threshold value, which is set to 80 in practical applications. This value is determined based on the average grayscale value of infrared LEDs under common low light and complex lighting conditions, providing a benchmark starting point for threshold adjustment. This represents the threshold adjustment coefficient, which is an integer (typically -2, -1, 0, 1, or 2). It is determined based on the deviation between the average gray value of the "prior feature region" in the image and the preset target gray value. For example, when the average gray value of the region is lower than the target value, A positive value is used to lower the threshold, ensuring that feature points are not misclassified as background. The threshold adjustment step size is set to 5, which is a fixed constant used to control the magnitude of each threshold adjustment. This avoids excessive adjustment magnitude leading to feature point loss or noise residue, while excessively small magnitude will affect the threshold adaptation efficiency.

[0049] The OpenCV find Contours function is called to extract contours. Contours with an area of ​​5 to 50 pixels² are selected as candidate feature points. If the number of candidate feature points extracted in a single extraction is not within the range of 4 to 12, the extraction process is repeated (up to 30 times) until the number requirement is met. Finally, the coordinates of the projection point of the infrared marker in the image are obtained.

[0050] 3.2 Feature Point Selection and Pose Calculation Feature point screening: For candidate feature points, first identify three collinear feature points, and then use formula (2): (2) Calculate parallelism (set the slope deviation threshold to 0.15; if the deviation exceeds this threshold, the group is removed); using formula (3): (3) Calculate the equidistance (set the distance deviation threshold to 0.10, and discard if the deviation exceeds the threshold); then use formula (4): (4) calculate The degree of alignment between the axis and the direction of gravity (the threshold for the degree of alignment deviation is set at 0.15; if the deviation exceeds this threshold, the component will be discarded).

[0051] Feature point localization: Let the midpoint of the three collinear points be point A, and the remaining feature points be point D. Rotate the line AD counterclockwise around point A as the axis. When the rotation angle is 90°, determine point B (closer to the starting direction of rotation) and point C, and clarify the correspondence between the four feature points. In equation (2), represents the vector from feature point A to feature point B, where A is the midpoint of the three collinear feature points in the "T" shaped visual marker, and B is one of the endpoints of the three collinear points. The numerical value of the vector is calculated by the coordinate difference of the two points in the camera image pixel coordinate system, reflecting the positional relationship and direction of the two points in the image plane. represents the vector from feature point C to feature point A, where C is the other endpoint of the three collinear points in the "T" shaped visual marker. The numerical value of the vector is also determined by the coordinate difference of the two points in the pixel coordinate system. is commonly used to determine the degree of collinearity of the three points. is the dot product of the vectors and . The calculation method is to multiply the corresponding components and then sum them up. The result can be used to reflect the angle relationship between the two vectors, and is one of the core calculation factors for determining whether the three points are collinear. represents the length of the vector , which is the Euclidean distance between feature point A and feature point B in the image pixel coordinate system. The calculation method is to take the square root of the sum of the squares of the components of the vector . It is used to standardize the dot product result and eliminate the interference of distance factors on parallelism determination. represents the length of the vector , which is the Euclidean distance between feature point C and feature point A in the image pixel coordinate system. The calculation method is consistent with , and is also used to standardize the dot product result to ensure the consistency and comparability of the parallelism calculation result. is a parallelism score mapping function, whose input is the calculation result of . The result is essentially the cosine value of the angle between the two vectors, with a value range of [-1, 1]. The output is a parallelism score of 0-100 points. When the input value is close to 1 (indicating that the two vectors are nearly parallel in the same direction), the output score is close to 100 points. When the input value is close to -1 (indicating that the two vectors are nearly anti-parallel), the output score is close to 0 points, quantifying the parallelism of the feature points.

[0052] In formula (3), is the length difference between the vectors and . The result reflects the difference between the distance between points A and B and the distance between points C and A. The smaller the difference, the closer the distance between A and B and the distance between A and C, indicating that the three points in the "T" shaped marker are equidistant. is the length sum of the vectors and , which is used to normalize the aforementioned length difference, eliminating the influence of data dimension on equidistance determination. is an equidistance score mapping function, whose input is The calculation result (whose absolute value reflects the relative deviation of the distance from A to B and from A to C) outputs an equidistance score of 0-100 points. When the absolute value of the input value is close to 0 (indicating that the distance deviation is extremely small), the output score is close to 100 points. When the absolute value of the input value is close to 1 (indicating that the distance deviation is extremely large), the output score is close to 0 points, thereby quantifying the equidistance of the feature points.

[0053] Pose solution: input the screened feature point coordinates into the PnP algorithm, combine the camera intrinsic parameters (obtained in advance by Zhang Zhengyou calibration method, focal length (f) 、 ), and the principal point coordinates (x0 , ), to solve the position and attitude (roll angle, pitch angle, and yaw angle) of each unmanned aerial vehicle relative to the ground platform.

[0054] In formula (4), represents the unit vector of the axis in the coordinate system of the moving target (such as a robot, e.g., an unmanned aerial vehicle) obtained by the PnP algorithm, whose direction is determined by the spatial position relationship of the feature points, reflecting the pointing direction of the axis of the moving target in the current attitude, which is a key parameter for judging whether the target attitude is physically reasonable; represents the unit vector of the gravity direction, which is preset to be perpendicular to the downward direction of the ground (such as (0, 0, -1) or (0, 0, 1) in the ground observation platform coordinate system, depending on the definition of the coordinate system) in actual application, and is a reference benchmark for judging whether the direction of the axis of the moving target is reasonable; is the dot product of the vector and the vector , which is calculated by multiplying the corresponding components and summing them up, and its result reflects the angle relationship between the axis of the moving target and the gravity direction. The closer the result is to -1, the closer the two directions are to the same direction; the closer the result is to 0, the closer the two directions are to being perpendicular; and the closer the result is to 1, the closer the two directions are to being opposite. represents the length of the vector , since is a unit vector with a fixed length of 1, it is mainly used for standardization in formula form to ensure the completeness and consistency of the calculation logic; represents the length of the vector , since is a unit vector with a fixed length of 1, it, together with , forms the denominator and participates in the standardization of the dot product result, making the calculation result more intuitive and comparable; is the degree of fit score mapping function, with the calculated result (the value range is [-1, 1], and the output is a fitness score of 0~100 points, when the input value is close to -1 (indicating that the motion target axis is close to anti-parallel with the direction of gravity), the output score is close to 100 points; when the input value is close to 1 (indicating that the motion target axis is close to parallel with the direction of gravity), the output score is close to 0 points, so as to quantify the physical rationality of the pose estimation result.

[0055] Comprehensive score calculation: according to the parallelism score weight 0.4, the equidistance score weight 0.3, the axis and the gravity direction fitness score weight 0.3, the feature point combination comprehensive score : (5) Select the highest score group with the score ≥80 as the final feature point of the pose solution, and input the PnP algorithm to solve the position and attitude (roll angle, pitch angle, yaw angle) of each unmanned aerial vehicle relative to the ground platform as the final result.

[0056] 3.3, Multi-camera cooperative scheduling and tracking Observation quality evaluation: the pose solution result is re-projected to the pixel coordinate system of each camera to obtain the projection point coordinates , the offset loss (6) is calculated by the inverse tangent function: , The smaller the offset loss is, the better the observation quality is; In formula (6), represents the offset loss, the value range is , which is used to quantify the influence of the offset degree of the motion target re-projection point and the camera optical center on the observation quality, The smaller the offset loss is, the better the observation quality is, and it is the core evaluation index of multi-camera cooperative scheduling; represents the projection point coordinates of the motion target pose re-projected to the camera pixel coordinate system, which is a non-negative real number (unit: pixel), calculated from the PnP pose solution result and the camera internal parameter, directly reflecting the actual position of the target in the image; is the principal point coordinates, the coordinates of the camera optical center in the pixel coordinate system, which is a pre-set known value (for example, when the camera resolution is 1280x720, it is usually set to (640, 360)), which is a reference point for judging the offset direction and amplitude of the projection point; , , is a function parameter used to adjust the potential well range of the inverse tangent function, which needs to be adjusted according to the task requirements.

[0057] Stage control: when the UAV cluster is rapidly reorganizing (speed 1.5 m / s), the overall observation quality is poor (Fig. 6) ), select the 2 cameras with the worst observation quality, and use a linear control model to: (7) solve the optimal control quantity to drive the gimbal to adjust the observation direction; when the UAV is hovering to perform material transfer tasks (speed ≤0.2 m / s), the overall observation quality is good (Fig. 6) ), only control the gimbal rotation of the camera with the worst observation quality, and keep the rest of the cameras stationary; In equation (7), represents the rate of change of the state quantity, where the state quantity is set as the observation quality (i.e., the offset loss ), so represents the rate of change of the observation quality (unit: ), reflecting the dynamic change trend of the observation quality over time, which is an important basis for determining whether the camera gimbal needs to be adjusted; represents the system matrix, which is a constant matrix (usually a 1x1 matrix, i.e., a single constant), determined by the system itself characteristics (such as the response speed of the camera gimbal, the motion characteristics of the moving target), used to describe the change law of the state quantity itself, for example, when is positive, the observation quality will spontaneously deteriorate over time and needs to be improved through control quantity intervention; represents the state quantity, set as the observation quality (offset loss ), with a value range of is the core input of the linear control model, directly reflecting the observation effect of the current camera on the target; represents the input matrix, which is a constant matrix (usually a 1x1 matrix), describing the influence degree of the control quantity on the state quantity, for example, The absolute value of the larger, the more significant the improvement effect of the unit control quantity on the observation quality, which needs to be calibrated according to the control sensitivity experiment of the camera gimbal; represents the control quantity, set as the camera gimbal rotation angle (unit: ° or rad), which is the output of the linear control model, directly determining the adjustment direction and amplitude of the camera gimbal, and by adjusting can change the observation direction of the camera and optimize the observation quality.

[0058] Cluster coordination: the ground control unit sends the pose information of each UAV to the UAV flight control system through the wireless data transmission module (model: SX1278), supporting the formation maintenance and obstacle avoidance of the UAV cluster, ensuring flight safety.

[0059] 4、Implementation effect During the 30-minute experiment, the drone swarm completed tasks such as formation flying, hovering transfer, and path switching, and the pose estimation results met the following requirements: Position accuracy: , The directional RMSE (Root Mean Square Error) is 0.04~0.06m. The RMSE for the direction is 0.03-0.05m; Attitude accuracy: Roll and pitch RMSE is 0.5°~1°, yaw RMSE is 1°~1.5°; System stability: No drone target loss throughout the entire process, camera gimbal response latency ≤80ms, adaptable to complex indoor lighting and weak texture environments.

[0060] Example 4 In another preferred embodiment, based on embodiments 1 to 3 above, this embodiment provides a multi-camera collaborative state estimation method based on active infrared visual features, which is an application of pose estimation for outdoor low-light robot inspection. The specific steps are as follows: 1. Application Scenarios This embodiment addresses the challenges of industrial park environments with low outdoor light (18:00-19:00 in the evening, light intensity 5-10 lux) and complex obstructions (including shadows from trees and buildings). It enables a ground observation platform to estimate the real-time pose of an inspection robot (used for pipeline and equipment status detection), ensuring that the robot accurately inspects along a preset path and avoiding positioning deviations and task interruptions caused by environmental interference.

[0061] 2. System Setup 2.1 Moving target end (inspection robot) The inspection robot (model: AGV-200 modified) has two sets of "T"-shaped active infrared vision markers arranged along the central axis on its top. Each set of markers consists of four infrared LEDs with a wavelength of 940nm (the 940nm wavelength can reduce the interference of red light components in natural light at dusk). The spacing between the LEDs is set to 12cm, and a light shield is installed on the outside of the LEDs to avoid the spread of light spots caused by direct ambient light from the side. The infrared LEDs are powered by the robot's built-in lead-acid battery and are equipped with a light sensor (model: BH1750) to detect the ambient light intensity in real time. When the light intensity is less than 8 lux, the LEDs automatically turn on the supplementary lighting function, with the brightness fixed at 90%, to ensure that the camera can capture a stable light spot.

[0062] 2.2 Ground-based observation terminal (observation platform) Five active visual cameras (Model: Hikvision DS-2CD3T46WD-I3, equipped with 940nm narrowband infrared filter) are deployed at intervals of 10m on both sides of the inspection path (total length 50m, width 3m) in the industrial park. The cameras are fixed at a height of 3m through a stand, ensuring that the observation field of view of adjacent cameras overlaps by ≥30%, avoiding blind spots in the inspection path. Each camera is equipped with a single-axis gimbal (only horizontal rotation, range 0°~360°) driven by an industrial-grade servo (Model: DS3218). The gimbal control signal is communicated with the ground control center through Ethernet. The ground control center deploys a multi-camera synchronization server to control the five cameras to synchronously collect images at a frame rate of 20fps through the sending of NTP synchronization signals, with a synchronization error ≤2ms, ensuring the time consistency of multi-camera data.

[0063] 3. Implementation process 3.1, image acquisition and preprocessing After the robot starts the inspection task, the infrared light bead is powered on, and five cameras synchronously collect images containing the infrared marker of the robot. The infrared filter filters out visible light and stray light in the natural light at dusk, highlighting the infrared light spot. Image preprocessing: Based on the pre-set coordinates of the inspection path, the "prior feature area" (width ±300 pixels) in the image along the path direction is delineated. The initial threshold value is adjusted according to the ambient light intensity feedback by the light sensor (5lux , 10lux ), ensuring clear separation of light spots in different light areas. Feature point extraction: Use the find Contours function of Open CV to extract contours, and select contours with an area of 5-50 pixels² as candidate feature points. The single extraction loop is repeated 30 times. If the number of candidate feature points is not within the range of 4-12, the threshold value is automatically adjusted for re-extraction, and the final projection point coordinates are obtained. 3.2, feature point selection and pose solving

[0064] Feature point selection: First, calculate the slope deviation of collinear three points through the parallelism formula, set the threshold value to 0.20 (slightly higher than indoor due to possible slight vibration in outdoor environment), and remove if the deviation exceeds; calculate the distance deviation through the equidistance formula, set the threshold value to 0.15, and ensure compliance with the "T" geometric feature; then calculate the deviation of z-axis and gravity direction through the fitting degree formula, set the threshold value to 0.25 (slightly higher than indoor due to possible wind influence in outdoor environment), and ensure compliance with the physical reasonableness of quantitative pose estimation results.

[0065] ​Feature point positioning: determine B and C points in the manner of "midpoint A-feature point D- anticlockwise rotation AD line", and clarify the corresponding relationship of the four feature points; Comprehensive score and pose solving: calculate the comprehensive score of feature point combination (parallelism 0.4, equidistance 0.3, axis fitting degree 0.3), select the highest score group with score ≥78 (outdoor environment interference is slightly larger, threshold is appropriately reduced), and input PnP algorithm to solve the robot pose.

[0066] 3.3, Multi-camera cooperative scheduling and tracking Observation quality evaluation: reproject the pose result to the pixel coordinate system of each camera, and calculate the offset loss When the robot moves quickly along the inspection path (speed 1 m / s), , select the 3 cameras with the worst observation quality, and minimize the cost function through the linear control model: (8) Solve the pan-tilt angle control quantity; when the robot moves slowly to detect the equipment (speed ≤0.3 m / s), , only control the pan-tilt of the camera with the worst observation quality; In formula (8), represents the cost function value, which is a non-negative real number, used to quantify the comprehensive cost of state quantity deviation and control quantity consumption in the future time, the goal is to minimize through optimization solving, to achieve the balance between "optimal observation quality" and "lowest control cost"; represents the current time index (such as in camera image frames, ), used to determine the starting time point of calculating the cost function, to ensure that the control strategy is synchronized with real-time observation data; represents the prediction step, which is a positive integer (can be set to 5-10), indicating that the cost function calculates the state and control quantity for the future time, The larger the prediction step is, the stronger the forward-looking of the control strategy is, but the calculation complexity is also higher, which needs to be adjusted according to the real-time requirement of the system; represents the state quantity at the time (offset loss ), the value range is , represents the square term of the observation quality deviation at the time, used to punish the state with poor observation quality, and the weight of this term is 1 (default), to ensure that the observation quality is prioritized; represents the regularization parameter, which is a non-negative real number (can be set to 0.1-0.2), used to balance the consumption of control quantity, represents the​ Square penalty term of time control amount (gimbal rotation angle), The greater the constraint on the control amount, the stronger the constraint, which can avoid mechanical wear and tear and stability decline caused by frequent and large rotation of the gimbal; Indicates the first The control amount (camera gimbal rotation angle) at the time is ° or rad, which is the core variable of the optimization solution of the cost function, and the optimal The optimal , accurate and efficient control of the camera gimbal is realized.

[0067] Path correction: The ground control center compares the pose estimation result with the preset inspection path, and if the deviation exceeds 0.1m, sends a path correction instruction to the robot through wireless communication to ensure that the robot accurately patrols along the path.

[0068] 4、Implementation effect In a one-hour outdoor dark light inspection experiment, the robot completed a total length of 50m of the inspection path, and the pose estimation and tracking effect met: Position accuracy: (path direction) RMSE is 0.05~0.07m, (perpendicular to the path direction) RMSE is 0.06~0.08m, (height direction) RMSE is 0.04~0.06m; Tracking stability: no robot target loss throughout the whole process, camera gimbal response delay ≤100ms, which can effectively avoid the shielding interference of trees and building shadows; Task adaptability: the deviation of the robot in the preset path inspection is ≤0.1m, which meets the accurate positioning requirements of the industrial park equipment detection.

[0069] Example 5 In another preferred embodiment, based on the above-mentioned embodiments 1 to 4, the present embodiment provides a multi-camera cooperative state estimation method based on active infrared visual features, which is the application of relative pose estimation of unmanned aerial vehicles and ground observation platforms in ground-air cooperative systems. The implementation process of the present embodiment will be described in detail. This example does not constitute any limitation on the present invention, and those skilled in the art can adjust the parameters and steps according to the actual scene.

[0070] 1、System building Moving target end: 2~3 groups of "T" shaped active infrared visual markers are arranged at appropriate positions of the unmanned aerial vehicle body (such as the belly, under the wings), each group of markers is composed of 4 infrared lamp beads with the same wavelength, to ensure uniform brightness and geometric position in accordance with the "T" shape design (the distance between the corner points is set according to the size of the unmanned aerial vehicle, which is 10cm in the example); Ground observation end: 4 active vision cameras are deployed on the ground observation platform (size 1.2m x 0.8m). The cameras are all equipped with 850nm infrared filters, and can realize 360° horizontal rotation and ±45° vertical rotation through the servo-driven gimbal. The camera deployment position is in a "square" distribution (1m spacing between adjacent cameras), ensuring that the observation field of view covers the unmanned aerial vehicle flight area (radius 5m, height 0~4m).

[0071] 2. Image acquisition and preprocessing Start the unmanned aerial vehicle and the ground observation platform. The infrared light beads are powered on and emit light. The 4 active vision cameras are started synchronously to acquire image videos containing the infrared markers of the unmanned aerial vehicle in real time at a frame rate of 30fps. Preprocess each frame of image: first determine the "prior feature region" (based on the initial position of the unmanned aerial vehicle and the camera field of view, define the image center ±300 pixels range), then perform dynamic threshold binarization - set the initial threshold , through the formula: (1) Adjust the threshold to ensure that the infrared light beads appear as clear white spots in the image and the environmental noise appears as black background. Call the find Contours function of Open CV to extract the contours in the binarized image, filter the contours with an area of 50~200 pixels² (corresponding to the size of the infrared light bead) as candidate feature points. If the number of candidate feature points extracted at a time is not within the range of 4~12, repeat the extraction process up to 30 times until the number requirement is met.

[0072] 3. Feature point matching and pose solving Geometrically filter the candidate feature points: first identify 3 collinear feature points, calculate their slope deviation (allowable deviation ≤0.05), and determine the "horizontal bar" part of the "T" marker; then calculate the distance between the 3 collinear points to ensure that the distance deviation between the two endpoints and the midpoint is ≤0.1cm, meeting the equidistance requirement; Determine the correspondence of the feature points: mark the midpoint of the 3 collinear points as point A, and the remaining candidate feature points as point D. Take point A as the axis and rotate AD line counterclockwise. When the line forms a 90° angle with the 3 collinear points, determine point B (close to the starting direction of rotation) and point C (far from the starting direction of rotation) in turn; Calculate the comprehensive score S (parallelism score, equidistance score, angle deviation score, each score range 0~100) for all 4-point combinations that meet the conditions, and select the highest score group with S≥85 as the final feature points; Input the coordinates of the final feature points into the PnP algorithm, combine the camera intrinsic parameters (obtained in advance through Zhang Zhengyou calibration method), and solve the position of the unmanned aerial vehicle relative to the ground observation platform With the attitude (roll angle, pitch angle, yaw angle), and check The angle between the axis and the direction of gravity (allowable deviation ≤5°), if the deviation is out of tolerance, select the feature point group again to solve.

[0073] 4. Multi-camera cooperative scheduling Observation quality calculation: re-project the pose solution results to the pixel coordinate system of each camera to get the projection point coordinates , camera optical center coordinates (camera resolution 1280x720), through the arctangent function: (6) Calculate the offset loss , The smaller the better the observation quality; Control weight distribution: set the total observation quality threshold ( The average value of the four cameras L), if > (corresponding to fast motion of the UAV), select The largest 2 cameras as the control object, and set the control weight to 0.5, and the weight of the remaining cameras to 0; if (corresponding to slow motion or hovering of the UAV), select the largest 1 camera L as the control object, and set the weight to 1.0, and the weight of the remaining cameras to 0; Gimbal angle control: construct a linear control model: (7) Design Instantaneous cost function (8) Solve the optimal control by minimizing , drive the servo to adjust the camera gimbal angle, and realize real-time tracking of the UAV.

[0074] 5. Experimental verification and results In the dark environment (light intensity <5 lux), GNSS denial environment (no satellite signal indoors), the UAV makes hovering, uniform flight (speed 1 m / s), variable speed flight (acceleration 0.5 m / s²) motion in the height range of 0~4m, and observes for 30 minutes: In the hovering state, the pose estimation RMSE is 0.03~0.05m; In the uniform flight state, the RMSE is 0.06~0.08m; In the variable speed flight state, the RMSE is 0.09~0.10m; ​No target loss occurs throughout the whole process, and the response delay of the camera holder is less than 100 ms, which verifies the high precision and high robustness of the application in a harsh environment.

[0075] Embodiment 6 In another preferred embodiment, based on the above-mentioned embodiments 1 to 5, the present embodiment provides a multi-camera cooperative state estimation method based on active infrared visual features, which will be described in detail below in combination with specific examples: This specific example is established in the background of relative pose estimation between the ground observation platform and the unmanned aerial vehicle in the air-ground cooperative system, Figure 1 for the unmanned aerial vehicle carrying active infrared visual markers, Figure 3 is a schematic diagram of the ground observation platform, on which four active visual cameras carrying infrared filters are arranged. The system extracts and matches feature points in the infrared camera image, as shown in Figure 5 and Figure 6 The pose estimation of the unmanned aerial vehicle relative to the platform is calculated, so as to complete the pose feedback of the unmanned aerial vehicle to itself, realize the control of the unmanned aerial vehicle and the cooperative scheduling of the camera.

[0076] Figure 1 is a physical diagram of the active infrared visual feature marker, which shows the actual arrangement form of the “T” shaped marker composed of infrared lamp beads on the unmanned aerial vehicle; Figure 2 is a schematic diagram of the “T” shaped marker, which labels the geometric position relationship of the four infrared lamp beads (corresponding to feature points A, B, C, D).

[0077] Figure 3 is a schematic diagram of the structure of the ground observation platform, which shows the deployment mode of the four active visual cameras carrying infrared filters on the platform; Figure 4 is a detail diagram of the active visual mechanism, which includes the camera and the gimbal structure driven by the steering engine, and embodies the adjustability of the camera observation direction.

[0078] Figure 5 and Figure 6 is a schematic diagram of feature point extraction and matching, which is the feature point matching result after dynamic threshold binarization, contour extraction and geometric screening. In the figure, the matching results 0, 1, 2 and 3 correspond to the A, B, C and D points on the “T” shaped marker respectively, and the 4, 5, 6 and 7 correspond to the A, B, C and D points on the second group of “T” shaped markers respectively, and so on. Figure 5 shows the stability of extraction and matching under the interference of similar markers; Figure 6 shows the matching result of simultaneously extracting multiple markers on a single unmanned aerial vehicle.

[0079] Figure 7The severe environment experiment verification graph includes an experiment scene (such as dark light, a complex shelter environment), a moving target trajectory and a state estimation error curve, and intuitively shows that the high-precision performance of the application is that the RMSE is less than 0.1m in the range of 4m height.

[0080] The processing of the infrared image mainly includes two parts of image feature extraction and matching. The key of image feature extraction is to binarize the black and white pictures obtained by the infrared camera. The designed "T" mark is composed of four infrared lamp beads, which are respectively located at the four corner points of the "T". However, due to the large difference in brightness of the infrared lamp beads at different distances and directions on the image, and the light noise in the environment will also be presented on the image, therefore, the dynamic threshold adjustment is performed on the prior possible region to perform binarization: (9) In formula (9), represents the ring iteration number; , are the dynamic threshold values of the first , loop iteration; is the number of feature points extracted in the first loop iteration process; in the embodiment, the threshold increment has a certain tolerance to the noise points appearing in the binarization process, which is manifested as that the feature points (12 in the embodiment) within the preset number of times (3 times) of the set mark points are accepted to enter the next step. If does not meet the tolerance range, the threshold is adjusted based on the current threshold and then the cycle continues. The single binarization threshold adjustment cycle is repeated at most 30 times, until a reasonable number of feature points or out-of-range feature points are extracted.

[0081] Then, the feature matching is performed, and the feature matching method based on geometric features proposed in the application can complete the recognition of features A, B, C and D in Figure 2 , and the specific recognition method is shown in embodiments 1 to 5. After obtaining the coordinates corresponding to the features, the pose of the unmanned aerial vehicle can be solved according to the PnP algorithm.

[0082] Based on the state estimation result estimated by the multiple cameras, the state estimation result is re-projected to the pixel coordinate system of each camera, and the coordinates are , the offset degree of the distance from the optical center is calculated: (10) In formula (10), , represent the distance values of the projection points x and y from the optical center in the pixel coordinate system, and the denominator 、 is used to normalize the distance value, which is convenient for extending to different resolution camera images. Since different regions in the image have a nonlinear relationship with the final estimation result, the arctangent function is designed to calculate its loss function: (11) In formula (11), the arctangent function is suitable for constructing the potential well function here because of its characteristics that the output tends to 0 when the input tends to 0, and the output value increases sharply after the input is greater than a certain value. 、 、 is a function parameter used to adjust the range of the potential well, which needs to be adjusted according to the task requirements. In this embodiment, 1, 0.7, and 0.66 are taken. Based on the loss function, the offset loss can be obtained: (12) The obtained offset loss is input into the controller and a linear model of the system is constructed: (7) represents the rate of change of the state quantity; the state quantity is the observation quality, and the control quantity is the pan-tilt angle. According to the overall observation quality and the single-camera observation quality, the selection of the motion pan-tilt is adjusted: (13) In formula (13), the overall observation quality is composed of the single-camera observation quality , which is used to describe the overall offset loss of the four cameras; and respectively describe the square sum of the Euclidean norm of the overall observation quality and the single-camera observation quality , which is used to measure the influence of the offset loss on the observation quality; wherein, represents the transpose of a vector or matrix.

[0083] (14) In formula (14), is the original weight of the original control quantity , which is updated to after calculating the single-camera and overall observation quality function .The output of the function only contains and , which respectively represent that the specified camera participates in scheduling and that the specified camera maintains the pose. The function The time controller and the cost function are (15) (8) In formula (15), the gain The linear relationship between the state quantity and the control quantity is constructed; the goal of optimizing the cost function of formula (8) is to solve the minimum control quantity, that is, (16) That is, solve: (17) Formula (17) represents a minimum value problem of a quadratic programming problem, and represents that when the partial derivative result is 0, the state quantity There is a minimum value, and the solution is (18) The gain solved by formula (18) is Finally, the control quantity of the time to be solved is obtained .

[0084] The identification algorithm and the multi-camera cooperative scheduling algorithm are tested in a harsh environment, as shown in Figure 4 In this environment, stable state estimation can be achieved within a flight height range of 4m, and the RMSE is less than 0.1m.

[0085] In the preferred scheme, the "T" shaped visual feature mark combination in step 1 is composed of four infrared lamp beads, and the four infrared lamp beads are respectively located at the four corner points of the "T" shape. Multiple "T" shaped visual feature mark combinations can be arranged on the moving target to improve the accuracy and robustness of the pose solution. The above settings ensure that each "T" shaped mark can provide independent visual positioning information. Through multi-view image matching and fusion algorithm, the occlusion and misidentification are effectively reduced, and high-precision target tracking and pose estimation can be realized even in complex environments, effectively coping with complex and variable scene requirements.

[0086] In the preferred scheme, the multi-camera deployed by the ground observation platform in step 1 is an active vision camera. The active vision camera can realize the rotation of the holder through the steering engine, so as to dynamically adjust the observation direction of the camera, expand the observation field range of the camera, improve the effectiveness of image information acquisition of the moving target, reduce the number of camera deployment, and at the same time enhance the adaptability of the system in dynamic scene, and balance the high precision and high efficiency of state estimation. The above settings enable the ground observation platform to continuously and stably track the target in a complex and variable environment. Even when the target moves quickly or the scene changes quickly, the continuity and accuracy of image acquisition can be ensured, providing a solid and reliable foundation for subsequent image processing and data analysis.

[0087] In a preferred solution, when the multi-camera cooperation module in step 2 acquires image videos, a multi-camera synchronous triggering mechanism is adopted to control all infrared cameras to collect image frames at the same time through a preset synchronization signal, so as to avoid position deviation of a moving target caused by time difference in collection of different cameras, ensure consistency of image data acquired by the multi-camera in the time dimension, provide time-synchronized basic data support for subsequent multi-camera observation quality evaluation and cooperative scheduling, and further improve the accuracy of multi-camera cooperative state estimation; the above settings also effectively reduce error accumulation caused by time asynchronization, enhance the system's ability to capture dynamic scenes, enable the system to maintain high-precision target tracking and behavior analysis capability in a complex and variable monitoring environment, and optimize the performance and reliability of the overall monitoring system.

[0088] In a preferred solution, the process of processing the images acquired in step 2 in step 3 is to first perform dynamic threshold adjustment on the region in the image where the prior feature points may exist to complete binarization, and then use the find Contours function of the Open CV open source library to extract feature point contours to obtain the coordinates of the projection points of the visual feature markers in the camera image; the above settings can significantly improve the accuracy and efficiency of feature point recognition; subsequently, by using these coordinate information, combining the camera intrinsic and extrinsic parameter matrices, and through a perspective transformation algorithm, the position and pose of the target object in the three-dimensional space can be accurately calculated.

[0089] In a preferred solution, the specific way of dynamic threshold adjustment in step 3 is to set an initial value of the threshold value, adjust the initial threshold value through a preset adjustment formula to adapt to the differences in brightness of infrared light beads and the influence of environmental light noise at different distances and directions; the above settings can ensure that the infrared sensor can accurately detect and trigger an alarm under different conditions, improve the stability and reliability of the system; at the same time, this solution also includes an automatic calibration function that can periodically verify and adjust the threshold value, further reducing the occurrence of false positives and false negatives.

[0090] In a preferred solution, the step 3 has a preset tolerance for stray points appearing in the binarization process, which is specifically manifested as accepting feature points within a preset number of set multiples into the next step of processing, and at the same time, the single feature point extraction loop is repeated for several times until a proper number of feature points are extracted or the feature points exceed the preset number range; the above settings aim to balance the accuracy and efficiency of feature point extraction, reduce misjudgment caused by stray point interference, and at the same time ensure that the number of feature points meets the subsequent analysis requirements, enhancing the robustness and adaptability of the algorithm.

[0091] In the preferred scheme, the step 4 of screening and matching the feature points extracted in step 3 specifically includes finding three collinear feature points and calculating their parallel degree, evaluating the distance between the feature points and calculating their equidistance, determining the specific position of the feature points, and checking the pose estimation result the shaft and the direction of gravity, and calculating the comprehensive score of all possible combinations of feature points; the above settings ensure the accuracy and robustness of feature point matching; the feature point combination with the highest comprehensive score is selected as the final matching result for subsequent pose estimation and optimization, thereby improving the accuracy and stability of the entire system.

[0092] In the preferred scheme, the step 4 of checking the pose estimation result the shaft and the direction of gravity, a preset alignment deviation threshold is set, and if the deviation of the pose estimation result the deviation of the shaft and the direction of gravity exceeds the threshold, it is determined that the pose solution result corresponding to the current feature point combination is invalid, and the feature point combination with the second highest score and exceeding the threshold is selected from all possible feature point combinations in step 4, and the pose solution based on the PnP algorithm is executed again, until an effective pose result is obtained the deviation of the shaft and the direction of gravity meets the threshold requirement, ensuring the physical reasonableness of the pose solution; the above settings not only improve the accuracy of pose estimation, but also effectively avoid the deviation of pose solution caused by incorrect feature point combination, enhancing the robustness and reliability of the system, and providing a solid foundation for subsequent three-dimensional reconstruction, navigation positioning and other applications.

[0093] In the preferred scheme, when calculating the comprehensive score of all possible combinations of feature points in step 4, the calculation dimensions of the comprehensive score at least include the feature point parallel degree score, the equidistance score, and the shaft and the direction of gravity alignment score, each dimension score is weighted and summed according to the preset weight to obtain the comprehensive score, and only when the comprehensive score is greater than or equal to the preset qualified threshold, the feature point combination has the qualification to participate in the subsequent PnP pose solution, thereby further screening the effective feature point combination and improving the pose solution accuracy; the above settings effectively avoid the pose solution error caused by improper selection of feature points, enhancing the robustness and accuracy of the system; at the same time, this scheme is flexible, and the weight and qualified threshold of each dimension score can be adjusted according to the specific application requirements to realize the optimal feature point combination screening strategy.

[0094] In the preferred solution, the specific position of the feature points in step 4 is determined as follows: the midpoint of the three collinear feature points is recorded as point A, and one feature point other than the three feature points is connected as point D. The AD line is rotated counterclockwise with A as the axis, and the other two feature points B and C are determined according to the order in the rotation process, thereby determining the specific correspondence of the four feature points. The above arrangement accurately defines the arrangement order and relative position relationship of the feature points, improving the accuracy and efficiency of feature matching. In the rotation process, by setting a reasonable rotation angle threshold, the positioning accuracy of the feature points can be further refined to ensure the smooth progress of the subsequent processing steps.

[0095] In the preferred solution, the specific process of calculating the degree of light center offset of the target distance in step 6 is as follows: the motion target pose is re-projected onto the camera pixel coordinate system to obtain the projection point coordinates, and the light center coordinates are preset known values. The difference between the projection point coordinates and the light center coordinates is calculated to obtain the degree of light center offset of the target distance. At the same time, since different regions in the image have a non-linear relationship with the final pose estimation result, an inverse tangent function is designed as a loss function to calculate the offset loss, so as to more accurately reflect the influence of the offset degree on the observation quality. The above arrangement effectively improves the accuracy and robustness of target tracking. In addition, the deep learning algorithm is combined to efficiently extract and match image features, further optimizing the efficiency of target detection and recognition, and ensuring stable tracking performance in complex and variable environments.

[0096] In the preferred solution, the specific strategy for controlling the rotation of the camera holder in stages in step 7 is as follows: when the moving target moves quickly, the overall observation quality is poor. At this time, 2-3 cameras with the worst observation quality are selected for tracking, and the rotation of the holder of these cameras is controlled to adjust the observation direction, and the remaining cameras remain stationary. When the moving target moves slowly or hovers, the overall observation quality is good. At this time, one camera with the worst observation quality is selected for tracking, and the rotation of its holder is controlled, and the remaining cameras remain stationary to ensure that the observation quality returns to the best state faster, reduces the overall fluctuation, avoids losing the target, and improves the stability of the tracking result. The above arrangement can dynamically adapt to targets in different motion states, effectively balancing tracking accuracy and resource consumption, and ensuring efficient observation and tracking efficiency in complex scenes.

[0097] In a preferred scheme, in the linear control model constructed in step 7, the state quantity is set as the observed mass, and the control quantity is set as the camera holder rotation angle, the offset loss calculated in step 6 is input into the controller to construct the linear model of the system; at the same time, the controller and the cost function at the setting time are designed, and the cost function is optimized and solved, the goal is to obtain the minimum control quantity, and then the control quantity of the camera holder rotation angle at this time is determined, and the accurate control of the camera holder rotation is realized; the above setting can significantly improve the stability and response speed of the system; in real-time application, the system can quickly adjust the camera holder rotation angle, effectively compensate the offset loss of the observed mass, ensure the high precision and stability of image acquisition, and lay a solid foundation for subsequent image processing and data analysis.

[0098] In summary, the application provides a multi-camera cooperative state estimation method based on active infrared visual features, which effectively solves the long-standing problems of insufficient precision and robustness in the field of motion target recognition, real-time positioning and attitude calculation between cooperative systems, especially unmanned aerial vehicles and ground observation platforms. In view of the limitations of passive visual markers in the prior art, such as being easily affected by environmental light, and performance degradation under dark or complex lighting conditions, the application adopts an active infrared visual marker design, which realizes stable feature recognition in harsh environments by arranging a specific geometric shape "T" shaped infrared lamp bead combination. At the same time, in view of the problems of limited observation field of view of single fixed camera scheme, high cost and poor environmental adaptability of multi-fixed camera scheme, the application innovatively proposes a multi-camera cooperative scheduling algorithm, which dynamically allocates camera control quantity weight according to the target motion state and observation quality, realizes efficient and stable target tracking, and significantly improves the environmental adaptability and dynamic performance of the system.

[0099] The application solves the deficiencies of the prior art in harsh environment adaptability, dynamic target tracking accuracy and system robustness by deeply fusing active infrared visual marker design, fine image feature processing, PnP pose solution and multi-camera cooperative scheduling technology. The low-cost and easy-to-deploy characteristics of the application make it widely applicable in unmanned aerial vehicle cluster cooperation, robot inspection, indoor positioning and other fields, showing significant engineering application value and promotion prospects. Specifically, the application not only replaces the traditional passive marker with an active infrared visual marker, effectively overcoming the problem of environmental light interference, but also improves the feature point recognition accuracy and stability in complex lighting and noise environments by designing a dynamic threshold adjustment binarization processing method and a specific geometric feature matching algorithm. Furthermore, the application creatively combines active infrared visual features with multi-camera cooperative observation to construct a high-precision and high-robustness state estimation system, achieving millimeter-level positioning accuracy in harsh environments such as GNSS denial and dim light. By dynamically scheduling the camera observation direction, the application optimizes the allocation of camera resources, reduces the number of cameras while achieving more efficient target tracking, and reduces the system cost and deployment difficulty, providing a new technical solution for robot positioning, unmanned aerial vehicle cooperation and other fields. Experimental verification shows that the RMSE of the scheme is less than 0.1m in the range of 4m flight height, the performance is excellent, and the effectiveness and practicality of the scheme in complex environments are proved, opening up a new technical path for target recognition and positioning in dynamic scenes.

Claims

1. A multi-camera cooperative state estimation method based on active infrared visual features, characterized in that, Includes the following steps: Step 1: Construct a visual feature marker combination with a specific "T" shaped geometry composed of infrared LED beads, place this marker combination on the moving target, and simultaneously deploy a multi-camera collaborative module equipped with infrared filters on the ground observation platform; Step 2: The multi-camera collaboration module acquires images and videos containing moving targets in real time through infrared cameras, and uses infrared filters to filter out most of the light noise in the environment; Step 3: Process the image obtained in Step 2 to obtain the coordinates of the projection points of the visual feature markers in the camera image; Step 4: Filter and match the feature points extracted in Step 3; Step 5: Select the set of feature points with the highest score in Step 4 that exceeds the threshold, and use the PnP algorithm to complete the pose calculation of the moving target; Step 6: Based on the moving target pose estimated in Step 5, reproject it onto the pixel coordinate system of each camera, and calculate the offset of the target from the optical center and the observation quality; Step 7: Based on the observation quality of a single camera and the total observation quality of all cameras obtained in Step 6, dynamically allocate the weights of the control variables in the linear control model, and control the rotation of the camera gimbal in stages to achieve tracking of moving targets.

2. The multi-camera cooperative state estimation method based on active infrared visual features according to claim 1, characterized in that: In step 1, the "T"-shaped visual feature marker combination consists of four infrared LEDs, which are located at the four corners of the "T". Multiple "T"-shaped visual feature marker combinations can be arranged on the moving target to improve the accuracy and robustness of pose calculation.

3. The multi-camera cooperative state estimation method based on active infrared visual features according to claim 2, characterized in that: In step 1, the multiple cameras deployed on the ground observation platform are active vision cameras. These active vision cameras can be driven by servo motors to rotate the gimbal, thereby dynamically adjusting the camera's observation direction, expanding the camera's field of view, improving the effectiveness of acquiring image information of moving targets, reducing the number of cameras deployed, and enhancing the system's adaptability in dynamic scenes, while taking into account both the high accuracy and efficiency of state estimation.

4. The multi-camera cooperative state estimation method based on active infrared visual features according to claim 1, characterized in that: In step 2, when the multi-camera collaboration module acquires images and videos, a multi-camera synchronization triggering mechanism is adopted. By using a preset synchronization signal, all infrared cameras are controlled to acquire image frames at the same time, avoiding deviations in the position of moving targets caused by time differences in acquisition by different cameras. This ensures the consistency of image data acquired by multiple cameras in the time dimension, providing basic data support for subsequent multi-camera observation quality assessment and collaborative scheduling, and further improving the accuracy of multi-camera collaborative state estimation.

5. The multi-camera cooperative state estimation method based on active infrared visual features according to claim 1, characterized in that: The process of processing the image obtained in step 2 in step 3 is as follows: first, the regions in the image that may have prior feature points are dynamically thresholded to complete binarization; then, the find Contours function of the OpenCV open-source library is used to extract the feature point contours to obtain the coordinates of the projection points of the visual feature markers in the camera image.

6. The multi-camera cooperative state estimation method based on active infrared visual features according to claim 5, characterized in that, The specific method for dynamic threshold adjustment in step 3 is as follows: set an initial value for the threshold, and adjust the initial threshold using a preset adjustment formula to adapt to the differences in the brightness of the infrared lamp beads at different distances and directions, as well as the influence of ambient light noise.

7. The multi-camera cooperative state estimation method based on active infrared visual features according to claim 6, characterized in that: In step 3, there is a preset tolerance for noise that appears during the binarization process. Specifically, feature points within a preset number of marker points (a set multiple) are allowed to proceed to the next step. At the same time, the loop of single feature point extraction will be repeated several times until a suitable number of feature points or feature points exceeding the preset number range are extracted.

8. The multi-camera cooperative state estimation method based on active infrared visual features according to claim 1, characterized in that: Step 4 involves filtering and matching the feature points extracted in step 3. Specifically, this includes finding three collinear feature points and calculating their parallelism, evaluating the distance between feature points and calculating their equidistance, determining the specific location of the feature points, and verifying the pose estimation results. The degree of alignment between the axis and the direction of gravity is calculated, and a comprehensive score is calculated for all possible combinations of feature points.

9. The multi-camera cooperative state estimation method based on active infrared visual features according to claim 8, characterized in that: In step 4, the pose estimation results are verified. When assessing the alignment between the axis and the direction of gravity, a preset alignment deviation threshold is used. If the pose estimation results show... If the deviation between the axis and the direction of gravity exceeds the threshold, the pose calculation result corresponding to the current feature point combination is deemed invalid. Step 4 is repeated, selecting the second-highest score feature point combination that exceeds the threshold from all possible feature point combinations, and the pose calculation and consistency check based on the PnP algorithm are performed again until a result is obtained. The effective pose result is obtained when the deviation between the axis and the direction of gravity meets the threshold requirement, ensuring the physical rationality of the pose calculation.

10. The multi-camera cooperative state estimation method based on active infrared visual features according to claim 8, characterized in that: In step 4, when calculating the comprehensive score for all possible combinations of feature points, the calculation dimensions of the comprehensive score include at least the feature point parallelism score, the isometry score, and... The score for the alignment of the axis with the direction of gravity is calculated by weighting the scores of each dimension according to preset weights and summing them to obtain a comprehensive score. Only when the comprehensive score is greater than or equal to a preset qualified threshold is the feature point combination qualified to participate in the subsequent PnP pose calculation. This further filters effective feature point combinations and improves the accuracy of pose calculation.

11. The multi-camera cooperative state estimation method based on active infrared visual features according to claim 8, characterized in that, The method for determining the specific location of the feature points in step 4 is as follows: the midpoint of the three collinear feature points is denoted as point A, and the feature point connecting these three feature points is denoted as point D. With point A as the axis, the AD line is rotated counterclockwise. Based on the order of rotation, the other two feature points, points B and C, are determined, thereby clarifying the specific correspondence of the four feature points.

12. The multi-camera cooperative state estimation method based on active infrared visual features according to claim 1, characterized in that, The specific process for calculating the target's distance from the optical center in step 6 is as follows: the pose of the moving target is reprojected onto the camera pixel coordinate system to obtain the coordinates of the projection point. The coordinates of the optical center are preset known values. The distance of the target from the optical center is obtained by calculating the difference between the coordinates of the projection point and the coordinates of the optical center. At the same time, since different regions in the image have a nonlinear relationship with the final pose estimation result, an arctangent function is designed as a loss function to calculate the offset loss, so as to more accurately reflect the impact of the offset on the observation quality.

13. The multi-camera cooperative state estimation method based on active infrared visual features according to claim 12, characterized in that, The specific strategy for controlling the camera gimbal rotation in step 7 is as follows: When the moving target moves quickly, the overall observation quality is poor. At this time, select 2-3 cameras with the worst observation quality for tracking, control the gimbal rotation of these cameras to adjust the observation direction, and keep the other cameras stationary. When the moving target moves slowly or hovers, the overall observation quality is good. At this time, select 1 camera with the worst observation quality for tracking, control its gimbal rotation, and keep the other cameras stationary to ensure that the observation quality returns to the best state more quickly, while reducing overall fluctuations, avoiding target loss, and improving the stability of tracking results.

14. The multi-camera cooperative state estimation method based on active infrared visual features according to claim 13, characterized in that: In the linear control model constructed in step 7, the state variable is set as the observation quality, and the control variable is set as the camera gimbal rotation angle. By inputting the offset loss calculated in step 6 into the controller, the linear model of the system is constructed. At the same time, the controller and cost function at a set time are designed, and the cost function is optimized to obtain the minimum control variable. Then, the control variable of the camera gimbal rotation angle at that time is determined, so as to achieve precise control of the camera gimbal rotation.

Images