Real-time positioning method for ground winding inspection robot

Abstract

The invention discloses a real-time positioning method for a ground aircraft-winding inspection robot, and the method comprises the steps: obtaining the wide-area point cloud data of an aircraft, and building a stable coordinate system with the aircraft as a reference through the recognition and registration of a plurality of semantic key points, such as a nose cone and a wingtip, and a pre-stored template; the method comprises the following steps: acquiring laser radar and camera data in real time, inputting the data into a pre-trained multi-modal deep learning model, and outputting probability distribution representing the current pose of a robot under an aircraft coordinate system by the model; in a filter, the mean value of the probability distribution is used as an observation value, the variance is used as dynamic observation noise, optimal fusion with motion prediction of the robot itself is carried out, and a final smooth pose is generated. According to the method, the real-time monitored pose uncertainty is utilized, the robot is actively controlled to adjust the position so as to improve the sensing quality when the observation view angle is poor, high-precision and high-robustness aircraft relative positioning is achieved, and the method has the active sensing ability.

Description

Technical Field

[0001] This invention relates to the field of robot positioning and automated inspection technology, and in particular to a real-time positioning method for a ground-based inspection robot. Background Technology

[0002] With the aviation industry's ever-increasing demands for safety and efficiency, the use of automated equipment for aircraft external inspections has become a mainstream development trend. Among various automation solutions, ground mobile robots demonstrate enormous application potential due to their inherent advantages in endurance, payload capacity, and ground operation safety. Currently, mainstream robot positioning technologies fall into two main categories: one is a high-precision absolute positioning scheme based on the Global Navigation Satellite System (GNSS), particularly combined with Real-Time Dynamic Differential (RTK) technology; the other is based on Simultaneous Localization and Mapping (SLAM) technology, which uses LiDAR or Vision sensors to build an environmental map in real time and track the robot's own pose within this map. These technologies have enabled autonomous navigation for robots to a certain extent, laying the foundation for automated inspections.

[0003] However, applying these existing technologies to aircraft bypass inspection scenarios fails to meet the requirements of high reliability and high precision. First, GNSS-based positioning methods suffer significant accuracy degradation or even failure in environments with weak satellite signals or multipath interference, such as indoor hangars or partially obstructed aprons, making stable operation in all weather conditions and scenarios impossible. Second, SLAM-based positioning methods are highly dependent on the richness and stability of environmental features. However, aircraft parking postures exhibit millimeter- to meter-level deviations each time. For a large aircraft, these deviations can cause significant misalignment between the robot's preset inspection trajectory and the aircraft's actual position. Furthermore, open environments like airport aprons lack sufficient static references, easily leading to SLAM system tracking failures or severe cumulative errors. More importantly, existing localization methods typically output a definite pose estimate, lacking a quantitative assessment of the uncertainty of this estimate. This makes it impossible for the robot to judge the reliability of the current localization result. At the same time, when faced with sensor instantaneous noise or perception degradation caused by poor viewing angle, it also leads to poor robustness of robot localization and the inability to proactively take strategies to improve perception quality. This directly restricts the accuracy and reliability of near-range defect identification. Summary of the Invention

[0004] The purpose of this section is to outline some aspects of embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0005] In view of the aforementioned existing problems, this invention is proposed. Therefore, this invention provides a real-time positioning method for a ground-based inspection robot, to solve the problems mentioned in the background art.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: a real-time positioning method for a ground-based inspection robot, comprising: The wide-area point cloud data of the aircraft is acquired, and multiple preset semantic key points are identified and located in the wide-area point cloud data. By registering the observation positions of the multiple semantic key points with the pre-stored key point map template of the corresponding aircraft model, the aircraft coordinate system based on the aircraft is calculated. After the aircraft coordinate system is determined, the point cloud data and image data collected by the lidar and camera on the robot at the current moment are acquired in real time. The point cloud data and image data at the current moment are input into a pre-trained multimodal deep learning model to output a probability distribution that characterizes the robot's current pose in the aircraft coordinate system. Based on the probability distribution, the real-time pose of the robot in the aircraft coordinate system is determined.

[0007] As a preferred embodiment of the real-time positioning method for a ground-based aircraft inspection robot described in this invention, the semantic key points are parts of the aircraft with stable three-dimensional structural features, selected from at least two of the following: the nose cone apex, wingtip, engine nacelle center, connection point between main landing gear and wing, and vertical tail apex.

[0008] As a preferred embodiment of the real-time positioning method for a ground-based inspection robot described in this invention, the registration includes: An algorithm based on random sample consistency is used to iteratively solve a rigid body transformation that minimizes the alignment error between the observed position of the semantic key point and the key point map template, and this rigid body transformation is used to determine the aircraft coordinate system.

[0009] As a preferred embodiment of the real-time positioning method for a ground-based inspection robot described in this invention, the method further includes, before inputting the current point cloud data and image data into a pre-trained multimodal deep learning model: The aircraft region in the image data is identified using an object detection model, and a two-dimensional bounding box is generated. Based on the two-dimensional bounding box and camera parameters, the target frustum in three-dimensional space is calculated; The point cloud data is filtered using the target view frustum to remove irrelevant background point clouds.

[0010] As a preferred embodiment of the real-time positioning method for a ground-based inspection robot according to the present invention, the probability distribution includes at least: A six-dimensional mean vector representing the robot's most likely pose; And a six-dimensional variance vector for quantifying the degree of uncertainty of the most likely pose in each dimension.

[0011] As a preferred embodiment of the real-time positioning method for a ground-based inspection robot described in this invention, the multimodal deep learning model is trained using a loss function with negative log-likelihood as the objective, and the loss function simultaneously optimizes the prediction accuracy of the mean vector and the numerical rationality of the variance vector during the training process.

[0012] As a preferred embodiment of the real-time positioning method for a ground-based aircraft inspection robot according to the present invention, wherein: determining the real-time pose of the robot in the aircraft coordinate system based on the probability distribution includes: The six-dimensional mean vector output by the multimodal deep learning model is used as the observation value, and the six-dimensional variance vector is converted into the observation noise covariance matrix. In a filter, the motion prediction values ​​from the inertial measurement unit or wheel speedometer are updated using the observed values ​​and the observed noise covariance matrix to generate the final smooth pose.

[0013] As a preferred embodiment of the real-time positioning method for a ground-based inspection robot described in this invention, the method further includes an active perception and judgment step, which includes: The value of the six-dimensional variance vector is monitored in real time. When at least one component or a combination thereof of the variance vector is continuously higher than a preset uncertainty threshold within a preset time period, the current observation perspective is determined to be poor.

[0014] As a preferred embodiment of the real-time positioning method for a ground-based inspection robot described in this invention, the method further includes, after determining that the current observation angle is unfavorable: Send a command to the robot's motion control system to execute a preset repositioning action; The relocation action moves the robot to a new observation position, acquiring new point cloud data and image data to reduce the uncertainty of positioning.

[0015] As a preferred embodiment of the real-time positioning method for ground-based inspection robots described in this invention, the multimodal deep learning model adopts a structure containing recurrent network units. When processing sensor data at the current moment, the structure also receives and utilizes hidden state information generated based on data from the previous moment to achieve temporal smoothness in pose prediction.

[0016] Compared with existing technologies, the beneficial effects of this solution are: 1. This invention establishes a stable coordinate system based on the aircraft itself through wide-area scanning and semantic key point registration in the initial stage. This allows all subsequent positioning to be performed within this coordinate system, eliminating dependence on the external environment (such as GNSS signals or fixed reference objects). This ensures that the robot can achieve consistent and high-precision relative positioning around the aircraft, regardless of where it is parked, laying the foundation for the accurate execution of inspection trajectories.

[0017] 2. Employing a multimodal deep learning model that integrates geometric information from LiDAR and texture / semantic information from the camera, this approach demonstrates greater robustness compared to single-sensor solutions when facing challenging scenarios such as varying lighting conditions and sparse aircraft surface features. Furthermore, visually guided point cloud filtering effectively eliminates dynamic and static interference in the airport environment, focusing localization on the aircraft itself and enhancing anti-interference capabilities.

[0018] 3. Furthermore, by outputting the probability distribution (mean and variance) of the pose through the deep learning model, not only is the pose estimated, but also a quantitative assessment of the reliability of this estimate is obtained. This uncertainty information is used in two ways: First, as dynamic weights in the filter, it enables adaptive fusion of observation data of different quality, improving the smoothness and accuracy of the final pose; Second, as a trigger signal, it drives the robot to execute an active perception strategy, improving the observation perspective through autonomous movement when the localization quality deteriorates, upgrading the system from passive perception to intelligent active localization, and improving the robot's task success rate in scenarios with degraded perception. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 This is a flowchart illustrating the real-time positioning method for a ground-based inspection robot according to an embodiment of the present invention. Detailed Implementation

[0020] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0021] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0022] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0023] This invention is described in detail with reference to the schematic diagrams. When detailing the embodiments of this invention, for ease of explanation, the cross-sectional views illustrating the device structure may be partially enlarged, not adhering to the usual scale. Furthermore, the schematic diagrams are merely examples and should not be construed as limiting the scope of protection of this invention. In actual fabrication, the three-dimensional spatial dimensions of length, width, and depth should be included.

[0024] Furthermore, in the description of this invention, it should be noted that the terms "upper," "lower," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are used solely for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. In addition, the terms "first," "second," or "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0025] Unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" in this invention should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; similarly, they can refer to mechanical connections, electrical connections, or direct connections, or indirect connections through an intermediate medium, or internal connections between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0026] Example 1 Reference Figure 1This is the first embodiment of the present invention, which provides a real-time positioning method for a ground-based inspection robot, comprising: S1. Acquire wide-area point cloud data of the aircraft, identify and locate multiple preset semantic key points in the wide-area point cloud data, and calculate the aircraft coordinate system based on the aircraft by registering the observation positions of multiple semantic key points with the pre-stored key point map template of the corresponding aircraft model. It should be noted that the goal of this step is to overcome the problem of the aircraft's inconsistent parking position each time, and to establish a stable and reliable reference benchmark for subsequent real-time tracking through a one-time, high-precision calibration. It is important to note that this step is not performed in real time, but rather when the robot begins its inspection task or when it first approaches the aircraft.

[0027] Furthermore, a wide-area feature scan is performed to obtain wide-area point cloud data of the aircraft.

[0028] Specifically, the inspection robot does not immediately begin close-range tracking, but instead executes a pre-set wide-area scanning program. In this program, the robot moves within a range of approximately 10 to 30 meters from the aircraft's main body, following a pre-defined sparse pathpoint, and pauses at several advantageous locations where different structural features of the aircraft can be observed (e.g., simultaneously seeing the nose and one wing, or the wing and tail). At each pause, the robot collects a frame of 3D point cloud data using its onboard LiDAR. By initially stitching together multiple frames of point cloud data collected along the way using the robot's own odometry information, a non-real-time, but globally consistent sparse point cloud set covering the main outline of the aircraft is ultimately formed. It should be noted that choosing a relatively long distance for scanning is necessary to ensure a sufficiently wide field of view for each scan, enabling the capture of multiple distant structural features at once. This provides stronger geometric constraints for subsequent global registration, thereby improving the stability and accuracy of the solution.

[0029] Furthermore, multiple pre-defined semantic key points are identified and located in the acquired wide-area point cloud data. It is important to emphasize that this process is feature extraction based on prior structural knowledge, rather than blind feature matching.

[0030] Specifically, based on the aircraft type information for this task (e.g., the aircraft model to be inspected is "A320" issued by the host system), the system loads the corresponding key point map template from the database. It is important to emphasize that this template can be extracted from the official 3D computer-aided design (CAD) model provided by the aircraft manufacturer, or obtained through high-precision 3D scanning and manual calibration of the actual aircraft. Furthermore, this template is a data structure that pre-stores the precise 3D coordinates of multiple semantic key points on the aircraft model in the standard aircraft coordinate system. These semantic key points are easily identifiable parts of the aircraft with stable 3D structural features. In this example, these semantic key points include: the apex of the nose cone, the left and right wingtips, the geometric center of the left and right engine nacelles, the connection center between the main landing gear strut and the wing, and the apex of the vertical stabilizer. These semantic key points are fixed and unchanging within the rigid structure of the aircraft itself, serving as ideal natural "anchor points."

[0031] Further, the wide-area point cloud data acquired in the previous step is processed to locate the actual 3D coordinates of these semantic key points. This process typically consists of two steps: The first step is to use point cloud segmentation algorithms, such as region growing clustering algorithms or Random Sample Consensus (RANSAC) algorithms, to perform coarse structured segmentation of the wide-area point cloud and identify macroscopic geometric components such as the nose, wings, and engine nacelles.

[0032] The second step involves applying specific geometric algorithms to precisely locate key points on the segmented component point cloud. For example, a quadratic surface fitting is performed on the point cloud of the nose region to determine the vertex coordinates of the surface and identify the "nose cone vertex"; a cylindrical model fitting is performed on the point cloud of the engine nacelle region to extract the intersection of its central axis and end face as the "engine nacelle center"; and for the wing contour point cloud, the "wingtip" is determined by calculating the curvature of each point or finding the point farthest from the fuselage. Through this process, a set of observation key points in the robot's current world coordinate system is obtained. .

[0033] Furthermore, by registering the observation positions with the template, the aircraft coordinate system is calculated, thereby determining an optimal rigid body transformation that can transform the observed keypoint set... Point set in the key point diagram template Achieving optimal alignment. This rigid body transformation defines the position and attitude of the aircraft coordinate system in the current world coordinate system. The registration process is implemented as follows: An iterative algorithm based on random sample consensus (specifically, an improved Iterative Closest Point (ICP) algorithm in this embodiment) is used to solve the transformation. The goal of this algorithm is to find the rotation matrix that minimizes the following error function. Translation vector : in, and are the optimal rotation matrix and translation vector to be solved, respectively. It is the first A three-dimensional coordinate vector of a semantic key point observed in a wide-area point cloud. Is with Correspondingly, in the key point diagram template, the first... The three-dimensional coordinate vectors of the key points. It is a robust kernel function (such as Huber loss) that aims to reduce the negative impact of large localization errors of individual keypoints (i.e., outliers) on the overall registration result, thereby enhancing the robustness of the algorithm. The iterative solution process of the algorithm is as follows: A minimal subset (e.g., 3 points) is randomly selected from the observation point set, along with their corresponding points in the template. An initial rigid body transformation is then calculated. This transformation is applied to all observation points, and the number of points whose distance to the template points is less than a certain threshold (i.e., the number of inliers) is counted. This process is repeated multiple times, and the transformation that yields the maximum number of inliers while minimizing inlier alignment error is ultimately selected. As the final solution.

[0034] It should be noted that this optimal rigid body transformation This refers to the transformation relationship from the aircraft coordinate system to the robot's current world coordinate system. At this point, the aircraft coordinate system has been successfully and accurately calibrated, providing a unified and reliable positioning benchmark for subsequent processing. Regardless of the aircraft's initial parking position, positioning during subsequent processing will be performed within this calibrated coordinate system centered on the aircraft.

[0035] S2. After the aircraft coordinate system is determined, the point cloud data and image data collected by the lidar and camera on the robot at the current moment are acquired in real time. The point cloud data and image data at the current moment are input into a pre-trained multimodal deep learning model to output a probability distribution that represents the robot's current pose in the aircraft coordinate system. It should be noted that after establishing a globally unique aircraft coordinate system in step S1, the goal of this step is to calculate the robot's pose relative to this coordinate system in real time with high frequency and high robustness. This process is achieved through a multimodal deep learning model capable of perceiving uncertainty.

[0036] Furthermore, during the robot's circling inspection movement, the control system triggers the lidar and camera at a fixed frequency (e.g., 10Hz) to synchronously acquire the current moment. Original lidar point cloud and camera images To improve the processing efficiency and accuracy of deep learning models, preprocessing of the raw data is necessary. The significance of this preprocessing lies in leveraging the semantic understanding capabilities of images to focus on key information in three-dimensional space, thereby eliminating irrelevant noise. This preprocessing process includes: The camera image at the current moment Input a lightweight, pre-trained object detection model (e.g., YOLOv5 or MobileNet-SSD). This model, trained on a large number of aircraft images, can quickly identify the main body of the aircraft in the image and output a 2D bounding box. Then, based on the camera's intrinsic parameter matrix (which includes focal length and principal point information) and the calibrated extrinsic parameter transformation relationship between the camera and the LiDAR, the two-dimensional bounding box on the image can be generated. When projected back into three-dimensional space, it forms a square pyramid with the lidar as its vertex and the projection of the bounding box in space as its base, which is the target view cone. The view frustum precisely defines the area where the aircraft is located within the camera's field of view in three-dimensional space. Finally, by traversing the original point cloud... For each point in the graph, determine whether its 3D coordinates are located within the target's view frustum. Internally, only points located inside the view frustum are retained, forming a filtered dataset containing only point clouds relevant to the aircraft's main body. .

[0037] It should be noted that this preprocessing operation greatly reduces the amount of data that the model needs to process and effectively filters out background noise point clouds such as the ground, distant jet bridges, and other ground support equipment, enabling the depth model to focus on learning pose information from the geometric features of the aircraft surface.

[0038] Furthermore, the preprocessed point cloud data and corresponding camera image data (or based on the bounding box) The cropped image region is used as input and fed into a pre-trained multimodal deep learning model. The purpose of this multimodal deep learning model is not only to predict the pose, but also to predict the uncertainty of the prediction result.

[0039] Specifically, in this embodiment, the multimodal deep learning model adopts a structure that includes recurrent network units (such as Long Short-Term Memory Network LSTM or Gated Recurrent Unit GRU). Its structure consists of three parts: a point cloud encoder (such as PointNet++) for extracting point cloud geometric features, an image encoder (such as ResNet) for extracting image texture and semantic features, and a recurrent decoder for fusing multimodal features and incorporating temporal information.

[0040] Specifically, the workflow of a multimodal deep learning model is as follows: At any moment The point cloud encoder and image encoder respectively from and Extract the high-dimensional feature vector, then concatenate the two feature vectors and add them to the previous time step. The passed-in hidden state vector They are fed together into the recurrent network unit. This design allows the model to utilize historical motion information when inferring the pose at the current moment, thereby ensuring the temporal smoothness of the output pose and effectively suppressing pose jumps caused by single-frame sensor noise.

[0041] It is important to emphasize that the output of this model is not a single six-dimensional pose vector, but rather the parameters of a probability distribution. Specifically, the model outputs two six-dimensional vectors: A mean vector This vector represents the model-predicted most probable pose of the robot in the aircraft coordinate system, where the first three dimensions represent translation. The latter three dimensions represent rotation (e.g., in Euler angles or quaternions).

[0042] A variance vector Each component of this vector quantifies the degree of uncertainty (variance) of the pose mean vector in its corresponding dimension. For example, a larger uncertainty value means that the model is less certain about the current prediction. The confidence level of the coordinates is low.

[0043] Furthermore, to enable the model to learn this meaningful uncertainty, it is trained using a loss function targeting negative log-likelihood (NLL). It is assumed that the true value of the pose is... If the probability distribution predicted by the model is Gaussian, then the loss function optimized during training is... It can be represented as: in, It represents the six degrees of freedom for traversing poses. and The models are respectively in The mean and variance predicted in each dimension. This is the true label value in that dimension. It's important to emphasize that this loss function consists of two parts: the first part... This is the prediction error normalized by variance. When the variance of the model prediction... When the uncertainty is large (i.e., when the model is uncertain), the penalty for prediction error is reduced, thereby encouraging the model to "acknowledge" its uncertainty when it perceives insufficient information. Part Two This is a regularization term used to penalize excessively large variance predictions, preventing the model from blindly predicting extremely large variances in order to minimize the first part. By optimizing the trade-off between these two objectives, the model training aims not only to predict accurately (mean close to the true value) but also to provide a reasonable quantitative assessment of its prediction accuracy (variance reflects the true uncertainty). Therefore, the final output of the model is a set of probability distribution parameters containing rich information. .

[0044] It is important to emphasize that pre-trained multimodal deep learning models are obtained through the following steps: High-precision 3D models of various aircraft types are imported using a 3D simulation platform (e.g., Unreal Engine or Gazebo). An airport tarmac scene is simulated in the simulation environment, and a robot model with virtual LiDAR and cameras is deployed. The virtual robot is programmed to perform diverse inspection paths around the aircraft, and rendered images, simulated point cloud data, and the robot's precise pose ground truth values ​​are directly exported from the simulation environment at each time step to construct a large-scale training dataset. The specific structure of this multimodal deep learning model is as follows: the point cloud input is processed by a PointNet++ network containing three Set Abstraction modules to extract a 128-dimensional geometric feature vector. The image input is processed by the backbone of a pre-trained ResNet-34 network to extract a 256-dimensional visual feature vector. These two feature vectors are concatenated and then reduced to 256 dimensions through a fully connected layer as the multimodal fusion feature. This fusion feature is then compared with the hidden state of the previous time step. Both are input into a two-layer GRU network with 256 hidden layers. Finally, the GRU output is passed through two parallel fully connected layers (MLPs) to regress a six-dimensional mean vector and a six-dimensional variance vector, respectively. During model training, the Adam optimizer is used with an initial learning rate of 1e-4 and a batch size of 32. The constructed training dataset is input into the model, and end-to-end training is performed using the aforementioned negative log-likelihood loss function for a total of 100 epochs. The model weights with the minimum loss on the validation set are saved, resulting in the pre-trained multimodal deep learning model.

[0045] S3. Determine the robot's real-time pose in the aircraft coordinate system based on the probability distribution; It's important to note that this step doesn't simply use the pose mean vector output from step S2 as the final result. Instead, it treats it as an external observation with uncertainties and optimally fuses it with the robot's prior motion information within a classic Bayesian filtering framework (such as the Extended Kalman Filter (EKF) or the Unscented Kalman Filter (UKF)). The physical meaning is that by combining high-frequency but cumulatively erroneous internal odometry (motion prediction) with low-frequency but globally accurate external perception (observation updates), a smooth and accurate final pose estimate is generated.

[0046] Specifically, in this embodiment, an extended Kalman filter (EKF) is used to implement this process.

[0047] Further, the execution state prediction step is performed.

[0048] Specifically, firstly, the robot's own internal sensors, such as inertial measurement units (IMUs) or wheel speedometers, are used to predict the current moment. The pose of the robot is then determined. Next, the robot's state vector is defined. It contains the robot's six-dimensional pose in the aircraft coordinate system, and is based on the pose from the previous moment. up to the current moment The motion increment read from the IMU or wheel speed gauge within the time interval. (e.g., linear velocity and angular velocity), through a nonlinear motion model To predict the current state: in, It was the previous moment The optimal state estimate after fusion, and This is the prior state estimate (i.e., the predicted value) at the current moment. Simultaneously, the uncertainty of the state (i.e., the covariance matrix) The propagation also occurs thereafter. The predicted covariance is: in, It is a motion model Regarding the status The Jacobian matrix describes how uncertainty propagates linearly with motion. It is the process noise covariance matrix, which quantifies the uncertainty of the motion model itself (e.g., errors introduced by wheel slippage, ground inequality, etc.) and is a pre-calibrated system parameter.

[0049] Further, an observation update step is performed. The predicted values ​​are corrected using the probability distribution output by the deep learning model in step S2.

[0050] Specifically, the six-dimensional mean vector output from step S2... Observations as filters Then the six-dimensional variance vector Convert to an observation noise covariance matrix In this embodiment, it is assumed that the observation noise between dimensions is uncorrelated. It is a diagonal matrix: It needs to be explained that the step of performing observation updates involves dynamically and quantitatively passing the uncertainty of the deep learning model's own predictions to the filter. When the model is very confident in its predictions for a certain dimension (corresponding to...), (The value is very small) The corresponding term is very small, so the filter will "trust" this observation more; conversely, when the model is uncertain ( If the value is large, the filter will reduce the weight of that observation and rely more on its own motion prediction.

[0051] Furthermore, the Kalman gain is calculated. It needs to be explained that this Kalman gain... It is a key weight matrix used to balance the importance of predicted and observed values: in, It is an observation model The Jacobian matrix. In the scheme of this invention, since the depth model directly outputs the pose, the observation model can be regarded as an identity mapping, therefore, It is an identity matrix. As can be seen in the formula, the Kalman gain is directly affected by... This mitigates the impact of [the factors], thus achieving adaptive weighting.

[0052] Furthermore, by utilizing this Kalman gain, the observation information is fused in to obtain the current time. Final, optimal posterior state estimate : At the same time, update the covariance matrix of the state: in, It is represented as an identity matrix.

[0053] It should be noted that this final output This refers to the robot's real-time, smooth, and high-precision pose in the aircraft coordinate system.

[0054] Furthermore, the present invention also includes an active sensing strategy for applying uncertain information, the execution process of which is as follows: Real-time monitoring of the variance vector directly output by step S2 Set a preset uncertainty threshold vector. When at least one component of the variance vector, or a combination thereof (such as its norm), is found to be consistently above a corresponding threshold for a preset time window (e.g., 0.5 seconds), the system determines that the current observation perspective is poor. For example, when a robot is facing a large, flat, geometrically unremarkable section of an airplane's fuselage skin, its degrees of freedom for tangential translation and rotation around the normal become difficult to constrain, leading to an increase in variance in the corresponding dimensions. The reading remains consistently high. Once the observation viewpoint is determined to be poor, the localization module sends a high-level command to the robot's motion control system to execute a pre-set relocation action. This action is not a blind movement, but a small, purposeful maneuver, such as "translating 0.5 meters to the left and rotating 10 degrees to the right." The purpose of this action is to actively change the robot's observation viewpoint, enabling it to recapture the geometric or textural features that effectively constrain all degrees of pose freedom (such as the edge of an airplane window, an access hatch, or a change in the curvature of a wing).

[0055] It should be noted that this proactive perception strategy transforms the robot from a passive observer into an intelligent agent capable of actively seeking information to ensure its own positioning accuracy, thereby improving the robot's success rate and reliability in the entire inspection task scenario.

[0056] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0057] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0058] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0059] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0060] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0061] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A real-time positioning method for a ground-based inspection robot, characterized in that, include: The wide-area point cloud data of the aircraft is acquired, and multiple preset semantic key points are identified and located in the wide-area point cloud data. By registering the observation positions of the multiple semantic key points with the pre-stored key point map template of the corresponding aircraft model, the aircraft coordinate system based on the aircraft is calculated. After the aircraft coordinate system is determined, the point cloud data and image data collected by the lidar and camera on the robot at the current moment are acquired in real time. The point cloud data and image data at the current moment are input into a pre-trained multimodal deep learning model to output a probability distribution that characterizes the robot's current pose in the aircraft coordinate system. Based on the probability distribution, the real-time pose of the robot in the aircraft coordinate system is determined.

2. The real-time positioning method for a ground-based inspection robot as described in claim 1, characterized in that, The semantic key points are parts of the aircraft that have stable three-dimensional structural features, selected from at least two of the following: the apex of the nose cone, the wingtip, the center of the engine nacelle, the connection point between the main landing gear and the wing, and the apex of the vertical tail.

3. The real-time positioning method for a ground-based inspection robot as described in claim 1, characterized in that, The registration includes: An algorithm based on random sample consistency is used to iteratively solve a rigid body transformation that minimizes the alignment error between the observed position of the semantic key point and the key point map template, and this rigid body transformation is used to determine the aircraft coordinate system.

4. The real-time positioning method for a ground-based inspection robot as described in claim 1, characterized in that, Before inputting the current point cloud data and image data into a pre-trained multimodal deep learning model, the following steps are also included: The aircraft region in the image data is identified using an object detection model, and a two-dimensional bounding box is generated. Based on the two-dimensional bounding box and camera parameters, the target frustum in three-dimensional space is calculated; The point cloud data is filtered using the target view frustum to remove irrelevant background point clouds.

5. The real-time positioning method for a ground-based inspection robot as described in claim 1, characterized in that, The probability distribution includes at least: A six-dimensional mean vector representing the robot's most likely pose; And a six-dimensional variance vector for quantifying the degree of uncertainty of the most likely pose in each dimension.

6. The real-time positioning method for a ground-based inspection robot as described in claim 5, characterized in that, The multimodal deep learning model is trained using a loss function that targets negative log-likelihood. During training, the loss function simultaneously optimizes the prediction accuracy of the mean vector and the numerical reasonableness of the variance vector.

7. The real-time positioning method for a ground-based inspection robot as described in claim 5, characterized in that, Determining the robot's real-time pose in the aircraft coordinate system based on the probability distribution includes: The six-dimensional mean vector output by the multimodal deep learning model is used as the observation value, and the six-dimensional variance vector is converted into the observation noise covariance matrix. In a filter, the motion prediction values ​​from the inertial measurement unit or wheel speedometer are updated using the observed values ​​and the observed noise covariance matrix to generate the final smooth pose.

8. The real-time positioning method for a ground-based inspection robot as described in claim 5, characterized in that, The method also includes an active perception and judgment step, which includes: The value of the six-dimensional variance vector is monitored in real time. When at least one component or a combination thereof of the variance vector is continuously higher than a preset uncertainty threshold within a preset time period, the current observation perspective is determined to be poor.

9. The real-time positioning method for a ground-based inspection robot as described in claim 8, characterized in that, After determining that the current observation angle is poor, the method also includes: Send a command to the robot's motion control system to execute a preset repositioning action; The relocation action moves the robot to a new observation position, acquiring new point cloud data and image data to reduce the uncertainty of positioning.

10. The real-time positioning method for a ground-based inspection robot as described in claim 1, characterized in that, The multimodal deep learning model employs a structure containing recurrent network units. When processing sensor data at the current moment, the structure also receives and utilizes hidden state information generated based on data from the previous moment to achieve temporal smoothness in pose prediction.

Images