VisionMama and EPnP-based vehicle-mounted unmanned aerial vehicle recovery entrance relative pose measurement method

Abstract

The invention discloses a vehicle-mounted unmanned aerial vehicle recovery entrance relative pose measurement method based on VisionMama and EPnP, and the method comprises the steps: obtaining a current frame image, enabling a camera to participate in an entrance three-dimensional reference point set, cutting an entrance image block according to a search window, and inputting the image block into a VisionMama feature point generation network in combination with a time sequence memory vector; regression is carried out on the entrance two-dimensional feature point set and the shielding visible weight, and the memory vector is updated; a two-dimensional and three-dimensional corresponding relation is established according to a point sequence, a weighted corresponding relation set is generated, an initial relative pose is solved by adopting weighted EPnP, pose validity judgment is carried out based on a re-projection error and an entering channel deviation, and a recovery closed-loop control quantity is generated; updating a next frame of search window parameter by using the effective pose projection; the method can provide continuous pose and effectiveness constraints for mobile platform recovery alignment-entry.

Description

Technical Field

[0001] This invention relates to the field of UAV vision measurement and recovery control technology, and in particular to a method for measuring the relative pose of a vehicle-mounted UAV recovery inlet based on VisionMamba and EPnP. Background Technology

[0002] Vehicle-mounted drones often need to be recovered while the vehicle is moving at low speeds for tasks such as urban road mobility, field operations, and temporary deployment. The recovery process typically relies on an onboard camera to acquire the position and attitude information of the recovery inlet in the image, and then uses the camera imaging model to convert the spatial pose of the inlet relative to the drone into the control input for alignment and entry. Due to pitch, roll, and positional disturbances caused by vehicle bumps, and the fact that nighttime, backlighting, rain, fog, reflections, and personnel obstruction can affect the appearance of the inlet, the observable features of the inlet may change rapidly over time. Therefore, the pose measurement link needs to have continuous output and availability constraints.

[0003] In existing technologies, most approaches to obtaining the pose of the retrieval entrance involve setting artificial markers or corner points on the entrance structure, obtaining two-dimensional features using threshold segmentation, corner detection, or marker recognition, establishing a correspondence between these features and the three-dimensional structural points of the entrance, and solving the relative pose using PnP-type algorithms (such as EPnP); or, under unmarked conditions, locating the entrance using edge / line detection, template matching, feature point matching, etc., and then solving the pose using a geometric model; some solutions also use deep learning networks to detect the entrance or perform keypoint regression, and combine this with Kalman filtering, optical flow tracking, etc., to achieve inter-frame smoothing and local region tracking, and finally output the estimated pose to the retrieval control module for alignment and entry.

[0004] The above-mentioned methods are prone to link instability under mobile carrier and complex imaging conditions: edge, corner or template-based methods are sensitive to changes in illumination, reflection and occlusion, and key point jumps are easy to occur when features are missing; learning-based detection / regression lacks explicit expression of key point reliability when there is local occlusion or rapid change of viewpoint, and two-dimensional and three-dimensional constraints are not robust enough to anomalies; if inter-frame tracking or ROI update lacks geometric consistency verification, it is easy to drift and affect subsequent cropping and pose calculation, thus making it difficult to continuously provide reliable pose input for alignment-entry closed loop.

[0005] Therefore, a relative pose measurement method for the vehicle-mounted UAV recovery inlet that can overcome the shortcomings of the existing technology is a problem that needs to be solved by those skilled in the art. Summary of the Invention

[0006] One objective of this invention is to propose a relative pose measurement method for vehicle-mounted UAV recovery inlet based on VisionMamba and EPnP. The core technical problem to be solved by this application is: how to stably obtain the relative pose of the UAV and the vehicle-mounted recovery inlet and have availability determination in recovery scenarios where the vehicle is moving at low speed and bumpy, and there are changes in lighting and occlusion, so as to support the closed-loop control process of recovery inlet alignment and entry.

[0007] The relative pose measurement method for the recovery inlet of a vehicle-mounted UAV based on VisionMamba and EPnP according to embodiments of the present invention includes:

[0008] S1. Obtain the current frame image, camera intrinsic parameters, and 3D reference point set of the retrieval entrance from the airborne camera, and determine the entrance search window parameters and temporal memory vector;

[0009] S2. Based on the entrance search window parameters, the entrance search image block is cropped from the current frame image. The entrance search image block and the temporal memory vector are input into the VisionMamba feature point generation network. The VisionMamba feature point generation network recursively updates the temporal memory vector through the Mamba temporal coding module and fuses it with the features of the entrance search image block. Then, the feature point regression module obtains the entrance two-dimensional feature point set and the occlusion visibility weight set, and at the same time obtains the updated temporal memory vector.

[0010] S3. Establish a two-dimensional and three-dimensional correspondence relationship based on the two-dimensional feature point set of the entrance and the three-dimensional reference point set of the recycling entrance, and generate a weighted correspondence relationship set based on the occlusion visibility weight set;

[0011] S4. Perform EPnP pose calculation on the weighted correspondence set to obtain the initial relative pose. Calculate the reprojection error based on the initial relative pose, camera intrinsic parameters, and the 3D reference point set of the retrieval entrance. Calculate the entrance center offset and entrance plane orientation deviation angle based on the initial relative pose to obtain the entry channel deviation. When the reprojection error is less than the first threshold and the entry channel deviation is less than the second threshold, determine the valid relative pose and calculate the alignment control error. If the conditions are not met, generate a valid pose flag as invalid and set the alignment control error to zero.

[0012] S5. Generate the recovery closed-loop control quantity based on the effective relative pose, the effective pose flag, and the alignment control error.

[0013] S6. Based on the effective relative pose, camera intrinsic parameters, and the retrieval entry 3D reference point set, the parameters of the next frame entry search window are projected into the current frame image, and the updated temporal memory vector is used as the temporal memory vector of the next frame for the VisionMamba feature point generation network to use.

[0014] Optionally, S1 is as follows:

[0015] During the current control cycle, the airborne camera is triggered to acquire the current frame image, which is a three-channel image. The camera intrinsic parameters corresponding to the airborne camera are read from the camera calibration results. The camera intrinsic parameters include focal length parameters and principal point parameters.

[0016] Read the three-dimensional reference point set of the recycling inlet, which is pre-calibrated in the coordinate system of the recycling inlet structure. The three-dimensional reference point set of the recycling inlet contains eight three-dimensional reference points arranged in point order, and the point order of the three-dimensional reference point set of the recycling inlet is consistent with the point order of the two-dimensional feature point set of the inlet.

[0017] The entry search window parameters obtained from the projection of the previous frame are used as the entry search window parameters of the current frame. The entry search window parameters include the center pixel coordinates of the window and the side length of the window. The side length of the window corresponds to the cropping size of the entry search image block of 256 pixels by 256 pixels. The entry search window parameters are then cropped to ensure that the entry search window parameters are within the pixel range of the current frame image. If the entry search window parameters of the previous frame are not obtained, the entry search window parameters are initialized with preset center pixel coordinates and preset side length.

[0018] The temporal memory vector updated in the previous frame is used as the temporal memory vector of the current frame and input into the VisionMamba feature point generation network. The temporal memory vector is a vector of length 256 and is used to pass the retrieval entry structure constraint between frames. When the temporal memory vector updated in the previous frame is not obtained, a zero vector of length 256 is used to initialize the temporal memory vector.

[0019] Optionally, S2 is as follows:

[0020] The pixel cropping region in the current frame image is determined based on the parameters of the ingress search window. The pixel cropping region is cropped and resampled to obtain a three-channel ingress search image block of 256 pixels by 256 pixels.

[0021] The entry search image block is input into the image feature encoding layer. A convolutional layer with a stride of four is used to output 64-channel features, and then a convolutional layer with a stride of two is used to output 128-channel features. The resulting two-dimensional feature map is flattened in grid order into a feature sequence of length 1024 and each sequence element is 128-dimensional.

[0022] The temporal memory vector is input into the linear transformation neuron and projected into a 128-dimensional memory feature. The memory feature is used as the first element of the feature sequence and concatenated with the feature sequence to form a feature sequence with memory. No detection box or segmentation mask is generated in the VisionMamba feature point generation network.

[0023] The feature sequence with memory is input into the Mamba temporal coding module. The feature sequence with memory is recursively encoded by a series of multi-layer Mamba blocks. Each Mamba block performs normalization, linear mapping to generate 256-dimensional intermediate features, gating neurons to generate gating coefficients, recursively updating the recursive features along the sequence order by state-space recursive neurons, and fusion of the recursive features and gating coefficients dimension by dimension and linear mapping back to 128 dimensions. The encoded feature sequence is obtained by adding it to the input through residual connection and updating the memory features of the first element simultaneously.

[0024] At the end of the Mamba temporal coding module, the updated memory features are restored to an updated temporal memory vector of length 256 through linear transformation neurons, and the encoded feature sequence and the updated temporal memory vector are output to the feature point regression module.

[0025] In the feature point regression module, pooling is performed on the encoded feature sequence other than the memory features to obtain 128-dimensional global features. The global features are concatenated with the updated temporal memory vector to obtain 384-dimensional fused features. The fused features are then passed through fully connected layers from 384 to 256 and from 256 to 24 to obtain 24-dimensional output. Based on the 24-dimensional output, an entry 2D feature point set and an occlusion visibility weight set are generated. The entry 2D feature point set contains the pixel coordinates of eight entry feature points and the point order is consistent with the point order of the recovery entry 3D reference point set. The occlusion visibility weight set contains weights that correspond one-to-one with the eight entry feature points, and the updated temporal memory vector is used as the temporal memory vector for the next frame.

[0026] Optionally, the pixel coordinates of the input two-dimensional feature point set output by the feature point regression module are restored and mapped to pixel coordinates in the current frame image using a feedback restoration function based on the input search window parameters and the input search image patch. Specifically, the feedback restoration function is:

[0027] ;

[0028] in, Indicates the current frame image The first coordinate system Pixel coordinates of each entry feature point Indicates the current frame number. This represents the index of the entry feature point, with values ​​ranging from one to eight. and Indicates the first Each entry feature point searches for an image patch at the entry point. Local pixel x-coordinate and local pixel y-coordinate in a coordinate system and Indicates the parameters of the entry search window. The x and y coordinates of the center pixel of the window. Indicates the parameters of the entry search window. The window side length, a constant Indicates the entry point for searching image blocks. The width and height pixel values ​​are used to establish a linear mapping relationship between local pixel coordinates and the window side length with consistent scale.

[0029] Optionally, S3 specifically refers to:

[0030] Read the two-dimensional feature point set of the entrance according to the point order of the three-dimensional reference point set of the entrance, and pair the eight entrance feature points in the two-dimensional feature point set with the eight three-dimensional reference points in the three-dimensional reference point set of the entrance in turn to form eight sets of two-dimensional and three-dimensional correspondences.

[0031] Read the occlusion-visible weight set according to the same point order, and associate the eight weights in the occlusion-visible weight set with eight sets of two-dimensional and three-dimensional correspondences to obtain the point-level weights of the correspondences.

[0032] The point-level weights are subjected to range limitation and normalization to make the point-level weights non-negative and the sum of the point-level weights equal to one. The corresponding relationships are then weighted and labeled using point-level weights.

[0033] The eight sets of two-dimensional and three-dimensional correspondences, including the two-dimensional feature points of the entrance, the three-dimensional reference points of the retrieval entrance, and the point-level weights, are combined into a weighted correspondence set, and the weighted correspondence set is output to the EPNP pose calculation.

[0034] Optionally, S4 specifically refers to:

[0035] Input the weighted correspondence set into the EPnP pose solution, and weight the correspondence constraints in the solution process according to the point-level weights in the weighted correspondence set to output the initial relative pose;

[0036] Based on the initial relative pose, camera intrinsic parameters, and the 3D reference point set of the retrieval entrance, a projection point set of the 3D reference point set of the retrieval entrance in the current frame image is generated. The pixel deviation between the projection point set and the 2D feature point set of the entrance under the same point order is calculated. The pixel deviation is weighted and aggregated according to the point-level weight to obtain the reprojection error.

[0037] The position of the entrance center in the camera coordinate system is determined by the initial relative pose of the 3D reference point set of the retrieval entrance. The entrance center is projected onto the current frame image to obtain the pixel coordinates of the entrance center. The difference between the pixel coordinates of the entrance center and the pixel coordinates of the window center of the entrance search window parameters is obtained to obtain the entrance center offset.

[0038] The plane orientation of the entrance plane in the camera coordinate system is determined based on the initial relative pose and defined by the three-dimensional reference point set of the retrieval entrance. The angle between the entrance plane orientation and the camera optical axis is calculated to obtain the entrance plane orientation deviation angle. The entrance center offset and the entrance plane orientation deviation angle are fused using a preset scaling factor to obtain the entry channel deviation.

[0039] The reprojection error is compared with the first threshold and the entry channel deviation is compared with the second threshold. When the comparison conditions are met, the initial relative pose is determined as the valid relative pose and the pose validity flag is set to valid. When the comparison conditions are not met, the pose validity flag is set to invalid and the valid relative pose is set to the valid relative pose of the previous frame.

[0040] When the pose valid flag is valid, the alignment control error is calculated and output based on the inlet center offset and the inlet plane orientation deviation angle. When the pose valid flag is invalid, the alignment control error is set to zero and output.

[0041] Optionally, based on the initial relative pose and the 3D reference point set of the retrieval entrance within the camera, a projection point set of the retrieval entrance 3D reference point set in the current frame image is generated. The pixel deviation between the projection point set and the entrance 2D feature point set under the same point order is calculated. Then, the pixel deviation is weighted and converged using a projection error feedback function based on point-level weights to obtain the reprojection error. The projection error feedback function is specifically:

[0042] ;

[0043] in, This represents the reprojection error, expressed in pixels. Represents the weighted correspondence set Midpoint index is Point-level weights, and Represents the projection point The x and y coordinates of pixels in the current frame image. and Represents the two-dimensional feature points of the entrance. The x and y coordinates of pixels in the current frame image. This represents the point index, with values ​​from one to eight.

[0044] Optional, S5 specifically includes:

[0045] Read the pose validity flag. If the pose validity flag is invalid, reset the closed-loop control quantity to zero and output it.

[0046] When the pose valid flag is valid, the relative position component and relative attitude component of the UAV relative to the recovery inlet are determined according to the valid relative pose. The relative position component and relative attitude component are combined with the alignment control error to obtain the closed-loop error.

[0047] The closed-loop error is mapped using a preset control gain parameter to generate a recovery closed-loop control quantity, which is then constrained by a preset control limit parameter before being output.

[0048] Optionally, step S6 specifically includes:

[0049] Based on the effective relative pose, camera intrinsic parameters, and the 3D reference point set of the retrieval entrance, image projection is performed on the eight 3D reference points of the 3D reference point set of the retrieval entrance to obtain the projection point set in the current frame image;

[0050] Calculate the rectangular region covering the projection point set, determine the window center pixel coordinates of the next frame entry search window parameters based on the rectangular region, and determine the window side length by superimposing a preset expansion amount on the larger of the width and height of the rectangular region.

[0051] Perform boundary cropping on the next frame entry search window parameters so that the cropping area corresponding to the window center pixel coordinates and the window side length is within the pixel range of the current frame image, and output the next frame entry search window parameters after boundary cropping for the next frame entry search image block cropping;

[0052] The updated temporal memory vector is directly assigned as the temporal memory vector for the next frame, and the temporal memory vector for the next frame is input into the VisionMamba feature point generation network as the temporal memory vector for the next frame.

[0053] The beneficial effects of this invention are:

[0054] (1) This proposal proposes an improved method for measuring the relative pose of the vehicle-mounted recycling entrance. By changing the visual observation of the entrance from the traditional results of the edge / line / template or detection box, it directly regresses eight fixed two-dimensional feature points of the entrance within the projection-updated entrance search window. Temporal recursion constraints are implemented by introducing a temporal memory vector that can be transferred across frames in the network. This design ensures that the inference of each frame revolves around the local area of ​​the recycling entrance, avoiding false detection interference introduced by the full-image search. At the same time, the Mamba temporal coding module is used to recursively update the memory vector and fuse it with local image features. Even when the vehicle bumps cause rapid changes in the viewing angle, or when the entrance is partially occluded and the local appearance is unstable, the temporal continuity of the key point output can still be maintained, reducing the key point jump caused by the degradation of features in a single frame. This is more in line with the need for continuous available observation in the "recycling while moving" scenario.

[0055] (2) This proposal puts forward a new geometric solution link of “occlusion visibility weight - weighted EPnP - validity gating”: the VisionMamba network outputs the occlusion visibility weight corresponding to each point while outputting the two-dimensional feature point set of the input and output. When constructing the two-dimensional and three-dimensional correspondence, the weight is bound according to the point order, and after performing non-negative constraints and normalization, a weighted correspondence set is formed. The point-level weight is directly introduced into the EPnP pose solution process, so that the feature points with higher visibility and more reliable positioning contribute more to the pose constraint. After the pose calculation, the solution does not directly use the calculation results for control. Instead, it reprojects the 3D reference point based on the camera intrinsic parameters and calculates the weighted projection error. At the same time, it calculates the entrance center offset and the entrance plane orientation deviation angle and merges them to form the entrance channel deviation. A valid pose flag is given through dual threshold judgment. When the judgment is invalid, it reverts to the valid pose of the previous frame and sets the alignment control error to zero. This reduces the risk of pose abrupt changes caused by abnormal key points or local occlusions affecting the closed-loop control. It establishes a clear availability constraint relationship between pose output and control use. This is different from the processing method that only relies on filtering smoothing or only uses PnP residual as a single criterion.

[0056] (3) This proposal puts forward a holistic approach for retrieval alignment-entry closed loop, which closes the "local clipping observation, temporal memory recursion, weighted EPnP pose calculation, pose validity determination, closed-loop control quantity generation, projection update search window and memory transfer" within the same control cycle link. Specifically, the scheme uses the effective relative pose to project the 3D reference point of the retrieval entry point onto the current frame image, calculates the rectangular area covering the projection point and superimposes the extension quantity to generate the entry search window parameters for the next frame, realizes ROI self-updating based on geometric consistency, and reduces the cumulative impact of tracking drift on subsequent clipping and inference; at the same time, the updated temporal memory vector is directly transferred to the next frame as network input to form explicit inter-frame state variables. This holistic link aligns the pose measurement output and the retrieval control quantity generation in terms of data and decision logic, and can provide relative pose and its validity information directly related to the retrieval action under mobile platform and complex vision conditions. Attached Figure Description

[0057] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0058] Figure 1 This is a flowchart of a method for measuring the relative pose of a vehicle-mounted UAV recovery inlet based on VisionMamba and EPnP, as proposed in this invention.

[0059] Figure 2This invention presents a flowchart of a method for measuring the relative pose of a vehicle-mounted UAV recovery entrance based on VisionMamba and EPnP, which involves acquiring the current frame image, camera intrinsic parameters, a three-dimensional reference point set of the recovery entrance, and determining the entrance search window parameters and temporal memory vector.

[0060] Figure 3 The flowchart of the relative pose measurement method for the entrance of a vehicle-mounted UAV recovery based on VisionMamba and EPnP proposed in this invention is as follows: cropping the entrance search image block and inputting it into the VisionMamba feature point generation network to obtain the entrance two-dimensional feature point set and occlusion visibility weight set.

[0061] Figure 4 The flowchart for establishing two-dimensional and three-dimensional correspondences and generating a weighted correspondence set is provided for a relative pose measurement method for the recovery inlet of a vehicle-mounted UAV based on VisionMamba and EPnP proposed in this invention.

[0062] Figure 5 This invention presents a flowchart illustrating the execution of EPnP pose calculation and pose validity determination in a relative pose measurement method for a vehicle-mounted UAV recovery inlet based on VisionMamba and EPnP.

[0063] Figure 6 The flowchart of the method for measuring the relative pose of a vehicle-mounted UAV recovery inlet based on VisionMamba and EPnP proposed in this invention is shown below, which generates the recovery closed-loop control quantity based on the effective relative pose.

[0064] Figure 7 The flowchart of the relative pose measurement method for the recovery entrance of a vehicle-mounted UAV based on VisionMamba and EPnP proposed in this invention is as follows: updating the entrance search window parameters based on the effective relative pose projection and passing the update timing memory vector.

[0065] Figure 8 This is a schematic diagram illustrating the vehicle-mounted recovery scenario and the self-updating of the entry search window parameters in a vehicle-mounted UAV recovery entry relative pose measurement method based on VisionMamba and EPnP proposed in this invention.

[0066] Figure 9 This is a schematic diagram of the cross-frame temporal memory vector recursive stabilization key points of a relative pose measurement method for the recovery inlet of a vehicle-mounted UAV based on VisionMamba and EPnP proposed in this invention.

[0067] Figure 10 This is a schematic diagram of the VisionMamba feature point generation network structure for a relative pose measurement method for the recovery inlet of a vehicle-mounted UAV based on VisionMamba and EPnP proposed in this invention. Detailed Implementation

[0068] In Example 1, reference Figures 1 to 10 A method for measuring the relative pose of a vehicle-mounted UAV recovery inlet based on VisionMamba and EPnP, comprising:

[0069] S1. Obtain the current frame image, camera intrinsic parameters, and 3D reference point set of the retrieval entrance from the airborne camera, and determine the entrance search window parameters and temporal memory vector;

[0070] S2. Based on the entrance search window parameters, the entrance search image block is cropped from the current frame image. The entrance search image block and the temporal memory vector are input into the VisionMamba feature point generation network. The VisionMamba feature point generation network recursively updates the temporal memory vector through the Mamba temporal coding module and fuses it with the features of the entrance search image block. Then, the feature point regression module obtains the entrance two-dimensional feature point set and the occlusion visibility weight set, and at the same time obtains the updated temporal memory vector.

[0071] S3. Establish a two-dimensional and three-dimensional correspondence relationship based on the two-dimensional feature point set of the entrance and the three-dimensional reference point set of the recycling entrance, and generate a weighted correspondence relationship set based on the occlusion visibility weight set;

[0072] S4. Perform EPnP pose calculation on the weighted correspondence set to obtain the initial relative pose. Calculate the reprojection error based on the initial relative pose, camera intrinsic parameters, and the 3D reference point set of the retrieval entrance. Calculate the entrance center offset and entrance plane orientation deviation angle based on the initial relative pose to obtain the entry channel deviation. When the reprojection error is less than the first threshold and the entry channel deviation is less than the second threshold, determine the valid relative pose and calculate the alignment control error. If the conditions are not met, generate a valid pose flag as invalid and set the alignment control error to zero.

[0073] S5. Generate the recovery closed-loop control quantity based on the effective relative pose, the effective pose flag, and the alignment control error.

[0074] S6. Based on the effective relative pose, camera intrinsic parameters, and the retrieval entry 3D reference point set, the parameters of the next frame entry search window are projected into the current frame image, and the updated temporal memory vector is used as the temporal memory vector of the next frame for the VisionMamba feature point generation network to use.

[0075] In this embodiment, step S1 specifically includes:

[0076] In one specific embodiment, this method operates within the closed-loop control cycle of the onboard moving and recovery task, and each control cycle is numbered as a time period. At any moment The controller triggers the airborne camera to complete one exposure and readout, obtaining the current frame image. Current frame image The three-channel image serves as the raw observation input for subsequent entry search image patch cropping and inference by the VisionMamba feature point generation network;

[0077] Obtaining the current frame image Then, the camera intrinsic parameters corresponding to the airborne camera are read from the camera calibration results. Camera internal parameters It must include at least the focal length parameter and the principal point parameter, used to project the 3D reference point set of the retrieval inlet onto the current frame image. The pixel coordinate system is used as an imaging model constraint for subsequent EPnP pose calculation, and camera intrinsics are also included. It can be directly reused when it remains unchanged between consecutive frames, avoiding the introduction of detection boxes or segmentation masks as intermediate quantities in the inference link;

[0078] Then, the set of three-dimensional reference points of the recovery inlet, pre-calibrated in the structural coordinate system of the recovery inlet, is read. 3D reference point set for recycling entry It contains eight 3D reference points arranged in point order. Each 3D reference point consists of three coordinate components. The point order is fixed during the system configuration phase and is used to constrain the point-to-point correspondence between the ingress 2D feature point set and the regress ingress 3D reference point set. This allows the ingress 2D feature point set output by the VisionMamba feature point generation network to be directly matched with the regress ingress 3D reference point set without relying on additional matching. Establish two-dimensional and three-dimensional correspondences, and further construct a weighted correspondence set to input into the EPNP pose calculation;

[0079] In the entry search window parameter determination stage, the system records the entry search window parameters obtained from the previous frame based on the effective relative pose projection as follows: and will As a parameter of the current frame entry search window Initial values, entry search window parameters This includes the center pixel coordinates of the window and the window side length. The window side length corresponds to the 256-pixel by 256-pixel cropping size of the entry search image patch, enabling subsequent processing based on... The cropped entry search image patch is scaled to match the input of the VisionMamba feature point generation network. If the system is in the first frame or has not obtained the entry search window parameters from the previous frame, the entry search window parameters are initialized using preset center pixel coordinates and preset window side lengths. This allows local observations to enter a tailorable and reasonable path from the initialization stage;

[0080] To ensure the availability of the cropped area for the image patch in the entry search, the system adjusts the parameters of the entry search window. Perform boundary cropping so that the cropping region, determined by the window center pixel coordinates and the window side length, is located within the current frame image. Within a pixel range, the boundary cropping process first determines the center position of the cropping region based on the center pixel coordinates of the window, then expands the region horizontally and vertically by half the side length of the window to obtain the four boundary positions of the cropping region, and then extends the region beyond the current frame image. The order of truncation at the pixel boundaries and then synchronous correction of the window center pixel coordinates determines the parameters of the resulting entry search window. It can be directly used to crop 256-pixel by 256-pixel entry search image blocks, avoiding the interruption of the inference link due to the cropping area going out of bounds caused by changes in perspective caused by vehicle bumps.

[0081] In the temporal memory vector determination stage, the system records the temporal memory vector updated in the previous frame as... and will As the temporal memory vector of the current frame Input into the VisionMamba feature point generation network, temporal memory vector This is a vector of length 256, specifically composed of 256 consecutive values. Each value serves as the internal memory channel state of the VisionMamba feature point generation network, carrying cross-frame retrieval entry structure constraint information. The previous frame updates the temporal memory vector. After the Mamba temporal coding module of the VisionMamba feature point generation network of the previous frame recursively updates the memory-laden feature sequence, the updated memory features are recovered by linear transformation neurons at the end of the module, thus forming a numerical vector input that can be directly passed to the next frame. If the system is in the first frame or has not obtained the updated temporal memory vector from the previous frame, the temporal memory vector is initialized with a zero vector of length 256. This enables the Mamba timing coding module to complete recursive startup when there is no historical memory input, and to obtain a stable updated timing memory vector by gradually updating it in subsequent frames.

[0082] In this embodiment, step S2 specifically includes:

[0083] In one specific embodiment, the current frame number is recorded as , at frame number The current frame image has been obtained. Entry search window parameters With temporal memory vector Among them, temporal memory vector This is a numerical vector of length 256. The values ​​in each dimension serve as the internal memory channel states of the VisionMamba feature point generation network, and the parameters of the entry search window are... This includes the window center pixel coordinates and the window side length. The window center pixel coordinates are used to determine the center of the cropping region, and the window side length is used to determine the coverage area of ​​the cropping region. First in the current frame image The pixel cropping region is determined in the middle. The four boundaries of the pixel cropping region are obtained by extending the window's side length by half in both the horizontal and vertical directions from the center pixel coordinates. Cropping is performed on the pixel cropping region to obtain a local image, and bilinear interpolation is used to resample the local image to obtain a three-channel entry search image block of 256 pixels by 256 pixels. The entry search image block is denoted as . ,Will As the image input to the VisionMamba feature point generation network, the network input is transformed from a full-image observation to a local observation around the retrieval entrance, and forms a consistent data link with the entrance search window parameters obtained by effective relative pose projection and transmitted between frames;

[0084] Search image blocks at the entry point The input image feature encoding layer first extracts 64-channel features through a convolutional layer with a stride of 4, and then extracts 128-channel features through a convolutional layer with a stride of 2, resulting in a two-dimensional feature map. Since the entry search image block is 256 pixels by 256 pixels, after two downsampling steps of 4 and 2, a 32 by 32 grid feature is obtained, which is then used to create the two-dimensional feature map. Flattened into feature sequences according to grid order Feature sequence The sequence length is 1024, and each sequence element is a 128-dimensional feature, which is used as the sequence input for the subsequent Mamba time-series coding module.

[0085] Temporal memory vector The input is a linearly transformed neuron, projected as a 128-dimensional memory feature, and the memory feature is denoted as... , memory features As the first element of the feature sequence and the feature sequence The sequences are spliced ​​together to form a feature sequence with memory, and this feature sequence with memory is denoted as... In the VisionMamba feature point generation network, no detection boxes or segmentation masks are generated; instead, feature sequences with memory are used. It is directly used as the input for subsequent recursive encoding;

[0086] Feature sequences with memory The input is a Mamba temporal coding module, which consists of multiple Mamba blocks connected in series. For each Mamba block, the input sequence is first normalized, and then the 128-dimensional features are mapped to 256-dimensional intermediate features through a linear transformation neuron. The 256-dimensional intermediate features are input into a gating neuron to obtain gating coefficients. The 256-dimensional intermediate features are then input into a state space recursive neuron in sequence to perform recursive updates to obtain recursive features. The recursive features and gating coefficients are fused dimension by dimension, and then projected back to the 128-dimensional features through a linear transformation neuron. The output sequence of the layer is obtained by adding the residual connection to the input of the layer, thus forming the encoded feature sequence. During the recursive process, the memory features corresponding to the first element of the sequence are updated synchronously, so that the memory features carry the structural memory of the retrieval entry and the stability constraints under short-term occlusion when transmitted across frames.

[0087] At the end of the Mamba temporal coding module, the updated memory features are taken and restored to an updated temporal memory vector of length 256 through a linear transformation neuron. This updated temporal memory vector is denoted as... Simultaneously output the encoded feature sequence and update the temporal memory vector. The feature point regression module updates the temporal memory vector. It can be used as a temporal memory vector input in the next frame to form an explicit inter-frame recursive variable transfer path;

[0088] In the feature point regression module, the average of the encoded feature sequences excluding memory features along the sequence length dimension is used to obtain a 128-dimensional global feature, which is denoted as... global features With updating the sequential memory vector The resulting 384-dimensional fused feature is obtained by concatenating the features. This fused feature is then sequentially input into fully connected layers 384 to 256 and 256 to 24, yielding a 24-dimensional output vector. This 24-dimensional output vector is denoted as... ,right The first sixteen dimensions are parsed in point order into the local pixel coordinates of the two-dimensional feature point set at the entrance in the coordinate system of the entrance search image block, resulting in the local pixel coordinates of the eight entrance feature points. ,right The latter eight dimensions are analyzed as an occlusion visibility weight set, and the Sigmoid activation function is used to limit the values ​​of the occlusion visibility weight set to zero and one. To convert local pixel coordinates into an entry two-dimensional feature point set in the current frame image coordinate system, the entry search window parameters are used. For each entry feature point, perform the following coordinate reconstruction, calculated based on the feedback reconstruction function:

[0089] ;

[0090] in, Indicates the current frame image The first coordinate system Pixel coordinates of each entry feature point Indicates the current frame number. This represents the index of the entry feature point, with values ​​ranging from one to eight. and Indicates the first Each entry feature point searches for an image patch at the entry point. Local pixel x-coordinate and local pixel y-coordinate in a coordinate system and Indicates the parameters of the entry search window. The x and y coordinates of the center pixel of the window. Indicates the parameters of the entry search window. The window side length, a constant Indicates the entry point for searching image blocks. The width and height pixel values ​​are used to establish a linear mapping relationship between local pixel coordinates and the window side length with consistent scale.

[0091] The entry two-dimensional feature point set consists of each The points are arranged in a point order consistent with the point order of the 3D reference point set at the retrieval entrance. This allows the 2D feature point set at the entrance, the occlusion visibility weight set, and the 3D reference point set at the retrieval entrance to directly construct a weighted correspondence set, which is then input into the EPnP pose calculation to update the temporal memory vector. As a direct input to the next frame's temporal memory vector, it enables continuous observation constraints that coordinate with the inter-frame transfer of parameters in the entry search window.

[0092] In this embodiment, step S3 specifically includes:

[0093] In one specific embodiment, regarding frame sequence number Having obtained the 2D feature point set of the entrance, the 3D reference point set of the entrance, and the occlusion visibility weight set, let the 2D feature point set of the entrance be denoted as... ,in It consists of eight entry feature points arranged in order, where each entry feature point is a pixel coordinate in the current frame image coordinate system. Each entry feature point is denoted as... and will The pixel x-coordinate and pixel y-coordinate are marked as and The three-dimensional reference point set of the recycling entrance is denoted as ,in It consists of eight three-dimensional reference points arranged in order. Each three-dimensional reference point represents the three-dimensional coordinates of the recovery inlet structure coordinate system. Each three-dimensional reference point is denoted as and will The three-dimensional coordinate components are denoted as , , Let the set of occlusion visibility weights be denoted as ,in Composed of eight weights ordered by points, the first... Each weight is denoted as , Generate numerical weights for the network output of VisionMamba feature points;

[0094] According to the three-dimensional reference point set of the recycling entrance Point sequence reading entry point for two-dimensional feature point set For point order index Pairing is performed sequentially from one to eight, and the entry two-dimensional feature points are... With three-dimensional reference point Eight sets of two-dimensional and three-dimensional correspondences are formed, and these eight sets of correspondences are arranged in point order to form a correspondence list. This correspondence list does not perform nearest neighbor matching or descriptor matching, but directly depends on the entry two-dimensional feature point set. With the three-dimensional reference point set of the recycling inlet The consistency of the point order completes the establishment of the correspondence;

[0095] Then, the occlusion-visibility weight set is read in the same point order. For each set of two-dimensional and three-dimensional correspondences, the point sequence index is... weight Associated with the point order index The correspondence is established, thereby generating point-level weights for each set of correspondences. The point-level weights are bound to the corresponding entry two-dimensional feature points, so that the weights can enter the subsequent pose calculation link along with the correspondence.

[0096] The point-level weights are subjected to range limiting and normalization. The range limiting follows the order of lower bound constraint first, then upper bound constraint, and then each... The values ​​are restricted to non-negative values ​​and not greater than one to obtain range-limited weights. After normalization, the eight range-limited weights are summed to obtain a total weight, which is denoted as... ,when When the value is greater than zero, divide the weight of each range limit by 0. Obtain the normalized weights and denote them as follows: ,when When equal to zero, the eight normalized weights are... The weights are all set to one-eighth, thus ensuring that the point-level weights are non-negative and that the sum of the point-level weights is one, thus normalizing the weights. As a weighted annotation parameter for the corresponding relationship, it maintains the same point order binding relationship with the entrance 2D feature points and 3D reference points.

[0097] The eight sets of two-dimensional and three-dimensional correspondences are combined with their corresponding normalized weights to form a weighted correspondence set, which is denoted as […]. ,in, Each element in the array is composed of two-dimensional feature points from the entry point. , Recycling entry three-dimensional reference point Point-level weights Composed of, and indexed by point order Arranged from one to eight, the weighted correspondence set Output to EPnP pose calculation.

[0098] In this embodiment, step S4 specifically includes:

[0099] In one specific embodiment, regarding frame sequence number The weighted correspondence set has been obtained. Camera internal parameters , Recycling entry three-dimensional reference point set Entrance 2D feature point set With entry search window parameters Among them, the weighted correspondence set It consists of eight sets of two-dimensional and three-dimensional correspondences and their point-level weights, with the point-level weights denoted as... Point order index Values ​​range from one to eight, representing the two-dimensional feature point set at the entrance. It contains the pixel coordinates of eight entry feature points arranged in point order, denoted as... 3D reference point set for recycling entry It contains eight three-dimensional reference points arranged in point order, denoted as... Entry search window parameters Includes the center pixel coordinates of the window and the side length of the window, where the center pixel coordinates are denoted as... ;

[0100] Weighted correspondence set Input EPnP pose calculation, and calculate based on point-level weights during the solution process. The constraint terms of each pair of correspondences are weighted so that correspondences with larger point-level weights contribute more to the pose solution. The EPnP pose solution outputs the initial relative pose. Initial relative pose From rotational components With translation components This structure is used to represent the pose transformation from the inlet structure coordinate system to the camera coordinate system.

[0101] In the reprojection error calculation stage, based on the initial relative pose... Camera internal parameters With the three-dimensional reference point set of the recycling inlet Generate a set of projection points, specifically, for each 3D reference point. First use and Transform it to the camera coordinate system to obtain the camera coordinate points, then perform perspective projection to obtain the normalized imaging plane coordinates. Perspective projection uses the depth component of the camera coordinate points to be normalized by division, and then uses the camera intrinsic parameters. The normalized imaging plane coordinates are mapped to the pixel coordinates of the current frame image to obtain the projection points. and will The pixel x-coordinate and pixel y-coordinate are marked as and , entrance two-dimensional feature points The pixel x-coordinate and pixel y-coordinate are marked as and The pixel deviations are weighted and aggregated according to point-level weights, that is, the reprojection error is calculated according to the projection error feedback function. :

[0102] ;

[0103] in, This represents the reprojection error, expressed in pixels. Represents the weighted correspondence set Midpoint index is Point-level weights, and Represents the projection point The x and y coordinates of pixels in the current frame image. and Represents the two-dimensional feature points of the entrance. The x and y coordinates of pixels in the current frame image. This represents the dotted index, with values ​​from one to eight.

[0104] In the calculation of the entrance center offset, based on the three-dimensional reference point set of the recycling entrance... Determine the position of the entrance center in the coordinate system of the recycling entrance structure, and denote the entrance center as... Entrance Center Eight three-dimensional reference points were used The coordinates are obtained by averaging the components, and the entrance center is then... Through the initial relative pose Transform to the camera coordinate system and project onto the current frame image to obtain the coordinates of the entrance center pixel. Mark the entrance center pixel coordinates as follows: , the center pixel coordinates of the entrance With entry search window parameters Window center pixel coordinates The two-dimensional difference is obtained by subtracting the pixels in the horizontal and vertical directions respectively. The square root of the sum of the squares of these two-dimensional differences is then used to obtain the entrance center offset, which is denoted as . ;

[0105] In the calculation of the inlet plane orientation deviation angle, based on the three-dimensional reference point set of the retrieval inlet... Determine the plane normal vector of the inlet plane in the coordinate system of the recycling inlet structure, and denote this normal vector as... Entrance plane normal vector Two edge vectors are constructed by selecting three non-collinear 3D reference points with distinct indexes from eight 3D reference points, and then performing a cross product on the two edge vectors to obtain the entrance plane normal vector. Using the initial relative pose rotational component Transform to the camera coordinate system to obtain the camera normal vector of the entrance plane, and denote the unit vector of the optical axis direction in the camera coordinate system as... For the entrance plane camera normal vector and Normalize both, calculate their dot product and limit the dot product to between -1 and +1, then use inverse cosine to obtain the entrance plane orientation deviation angle, denoted as . ;

[0106] Use preset ratio coefficient Orient the entrance plane toward the deviation angle The angular deviation component is converted to pixel scale, and the angular deviation component is taken as... and The product of the angular deviation component and the inlet center offset. The input channel deviation is obtained by adding the two values, and the input channel deviation is denoted as . ,in, The input is an angle value, and the output is a pixel scale value, which is a preset scaling factor to adjust the input channel deviation. It can be directly compared numerically with the second threshold;

[0107] In the pose validity determination stage, the first threshold is read. With the second threshold Reprojection error With the first threshold Compare and enter channel deviation With the second threshold When comparing, Less than the first threshold and Less than the second threshold At that time, the initial relative pose will be Determined as valid relative pose And record the valid pose marker as And set it to valid; if the comparison condition is not met, set the pose valid flag. Set to invalid and change the valid relative pose. Set as the effective relative pose of the previous frame ,in, Indicates frame number The corresponding effective relative pose, Indicates frame number The corresponding effective relative pose;

[0108] In the calculation of alignment control error, the alignment control error is denoted as... When the pose is valid For validity, based on the center pixel coordinates of the entrance. Pixel coordinates of the window center The lateral and longitudinal differential structures construct in-plane alignment error components, and the entrance plane is oriented towards the deviation angle. As an attitude alignment error component, it together with the in-plane alignment error component constitutes the alignment control error. And output to the closed-loop control quantity generation stage, when the pose valid flag is displayed. If invalid, the alignment control error amount will be... Set to zero and output.

[0109] In this embodiment, step S5 specifically includes:

[0110] In one specific embodiment, regarding frame sequence number Valid pose markers have been obtained. Effective relative pose Alignment control error Among them, the effective pose marker Effective relative pose is used to characterize whether the effective relative pose can be used for closed-loop control. From rotational components With translation components Composition, alignment control error Including in-plane alignment error components and attitude alignment error components, the recovery closed-loop control quantity is denoted as... The number of control channels that recover closed-loop control quantities is denoted as ,but Depend on The values ​​of each control channel are used as the output of the flight control interface;

[0111] Read the valid pose flag When the pose is valid If invalid, the closed-loop control quantity will be recovered. Set it to the zero vector and output it; the zero vector is where all control channels are zero. A numerical vector is used to ensure that the control output remains numerically consistent with the closed-loop control interface when the pose is invalid.

[0112] When the pose is valid When valid, based on the valid relative pose Determine the relative position and attitude components of the UAV relative to the recovery inlet; specifically, for the rotation component... Performing a transpose yields the inverse rotation, which is denoted as . The inverse translation is calculated using matrix multiplication. ,make The three-dimensional relative position of the drone with respect to the recovery inlet will be... Read the relative position components according to the fixed coordinate axis order, denoted as... ,in , , These are numerical values ​​that can be directly used to control the mapping;

[0113] For relative attitude components, adopt Perform ZYX Euler angle decomposition, and The matrix elements are denoted as ,in It is a row index with values ​​from one to three. For column indexes with values ​​from one to three, first... Limit the calculation to the range of negative one to positive one, and then calculate the pitch relative attitude components. Then calculate the roll relative attitude components. Then calculate the heading relative attitude components. Thus, the three-axis relative attitude components are obtained. ,in For the roll relative attitude component, For pitch relative attitude components. For the heading-relative attitude component;

[0114] The relative position component, relative attitude component, and alignment control error are combined. The closed-loop error is obtained by combining the values; specifically, it is derived from the alignment control error. The alignment error components in the plane and the attitude alignment error components are read in a fixed order, denoted as follows: ,in and This represents the difference between the horizontal and vertical coordinates of the entry center pixel and the window center pixel. Let the attitude alignment error component corresponding to the entrance plane orientation deviation angle be denoted as . And constructed in the following fixed order sequence of components: This makes the closed-loop error amount Simultaneously carry relative pose information and alignment information within the same control cycle;

[0115] The closed-loop error is measured using preset control gain parameters. Mapping is performed to generate closed-loop control quantities for recycling. The preset control gain parameters include a position gain parameter set, an attitude gain parameter set, and an alignment gain parameter set, which are used to respectively... , and Mapped to the closed-loop control quantity for recycling The mapping process uses a weighted summation for each control channel value, i.e., for the closed-loop error. Each component is multiplied by the control gain corresponding to the control channel and summed to obtain the control output of the control channel. Then, the closed-loop control quantity is recovered. The preset control limiting parameter constraint is executed. The preset control limiting parameter includes the upper limit value and the lower limit value of each control channel. For each control channel, if the control channel value is greater than the upper limit value, the control channel value is set to the upper limit value. If the control channel value is less than the lower limit value, the control channel value is set to the lower limit value. The limited recovery closed-loop control quantity is obtained and output.

[0116] In this embodiment, step S6 specifically includes:

[0117] In one specific embodiment, regarding frame sequence number Valid relative pose has been obtained Camera internal parameters , Recycling entry three-dimensional reference point set Update the sequential memory vector With the current frame image The pixel range, where the width pixel value of the current frame image is denoted as . The height pixel value is denoted as Effective relative pose Used to characterize the pose transformation from the retrieval inlet structure coordinate system to the camera coordinate system, camera intrinsic parameters Used to map points in the camera coordinate system to image pixel coordinates, and to retrieve the entry 3D reference point set. Composed of eight three-dimensional reference points in point order, updating the temporal memory vector. For transmitting timing constraints across frames, the next frame entry search window parameter is denoted as... Includes the pixel coordinates of the window center. With window side length ;

[0118] Effective relative pose Camera internal parameters With the three-dimensional reference point set of the recycling inlet As input, The image is projected onto eight 3D reference points to obtain the set of projected points in the current frame image. This set of projected points is denoted as... , of which The pixel coordinates of each projection point are: Point order index The values ​​range from one to eight. The projection process is as follows: First use... The rotation and translation components transform the 3D reference point to the camera coordinate system, and then the camera intrinsic parameters are used to transform the 3D reference point to the camera coordinate system. Complete perspective projection and obtain pixel coordinates to form... ;

[0119] For the projection point set Calculate the covered rectangular area and read all. Find the minimum and maximum values ​​of the pixel's horizontal and vertical coordinates to obtain the left and right boundaries and the top and bottom boundaries. Denote the minimum and maximum horizontal values ​​as... , Let the minimum and maximum values ​​in the vertical direction be denoted as , Let the width of the rectangle be denoted as Let the height of the rectangle be denoted as Both are pixel values. The center pixel coordinates of the next frame window are set to the center position of the rectangle, i.e. Set the average position of the left and right boundaries and the top and bottom boundaries, and set the side length of the next frame window to the larger of the rectangle's width and height, plus a preset expansion amount. The preset expansion amount is denoted as... And the unit is pixels, thus obtaining This forms the initial values ​​of the entry search window parameters for the next frame. ;

[0120] Parameters for searching the entry window of the next frame Perform boundary clipping to ensure that the clipping square, defined by the window's center pixel coordinates and the window's side length, lies entirely within the pixel range of the current frame's image. Specifically, first apply a feasibility constraint to the window's side length: when Greater than or At that time, Set as Then apply an interval constraint to the window center: limit the x-coordinate of the window center to be no less than and not greater than The value limits the vertical coordinate of the window center to no less than and not greater than The value is used to obtain the next frame's entry search window parameters after boundary clipping, after completing the constraints. This is still denoted as... , will The output is used for cropping the ingress search image patch in the next frame, so that the scale and position of the ingress search image patch are directly correlated with the geometric observation of the current frame;

[0121] The updated temporal memory vector is directly assigned to the temporal memory vector of the next frame, denoted as . In frame number At the processing entry point, the next frame's temporal memory vector is... with by The entry search image patch obtained from the cropped next frame is simultaneously input into the VisionMamba feature point generation network and the temporal memory vector. In the Mamba temporal coding module, it participates in recursive updates and is fused with the local features of the entry search image patch to form a fused feature carrying cross-frame structural memory and current frame appearance evidence. Thus, the inter-frame transfer of entry search window parameters and the explicit transfer of temporal memory vectors form a continuous link in the network structure, ensuring that the entry two-dimensional feature point set and occlusion visible weight set of the next frame are generated under stable observation conditions, and providing continuous input for the construction of weighted correspondence set and EPnP pose calculation.

[0122] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for measuring the relative pose of a vehicle-mounted UAV recovery inlet based on VisionMamba and EPnP, characterized in that, include: S1. Obtain the current frame image, camera intrinsic parameters, and 3D reference point set of the retrieval entrance from the airborne camera, and determine the entrance search window parameters and temporal memory vector; S2. Based on the entrance search window parameters, the entrance search image block is cropped from the current frame image. The entrance search image block and the temporal memory vector are input into the VisionMamba feature point generation network. The VisionMamba feature point generation network recursively updates the temporal memory vector through the Mamba temporal coding module and fuses it with the features of the entrance search image block. Then, the feature point regression module obtains the entrance two-dimensional feature point set and the occlusion visibility weight set, and at the same time obtains the updated temporal memory vector. S3. Establish a two-dimensional and three-dimensional correspondence relationship based on the two-dimensional feature point set of the entrance and the three-dimensional reference point set of the recycling entrance, and generate a weighted correspondence relationship set based on the occlusion visibility weight set; S4. Perform EPnP pose calculation on the weighted correspondence set to obtain the initial relative pose. Calculate the reprojection error based on the initial relative pose, camera intrinsic parameters, and the 3D reference point set of the retrieval entrance. Calculate the entrance center offset and entrance plane orientation deviation angle based on the initial relative pose to obtain the entry channel deviation. When the reprojection error is less than the first threshold and the entry channel deviation is less than the second threshold, determine the valid relative pose and calculate the alignment control error. If the conditions are not met, generate a valid pose flag as invalid and set the alignment control error to zero. S5. Generate the recovery closed-loop control quantity based on the effective relative pose, the effective pose flag, and the alignment control error. S6. Based on the effective relative pose, camera intrinsic parameters, and the retrieval entry 3D reference point set, the parameters of the next frame entry search window are projected into the current frame image, and the updated temporal memory vector is used as the temporal memory vector of the next frame for the VisionMamba feature point generation network to use.

2. The method for relative pose measurement of the vehicle-mounted UAV recovery inlet based on VisionMamba and EPnP according to claim 1, characterized in that, S1 specifically refers to: During the current control cycle, the airborne camera is triggered to acquire the current frame image, which is a three-channel image. The camera intrinsic parameters corresponding to the airborne camera are read from the camera calibration results. The camera intrinsic parameters include focal length parameters and principal point parameters. Read the three-dimensional reference point set of the recycling inlet, which is pre-calibrated in the coordinate system of the recycling inlet structure. The three-dimensional reference point set of the recycling inlet contains eight three-dimensional reference points arranged in point order, and the point order of the three-dimensional reference point set of the recycling inlet is consistent with the point order of the two-dimensional feature point set of the inlet. The entry search window parameters obtained from the projection of the previous frame are used as the entry search window parameters of the current frame. The entry search window parameters include the center pixel coordinates of the window and the side length of the window. The side length of the window corresponds to the cropping size of the entry search image block of 256 pixels by 256 pixels. The entry search window parameters are then cropped to ensure that the entry search window parameters are within the pixel range of the current frame image. If the entry search window parameters of the previous frame are not obtained, the entry search window parameters are initialized with preset center pixel coordinates and preset side length. The temporal memory vector updated in the previous frame is used as the temporal memory vector of the current frame and input into the VisionMamba feature point generation network. The temporal memory vector is a vector of length 256 and is used to pass the retrieval entry structure constraint between frames. When the temporal memory vector updated in the previous frame is not obtained, a zero vector of length 256 is used to initialize the temporal memory vector.

3. The method for measuring the relative pose of a vehicle-mounted UAV recovery inlet based on VisionMamba and EPnP according to claim 1, characterized in that, S2 specifically refers to: The pixel cropping region in the current frame image is determined based on the parameters of the ingress search window. The pixel cropping region is cropped and resampled to obtain a three-channel ingress search image block of 256 pixels by 256 pixels. The entry search image block is input into the image feature encoding layer. A convolutional layer with a stride of four is used to output 64-channel features, and then a convolutional layer with a stride of two is used to output 128-channel features. The resulting two-dimensional feature map is flattened in grid order into a feature sequence of length 1024 and each sequence element is 128-dimensional. The temporal memory vector is input into the linear transformation neuron and projected into a 128-dimensional memory feature. The memory feature is used as the first element of the feature sequence and concatenated with the feature sequence to form a feature sequence with memory. No detection box or segmentation mask is generated in the VisionMamba feature point generation network. The feature sequence with memory is input into the Mamba temporal coding module. The feature sequence with memory is recursively encoded by a series of multi-layer Mamba blocks. Each Mamba block performs normalization, linear mapping to generate 256-dimensional intermediate features, gating neurons to generate gating coefficients, recursively updating the recursive features along the sequence order by state-space recursive neurons, and fusion of the recursive features and gating coefficients dimension by dimension and linear mapping back to 128 dimensions. The encoded feature sequence is obtained by adding it to the input through residual connection and updating the memory features of the first element simultaneously. At the end of the Mamba temporal coding module, the updated memory features are restored to an updated temporal memory vector of length 256 through linear transformation neurons, and the encoded feature sequence and the updated temporal memory vector are output to the feature point regression module. In the feature point regression module, pooling is performed on the encoded feature sequence other than the memory features to obtain 128-dimensional global features. The global features are concatenated with the updated temporal memory vector to obtain 384-dimensional fused features. The fused features are then passed through fully connected layers from 384 to 256 and from 256 to 24 to obtain 24-dimensional output. Based on the 24-dimensional output, an entry 2D feature point set and an occlusion visibility weight set are generated. The entry 2D feature point set contains the pixel coordinates of eight entry feature points and the point order is consistent with the point order of the recovery entry 3D reference point set. The occlusion visibility weight set contains weights that correspond one-to-one with the eight entry feature points, and the updated temporal memory vector is used as the temporal memory vector for the next frame.

4. The method for measuring the relative pose of a vehicle-mounted UAV recovery inlet based on VisionMamba and EPnP according to claim 3, characterized in that, The pixel coordinates of the input two-dimensional feature point set output by the feature point regression module are restored and mapped to pixel coordinates in the current frame image using a feedback restoration function based on the input search window parameters and the input search image patch. Specifically, the feedback restoration function is: ； in, Indicates the current frame image The first coordinate system Pixel coordinates of each entry feature point Indicates the current frame number. This represents the index of the entry feature point, with values ​​ranging from one to eight. and Indicates the first Each entry feature point searches for an image patch at the entry point. Local pixel x-coordinate and local pixel y-coordinate in a coordinate system and Indicates the parameters of the entry search window. The x and y coordinates of the center pixel of the window. Indicates the parameters of the entry search window. The window side length, a constant Indicates the entry point for searching image blocks. The width and height pixel values ​​are used to establish a linear mapping relationship between local pixel coordinates and the window side length with consistent scale.

5. The method for relative pose measurement of the vehicle-mounted UAV recovery inlet based on VisionMamba and EPnP according to claim 1, characterized in that, S3 specifically refers to: Read the two-dimensional feature point set of the entrance according to the point order of the three-dimensional reference point set of the entrance, and pair the eight entrance feature points in the two-dimensional feature point set with the eight three-dimensional reference points in the three-dimensional reference point set of the entrance in turn to form eight sets of two-dimensional and three-dimensional correspondences. Read the occlusion-visible weight set according to the same point order, and associate the eight weights in the occlusion-visible weight set with eight sets of two-dimensional and three-dimensional correspondences to obtain the point-level weights of the correspondences. The point-level weights are subjected to range limitation and normalization to make the point-level weights non-negative and the sum of the point-level weights equal to one. The corresponding relationships are then weighted and labeled using point-level weights. The eight sets of two-dimensional and three-dimensional correspondences, including the two-dimensional feature points of the entrance, the three-dimensional reference points of the retrieval entrance, and the point-level weights, are combined into a weighted correspondence set, and the weighted correspondence set is output to the EPNP pose calculation.

6. The method for measuring the relative pose of a vehicle-mounted UAV recovery inlet based on VisionMamba and EPnP according to claim 1, characterized in that, S4 specifically refers to: Input the weighted correspondence set into the EPnP pose solution, and weight the correspondence constraints in the solution process according to the point-level weights in the weighted correspondence set to output the initial relative pose; Based on the initial relative pose, camera intrinsic parameters, and the 3D reference point set of the retrieval entrance, a projection point set of the 3D reference point set of the retrieval entrance in the current frame image is generated. The pixel deviation between the projection point set and the 2D feature point set of the entrance under the same point order is calculated. The pixel deviation is weighted and aggregated according to the point-level weight to obtain the reprojection error. The position of the entrance center in the camera coordinate system is determined by the initial relative pose of the 3D reference point set of the retrieval entrance. The entrance center is projected onto the current frame image to obtain the pixel coordinates of the entrance center. The difference between the pixel coordinates of the entrance center and the pixel coordinates of the window center of the entrance search window parameters is obtained to obtain the entrance center offset. The plane orientation of the entrance plane in the camera coordinate system is determined based on the initial relative pose and defined by the three-dimensional reference point set of the retrieval entrance. The angle between the entrance plane orientation and the camera optical axis is calculated to obtain the entrance plane orientation deviation angle. The entrance center offset and the entrance plane orientation deviation angle are fused using a preset scaling factor to obtain the entry channel deviation. The reprojection error is compared with the first threshold and the entry channel deviation is compared with the second threshold. When the comparison conditions are met, the initial relative pose is determined as the valid relative pose and the pose validity flag is set to valid. When the comparison conditions are not met, the pose validity flag is set to invalid and the valid relative pose is set to the valid relative pose of the previous frame. When the pose valid flag is valid, the alignment control error is calculated and output based on the inlet center offset and the inlet plane orientation deviation angle. When the pose valid flag is invalid, the alignment control error is set to zero and output.

7. The method for measuring the relative pose of a vehicle-mounted UAV recovery inlet based on VisionMamba and EPnP according to claim 1, characterized in that, Based on the initial relative pose and the 3D reference point set of the retrieval entrance within the camera, a projection point set of the 3D reference point set of the retrieval entrance in the current frame image is generated. The pixel deviation between the projection point set and the 2D feature point set of the entrance under the same point order is calculated. Then, based on the point-level weights, the pixel deviation is weighted and converged using a projection error feedback function to obtain the reprojection error. The projection error feedback function is specifically as follows: ； in, This represents the reprojection error, expressed in pixels. Represents the weighted correspondence set Midpoint index is Point-level weights, and Represents the projection point The x and y coordinates of pixels in the current frame image. and Represents the two-dimensional feature points of the entrance. The x and y coordinates of pixels in the current frame image. This represents the point index, with values ​​from one to eight.

8. The method for measuring the relative pose of a vehicle-mounted UAV recovery inlet based on VisionMamba and EPnP according to claim 1, characterized in that, S5 specifically refers to: Read the pose validity flag. If the pose validity flag is invalid, reset the closed-loop control quantity to zero and output it. When the pose valid flag is valid, the relative position component and relative attitude component of the UAV relative to the recovery inlet are determined according to the valid relative pose. The relative position component and relative attitude component are combined with the alignment control error to obtain the closed-loop error. The closed-loop error is mapped using a preset control gain parameter to generate a recovery closed-loop control quantity, which is then constrained by a preset control limit parameter before being output.

9. The method for measuring the relative pose of a vehicle-mounted UAV recovery inlet based on VisionMamba and EPnP according to claim 1, characterized in that, Step S6 is as follows: Based on the effective relative pose, camera intrinsic parameters, and the 3D reference point set of the retrieval entrance, image projection is performed on the eight 3D reference points of the 3D reference point set of the retrieval entrance to obtain the projection point set in the current frame image; Calculate the rectangular region covering the projection point set, determine the window center pixel coordinates of the next frame entry search window parameters based on the rectangular region, and determine the window side length by superimposing a preset expansion amount on the larger of the width and height of the rectangular region. Perform boundary cropping on the next frame entry search window parameters so that the cropping area corresponding to the window center pixel coordinates and the window side length is within the pixel range of the current frame image, and output the next frame entry search window parameters after boundary cropping for the next frame entry search image block cropping; The updated temporal memory vector is directly assigned as the temporal memory vector for the next frame, and the temporal memory vector for the next frame is input into the VisionMamba feature point generation network as the temporal memory vector for the next frame.

Images