Dual-motion platform interaction method and system based on non-cooperative mode

Abstract

The application relates to a double-motion platform interaction method and system based on a non-cooperative mode, which comprises the following steps: selecting a target image in an obtained target image sequence as an initial frame image, solving the relative pose parameters of a target in the frame image by adopting a pose detection algorithm, sequentially solving the relative pose parameters of the target in multiple frame images after the initial frame image, taking the relative pose parameters of the target obtained based on the initial frame image as initial pose parameters, and obtaining the relative pose parameters of the target in subsequent frame images after the initial frame image in the target image sequence according to the initial pose parameters and the pose tracking algorithm, so as to realize tracking of the target, and simultaneously, a consistency discrimination strategy is adopted for verification when the pose tracking algorithm is adopted. The method can accurately track the pose of another motion platform in the non-cooperative mode.

Description

Technical Field

[0001] This application relates to the fields of image measurement and visual navigation technology, and in particular to a dual-motion platform interaction method and system based on a non-cooperative mode. Background Technology

[0002] Completing dual-motion platform interaction tasks requires accurate and reliable relative pose estimation methods. Currently, typical dual-motion platform interaction tasks, such as autonomous landing of rotary-wing UAVs on a moving platform and space rendezvous and docking, all require one or both platforms to accurately grasp their own position and attitude information relative to a certain world coordinate system (which is usually fixed to the other platform). Obtaining accurate and reliable position and attitude information is a crucial step for platform navigation, guidance, and control; therefore, it is necessary to find an accurate, reliable, and efficient method and device to solve for the position and attitude information between the two platforms.

[0003] Currently, methods for determining the relative pose of dual-motion platforms are categorized into two types based on whether the platforms exchange absolute pose information via a communication link. One type involves first obtaining the platform's own absolute pose through GNSS, inertial navigation, or a combined satellite-inertial navigation system, and then exchanging absolute pose information with another platform to obtain the relative pose. In this type of method, GNSS acquisition of absolute pose is unusable under GNSS-denied conditions such as confined spaces, terrain obstruction, or human interference. Inertial navigation measurement errors accumulate over time, and communication between the two platforms is limited by terrain conditions, weather conditions, and human interference, resulting in low reliability. The other type is vision-based measurement methods. These methods are further divided into cooperative and non-cooperative measurement methods, depending on whether additional markers are placed on the platform. Cooperative measurement methods generally require the placement of cooperative markers on the platform, such as pyramidal reflector cross arrays, T-shaped infrared thermal radiation cooperative markers, and H-shaped cooperative markers. This method mainly extracts the point and line features of the cooperation markers, and then calculates the relative position and attitude between the two-motion platforms based on the geometric characteristics of the cooperation markers and the imaging relationship. Because the method uses cooperation markers, the features are obvious. However, the additional deployment of markers will introduce modifications to the original design of the platform and reduce the concealment of the interaction task of the two-motion platform in a specific environment.

[0004] Due to factors such as potential communication link interference, mission concealment requirements, and platform structure design stability requirements in actual tasks, the above methods cannot be adapted to common dual-action platform interaction tasks. Summary of the Invention

[0005] Therefore, it is necessary to provide a dual-motion platform interaction method and system based on a non-cooperative mode that can solve for precise pose in order to address the above-mentioned technical problems.

[0006] A method for interaction between two motion platforms in a non-cooperative mode is provided. The method is implemented on two mutually targeting motion platforms in a non-cooperative mode without additional markers. The method includes: Acquire a target image sequence, the target image sequence comprising multiple frames of target images arranged in chronological order; A target image frame is selected from the target image sequence as the initial frame image. Based on the initial frame image, a pose detection algorithm is used to solve for the relative pose parameters of the target in the frame image. Then, the relative pose parameters of the target in multiple frames after the initial frame image are solved sequentially. The pose detection algorithm includes: using a trained detection neural network to detect target key points in the target image to obtain target key point detection results; and obtaining the relative pose parameters of the world coordinate system and the camera coordinate system fixed to the target by solving the PnP problem based on the camera intrinsic parameters, the target 3D model, the world coordinate system fixed to the target, and the target key point detection results. The target relative pose parameters obtained based on the initial frame image are used as initial pose parameters. Based on the initial pose parameters and all subsequent frame images in the target image sequence after the initial frame image, a pose tracking algorithm is used to obtain the relative pose parameters of the target in the subsequent frame images, so as to realize the tracking of the target. When the relative pose parameters of the target in the candidate frame image are obtained by the pose tracking algorithm, the tracking result is also verified by a consistency discrimination strategy based on the relative pose parameters in the target image of the corresponding frame obtained by the pose detection algorithm. If the verification standard is not met, the initial pose parameters in the pose tracking algorithm are updated by the relative pose parameters of the target in the current frame obtained by the pose detection algorithm.

[0007] In one embodiment, the step of using a trained detection neural network to detect target key points in the target image, and obtaining target key point detection results, includes: The trained target recognition network is used to identify the targets in the target image to obtain the target recognition results; A trained target key point neural network based on a heatmap key point detection algorithm is used to detect target key points in the target image based on the target recognition results, thereby obtaining the target key point detection results.

[0008] In one embodiment, during the training of the target recognition network and the target keypoint neural network: A training dataset is established based on the target 3D model using a graphics simulation engine. The training dataset includes multiple target simulation images. During the image rendering process, the background of the images in the public dataset and the texture of the target 3D model are randomly processed. The ground truth labels for each target simulation image include: the bounding box information of the target, the key point index value, and the image coordinates of the target key points.

[0009] In one embodiment, during the heatmap regression process in the heatmap key point detection algorithm, a probabilistic orientation is used to eliminate quantization errors in the heatmap regression.

[0010] In one embodiment, obtaining the relative pose parameters of the target by solving a PnP problem based on the target's 3D model, a world coordinate system fixed to the target, and the target's key point detection results includes: Based on the target 3D model, the RANSAC algorithm is used to remove outlier points from the target key point detection results; The coordinate positions of the target key points on the two-dimensional image are determined based on the target key point detection results after removing outliers; Based on the two-dimensional coordinates of the target key points, the world coordinate system fixed to the target, and the three-dimensional model of the target, the relative pose parameters of the target are obtained by solving the PnP problem.

[0011] In one embodiment, the pose tracking algorithm includes: The target 3D model is rendered under the initial pose parameters, and the model outline is obtained by projecting it onto a 2D image. The scale of the projected contour is calculated based on the model contour rendering, and sampling points are determined on the model contour rendering. A scale-adaptive local region model is established in subsequent frame images based on the location of the sampling points and the scale of the projected contour. The segmentation energy function of the scale-adaptive local region model is optimized to obtain the relative pose parameters of the target in subsequent frame images, and the relative pose parameters are iteratively updated.

[0012] In one embodiment, the consistency determination strategy includes: For the same frame of target image, calculate the intersection-union ratio between the reprojected two-dimensional bounding box of the tracking result obtained by the pose tracking algorithm and the target recognition box obtained by the target recognition network in the pose detection algorithm, and compare the calculation result with a first preset threshold. The algorithm also calculates the distance between the two-dimensional projection point of the target key point corresponding to the tracking result obtained by the pose tracking algorithm and the target key point position obtained by the pose detection algorithm based on the target key point neural network, and compares the calculation result with the second preset threshold.

[0013] In one embodiment, when verifying the tracking results obtained by the pose tracking algorithm using the consistency discrimination strategy: When the calculated result of the intersection-union ratio is greater than or equal to the first preset threshold, and the distance between the positions of the top preset number of target key points with confidence in the target key point detection results obtained by the target key point neural network and the two-dimensional projection point is less than or equal to the second preset threshold, then the verification standard is met. Otherwise, it does not meet the verification criteria.

[0014] This application also provides a dual-motion platform interaction system based on a non-cooperative mode. The system includes two mutually targeting motion platforms. Each motion platform includes a camera device and a data processing unit. The camera device captures images of the target to obtain a target image sequence and sends the target image sequence to the data processing unit. The data processing unit implements the dual-motion platform interaction method described above.

[0015] In one embodiment, the data processing unit includes a GPU module and a CPU module, wherein the pose detection algorithm is implemented in the GPU module and the pose tracking algorithm is implemented in the CPU module.

[0016] A dual-motion platform interaction device based on a non-cooperative mode, the device comprising: A target image sequence acquisition module is used to acquire a target image sequence, the target image sequence comprising multiple frames of target images arranged in chronological order; The pose detection module is used to select a target image frame as the initial frame image in the target image sequence, and based on the initial frame image, use a pose detection algorithm to solve for the relative pose parameters of the target in the frame image, and then solve for the relative pose parameters of the target in multiple frames after the initial frame image in sequence. The pose detection algorithm includes: using a trained detection neural network to detect target key points in the target image to obtain target key point detection results, and obtaining the relative pose parameters of the world coordinate system and the camera coordinate system fixed to the target by solving the PnP problem based on the camera intrinsic parameters, the target 3D model, the world coordinate system fixed to the target, and the target key point detection results. The pose tracking module is used to take the target relative pose parameters obtained based on the initial frame image as the initial pose parameters, and use a pose tracking algorithm to obtain the relative pose parameters of the target in the subsequent frame images according to the initial pose parameters and all subsequent frame images in the target image sequence after the initial frame image, so as to realize the tracking of the target. The pose tracking verification module is used to verify the tracking results by using a consistency discrimination strategy based on the relative pose parameters of the target in the candidate frame image obtained by the pose tracking algorithm. If the results do not meet the verification criteria, the initial pose parameters in the pose tracking algorithm are updated with the relative pose parameters of the target in the current frame obtained by the pose detection algorithm.

[0017] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps: Acquire a target image sequence, the target image sequence comprising multiple frames of target images arranged in chronological order; A target image frame is selected from the target image sequence as the initial frame image. Based on the initial frame image, a pose detection algorithm is used to solve for the relative pose parameters of the target in the frame image. Then, the relative pose parameters of the target in multiple frames after the initial frame image are solved sequentially. The pose detection algorithm includes: using a trained detection neural network to detect target key points in the target image to obtain target key point detection results; and obtaining the relative pose parameters of the world coordinate system and the camera coordinate system fixed to the target by solving the PnP problem based on the camera intrinsic parameters, the target 3D model, the world coordinate system fixed to the target, and the target key point detection results. The target relative pose parameters obtained based on the initial frame image are used as initial pose parameters. Based on the initial pose parameters and all subsequent frame images in the target image sequence after the initial frame image, a pose tracking algorithm is used to obtain the relative pose parameters of the target in the subsequent frame images, so as to realize the tracking of the target. When the relative pose parameters of the target in the candidate frame image are obtained by the pose tracking algorithm, the tracking result is also verified by a consistency discrimination strategy based on the relative pose parameters in the target image of the corresponding frame obtained by the pose detection algorithm. If the verification standard is not met, the initial pose parameters in the pose tracking algorithm are updated by the relative pose parameters of the target in the current frame obtained by the pose detection algorithm.

[0018] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Acquire a target image sequence, the target image sequence comprising multiple frames of target images arranged in chronological order; A target image frame is selected from the target image sequence as the initial frame image. Based on the initial frame image, a pose detection algorithm is used to solve for the relative pose parameters of the target in the frame image. Then, the relative pose parameters of the target in multiple frames after the initial frame image are solved sequentially. The pose detection algorithm includes: using a trained detection neural network to detect target key points in the target image to obtain target key point detection results; and obtaining the relative pose parameters of the world coordinate system and the camera coordinate system fixed to the target by solving the PnP problem based on the camera intrinsic parameters, the target 3D model, the world coordinate system fixed to the target, and the target key point detection results. The target relative pose parameters obtained based on the initial frame image are used as initial pose parameters. Based on the initial pose parameters and all subsequent frame images in the target image sequence after the initial frame image, a pose tracking algorithm is used to obtain the relative pose parameters of the target in the subsequent frame images, so as to realize the tracking of the target. When the relative pose parameters of the target in the candidate frame image are obtained by the pose tracking algorithm, the tracking result is also verified by a consistency discrimination strategy based on the relative pose parameters in the target image of the corresponding frame obtained by the pose detection algorithm. If the verification standard is not met, the initial pose parameters in the pose tracking algorithm are updated by the relative pose parameters of the target in the current frame obtained by the pose detection algorithm.

[0019] The aforementioned dual-motion platform interaction method and system based on a non-cooperative mode selects one frame of the target image sequence as the initial frame image. Based on the initial frame image, a pose detection algorithm is used to solve for the relative pose parameters of the target in that frame image. Then, the relative pose parameters of the target in subsequent frames are solved sequentially. The pose detection algorithm includes: using a trained detection neural network to detect target key points in the target image, obtaining the target key point detection results; and, based on the target's 3D model, the world coordinate system fixed to the target, and the target key point detection results, solving a PnP problem to obtain the target's relative pose parameters. Then, the... Using the target's relative pose parameters obtained from the initial frame image as initial pose parameters, and based on these initial pose parameters and all subsequent frame images in the target image sequence after the initial frame image, a pose tracking algorithm is employed to obtain the target's relative pose parameters in these subsequent frame images, thus achieving target tracking. Simultaneously, when obtaining the target's relative pose parameters in candidate frame images using the pose tracking algorithm, a consistency judgment strategy is applied to verify the tracking results against the relative pose parameters in the corresponding frame target image obtained by the pose detection algorithm. If the verification criteria are not met, the initial pose parameters in the pose tracking algorithm are updated using the target's relative pose parameters in the current frame obtained by the pose detection algorithm. This method can accurately track the pose of another motion platform in non-cooperative mode, providing precise and reliable relative pose calculation information for dual-motion platform interaction, thereby supporting the successful completion of dual-motion platform interaction tasks. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating a dual-action platform interaction method based on a non-cooperative mode in one embodiment. Figure 2 This is a schematic diagram of a quantization error optimization method using random rounding in one embodiment; Figure 3 This is a schematic diagram of the fusion framework of pose detection algorithm and pose tracking algorithm in one embodiment; Figure 4 This is a schematic diagram of the pose tracking algorithm framework in one embodiment; Figure 5 This is a schematic diagram of the intersection-union ratio and pixel distance in a consistency discrimination strategy in one embodiment; Figure 6 This is a flowchart illustrating a dual-action platform interaction method in a non-cooperative mode, as shown in another embodiment. Figure 7 This is a structural block diagram of a dual-motion platform interaction device based on a non-cooperative mode in one embodiment. Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0022] To address the issue that existing methods for relative pose detection of dual-motion platforms applied in cooperative modes cannot be applied in non-cooperative modes, this embodiment addresses this problem. Figure 1 As shown, a dual-motion platform interaction method based on non-cooperative mode is provided. This method is implemented in two non-cooperative motion platforms that are mutually targeted, and specifically includes the following steps: Step S100: Obtain the target image sequence, which includes multiple target images arranged in chronological order.

[0023] Step S110: Select a target image frame from the target image sequence as the initial frame image. Based on the initial frame image, use a pose detection algorithm to solve for the relative pose parameters of the target in that frame image. Then, solve for the relative pose parameters of the target in multiple frames after the initial frame image. The pose detection algorithm includes: using a trained detection neural network to detect target key points in the target image and obtaining target key point detection results; and obtaining the relative pose parameters of the world coordinate system and the camera coordinate system fixed to the target by solving the PnP problem based on the camera intrinsic parameters, the target 3D model, the world coordinate system fixed to the target, and the target key point detection results.

[0024] Step S120: The target relative pose parameters obtained based on the initial frame image are used as the initial pose parameters. Based on the initial pose parameters and all subsequent frame images in the target image sequence after the initial frame image, a pose tracking algorithm is used to obtain the target relative pose parameters in the subsequent frame images, so as to achieve target tracking.

[0025] In step S130, when the relative pose parameters of the target in the candidate frame image are obtained by the pose tracking algorithm, the tracking result is also verified by a consistency discrimination strategy based on the relative pose parameters in the target image of the corresponding frame obtained by the pose detection algorithm. If the verification standard is not met, the initial pose parameters in the pose tracking algorithm are updated by the relative pose parameters of the target in the current frame obtained by the pose detection algorithm.

[0026] In this embodiment, the process described above can actually be understood as the steps of an online target pose detection process. By implementing the above method on two platforms that are in motion and do not communicate with each other in any way, the real-time pose parameters of the other platform as the target can be obtained, thereby providing support for the smooth completion of subsequent interactive tasks between the two platforms.

[0027] Before performing the online target pose detection described above, some offline preparations are required. This includes setting the lens focal length of the platform's camera equipment (camera or video camera, exemplified by a camera in this example), calibrating the camera equipment, and determining its intrinsic parameters; determining the transformation matrix between the target platform's N / A coordinate system and the world coordinate system; obtaining the target platform's 3D model; and training the target detection network. The training of the target detection network will be described in detail later, while the other offline preparation steps can be understood as existing technologies, and will only be briefly described below.

[0028] As an example, based on the characteristics of the interactive task and the principle of margin design, when the platform target is designed at 5km and centered on the image plane, the horizontal pixel size is 40 pixels. Based on the camera pixel size... Given that the maximum width of the target main structure is 20m, the rated focal length of the lens is calculated as follows:

[0029] Adjust the camera's focal length according to the calculated focal length method described above, and set the camera aperture to an appropriate size. Then, calibrate the camera to determine its intrinsic parameters.

[0030] In this embodiment, the coordinates of key points on the target platform in the total station's northeast-sky coordinate system are measured using a total station electronic distance measuring instrument. Three key points on the target platform are selected to determine the world coordinate system fixedly connected to the target platform. The transformation relationship is determined based on the coordinates of the three key points in the northeast-sky coordinate system, thereby obtaining the coordinates of all key points in the world coordinate system.

[0031] The three key points are usually selected from the origin of the world coordinate system. A point on the X-axis of the world coordinate system A point on the upper half-plane of the world coordinate system XOY Thus, the transformation matrix from the total station's northeast-sky coordinate system to the world coordinate system is determined:

[0032] in,

[0033]

[0034]

[0035]

[0036] In step S100, the camera configured by the platform takes another moving platform as the target to take pictures, and obtains multiple frames of target images that record different poses of the target platform over a period of time.

[0037] In step S110, the trained detection neural network is used to detect target key points in the target image. The target key point detection result is obtained by sequentially detecting target key points through two trained neural networks. First, the trained target recognition network is used to recognize the target in the target image, obtaining the target recognition result. Then, the trained target key point neural network, based on a heatmap key point detection algorithm, detects target key points in the target image according to the target recognition result, obtaining the target key point detection result.

[0038] Since this method is applied to a non-cooperative mode, meaning that no cooperative markers are placed on the target platform, this method uses a trained deep neural network to identify key points of the target platform based on its structural features.

[0039] In this embodiment, when training the target recognition network and the target keypoint neural network: a graphics simulation engine is used to establish a training dataset based on the target 3D model. The training dataset includes multiple target simulation images. During image rendering, the background of the images in the public dataset and the texture of the target 3D model are randomly processed. The ground truth labels for each target simulation image include: the bounding box information of the target, the keypoint index value, and the image coordinates of the target keypoints.

[0040] Specifically, to enable the neural network to better adapt to the task scenario and possess strong robustness against various background disturbances, a quantitative set of images with ground truth labels is generated as the training set for the network model. First, the Blenderproc graphics simulation engine is used, combined with the target's 3D model, to establish a simulation dataset. To ensure the network learns the target's main structural features, during image rendering, images from the COCO dataset are used to randomize the background of the training set images and the texture of the target 3D model, thereby improving the network model's generalization performance.

[0041] In this embodiment, when providing the true values ​​of key points for each target simulation image, key points are selected based on the task characteristics and structural features of the target platform. For example, when a rotorcraft lands on a moving landing platform such as a bus, it typically approaches from the side and lands vertically. Therefore, the key points are usually selected from the structural features of the bus's side, such as the corner points of the rearview mirrors and windows, as well as the structural features of the top, such as the corner points of the sunroof and the corner points of the car's exterior shape.

[0042] In this embodiment, both the target recognition network and the target key point neural network adopt convolutional neural networks.

[0043] In one embodiment, a YOLOv5-based object detection algorithm is used. The YOLO series of algorithms places the entire image in an instance and predicts the coordinates of these bounding boxes and their class probabilities. It is a one-stage object detection algorithm with the advantages of high efficiency, flexibility and good generalization performance.

[0044] In this embodiment, a heatmap-based keypoint detection algorithm is used in the target keypoint neural network. In the heatmap regression process, the quantization error in the heatmap regression is eliminated by using probability direction.

[0045] Specifically, a heatmap, or confidence map, is a simple and effective representation of keypoint locations. Given the numerical coordinates x of the i-th keypoint, this keypoint corresponds to a specific heatmap h. Through the heatmap representation, the keypoint localization problem can be transformed into a heatmap regression problem, which includes two parts: encoding and decoding. The encoding process maps the ground truth coordinates x of the keypoint to the ground truth coordinates h of the heatmap, and the decoding process maps the predicted values ​​from the heatmap back to the predicted values ​​x of the coordinates. However, existing heatmap regression methods require high-resolution images and rely on empirical compensation to alleviate quantization problems during inference. Heatmap-based regression typically includes two key steps: heatmap preheating and the conversion from heatmap to numerical coordinates.

[0046] To preserve score information when using heatmap representation, multiple activation points are used around the activation points of the ground truth keypoints. A probabilistic method is employed to eliminate quantization errors in heatmap regression. Specifically, the score portion of the score coordinates is encoded to different activation points with varying activation probabilities. The score portion of the ground truth keypoint coordinates can be reconstructed from the predicted heatmap using the activation probabilities of all activation points. The unbiased encoding and decoding operations ensure that the quantization system is lossless, eliminating quantization errors and enabling efficient and high-precision detection of target keypoints.

[0047] During training, when the ground truth numerical coordinates x are floating-point numbers, a specific Gaussian kernel matrix needs to be calculated for each keypoint, which significantly increases the training load of the heatmap regression model. Currently, to address this issue, existing techniques efficiently generate ground truth heatmaps by quantizing numerical coordinates to integers and sharing a standard kernel matrix. However, this results in the inability to encode the fractional portion of the numerical coordinates, leading to errors.

[0048] Typical heatmap regression methods cannot effectively handle the decimal part of numerical coordinates, leading to positioning errors. The positioning error of a heatmap is usually expressed as follows:

[0049] In the above formula, Represents the predicted point location. The location of the truth point. This indicates the location of the extreme points on the heatmap. For heatmap error terms, This is the quantization error term.

[0050] In heatmap regression, to preserve score information when using heatmap representation, multiple activation points are used around the activation points of the ground truth keypoints. A probabilistic method is used to eliminate quantization errors in heatmap regression. Specifically, the fractional portion of the score coordinates is encoded to different activation points with different activation probabilities, such as... Figure 2 As shown.

[0051] Given the true coordinates of the key points The downsampling step size 's' is set, and the resulting activation points are typically floating-point numbers that cannot be found in the heatmap. True activation points are randomly assigned to an optional activation point nearby, with the activation probability determined by the fractional part of the true activation point. The fractional part of the true coordinates of the keypoints can be reconstructed from the predicted map using the activation probabilities of all activation points. Unbiased encoding and decoding operations ensure that the quantization system is lossless and free from quantization error.

[0052] like Figure 2 As shown, a random rounding method is used to optimize quantization error and solve the problem of accurate detection of key points on the platform. A true activation point is randomly assigned to one of the surrounding optional activation points, and the activation probability is determined by the fractional part of the true activation point. Therefore, the fractional part of the true coordinates of the key point can be reconstructed from the predicted heatmap using the activation probabilities of all activation points, i.e.:

[0053] Using this method, the unbiased operation of encoding and decoding makes the quantization error system lossless, that is, there is no quantization error, which can efficiently and accurately detect the key points of the target platform.

[0054] In this embodiment, when performing pose tracking on the target platform, the pose detection algorithm and the pose tracking algorithm are actually fused together to improve the accuracy of the results. The specific fusion process of the two algorithms is as follows: Figure 3 As shown.

[0055] In step S110, a target image is first selected from the target image sequence using a pose detection algorithm. The target keypoints in this image are detected using a trained neural network. Then, based on the prior-obtained 3D target model, the world coordinate system fixed to the target, and the target keypoint detection results, the relative pose parameters of the target are obtained by solving a PnP problem. The obtained relative pose parameters are actually the target's position coordinates in the world coordinate system. These relative pose coordinates are used as the initial pose parameters for the pose tracking calculation in step S120. Subsequent target images are used to continuously track the target pose. To ensure the reliability of the pose tracking algorithm results, the verification method in step S130 is used to verify the results. It can also be understood that step S130 is not actually implemented after step S120, but rather is a part of step S120.

[0056] Specifically, based on the prior obtained target 3D model, the world coordinate system fixed to the target, and the target key point detection results, the relative pose parameters of the target are obtained by solving the PnP problem. This includes: first, based on the target 3D model, using the RANSAC algorithm to remove outliers from the target key point detection results; then, determining the coordinate positions of the target key points in the world coordinate system based on the target key point detection results after removing outliers; and finally, based on the coordinate positions of the target key points and the target's prior 3D model, obtaining the target's relative pose parameters by solving the PnP problem.

[0057] In this embodiment, the target key points obtained by the pose detection algorithm are multiple, and the pose of the target platform is determined by the position coordinates of the multiple target key points in the world coordinate system.

[0058] In fact, in this embodiment, after obtaining the initial pose parameters of the target using the pose detection algorithm, the target pose in subsequent target images is further detected for verification of the pose tracking algorithm.

[0059] Specifically, in step S120, as follows Figure 4As shown, the pose tracking algorithm includes: rendering the target 3D model under initial pose parameters and projecting it onto a 2D image to obtain the model contour rendering; calculating the scale of the projected contour based on the model contour rendering; determining sampling points on the model contour rendering; establishing a scale-adaptive local region model in subsequent frame images based on the position of the sampling points and the scale of the projected contour; optimizing and solving the segmentation energy function of the scale-adaptive local region model to obtain the relative pose parameters of the target in subsequent frame images; and iteratively updating the relative pose parameters.

[0060] Specifically, in the pose tracking algorithm, based on the initial pose parameters of the target, a scale-adaptive local region model is established in the subsequent target image. By continuously iteratively optimizing and solving its segmentation energy function, the relative pose parameters of the target in the subsequent frame target image are obtained. After obtaining the tracking result, the pose detection algorithm will verify the detection result obtained for the same frame image and finally output a reliable tracking result.

[0061] In this embodiment, if the pose tracking result verification does not meet the standard, it indicates that pose tracking has failed. In this case, the pose detection result can be re-obtained from the pose detection algorithm as the initial pose parameters for re-pose tracking. Similarly, when the pose tracking algorithm is interrupted and restarted, the pose detection result is also obtained as the initial pose parameters for re-pose tracking.

[0062] In this embodiment, the pose detection algorithm specifically adopts a 6D pose detection algorithm, and the pose tracking algorithm specifically adopts a 6D pose tracking algorithm.

[0063] In this embodiment, the proposed consistency discrimination strategy includes: for the same frame of target image, calculating the intersection-union ratio (IUGR) between the reprojected two-dimensional bounding box of the tracking result obtained by the pose tracking algorithm and the target recognition box obtained by the target recognition network in the pose detection algorithm, and comparing the calculated result with a first preset threshold. Simultaneously, calculating the distance between the two-dimensional projection points of the target key points corresponding to the tracking result obtained by the pose tracking algorithm and the target key point positions obtained by the target key point neural network in the pose detection algorithm, and comparing the calculated result with a second preset threshold.

[0064] Specifically, when verifying the tracking results obtained by the pose tracking algorithm using a consistency discrimination strategy: if the calculated result of the intersection-union ratio is greater than or equal to the first preset threshold, and the distance between the positions of the top preset number of target key points with confidence in the target key point detection results obtained by the target key point neural network and the two-dimensional projection point is less than or equal to the second preset threshold, then the verification standard is met; otherwise, the verification standard is not met.

[0065] like Figure 5 As shown, the consistency discrimination strategy of the pose tracking algorithm consists of the intersection-union ratio (IU) threshold and the distance between corresponding keypoints. In the target detection algorithm, the target detection box region based on deep learning is represented as... The reprojected 2D bounding box region of the pose tracking result obtained by the pose tracking algorithm is represented as: Then the intersection and comparison is:

[0066] In target keypoint detection, the keypoint detection results based on deep learning are as follows: The 2D projection points of the corresponding key points obtained by the pose tracking algorithm are: The distance to the corresponding key point is:

[0067] In one embodiment, the first preset threshold can be set to 0.8, the second preset threshold can be set to 5 pixels, and the distance between the top five target key points with the highest confidence in the target key point detection results and the projection point can be calculated. When the pose tracking result meets both of these criteria, it indicates that the current result is reliable.

[0068] In another embodiment, the implementation steps of the dual-motion platform interaction method mentioned herein may also be as shown in Figure 6.

[0069] In the aforementioned dual-motion platform interaction method based on a non-cooperative mode, a suitable camera and lens are first selected and the lens focal length is determined according to the characteristics of the interaction task and the principle of margin design. Then, a mature calibration method is used to calibrate the camera and obtain its intrinsic parameters. Next, the transformation matrix of the camera coordinate system relative to its own platform coordinate system, i.e., the camera mounting matrix, is determined through optical calibration methods. Key points with obvious structural features are selected on the target platform, and the coordinates of the key points in the world coordinate system are obtained through the target 3D model or a precise surveying instrument. After obtaining a simulation dataset using the target 3D model and a simulation engine, or obtaining a real dataset by actually simulating the dual-motion platform interaction task, a mature convolutional neural network is used to achieve robust detection of the platform target. Then, a key point detection algorithm based on heatmaps is used to detect and locate the key points of the target platform. Finally, based on the prior information of the target platform 3D model, RANSAC is used to remove outliers, and the 6D pose of the target platform in the image is achieved by solving the PnP problem. Based on the continuity of motion, after the 6D pose detection algorithm calculates the initial pose, the results obtained from the 6D pose tracking algorithm are fused to achieve iterative refinement of the pose. This method requires only an optical camera, a lens, and a processor capable of supporting parallel computing needs for deep learning-based pose detection on a GPU and adaptive local region model-based pose tracking on a CPU. The device is low-cost, simple to operate, and highly reliable. This method and device can reliably and accurately calculate relative poses in dual-motion platform interaction tasks, thus providing precise navigation information for dual-motion platform interaction.

[0070] This method also provides a dual-motion platform interaction system based on a non-cooperative mode. This system includes two mutually targeting motion platforms, each equipped with a camera and a data processing unit. The camera captures images of the target to obtain a sequence of target images, which is then sent to the data processing unit. The data processing unit implements the aforementioned dual-motion platform interaction method. This method can be implemented on, but is not limited to, the following dual-motion platforms: two spacecraft rendezvous and docking, where cameras can be installed on each platform to obtain precise relative poses. This method is used to obtain the transformation relationship between the other and itself (in reality, one side obtains the transformation relationship between its camera coordinate system and a world coordinate system fixed to the other; if the relative poses of the two spacecraft are required, the pose transformation relationship between the camera and its own spacecraft system also needs to be calibrated); it can also be a rotary-wing UAV and a mobile landing platform, etc.

[0071] In this embodiment, a camera and lens are selected based on different dual-motion platform interaction scenarios, meeting the platform size, weight, and power consumption requirements. The camera is required to operate stably in various harsh environments and possess high definition, ease of installation, and ease of use. In actual interaction tasks, a fixed-focus lens is generally used, and a suitable focal length is selected based on the characteristics of the interaction task while adhering to the margin design principle. This ensures rapid detection of targets within a large field of view.

[0072] In this embodiment, the data processing unit includes a GPU module and a CPU module, wherein the pose detection algorithm is implemented in the GPU module and the pose tracking algorithm is implemented in the CPU module.

[0073] Specifically, the overall algorithm divides image processing into two parallel threads: pose detection and pose tracking. The pose detection thread primarily runs on the deep learning accelerator processor. After multiple experimental tests, this thread processes each frame of image in an average of 32.4ms. Specifically, the object detection module runs on NPU1, with an average processing time of 28.5ms per frame, the keypoint detection module runs on NPU2, with an average processing time of 3.6ms per frame, and the pose calculation based on keypoints runs on the CPU, taking 0.3ms per frame. In the pose tracking thread, after the target 3D model and initial pose input are received, the model is first rendered and projected on the GPU. This initialization process of reading the 3D model when starting the thread is time-consuming and requires pre-initialization. The average rendering time for subsequent images is approximately 3.2ms per frame. Iterative optimization and solution of pose parameters based on image information takes an average of 36.5ms per frame, and consistency judgment also takes 1~2ms. According to statistics, the average processing time of the overall algorithm for a single frame image on the processor is 46.5ms, which basically meets the requirements for real-time processing.

[0074] In this embodiment, the control computer of the dual-motion platform is configured with an ARM+GPU-based hardware architecture, and the software architecture supports the Linux system and provides software library support for CUDA, cuDNN and TensorRT.

[0075] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0076] In one embodiment, such as Figure 7 As shown, a dual-motion platform interaction device based on a non-cooperative mode is provided, including: a target image sequence acquisition module 200, a pose detection module 210, a pose tracking module 220, and a pose tracking verification module 230, wherein: The target image sequence acquisition module 200 is used to acquire a target image sequence, the target image sequence including multiple frames of target images arranged in chronological order; The pose detection module 210 is used to select a target image frame as an initial frame image in the target image sequence, and based on the initial frame image, use a pose detection algorithm to solve the relative pose parameters of the target in the frame image, and then solve the relative pose parameters of the target in multiple frames after the initial frame image in sequence. The pose detection algorithm includes: using a trained detection neural network to detect target key points in the target image to obtain target key point detection results, and obtaining the relative pose parameters of the world coordinate system and the camera coordinate system fixed to the target by solving the PnP problem based on the camera intrinsic parameters, the target 3D model, the world coordinate system fixed to the target, and the target key point detection results. The pose tracking module 220 is used to take the target relative pose parameters obtained based on the initial frame image as the initial pose parameters, and use a pose tracking algorithm to obtain the relative pose parameters of the target in the subsequent frame images according to the initial pose parameters and all subsequent frame images in the target image sequence after the initial frame image, so as to realize the tracking of the target. The pose tracking verification module 230 is used to verify the tracking results by using a consistency discrimination strategy based on the relative pose parameters of the target in the candidate frame image obtained by the pose tracking algorithm when the relative pose parameters of the target in the corresponding frame image obtained by the pose detection algorithm are obtained by using the pose tracking algorithm to obtain the relative pose parameters of the target in the current frame. If the verification results do not meet the verification criteria, the initial pose parameters in the pose tracking algorithm are updated by using the relative pose parameters of the target in the current frame obtained by the pose detection algorithm.

[0077] Specific limitations regarding the dual-action platform interaction device in non-cooperative mode can be found in the limitations of the dual-action platform interaction method in non-cooperative mode described above, and will not be repeated here. Each module in the aforementioned dual-action platform interaction device in non-cooperative mode can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0078] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 8 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a dual-platform interaction method based on a non-cooperative mode. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0079] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0080] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps: Acquire a target image sequence, the target image sequence comprising multiple frames of target images arranged in chronological order; A target image frame is selected from the target image sequence as the initial frame image. Based on the initial frame image, a pose detection algorithm is used to solve for the relative pose parameters of the target in the frame image. Then, the relative pose parameters of the target in multiple frames after the initial frame image are solved sequentially. The pose detection algorithm includes: using a trained detection neural network to detect target key points in the target image to obtain target key point detection results; and obtaining the relative pose parameters of the world coordinate system and the camera coordinate system fixed to the target by solving the PnP problem based on the camera intrinsic parameters, the target 3D model, the world coordinate system fixed to the target, and the target key point detection results. The target relative pose parameters obtained based on the initial frame image are used as initial pose parameters. Based on the initial pose parameters and all subsequent frame images in the target image sequence after the initial frame image, a pose tracking algorithm is used to obtain the relative pose parameters of the target in the subsequent frame images, so as to realize the tracking of the target. When the relative pose parameters of the target in the candidate frame image are obtained by the pose tracking algorithm, the tracking result is also verified by a consistency discrimination strategy based on the relative pose parameters in the target image of the corresponding frame obtained by the pose detection algorithm. If the verification standard is not met, the initial pose parameters in the pose tracking algorithm are updated by the relative pose parameters of the target in the current frame obtained by the pose detection algorithm.

[0081] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Acquire a target image sequence, the target image sequence comprising multiple frames of target images arranged in chronological order; A target image frame is selected from the target image sequence as the initial frame image. Based on the initial frame image, a pose detection algorithm is used to solve for the relative pose parameters of the target in the frame image. Then, the relative pose parameters of the target in multiple frames after the initial frame image are solved sequentially. The pose detection algorithm includes: using a trained detection neural network to detect target key points in the target image to obtain target key point detection results; and obtaining the relative pose parameters of the world coordinate system and the camera coordinate system fixed to the target by solving the PnP problem based on the camera intrinsic parameters, the target 3D model, the world coordinate system fixed to the target, and the target key point detection results. The target relative pose parameters obtained based on the initial frame image are used as initial pose parameters. Based on the initial pose parameters and all subsequent frame images in the target image sequence after the initial frame image, a pose tracking algorithm is used to obtain the relative pose parameters of the target in the subsequent frame images, so as to realize the tracking of the target. When the relative pose parameters of the target in the candidate frame image are obtained by the pose tracking algorithm, the tracking result is also verified by a consistency discrimination strategy based on the relative pose parameters in the target image of the corresponding frame obtained by the pose detection algorithm. If the verification standard is not met, the initial pose parameters in the pose tracking algorithm are updated by the relative pose parameters of the target in the current frame obtained by the pose detection algorithm.

[0082] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0083] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0084] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A two-way platform interaction method based on a non-cooperative mode, characterized in that, The method is implemented in two non-cooperative motion platforms that are mutually targeting each other. The method includes: Acquire a target image sequence, the target image sequence comprising multiple frames of target images arranged in chronological order; A target image frame is selected from the target image sequence as the initial frame image. Based on the initial frame image, a pose detection algorithm is used to solve for the relative pose parameters of the target in the frame image. Then, the relative pose parameters of the target in multiple frames after the initial frame image are solved sequentially. The pose detection algorithm includes: using a trained detection neural network to detect target key points in the target image to obtain target key point detection results; and obtaining the relative pose parameters of the world coordinate system and the camera coordinate system fixed to the target by solving the PnP problem based on the camera intrinsic parameters, the target 3D model, the world coordinate system fixed to the target, and the target key point detection results. The target relative pose parameters obtained based on the initial frame image are used as initial pose parameters. Based on the initial pose parameters and all subsequent frame images in the target image sequence after the initial frame image, a pose tracking algorithm is used to obtain the relative pose parameters of the target in the subsequent frame images, so as to realize the tracking of the target. When obtaining the relative pose parameters of the target in the candidate frame image using the pose tracking algorithm, the tracking result is also verified by a consistency discrimination strategy based on the relative pose parameters in the target image of the corresponding frame obtained by the pose detection algorithm. If the verification criteria are not met, the initial pose parameters in the pose tracking algorithm are updated with the relative pose parameters of the target in the current frame obtained by the pose detection algorithm. The consistency discrimination strategy includes: for the same frame target image, calculating the intersection-union ratio between the reprojected two-dimensional bounding box of the tracking result obtained by the pose tracking algorithm and the target recognition box obtained by the target recognition network in the pose detection algorithm, and comparing the calculation result with a first preset threshold; and also calculating the distance between the two-dimensional projection point of the target key point corresponding to the tracking result obtained by the pose tracking algorithm and the target key point position obtained by the target key point neural network in the pose detection algorithm, and comparing the calculation result with a second preset threshold.

2. The dual-motion platform interaction method according to claim 1, characterized in that, The step of using a trained detection neural network to detect target key points in the target image, and obtaining target key point detection results, includes: The trained target recognition network is used to identify the targets in the target image to obtain the target recognition results; A trained target key point neural network based on a heatmap key point detection algorithm is used to detect target key points in the target image based on the target recognition results, thereby obtaining the target key point detection results.

3. The dual-motion platform interaction method according to claim 2, characterized in that, When training the target recognition network and the target key point neural network: A training dataset is established based on the target 3D model using a graphics simulation engine. The training dataset includes multiple target simulation images. During the image rendering process, the background of the images in the public dataset and the texture of the target 3D model are randomly processed. The ground truth labels for each of the target simulation images include: the target's bounding box information, key point index values, and the target key point image coordinates.

4. The dual-motion platform interaction method according to claim 3, characterized in that, In the heatmap regression process of the heatmap key point detection algorithm, the probability direction is used to eliminate the quantization error in the heatmap regression.

5. The dual-motion platform interaction method according to claim 4, characterized in that, The process of obtaining the relative pose parameters of the target by solving a PnP problem based on the target's 3D model, a world coordinate system fixed to the target, and the target's key point detection results includes: Based on the target 3D model, the RANSAC algorithm is used to remove outlier points from the target key point detection results; The coordinate positions of the target key points on the two-dimensional image are determined based on the target key point detection results after removing outliers; Based on the two-dimensional coordinates of the target key points, the world coordinate system fixed to the target, and the three-dimensional model of the target, the relative pose parameters of the target are obtained by solving the PnP problem.

6. The dual-motion platform interaction method according to claim 5, characterized in that, The pose tracking algorithm includes: The target 3D model is rendered under the initial pose parameters, and the model outline is obtained by projecting it onto a 2D image. The scale of the projected contour is calculated based on the model contour rendering, and sampling points are determined on the model contour rendering. A scale-adaptive local region model is established in subsequent frame images based on the location of the sampling points and the scale of the projected contour. The segmentation energy function of the scale-adaptive local region model is optimized to obtain the relative pose parameters of the target in subsequent frame images, and the relative pose parameters are iteratively updated.

7. The dual-motion platform interaction method according to claim 5, characterized in that, When using the aforementioned consistency discrimination strategy to verify the tracking results obtained by the pose tracking algorithm: When the calculated result of the intersection-union ratio is greater than or equal to the first preset threshold, and the distance between the positions of the top preset number of target key points with confidence in the target key point detection results obtained by the target key point neural network and the two-dimensional projection point is less than or equal to the second preset threshold, then the verification standard is met. Otherwise, it does not meet the verification criteria.

8. A dual-action platform interaction system based on a non-cooperative mode, characterized in that: The system includes two mutually targeting motion platforms. Each motion platform includes a camera device and a data processing unit. The camera device captures images of the target to obtain a target image sequence and sends the target image sequence to the data processing unit. The data processing unit implements the dual-motion platform interaction method according to any one of claims 1 to 7.

9. The dual-motion platform interaction system according to claim 8, characterized in that, The data processing unit includes a GPU module and a CPU module, wherein the pose detection algorithm is implemented in the GPU module and the pose tracking algorithm is implemented in the CPU module.

Images