Radar and camera fusion-based refueling taper sleeve pose measurement method and system

Abstract

The invention discloses a refueling taper sleeve pose measurement method and system based on radar and camera fusion, and the method comprises the steps: carrying out the space-time joint calibration of an industrial camera and a laser radar, and obtaining a transformation matrix from a camera coordinate system to a mechanical arm coordinate system; a mechanical arm is used for collecting multiple sets of taper sleeve images of the to-be-detected refueling taper sleeve under different poses, and coordinates and space projection information of the inner and outer circular faces of the taper sleeve are obtained; identifying an inner circle target and an outer circle target of the to-be-detected refueling taper sleeve from an image acquired by the industrial camera to obtain a space attitude angle; optimizing the space attitude angle in a pre-trained multi-layer perceptron model, and determining the orientation of the to-be-tested refueling taper sleeve; a point cloud segmentation neural network model is utilized to segment a point cloud data set belonging to the outer circle of the to-be-measured refueling taper sleeve from the laser radar point cloud, the space coordinate of the circle center of the bottom surface of the refueling taper sleeve is obtained, various data are integrated to obtain the pose information of the to-be-measured refueling taper sleeve in a camera coordinate system, and the pose measurement of the three-dimensional space of the target taper sleeve is completed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent measurement of aviation equipment, in particular to a refueling cone sleeve pose measurement method and system based on fusion of laser radar and monocular camera. BACKGROUND

[0002] The autonomous aerial refueling technology is a key means to improve the long-range combat and sustained combat capability of unmanned aerial vehicles. The relative positioning technology in the docking stage is one of the most core technologies in the autonomous refueling process. The current research mainly focuses on the capture, stable tracking and spatial pose solution of the refueling cone sleeve in the docking stage. In order to achieve accurate relative positioning, scholars at home and abroad have proposed various technical routes, mainly including satellite navigation, laser radar, laser light field and machine vision based methods. Among them, the visual relative positioning technology is increasingly applied to the docking stage of autonomous aerial refueling due to its advantages of simple structure, low cost and no need for special modification.

[0003] However, the existing single sensor measurement scheme has many shortcomings. The visual-based technology is greatly affected by environmental factors such as light and weather, while the laser radar-based scheme lacks in resolution and detail capture. The traditional explicit pose solution method based on ideal geometric modeling often has large errors due to limitations such as non-ideal cone sleeve structure, complex environment, etc., and it is difficult to meet the actual demand. Although laser radar provides sparse three-dimensional point cloud information, it is limited by problems such as laser point density, resolution and angle scanning interval. When the target structure visibility is poor or the attitude inclination angle is large, it is difficult to obtain sufficient point cloud support, causing unstable attitude estimation. In addition, the camera itself calibration error will further amplify the forward geometric solution error.

[0004] CN114897984A discloses a method for visual alignment of a refueling probe and a refueling cone sleeve in autonomous aerial refueling of unmanned aerial vehicles. The method obtains data through an optical sensor device located directly above the refueling probe, and outputs the relative spatial distance measurement alignment information of the refueling probe and the refueling cone sleeve in three-dimensional space after processing. This method simplifies the refueling cone sleeve model to a key point model using the structure of the refueling cone sleeve, can detect the inner and outer contours of the cone sleeve, and can measure the relative position three-dimensional alignment information of the refueling cone sleeve and the refueling probe. However, there is still room for optimization in target detection and target tracking algorithms, and it is difficult to maintain high precision in complex environments.

[0005] Therefore, a spatial pose measurement method is needed that can fuse multi-source sensor data and adapt to complex environments. SUMMARY

[0006] The present application aims at the problems of insufficient accuracy of the attitude solution method based on ideal geometric modeling in the prior art and poor stability of the measurement scheme based on a single sensor in a complex environment, and provides a refueling cone sleeve pose measurement method and system based on radar and camera fusion.

[0007] The first aspect of the present application provides a refueling cone sleeve pose measurement method based on radar and camera fusion,

[0008] Step one: use an industrial camera and a laser radar to collect multiple sets of image and point cloud information; calibrate the internal parameters of the industrial camera to obtain the internal parameter matrix and distortion coefficient of the industrial camera; and perform space-time joint calibration on the industrial camera and the laser radar to transform the spatial coordinates of the radar point cloud to the camera coordinate system; pair each set of collected image and pose data, calibrate the pose of the mechanical arm device in the camera coordinate system by using the hand-eye calibration method, obtain the transformation matrix from the camera coordinate system to the mechanical arm coordinate system, and complete the space-time joint calibration among the laser radar, the industrial camera and the mechanical arm device;

[0009] Step two: set the to-be-measured refueling cone sleeve at the end of the mechanical arm, so that the end of the mechanical arm device collects multiple sets of cone sleeve images at different poses; use the transformation matrix from the camera coordinate system to the mechanical arm coordinate system obtained in step one to convert the mechanical arm end pose angle information corresponding to the cone sleeve image to the camera coordinate system; input the cone sleeve image corresponding to the pose angle into the pre-trained neural network-based target detection model to identify the inner and outer circular surfaces of the cone sleeve, so as to obtain the coordinates and spatial projection information of the inner and outer circular surfaces of the to-be-measured refueling cone sleeve;

[0010] Step three: based on the function mapping relationship between the to-be-measured refueling cone sleeve coordinates and pose angles in the image coordinate system obtained in step two, identify the inner and outer circular targets of the to-be-measured refueling cone sleeve from the images collected by the industrial camera, obtain the spatial pose angle, extract image features to establish a ten-dimensional image feature dataset, and establish a ten-dimensional image feature dataset; the ten-dimensional image feature dataset includes the center coordinates, target bounding box width and height, image center difference, area and aspect ratio information extracted from the images obtained in step two;

[0011] Step four: input the ten-dimensional image feature dataset established in step three into the pre-trained multilayer perceptron model to optimize the spatial pose angle obtained in step three, output the predicted pose angle of the to-be-measured refueling cone sleeve in space through the model, and determine the orientation of the to-be-measured refueling cone sleeve;

[0012] Step five: use a point cloud segmentation neural network model to segment the point cloud data set belonging to the outer circle of the to-be-measured refueling cone sleeve from the laser radar point cloud, perform plane fitting and circular fitting on the segmented outer circle region point cloud, perform geometric modeling on the outer circular surface of the cone sleeve, obtain the spatial coordinates of the center of the bottom surface of the refueling cone sleeve, and thus determine the position of the to-be-measured refueling cone sleeve.

[0013] Step Six: Using the transformation matrix from the camera coordinate system to the robotic arm coordinate system obtained in Step One, the spatial coordinates of the refueling cone sleeve to be tested obtained in Step Five are transformed to the camera coordinate system. Thus, based on the optimized spatial pose obtained in Step Four and the position information of the refueling cone sleeve to be tested in the camera coordinate system, the pose information of the refueling cone sleeve to be tested in the camera coordinate system is obtained, and the pose measurement of the target cone sleeve in three-dimensional space is completed.

[0014] In one embodiment, step one includes:

[0015] Adjust the pose of the calibration board and use an industrial camera and lidar to collect multiple sets of images and point cloud information;

[0016] The Zhang Zhengyou calibration method is used to calibrate the intrinsic parameters of the industrial camera. The homography matrix from the calibration plate plane to the image plane is solved. By constraining the homography matrix of multiple images, the intrinsic parameter matrix and distortion coefficients are solved. Then, the intrinsic parameter matrix and distortion coefficients of the industrial camera are optimized by the maximum likelihood estimation method.

[0017] The industrial camera and lidar are connected to the PTP timing clock. A projection relationship is established between each set of images acquired by the industrial camera and lidar and the point cloud data. The PTP timestamp is parsed, the time synchronization error of the camera and the lidar is calculated, and the transformation relationship between the camera coordinate system and the lidar coordinate system is established.

[0018] Hand-eye calibration is performed on the industrial camera and robotic arm device. Each set of acquired images and pose data is paired, and the transformation relationship between the camera coordinate system and the robotic arm device is established.

[0019] Furthermore, the ten-dimensional image feature dataset in step three includes the X-coordinates of the centers of the inner and outer circles of the refueling cone sleeve in the camera coordinate system. The Y-coordinates of the centers of the inner and outer circles of the refueling cone sleeve in the camera coordinate system. And the width and height of the inner and outer circular detection boxes of the cone sleeve in the image. Area of ​​outer circular detection frame Aspect Ratio of Outer Circular Target Boundary ; where the area of ​​the outer circular detection frame Aspect Ratio of Outer Circular Target Boundary .

[0020] Furthermore, in step one, each transformation matrix / transformation relation obtained is optimized using maximum likelihood estimation.

[0021] In one embodiment, the establishment of the target detection model in step two includes: acquiring multiple sets of images and their corresponding point cloud information; converting the acquired multiple sets of robotic arm end-effector attitude angle information into the camera coordinate system; and inputting the cone sleeve image corresponding to each attitude angle into the neural network after correction using distortion coefficients to train the cone sleeve detection model for target recognition of the inner and outer circular surfaces of the cone sleeve, thereby obtaining a target detection model that can recognize the inner and outer circular surfaces of the cone sleeve respectively.

[0022] In one embodiment, the multilayer perceptron model in step four uses an end-to-end regression method based on neural networks to optimize the spatial attitude angles obtained in step three. The multilayer perceptron model is constructed based on a regression method that learns the mapping relationship between image features and conical attitudes through neural networks. It uses the ReLU activation function and the mean squared error as the regression loss function. The ten-dimensional image feature dataset established in step three is used as input to perform regression training for the two attitude angles.

[0023] In one embodiment, step five specifically includes:

[0024] The robotic arm is controlled to position the refueling cone sleeve under test in different poses, and multiple sets of point cloud data are collected. The statistical outlier removal algorithm is used for noise reduction and labeling. The pre-processed point cloud data and its labels are input into a pre-trained point cloud segmentation neural network model to perform semantic segmentation on the point cloud data, and segment out the point cloud data set belonging to the outer circle of the refueling cone sleeve under test.

[0025] Then, the least squares method of three-dimensional plane fitting algorithm is used to perform plane fitting on the point cloud of the segmented part belonging to the outer circle of the refueling cone sleeve, and then perform circular fitting to obtain the spatial coordinates of the center of the outer circle of the bottom surface of the refueling cone sleeve, and use it to represent the spatial position of the refueling cone sleeve.

[0026] In one implementation, the point cloud target recognition model in step five is a deep learning convolutional neural network, which achieves local feature training through a hierarchical network structure.

[0027] A second aspect of the present invention provides a refueling cone pose measurement system based on radar and camera fusion, comprising a lidar, an industrial camera, a standard calibration plate, a processing unit, and a robotic arm device; wherein the lidar, industrial camera, and robotic arm device are all communicatively connected to the processing unit via lines;

[0028] The lidar is used to acquire point cloud information of the target cone sleeve under test, the industrial camera is used to acquire images of the target cone sleeve under test, and the end of the robotic arm device is equipped with the refueling cone sleeve to be tested, which is used to simulate the relative motion during the refueling process by adjusting the pose of the end of the refueling cone sleeve under test, and transmit the pose of the cone sleeve to the processing unit; the processing unit receives information from the lidar, industrial camera and robotic arm device, and performs image processing, model training, pose regression, point cloud segmentation and pose estimation on the received information, thereby obtaining the pose measurement of the target cone sleeve in three-dimensional space.

[0029] Furthermore, the processing unit includes a calibration module, an image target detection module, an image feature construction module, a pose regression module, a point cloud target segmentation module, a point cloud circular fitting module, and a pose output module;

[0030] The calibration module is used to acquire multiple sets of images and point cloud information, perform intrinsic parameter calibration on the industrial camera, obtain the intrinsic parameter matrix and distortion coefficient of the industrial camera, and perform spatiotemporal joint calibration on the industrial camera and lidar, transforming the spatial coordinates of the lidar point cloud to the camera coordinate system; it establishes a pairing between each set of acquired images and pose data, and uses a hand-eye calibration method to calibrate the pose of the robotic arm device in the camera coordinate system, obtaining the transformation matrix from the camera coordinate system to the robotic arm coordinate system;

[0031] The image target detection module is used to transform the robot arm end attitude angle information corresponding to the cone sleeve image into the camera coordinate system according to the transformation matrix from the camera coordinate system to the robot arm coordinate system obtained by the calibration module; and input the cone sleeve image corresponding to the attitude angle into the pre-trained neural network-based target detection model to identify the inner and outer circular surfaces of the cone sleeve, thereby obtaining the coordinates and spatial projection information of the inner and outer circular surfaces of the refueling cone sleeve to be tested.

[0032] The image feature construction module is used to obtain the spatial attitude angle based on the function mapping relationship between the coordinates and attitude angle of the refueling cone sleeve under test in the image coordinate system obtained by the image target detection module, identify the inner and outer circle targets of the refueling cone sleeve under test from the image acquired by the industrial camera, and extract image features, including center coordinates, target bounding box width and height, image center difference, area and aspect ratio.

[0033] The attitude regression module is used to optimize the spatial attitude angles obtained by the image feature construction module by inputting the image features from the image feature construction module into the pre-trained multilayer perceptron model, and to determine the orientation of the refueling cone in space by outputting the predicted attitude angle of the refueling cone in space through the model.

[0034] The point cloud target segmentation module uses a point cloud segmentation neural network to segment the point cloud data set of the outer circle of the refueling cone sleeve to be tested from the LiDAR point cloud; it performs plane fitting and circle fitting on the segmented outer circle area point cloud, performs geometric modeling on the outer circle surface of the cone sleeve, and obtains the spatial coordinates of the center of the bottom surface of the refueling cone sleeve.

[0035] The pose output module is used to transform the spatial coordinates obtained by the point cloud target segmentation module to the camera coordinate system using the transformation matrix of the calibration module. In this way, based on the spatial attitude angle obtained by the attitude regression module and the position information of the refueling cone under test in the camera coordinate system, the pose information of the refueling cone under test in the camera coordinate system can be obtained.

[0036] Furthermore, the industrial camera and the lidar have the same acquisition frequency and synchronized data timestamps, enabling joint measurement of the position and attitude of the refueling cone sleeve under test.

[0037] Furthermore, the calibration board is set within the field of view of the industrial camera and the lidar to achieve the calibration of the intrinsic parameters of the industrial camera and the calibration of the spatial coordinates of the lidar and the camera.

[0038] The beneficial effects of this invention are as follows:

[0039] 1. By combining images and point cloud information acquired by industrial cameras and LiDAR, and using a regression method that learns the mapping relationship between cone sleeve image features and posture through neural networks, the problem of traditional explicit posture calculation methods based on ideal geometric modeling being limited by factors such as non-ideal cone sleeve structure and complex environment is overcome. This significantly reduces the uncertainty of posture calculation caused by model error or measurement error, and improves the robustness and accuracy of spatial pose calculation.

[0040] 2. This invention introduces multi-dimensional explicit image geometric features, such as the coordinates of the inner and outer circle centers, the length and width of the detection box, the difference in image center offset, the area ratio, and the aspect ratio, and combines them with a deep point cloud segmentation network to identify the cone sleeve target region. Furthermore, it uses a combination of plane fitting and circular fitting to perform geometric modeling on the outer circular surface of the cone sleeve, which effectively reduces the calculation error due to the non-ideal structure of the cone sleeve and improves the accuracy of attitude estimation.

[0041] 3. Compared with traditional single-sensor measurement schemes, this invention achieves information complementarity and enhancement through the fusion of multi-sensor data. Especially in complex dynamic environments, it can provide more reliable spatial pose measurement results and effectively solve the problem of poor environmental adaptability of single sensors.

[0042] 4. This invention uses a target detection network to detect inner and outer circular targets in the cone sleeve image and trains a regression model through a data-driven approach, avoiding dependence on precise measurement of the cone sleeve's physical structural parameters. This overcomes the error sources caused by inaccurate structural modeling in traditional methods and improves the accuracy and stability of attitude calculation.

[0043] 5. By establishing a unified camera coordinate system and achieving spatiotemporal alignment between radar point cloud and image data, this invention achieves consistency in spatial coordinates and semantic information of multi-source sensor data, ensuring the accuracy and reliability of data fusion and providing a solid foundation for subsequent attitude regression. Attached Figure Description

[0044] Figure 1 schematically illustrates the structure of the cone sleeve of the target under test, wherein Figure 1a For the ideal conical sleeve geometric model, Figure 1b This is the actual geometric model of the conical sleeve;

[0045] Figure 2 This is a flowchart of the refueling cone pose measurement method based on radar and camera fusion as described in this invention;

[0046] Figure 3 This is a schematic diagram of the refueling cone pose measurement system based on radar and camera fusion;

[0047] Figure 4 It is an imaging model of the refueling cone sleeve to be tested, captured by a monocular industrial camera.

[0048] Figure 5 The image of the refueling cone sleeve to be tested is projected onto the camera coordinate plane. Imaging model;

[0049] Figure 6 The image of the refueling cone sleeve to be tested is projected onto the camera coordinate plane. Imaging model;

[0050] Figure 7 It is the distance between the two circular surfaces of the fuel filling cone sleeve to be tested. The distance between the two circular surfaces of the fuel filling cone sleeve to be tested exist Projection of a plane The distance between the two circular surfaces of the fuel filling cone sleeve to be tested exist Projection of a plane The geometric relationship between them;

[0051] Figure 8 This is a schematic diagram of the multilayer perceptron model established in step four. Wherein:

[0052] 1: LiDAR; 2: Industrial camera; 3: Calibration board; 4: Processing unit; 5: Robotic arm device; 6: Oiling cone sleeve. Detailed Implementation

[0053] To make the objectives, technical solutions, beneficial effects, and significant advancements of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings provided in the examples of the present invention. Obviously, all the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] In the description of this application, unless otherwise expressly specified and limited, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance; the term "multiple" refers to two or more; unless otherwise specified or explained, the terms "connected," "fixed," etc., should be interpreted broadly. For example, "connected" can be a fixed connection, a detachable connection, an integral connection, or an electrical connection; "connected" can be a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0055] This embodiment provides a refueling drogue attitude measurement method and system based on radar and camera fusion, which is used for high-precision relative positioning and complex dynamic docking between the tanker and the receiver aircraft during autonomous aerial refueling in military and civilian aircraft.

[0056] Traditional single-sensor measurement methods, such as vision-based techniques, are greatly affected by environmental factors such as lighting and weather, while lidar-based methods are lacking in resolution and detail capture. Therefore, the refueling cone pose measurement method based on lidar and monocular camera fusion described in this embodiment adopts multi-sensor fusion measurement. By integrating data from different types of sensors, it can achieve information complementarity and enhancement, which helps to promote the application and development of multi-sensor fusion technology in complex environment measurement.

[0057] like Figure 3 As shown, a refueling cone pose measurement system based on radar and camera fusion includes a lidar 1, a monocular industrial camera 2, a standard calibration plate 3, a processing unit 4, and a robotic arm device 5. The processing unit 4 houses a target detection model and a regression solution model. The lidar 1, industrial camera 2, and robotic arm device 5 are all communicatively connected to the processing unit 4 via wiring.

[0058] The robotic arm device 5 is used to acquire the reference value of the refueling cone sleeve 6. The refueling cone sleeve is installed at the end of the robotic arm, which is a six-degree-of-freedom robotic arm. Manual operation adjusts the spatial pose of the target cone sleeve to simulate the relative motion during refueling, allowing the target cone sleeve to adjust to different postures. The processing unit 4 acquires the position and posture of the cone sleeve in the robotic arm coordinate system. The lidar 1 is an area array radar with a sampling frequency of 10Hz; the industrial camera 2 has an image resolution of 1920×1080 and a frame rate synchronization of 10Hz. The lidar 1 and the industrial camera 2 are colinearly installed in a fixed position to acquire the image and point cloud information of the target cone sleeve. They are synchronized in time through an external triggering system. The industrial camera and lidar have the same acquisition frequency of 10Hz and synchronized data timestamps, enabling joint measurement of the position and posture of the refueling cone sleeve. The calibration plate 3 is set within the field of view to achieve coordinate system unification and camera calibration. The calibration plate 3 is a standard checkerboard calibration plate with 10×8 squares, each square having an actual physical size of 80mm×80mm. Specifically, the system adjusts the distance and posture within the field of view to achieve the internal parameter calibration of the industrial camera and the unified calibration of the spatial coordinates of the LiDAR and the camera; the processing unit 4 is connected to the mechanical wall device 5 by wiring to control the robotic arm device 5 and data processing, including image processing, model training, pose regression, point cloud segmentation and pose estimation, etc.

[0059] like Figure 2 As shown, the measurement method using the above-mentioned radar and camera fusion-based refueling cone pose measurement system includes:

[0060] Step 1: Spatiotemporal joint calibration among LiDAR, industrial camera, and robotic arm

[0061] S11: First, install the LiDAR 1 and industrial camera 2 in fixed positions, and place a checkerboard calibration board (as shown in Figure 1) within the common field of view of the industrial camera and LiDAR. Adjust the position and orientation of the calibration board to ensure that its position basically covers the entire field of view of the image, and align the coordinate system with the camera coordinate system. After adjusting the position and orientation of the calibration board, control the industrial camera 2 and LiDAR 1 through the processing unit 4 to acquire multiple sets of images and point cloud information respectively.

[0062] S12: Perform intrinsic parameter calibration on industrial camera 2 to obtain the intrinsic parameter matrix and distortion coefficients of industrial camera 2. The calibration process is as follows:

[0063] S121: The intrinsic parameters of the industrial camera are calibrated using Zhang Zhengyou's calibration method. The processing unit calculates the homography matrix from the checkerboard plane to the image plane for each image acquired in step S11. satisfy:

[0064] (1)

[0065] in For the camera intrinsic parameter matrix, Let be a rotation matrix. It is a translation vector. To calibrate the grid size, These are the pixel coordinates of the corner points of the chessboard. The coordinates of the corner points within the chessboard are given.

[0066] Solving for the intrinsic parameter matrix by constraining the homography matrix of multiple images. Value:

[0067] (2)

[0068] in , Camera focal length The coordinates of the main point.

[0069] S122: Optimize the intrinsic parameter matrix

[0070] To account for errors, multiple images are used to optimize the solution results, and maximum likelihood estimation is employed to optimize the intrinsic parameter matrix. distortion coefficient and extrinsic parameter matrix Minimize reprojection error:

[0071] (3)

[0072] in For the first The first image The observed pixel coordinates of each corner point Here, M represents the world coordinates of the corner points, N is the number of images, and M is the number of corner points in each image. It is the first The rotation matrix obtained from the image. It is the first The translation vector is obtained from the images. The optimized intrinsic parameter matrix of the industrial camera is obtained by minimizing the reprojection error. and distortion coefficient .

[0073] S13: To achieve spatiotemporal alignment between visual image information and radar point cloud data, spatiotemporal joint calibration is performed on industrial camera 2 and lidar 1 to establish a precise transformation relationship between their coordinate systems. The calibration process is as follows:

[0074] S131: Connect the industrial camera and lidar to the PTP timing clock, and send a synchronization trigger signal to both of them to achieve time synchronization between the sensors.

[0075] S132: The processing unit establishes a projection relationship between each group of images and point cloud data acquired in step S11, satisfying the following:

[0076] (4)

[0077] in, These are the pixel coordinates of the corner points of the chessboard. It is the industrial camera projection model This is the transformation matrix from the industrial camera coordinate system to the lidar coordinate system. These are the three-dimensional coordinates of the corresponding calibration board corner points in the lidar point cloud.

[0078] S133: Optimize the transformation matrix from the camera coordinate system to the lidar coordinate system.

[0079] To account for errors, multiple sets of images and point clouds are used to optimize the solution results. Maximum likelihood estimation is employed to optimize the transformation matrix from the camera coordinate system to the lidar coordinate system. , For the rotation matrix from lidar to camera, To minimize the reprojection error from the translation matrix from lidar to the camera:

[0080] (5)

[0081] in For the first The first image The observed pixel coordinates of each corner point For the first In the group of point clouds The three-dimensional coordinates of each corner point are obtained. The optimized transformation matrix from the industrial camera coordinate system to the LiDAR coordinate system is obtained by minimizing the reprojection error. .

[0082] S14: The industrial camera and the robotic arm used to fix the cone sleeve are jointly calibrated by camera and robotic arm, so as to unify the coordinate system of the industrial camera and the coordinate system of the robotic arm.

[0083] To facilitate the subsequent acquisition of the spatial attitude solution model of the target cone sleeve using parametric regression, the six-axis robotic arm connecting to the target cone sleeve needs to be calibrated in the same coordinate system as the camera. This application employs joint camera-robotic arm calibration, i.e., hand-eye calibration, to determine the extrinsic parameter transformation between the two. The calibration process is as follows:

[0084] S141: Fix the calibration plate to the end of the six-axis robotic arm so that the calibration plate is within the common field of view of the camera;

[0085] The position and posture of the six-axis robotic arm are manually adjusted, the position and posture parameters are recorded and uploaded to the processing unit 4, and the processing unit 4 controls the industrial camera to acquire images of the corresponding posture.

[0086] S142: The processing unit pairs each acquired image with pose data and solves the hand-eye calibration equations:

[0087] (6)

[0088] (7)

[0089] (8)

[0090] in, For the first The camera observes the pose of the calibration board. For the first Group, the pose of the robotic arm's end effector relative to the robotic arm base, This is the transformation matrix from the camera coordinate system to the robot arm's base coordinate system.

[0091] S143: Optimize the transformation matrix from the camera coordinate system to the robot arm coordinate system.

[0092] To optimize the solution results, multiple sets of data and images are used to account for errors. Maximum likelihood estimation is employed to optimize the transformation matrix from the camera coordinate system to the robot arm coordinate system, minimizing the reprojection error.

[0093] (9)

[0094] The transformation matrix from the camera coordinate system to the robot coordinate system is obtained by minimizing the reprojection error. .

[0095] Through the above calibration steps, the spatiotemporal joint calibration of the LiDAR, industrial camera, and robotic arm was completed. All data in the entire system, including image features, cone attitude angle labels, and 3D spatial coordinates extracted from point clouds, were unified into the camera coordinate system.

[0096] The purpose of step S12, LiDAR-camera extrinsic parameter calibration, is to transform the spatial coordinate data in the LiDAR point cloud into the camera coordinate system, thereby effectively fusing it with the target location identified in the image. Step S13, camera-robotic arm hand-eye calibration, transforms the cone-shaped attitude angle labels (Pitch and Yaw) recorded in the robotic arm base coordinate system into attitude information in the camera coordinate system, which can then be used as supervisory signals for training the attitude regression network. By establishing a unified reference system, the consistency of multi-source sensor data in spatial coordinates and semantic information is ensured. Therefore, step one is the fundamental prerequisite for data fusion and regression modeling in subsequent steps.

[0097] Step 2: Use the neural network target recognition algorithm deployed in the processing unit to identify the inner and outer circles of the refueling cone sleeve in the image and derive its geometric feature parameters such as image coordinates and target size.

[0098] S21: Set the refueling cone sleeve to be tested at the end of the robotic arm device, control the end of the robotic arm device 5 to be in different positions and different postures, and collect multiple sets of cone sleeve images under different postures through the industrial camera 2. Transmit the multiple sets of images and the corresponding robotic arm end posture angle information pitch and yaw to the processing unit for processing.

[0099] Due to the symmetrical structure of the refueling cone sleeve, the cone sleeve rotates around its own axis without changing. Therefore, the roll angle of the refueling cone sleeve under test in space can be considered to be zero at all times. The simplified spatial attitude can be represented by two parameters: pitch angle and yaw angle.

[0100] S22: Use the transformation matrix from the camera coordinate system to the robotic arm coordinate system obtained in step one. The pitch and yaw angles of the robotic arm end effector corresponding to the image obtained in S21 are transformed into the camera coordinate system and denoted as follows: and .

[0101] S23: Distortion coefficients obtained using step S12 of step one. The image acquired by S21 is corrected to make it closer to the true geometric projection.

[0102] S24: The image corrected in step S23 is labeled using the open-source LabelImg image labeling platform and the labels are exported. During labeling, an object detection box is used to select the target object, and the refueling cone sleeve is refined into two independent targets: the inner circle and the outer circle of the cone sleeve. Then, the center coordinates are normalized to make them independent of the image size.

[0103] like Figure 4 As shown in the figure For the image coordinate system, For the camera coordinate system, To establish a refueling cone sleeve coordinate system, For the camera's imaging plane, To determine the position of the center of the inner circle of the oil filling cone sleeve. This is the imaging position of the center of the inner circle of the refueling cone sleeve. The label format is... Where 0 and 1 represent the serial number of the object to be detected, 0 represents the outer circular surface of the cone sleeve, and 1 represents the inner circular surface of the cone sleeve. and The projections of the centers of the inner and outer circles of the refueling cone sleeve into the image coordinate system. Directional coordinates and After projecting the centers of the inner and outer circles of the refueling cone sleeve into the image coordinate system... Directional coordinates These represent the width and height of the inner and outer circular detection frames of the cone sleeve in the image, respectively.

[0104] S25: Input the acquired images and corresponding labels into the neural network to train the target detection model for the conical sleeve, obtaining a target detection model capable of recognizing the inner and outer circular surfaces of the conical sleeve. Input the image of the refueling conical sleeve taken by an industrial camera into the trained target detection model, and the corresponding coordinates of the inner and outer circles and the detection box size parameters will be output.

[0105] Ideally, the circle of the conical sleeve is... Figure 1a The image shows a perfect circle. However, in practical applications, to prevent collision damage, the refueling cone sleeve needs to be wrapped with a protective sleeve. Furthermore, the cone sleeve undergoes geometric deformation during use, resulting in the outer circle of the cone sleeve not being an ideal perfect circle (in the image coordinate system). This is called the "actual geometric model of the cone sleeve under test," as shown below. Figure 1b As shown. Compared with the traditional elliptical feature extraction method, using the target detection model to identify the inner and outer circular surfaces of the conical sleeve can effectively reduce the calculation error when the conical sleeve structure is not ideal.

[0106] The target detection model uses the YOLOv11 target detection model and employs binary cross-entropy (BCE) loss as a classifier for multi-class classification. The target feature is set as two parallel circular planes, with the line connecting the centers of the two circles... Perpendicular to the two circular planes.

[0107] Step 3: Without depth information, using the camera intrinsic parameters calibrated in Step 1 and the coordinates of the refueling cone obtained in Step 2 in the image coordinate system, establish the relationship between the attitude angle obtained in Step 2 and the cone in the image coordinate system based on the principles of geometric modeling and projection transformation, and solve the attitude angle of the refueling cone in reverse.

[0108] likeFigures 5-7 As shown, and These are the X-coordinates of the centers of the inner and outer circles of the oil filling cone sleeve to be measured, respectively, in the camera coordinate system. and These are the Y-coordinates of the centers of the inner and outer circles of the oil filling cone sleeve to be measured, respectively, in the camera coordinate system. and These are the Z-coordinates of the centers of the inner and outer circles of the oil filling cone sleeve to be measured, respectively, in the camera coordinate system. and To measure the pitch and yaw angles of the fuel cone sleeve to be tested, The distance between the two circular surfaces of the oil filling cone sleeve to be tested exist Projection of a plane The distance between the two circular surfaces of the oil filling cone sleeve to be tested exist Projection onto a plane.

[0109] Based on the distance between the two circular surfaces of the fuel cone sleeve to be tested The distance between the two circular surfaces of the fuel filling cone sleeve to be tested exist Projection of a plane The distance between the two circular surfaces of the fuel filling cone sleeve to be tested exist Projection of a plane The geometric relationship between them can be obtained as follows:

[0110] (10)

[0111] (11)

[0112] (12)

[0113] (13)

[0114] (14)

[0115] in, Obtained from step two, The distance between the two circular surfaces of the fuel filling cone sleeve to be measured is obtained through measurement. These are camera intrinsic parameters, obtained through camera intrinsic parameter calibration. The unknown variables in the equation are... , , , , , , There are a total of 10 equations, and 10 sets of independent equations are derived from geometric relationships. Therefore, the above system of equations has a unique solution. Because... Since they are all constant values, then we can... The attitude angle of the refueling cone sleeve to be measured Establish function mapping relationships:

[0116] (15)

[0117] (16).

[0118] Step 4: Optimize spatial attitude angles (pitch, yaw) using an end-to-end regression method based on neural networks.

[0119] Due to factors such as non-ideal target structure, detection errors, image noise, and complex nonlinear attitude, the attitude angles of the refueling cone sleeve obtained in step three are prone to significant numerical instability. Therefore, this step employs an end-to-end regression method based on neural networks. A regression model is trained on a supervised dataset containing a large number of known attitude samples to automatically learn the nonlinear mapping relationship between image observation features and spatial attitude angles (pitch, yaw), thereby improving the accuracy and robustness of attitude estimation.

[0120] A regression method based on neural network learning of the mapping relationship between image features and cone sleeve posture is used to construct a multilayer perceptron (MLP) model. The posture parameters of the refueling cone sleeve to be tested, which are fixed on the robotic arm and obtained in step two, are used to train the regression model to estimate the predicted posture angle.

[0121] like Figure 8 As shown, the Multilayer Perceptron (MLP) model contains an input layer of dimension 10, two hidden layers with 64 neurons each, a ReLU activation function, and an output layer of dimension 2, regressing two pose angles respectively. The input variable of the model is a 10-dimensional feature vector, which is as follows:

[0122] The width and height of the detection frame of the inner and outer circles of the fuel filling cone sleeve to be tested in the image are used. and the area of ​​the outer circular detection frame Aspect Ratio of Outer Circular Target Boundary After normalizing the above geometric features and A 10-dimensional parameter dataset, i.e., a 10-dimensional feature vector, is constructed. These features originate from the object detection box attributes obtained by processing the image of the inner and outer circles of the fuel filler cone sleeve in step two using the object detection model. The 10-dimensional feature vector is then input into the neural network to enhance the scale consistency and training stability of the features.

[0123] A multilayer perceptron is a typical feedforward neural network structure, consisting of multiple fully connected layers stacked sequentially. It is primarily used for modeling and fitting nonlinear functions, and its mathematical expression is as follows:

[0124] (17)

[0125] in For the input feature vector, For the first Layer weight matrix and bias vector Nonlinear activation function This is the output feature vector. It is a 10-dimensional feature vector, containing and , Output attitude angle in 2D These two pose angles are normalized to the interval [-1, 1] during training to improve the numerical stability of the model, and then denormalized to angle values ​​during the inference phase. The ReLU activation function has the following form:

[0126] (18)

[0127] The ReLU (Rectified Linear Unit) activation function primarily introduces nonlinear modeling capabilities into neural networks, enabling them to fit more complex nonlinear mappings. Compared to traditional Sigmoid or Tanh functions, ReLU offers better numerical stability and convergence speed, effectively mitigating the vanishing gradient problem. The embodiment described uses ReLU as the activation function for each hidden layer, improving the convergence efficiency and model expressiveness of the multilayer perceptron in pose angle regression.

[0128] To measure the error between the predicted pose angle and the angle in the true label, mean squared error was used as the regression loss function during training.

[0129] (19)

[0130] This loss function drives the network to continuously optimize during gradient descent, causing the predicted results to gradually approach the true values. This yields a spatial attitude calculation model for the refueling cone. By inputting an image of the cone captured by an industrial camera into the system, the model can output the predicted attitude angles. .

[0131] Step 5: Using a combination of planar fitting and circular fitting, geometrically model the outer circular surface of the conical sleeve to obtain the spatial position of its bottom center in the three-dimensional coordinate system.

[0132] S51: Preprocessing point cloud data

[0133] The robotic arm is controlled to position the refueling cone sleeve under test in different positions and postures, and multiple sets of point cloud data are collected.

[0134] Because the raw point cloud data acquired by LiDAR 1 often contains many background points, noise points, and dynamic interference factors, a statistical outlier removal algorithm is used to remove isolated noise points from the LiDAR point cloud to ensure subsequent identification and positioning accuracy. Statistical outlier removal is a commonly used point cloud denoising method that determines whether a point is an outlier based on local statistical features. For each point in the point cloud... Find the k nearest points to it, and calculate the distance from these points to it. geometric distance , ,..., And calculate the average distance:

[0135] (20)

[0136] Then calculate the average distance of all points. and standard deviation Set threshold When the following conditions are met:

[0137] (twenty one)

[0138] If the point is not found in the point cloud, it is retained; otherwise, it is removed. This method can effectively remove outliers in the point cloud caused by occlusion, sensor errors, etc., while preserving the continuity of the local structure.

[0139] All preprocessed point cloud data were labeled using the open-source CloudCompare point cloud annotation platform, and the labels were exported. The label format is as follows: Where 0 is the serial number of the object to be detected, representing the outer circle of the conical sleeve. The outer diameter of the oil filling cone sleeve to be tested The spatial coordinates of the points For the first Point cloud density at each point.

[0140] S52: Identify and segment the refueling cone sleeve to be tested

[0141] Leveraging the advantages of convolutional neural networks in target recognition, a PointNet++ deep learning network is established to train a point cloud segmentation neural network model. The model's sampling layer uses Farthest Point Sampling (FPS), the grouping layer uses Ball Query or K-Nearest Neighbors (K-NN), and the feature propagation layer uses Inverse Distance Weighted Interpolation combined with Skip Connection. The collected point cloud data and the corresponding labels exported in step S421 are input into the established point cloud segmentation neural network model to train the detection model for the refueling cone sleeve. The point cloud belonging to the outer circle of the refueling cone sleeve is segmented from other point clouds, resulting in a point cloud segmentation neural network model that can identify the point cloud of the outer circle of the refueling cone sleeve and segment it from other point clouds.

[0142] Input the point cloud data scanned by the LiDAR into the pre-trained point cloud segmentation neural network model, perform semantic segmentation on the point cloud data, and segment out the point cloud set belonging to the outer circle of the refueling cone sleeve to be tested.

[0143] S53: Because the refueling cone sleeve has arbitrary orientations in space, its bottom surface is not always aligned with the ground or the lidar reference axis, making traditional methods based on minimum Z or height thresholds difficult to adapt. Therefore, a three-dimensional plane fitting algorithm based on the least squares method is adopted. The point cloud of the segmented outer circle of the refueling cone sleeve is subjected to plane fitting and then circle fitting to estimate the position of the center of the outer circle of the refueling cone sleeve, which represents the spatial position of the refueling cone sleeve.

[0144] Suppose that the point cloud segmented from S52, belonging to the outer circle of the refueling cone sleeve to be measured, contains N three-dimensional points. Its center of mass is:

[0145] (twenty two)

[0146] Construct the covariance matrix:

[0147] (twenty three)

[0148] For covariance matrix Perform eigenvalue decomposition and extract the eigenvector corresponding to the smallest eigenvalue. As the normal vector of the fitting plane, the plane containing the base of the cone is thus determined:

[0149] (twenty four)

[0150] To map 3D points to the planar coordinate system required for fitting a 2D circle, a set of orthogonal bases needs to be established on the fitting plane. satisfy

[0151] (25)

[0152] Select the centroid of the point cloud Let the origin of the local coordinate system be an arbitrary unit vector, and let its direction be orthogonal to the normal vector. Then calculate:

[0153] (26)

[0154] get Two orthogonal direction vectors are used to construct a local two-dimensional coordinate system.

[0155] S54: Project the point cloud onto a plane and perform circular fitting.

[0156] After constructing the local two-dimensional coordinate system, project all points onto this coordinate system:

[0157] (27)

[0158] (28)

[0159] This yields N projection points. The resulting two-dimensional point set is used for subsequent circular fitting in a two-dimensional plane using the RANSAC method.

[0160] In a two-dimensional plane, assume the center of the torus is... , radius is Then each projection point satisfies:

[0161] (29)

[0162] Substitute all points and construct a linear least squares objective function:

[0163] (30)

[0164] when When the minimum value is obtained, the coordinates of the fitted circle center are obtained. and radius This allows us to calculate the actual coordinates of the center of the circle in three-dimensional space.

[0165] (31)

[0166] in This refers to the coordinates of the outer circle center of the fueling cone sleeve in the lidar coordinate system.

[0167] Step Six: Utilize the transformation matrix obtained in Step One during the camera-LiDAR joint calibration. Transform the spatial coordinates of the refueling cone sleeve to be tested into the camera coordinate system, and then use the optimized spatial attitude angles obtained in step four. And the obtained position information of the refueling cone sleeve under test in the camera coordinate system. Thus, the pose information of the target cone sleeve in the camera coordinate system is obtained. The pose measurement of the target cone sleeve in three-dimensional space was completed.

[0168] The above embodiment acquires image and point cloud information through spatial interaction with the robotic arm housing the refueling cone. First, the intrinsic parameters of the industrial camera are calibrated to obtain the camera's intrinsic parameter matrix and distortion coefficients, facilitating subsequent image processing. Next, spatiotemporal joint calibration is performed on the industrial camera and LiDAR to establish a precise transformation relationship between their coordinate systems, thus unifying the data in a unified coordinate system. Finally, hand-eye calibration is performed on the industrial camera and the robotic arm used to fix the cone, unifying the camera coordinate system with the robot coordinate system, facilitating subsequent cone spatial attitude calculation. Then, images of the cone in different poses are captured by the industrial camera. A neural network target recognition algorithm deployed on the processing unit identifies the inner and outer circles of the target cone in the images and derives its geometric feature parameters such as image coordinates and target size. Combined with the existing attitude parameters of the cone fixed on the robotic arm, a regression model is trained to estimate the attitude angle. Further, point cloud data is acquired using the LiDAR, and the cone target is segmented using deep learning methods to extract its spatial position information. Finally, the cone's position and attitude information are unified in the camera coordinate system and output.

[0169] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style of the specification is merely for clarity. Those skilled in the art should regard the specification as a whole, and the technical solutions in the embodiments can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for measuring the pose of a refueling cone based on radar and camera fusion, characterized in that, Includes the following steps: Step 1: Collect multiple sets of images and point cloud information using an industrial camera (2) and a lidar (1); perform intrinsic parameter calibration on the industrial camera (2) to obtain the intrinsic parameter matrix and distortion coefficients of the industrial camera (2); and perform spatiotemporal joint calibration on the industrial camera (2) and the lidar (1) to transform the spatial coordinates of the lidar point cloud into the camera coordinate system; establish a pairing between each set of images and pose data, and use the hand-eye calibration method to calibrate the pose of the robotic arm device in the camera coordinate system to obtain the transformation matrix from the camera coordinate system to the robotic arm coordinate system, thus completing the spatiotemporal joint calibration between the lidar, the industrial camera, and the robotic arm device. Step 2: Set the refueling cone sleeve to be tested at the end of the robotic arm, so that the end of the robotic arm device (5) can collect multiple sets of cone sleeve images in different poses; Using the transformation matrix from the camera coordinate system to the robotic arm coordinate system obtained in step one, the attitude angle information of the robotic arm end corresponding to the cone sleeve image is transformed into the camera coordinate system; the cone sleeve image corresponding to the attitude angle is input into the pre-trained neural network-based target detection model to identify the inner and outer circular surfaces of the cone sleeve, thereby obtaining the coordinates and spatial projection information of the inner and outer circular surfaces of the refueling cone sleeve to be tested; Step 3: Based on the function mapping relationship between the coordinates and attitude angles of the refueling cone sleeve under test in the image coordinate system obtained in Step 2, identify the inner and outer circle targets of the refueling cone sleeve under test from the images acquired by the industrial camera, obtain the spatial attitude angle, and extract image features to establish a ten-dimensional image feature dataset; the ten-dimensional image feature dataset includes the center coordinates, target bounding box width and height, image center difference, area and aspect ratio information extracted from the images obtained in Step 2; Step 4: Input the ten-dimensional image feature dataset established in Step 3 into the pre-trained multilayer perceptron model to optimize the spatial attitude angle obtained in Step 3. The model outputs the predicted attitude angle of the refueling cone in space to determine the orientation of the refueling cone. Step 5: Use a point cloud segmentation neural network model to segment the point cloud data set belonging to the outer circle of the refueling cone sleeve to be tested from the lidar point cloud. Perform plane fitting and circle fitting on the segmented outer circle area point cloud, perform geometric modeling on the outer circle surface of the cone sleeve, obtain the spatial coordinates of the center of the bottom surface of the refueling cone sleeve, and thus determine the position of the refueling cone sleeve to be tested. Step Six: Using the transformation matrix from the camera coordinate system to the robotic arm coordinate system obtained in Step One, the spatial coordinates of the refueling cone sleeve to be tested obtained in Step Five are transformed to the camera coordinate system. Thus, based on the optimized spatial pose obtained in Step Four and the position information of the refueling cone sleeve to be tested in the camera coordinate system, the pose information of the refueling cone sleeve to be tested in the camera coordinate system is obtained, and the pose measurement of the target cone sleeve in three-dimensional space is completed.

2. The refueling cone pose measurement method based on radar and camera fusion according to claim 1, characterized in that, Step one specifically includes: Adjust the pose of the calibration board and use an industrial camera (2) and a lidar (1) to collect multiple sets of images and point cloud information; The Zhang Zhengyou calibration method is used to calibrate the intrinsic parameters of the industrial camera. The homography matrix from the calibration plate plane to the image plane is solved. By constraining the homography matrix of multiple images, the intrinsic parameter matrix and distortion coefficients are solved. Then, the intrinsic parameter matrix and distortion coefficients of the industrial camera are optimized by the maximum likelihood estimation method. The industrial camera and lidar are connected to the PTP timing clock. A projection relationship is established between each set of images acquired by the industrial camera and lidar and the point cloud data. The PTP timestamp is parsed, the time synchronization error of the camera and the lidar is calculated, and the transformation relationship between the camera coordinate system and the lidar coordinate system is established. Hand-eye calibration is performed on the industrial camera and robotic arm device. Each set of acquired images and pose data is paired, and the transformation relationship between the camera coordinate system and the robotic arm device is established.

3. The refueling cone pose measurement method based on radar and camera fusion according to claim 1, characterized in that, The ten-dimensional image feature dataset in step three includes the X-coordinates of the centers of the inner and outer circles of the refueling cone sleeve in the camera coordinate system. The Y-coordinates of the centers of the inner and outer circles of the refueling cone sleeve in the camera coordinate system. And the width and height of the inner and outer circular detection boxes of the cone sleeve in the image. Area of ​​outer circular detection frame Aspect Ratio of Outer Circular Target Boundary ; where the area of ​​the outer circular detection frame Aspect Ratio of Outer Circular Target Boundary .

4. The refueling cone pose measurement method based on radar and camera fusion according to claim 1, characterized in that, The establishment of the target detection model in step two includes: acquiring multiple sets of images and their corresponding point cloud information; converting the acquired multiple sets of robotic arm end-effector attitude angle information into the camera coordinate system; and inputting the cone sleeve images corresponding to each attitude angle into the neural network after correction using distortion coefficients to train the cone sleeve detection model for target recognition on the inner and outer circular surfaces of the cone sleeve, thereby obtaining a target detection model that can recognize the inner and outer circular surfaces of the cone sleeve respectively.

5. The refueling cone pose measurement method based on radar and camera fusion according to claim 1, characterized in that, In step four, the multilayer perceptron model uses an end-to-end regression method based on neural networks to optimize the spatial attitude angles obtained in step three. The multilayer perceptron model is constructed based on a regression method that learns the mapping relationship between image features and conical attitudes through neural networks. It uses the ReLU activation function and the mean squared error as the regression loss function. The ten-dimensional image feature dataset established in step three is used as input to perform regression training for the two attitude angles.

6. The refueling cone pose measurement method based on radar and camera fusion according to claim 1, characterized in that, Step five specifically includes: The robotic arm is controlled to position the refueling cone sleeve under test in different poses, and multiple sets of point cloud data are collected. The statistical outlier removal algorithm is used for noise reduction and labeling. The pre-processed point cloud data and its labels are input into a pre-trained point cloud segmentation neural network model to perform semantic segmentation on the point cloud data, and segment out the point cloud data set belonging to the outer circle of the refueling cone sleeve under test. Then, the least squares method of three-dimensional plane fitting algorithm is used to perform plane fitting on the point cloud of the segmented part belonging to the outer circle of the refueling cone sleeve, and then perform circular fitting to obtain the spatial coordinates of the center of the outer circle of the bottom surface of the refueling cone sleeve, and use it to represent the spatial position of the refueling cone sleeve.

7. A refueling cone pose measurement system based on radar and camera fusion, characterized in that, It includes a lidar (1), an industrial camera (2), a standard calibration board (3), a processing unit (4), and a robotic arm device (5); the lidar (1), the industrial camera (2), and the robotic arm device (5) are all connected to the processing unit (4) via lines; The lidar is used to acquire point cloud information of the target cone sleeve under test. The industrial camera (2) is used to acquire images of the target cone sleeve under test. The robotic arm device (5) has a refueling cone sleeve under test at its end. It is used to simulate the relative motion during the refueling process by adjusting the pose of the refueling cone sleeve at the end and transmit the pose of the cone sleeve to the processing unit (4). The processing unit (4) receives information from the lidar (1), the industrial camera (2) and the robotic arm device (5), and performs image processing, model training, pose regression, point cloud segmentation and pose estimation on the received information to obtain the pose measurement of the target cone sleeve in three-dimensional space.

8. The refueling cone pose measurement system based on radar and camera fusion according to claim 7, characterized in that, The processing unit (4) includes a calibration module, an image target detection module, an image feature construction module, a pose regression module, a point cloud target segmentation module, a point cloud circular fitting module, and a pose output module. The calibration module is used to collect multiple sets of images and point cloud information, perform intrinsic parameter calibration on the industrial camera, obtain the intrinsic parameter matrix and distortion coefficient of the industrial camera (2), and perform spatiotemporal joint calibration on the industrial camera (2) and the lidar (1), transforming the spatial coordinates of the lidar point cloud into the camera coordinate system; pairing each set of images and pose data is established, and the pose of the robotic arm device is calibrated in the camera coordinate system using the hand-eye calibration method, thereby obtaining the transformation matrix from the camera coordinate system to the robotic arm coordinate system; The image target detection module is used to transform the end-effector attitude angle information of the robotic arm corresponding to the cone sleeve image into the camera coordinate system based on the transformation matrix from the camera coordinate system to the robotic arm coordinate system obtained by the calibration module. The cone sleeve image corresponding to the attitude angle is input into a pre-trained neural network-based target detection model to identify the inner and outer circular surfaces of the cone sleeve, thereby obtaining the coordinates and spatial projection information of the inner and outer circular surfaces of the refueling cone sleeve to be tested. The image feature construction module is used to obtain the spatial attitude angle based on the function mapping relationship between the coordinates and attitude angle of the refueling cone sleeve under test in the image coordinate system obtained by the image target detection module, identify the inner and outer circle targets of the refueling cone sleeve under test from the image acquired by the industrial camera, and extract image features, including center coordinates, target bounding box width and height, image center difference, area and aspect ratio. The attitude regression module is used to optimize the spatial attitude angles obtained by the image feature construction module by inputting the image features from the image feature construction module into the pre-trained multilayer perceptron model, and to determine the orientation of the refueling cone in space by outputting the predicted attitude angle of the refueling cone in space through the model. The point cloud target segmentation module uses a point cloud segmentation neural network to segment the point cloud data set of the outer circle of the refueling cone sleeve to be tested from the LiDAR point cloud; it performs plane fitting and circle fitting on the segmented outer circle area point cloud, performs geometric modeling on the outer circle surface of the cone sleeve, and obtains the spatial coordinates of the center of the bottom surface of the refueling cone sleeve. The pose output module is used to transform the spatial coordinates obtained by the point cloud target segmentation module to the camera coordinate system using the transformation matrix of the calibration module. In this way, based on the spatial attitude angle obtained by the attitude regression module and the position information of the refueling cone under test in the camera coordinate system, the pose information of the refueling cone under test in the camera coordinate system can be obtained.

9. The refueling cone pose measurement system based on radar and camera fusion according to claim 7, characterized in that, The industrial camera and lidar have the same acquisition frequency and synchronized data timestamps, enabling joint measurement of the position and attitude of the refueling cone sleeve under test.

10. The refueling cone pose measurement system based on radar and camera fusion according to claim 7, characterized in that, The calibration plate (3) is set within the field of view of the industrial camera and the lidar, and is used to realize the internal parameter calibration of the industrial camera and the unified calibration of the spatial coordinates of the lidar and the camera.

Images