Method and system for multi-view three-dimensional reconstruction and flexible docking based on vision and force sensing
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-04-28
- Publication Date
- 2026-08-11
Smart Images
Figure CN116524128B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of automatic docking technology, and more specifically, relates to a method and system for multi-view three-dimensional reconstruction and flexible docking based on vision and force perception. Background Technology
[0002] Charging port docking is a primary method for robots to automatically charge new energy vehicles. In fixed-position docking, the vehicle needs to be parked in a precise location to achieve automatic charging. However, this method has high requirements for parking accuracy, resulting in poor adaptability.
[0003] When using vision for localization, the charging port in the RGB image is not a regular black shape, and the features of the area around the charging port are not obvious, making it difficult to directly segment the charging port pixels by color. Furthermore, when the vision sensor acquires images of the charging port, due to factors such as lighting or camera performance, some areas may lack depth information. The completeness of depth information affects the accuracy of the docking pose; missing crucial depth information can lead to docking failure. In addition to the charging port, the depth information acquired by the vision sensor also includes a lot of irrelevant surrounding information, resulting in many unnecessary calculations during algorithm processing. After the robotic arm's end effector reaches the docking pose acquired visually, the camera is too close to the charging port surface to perceive the pose deviation visually. If docking proceeds along the originally planned path, the pose deviation can cause the robotic arm's end effector to squeeze against the charging port, potentially causing hardware damage. Controlling the robotic arm to return and re-acquire the docking pose visually requires a significant amount of time.
[0004] It is evident that existing technologies suffer from technical problems such as missing depth information, deviations in the docking pose obtained visually, and low docking success rates. Summary of the Invention
[0005] In response to the above-mentioned defects or improvement needs of existing technologies, this invention provides a multi-view three-dimensional reconstruction and flexible docking method and system based on vision and force perception, thereby solving the technical problems of missing depth information, deviation of docking pose obtained by vision, and low docking success rate in existing technologies.
[0006] To achieve the above objectives, according to one aspect of the present invention, a multi-view 3D reconstruction and flexible docking method based on vision and force perception is provided, comprising the following steps:
[0007] (1) Acquire the first RGB image and the first depth image of the robotic arm at the initial position, detect the position of the interface in the first RGB image, and calculate the coordinates of the interface in the world coordinate system by combining the depth value of the interface center in the first depth image;
[0008] (2) The distance between the robotic arm and the interface in the world coordinate system is a preset value. The second RGB image and the second depth image of the robotic arm at this location are acquired from multiple perspectives. The position of the interface in the second RGB image under each perspective is detected. The detection results under each perspective are expanded by multiple times to obtain the region of interest under each perspective. The region of interest under each perspective is mapped to the second depth image under the same perspective and then three-dimensional reconstruction is performed to obtain the reconstructed point cloud. The interface plane is extracted from the reconstructed point cloud, thereby generating the docking pose.
[0009] (3) The robotic arm moves to the docking position and monitors the contact force during the docking process. When the contact force is not within the set range, the position of the robotic arm end is adjusted to achieve flexible docking.
[0010] Furthermore, in step (2), the number of second RGB images of the robotic arm at that location acquired from N perspectives is N, and the number of second depth images of the robotic arm at that location acquired from N perspectives is N, where N is greater than or equal to 2. Acquiring images from multiple perspectives can avoid feature loss caused by light pollution.
[0011] Furthermore, the detection of the interface position in the second RGB image from each viewpoint uses a deep learning-based object detection method, specifically:
[0012] Input the second RGB image from each viewpoint into the position detection model to obtain the position of the interface in the second RGB image from each viewpoint;
[0013] The position detection model includes an input layer, a backbone neural network, a neck neural network, and a head neural network. The position detection model is trained in the following manner:
[0014] A training set was formed by collecting RGB sample images from different scenes and perspectives, and the positions of the interfaces in the RGB sample images were marked.
[0015] The training set is input into the location detection model. The input layer performs data augmentation on the RGB sample images. The backbone neural network extracts features from the images output by the input layer. The neck neural network fuses the features extracted by the backbone neural network. The head neural network uses the fused features to output the predicted location of the interface. The location detection model is trained with the goal of minimizing the error between the predicted location of the interface and the marked location of the interface. The training continues until convergence, resulting in a well-trained location detection model.
[0016] Furthermore, the method for detecting the interface position in the second RGB image under each viewpoint uses a method based on HSV color space segmentation, specifically:
[0017] Collect interface image samples to obtain the HSV space value of the interface, convert the second RGB image from RGB color space to HSV color space from each viewpoint, calculate the minimum bounding rectangle of the HSV space value region of the interface in the second RGB image, and obtain the position of the interface.
[0018] Furthermore, the backbone network includes a CBS module, an ELAN module, and an MP module. The CBS module includes convolutional layers, normalization layers, and activation functions. The ELAN module includes multiple CBS modules, and the MP module includes max pooling and multiple CBS modules. The neck neural network includes a CBS module, an MP module, an ELAN-W module, an upsampling module, a Cat module, and an SPPCSPC module. The difference between the ELAN-W module and the ELAN module is the number of outputs selected. The Cat module is used for feature fusion. The SPPCSPC module obtains a larger receptive field through max pooling. The head neural network includes a residual structure.
[0019] Furthermore, the detection interface uses an HSV color space segmentation method or a deep learning-based object detection method at the position of the first RGB image.
[0020] Furthermore, the three-dimensional reconstruction includes the following steps:
[0021] (21) The depth map set Image composed of the second depth maps from each viewpoint, the region of interest set Region composed of the regions of interest from each viewpoint, the camera pose set Pose used for acquisition from multiple viewpoints, and the coordinates P of the interface in the world coordinate system are combined. w (x w y w , z w ) as input to the improved TSDF algorithm;
[0022] (22) P w A cuboid bounding box containing the interface is constructed around the center, with length, width, and height of L, W, and H respectively. The cuboid bounding box is then divided into multiple voxels.
[0023] (23) Traverse all voxels v, and find the corresponding point P of v in the world coordinate system. v Find P v Coordinate P in the camera coordinate system c It also provides camera intrinsic parameters to calculate the second depth map of the i-th view. i Pixel P in d ;
[0024] (24) Determine P d Is it located in the image? i Region of interest R dIf not, then the signed distance function value of the voxel is sdf(P). v Set ) to 1, return to step (23), if it is, proceed to step (25);
[0025] (25)P d Dot in image i The depth value in P is val(d). c If the distance from the origin of the camera coordinate system is dis(c), then sdf(P) v =val(d) - dis(c), set the cutoff distance to u, and obtain point P using the following formula. v Truncation of symbolic distance function value tsdf(P) v ):
[0026]
[0027] (26) Define the projected ray from the camera origin and P v Let the angle between the surfaces be θ, and calculate point P. v The weight is w(P) v =cos(θ) / dis(c), repeat steps (23) to (26) until all voxels have been traversed;
[0028] (27) The tsdf(P) under the current perspective v ), w(P v ) and the global truncation symbolic distance function value TSDF(P v ) and global weight W(P v ) fusion, TSDF(P v ) and W(P v All are initialized to 0:
[0029]
[0030] (28) Switch to the depth map, region of interest and camera pose of the next view until the depth map set, region of interest set and camera pose set have been traversed.
[0031] Further, step (3) includes:
[0032] The robotic arm moves to the docking position and monitors the contact force during the docking process;
[0033] When the end of the robotic arm first enters the docking interface, the end of the robotic arm is controlled to make an initial probe along the z-axis of the docking pose. If the contact force is less than or equal to the maximum contact force of the initial probe, a second probe is made. If the contact force is greater than the maximum contact force of the initial probe, the translation amount of the end of the robotic arm is adjusted. If the contact force after adjustment is greater than the maximum contact force of the initial probe, step (2) is entered. If the contact force after adjustment is less than or equal to the maximum contact force of the initial probe, a second probe is made.
[0034] After the initial test is normal, the docking point at the end of the robotic arm continues to conduct a second test along the z-axis of the docking pose. If the contact force during the second test is less than or equal to the maximum contact force during the second test, the docking is performed using the accurate pose corresponding to this contact force. If the contact force during the second test is greater than the maximum contact force during the second test, the rotation of the end of the robotic arm is adjusted. If the contact force after adjustment is greater than the maximum contact force during the second test, then proceed to step (2). If the contact force after adjustment is less than or equal to the maximum contact force during the second test, the docking is performed using the accurate pose corresponding to this contact force.
[0035] When using accurate pose docking, if the contact force is greater than the maximum contact force for docking, proceed to step (2). If the step error between the actual step displacement at the docking point of the robotic arm end and the given step displacement is greater than the threshold, adjust the step value at the docking point of the robotic arm end. When the contact force is within the set range, the step error is less than or equal to the threshold, and the robotic arm end is connected to the docking interface, the docking is successful.
[0036] Furthermore, the method for adjusting the pose of the robotic arm's end effector is as follows:
[0037] When the contact force is outside the set range, the position of the robotic arm's end effector is adjusted through the following actions.
[0038]
[0039] ε = max(1 - p × t, 0)
[0040] Where t represents the number of adjustment failures, p is the adjustable hyperparameter, ε is the dynamic adjustment value, the priority adjustment pool is the preset action, random means to generate random action, rand(1) means to generate a random number, and the action is the translation amount or rotation amount of the robotic arm end effector.
[0041] Furthermore, if the interface is a charging port, then the end of the robotic arm is a charging head; if the interface is a water inlet, then the end of the robotic arm is a water supply pipe outlet; if the interface is a fuel inlet, then the end of the robotic arm is a fuel supply pipe outlet; and if the interface is an air inlet, then the end of the robotic arm is an air supply pipe outlet.
[0042] According to another aspect of the present invention, a multi-view three-dimensional reconstruction and flexible docking system based on vision and force is provided, comprising: a visual positioning module and a flexible docking module, wherein the visual positioning module includes a preliminary positioning module and a secondary positioning module;
[0043] The preliminary positioning module is used to acquire the first RGB image and the first depth image of the robotic arm at the initial position, detect the position of the interface in the first RGB image, and calculate the coordinates of the interface in the world coordinate system by combining the depth value of the interface center in the first depth image.
[0044] The secondary positioning module is used to move the robotic arm from its initial position to a distance of a preset value from the coordinates of the docking interface in the world coordinate system. It acquires the second RGB image and the second depth image of the robotic arm at that location from multiple perspectives, detects the position of the docking interface in the second RGB image under each perspective, expands the detection results under each perspective by multiple times to obtain the region of interest under each perspective, maps the region of interest under each perspective to the second depth image under the same perspective, and then performs three-dimensional reconstruction to obtain the reconstructed point cloud. The docking interface plane is extracted from the reconstructed point cloud, thereby generating the docking pose.
[0045] The flexible docking module is used to move the robotic arm to the docking position, monitor the contact force during the docking process, and adjust the position of the robotic arm end when the contact force is not within the set range to achieve flexible docking.
[0046] According to another aspect of the present invention, an application of a multi-view three-dimensional reconstruction and flexible docking method based on vision and force perception is provided. The method is a flexible docking method, which is applied to charging, refueling, water filling and gas filling. When applied to charging, the charging head at the end of the robotic arm docks with the charging port. When applied to refueling, the refueling pipe outlet at the end of the robotic arm docks with the refueling port. When applied to water filling, the water filling pipe outlet at the end of the robotic arm docks with the water inlet. When applied to gas filling, the gas filling pipe outlet at the end of the robotic arm docks with the gas inlet.
[0047] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects:
[0048] (1) The visual positioning of this invention is divided into two stages: preliminary positioning and secondary positioning. In the preliminary positioning stage, the position of the interface is detected in the RGB image, and then the coordinates of the interface in the world coordinate system are initially calculated using the depth information of the interface center. In the secondary positioning stage, RGB images and depth maps are acquired from multiple perspectives. Acquiring images from multiple perspectives can avoid feature loss caused by light pollution. After detecting the position of the interface in the RGB image, the detection result is expanded into a region of interest and mapped onto the depth map. Then, local 3D reconstruction is performed on the region of interest, which solves the problem of missing depth information when directly acquiring point clouds and filters out redundant point clouds in non-regions of interest, thus improving computational efficiency. During the docking process, the contact force during the docking process is monitored. When the contact force is abnormal, the pose of the end effector of the robotic arm is adjusted, thereby achieving flexible docking. The multi-view 3D reconstruction and flexible docking method based on vision and force perception can acquire the complete point cloud and accurate pose of the interface, improve the success rate of docking tasks and the compliance of the robotic arm, and ensure the safety of the docking process.
[0049] (2) Both preliminary and secondary localization require detecting the position of the interface in the RGB image. This invention provides several methods for detecting the position. Among them, the deep learning-based target detection method detects the pixel position of the interface in the RGB image, solving the problems of unclear interface features and the large computational burden of manually designed features, and has good adaptability to different environments. The HSV color space segmentation method has good detection performance when the color difference between the interface and its surrounding area is obvious.
[0050] (3) Although depth cameras can obtain point cloud data through their SDK, the point cloud acquired by the SDK is the entire point cloud collected from the entire viewpoint, which includes a lot of useless information. Directly processing point clouds from multiple views would lead to a serious waste of computing resources and poor real-time performance. In real-world environments, factors such as lighting can affect the completeness of the acquired depth information, thus affecting the accuracy of pose acquisition. This invention performs 3D reconstruction of the region of interest to obtain the complete point cloud of the charging port. This solves both the problem of missing depth information when depth cameras acquire images and the problem of low real-time performance caused by excessive computing power when using global point clouds. The Region of Interest (ROI) 3D reconstruction algorithm improves the main step (24) of TSDF by pruning. For the TSDF algorithm, after specifying the bounding box L, W, and H, it calculates the TSDF value of all voxels within the bounding box. The reconstruction result includes not only the point cloud of the ROI but also redundant point clouds of the non-ROI, which may result in the plane obtained when segmenting the charging port plane being an invalid plane of the non-ROI. For the ROI 3D reconstruction algorithm, due to the pruning improvement, most of the non-ROI point clouds are filtered out, and the reconstructed ROI point cloud retains the complete point cloud of the charging port, reducing the invalid calculation of non-key points, and using the ROI point cloud to segment the charging port plane.
[0051] (4) During flexible docking, this invention detects contact force for collision protection, preventing equipment squeezing and damage caused by docking posture deviations. Correction is achieved by fine-tuning the posture of the robotic arm's end effector, thus refining the docking posture and ensuring accurate entry of the end effector into the docking interface. During docking, an accurate docking posture is obtained through initial and secondary probes. Position and force detection are used as criteria for judging the success of the stepping process when docking with the accurate posture, thereby avoiding equipment damage and improving the docking success rate.
[0052] (5) When an abnormal contact force is detected during the trial, the flexible docking will reduce its reliance on the priority adjustment pool as the number of failed fine-tuning attempts increases, and will instead tend to use a random exploration approach to find the accurate docking pose. When fine-tuning and correcting the docking pose, an ε-greedy-based approach is used. Compared to random adjustment, this approach utilizes the successful adjustment scheme when correction is successful, thus improving the efficiency of fine-tuning.
[0053] (6) The present invention has a variety of applications. It can be applied to automatic charging of new energy vehicles, and can also be used to automatically connect water pipes to vehicles with water inlets to achieve automatic water filling. It can also be used to automatically connect refueling interfaces to cars, airplanes and rail transit that need refueling to achieve automatic refueling. It can also be used to automatically connect gas inlets to equipment that needs gas filling to achieve automatic gas filling. Attached Figure Description
[0054] Figure 1 This is a flowchart of a multi-view 3D reconstruction and flexible docking method based on vision and force perception provided in an embodiment of the present invention;
[0055] Figure 2 This is a functional diagram of the preliminary positioning module provided in an embodiment of the present invention;
[0056] Figure 3 This is a functional schematic diagram of the secondary positioning module provided in an embodiment of the present invention;
[0057] Figure 4 This is a flowchart of three-dimensional reconstruction of the region of interest provided in an embodiment of the present invention;
[0058] Figure 5 Image (a) is a TSDF 3D reconstruction effect diagram provided in an embodiment of the present invention;
[0059] Figure 5 Image (b) is a 3D reconstruction effect of the region of interest provided in an embodiment of the present invention;
[0060] Figure 6 This is a flowchart of the flexible docking process provided in an embodiment of the present invention;
[0061] Figure 7 This is a flowchart of the flexible fine-tuning process provided in the embodiments of the present invention. Detailed Implementation
[0062] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.
[0063] like Figure 1 As shown, a multi-view 3D reconstruction and flexible docking method based on vision and force perception includes the following steps:
[0064] (1) Acquire the first RGB image and the first depth image of the robotic arm at the initial position, detect the position of the interface in the first RGB image, and calculate the coordinates of the interface in the world coordinate system by combining the depth value of the interface center in the first depth image;
[0065] (2) The distance between the robotic arm and the interface in the world coordinate system is a preset value. The second RGB image and the second depth image of the robotic arm at this location are acquired from multiple perspectives. The position of the interface in the second RGB image under each perspective is detected. The detection results under each perspective are expanded by multiple times to obtain the region of interest under each perspective. The region of interest under each perspective is mapped to the second depth image under the same perspective and then three-dimensional reconstruction is performed to obtain the reconstructed point cloud. The interface plane is extracted from the reconstructed point cloud, thereby generating the docking pose.
[0066] (3) The robotic arm moves to the docking position and monitors the contact force during the docking process. When the contact force is not within the set range, the position of the robotic arm end is adjusted to achieve flexible docking.
[0067] During secondary positioning, the robotic arm moves from its initial position to a distance S relative to the interface in the world coordinate system. The left and right coordinates of this location are used as the start and end points of the secondary positioning. When planning the motion between these start and end points, the robotic arm acquires a second RGB image and a second depth image at that location from multiple perspectives.
[0068] Example 1
[0069] Taking intelligent charging of new energy vehicles as an example, the present invention will be described in detail.
[0070] The charging port for new energy vehicles is not completely perpendicular to the ground, but rather at an angle to the ground. In the docking task of Embodiment 1 of the present invention,
[0071] To more clearly describe the docking task of the new energy vehicle charging port, a coordinate system is established for the new energy charging port and its pose is described. The upper surface plane of the charging port is defined as a plane. α The connection surface between the charging port and the bracket is defined as plane. β ,have:
[0072] plane α ||plane β
[0073] Let the center point P(x, y, z) of the charging port be defined as the origin, and the surface normal vector of the charging port be defined as... The direction is the y-axis, perpendicular to the y-axis and parallel to the ground plane. playground Let the vector be the x-axis. According to the right-hand rule, the direction of the z-axis is obtained, that is:
[0074]
[0075] For the charging port docking task, a docking pose needs to be generated for the charging head docking. The three-dimensional coordinates of the docking pose are the center point P(x, y, z) of the charging port, pointing inward along the normal direction of the charging port plane. This pose is described as follows:
[0076]
[0077] Where x, y, and z are the coordinates of the center point P of the charging port. In the docking scenario, ω = 0, which indicates the direction of the top and bottom edges of the charging port. plane parallel to the ground playground .
[0078] Based on the above definition, the charging port docking problem is transformed into the following: Using the acquired RGB image and depth map as input, the charging port pose is calculated using a charging port pose estimation method. charge This information is then converted into the docking pose in the world coordinate system, providing a positional basis for docking planning.
[0079] For a charging head with a depth of L = 3.5cm, the standard for successful docking is:
[0080] 1) From the center point P(x, y, z) of the charging port, along the normal line of the charging port plane When the direction enters L, the error is less than 1mm;
[0081] 2) The actual distance between the target center point Q(x′, y′, z′) of the charging head and the center point P(x, y, z) of the charging port in the x, y, and z directions is less than ε. max , where ε max =2mm, that is:
[0082]
[0083] 3) Since the visual sensor cannot determine the docking status when the charging head reaches the charging port plane, meaning the docking posture error cannot be detected by the visual sensor, a force sensor is introduced for determination. Therefore, the third condition for successful docking is a force along the normal direction at F... min and F max Between, where F min =60N, F max =80N, that is:
[0084] F min ≤|F z |≤F max
[0085] A multi-view 3D reconstruction and flexible docking system based on vision and force perception includes a visual positioning module and a flexible docking module. The visual positioning module includes a preliminary positioning module and a secondary positioning module.
[0086] like Figure 2 As shown, the preliminary positioning module is used to collect the RGB image and depth image (visualized as a gray view) of the robotic arm at the initial position as input, detect the position of the charging port in the RGB image, and combine the depth of the depth image at the center point to preliminarily calculate the coordinates of the charging port in the world coordinate system.
[0087] like Figure 3 As shown, the secondary localization module takes RGB images and depth maps acquired from multiple viewpoints as input. First, it uses deep learning to detect the pixel positions of the charging port in the RGB images. To avoid the RGB target detection not fully encompassing the charging port, the detection results are expanded by a factor of N to obtain the region of interest (ROI). Then, the ROI is mapped onto the depth map, which includes the charging port and a small portion of the surrounding plane. Next, 3D reconstruction is performed on the depth map region of the ROI. Then, the plane fitted to the charging port point cloud is separated to obtain the charging port plane equation. Finally, the charging port pose is estimated using the charging port plane equation and the center point. charge And generate the docking pose.
[0088] Both the initial localization module and the secondary localization module need to detect the pixel position of the charging port in the RGB image. Two-dimensional object detection can be achieved using traditional image-based methods or deep learning-based methods. Traditional image-based methods can utilize HSV color space segmentation. The core idea is to acquire the HSV space value of the charging port by collecting image samples, and then calculate its minimum bounding rectangle to obtain the largest object (i.e., the charging port) in the selected HSV interval of the RGB image.
[0089] The target detection method based on the HSV color space has good detection performance when the color difference between the charging port and its surrounding area is obvious in the experimental scenario, and the largest area is the charging port. However, in real-world scenarios, there are many scenarios where the HSV color space of the charging port and its surrounding area is very similar, such as black obstructions and black car bodies. Moreover, this method is greatly affected by lighting and exposure parameters, so the method is not very effective. In this case, the largest plane obtained by segmentation is not the charging port.
[0090] It is evident that charging ports cannot be segmented simply by color; instead, feature detection is required. This can be categorized into traditional feature-based and deep learning-based object detection algorithms. Traditional feature-based algorithms typically extract feature points from specific regions of the image, such as SIFT and Haar, and then classify them using machine learning algorithms like Support Vector Machines (SVM) to achieve object detection. However, these algorithms require extensive manual parameter tuning, have high time complexity, and are difficult to adapt to complex scenarios.
[0091] Deep learning-based object detection methods can be categorized into one-stage and two-stage methods based on whether they use region proposals (RPs, which generate candidate boxes that may contain the target object). The YOLO series and R-CNN series are representative examples of these methods, respectively. One-stage methods do not require region proposal generation, resulting in simpler models and faster detection speeds, but their detection accuracy is slightly lower than that of two-stage networks. Conversely, two-stage networks sacrifice some real-time performance for region proposal generation, thereby achieving higher detection accuracy. Since docking tasks require high real-time performance, and the accurate docking pose in the secondary localization of the charging port ultimately relies on depth information, while RGB image-based charging port detection only plays an auxiliary role, a certain accuracy error can be tolerated in the RGB object detection stage. Therefore, this embodiment of the invention uses a one-stage object detection algorithm to detect the charging port, specifically the latest improved version of the YOLO series, YOLOv7.
[0092] YOLOv7 outperforms most known target detectors in both accuracy and speed in the range of 5 FPS to 160 FPS. It largely inherits from YOLOv5, but is more accurate and about 120% faster.
[0093] The YOLOv7 network structure can be mainly divided into four parts: Input, Backbone, Neck, and Head. The Input network handles preprocessing and data augmentation of the input image; the Backbone network extracts features from the processed image; the Neck network fuses features to obtain large, medium, and small features; and the Head network is the prediction end, using the fused features to output the prediction result. The Backbone network mainly consists of three modules: CBS, ELAN, and MP. CBS (Convolutional Layer), BN (Batch Normalization), and Silu (an activation function) are mainly used to change the number of channels, feature extraction, and downsampling. The ELAN module consists of multiple CBS modules and controls the shortest and longest distances of gradient paths, allowing the network to efficiently learn more features. The MP module consists of MaxPool and multiple CBS modules and is mainly used for downsampling. In the neck neural network, in addition to the CBS and MP modules found in the backbone neural network, it also includes ELAN-W, UpSampling, Cat, and SPPCSPC modules. The ELAN-W module is very similar to the ELAN module in the backbone neural network, differing only in the number of outputs selected; its function is also to make the network learn features more efficiently. UpSampling is an upsampling module; the Cat module is mainly used for feature fusion; and the SPPCSPC module's role is to obtain a larger receptive field through max pooling, enabling it to better adapt to images of different resolutions. The head neural network only contains the REPConv module, which differs somewhat in training and prediction, its ideas being derived from RepVGG. [9] A special residual structure was designed to assist training. This residual structure is uniquely designed so that during prediction, the complex residual structure is equivalent to a 3×3 convolution, thereby reducing the complexity of the network without sacrificing its prediction performance.
[0094] Compared to other algorithms in the YOLO series, the network optimizations in YOLOv7 mainly focus on the following aspects:
[0095] 1) In feature extraction, the proposed multi-branch stacking structure and downsampling structure were used, and the skip connection structure of the model is denser than that of YOLOv5.
[0096] 2) CSP is introduced into the SPP structure to form SPPCSPC modules, thereby expanding the receptive field to adapt to images of different resolutions;
[0097] 3) The RepVGG-based structure introduces RepConv, which reduces the number of network parameters;
[0098] 4) YOLOv7 introduces an adaptive multiple positive sample matching mechanism. During training, each ground truth box can be predicted from multiple anchor boxes. After adjusting the anchor boxes, a predicted box is obtained, and the IOU and class are calculated to find the most suitable anchor box. This mechanism speeds up the model's training efficiency.
[0099] When creating the charging port dataset, it is first necessary to control the robotic arm to move to different poses to acquire RGB images from different angles. Multiple sets of white, red, and black backgrounds are set to simulate vehicles of different colors in real-world scenarios, and different lighting conditions are set to simulate environmental interference. Furthermore, to demonstrate the application of the algorithm in real-world scenarios, in addition to charging ports acquired in the laboratory, images of charging ports from real-world new energy vehicles are also added to the dataset for training.
[0100] When creating the dataset, the open-source tool labelImg is used for annotation. Rectangular boxes are marked at the charging ports in the images, and XML files are generated to store information such as the label category and coordinates as labels.
[0101] Since the target detection used in this invention is supervised learning, it requires a large dataset for training. However, the charging port dataset for this invention requires continuous control of a robotic arm to change its pose for image acquisition, and all generated images need to be labeled, resulting in low efficiency. Therefore, it is necessary to expand the images and labels in the dataset. Expansion methods include rotation, mirroring, scaling, cropping, and blurring.
[0102] In the preliminary localization module, the position of the charging port in the RGB image is first obtained, and the center pixel P of the detection result is then... o (u, v) is used as the initial pose coordinates, and the depth information corresponding to this pixel is obtained through the registered depth map. To avoid missing depth information, the center pixel is expanded into a region of 9 pixels, and the average depth information of the 9 pixels is calculated. (If depth information is missing, ignore this pixel), and set z = Substitute the coordinates P of the charging port in the camera coordinate system into the calculation. c (x c y c , z c The coordinates of the charging port, P, are obtained by transforming the camera's extrinsic parameters to the world coordinate system. w (x w y w , z w This provides a location basis for image acquisition and 3D reconstruction of regions of interest for secondary positioning.
[0103] In the secondary localization module, after inputting the RGB image and depth map acquired from multiple perspectives, the pixel position of the charging port in the RGB image is first obtained, and the precise pose of the charging port is obtained through the following three steps:
[0104] (1) Point cloud segmentation of the charging port area
[0105] When detecting the location of the charging port in RGB images acquired from multiple views, the bounding box of the detection result may not completely encompass the charging port. To prevent the critical portion of the charging port from being missed when guiding point cloud segmentation with the RGB detection results, the bounding box of the detection result needs to be expanded by N times, taking N=1.2, and this region is called the region of interest R. o .
[0106] Since the RGB image and the depth map are already aligned, the region of interest can be mapped onto the depth map, and the mapped region can be defined as the region of interest R in the depth map. d Then R d It contains both the full depth information of the charging port and a small amount of depth information of the charging port connection panel, and the area of the region of interest is approximately N times the area of the charging port. 2 =1.44 times.
[0107] (2) Charging port point cloud completion based on 3D reconstruction of region of interest
[0108] Although the Intel RealSense D435i depth camera can obtain point cloud data through its SDK, the point cloud acquired by the SDK is the entire point cloud collected from the entire viewpoint, which includes a lot of useless information. In the task of this chapter, we only need to process and analyze the region of interest. Directly processing point clouds from multiple views would lead to a serious waste of computing resources and poor real-time performance. In addition, in the real environment of charging new energy vehicles, there will be influencing factors such as lighting, which will reduce the completeness of the depth information acquired, thus affecting the accuracy of pose acquisition.
[0109] To address the aforementioned issues, a 3D reconstruction of the region of interest (ROI) is performed to obtain the complete point cloud of the charging port. However, the common 3D reconstruction method, KinectFusion, is a large-scale reconstruction method, while the ROI targeted by this invention is relatively small. Therefore, this invention uses an improved TSDF (Truncated Signed Distance Function) algorithm. The basic idea of the TSDF algorithm is to divide the 3D space into a series of voxels, each used to record its distance information from the object surface. At each depth map input, the distance value between the voxel and the depth image is calculated and weighted and fused with the previous voxel values. The information in the voxels is updated based on the pixel values in the current depth image and the distance values in the existing voxels, thereby obtaining a more accurate 3D model.
[0110] The docking task only requires 3D reconstruction of the region of interest (ROI) after the charging port expansion. This invention proposes a method for 3D reconstruction of the ROI based on the TSDF algorithm. First, the world coordinate system, the end-effector coordinate system, and the camera coordinate system are defined as O1, O2, O3, O4, O5, O6, O7, O8, O9, ... w {x, y, z}, O e {x, y, z} and O c {x, y, z}, since the camera pose in the world coordinate system needs to be obtained during reconstruction, a homogeneous transformation matrix is used. Describe the camera pose in the world coordinate system, and define a set of depth maps Image = {image1, image2, ..., image...} acquired from n viewpoints. n The corresponding set of regions of interest is Region = {R1, R2, ..., R}. n}, during data acquisition, the camera pose set Pose = {pose1, pose2, ..., pose} n},Right now The elements in the Region are of type R(x, y, w, h), where x and y represent the pixel coordinates of the top-left vertex of the Region of Interest, and w and h represent the length and width of the Region of Interest, respectively. The flow of the Region of Interest 3D reconstruction algorithm is as follows: Figure 4 As shown.
[0111] according to Figure 4 The algorithm flow for 3D reconstruction of interest is as follows:
[0112] 1) Combine the depth map set Image, the region of interest set Region, the camera pose set Pose, and the initial localization-obtained charging port center coordinates P. w (x w y w , z w ) as input to the improved TSDF algorithm;
[0113] 2) With P w Construct a cuboid bounding box with length L, width W, and height H, centered on the target location, ensuring it contains the charging port to be reconstructed. Divide the cuboid bounding box into several voxels, each with a side length L. v The cube, obviously, the cuboid is divided into... Individual factors;
[0114] 3) Traverse all voxels v, and find the corresponding point P of v in the world coordinate system. v They can be transformed separately. Find P v Coordinate P in the camera coordinate system c It provides camera intrinsic parameters to calculate its image i Pixel P in d ;
[0115] 4) Determine P d Is it located in the depth map image? i Region of interest R d If it is not present, it proves that the point of voxel v in space is not part of the charging port point cloud. Then directly calculate the voxel's SDF value (Signed Distance Function) sdf(P). v Set the value to 1 and return to step 3; otherwise, continue processing.
[0116] 5) By P d Dot in image i The depth value in P is val(d). c The distance from the origin of the camera coordinate system is dis(c), which can be calculated as sdf(P). v Given val(d) - dis(c), and setting the cutoff distance to u, point P can be obtained using the following formula: v Truncating the symbolic distance function value:
[0117]
[0118] 6) Define the projected ray from the camera origin and P v Let θ be the angle between the surfaces on which the point P lies. v The weight is w(P) v =cos(θ) / dis(c), repeat steps 3) to 6) until all voxels have been traversed;
[0119] 7) Set the tsdf(P) of the current frame. v ), w(P v ) and global TSDF (P v ) and W(P vThe fusion method is shown in the following formula, where the global TSDF(P) v ) and W(P v All are initialized to 0:
[0120]
[0121] 8) If data from multiple perspectives has not been fully traversed, switch to the depth map of the next perspective. i+1 Region of Interest R i+1 and pose acquisition i+1 This continues until all image and pose data have been traversed.
[0122] As can be seen from the above steps, the 3D reconstruction algorithm for the region of interest (ROI) has made pruning improvements to the TSDF algorithm, primarily in step 4). For the TSDF algorithm, such as... Figure 5 As shown in (a), after specifying the cuboid bounding box L, W, and H, the TSDF value of all voxels within the bounding box is calculated. The reconstruction result includes not only the point cloud of the region of interest but also redundant point clouds of the non-region of interest, which may result in the plane obtained during charging port plane segmentation being an invalid plane of the non-region of interest. For the 3D reconstruction algorithm of the region of interest, such as Figure 5 As shown in (b), most of the non-interest region point clouds were filtered out, and the reconstructed interest region point cloud retained the complete point cloud of the charging port, reducing the invalid calculation of non-critical points, and the charging port plane was segmented using the interest region point cloud.
[0123] (3) Charging port pose acquisition
[0124] After completing the key point cloud of the charging port through 3D reconstruction of the region of interest, the large number of point clouds can lead to a large amount of computation. Therefore, downsampling is required. The specific method is to use a voxelized mesh filter. By downsampling the dense point cloud, the computational efficiency can be improved while preserving most of the feature information.
[0125] In the docking task, after obtaining the detection results in the RGB detection phase, only the region of interest (ROI) point cloud, expanded by N times, is processed. The area of the ROI is N times the area of the detection result. 2 =1.44 times, therefore the plane with the largest area in the point cloud is the charging port plane. Thus, the RANSAC method can be used to fit the largest plane as the charging port plane, obtaining the fitted plane α in the camera coordinate system. The equation of α is Ax + By + Cz + D = 0, and the normal vector is... During the docking task, the docking pose should be perpendicular to the charging port surface and facing inwards, unlike the 4-DOF pose that is always vertically downwards during the grasping task. The calculated fitting plane normal vector The direction is uncertain; the solution for the normal vector could be perpendicular to the fitting plane α outwards or perpendicular to α inwards, with the latter indicating the docking pose. The charging port in the image acquired in the camera coordinate system should be aligned with the positive direction of the camera coordinate system's z-axis. If C > 0, it indicates the normal vector... The direction is consistent with the z-axis direction, so no correction is needed; it remains the same. If C < 0, then it represents the normal vector. The direction of is opposite to the z-axis direction, and the normal vector of the fitting plane α should be corrected to .
[0126] The point cloud set P of the charging port, separated by the RANSAC method, can be used to calculate the center coordinates of the charging port. Assume P consists of n points, where the coordinates of each point are P. i (x i y i , z i ), and for any i∈[1,n], we have P i If ∈P, then the center coordinates P of the charging port plane can be calculated using the following formula. o (x o y o , z o ):
[0127]
[0128] Through calculation and P o (x o y o , z o Together, they form the charging port pose in the camera coordinate system. The pose is obtained by transforming the camera extrinsic parameters to the world coordinate system and normalizing the normal vector. in This is the normalized normal vector of the charging port plane.
[0129] However, ROS cannot input only the surface normal vector as the pose; it requires either a translation vector plus Euler angles or a translation vector plus quaternions. Therefore, the normal vector needs to be transformed. Vector to Euler angle conversion usually requires determining two direction vectors, but currently only the normal vector is available. In one direction, the calculation result is uncertain. The direction of the upper and lower edges of the charging port is specified in the docking task. Always parallel to the ground, represented by a vector. Therefore, its pose can be calculated using vector product. First, assume... The estimated vector is pass and Determining a definite direction using the vector product method in for Then through and Find the accurate vector product Right now:
[0130]
[0131] The result calculated using the above formula The rotation matrix of a vector can be calculated as follows:
[0132]
[0133] Finally, the rotation matrix is converted into Euler angles (φ, θ, ψ), and then compared with the center coordinates (x, y) of the charging port. w y w , z w Forming a 6-DOF docking pose d (x w y w , z w ,φ,θ,ψ).
[0134] like Figure 6 As shown, the flexible docking module first acquires the docking pose in the world coordinate system through vision, and then uses the MoveIt function package for trajectory planning to control the charging head at the end of the robotic arm to move to the docking pose. When the charging head first enters the charging port, the robotic arm is first controlled to make an initial probe inward along the z-axis of the docking pose. If the contact force |F| is applied at this time... z |>F fmax (where F) fmax To initially test the maximum contact force and F fmax =30N), indicating a deviation in the docking pose acquired visually. In this case, the translation amount is adjusted via the flexible adjustment module. If the adjustment fails, the robotic arm stops planning and re-acquires and calculates the docking pose; otherwise, it indicates that the translation fine-tuning is normal and docking can continue. After the initial probe is normal, a second probe is performed along the z-axis of the docking pose. If an abnormal contact force (i.e., |F|) is still found during the second probe, the robot will proceed. z |>F smax F smax For the second test of the maximum contact force and F smax =50N), then the rotation amount is adjusted through the flexible adjustment module. If the adjustment fails, the robotic arm stops planning and re-acquires the docking pose through vision. If the adjustment is successful, docking with the accurate pose begins. When docking with the accurate pose, the charging head takes a step Δ each time. Position and force detection are designed as the criteria for judging the success of the stepping process. When the force sensor detects a force |F in the z direction... z |>F maxWhen the value is 80N, it indicates an anomaly in the docking process, posing a risk of equipment damage. Therefore, the planning is stopped immediately, and the robotic arm is controlled to retract to the visual image acquisition point to re-acquire the docking pose. Furthermore, the pose of the end effector in the current state is calculated using the forward kinematics of the robotic arm, thereby calculating the actual step displacement Δz. If |Δ-Δz|>σ, it indicates a large error between the actual step displacement Δz and the given step displacement Δ. In this case, the step value needs to be readjusted for correction, ensuring the charging head steps Δ-Δz along the normal direction, thus ensuring that the displacement of each step meets expectations and the error is less than σ=0.1mm. Since the single step error is controlled within σ=0.1mm, the number of steps in this invention is N=7, and the maximum error of the entire process is σ′=N.σ=0.7mm, which is within the allowable docking error range [0, σ max [Inside.] Upon successful docking, the contact force should satisfy 60N ≤ |F. z The specification is ≤80N. In addition, the final criterion for judging whether the docking is successful is the detection of a change in charging power. If the charging power P>5KW, the docking is considered successful.
[0135] When abnormal contact force is detected during the initial or secondary probe, it indicates that the visually acquired docking pose translational deviation ε t >ε tmax =1mm or rotational deviation ε r >ε rmax =1°, but in most cases the docking pose deviation will not differ much, requiring only slight adjustments. If the visually acquired pose is returned and the docking pose is recalculated, the docking process will be very inefficient. Therefore, a flexible fine-tuning module is needed to adjust the docking pose, such as... Figure 7 As shown, when an abnormal contact force is detected during the trial, the robotic arm returns to the initial trial position and begins fine-tuning. The fine-tuning motion is obtained using the following formula:
[0136]
[0137] After obtaining the corrected pose, the robotic arm is controlled to reach that pose and move linearly Δ along the z-axis of the adjusted docking pose. If the contact force |F| in the z-direction after the movement... z | <F max If the result is positive, the fine-tuning is effective, and docking can continue; otherwise, the fine-tuning is ineffective. Since the docking task does not allow the robotic arm to fine-tune indefinitely, an upper limit C is set for the number of fine-tuning attempts. In this invention, C = 150. If the number of fine-tuning attempts exceeds C, it means that the docking pose deviation generated by vision cannot be corrected through fine-tuning, and the pose needs to be re-acquired through vision; if the upper limit has not been reached, the fine-tuning strategy continues. This invention sets a dynamic ε:
[0138] ε = max(1 - p × t, 0)
[0139] Where t represents the number of fine-tuning failures, and p is an adjustable hyperparameter. The larger the p value, the lower the reliability of the priority adjustment pool. In this invention, p = 0.005. It can be seen that after setting the hyperparameter p, as the number of fine-tuning failures increases, the flexible docking model reduces its dependence on the priority adjustment pool and tends to use a random exploration approach to find the accurate docking pose.
[0140] In addition to its application in the automatic charging of new energy vehicles as described in Example 1, this invention can also be used to automatically connect water supply pipes to vehicles with water inlets for automatic water filling. It can also be used to automatically connect refueling interfaces to cars, airplanes, and rail vehicles requiring refueling for automatic refueling. Furthermore, it can be used to automatically refuel equipment requiring gas. If aircraft require the injection of special liquids, this invention's system can also be used.
[0141] Example 2
[0142] To quantify the accuracy of the docking pose obtained by the deep learning and local 3D reconstruction methods presented in this chapter, comparative experiments were conducted on three methods: SFIT feature extraction + PNP pose estimation, deep learning detection + SDK-generated point cloud, and deep learning detection + local 3D reconstruction point cloud. The overall process for each method involved obtaining the docking pose from an input RGB image and depth information. During the comparative experiments, six experimental scenarios were set for each method: white background (vehicle body), black background (vehicle body), red background (vehicle body), blue background (vehicle body), strong light, and weak light. Each experiment was repeated five times. The main metrics studied in the comparative experiments were the pose translation error and the time consumed in generating the pose. The experimental results are shown in Table 1.
[0143] Table 1. Experimental results of visual positioning of charging port
[0144]
[0145]
[0146] Based on the experimental results in Table 1, the following analysis is made: 1) The method of obtaining docking pose using SFIT feature extraction + PNP pose estimation cannot guarantee that the average translation error ε ≤ ε maxFurthermore, the docking pose calculated under black background and different lighting conditions has greater errors and cannot adapt to all scenarios; 2) In the method of obtaining point clouds using deep learning + SDK, the average error was reduced by 68.7% in black background scenarios, and by 43.6% and 84.0% in strong and weak light scenarios, respectively, with an overall average error reduction of 61.7%. In addition, since the target detection algorithm used in this paper has good real-time performance and can quickly obtain point clouds directly from the camera SDK, the time for obtaining docking pose was also reduced by 26%. However, the docking pose error caused by the missing charging port point cloud in strong light scenarios is still very large; 3) After introducing the method of 3D reconstruction of region of interest to complete the point cloud near the charging port, although some real-time performance was sacrificed, the average error was reduced by 59.1% in strong light scenarios, and the overall average error was reduced by 30.4%, which to some extent solved the problem of excessive pose error in strong light scenarios.
[0147] Example 3
[0148] In the flexible docking experiment, this chapter compares and analyzes three docking methods: no fine-tuning, random fine-tuning, and ε-greedy-based flexible fine-tuning. Because the docking pose obtained through vision does not always have an error ε > ε0. max The error varies each time, so it's necessary to control the variables by controlling the magnitude of the error. This invention primarily studies the adjustment of translational error in its experimental design. In addition to the pose obtained visually, three additional sets of manually set error poses (i.e., adding offsets to the accurate pose) were added. A total of four pose types were set for comparison experiments for each method, as detailed below:
[0149] a) Type 1: The docking pose obtained by the visual positioning method in this chapter;
[0150] b) Type 2: Set the x-direction +1.8mm as the error (maximum average translation error);
[0151] c) Type 3: Set the y-direction setting to -2.8mm as the error (maximum translation error).
[0152] d) Type 4: Set both x-direction -2.8mm and y-direction +2.6mm as errors (combined case).
[0153] Each control experiment was repeated 20 times, and the number of fine-tuning attempts (C) was limited to 150. The experimental results are shown in Table 2.
[0154] Table 2. Experimental results of flexible docking of charging ports
[0155]
[0156] *Note: - indicates that the average docking time is an invalid indicator when the docking success rate is 0%.
[0157] Based on the experimental results in Table 2, the following analysis is made: 1) The method without fine-tuning has a success rate of 45% in pose type 1 acquired visually. Since no pose adjustment is performed, the average docking time is short. Its docking performance depends entirely on the accuracy of the visual positioning algorithm. When a value greater than ε is manually set... max In pose types 2-4 with errors, the docking success rate is 0% because the pose deviation cannot be adjusted. 2) The random fine-tuning method improves the docking success rate by 55.0% for pose type 1 obtained visually, and also improves the docking success rate when the error is greater than ε. max Fine-tuning was performed under certain conditions, achieving a breakthrough in successful docking from scratch. However, this sacrificed some real-time performance and required a high number of fine-tuning operations. 3) The ε-greedy-based flexible fine-tuning method, compared to the non-fine-tuning method, sacrificed some docking time, making it suitable for errors greater than ε. max In this scenario, compared to the random fine-tuning method, this method utilizes the experience of successful fine-tuning, reducing the average number of fine-tuning attempts by 66.2%, increasing the average docking success rate by 6.25%, and reducing the average docking time by 59.1%. Experimental results show that the ε-greedy-based flexible fine-tuning method can improve the docking success rate and robustness of the robotic arm in real-world environments, but fine-tuning attempts require a certain amount of time.
[0158] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A multi-view 3D reconstruction and flexible docking method based on vision and force perception, characterized in that, Includes the following steps: (1) Acquire the first RGB image and the first depth image of the robotic arm at the initial position, detect the position of the interface in the first RGB image, and calculate the coordinates of the interface in the world coordinate system by combining the depth value of the interface center in the first depth image; (2) The distance between the robotic arm and the interface in the world coordinate system is a preset value. The second RGB image and the second depth image of the robotic arm at this location are collected from multiple perspectives. The position of the interface in the second RGB image under each perspective is detected. The detection results under each perspective are expanded by multiple times to obtain the region of interest under each perspective. The region of interest under each perspective is mapped to the second depth image under the same perspective and then three-dimensional reconstruction is performed to obtain the reconstructed point cloud. The interface plane is extracted from the reconstructed point cloud, thereby generating the docking pose. (3) The robotic arm moves to the docking position and monitors the contact force during the docking process. When the contact force is not within the set range, the position of the robotic arm end is adjusted to achieve flexible docking. The three-dimensional reconstruction includes the following steps: (21) A set of depth maps composed of the second depth maps from each viewpoint A set of regions of interest composed of regions of interest from various perspectives. A set of camera poses used for acquisition from multiple perspectives and the coordinates of the interface in the world coordinate system As input to the improved TSDF algorithm; (22) with Construct a cuboid bounding box containing the interface at the center, with length, width, and height of [missing information]. , , The rectangular bounding box is divided into multiple voxels; (23) Traverse all voxels , Corresponding point in the world coordinate system Find coordinates in the camera coordinate system and provides camera intrinsic parameters to calculate its in the first... Second depth map from a different perspective pixels in ; (24) Judgment Is it located in Region of interest If not, then the signed distance function value of the voxel. Set to 1 and return to step (23). If it is already there, proceed to step (25). (25) Dot at The depth value in , The distance from the origin of the camera coordinate system is ,but Set the cutoff distance to The point is obtained by the following formula. Truncation of symbolic distance function value : (26) Define the projected ray from the camera origin and The included angle of the surface is Calculation points The weight is Repeat steps (23) to (26) until all voxels have been traversed; (27) From the current perspective , Distance function value of global truncation symbol and global weight Integration and All initialized to 0: (28) Switch to the depth map, region of interest and camera pose of the next view until the depth map set, region of interest set and camera pose set have been traversed.
2. The method for multi-view 3D reconstruction and flexible docking based on vision and force perception as described in claim 1, characterized in that, The location of the interface in the second RGB image under each viewpoint is detected using a deep learning-based object detection method, specifically: Input the second RGB image from each viewpoint into the position detection model to obtain the position of the interface in the second RGB image from each viewpoint; The position detection model includes an input layer, a backbone neural network, a neck neural network, and a head neural network. The position detection model is trained in the following manner: A training set was formed by collecting RGB sample images from different scenes and perspectives, and the positions of the interfaces in the RGB sample images were marked. The training set is input into the location detection model. The input layer performs data augmentation on the RGB sample images. The backbone neural network extracts features from the images output by the input layer. The neck neural network fuses the features extracted by the backbone neural network. The head neural network uses the fused features to output the predicted location of the interface. The location detection model is trained with the goal of minimizing the error between the predicted location of the interface and the marked location of the interface. The training continues until convergence, resulting in a well-trained location detection model.
3. The method for multi-view 3D reconstruction and flexible docking based on vision and force perception as described in claim 1, characterized in that, The method for detecting the interface position in the second RGB image under each viewpoint uses a method based on HSV color space segmentation, specifically: Collect interface image samples to obtain the HSV space value of the interface, convert the second RGB image from RGB color space to HSV color space from each viewpoint, calculate the minimum bounding rectangle of the HSV space value region of the interface in the second RGB image, and obtain the position of the interface.
4. A multi-view 3D reconstruction and flexible docking method based on vision and force perception as described in any one of claims 1-3, characterized in that, The detection interface uses an HSV color space-based segmentation method or a deep learning-based object detection method at the location of the first RGB image.
5. A multi-view 3D reconstruction and flexible docking method based on vision and force perception as described in any one of claims 1-3, characterized in that, Step (3) includes: The robotic arm moves to the docking position and monitors the contact force during the docking process; When the end of the robotic arm first enters the docking interface, the end of the robotic arm is controlled to make an initial probe along the z-axis of the docking pose. If the contact force is less than or equal to the maximum contact force of the initial probe, a second probe is made. If the contact force is greater than the maximum contact force of the initial probe, the translation amount of the end of the robotic arm is adjusted. If the contact force after adjustment is greater than the maximum contact force of the initial probe, step (2) is entered. If the contact force after adjustment is less than or equal to the maximum contact force of the initial probe, a second probe is made. After the initial test showed no abnormalities, the robotic arm's end effector continued to follow the docking posture. A second probe is made in the axial direction. If the contact force during the second probe is less than or equal to the maximum contact force during the second probe, the docking is performed using the accurate pose corresponding to this contact force. If the contact force during the second probe is greater than the maximum contact force during the second probe, the rotation of the end of the robotic arm is adjusted. If the contact force after adjustment is greater than the maximum contact force during the second probe, then proceed to step (2). If the contact force after adjustment is less than or equal to the maximum contact force during the second probe, the docking is performed using the accurate pose corresponding to this contact force. When docking with accurate pose, if the contact force is greater than the maximum contact force for docking, proceed to step (2). If the step error between the actual step displacement at the docking point of the robotic arm end and the given step displacement is greater than the threshold, adjust the step value at the docking point of the robotic arm end. When the contact force is within the set range, the step error is less than or equal to the threshold, and the robotic arm end is connected to the docking interface, the docking is successful.
6. A multi-view 3D reconstruction and flexible docking method based on vision and force perception as described in any one of claims 1-3, characterized in that, The method for adjusting the pose of the robotic arm's end effector is as follows: When the contact force is outside the set range, adjust the position of the robotic arm's end effector using the following actions: in, This indicates the number of times the adjustment failed. For adjustable hyperparameters, To dynamically adjust values, the priority adjustment pool is set to pre-defined actions, and "random" indicates that actions are generated randomly. This indicates the generation of a random number, with the action being either the translation or rotation of the robotic arm's end effector.
7. A multi-view 3D reconstruction and flexible docking method based on vision and force perception as described in any one of claims 1-3, characterized in that, If the interface is a charging port, then the end of the robotic arm is a charging head; if the interface is a water inlet, then the end of the robotic arm is a water supply pipe outlet; if the interface is a fuel inlet, then the end of the robotic arm is a fuel supply pipe outlet; if the interface is an air inlet, then the end of the robotic arm is a gas supply pipe outlet.
8. A multi-view 3D reconstruction and flexible docking system based on vision and force perception, used to execute the multi-view 3D reconstruction and flexible docking method based on vision and force perception as described in any one of claims 1-7, characterized in that, include: The visual positioning module includes a preliminary positioning module and a secondary positioning module. The preliminary positioning module is used to acquire the first RGB image and the first depth image of the robotic arm at the initial position, detect the position of the interface in the first RGB image, and calculate the coordinates of the interface in the world coordinate system by combining the depth value of the interface center in the first depth image. The secondary positioning module is used to move the robotic arm from its initial position to a distance of a preset value from the coordinates of the docking interface in the world coordinate system. It acquires the second RGB image and the second depth image of the robotic arm at that location from multiple perspectives, detects the position of the docking interface in the second RGB image under each perspective, expands the detection results under each perspective by multiple times to obtain the region of interest under each perspective, maps the region of interest under each perspective to the second depth image under the same perspective, and then performs three-dimensional reconstruction to obtain the reconstructed point cloud. The docking interface plane is extracted from the reconstructed point cloud, thereby generating the docking pose. The flexible docking module is used to move the robotic arm to the docking position, monitor the contact force during the docking process, and adjust the position of the robotic arm end when the contact force is not within the set range to achieve flexible docking.
9. An application of a multi-view 3D reconstruction and flexible docking method based on vision and force perception, characterized in that, The method is the flexible docking method according to any one of claims 1-7. The method is applied to charging, refueling, watering or gasing. When applied to charging, the charging head at the end of the robotic arm docks with the charging port. When applied to refueling, the refueling pipe outlet at the end of the robotic arm docks with the refueling port. When applied to watering, the water pipe outlet at the end of the robotic arm docks with the water inlet. When applied to gasing, the gas pipe outlet at the end of the robotic arm docks with the gas inlet.