Unmanned forklift autonomous charging method capable of accurately and visually identifying position and posture of charging port

Through deep learning-driven semantic segmentation and 6D pose regression network, the problems of position recognition accuracy and motion planning of unmanned forklift charging technology are solved, and efficient and independent charging of unmanned forklifts are achieved, which improves charging efficiency and accuracy.

CN118942080AInactive Publication Date: 2024-11-12SUZHOU AITEN INTELLIGENT TECH CO LTD

Patent Information

Application Number
CN202411422812.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-12
Publication Date
2024-11-12
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The existing unmanned forklift charging technology has problems such as cumbersome plug-in and unplugging operations, low wireless charging efficiency, high equipment cost, poor compatibility and insufficient position recognition accuracy in complex scenarios, making it difficult to achieve efficient and autonomous charging.

Method used

Using a deep learning-driven method, the side adapter network for open vocabulary semantic segmentation and the deep-guided 6D pose regression network are used to achieve accurate recognition of the charging port position, and combined with motion planning and control, the autonomous charging of the unmanned forklift is completed.

Benefits of technology

The feature extraction accuracy and position estimation accuracy of the charging port in complex scenarios are improved, the smooth and efficient motion planning of the mobile charging pile is realized, the autonomous charging task of the unmanned forklift is completed, and the operation complexity and equipment cost are reduced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118942080A_ABST
    Figure CN118942080A_ABST
Patent Text Reader

Abstract

The invention provides an unmanned forklift autonomous charging method capable of accurately and visually identifying the pose of a charging port, and the method comprises the four steps: hardware transformation, side adapter network construction and training facing open vocabulary semantic segmentation, deep-guided 6D pose regression network and mobile charging piles and charging heads to complete the accurate butt joint of the charging heads and the charging port. According to the method, the situation that the charging port is not in a visual image can be reasoned by using the advantage of semantic segmentation, the accuracy of feature extraction of the charging port in a monocular camera is greatly improved, and more accurate pose estimation is realized on the basis, so that smooth and efficient motion planning of the mobile charging pile is realized; and the autonomous charging task of the intelligent parking lot is completed.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of autonomous charging of unmanned forklifts, and in particular to an autonomous charging method for unmanned forklifts capable of accurately visually identifying the position of a charging port. Background Art

[0002] As an emerging technology in the field of industry and logistics, unmanned forklifts have developed rapidly in recent years. This product, which combines forklift technology and AGV (automatic guided vehicle) technology, can not only realize point-to-point material handling, but also be competent for logistics and transportation tasks in multiple production links such as high-rise warehouses, off-site receiving areas, and production line transfers, and is widely used in the field of automated warehousing and logistics.

[0003] How to charge unmanned forklifts is a major difficulty in current applications. The traditional charging method relies on manual plugging and unplugging of cables, which is cumbersome, error-prone, and inefficient. Since the plugging and unplugging force varies from person to person, it is also easy to cause problems such as damage to the charging plug and poor contact, and the maintenance cost is high. The high-end charging method is achieved through wireless charging technology, which does not require plugging and unplugging the charging plug. However, when used, the energy conversion efficiency of wireless charging is low, the charging speed is slow, the power transmission stability is poor, and there are problems such as equipment compatibility. At the same time, the equipment cost of wireless charging is high, and the requirements in terms of space planning, safety regulations, and environmental adaptability are more stringent, making it difficult to promote and use.

[0004] Generally speaking, an automatic charging system needs to go through four steps to respond to a charging request: "find the charging port - insert the charging gun into the charging port - start charging". First, to find the charging port, you need to obtain the approximate location of the charging port, and then use visual detection to obtain the precise position of the charging port to provide a control target for subsequent motion planning. The first step of visual search currently still relies on human experts to conduct regularized and process-based design by combining prior knowledge. When the target point appears within the camera range, the target detection and pose regression phase begins. For target detection, the more traditional and stable method is to extract the target bounding box by calling algorithms such as Yolo (Jiang, Peiyuan, et al. "A Review of Yolo algorithm developments." Procedia computer science 199 (2022): 1066-1073.) or Faster R-CNN (Girshick, Ross. "Fast r-cnn." Proceedings of the IEEE international conference on computer vision. 2015.), which have achieved a balance between speed and accuracy. However, in more complex scenes, the above algorithms are particularly deficient in feature point positioning accuracy. On this basis, the classic set method for pose estimation is mainly the PnP algorithm, which calculates the pose of the object by specifying the coordinates of known points in 3D space, that is, the corresponding 2D image projection points. However, this method requires setting standard points in the environment for calibration, which is difficult to deploy in actual application scenarios.With the popularity of neural networks, some methods that use deep learning frameworks to extract image features and then regress the 6D pose of objects have gradually been applied (Shi, Guanya, et al. "Fast uncertainty quantification for deep object pose estimation." 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021.), and there are models that directly use deep learning to regress the pose of objects from images, extract features through convolutional neural networks and predict pose parameters (Kendall, Alex, Matthew Grimes, and Roberto Cipolla. "Posenet: A convolutional network for real-time 6-dof camera relocalization." Proceedings of the IEEE international conference on computer vision. 2015.). However, most methods based on deep learning rely on huge training data sets, and the effect is often not ideal when the target object is occluded.

[0005] At the same time, after realizing the 6D pose extraction of the charging port, it is necessary to plan and control the motion path of the mobile charging pile. Usually, the A-star algorithm or randomized algorithms such as RRT and PRM are used for random sampling to plan the path. However, the above algorithms have certain limitations in handling dynamic obstacles and are very easy to fall into local optimality. Therefore, some intelligent algorithms have been developed, such as deep reinforcement learning (Li, **angjian, Huashan Liu, and Menghua Dong. "A general framework of motion planning for redundant robot manipulator based on deep reinforcement learning." IEEE Transactions on Industrial Informatics18.8 (2021): 5253-5263.) and imitation learning (Mandlekar, Ajay, et al. "Human-in-the-looptask and motion planning for imitation learning." Conference on RobotLearning. PMLR, 2023.) models, which rely on interaction with the environment or expert teaching to quickly and accurately achieve the optimal planning and control decisions. However, there are still the disadvantages of learning-based methods, that is, they have high requirements for data and computing power, and it is difficult to handle dynamic environments and rely on a good simulation environment, which is difficult to do in open environments. Summary of the invention

[0006] The purpose of the present invention is to solve one or more problems in the prior art and to provide an autonomous charging method for an unmanned forklift that can accurately visually identify the position of a charging port. In this method, accurate recognition of the position of the charging port is achieved based on deep learning drive, and on this basis, efficient motion planning and control of the charging pile are achieved, which can successfully complete the autonomous charging task of the unmanned forklift.

[0007] In order to achieve the above-mentioned object, the technical solution adopted by the present invention is an autonomous charging method for an unmanned forklift capable of accurately visually identifying the position of a charging port, comprising the following steps: S1. Hardware transformation, including charging pile transformation, camera deployment, and coordinate system construction; The modification of the charging pile is used to enable the charging pile to travel freely within the charging area, and to enable the charging head on the charging pile to move relative to the charging pile in at least four degrees of freedom; Camera deployment refers to placing a monocular camera on the charging pile to acquire visual images; Constructing a coordinate system includes constructing a global coordinate system with the earth as reference , and, construct a local coordinate system with the modified charging pile as reference ; S2. Construction and training of side adapter networks for open vocabulary semantic segmentation, including: S2.1. Contrasting language-image pre-training model: First, image information and text information are mapped to the same feature space, and then a set of <preset image, preset text> pairs are given to pair the image information with the preset image, that is, , where image information refers to the image information obtained by the monocular camera, and text information refers to the charging pile in the overall coordinate system when the image information is taken. The preset picture refers to the image of at least a part of the unmanned forklift entering the charging area taken in advance, and the preset text refers to the position of the shooting point in the overall coordinate system when the preset picture is taken. The following location information; S2.2. Construction of side adapter network: The layer features of the shallow contrastive language-image pre-trained model are fused into the side adapter network, and the attention bias is applied to the remaining deeper pre-trained model layers for recognition. After the input image information is segmented into 16×16, a linear embedding layer is used to form a visual marker for projection, which is connected with the learnable query and fed to the subsequent Transformer layer. The output of the final side adapter network is a semantic segmentation map; S2.3. Template matching and charging port discovery: Compare the semantic segmentation map obtained in S2.2 with the 3D image of the unmanned forklift to obtain the local coordinate system of the unmanned forklift. Rough pose under , and then obtain the rough position of the charging port by checking the model of the unmanned forklift , and the coordinate system of the unmanned forklift is obtained by coordinate transformation Rough pose under ,in: ; S2.4. Rough pose Get the charging pile in the overall coordinate system The target posture under the control of the camera is determined, and the charging pile is moved to the target posture until the semantic segmentation map of the monocular camera image information contains the relevant semantic information of the charging port, and then this step is terminated; S3. Depth-guided 6D pose regression network, including: S3.1. Depth information capture: Two depth estimation networks are established to compare the errors of the two generated depth maps, distinguish the areas with uncertain depth, and use the self-attention mechanism to optimize the areas; the depth maps generated by the two networks are represented as , calculate the difference of the depth map and define the area where the difference exceeds the threshold as the uncertain area, and use the cross attention mechanism to calculate the depth Query, with uncertain area As the key and value, the calculation of the attention map is: ; in represents the attention map of the m-th head, represents the normalization function, Represents a multi-layer perceptron and obtains a multi-head cross attention map: ;in are the output linear layer parameters at the end of the network architecture; The final depth feature map is: ; This is used to construct the depth loss , first define the photometric reconstruction loss function: ; in represents the structural similarity calculated on a 3×3 pixel window, is a balancing parameter used to balance the proportion of the structural similarity loss function and the reconstruction loss function to the total loss function. Reconstructing the image Defined as: ; in represents the sampling operator, represents the relative position of the camera, Representative image The projection depth, represents the intrinsic parameter matrix of the camera, which is the same for all images, where The 2D coordinates used to calculate the depth of the image projection: ; S3.2. Feature extraction: Represents a pixel in an RGB image, express The adjacent depth set of Represents adjacent semantic segmentation categories, using nonlinear functions To aggregate Pixel-depth local features, where Represents the neural network parameters that will be used for subsequent training in the nonlinear function of the current pixel. Similarly, using to aggregate depth-pixel features, where Representing pixels The pixel features of Then fuse the two to obtain local features: ; S3.3. Key point positioning: Based on the structural design of the charging port, the hole position is selected as the 2D key point and used for subsequent pose estimation; S3.4. Obtain 6D pose: from n 3D key points , where each Corresponds to a set of 2D positions in the image , establish the loss function for 2D-3D matching: ; in express The actual 2D keypoint position of the pixel, is the number of 3D key points, represent The corresponding number in the 2D keypoint template, The error between each 2D key point corresponding to each 3D key point represented by θ and the actual 2D key point position is used as the loss function for network training. To alleviate the dense / sparse requirements for correspondence, a graph convolutional network is built to directly regress 6D pose from 2D-3D correspondences: ; in represents the rotation matrix, Represents the absolute pose, and the loss function is defined as: ; in is the estimated rotation matrix and translation vector, followed by the true value; S3.5. Loss function and training, the loss function is as follows: ;in, Represents the importance of the three loss functions to the overall loss, which can be adjusted according to actual engineering needs; S3.6.Charging port posture output: ; S4. Move the charging pile and charging head to complete the precise docking between the charging head and the charging port, including: S4.1. Mobile charging pile modeling: The state of the system is represented by the six-degree-of-freedom position of the charging head in the world coordinate system , then the state kinematic model is: ; in is the sampling time; S4.2. Optimization problem modeling: ; in The target charging port position obtained by S3.6 Transformed into numerical values ​​in Cartesian space, Indicates that the positions of all joints are within the safe area, that is, the obstacle avoidance constraint, and finally optimizes the process, input regularization and terminal cost comprehensively; in the objective function Represents the error between the reference pose and the actual pose, , The weight matrices representing the process and final accuracy, Represents the control variable in the optimization control problem, which is mainly used to control the torque of each joint of the robot arm so that it can reach the target posture; S4.3. Obstacle avoidance constraint construction: Starting from the movement of the charging pile, if a new semantic information appears in the semantic segmentation map, it is necessary to construct constraints for the newly appeared obstacles, use the semantic segmentation map as a feature point mask, use the pose estimation in S3 to model the obstacles, and Every spatial point in Construct constraints to represent a certain danger zone around the obstacle point: ; in, Represents the safety radius designed according to actual engineering requirements to achieve obstacle avoidance.

[0008] Preferably, in step S1, the charging pile modification is achieved by adding a driven wheeled chassis to the existing charging pile, and adding a moving and driving mechanism to the charging head on the existing charging pile; and the monocular camera is arranged on the front part of the charging pile.

[0009] Preferably, step S2.1 further includes: Vision encoder and text encoder The step of reducing the distance between the paired image information and the text information corresponding to the preset picture and the preset text, and simultaneously increasing the distance between the unpaired image information and the text information corresponding to the preset picture and the preset text; wherein the visual encoder The working mode is: first divide the image information into a series of small image blocks, combine the position information of the small image blocks in the preset picture, use gradient training linear projection to project them into one dimension, and then use the self-attention model network to encode them; text encoder The working mode is: use the self-attention model network to encode text information and preset text; thereby and text Converted into hidden features , and finally use the contrast loss function to reduce The cosine distance between: ; in, Represents a batch of data, including image data and text data , Represents the cosine similarity between two features, is a batch of alternative texts, including a positively matched alternative text and negative alternative texts, which is a hyperparameter for scaling similarity is a hyperparameter for scaling similarity.

[0010] Preferably, step S2.2 specifically includes: S2.2.1. Mask prediction: First, two independent 3-layer multi-layer perceptrons (MLPs) are used to project the query token and the visualization token into 256-dimensional features, where the query token is ,in To query the number of tags, the visualization tags are , the predicted mask can be obtained by performing an inner product operation on the two: ; in ; S2.2.2. Attention bias: query and visualization tags are projected through a 3-layer MLP and ,in is the number of attention heads in the VIT pre-trained model, and the attention bias is obtained: ; in ; S2.2.3. Mask recognition: After completing mask prediction and generating attention bias, use attention bias to judge the category of the predicted mask, and use this as the basis for semantic segmentation. Then design a category of classification tags, which are updated unidirectionally by the visual tags, but do not affect the visual tags. The tags are updated by adding attention bias: ; in is the number of layers, Representative Attention head, , query and value embeddings representing categorical labels, is a query for visual labels, which can eventually get the class prediction of the mask , is the number of categories; S2.2.4. Semantic segmentation map generation: Using mask recognition and mask prediction, we can get the semantic segmentation map: ; in .

[0011] Preferably, when completing step S3.1, in order to encourage adjacent pixels to have similar gradients, it also includes a step of using an edge-aware depth smoothing loss weighted by image gradient to improve object edge prediction, and the calculation formula is: ; in represents the gradient in the axis direction, , represents the normalized inverse depth, then the final loss function is: .

[0012] Due to the application of the above technical solution, the present invention has the following advantages compared with the prior art: The present invention can use the advantages of semantic segmentation to infer the charging port without a field of view, greatly improve the feature extraction of the charging port in the monocular camera, and achieve more accurate pose estimation based on this. This can achieve smooth and efficient motion planning of the mobile charging pile and complete the autonomous charging task of the smart parking lot.

[0013] The present invention involves using deep learning to complete target detection and pose estimation, and based on this, motion planning is used to achieve autonomous charging of unmanned forklifts. The specific design includes first building a hardware facility system to achieve autonomous charging. On this basis, a side-facing adapter network is designed to perform open vocabulary semantic segmentation on the image of the mobile charging pile when observing the forklift to be charged, and combined with the prior forklift parking position coordinates and forklift model, the rough position of the charging port is obtained, and the charging pile is moved to a position where the charging port can be observed; then, a deep-guided 6D target pose regression network is used in combination with the semantic segmentation map obtained in the previous stage to obtain the precise pose of the charging port; finally, the information obtained is used to complete the motion planning and control of the mobile charging pile using a controller based on model predictive control. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 Flowchart of the algorithm for accurate visual recognition of charging ports and autonomous charging of unmanned forklifts driven by deep learning.

[0015] Figure 2Schematic diagram of the autonomous charging hardware system.

[0016] Figure 3 Schematic diagram of the side adapter network architecture for open vocabulary semantic segmentation.

[0017] Figure 4 Schematic diagram of the contrastive language-image pre-training model structure.

[0018] Figure 5 Schematic diagram of the depth-guided 6D object pose regression network structure. DETAILED DESCRIPTION

[0019] Figure 1 It is a flowchart for illustrating the specific implementation method of the embodiment for charging port recognition and autonomous charging of unmanned forklifts. The core technology is to first control the mobile charging pile to travel to a position where the unmanned forklift can be observed based on the position information of the unmanned forklift parked in the parking space, and use the side adapter network for open vocabulary semantic segmentation to obtain images from the monocular camera installed on the mobile charging pile for semantic segmentation, and match it with the template of the unmanned forklift model to determine the approximate position of the charging port, and guide the charging pile to move to a position where the charging port can be directly observed by the monocular camera. On this basis, a deep-driven neural network is used in combination with a semantic segmentation feature set to estimate the 6D posture of the charging port. Finally, model predictive control is used to take the relevant information of the charging port as the target posture, combined with real-time visual updates of obstacle constraints to perform motion planning and control of the mobile charging pile.

[0020] S1. First of all, the hardware system for realizing autonomous charging is the deep learning-driven accurate recognition of charging ports and the autonomous charging hardware system for unmanned forklifts. The overall hardware system is as follows: Figure 2 As shown, it specifically includes the following sub-steps: Mobile charging pile transformation: Mainly transform the existing fixed charging piles. First, add a wheeled chassis to facilitate its free travel in the charging area. On this basis, combine different forklift models to increase the charging docking part to 4 degrees of freedom to meet different docking target postures.

[0021] Monocular camera deployment: A monocular camera VEN-830-22U3C is installed on the front part of the mobile charging pile to acquire visual images. In step (2), it mainly determines the current posture of the vehicle body to roughly infer the charging port, and assists in extracting the posture of the charging port in step (3).

[0022] Positioning coordinate system construction: The coordinate system constructed for subsequent pose estimation and motion planning mainly includes the following two coordinate systems: Overall coordinate system: It is an absolutely stationary overall coordinate system, expressed as , mainly serving the subsequent motion planning of mobile charging piles.

[0023] Local charging pile coordinate system: It is the local coordinate system of relative motion with the mobile charging pile chassis as the reference, expressed as , which mainly serves the subsequent posture estimation of unmanned forklifts and charging ports.

[0024] S2. Construction and training of the side adapter network for open vocabulary semantic segmentation: Since the monocular camera VEN-830-22U3C equipped on the mobile charging pile in the unmanned forklift charging area cannot capture the image of the charging port in the first time of responding to the charging request, it is necessary to perform semantic segmentation on the photos taken by the charging pile at the initial position to obtain the relative rough position of the charging port in the global static coordinate system. The overall network structure is as follows Figure 3 As shown, it specifically includes the following sub-steps: S2.1. Contrasting language-image pre-training model: Since the search for charging ports requires certain semantic information, a contrasting language-image pre-training model is first constructed to map image information and text information to the same feature space. The model structure diagram is shown in the figure below. Figure 4 When building and training the model, a set of <image, text> pairs is given, that is, , the goal is to pass the visual encoder and image encoder The two encoders are explained as follows: Visual Encoder: Using the Vision Transformer (VIT) model, the image is divided into a series of small image blocks. Combined with the position information in the image, a linear projection that can be trained for gradients is used to project it into 1D, and then it is encoded using a common self-attention model (Transformer) network.

[0025] Text encoder: Since text information only has one-dimensional information, only the self-attention model is used for encoding.

[0026] The above two encoders convert the image and text Converted into hidden features , and finally use the contrast loss function to reduce The cosine distance between: ;

[0027] in Represents a batch of data, including image data and text data , Represents the cosine similarity between two features, is a batch of alternative texts, including a positively matched alternative text and Negative alternative text, is a hyperparameter for scaling similarity.

[0028] S2.2. Side adapter network construction: In order to give full play to the contrastive language-image pre-trained model (hereinafter referred to as the pre-trained model) proposed in (1.1), a side adapter network is implemented through a lightweight visual converter. The layer features of the shallow pre-trained model are fused into the side adapter network, and the attention bias is applied to the remaining deeper pre-trained model layers for recognition.

[0029] The model structure of the side adapter network is as follows Figure 3 As shown in Figure 2. After the input image is segmented into 16×16, it passes through a linear embedding layer to form a visual mark for projection, which is connected with the learnable query and fed into the subsequent Transformer layer. The output of the final side adapter network is the mask proposal and the corresponding attention bias for mask recognition.

[0030] S2.2.1 Mask prediction: First, two independent 3-layer multilayer perceptrons (MLP) project the query tag and the visualization tag into 256-dimensional features, where the query tag is ( is the number of query tags), the visualization tags are , so the predicted mask can be obtained by performing an inner product operation on the two:

[0031] in

[0032] S2.2.2.2 Attention Bias: Similar to mask prediction, query tags and visualization tags are projected through a 3-layer MLP and ,in is the number of attention heads in the VIT pre-trained model, and the attention bias is obtained:

[0033] in

[0034] S2.2.3 Mask recognition: After completing mask prediction and generating attention bias, attention bias needs to be used to judge the category of the predicted mask and use it as the basis for semantic segmentation. Design a class of classification tags that are updated unidirectionally by visual tags without affecting the visual tags. Update the tags by adding attention bias:

[0035] in is the number of layers, Representative Attention head, , query and value embeddings representing categorical labels, is a query for visual labels. Finally, we can get the class prediction of the mask , is the number of categories.

[0036] S2.2.4 Semantic segmentation map generation: Using mask recognition and mask prediction, the segmentation map can be obtained:

[0037] in , and finally use dice loss and binary cross entropy loss for training: ; The left side of the equation is the overall loss function of semantic segmentation , the three terms on the right side of the equation are respectively calculated by dice loss , and binary cross entropy loss to supervise the generation of masks, and The recognition of masks is supervised by binary cross entropy loss. , , To balance the weight parameters of the three loss functions, they can be adjusted according to actual engineering needs;

[0038] The gradient flows in Figure 3 is displayed in the

[0039] S2.3. Template matching and charging port discovery: In order to find the charging port more quickly, the semantic segmentation map obtained in (2.2) is compared with the 3D image of the forklift to obtain the local coordinate system of the unmanned forklift. Rough pose under , and obtain the rough position of the charging port through the prior knowledge of the forklift model , and obtain the position and posture in the global coordinate system through coordinate transformation:

[0040] in Represents the rough pose. The target pose of the mobile charging pile is obtained with the rough pose, and the motion planning method in step (4) is used to control it to a position where the charging port can be observed, until the segmentation map in the image of the monocular camera carried contains the relevant semantic information of the charging port, and then this step is ended. S3. Depth-guided 6D pose regression network: After completing the relevant sub-steps of step (2), so that the relevant semantic information of the charging port appears in the monocular camera image carried by a mobile charging pile, in order to estimate the 6D pose from the monocular image, the traditional key point positioning adopts a voting-based structure, which cannot be combined with depth information for regression. Therefore, the present invention mainly incorporates depth information to enhance the robustness and accuracy of the corresponding 2D-3D pose estimation. The network architecture is as follows Figure 4 As shown, it mainly includes the following steps and core elements: S3.1. Depth information capture: By establishing two depth estimation networks, the two generated depth maps are compared to determine the area with uncertain depth. The self-attention mechanism is used to optimize this area. The depth maps generated by the two networks are represented as Calculate the difference of the depth map and define the area where the difference exceeds the threshold as the uncertain area, use the cross attention mechanism to solve part of it, and divide the depth As a query, the uncertainty graph As the key and value, the calculation of the attention map can be expressed as:

[0041] in represents the attention map of the m-th head, represents the normalization function, Represents a multi-layer perceptron, then we can get a multi-head cross attention map: ,in are the output linear layer parameters at the end of the network architecture;

[0042] The final deep feature map can be expressed as:

[0043] This is used to construct the depth loss , first define the photometric reconstruction loss function:

[0044] in represents the structural similarity calculated on a 3×3 pixel window, is a balancing parameter used to balance the proportion of the structural similarity loss function and the reconstruction loss function to the total loss function, and reconstruct the image Defined as:

[0045] in represents the sampling operator, represents the relative position of the camera, Representative image The projection depth, represents the intrinsic parameter matrix of the camera, which is the same for all images, where The 2D coordinates used to calculate the depth of the image projection:

[0046] In order to encourage neighboring pixels to have similar gradients, an edge-aware deep smoothing loss weighted by image gradient is used to improve object edge prediction:

[0047] in represents the gradient in the axis direction, , represents the normalized inverse depth, and the final loss function can be expressed as:

[0048] S3.2. Feature extraction: After completing the deep feature extraction, it is necessary to accurately extract and segment the target. Represents a pixel in an RGB image, express The adjacent depth set of Indicates the adjacent semantic segmentation category. The target extraction including semantic segmentation can solve the problem of if it is in the camera field of view. However, when there is a certain occlusion on the charging port, it is easy to cause the general target feature point extraction network to be unable to recognize the target. Finally, the nonlinear function is used To aggregate Pixel-depth local features, where Represents the neural network parameters that will be used for subsequent training in the nonlinear function of the current pixel. Similarly, using to aggregate depth-pixel features, where Representing pixels The pixel features of , and then fuse the two to obtain local features: ;

[0049] 3.3. Key point positioning: Combined with the structural design of the charging port, we select the more obvious hole positions as 2D key points and use them for subsequent pose estimation.

[0050] S3.4. Obtaining 6D pose: This is mainly done by finding the transformation relationship between key points. , where each Corresponds to a set of 2D positions in the image , establish the loss function for 2D-3D matching: ; in express The actual 2D keypoint position of the pixel, is the number of 3D key points, represent The corresponding number in the 2D keypoint template, The error between each 2D key point corresponding to each 3D key point represented by θ and the actual 2D key point position is used as the loss function for network training. To alleviate the requirement for dense / sparse correspondences, a graph convolutional network is built to directly regress 6D poses from 2D-3D correspondences: ; in represents the rotation matrix, Represents the absolute pose, and the loss function is defined as: ; in is the estimated rotation matrix and translation vector, followed by the true value; S3.5. Loss function and training: The final loss function is as follows: ;in, Represents the importance of the three loss functions to the overall loss, which can be adjusted according to actual engineering needs;

[0051] S3.6. Output of charging port posture: According to step (2) combined with the above design scheme of step (3), the precise posture of the charging port in the overall coordinate system can be obtained:

[0052] S4. After obtaining the precise position and posture of the target charging port, it is necessary to control the mobile charging pile to complete the task of precise docking. Due to the existence of the vehicle structure, it is necessary to consider both obstacle avoidance and target tracking when implementing the docking task, which is a motion planning problem. It mainly includes the following sub-steps: S4.1. Mobile charging pile modeling: The state of the system is represented by the six-degree-of-freedom position of the charging docking end in the global coordinate system. , the control actions represented by the actuator mainly include the movement speed and direction of the wheeled chassis and the angular velocity of the four-degree-of-freedom steering gear of the charging docking part. Based on the above analysis, the state kinematic model can be obtained:

[0053] in is the sampling time.

[0054] S4.2. Optimization problem modeling: The optimization objectives mainly include three tasks: approaching the target while avoiding high speed as much as possible; avoiding obstacles; and keeping the speed within a safe range. Therefore, the optimization problem for the optimal system input sequence can be expressed as:

[0055] in The target charging port position obtained by S3.6 Transformed into numerical values ​​in Cartesian space, Indicates that the positions of all joints are within the safe area, that is, the obstacle avoidance constraint, and finally optimizes the process, input regularization and terminal cost comprehensively; in the objective function Represents the error between the reference pose and the actual pose, , The weight matrices representing the process and final accuracy, Represents the control variable in the optimization control problem, which is mainly used to control the torque of each joint of the robot arm so that it can reach the target posture.

[0056] S4.3. Obstacle avoidance constraint construction: When starting from the mobile charging station, if a new semantic information appears in the semantic segmentation, it is necessary to construct constraints for the newly appeared obstacles. The semantic segmentation is used as the feature point mask, and the pose estimation in step (3) is used to build the obstacle model. Every spatial point in Construct constraints to represent a certain danger zone around the obstacle point:

[0057] in, Represents the safety radius designed according to actual engineering requirements to achieve obstacle avoidance.

[0058] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with this technology to understand the contents of the present invention and implement them accordingly. They cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. An autonomous charging method for an unmanned forklift capable of accurately visually identifying the position of a charging port, characterized in that: The following steps are involved: S1. Hardware transformation, including charging pile transformation, camera deployment, and coordinate system construction; The modification of the charging pile is used to enable the charging pile to travel freely within the charging area, and to enable the charging head on the charging pile to move relative to the charging pile in at least four degrees of freedom; Camera deployment refers to placing a monocular camera on the charging pile to acquire visual images; Constructing a coordinate system includes constructing a global coordinate system with the earth as reference , and, construct a local coordinate system with the modified charging pile as reference ; S2. Construction and training of side adapter networks for open vocabulary semantic segmentation, including: S2.

1. Contrasting language-image pre-training model: First, image information and text information are mapped to the same feature space, and then a set of <preset image, preset text> pairs are given to pair the image information with the preset image, that is, , where image information refers to the image information obtained by the monocular camera, and text information refers to the charging pile in the overall coordinate system when the image information is taken. The preset picture refers to the image of at least a part of the unmanned forklift entering the charging area taken in advance, and the preset text refers to the position of the shooting point in the overall coordinate system when the preset picture is taken. The following location information; S2.

2. Construction of side adapter network: The layer features of the shallow contrastive language-image pre-trained model are fused into the side adapter network, and the attention bias is applied to the remaining deeper pre-trained model layers for recognition. After the input image information is segmented into 16×16, a linear embedding layer is used to form a visual marker for projection, which is connected with the learnable query and fed to the subsequent Transformer layer. The output of the final side adapter network is a semantic segmentation map; S2.

3. Template matching and charging port discovery: Compare the semantic segmentation map obtained in S2.2 with the 3D image of the unmanned forklift to obtain the local coordinate system of the unmanned forklift. Rough pose under , and then obtain the rough position of the charging port by checking the model of the unmanned forklift , and the coordinate system of the unmanned forklift is obtained by coordinate transformation Rough pose under ,in: ; S2.

4. Rough pose Get the charging pile in the overall coordinate system The target posture under the control of the camera is determined, and the charging pile is moved to the target posture until the semantic segmentation map of the monocular camera image information contains the relevant semantic information of the charging port, and then this step is terminated; S3. Depth-guided 6D pose regression network, including: S3.

1. Depth information capture: Two depth estimation networks are established to compare the errors of the two generated depth maps, distinguish the areas with uncertain depth, and use the self-attention mechanism to optimize the areas; the depth maps generated by the two networks are represented as , calculate the difference of the depth map and define the area where the difference exceeds the threshold as the uncertain area, and use the cross attention mechanism to calculate the depth Query, with uncertain area As the key and value, the calculation of the attention map is: ; in represents the attention map of the m-th head, represents the normalization function, Represents a multi-layer perceptron and obtains a multi-head cross attention map: ;in are the output linear layer parameters at the end of the network architecture; The final depth feature map is: ; This is used to construct the depth loss , first define the photometric reconstruction loss function: ; in represents the structural similarity calculated on a 3×3 pixel window, is a balancing parameter used to balance the proportion of the structural similarity loss function and the reconstruction loss function to the total loss function. Reconstructing the image Defined as: ; in represents the sampling operator, represents the relative position of the camera, Representative image The projection depth, represents the intrinsic parameter matrix of the camera, which is the same for all images, where The 2D coordinates used to calculate the depth of the image projection: ; S3.

2. Feature extraction: Represents a pixel in an RGB image, express The adjacent depth set of Represents adjacent semantic segmentation categories, using nonlinear functions To aggregate Pixel-depth local features, where Represents the neural network parameters that will be used for subsequent training in the nonlinear function of the current pixel. Similarly, using to aggregate depth-pixel features, where Representing pixels The pixel features of Then fuse the two to obtain local features: ; S3.

3. Key point positioning: Based on the structural design of the charging port, the hole position is selected as the 2D key point and used for subsequent pose estimation; S3.

4. Obtain 6D pose: from n 3D key points , where each Corresponds to a set of 2D positions in the image , establish the loss function for 2D-3D matching: ; in express The actual 2D keypoint position of the pixel, is the number of 3D key points, represent The corresponding number in the 2D keypoint template, The error between each 2D key point corresponding to each 3D key point represented by θ and the actual 2D key point position is used as the loss function for network training. To alleviate the requirement for dense / sparse correspondences, a graph convolutional network is built to directly regress 6D poses from 2D-3D correspondences: ; in represents the rotation matrix, Represents the absolute pose, and the loss function is defined as: ; in is the estimated rotation matrix and translation vector, followed by the true value; S3.

5. Loss function and training, the loss function is as follows: ;in, Represents the importance of the three loss functions to the overall loss, which can be adjusted according to actual engineering needs; S3.6.Charging port posture output: ; S4. Move the charging pile and charging head to complete the precise docking between the charging head and the charging port, including: S4.

1. Mobile charging pile modeling: The state of the system is represented by the six-degree-of-freedom position of the charging head in the world coordinate system , then the state kinematic model is: ; in is the sampling time; S4.

2. Optimization problem modeling: ; in The target charging port position obtained by S3.6 Transformed into numerical values ​​in Cartesian space, Indicates that the positions of all joints are within the safe area, that is, the obstacle avoidance constraint, and finally optimizes the process, input regularization and terminal cost comprehensively; in the objective function Represents the error between the reference pose and the actual pose, , The weight matrices representing the process and final accuracy, Represents the control variable in the optimization control problem, which is mainly used to control the torque of each joint of the robot arm so that it can reach the target posture; S4.

3. Obstacle avoidance constraint construction: Starting from the movement of the charging pile, if a new semantic information appears in the semantic segmentation map, it is necessary to construct constraints for the newly appeared obstacles, use the semantic segmentation map as a feature point mask, use the pose estimation in S3 to model the obstacles, and Every spatial point in Construct constraints to represent a certain danger zone around the obstacle point: ; in, Represents the safety radius designed according to actual engineering requirements to achieve obstacle avoidance.

2. The unmanned forklift autonomous charging method capable of accurately visually identifying the position of the charging port according to claim 1 is characterized in that: In step S1, the charging pile is modified by adding a driven wheeled chassis to the existing charging pile, and adding a moving and driving mechanism to the charging head on the existing charging pile; the monocular camera is arranged on the front part of the charging pile.

3. The unmanned forklift autonomous charging method capable of accurately visually identifying the position of the charging port according to claim 1 is characterized in that: Step S2.1 also includes: Vision encoder and text encoder The step of reducing the distance between the paired image information and the text information corresponding to the preset picture and the preset text, and simultaneously increasing the distance between the unpaired image information and the text information corresponding to the preset picture and the preset text; wherein the visual encoder The working mode is: first divide the image information into a series of small image blocks, combine the position information of the small image blocks in the preset picture, use gradient training linear projection to project them into one dimension, and then use the self-attention model network to encode them; text encoder The working mode is: use the self-attention model network to encode text information and preset text; thereby and text Converted into hidden features , and finally use the contrast loss function to reduce The cosine distance between: ; in, Represents a batch of data, including image data and text data , Represents the cosine similarity between two features, is a batch of alternative texts, including a positively matched alternative text and negative alternative texts, which is a hyperparameter for scaling similarity is a hyperparameter for scaling similarity.

4. The method for autonomous charging of an unmanned forklift capable of accurately visually identifying the position of a charging port according to claim 1, characterized in that: Step S2.2 specifically includes: S2.2.

1. Mask prediction: First, two independent 3-layer multi-layer perceptrons (MLPs) are used to project the query token and the visualization token into 256-dimensional features, where the query token is ,in To query the number of tags, the visualization tags are , the predicted mask can be obtained by performing an inner product operation on the two: ; in ; S2.2.

2. Attention bias: query and visualization tags are projected through a 3-layer MLP and ,in is the number of attention heads in the VIT pre-trained model, and the attention bias is obtained: ; in ; S2.2.

3. Mask recognition: After completing mask prediction and generating attention bias, use attention bias to judge the category of the predicted mask, and use this as the basis for semantic segmentation. Then design a category of classification tags, which are updated unidirectionally by the visual tags, but do not affect the visual tags. The tags are updated by adding attention bias: ; in is the number of layers, Representative A head of attention, , query and value embeddings representing categorical labels, is a query for visual labels, which can eventually get the class prediction of the mask , is the number of categories; S2.2.

4. Semantic segmentation map generation: Using mask recognition and mask prediction, we can get the semantic segmentation map: ; in .

5. The method for autonomous charging of an unmanned forklift capable of accurately visually identifying the position of a charging port according to claim 1, characterized in that: When completing step S3.1, in order to encourage adjacent pixels to have similar gradients, a step of improving object edge prediction by using an edge-aware depth smoothing loss weighted by image gradient is also included, and the calculation formula is: ; in represents the gradient in the axis direction, , represents the normalized inverse depth, then the final loss function is: 。

Citation Information

Patent Citations

  • Automatic recharging control method for hydraulic steering Ackerman type robot

    CN113581285A

  • Goods shelf pose recognition method, device and equipment and storage medium

    CN115511965A

  • Indoor service robot repositioning method based on dense feature matching

    CN115578426A

  • 6D pose estimation method applied to robot grabbing task

    CN118470114A

Cited By

  • Excavator pose recognition method based on 3d vision

    CN119515981A

  • A 3d vision-based pose recognition method for excavators

    CN119515981B

  • Vision-based operation management system

    CN119668303A

  • Charging equipment, charging control method and device thereof and medium

    CN120828690A

  • A charging gun plug-pull fatigue compensation and pose self-adaptive correction method

    CN123010397A