Hole tray sensing and sensing method and device and electronic equipment

Abstract

The invention relates to a hole tray sensing method and device and electronic equipment, and the method comprises the steps: obtaining a complete to-be-grabbed point cloud structure based on remote RGBD prior region estimation and mechanical arm RGBD camera close-range multi-view shooting; outputting an optimal grabbing posture according to feature extraction and grabbing posture simulation; and based on space search and reinforcement learning path planning, determining the optimal storage point position and path planning of the plug tray. According to the method, the plug trays completely covered by densely growing crops can be visualized, when the plug trays are closely arranged and stacked on an original culture table, the grabbing key point and pose information of the single plug tray can be conveniently and accurately judged by adopting an iterative simulation mode of combining feature extraction with grabbing postures, and meanwhile, the grabbing key point and pose information of the single plug tray can be accurately judged. The optimal storage point position of the hole tray can be quickly selected through combination of space search and convenience of reinforcement learning path planning, and the hole tray is transferred according to the planned path.

Description

Technical Field

[0001] The present invention relates to the field of plug tray transfer, and in particular to a plug tray sensing and sensing method, device and electronic equipment. Background Art

[0002] When grabbing, transferring and restacking the plug trays on the culture table, the perception and positioning of the plug trays are the key difficulties to achieve this goal. Since the plug trays are closely arranged and stacked on the original culture table, it is difficult for the system to accurately determine the key points and posture information of the grabbing of a single plug tray. At the same time, compared with empty plug trays and growing culture plug trays, for plug trays that have been cultured and are waiting to be harvested, since these plug trays are completely covered by densely growing crops, the method of relying on RGB images for single recognition is no longer applicable and difficult to visualize. Therefore, it needs to be improved. Summary of the invention

[0003] Based on this, it is necessary to provide a perception and sensing method, device and electronic device for the plug trays, in order to address the problem that it is difficult to accurately determine the grasping key points and posture information of a single plug tray when the plug trays are closely arranged and stacked on the original culture table. At the same time, for the plug trays that have been cultured and are waiting to be harvested, since these plug trays have been completely covered by densely growing crops, the method of relying on RGB images for separate identification is no longer applicable and is difficult to visualize.

[0004] The present invention provides a method for sensing a hole tray, the method comprising:

[0005] Based on long-range RGBD prior area estimation and close-range multi-view shooting by the robot arm RGBD camera, the complete point cloud structure to be grasped is obtained;

[0006] Output the optimal grasping posture based on feature extraction and grasping posture simulation;

[0007] Based on spatial search and reinforcement learning path planning, the optimal storage point location and path planning of the plug tray are determined.

[0008] In one embodiment, the method of obtaining a complete point cloud structure to be captured based on long-range RGBD priori region estimation and close-range multi-view shooting of a robotic arm RGBD camera includes:

[0009] Obtain remote RGBD information and perform denoising and smoothing processing on the remote RGBD information;

[0010] Roughly divide the area according to the size of a single plug tray platform, extract the segmentation map, and calibrate the suspected center point;

[0011] Acquire images taken at multiple angles in the suspected center point area, where the suspected center point area includes the outer edge and the outer long side area of ​​the hole plate;

[0012] Reconstruct point cloud data from multiple perspectives to generate a complete 3D point cloud;

[0013] Based on the Nerf neural network, the position, radius, color and opacity information of scene points are rendered as Gaussian field parameters to obtain a fine 3D structure.

[0014] In one embodiment, the obtaining of remote RGBD information includes:

[0015] Acquire leaf image data of plants at different growth stages;

[0016] Mark the key point information of the connection between seedlings and the center point of the crop plant in the leaf image;

[0017] The leaf image data is enhanced by rotation, flipping, scaling and color adjustment.

[0018] In one embodiment, the calibrating the suspected center point includes:

[0019] Acquire a scene image of dense vegetation;

[0020] Based on the attention mechanism, the scene image of a single plant is extracted from dense plants;

[0021] The scene image of the single plant is input into the YOLOv8-Pose model, and the position information of the captured key points is output, where the position information includes the two-dimensional pixel coordinates and the credibility score of the plant.

[0022] In one embodiment, the calibrating the suspected center point further includes:

[0023] The scene image with missing leaf data is input into a semi-supervised learning segmentation model, and the labeling information of the scene image is output. The semi-supervised learning segmentation model is trained with a small amount of labeled data and a large amount of unlabeled data.

[0024] In one embodiment, the outputting of the optimal grasping posture according to feature extraction and grasping posture simulation includes:

[0025] Based on geometric and semantic constraints, key frames are selected and weighted calculations are performed in combination with time decay coefficients to optimize camera pose and scene parameters.

[0026] Use feature extraction network and analyze point cloud area;

[0027] Simulate the position and posture of different robot arm ends, calculate the grasping and lifting quality score under each posture, and output the optimal grasping posture.

[0028] In one embodiment, the method of determining the optimal storage point location and path planning of the plug tray based on spatial search and reinforcement learning path planning includes:

[0029] Obtain 3D spatial information of the shelves and new culture platforms, and filter out free areas based on geometric constraints;

[0030] The KD tree algorithm is used to divide the free area, and combined with the particle swarm optimization algorithm, the optimal storage point of the plug tray is determined;

[0031] Based on the A-star algorithm, calculate the shortest path in a static environment;

[0032] Based on monocular depth estimation and multi-view fusion depth completion, the plant 3D positioning technology can determine the point cloud information of target seedlings and crops;

[0033] Dynamically adjust path planning based on the adaptive path planning system of the deep Q network.

[0034] The present invention also provides a sensing and sensing device for a plug tray, comprising:

[0035] The acquisition module is used to obtain the complete point cloud structure to be grasped based on the long-range RGBD prior area estimation and the close-range multi-view shooting of the robot arm RGBD camera;

[0036] An output module is used to output the optimal grasping posture according to feature extraction and grasping posture simulation;

[0037] The determination module is used to determine the optimal storage point position and path planning of the plug tray based on spatial search and reinforcement learning path planning.

[0038] The present invention further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements any of the above-mentioned hole tray perception and sensing methods when executing the computer program.

[0039] The present invention also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the sensing and detecting method of the hole tray as described above is implemented.

[0040] The above-mentioned perception and sensing method, device and electronic equipment of the plug tray adopts a long-range RGBD prior area to determine the suspected area, and then adopts close-range multi-view shooting in the suspected area, so as to facilitate the completion of the point cloud and assist positioning, so as to visualize the plug tray completely covered by densely grown crops. When the plug trays are closely arranged and stacked on the original culture table, the feature extraction is combined with the iterative simulation method of grasping posture to facilitate the accurate judgment of the grasping key points and posture information of a single plug tray. At the same time, through the convenience of spatial search combined with reinforcement learning path planning, the optimal storage point position of the plug tray can be quickly selected, and the plug tray can be transferred according to the planned path.

[0041] For cultivated plants that grow densely and obstruct the seedlings themselves, a feature extraction network based on the attention mechanism is used, and image samples under different lighting conditions, different shooting angles and different backgrounds are collected to improve the accuracy of semantic segmentation; to address the high cost of labeling training data for instance segmentation, a semi-supervised approach is used to solve the problem of too little image labeling data for existing seedlings and seedling cultivation scenes; self-supervised monocular depth estimation technology is introduced to complete the coarse positioning of target crops and seedlings; then, RGBD information fusion depth completion and multi-view fusion three-dimensional semantic reconstruction based on deep learning are performed to perform three-dimensional registration of the target and enhance the accuracy of target positioning; based on the feature extraction of the above-mentioned 3D positioning segmentation point cloud results, the most suitable grasping posture is generated end-to-end to improve the success rate of precise seedling transplanting and movement. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 It is a schematic diagram of the flow of the sensing and sensing method of the hole tray in one embodiment;

[0044] Figure 2 A schematic diagram of a process for obtaining a complete point cloud structure to be captured in one embodiment;

[0045] Figure 3 A flowchart of hole tray perception and positioning under severe occlusion conditions based on two-stage step-by-step reasoning in one embodiment;

[0046] Figure 4 A schematic diagram of a process for obtaining remote RGBD information in an embodiment;

[0047] Figure 5 A schematic diagram of a process for calibrating a suspected center point in an embodiment;

[0048] Figure 6 A schematic diagram of a process of outputting an optimal grasping posture in one embodiment;

[0049] Figure 7 A schematic diagram of a process for determining the optimal storage point position of a plug tray and path planning in an embodiment;

[0050] Figure 8 A flowchart of hole tray placement point planning based on spatial search and reinforcement learning path planning in one embodiment;

[0051] Fig. 9 is a schematic diagram of the structure of a sensing and sensing device of a hole tray in one embodiment;

[0052] Fig.10 The figure is a diagram showing the internal structure of an electronic device according to an embodiment. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] The traditional sensing and positioning system of the plug tray has the following disadvantages:

[0055] 1. Insufficient accuracy and robustness of image recognition

[0056] Strong dependence on lighting: Image recognition systems usually rely on visual sensors (such as RGB cameras, depth cameras, etc.), but the recognition accuracy drops significantly under different lighting conditions, especially in environments with large lighting changes. For the complex background in the plug tray planting environment, the existing image recognition algorithms often cannot stably identify the target.

[0057] Object occlusion problem: The growth of plants may cause the target object to be partially occluded. The existing image recognition system has poor robustness when dealing with partially occluded or overlapping objects, which affects accurate recognition and positioning.

[0058] Slow recognition speed: When the image processing and recognition algorithms run at high precision, the amount of calculation is large, which may lead to slow image recognition and affect the real-time performance of the system, especially when dealing with dynamic or complex scenes.

[0059] 2. Robotic arm accuracy problem

[0060] Insufficient positioning and grasping accuracy: In actual applications, the end effector of the robot arm (such as the gripper) may not be able to accurately locate and grasp the target object due to sensor errors, deviations in the mechanical structure, etc. Especially when performing small or repetitive tasks, accuracy deviations can lead to job failures or low production efficiency.

[0061] Insufficient force feedback: Some robotic arm systems lack an effective force feedback mechanism and are unable to sense the contact status with the target object in real time, resulting in inaccurate grasping force control, which can easily cause plant damage or operation failure.

[0062] Insufficient rigidity and flexibility: The rigid design of the robotic arm may limit its adaptability in complex tasks, especially in environments that require flexible operations. The robotic arm has difficulty adapting to the processing needs of different shapes or materials.

[0063] 3. Insufficient trajectory planning of the robotic arm

[0064] Low efficiency of trajectory planning algorithms: Existing trajectory planning algorithms are inefficient when processing complex operation paths, especially when it comes to complex path planning involving multiple points and multiple postures, which may result in long calculation times and affect the overall operation speed. For heterogeneous tasks, existing trajectory planning algorithms lack flexibility and are difficult to adjust in real time.

[0065] Weak collision detection and obstacle avoidance capabilities: Although some systems have basic obstacle avoidance functions, in complex environments, the robot's trajectory planning may lack sophisticated collision detection capabilities, which may lead to collisions or misoperations during operations, affecting operational efficiency and safety.

[0066] Poor adaptability to environmental changes: Existing trajectory planning technologies usually assume that the working environment is relatively stable. However, in actual planting environments, factors such as plant growth and changes in obstacles will affect the accuracy and efficiency of planning. Existing systems have poor adaptability in this regard.

[0067] 4. Low efficiency and slow system response

[0068] Multi-task scheduling is not intelligent: The existing system has low scheduling efficiency when facing multiple tasks in parallel, especially in complex loading and unloading, planting and handling tasks. The system often relies on manual intervention or simple automation processes, resulting in reduced work efficiency of the production line.

[0069] Feedback lag between perception and execution: There is a time delay in the feedback between the perception system (such as image recognition, sensor data) and the robot arm execution, which affects the system's real-time response ability. Especially in scenarios where actions need to be adjusted quickly, the system's response speed is insufficient, reducing efficiency.

[0070] Data processing bottleneck: The processing of image data, sensor data, etc. may require high computing power, but existing hardware facilities may have processing bottlenecks, resulting in slow overall system response. Especially under high load or large-scale tasks, processing delays will significantly reduce production efficiency.

[0071] Combine the following Figure 1-Figure 10 The present invention describes the sensing and detecting method, device and electronic device of the plug tray.

[0072] like Figure 1 As shown, in one embodiment, a method for sensing a hole tray comprises the following steps:

[0073] Step S100, based on long-range RGBD priori area estimation and close-range multi-view shooting of the robot arm RGBD camera, a complete point cloud structure to be captured is obtained.

[0074] RGBD refers to an image or data format that contains color (RGB) and depth (Depth) information. RGB refers to the three color channels of the image, red (Red), green (Green) and blue (Blue), which can be combined to represent a full-color image; the D (Depth) depth channel contains the depth information of each pixel in the scene, usually expressed in physical distance from the camera (for example, millimeters or meters). The depth information allows the understanding of the three-dimensional structure in the scene.

[0075] For the plug tray platform that has completed cultivation and is waiting to be harvested, the plug tray is completely covered by densely grown crop plants and is invisible. In addition, the densely cultivated crop plants overlap and block each other, making it difficult to obtain their complete 3D information and to label and semantically segment a single plant on the RGB image. Therefore, in response to this situation, a remote RGBD prior area estimation combined with a robotic arm RGBD camera close-range multi-view shooting method is used to facilitate point cloud completion and auxiliary positioning.

[0076] Step S200: outputting the optimal grasping posture according to feature extraction and grasping posture simulation.

[0077] By selecting the optimal gripping posture, accurate tray gripping is facilitated.

[0078] Step S300, based on spatial search and reinforcement learning path planning, determine the optimal storage point position and path planning of the plug tray.

[0079] For the plug trays that have been grasped, they need to be placed on the shelf where the robot itself is placed, that is, taken out from the shelf and moved to the new workbench. The optimal placement point needs to be planned so that the plug trays can be closely arranged at the target position. To this end, through spatial search combined with the convenience of reinforcement learning path planning, the optimal storage point position of the plug tray can be quickly selected, and the plug trays can be transferred according to the planned path.

[0080] The perception and sensing method of the plug tray adopts a long-range RGBD prior area to determine the suspected area, and then adopts close-range multi-view shooting in the suspected area, so as to facilitate the completion of the point cloud and assist positioning, so as to visualize the plug tray completely covered by densely grown crops. When the plug trays are closely arranged and stacked on the original culture table, the feature extraction is combined with the iterative simulation method of grasping posture to facilitate the accurate judgment of the grasping key points and posture information of a single plug tray. At the same time, through the convenience of spatial search combined with reinforcement learning path planning, the optimal storage point position of the plug tray can be quickly selected, and the plug tray can be transferred according to the planned path.

[0081] In this embodiment, see Figure 2 and Figure 3 , based on the long-range RGBD prior area estimation and the close-range multi-view shooting of the robot arm RGBD camera, the complete point cloud structure to be grasped is obtained, including the following steps:

[0082] Step S110, acquiring remote RGBD information, and performing denoising and smoothing processing on the remote RGBD information.

[0083] By denoising and smoothing the long-range RGBD information, the accuracy of the RGBD information can be easily improved. LiDAR can be used instead of RGBD camera to obtain more accurate depth information and distance measurement.

[0084] Step S120, roughly dividing the area according to the size of the single hole tray platform, extracting the segmentation map, and calibrating the suspected center point.

[0085] In the practical stage, after obtaining the corresponding long-range RGBD information through the camera and performing denoising and smoothing, combined with the size of a single plug tray platform obtained in advance, the plant area photographed from the edge is roughly divided according to the length and width information to extract the segmentation map, which is convenient for giving the suspected single plug tray center point area closest to the working area of ​​the robotic arm, so that the robotic arm can approach this area for fine shooting.

[0086] Step S130, acquiring images shot at multiple angles in the suspected center point area, where the suspected center point area includes the outer edge and the outer long side area of ​​the hole plate.

[0087] By making the robotic arm mainly take multi-angle shots in the area suspected to be the center of the hole tray, further multi-view reconstruction is performed on some visible areas, such as the long sides of the hole tray that are relatively far outward, to facilitate the acquisition of detailed 3D information.

[0088] Step S140: reconstruct the point cloud data from multiple perspectives to generate a complete 3D point cloud.

[0089] Multi-view reconstruction is performed within the rectangular block area given by the above coarse positioning to facilitate the generation of a relatively complete point cloud area.

[0090] Step S150, based on the Nerf neural network, the location, radius, color and opacity information of the scene point are used as Gaussian field parameters for rendering, and a fine 3D structure is obtained.

[0091] The Nerf neural network is used to render the point cloud structure into a detailed reconstruction, making it easier to obtain the complete point cloud structure to be captured.

[0092] In this embodiment, see Figure 4 , obtain remote RGBD information, including the following steps:

[0093] Step S111, acquiring leaf image data of plants at different growth stages.

[0094] The leaf images cover images under different lighting conditions, different shooting angles and different backgrounds. These image datasets provide diverse training samples, which help improve the generalization ability of the model.

[0095] Step S112, marking key point information of the connecting points between seedlings and the center points of crop plants in the leaf image.

[0096] It is convenient to annotate images in detail.

[0097] Step S113, performing enhancement processing of rotation, flipping, scaling and color adjustment on the blade image data.

[0098] It is convenient to enhance the robustness of the model. These data enhancement technologies can simulate various transformations in practical applications and improve the performance of the model in complex environments.

[0099] In this embodiment, see Figure 5 , calibrating the suspected center point includes the following steps:

[0100] Step S121, acquiring a scene image of dense plants.

[0101] The dense vegetation scene is captured by a camera or sensor, usually as an RGB image.

[0102] Step S122: extracting a scene image of a single plant from dense plants based on the attention mechanism.

[0103] In order to solve the problem of poor instance segmentation under complex background and occlusion conditions, an attention mechanism will be added to the network architecture. Through weighted feature selection, the model's attention to important features will be effectively improved, irrelevant background and noise will be suppressed, thereby improving segmentation accuracy and accurately extracting individual seedlings from dense seedling plants at high speed. The attention mechanism was first applied in the field of natural language processing to enhance the model's ability to focus on key information, and was subsequently introduced into computer vision tasks. After integrating the attention mechanism into the backbone network of the segmentation network, it can improve feature expression capabilities, enable the model to more accurately distinguish between instances and backgrounds, optimize the ability to understand object relationships and layouts, and enhance boundary refinement, thereby improving segmentation accuracy. It is expected that by integrating it into the segmentation network, the model's adaptability in complex backgrounds will be enhanced, seedlings will be accurately segmented, and the accuracy and efficiency of the seedling capture step will be ensured.

[0104] Step S123, input the scene image of a single plant into the YOLOv8-Pose model, and output the position information of the captured key points, where the position information includes the two-dimensional pixel coordinates and credibility score of the plant.

[0105] YOLOv8-Pose adds a key point detection module based on YOLOv8, which can identify the key nodes of objects (such as human joints, plant nodes, etc.), thereby achieving more detailed recognition and positioning. To improve the detection performance of the model, the study also introduced depth information, combining depth data with RGB images, and optimizing the model's feature learning and overall framework.

[0106] The YOLOv8-Pose network model is used to detect key points, accurately distinguish between seedlings and plants, and mark the key points for grasping. At the same time, a prediction branch for grasping decisions is added. Combined with the multi-task learning framework, the model can simultaneously perform target detection and plant grasping feasibility assessment, thereby realizing automatic recognition of grasping points and guiding the robot to perform precise grasping operations.

[0107] In this embodiment, calibrating the suspected center point also includes the following steps:

[0108] Step S124, inputting the scene image with missing blade data into a semi-supervised learning segmentation model, and outputting the labeling information of the scene image, wherein the semi-supervised learning segmentation model is trained with a small amount of labeled data and a large amount of unlabeled data.

[0109] Traditional instance segmentation methods rely on a large amount of labeled data for training. Although they perform well on large-scale labeled data sets, their performance will drop significantly when there is insufficient labeled data. In complex agricultural scenarios such as seedling loading and unloading, the labeling of the center point of the plant, the outline selection and labeling of the entire plant target, etc. are difficult and time-consuming, and are easily affected by subjective factors. For this reason, semi-supervised learning has become an effective solution. It combines a small amount of labeled data with a large amount of unlabeled data for training, which greatly improves the segmentation effect with limited data resources. The specific plan includes: first prepare labeled data and unlabeled data. The labeled data selects a small number of typical target crops and seedling pictures from various perspectives on various seedlings for manual labeling. The unlabeled data covers different time periods and different angles, with the focus on vegetable seedlings taken from a bird's-eye view, to increase the diversity of the data.

[0110] In terms of model selection, a segmentation model suitable for semi-supervised learning is used, such as a method based on consistency regularization and pseudo-label self-training. The training process is divided into four steps:

[0111] 1. In the pre-training stage, a small amount of labeled data is used to pre-train the model to obtain basic segmentation capabilities of seedlings and plant center points.

[0112] 2. Consistency regularization: by performing image enhancement on unlabeled data at various capture times and weather environments, the model ensures the consistency of the prediction results of the enhanced version.

[0113] 3. Pseudo-label generation: use the model to predict unlabeled data to generate pseudo-labels as additional supervisory signals for training.

[0114] 4. Joint training, combining labeled data and pseudo-labeled data for model optimization. When optimizing the model, combine supervised and unsupervised loss functions, design a comprehensive loss function, including cross entropy loss and consistency loss, and adjust hyperparameters such as learning rate and consistency regularization coefficient through cross-validation to ensure optimal model performance.

[0115] In this embodiment, see Figure 3 and Figure 6 ,According to the feature extraction and grasping posture simulation, the optimal grasping posture is output, including the following steps:

[0116] Step S210, based on geometric and semantic constraints, key frames are selected and weighted calculation is performed in combination with the time decay coefficient to optimize the camera pose and scene parameters.

[0117] By selecting key frames according to geometric and semantic constraints and weighting them with time decay coefficients, the final point cloud is mapped as a whole. Then, the scene parameters are optimized according to the estimated camera pose, which facilitates high-precision 3D mapping and semantic segmentation, and then facilitates the transfer of a relatively complete plant tray system to the grasping point reasoning structure.

[0118] Step S220, using a feature extraction network and analyzing the point cloud area.

[0119] The feature extraction network is used to simulate the sampling of different robot arm end positions and postures close to the bottom of the point cloud area and extract part of the point cloud under the corresponding perspective, so as to output the corresponding grasping and lifting quality score.

[0120] Step S230, simulating the positions and postures of different robot arm ends, calculating the grasping and lifting quality score under each posture, and outputting the optimal grasping posture.

[0121] Through iterative simulation, the optimal grasping posture can be easily selected. Combined with high-precision sensors (such as force sensors, visual sensors, laser rangefinders, etc.), the posture and position of the robotic arm can be accurately sensed to ensure the high precision of the grasping action.

[0122] In this embodiment, see Figure 7 and Figure 8 , based on spatial search and reinforcement learning path planning, determine the optimal storage point location and path planning of the plug tray, including the following steps:

[0123] Step S310, obtaining 3D spatial information of the shelf and the new culture platform, and screening out free areas according to geometric constraints.

[0124] By screening out the free areas, it is easy to determine the optional space for the destination of the plug tray transfer.

[0125] Step S320, using the KD tree algorithm to divide the free area, and combining it with the particle swarm optimization algorithm to determine the optimal storage point position of the plug tray.

[0126] The KD tree is an efficient spatial segmentation data structure that can quickly search for qualified free areas in three-dimensional space and provide data support for subsequent path planning. The particle swarm optimization algorithm finds the optimal solution by simulating the movement of particles in the search space. The system calculates the space utilization, handling efficiency and environmental adaptability of each candidate storage point in multiple iterations, and finally selects the optimal storage point location.

[0127] In the storage point optimization process, genetic algorithm can be used as an alternative to particle swarm optimization algorithm to provide different optimization paths.

[0128] Step S330, calculating the shortest path in a static environment based on the A-star algorithm.

[0129] The A-star algorithm can calculate the shortest path in a static environment, avoid known obstacles, and provide a basis for path planning. The heuristic search strategy of the A-star algorithm makes path planning more efficient in a static environment.

[0130] Step S340, based on the plant 3D positioning technology of monocular depth estimation and multi-view fusion depth completion, determine the point cloud information of the target seedlings and crops.

[0131] In order to ensure the accuracy of seedling positioning and movement, two-stage target positioning is planned. First, the seedling platform is roughly positioned based on RGB shooting by the onboard camera, and the approximate position is determined based on the empty seedling segmentation results or the crop plant segmentation and counting results based on the fast monocular depth estimation combined with the semantic segmentation of the target RGB image; secondly, when the robotic arm approaches the target seedling and prepares for transplanting, the 3D camera at the end of the robotic arm performs depth completion based on multi-view fusion, constructs a fine point cloud of the seedling crop system, assists in the precise positioning of the target seedling clamping point, assists in the generation of the grasping point, and matches the previous depth estimation of this part based on the close-range 3D shooting information to perform the next step of fine adaptive handling operations.

[0132] 1. Coarse positioning of seedling platform based on self-supervised monocular depth estimation

[0133] In the coarse positioning stage, a relatively complete and accurate depth map is estimated through self-supervised monocular depth estimation to replace the incomplete and less reliable depth map obtained by the depth camera, thereby improving the accuracy of coarse positioning of seedling targets. Since the Manydepth framework proposed by Watson et al. works well on open source datasets and innovatively uses adaptive cost volumes to alleviate the scale ambiguity problem during self-supervised training, this study will use the Manydepth framework to achieve depth estimation of the target seedlings to be moved.

[0134] Manydepth uses the geometric constraints between consecutive time frames as supervision signals to transform monocular depth estimation into an image reconstruction problem; and measures the geometric compatibility of pixel depth between the current frame and the adjacent frame through adaptive cost volume, overcoming the defect of scale ambiguity in unsupervised training. Manydepth also introduces binary masks and teacher-student training models to mark the credibility of cost volumes in different regions, and uses the single-frame depth estimation network (teacher model) as the standard in untrustworthy areas to prevent overfitting of the cost volume. In addition, it also solves the problem of missing the previous frame and camera stillness during training. Finally, the cost volume is sent to the encoder-decoder, and the output disparity is transformed to obtain the estimated depth. During the training and evaluation process, Manydepth introduces image reconstruction loss, disparity smoothing loss and consistency loss, among which the consistency loss makes the network ignore the unreliable cost volume and refer to the estimated results of the single-frame depth estimation network.

[0135] Before offline training, this project will first simulate the robot's movements through a monocular camera, collect pixel-aligned RGB images and depth image sequences of the seedling targets to be transplanted, and create a data set. Then, in the process of training the model, the effectiveness of the feature extraction network using pre-trained weights, cost volume, and single-frame depth estimation network (teacher model) will be verified, and compared horizontally with other methods. After completing offline training and estimating the depth map based on the Manydepth framework, combined with the previous segmentation results of seedling targets of various sizes on the RGB map, the depth corresponding to the pixels in the segmentation box can be obtained, thereby achieving coarse positioning of the seedling targets. This method can reduce the computational cost of annotation and depth data processing during training.

[0136] 2. Fine positioning based on depth completion and multi-view fusion

[0137] When the robot arm approaches the target, fine positioning requires more accurate depth information than the coarse positioning stage. Therefore, this project will use depth completion to assist in the accurate adaptive positioning of seedlings of different sizes. When performing fine positioning, the end of the robot arm should be close to the target seedling, and based on the close-range 3D shooting information, match the previous depth estimation of the part, and perform the next step of fine segmentation.

[0138] Referring to the CompletionFormer framework proposed by Zhang et al., local features and global context information are introduced simultaneously in the task of depth completion, which can achieve better depth completion. The algorithm takes an incomplete depth map and the corresponding RGB image as input, and extracts multi-scale features through the U-Net backbone network constructed by the JCAT block, realizing the interaction of depth and image information at multiple scales, so that features at different stages can be fused at full resolution. Then, the decoder will provide full-resolution features for initial depth prediction, and finally use the spatial propagation network (SPN) to refine the initial estimation results.

[0139] In addition, the accuracy of 3D target representation can be further improved by performing point cloud registration through multi-view fusion, and the semantic segmentation information can be directly projected to the 3D point cloud. It is planned to use an explicit isotropic 3D Gaussian function to represent the position, radius, color, opacity and other information of each scene point, project the 3D Gaussian distribution to the image plane, calculate the color and depth of the pixel according to the transparency and influence range of the Gaussian distribution, and use the maximum volume rendering method to mix the pixels, so as to construct the overall 3D information of the target seedlings and the crops that may exist on them according to the multiple perspectives taken by the end of the robotic arm, and construct the overall detailed 3D positioning of the loading and unloading movement. The system framework can complete the tracking and mapping of the target based on the Gaussian field. Tracking is to estimate the camera pose of each frame of the image and fix the scene parameters. At this time, the system optimizes the camera pose by minimizing the loss function between the rendered image and the real image. The loss function is:

[0140]

[0141] Among them, Σ pix It is the sum of each pixel in the image, which means that the loss function evaluates each pixel in the image. pix >T S : is a threshold condition, which means that the corresponding loss will be calculated only when the transparency (Sil) of the pixel is greater than a certain threshold Ts. That is, the loss is calculated only for significant pixels (for example, pixels with high transparency); D ,λ C ,λ S : These are weight coefficients that control the contribution of different loss terms (depth, color, semantic) to the total loss function. Their role is to balance different types of errors and ensure the relative importance of each part in the optimization process; The ground truth values ​​(Ground Truth) representing depth, color, and semantic labels are the values ​​in the real world and are used to compare with the rendering results; D pix , C pix , S pix: respectively represent the depth, color and semantic label values ​​calculated by the model, that is, the estimated values ​​in the current image; These represent the errors of depth, color, and semantic labels, i.e., the absolute difference between the ground truth and the model’s prediction. The loss function optimizes the model by minimizing these errors; depth error This measure is the error of the depth map, which is crucial for optimizing the camera pose and accurately reconstructing 3D objects; color error This measure of the difference in color between the rendered image and the real image helps ensure color consistency; semantic error This measure measures the difference in semantic segmentation information to ensure that the semantic content (such as object category, etc.) of the rendered image is consistent with the real image.

[0142] The system then selects keyframes based on geometric and semantic constraints and weights them in combination with the time decay coefficient to improve the quality of the map. When building the map, the scene parameters are optimized based on the estimated camera pose to achieve high-precision 3D mapping and semantic segmentation. Gaussian densification is introduced during the mapping process, that is, new Gaussian points are introduced into the scene to improve the reconstruction accuracy; at the same time, Gaussian parameters are optimized by minimizing the loss function between the rendered image and the real image, including RGB-D and semantic information. Accordingly, this method is based on the explicit representation of the 3D Gaussian field, and can simultaneously achieve high-precision scene reconstruction and semantic segmentation; since the channel directly contains the semantic information of the 3D scene, specific objects in the scene can be easily selected and manipulated; and by editing the Gaussian parameters, operations such as moving, rotating, and removing objects can be achieved, enhancing the real-time and sensitivity of 3D reconstruction.

[0143] Based on the point cloud information of the target seedlings and crops formed by the two-step positioning, their precise positioning is completed; and this point cloud information is used as input to sample the point cloud information of the seedlings after semantic segmentation, and a feature extraction network with PointNet as the backbone is used to extract features from the viewpoint cloud, and the corresponding transplanting stability quality score is output. Finally, the posture with the highest score is selected to obtain the most suitable posture information of the grasping or clamping point.

[0144] Step S350, dynamically adjusting the path planning according to the adaptive path planning system of the deep Q network.

[0145] The deep reinforcement learning (DQN) method uses a deep Q network to dynamically adjust path planning. DQN learns about dynamically changing obstacles and unknown situations in the environment, allowing the robot to adjust the path in real time to avoid collisions and improve work efficiency. The DQN network optimizes path selection under different states through state and action pairs, thereby achieving adaptive adjustment of path planning.

[0146] In path planning, other reinforcement learning algorithms such as PPO (Proximal Policy Optimization) can replace DQN to improve the convergence and stability of path planning.

[0147] The present invention focuses on the grabbing, transferring and restacking of the plug trays on the culture table, and the perception and positioning of the plug trays are the key difficulties in achieving this goal. Since the plug trays are closely arranged and stacked on the original culture table, it is difficult for the system to accurately determine the grabbing key points and posture information of a single plug tray. At the same time, when the plug trays are densely placed, the edge information used to segment and locate a single plug tray is often too small to be extracted.

[0148] In order to solve the problem of sensing and positioning the empty trays to be loaded in densely placed conditions, the robot needs to perform image segmentation on the entire tray culture area when lifting the trays to the culture area and transplanting area, identify independent trays, and determine the center point of empty trays or early cultivation trays. At the same time, accurately identifying empty trays, growing crops, and key edge points is also the focus of this project.

[0149] Compared with empty and growing culture trays, the method of relying on RGB images alone to identify the trays that have been cultured and are waiting to be harvested is no longer applicable. This is because these trays are completely covered by densely growing crops and are difficult to visualize. At the same time, the point cloud data generated by traditional RGB-D cameras when shooting plants often has serious information missing, so the method of projecting 2D semantic segmentation to 3D cannot accurately derive the appropriate grasping points. We will focus on developing an active multi-view perception and image 3D completion method to perform remote rough estimation and close-range correction of severely occluded point cloud areas. This method first obtains more complete 3D information reasoning, combines the length and width of the known tray platform in advance, and performs segmentation of the suspected single platform area and determination of the center point position, completing the rough segmentation and rough positioning of the tray. On this basis, combined with the Nerf algorithm, multi-view image shooting is performed near the suspected center point to reconstruct the corresponding point cloud information. This method effectively fills in the 3D information of the platform of the harvested plug tray near the long side, and finally hands it over to the grasping point generator for grasping point derivation, solving the problem that densely placed plug trays are difficult to identify one by one in severe occlusion scenes.

[0150] In addition, the robot chassis is equipped with a cabinet-type shelf structure for transporting plug trays, and new plug tray placement positions also need to be arranged on the target's new culture table. After completing the lifting of the plug tray, the robot needs to plan to arrange multiple plug trays tightly again to complete the stacking task. In this process, effective identification must be performed based on the shelves, new culture tables, and already placed plug trays to determine the location of the most densely arranged stacking points. To this end, the robot needs to obtain and identify the point cloud information of the shelves and new culture tables through RGB-D information, and use the optimal path planning algorithm to derive the most suitable placement position for the plug trays at the moment, ensuring that the robot can efficiently complete the automatic stacking task. Through the optimal placement point search, the system will locate the target storage point based on the real-time updated RGB-D information, so that the crops can be tightly arranged at the new cultivation point and shelves, improving overall work efficiency.

[0151] The sensing and sensing method of the plug tray provided by the present invention has the following effects:

[0152] 1. Image recognition advantages:

[0153] High-precision recognition: The image recognition system can accurately detect and identify the specific position, shape and direction of the hole tray, providing reliable data for the accurate grasping of the robotic arm.

[0154] Strong environmental adaptability: Through deep learning and computer vision technology, it can effectively identify and adjust under complex environmental conditions, which helps to improve the stability and reliability of the system.

[0155] Real-time feedback: The image recognition system can monitor the working status in real time, provide timely feedback on position and posture information, enable rapid adjustments, and avoid incorrect grasping or dropping.

[0156] 2. Advantages of robotic arm precision:

[0157] High-precision positioning: Combined with high-precision sensors (such as force sensors, visual sensors, laser rangefinders, etc.), it can accurately sense the posture and position of the robotic arm to ensure high-precision grasping action.

[0158] Strong adaptability: The robotic arm can automatically make fine adjustments based on the real-time feedback from the perception system to adapt to different hole trays and environmental changes to ensure stability.

[0159] Reduce human errors: High-precision control through automated systems reduces errors caused by manual operations and improves overall operation accuracy.

[0160] 3. Advantages of robot trajectory planning:

[0161] Dynamic trajectory planning: The system can dynamically adjust the trajectory based on real-time data, target position and path obstacles to optimize the working path of the robot arm and avoid unnecessary waiting time and collision risks.

[0162] Efficient path generation: Using advanced path planning algorithms (such as A-star algorithm, RRT algorithm, etc.), the shortest or most optimized path can be generated to improve the working efficiency of the robotic arm.

[0163] Task flexibility: Through flexible trajectory planning, the robot arm can perform a variety of tasks (such as loading and unloading, sorting, sorting, etc.), and has strong versatility and adaptability.

[0164] 4. Efficiency improvement advantages:

[0165] Automation improves production efficiency: The efficient collaboration of the robot arm’s perception and sensor system makes the entire operation process more automated, reducing the need for human intervention and improving work efficiency.

[0166] Multi-task parallelism: The system can process multiple trays or tasks at the same time, reducing the time wasted on single-task operations and further improving overall efficiency.

[0167] Reduce waiting time: Real-time perception and feedback mechanism reduces the waiting time of the robot arm during operation, and improves the overall response speed and operation rate of the system.

[0168] The sensing and sensing device of the plug tray provided by the present invention is described below. The sensing and sensing device of the plug tray described below and the sensing and sensing method of the plug tray described above can be referred to each other.

[0169] like Fig. 9 As shown, in one embodiment, a sensing and detecting device for a hole tray includes an acquisition module 910 , an output module 920 and a determination module 930 .

[0170] The acquisition module 910 is used to acquire a complete point cloud structure to be captured based on long-range RGBD priori area estimation and close-range multi-view shooting of the robot arm RGBD camera.

[0171] The output module 920 is used to output the optimal grasping posture according to feature extraction and grasping posture simulation.

[0172] The determination module 930 is used to determine the optimal storage point position and path planning of the plug tray based on spatial search and reinforcement learning path planning.

[0173] In this embodiment, the acquisition module 910 is specifically used for:

[0174] Obtain remote RGBD information and perform denoising and smoothing processing on the remote RGBD information;

[0175] Roughly divide the area according to the size of a single plug tray platform, extract the segmentation map, and calibrate the suspected center point;

[0176] Acquire images taken at multiple angles in the suspected center point area, where the suspected center point area includes the outer edge and the outer long side area of ​​the hole plate;

[0177] Reconstruct point cloud data from multiple perspectives to generate a complete 3D point cloud;

[0178] Based on the Nerf neural network, the position, radius, color and opacity information of scene points are rendered as Gaussian field parameters to obtain a fine 3D structure.

[0179] In this embodiment, remote RGBD information is obtained, specifically for:

[0180] Acquire leaf image data of plants at different growth stages;

[0181] Mark the key point information of the connection between seedlings and the center point of the crop plant in the leaf image;

[0182] The leaf image data is enhanced by rotation, flipping, scaling and color adjustment.

[0183] In this embodiment, the suspected center point is calibrated, specifically for:

[0184] Acquire a scene image of dense vegetation;

[0185] Based on the attention mechanism, the scene image of a single plant is extracted from dense plants;

[0186] The scene image of a single plant is input into the YOLOv8-Pose model, and the location information of the captured key points is output. The location information includes the two-dimensional pixel coordinates and credibility score of the plant.

[0187] In this embodiment, the calibration of the suspected center point is also specifically used for:

[0188] The scene image with missing leaf data is input into the semi-supervised learning segmentation model, and the labeling information of the scene image is output. The semi-supervised learning segmentation model is trained with a small amount of labeled data and a large amount of unlabeled data.

[0189] In this embodiment, the output module 920 is specifically used for:

[0190] Based on geometric and semantic constraints, key frames are selected and weighted calculations are performed in combination with time decay coefficients to optimize camera pose and scene parameters.

[0191] Use feature extraction network and analyze point cloud area;

[0192] Simulate the position and posture of different robot arm ends, calculate the grasping and lifting quality score under each posture, and output the optimal grasping posture.

[0193] In this embodiment, the determination module 930 is specifically configured to:

[0194] Obtain 3D spatial information of the shelves and new culture platforms, and filter out free areas based on geometric constraints;

[0195] The KD tree algorithm is used to divide the free area, and combined with the particle swarm optimization algorithm, the optimal storage point of the plug tray is determined;

[0196] Based on the A-star algorithm, calculate the shortest path in a static environment;

[0197] Based on monocular depth estimation and multi-view fusion depth completion, the plant 3D positioning technology can determine the point cloud information of target seedlings and crops;

[0198] Dynamically adjust path planning based on the adaptive path planning system of the deep Q network.

[0199] The perception and sensing device of the plug tray uses a feature extraction network based on the attention mechanism, as well as image sample collection covering different lighting conditions, different shooting angles and different backgrounds, to improve the accuracy of semantic segmentation for cultivated plants that grow densely and obscure the seedlings themselves. In order to address the high cost of labeling training data for instance segmentation, the semi-supervised idea is adopted to solve the problem of too little labeling data for existing seedlings and plant images in seedling cultivation scenes.

[0200] Self-supervised monocular depth estimation technology is introduced to complete the coarse positioning of target crops and seedlings. Then, RGBD information fusion depth completion and multi-view fusion 3D semantic reconstruction based on deep learning are carried out to perform three-dimensional alignment of the target and enhance the accuracy of target positioning. Based on the feature extraction of the above-mentioned 3D positioning segmentation point cloud results, the most suitable grasping posture is generated end-to-end to improve the success rate of precise seedling transplanting and movement.

[0201] Fig.10 An example of a physical structure diagram of an electronic device is shown. The electronic device may be a smart terminal, and its internal structure diagram may be as follows: Fig.10 As shown. The electronic device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a perception and sensing method of a hole tray is implemented, and the method includes:

[0202] Based on long-range RGBD prior area estimation and close-range multi-view shooting by the robot arm RGBD camera, the complete point cloud structure to be grasped is obtained;

[0203] Output the optimal grasping posture based on feature extraction and grasping posture simulation;

[0204] Based on spatial search and reinforcement learning path planning, the optimal storage point location and path planning of the plug tray are determined.

[0205] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present invention, and does not constitute a limitation on the electronic device to which the scheme of the present invention is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0206] On the other hand, the present invention also provides a computer storage medium storing a computer program, which implements a method for sensing a hole tray when the computer program is executed by a processor, the method comprising:

[0207] Based on long-range RGBD prior area estimation and close-range multi-view shooting by the robot arm RGBD camera, the complete point cloud structure to be grasped is obtained;

[0208] Output the optimal grasping posture based on feature extraction and grasping posture simulation;

[0209] Based on spatial search and reinforcement learning path planning, the optimal storage point location and path planning of the plug tray are determined.

[0210] In another aspect, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and when the processor executes the computer instructions, a method for sensing a hole tray is implemented, the method comprising:

[0211] Based on long-range RGBD prior area estimation and close-range multi-view shooting by the robot arm RGBD camera, the complete point cloud structure to be grasped is obtained;

[0212] Output the optimal grasping posture based on feature extraction and grasping posture simulation;

[0213] Based on spatial search and reinforcement learning path planning, the optimal storage point location and path planning of the plug tray are determined.

[0214] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory.

[0215] By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0216] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0217] The above-described embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the attached claims.

Claims

1. A method for sensing a hole tray, characterized in that: The method comprises: Based on long-range RGBD prior area estimation and close-range multi-view shooting by the robot arm RGBD camera, the complete point cloud structure to be grasped is obtained; Output the optimal grasping posture based on feature extraction and grasping posture simulation; Based on spatial search and reinforcement learning path planning, the optimal storage point location and path planning of the plug tray are determined.

2. The method for sensing a hole tray according to claim 1, characterized in that: The method of obtaining a complete point cloud structure to be captured based on long-range RGBD priori area estimation and close-range multi-view shooting by the robot arm RGBD camera includes: Obtain remote RGBD information and perform denoising and smoothing processing on the remote RGBD information; Roughly divide the area according to the size of a single plug tray platform, extract the segmentation map, and calibrate the suspected center point; Acquire images taken at multiple angles in the suspected center point area, where the suspected center point area includes the outer edge and the outer long side area of ​​the hole plate; Reconstruct point cloud data from multiple perspectives to generate a complete 3D point cloud; Based on the Nerf neural network, the position, radius, color and opacity information of scene points are rendered as Gaussian field parameters to obtain a fine 3D structure.

3. The method for sensing a hole tray according to claim 2, characterized in that: The obtaining of remote RGBD information includes: Acquire leaf image data of plants at different growth stages; Mark the key point information of the connection between seedlings and the center point of the crop plant in the leaf image; The leaf image data is enhanced by rotation, flipping, scaling and color adjustment.

4. The method for sensing a hole tray according to claim 3, characterized in that: The calibrating of the suspected center point comprises: Acquire a scene image of dense vegetation; Based on the attention mechanism, the scene image of a single plant is extracted from dense plants; The scene image of the single plant is input into the YOLOv8-Pose model, and the position information of the captured key points is output, where the position information includes the two-dimensional pixel coordinates and the credibility score of the plant.

5. The method for sensing a hole tray according to claim 4, characterized in that: The calibrating of the suspected center point also includes: The scene image with missing leaf data is input into a semi-supervised learning segmentation model, and the labeling information of the scene image is output. The semi-supervised learning segmentation model is trained with a small amount of labeled data and a large amount of unlabeled data.

6. The method for sensing a plug tray according to any one of claims 1 to 5, characterized in that: The method of outputting the optimal grasping posture according to feature extraction and grasping posture simulation includes: Based on geometric and semantic constraints, key frames are selected and weighted calculations are performed in combination with time decay coefficients to optimize camera pose and scene parameters. Use feature extraction network and analyze point cloud area; Simulate the position and posture of different robot arm ends, calculate the grasping and lifting quality score under each posture, and output the optimal grasping posture.

7. The method for sensing a hole tray according to claim 6, characterized in that: The method of determining the optimal storage point position and path planning of the plug tray based on spatial search and reinforcement learning path planning includes: Obtain 3D spatial information of the shelves and new culture platforms, and filter out free areas based on geometric constraints; The KD tree algorithm is used to divide the free area, and the particle swarm optimization algorithm is combined to determine the optimal storage point of the plug tray; Based on the A-star algorithm, calculate the shortest path in a static environment; Based on monocular depth estimation and multi-view fusion depth completion, the plant 3D positioning technology can determine the point cloud information of target seedlings and crops; Dynamically adjust path planning based on the adaptive path planning system of the deep Q network.

8. A sensing and sensing device for a plug tray, characterized in that: include: The acquisition module is used to obtain the complete point cloud structure to be grasped based on the long-range RGBD prior area estimation and the close-range multi-view shooting of the robot arm RGBD camera; An output module is used to output the optimal grasping posture according to feature extraction and grasping posture simulation; The determination module is used to determine the optimal storage point location and path planning of the plug tray based on spatial search and reinforcement learning path planning.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the sensing method for a plug tray according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the sensing method for a plug tray according to any one of claims 1 to 7 are implemented.

Images