Perception and sensing method, device and electronic equipment for plug trays

Abstract

The application relates to a kind of perception and sensing method, device and electronic equipment of plug, the method comprises: based on remote RGBD prior area estimation and mechanical arm RGBD camera close-range multi-view shooting, obtain complete to be grabbed point cloud structure;According to feature extraction and grabbing posture simulation, output optimal grabbing posture;Based on space search and reinforcement learning path planning, determine the optimal storage point position of plug and path planning.The application can make the plug of the crop that is grown densely completely covered visualized, when the plug is closely arranged and stacked on the original culture table, by using the iterative simulation mode of feature extraction combined with grabbing posture, the grabbing key point and pose information of single plug are conveniently and accurately judged, at the same time, by the convenience of space search combined with reinforcement learning path planning, the optimal storage point position of plug can be quickly selected, and the plug is transferred according to the planned path.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of plug tray transfer, in particular to a plug tray sensing and sensing method, device and electronic equipment. BACKGROUND

[0002] When plug trays are grabbed, transferred and re-stacked on a culture table, the sensing and positioning of the plug trays are key difficulties to achieve this goal. Since the plug trays are closely arranged and stacked on the original culture table, it is difficult for the system to accurately determine the grabbing key points and pose information of a single plug tray. Meanwhile, compared with empty plug trays and growing culture plug trays, for plug trays that have been cultured and are waiting for harvesting, since these plug trays have been completely covered by densely grown crops, the method of relying on RGB images alone is no longer applicable and is difficult to visualize. Therefore, it needs to be improved. SUMMARY

[0003] Therefore, it is necessary to provide a plug tray sensing and sensing method, device and electronic equipment to solve the problems that when the plug trays are closely arranged and stacked on the original culture table, it is difficult to accurately determine the grabbing key points and pose information of a single plug tray, and for plug trays that have been cultured and are waiting for harvesting, since these plug trays have been completely covered by densely grown crops, the method of relying on RGB images alone is no longer applicable and is difficult to visualize.

[0004] The present application provides a plug tray sensing and sensing method, which comprises:

[0005] Based on remote RGBD prior region estimation and mechanical arm RGBD camera close-range multi-view shooting, complete to-be-grabbed point cloud structure is obtained;

[0006] According to feature extraction and grabbing pose simulation, the optimal grabbing pose is output;

[0007] Based on spatial search and reinforcement learning path planning, the optimal storage point position of the plug tray and the path planning are determined.

[0008] In one embodiment, the method based on remote RGBD prior region estimation and mechanical arm RGBD camera close-range multi-view shooting to obtain complete to-be-grabbed point cloud structure comprises:

[0009] Remote RGBD information is obtained, and the remote RGBD information is denoised and smoothed;

[0010] According to the size of a single plug tray platform, the region is roughly divided, and the segmentation map is extracted, and the suspected center point is calibrated;

[0011] Images shot at multiple angles in the suspected center point region are obtained, and the suspected center point region includes the outer edge and the outer plug tray long side region;

[0012] Multi-view reconstruction of point cloud data to generate a complete 3D point cloud;

[0013] Based on the Nerf neural network, the position, radius, color and opacity information of the scene point are rendered as Gaussian field parameters, and a fine 3D structure is obtained.

[0014] In one of the embodiments, the remote RGBD information is obtained, including:

[0015] Obtain leaf image data of different growth stages of the plant;

[0016] Label the key point information of the connection between seedlings and the center point of the crop plant in the leaf image.

[0017] The leaf image data is enhanced by rotation, flipping, scaling and color adjustment.

[0018] In one of the embodiments, the suspected center point is calibrated, including:

[0019] Obtain the scene image of dense plants;

[0020] Based on the attention mechanism, the scene image of a single plant is extracted from the dense plants;

[0021] The scene image of the single plant is input into the YOLOv8-Pose model, and the position information of the grasping key point is output, including the two-dimensional pixel coordinates of the plant and the confidence score.

[0022] In one of the embodiments, the suspected center point is calibrated, including:

[0023] Input the scene image with missing leaf data into a semi-supervised learning segmentation model to output the annotation information of the scene image, wherein the semi-supervised learning segmentation model is trained with a small amount of labeled data and a large amount of unlabeled data.

[0024] In one of the embodiments, the optimal grasping pose is output according to the feature extraction and grasping pose simulation, including:

[0025] Based on geometric and semantic constraints, select key frames and combine time decay coefficients for weighted calculation to optimize camera pose and scene parameters;

[0026] A feature extraction network is used to analyze the point cloud region;

[0027] Simulate the position and pose of different mechanical arm ends, calculate the grasping and lifting quality score under each pose, and output the optimal grasping pose.

[0028] In one of the embodiments, the optimal storage point position of the plug tray and path planning are determined based on spatial search and reinforcement learning path planning, comprising:

[0029] 3D spatial information of the shelf and the new culture table is acquired, and idle areas are screened out according to geometric constraints;

[0030] The idle areas are divided by using a K-D tree algorithm, and the optimal storage point position of the plug tray is determined in combination with a particle swarm optimization algorithm;

[0031] The shortest path in the static environment is calculated based on an A-star algorithm;

[0032] Plant 3D positioning technology based on monocular depth estimation and multi-view fusion depth completion is used to determine point cloud information of the target seedling and the crop;

[0033] The path planning is dynamically adjusted according to a self-adaptive path planning system based on a deep Q network.

[0034] The application further provides a perception and sensing device for a plug tray, comprising:

[0035] An acquisition module is configured to acquire a complete to-be-grabbed point cloud structure based on remote RGBD prior area estimation and close-range multi-view shooting of an RGBD camera of a mechanical arm;

[0036] An output module is configured to output an optimal grabbing posture according to feature extraction and grabbing posture simulation.

[0037] A determination module is configured to determine the optimal storage point position of the plug tray and path planning based on spatial search and reinforcement learning path planning.

[0038] The application further provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the perception and sensing method for the plug tray as described in any of the above embodiments when executing the computer program.

[0039] The application further provides a computer readable storage medium storing a computer program, wherein the computer program is executed by a processor to implement the perception and sensing method for the plug tray as described in any of the above embodiments.

[0040] The sensing method, device and electronic equipment of the plug tray determine a suspected area by adopting a remote RGBD prior area, and then adopt close-range multi-view shooting in the suspected area, so as to facilitate the completion of the point cloud, assist in positioning, make the densely grown crops of the plug tray completely visualized, when the plug trays are closely arranged and stacked on the original culture table, facilitate the accurate determination of the grasping key points and pose information of the single plug tray by adopting the iterative simulation mode of feature extraction combined with the grasping posture, and the optimal storage point position of the plug tray can be quickly selected by the convenience of space search combined with reinforcement learning path planning, and the plug tray is transferred according to the planned path.

[0041] For the densely grown and sheltered seedlings, a feature extraction network based on an attention mechanism is used, and image sample collection under different light conditions, different shooting angles and different backgrounds is used to improve the accuracy of semantic segmentation; in view of the high cost of instance segmentation training data labeling, the semi-supervised idea is used to solve the problem of too little seedling and seedling cultivation scene plant image labeling data; the self-supervised monocular depth estimation technology is introduced to complete the rough positioning of the target crops and seedlings; then, the RGBD information fusion depth completion and multi-view fusion three-dimensional semantic reconstruction based on deep learning are carried out, the three-dimensional registration of the target is carried out, and the positioning accuracy of the target is enhanced; according to the 3D positioning segmentation point cloud result feature extraction, the most suitable grasping pose is generated end to end, and the success rate of accurate seedling transplanting and moving is improved. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the present application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0043] Figure 1 The sensing method flow chart of the plug tray in an embodiment;

[0044] Figure 2 The flow chart of obtaining complete to-be-grasped point cloud structure in an embodiment;

[0045] Figure 3 The plug tray sensing and positioning flow chart under the serious occlusion condition based on two-stage step-by-step reasoning in an embodiment;

[0046] Figure 4 The flow chart of obtaining remote RGBD information in an embodiment;

[0047] Figure 5 The flow chart of calibrating suspected center points in an embodiment;

[0048] Figure 6 Optimal grasping pose flowchart for output in an embodiment;

[0049] Figure 7 Optimal storage point position and path planning flowchart for plug for an embodiment;

[0050] Figure 8 Plug placement point planning flowchart for an embodiment based on spatial search and reinforcement learning path planning;

[0051] Figure 9 Perception and sensing device structure diagram for a plug for an embodiment;

[0052] Figure 10 Internal structure diagram of an electronic device for an embodiment. DETAILED DESCRIPTION

[0053] To make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, rather than all embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0054] The traditional perception and positioning system of the plug has the following shortcomings:

[0055] 1. Insufficient precision and robustness of image recognition

[0056] Strong dependence on light: Image recognition systems usually rely on visual sensors (such as RGB cameras, depth cameras, etc.), but in different lighting conditions, especially in environments with large changes in lighting, the recognition accuracy decreases significantly. For the complex background in the plug planting environment, the image recognition algorithm of the prior art often cannot stably recognize the target.

[0057] Object occlusion problem: The growth of plants may cause part of the target object to be occluded, and the existing image recognition system has poor robustness in handling partially occluded or overlapping objects, affecting accurate recognition and positioning.

[0058] Slow recognition speed: Image processing and recognition algorithms run at high precision, which requires a large amount of calculation, which may result in slow image recognition speed, affecting the real-time performance of the system, especially when processing dynamic or complex scenes.

[0059] 2. Mechanical arm precision problem

[0060] Insufficient positioning and grasping accuracy: The end effector of the robotic arm (such as grippers) may not be able to accurately position and grasp the target object due to sensor errors, mechanical structure deviations, and other reasons in actual application. Especially when performing fine or repetitive tasks, precision deviations can cause job failure or low production efficiency.

[0061] Insufficient force feedback: Some robotic arm systems lack effective force feedback mechanisms, which cannot perceive the contact state with the target object in real time, resulting in inaccurate grasping force control and easy damage to plants or operation failure.

[0062] Insufficient rigidity and flexibility: The rigid design of the robotic arm may limit its adaptability in complex tasks, especially in environments requiring flexible operation, making it difficult for the robotic arm to adapt to different shape or material handling requirements.

[0063] 3. Deficiency of robotic arm trajectory planning

[0064] Low efficiency of trajectory planning algorithm: Existing trajectory planning algorithms are low in efficiency when dealing with complex operation paths, especially when involving complex path planning with multiple points and attitudes, which may result in longer calculation time, affecting the overall operation speed. For heterogeneous tasks, existing trajectory planning algorithms lack flexibility and are difficult to adjust in real time.

[0065] Weak collision detection and obstacle avoidance capability: Although some systems have basic obstacle avoidance function, in complex environment, the trajectory planning of the robotic arm may lack fine collision detection capability, resulting in collision or misoperation during operation, affecting operation efficiency and safety.

[0066] Poor adaptability to environmental changes: Existing trajectory planning technology usually assumes a stable working environment, but in actual planting environment, factors such as plant growth and obstacle changes will affect the accuracy and efficiency of planning, and existing systems have poor adaptability in this respect.

[0067] 4. Low efficiency and slow system response

[0068] Unintelligent multi-task scheduling: Existing systems have low scheduling efficiency when facing multiple tasks in parallel, especially in complex feeding, planting and handling tasks, the system often relies on manual intervention or simple automated process, resulting in low production line efficiency.

[0069] Feedback lag between perception and execution: There is a time delay in the feedback between the perception system (such as image recognition, sensor data) and the robotic arm execution, affecting the real-time response capability of the system. Especially in scenarios requiring rapid action adjustment, the system's response speed is insufficient, reducing efficiency.

[0070] Data processing bottleneck: The processing of image data, sensor data, etc. may require high computing power, and existing hardware facilities may have processing bottlenecks, resulting in slow overall system response, especially under high load or large-scale tasks, and processing delays will significantly reduce production efficiency.

[0071] The following describes the perception and sensing method, device and electronic equipment of the plug tray of the application. Figures 1-10 The perception and sensing method, device and electronic equipment of the plug tray of the application are described.

[0072] As shown in the embodiment, a perception and sensing method of a plug tray comprises the following steps: Figure 1

[0073] Step S100, based on remote RGBD prior region estimation and mechanical arm RGBD camera close-range multi-view shooting, complete point cloud structure to be grasped is obtained.

[0074] RGBD refers to an image or data format containing color (RGB) and depth (Depth) information. Among them, RGB refers to the red (Red), green (Green) and blue (Blue) three color channels of the image, which can represent the full-color image when combined together; D (Depth) depth channel contains the depth information of each pixel in the scene, usually expressed in physical distance (e.g. millimeter or meter) from the camera, and the depth information allows to understand the three-dimensional structure in the scene.

[0075] For the plug tray platform that has completed cultivation and is waiting for harvesting, the plug tray is completely covered by densely grown crop plants and is not visible itself. The densely grown crop plants are stacked and blocked, making it difficult to obtain their complete 3D information and also difficult to label and segment single plants on the RGB image. Therefore, for this situation, remote RGBD prior region estimation combined with mechanical arm RGBD camera close-range multi-view shooting is adopted to facilitate point cloud completion and auxiliary positioning.

[0076] Step S200, according to feature extraction and grasping pose simulation, output the optimal grasping pose.

[0077] By selecting the optimal grasping pose, precise plug tray grasping is facilitated.

[0078] Step S300, based on spatial search and reinforcement learning path planning, determine the optimal storage point position and path planning of the plug tray.

[0079] ​For the hole disc that has completed grabbing, it needs to be placed on the shelf placed by the robot itself, that is, to be moved from the shelf to the new workbench, and the optimal placement point needs to be planned so that the hole disc can be closely arranged at the target position. Therefore, through the convenience of spatial search combined with reinforcement learning path planning, the optimal storage point position of the hole disc can be quickly selected, and the hole disc is transferred according to the planned path.

[0080] The perception and sensing method of the hole disc determines the suspected area by using a remote RGBD prior area, and then uses close-range multi-view shooting in the suspected area to facilitate the completion of the point cloud and assist in positioning, so that the hole disc with densely growing crops is completely visualized. When the hole discs are closely arranged and stacked on the original culture table, the feature extraction combined with the iterative simulation of the grabbing posture is used to conveniently and accurately determine the grabbing key points and pose information of a single hole disc. At the same time, through the convenience of spatial search combined with reinforcement learning path planning, the optimal storage point position of the hole disc can be quickly selected, and the hole disc is transferred according to the planned path.

[0081] In the embodiment, referring to Figure 2 and Figure 3 , based on remote RGBD prior area estimation and close-range multi-view shooting of the robot RGBD camera, a complete to-be-grabbed point cloud structure is obtained, including the following steps:

[0082] In step S110, remote RGBD information is obtained, and the remote RGBD information is denoised and smoothed.

[0083] By denoising and smoothing the remote RGBD information, the accuracy of the RGBD information can be improved. A laser radar can be used instead of an RGBD camera to obtain more accurate depth information and distance measurement.

[0084] In step S120, the area is roughly divided according to the size of the single hole disc platform, and the segmentation map is extracted, and the suspected center point is calibrated.

[0085] In the practical stage, after the corresponding remote RGBD information is obtained by the camera and denoised and smoothed, the segmentation map is extracted according to the rough division of the plant area photographed from the edge according to the length and width information, and the suspected single hole disc center point area closest to the robot working area is given, so that the robot performs fine shooting near this area.

[0086] In step S130, the image is obtained by multi-angle shooting in the suspected center point area, and the suspected center point area includes the outer edge and the outer hole disc long side area.

[0087] By making the robot arm mainly in the suspected hole disc center point area multi-angle shooting, the multi-view reconstruction is further carried out for the visible area such as the long side of the hole disc outside, and the fine 3D information is obtained.

[0088] Step S140, multi-view reconstruction point cloud data is performed to generate a complete 3D point cloud.

[0089] Multi-view reconstruction is performed in the rectangular block region given by the above coarse positioning, and a more complete point cloud region is generated.

[0090] Step S150, based on the Nerf neural network, the position, radius, color and opacity information of the scene point are rendered as Gaussian field parameters, and the fine 3D structure is obtained.

[0091] Through the Nerf neural network rendering, the complete point cloud structure to be grasped is obtained.

[0092] In the embodiment, referring to Figure 4 , the remote RGBD information is obtained, including the following steps:

[0093] Step S111, leaf image data of different growth stages of the plant is obtained.

[0094] The leaf image covers images under different light conditions, different shooting angles and different backgrounds. These image data sets provide diversified training samples, which help to improve the generalization ability of the model.

[0095] Step S112, the key point information of the connecting part between seedlings and the center point of the crop plant of the leaf image is labeled.

[0096] It is convenient to label the image in detail.

[0097] Step S113, the leaf image data is enhanced by rotation, flipping, scaling and color adjustment.

[0098] It is convenient to enhance the robustness of the model. These data enhancement techniques can simulate various transformations in actual applications and improve the performance of the model in complex environments.

[0099] In the embodiment, referring to Figure 5 , the suspected center point is calibrated, including the following steps:

[0100] Step S121, the scene image of dense plants is obtained.

[0101] The scene image of dense plants is captured by a camera or a sensor, usually an RGB image.

[0102] Step S122, based on the attention mechanism, the scene image of a single plant is extracted from the dense plants.

[0103] In order to solve the problem of poor instance segmentation effect under complex background and occlusion conditions, an attention mechanism is added to the network architecture. By weighting feature selection, the model's attention to important features is effectively improved, irrelevant background and noise are suppressed, and the segmentation accuracy is improved. Single seedlings can be accurately extracted from dense seedling plants at high speed. Attention mechanism is first applied to natural language processing field, aiming to enhance the model's focus on key information, and then introduced into computer vision tasks. After integrating the attention mechanism into the backbone network of the segmentation network, it can improve the feature expression ability, make the model more accurately distinguish instances and background, optimize the understanding of object relationship and layout, and strengthen the boundary refinement, improve the segmentation accuracy. It is expected that by integrating it into the segmentation network, the adaptability of the model in complex background is enhanced, and the seedling plants are accurately segmented, ensuring the accuracy and efficiency of the seedling grabbing step.

[0104] In step S123, the scene image of a single plant is input into the YOLOv8-Pose model, and the position information of the grabbing key point is output. The position information includes the two-dimensional pixel coordinates of the plant and the confidence score.

[0105] YOLOv8-Pose adds a key point detection module based on YOLOv8, which can identify the key nodes of objects (such as human joints, plant nodes, etc.), thereby achieving more detailed recognition and positioning. To improve the detection performance of the model, the research also introduces depth information, which combines depth data with RGB images to optimize the model's feature learning and overall framework.

[0106] Through the YOLOv8-Pose network model, key point detection is performed to accurately distinguish seedling plants and mark the grabbing key points. At the same time, a new prediction branch for grabbing decision is added, which combines with the multi-task learning framework, so that the model can perform target detection and plant grabbing feasibility evaluation simultaneously, thereby realizing automatic identification of grabbing points and guiding the robot to perform accurate grabbing operations.

[0107] In this embodiment, calibrating the suspected center point further includes the following steps:

[0108] In step S124, the scene image with missing leaf data is input into the semi-supervised learning segmentation model, and the annotation information of the scene image is output. The semi-supervised learning segmentation model is trained with a small amount of labeled data and a large amount of unlabeled data.

[0109] Traditional instance segmentation methods rely on a large amount of labeled data for training, and although they perform well on large-scale labeled datasets, their performance will decrease significantly in the case of insufficient labeled data. In complex agricultural scenarios such as seedling unloading, the labeling of plant center points, the contour selection and labeling of the entire plant target are not only difficult and time-consuming, but also easily affected by subjective factors. Therefore, semi-supervised learning becomes an effective solution, which combines a small amount of labeled data and a large amount of unlabeled data for training, greatly improving the segmentation effect under limited data resources. The specific scheme includes: first, prepare labeled data and unlabeled data, select a small amount of typical target crops and seedling pictures under various perspectives for manual labeling, and the unlabeled data covers different time periods, different angles, and mainly the overhead perspective of the vegetable seedling, increasing the diversity of the data.

[0110] In terms of model selection, a segmentation model suitable for semi-supervised learning is used, such as a method based on consistency regularization and pseudo-label self-training. The training process is divided into four steps:

[0111] 1. Pre-training stage, use a small amount of labeled data to pre-train the model, and obtain the basic vegetable seedling and plant center point segmentation capability.

[0112] 2. Consistency regularization, through various image enhancement of unlabeled data under different grabbing times and weather conditions, ensure the consistency of the model's prediction results for enhanced versions.

[0113] 3. Pseudo-label generation, use the model to predict unlabeled data to generate pseudo-labels as additional supervision signals for training.

[0114] 4. Joint training, combine labeled data and pseudo-label data to optimize the model. When optimizing the model, combine supervised and unsupervised loss functions to design a comprehensive loss function, including cross-entropy loss and consistency loss, and adjust hyperparameters such as learning rate and consistency regularization coefficient through cross-validation to ensure the best model performance.

[0115] In this embodiment, referring to Figure 3 and Figure 6 , according to feature extraction and grabbing pose simulation, the optimal grabbing pose is output, including the following steps:

[0116] Step S210, based on geometric and semantic constraints, select key frames and combine time decay coefficients for weighted calculation to optimize camera pose and scene parameters.

[0117] By selecting key frames according to geometric and semantic constraints and combining time decay coefficients for weighting, the final point cloud overall mapping is performed, and then the scene parameters are optimized according to the estimated camera pose, so as to facilitate high-precision 3D mapping and semantic segmentation, and then facilitate the transmission of a relatively complete plant plug system to the grabbing point reasoning structure.

[0118] In step S220, a feature extraction network is used to analyze the point cloud region.

[0119] The feature extraction network is used to simulate the different mechanical arm end position and posture near the bottom of the point cloud region and extract part of the point cloud under the corresponding view angle, so as to facilitate the output of the corresponding grabbing and lifting quality score.

[0120] In step S230, different mechanical arm end positions and postures are simulated, and the grabbing and lifting quality score under each posture is calculated, and the optimal grabbing posture is output.

[0121] Through iterative simulation, the optimal grabbing posture is selected, and combined with high-precision sensors such as force sensors, vision sensors, laser range finders, etc., the posture and position of the mechanical arm can be accurately perceived, and the high precision of the grabbing action is ensured.

[0122] In this embodiment, referring to Figure 7 and Figure 8 , based on spatial search and reinforcement learning path planning, the optimal storage point position of the plug and the path planning are determined, including the following steps:

[0123] In step S310, the 3D space information of the shelf and the new culture table is obtained, and the idle area is selected according to the geometric constraint.

[0124] By selecting the idle area, the optional space of the plug transfer destination can be determined.

[0125] In step S320, the K-D tree algorithm is used to divide the idle area, and combined with the particle swarm optimization algorithm, the optimal storage point position of the plug is determined.

[0126] K-D tree is a high-efficiency spatial partition data structure, which can quickly search the idle area that meets the conditions in three-dimensional space, providing data support for subsequent path planning. The particle swarm optimization algorithm finds the optimal solution by simulating the motion of particles in the search space. The system calculates the space utilization, carrying efficiency and environmental adaptability of each candidate storage point in multiple iterations, and finally selects the optimal storage point position.

[0127] In the storage point optimization process, genetic algorithm can be used as an alternative to particle swarm optimization algorithm to provide different optimization paths.

[0128] Step S330, based on A-star algorithm, the shortest path in the static environment is calculated.

[0129] A-star algorithm can calculate the shortest path in the static environment, avoid known obstacles, and provide the basis for path planning. The heuristic search strategy of A-star algorithm makes the path planning more efficient in static environment.

[0130] Step S340, based on monocular depth estimation and multi-view fusion depth completion of plant 3D positioning technology, the point cloud information of target seedling and crop is determined.

[0131] In order to ensure the accuracy of seedling positioning and moving, two-stage target positioning is proposed. First, the seedling platform is roughly positioned by RGB shooting based on airborne camera, and the approximate position is determined according to the segmentation result of empty seedling or the segmentation and counting result of crop plant based on fast monocular depth estimation combined with target RGB picture semantic segmentation. Second, when the mechanical arm is close to the target seedling to prepare for transplanting action, based on the 3D camera at the end of the mechanical arm, the depth completion based on multi-view fusion is carried out to construct the fine point cloud of seedling crop system, which assists to realize the precise positioning of target seedling clamping point and the generation of grabbing point. Based on the close-range 3D shooting information, the previous depth estimation of this part is matched, and the next fine adaptive handling operation is performed.

[0132] 1. Rough positioning of seedling platform based on self-supervised monocular depth estimation

[0133] In the rough positioning stage, the self-supervised monocular depth estimation is used to obtain a complete and accurate depth map, which replaces the depth map obtained by the depth camera with missing and low reliability, thereby improving the accuracy of rough positioning of seedling target. Since the Manydepth framework proposed by Watson et al. has good effect on open source dataset, and innovatively relieves the problem of scale ambiguity in self-supervised training through adaptive cost volume, this study will realize the depth estimation of the target seedling to be moved based on the Manydepth framework.

[0134] Manydepth takes the geometric constraints between consecutive time frames as the supervision signal, converts monocular depth estimation into an image reconstruction problem; and measures the geometric compatibility of the pixel depth between the current frame and the adjacent frame through the adaptive cost volume, overcoming the defect of scale ambiguity in unsupervised training. Manydepth also introduces a binary mask and a teacher-student training model to mark the reliability of the cost volume in different regions, and uses the single-frame depth estimation network (teacher model) as a reference in the untrusted area to prevent overfitting of the cost volume. In addition, it also solves the problem of lack of last frame and camera stillness in the training process. Finally, the cost volume is sent to the encoder-decoder, and the output disparity is transformed to obtain the estimated depth. In the training and evaluation process, Manydepth introduces image reconstruction loss, disparity smoothness loss and consistency loss, among which the consistency loss makes the network ignore unreliable cost volumes and refer to the estimated results of the single-frame depth estimation network.

[0135] Before offline training, this project will first simulate the robot's action through a monocular camera, collect pixel-aligned target RGB image and depth image sequences of seedlings to be transplanted, and make a dataset. Then, the effectiveness of the pre-trained weight, cost volume, and single-frame depth estimation network (teacher model) will be verified during the training model process, and compared with other methods. After completing offline training and estimating depth based on the Manydepth framework, combined with the segmentation results of multiple size seedling targets on the RGB image previously, the depth corresponding to the pixels in the segmentation frame can be obtained, thereby realizing the coarse positioning of the seedling target. This method can reduce the labeling and depth data processing calculation cost in the training process.

[0136] 2. Fine positioning based on depth completion and multi-view fusion

[0137] When the robot arm approaches the target, fine positioning requires more accurate depth information than the coarse positioning stage, so this project will use depth completion to assist in achieving accurate adaptive positioning of different size seedlings to be transported. When fine positioning is performed, the end of the robot arm should be close to the target seedling. Based on the 3D shooting information at close range, match the previous depth estimation of this part, and perform the next fine segmentation operation.

[0138] Referring to the CompletionFormer framework proposed by Zhang et al., local features and global context information are introduced simultaneously in the task of depth completion, which can better realize depth completion. The algorithm takes an incomplete depth map and the corresponding RGB image as input, extracts multi-scale features through the U-Net backbone network constructed by JCAT blocks, realizes the interaction of depth and image information at multiple scales, and enables the features at different stages to be fused at full resolution. Then, the decoder provides full-resolution features for initial depth prediction, and finally the spatial propagation network (SPN) is used to refine the initial estimation result.

[0139] In addition, the point cloud registration can be further improved by multi-view fusion to improve the accuracy of 3D target representation and directly project semantic segmentation information to 3D point cloud. It is proposed to use an explicit isotropic 3D Gaussian function to represent the position, radius, color and opacity of each scene point, project the 3D Gaussian distribution to the image plane, calculate the color and depth of the pixel according to the transparency and influence range of the Gaussian distribution, and use the maximum volume rendering method to mix the pixels, so as to construct the overall 3D information of the target seedling and the crops that may exist on it according to the multiple perspectives of the end of the mechanical arm, and construct the overall detailed 3D positioning of the up and down movement. The system framework can complete the tracking and mapping of the target according to the Gaussian field. Tracking means estimating the camera pose of each frame of image, and fixing the scene parameters. At this time, the system optimizes the camera pose by minimizing the loss function between the rendered image and the real image, and the loss function is:

[0140]

[0141] where Σ pix is the sum over each pixel in the image, meaning that the loss function will evaluate each pixel in the image; Sil pix > T S : is a threshold condition, indicating that the loss will only be calculated for a pixel if its transparency (Sil) is greater than a certain threshold Ts. That is, only significant pixels (e.g. pixels with high transparency) are evaluated for loss; λ D , λ C , λ S : These are weight coefficients that control the contribution of different loss terms (depth, color, semantics) to the total loss function. Their role is to balance different types of errors and ensure the relative importance of each part in the optimization process; D pix , C pix , S pixrespectively represent the depth, color and semantic label values calculated by the model, i.e. the estimated values in the current image; These errors represent the absolute difference between the ground truth values and the model prediction values, and the loss function optimizes the model by minimizing these errors; depth error This measures the error of the depth map, and the depth error is crucial for optimizing the camera pose and accurate reconstruction of three-dimensional objects; color error This measures the difference in color between the rendered image and the real image, helping to ensure color consistency; semantic error This measures the difference in semantic segmentation information, ensuring that the semantic content (such as object categories, etc.) of the rendered image is consistent with the real image.

[0142] Subsequently, the system selects key frames based on geometric and semantic constraints and combines time decay coefficients for weighting to improve mapping quality. When mapping, scene parameters are optimized based on estimated camera poses to achieve high-precision 3D mapping and semantic segmentation. During the mapping process, Gaussian densification is introduced, i.e. new Gaussian points are introduced into the scene to improve reconstruction accuracy; at the same time, the loss function between the rendered image and the real image is minimized to optimize the Gaussian parameters, including RGB-D and semantic information. Based on this, this method can achieve high-precision scene reconstruction and semantic segmentation based on the explicit representation of 3D Gaussian field; since the channel directly contains the semantic information of the 3D scene, it can easily select and manipulate specific objects in the scene; and by editing the Gaussian parameters, operations such as object movement, rotation and removal can be achieved, enhancing the real-time and sensitivity of 3D reconstruction.

[0143] Based on the point cloud information of the target seedlings and crops formed by the two-step positioning, the precise positioning is completed; and the point cloud information of the seedlings after semantic segmentation is sampled as input, the feature extraction network with PointNet as the backbone is used to extract features from the point cloud, and the corresponding transplant stability quality score is output, and finally the pose with the highest score is selected as the most appropriate pose information of the grabbing or clamping point.

[0144] Step S350, according to the adaptive path planning system of the depth Q network, dynamically adjust the path planning.

[0145] The deep reinforcement learning (DQN) method uses a deep Q network to dynamically adjust the path planning. DQN learns the dynamic changes of obstacles and unknown situations in the environment, enabling the robot to adjust the path in real time, avoid collisions and improve work efficiency. The DQN network optimizes the path selection under different states through state and action pairs, thereby realizing adaptive adjustment of path planning.

[0146] In path planning, other reinforcement learning algorithms such as PPO (Proximal Policy Optimization) can be used instead of DQN to improve the convergence and stability of path planning.

[0147] The present application focuses on the grabbing, storing and restacking of plug trays on the culture table, and the perception and positioning of the plug trays are key difficulties in achieving this goal. Since the plug trays are closely arranged and stacked on the original culture table, it is difficult for the system to accurately determine the grabbing key points and pose information of a single plug tray. At the same time, when the plug trays are densely placed, the edge information used to segment and locate a single plug tray is often too small to extract.

[0148] In order to solve the problem of perception and positioning of the empty plug trays in the densely placed state, the robot needs to perform image segmentation on the entire plug tray culture area during the process of lifting the plug trays to the culture area and the transplanting area, identify the independent plug trays and determine the center point position of the empty plug trays or the pre-cultivation plug trays. At the same time, accurately identifying the empty plug trays, growing crops and their edge key points is also the focus of this project.

[0149] Compared with empty plug trays and growing plug trays, the method of relying on RGB images alone is no longer applicable for plug trays that have been cultured and are waiting for harvesting. This is because these plug trays have been completely covered by densely growing crops and are difficult to visualize. At the same time, the point cloud data generated by the traditional RGB-D camera often has serious information missing, so the method of relying solely on 2D semantic segmentation to 3D projection cannot accurately derive the appropriate grabbing point. We will focus on developing an active multi-view perception and image 3D completion method to remotely estimate and closely correct the severely occluded point cloud area. This method first obtains more complete 3D information reasoning, combines the length and width dimensions of the pre-known plug tray platform, performs segmentation of the suspected single platform area and determination of the center point position, completes the rough segmentation and rough positioning of the plug tray. On this basis, combined with the Nerf algorithm, multi-view image shooting is performed near the suspected center point to reconstruct the corresponding point cloud information. This method effectively fills in the 3D information of the harvesting plug tray platform near the long side, and finally hands over to the grabbing point generator for grabbing point derivation, solving the problem of densely placed plug trays in severely occluded scenes that are difficult to identify one by one.

[0150] In addition, the robot chassis is equipped with a cabinet rack structure for transporting the plug trays, and a new plug tray placement position needs to be arranged on the target new cultivation table. After completing the lifting of the plug tray, the robot needs to plan to re-closely arrange multiple plug trays to complete the stacking task. In this process, effective identification of the rack, the new cultivation table and the placed plug tray is necessary to determine the closest stacking point position. For this purpose, the robot needs to acquire and identify the point cloud information of the rack and the new cultivation table through RGB-D information, and use the optimal path planning algorithm to deduce the current most suitable plug tray placement position, ensuring that the robot can efficiently complete the automatic stacking task. Through optimal placement point search, the system will locate the target storage point based on real-time updated RGB-D information, so that crops can be closely arranged on the new cultivation point and the rack, improving the overall work efficiency.

[0151] The perception and sensing method of the plug tray provided by the application has the following effects:

[0152] 1. Image recognition advantage:

[0153] High-precision recognition: The image recognition system can accurately detect and recognize the specific position, shape and direction of the plug tray, providing reliable data for accurate grabbing of the mechanical arm.

[0154] Strong environmental adaptability: Through deep learning and computer vision technology, it can effectively identify and adjust in complex environmental conditions, helping to improve the stability and reliability of the system.

[0155] Real-time feedback: The image recognition system can monitor the working state in real time and feedback the position and attitude information in time to realize rapid adjustment and avoid incorrect grabbing or falling.

[0156] 2. Mechanical arm precision advantage:

[0157] High-precision positioning: Combined with high-precision sensors such as force sensors, vision sensors and laser range finders, it can accurately perceive the attitude and position of the mechanical arm to ensure high precision of the grabbing action.

[0158] Strong self-adaptation: The mechanical arm can automatically fine-tune according to the real-time feedback of the perception system to adapt to different plug trays and environmental changes to ensure stability.

[0159] Reduce human error: Through high-precision control of the automatic system, the error caused by manual operation is reduced, and the overall operation precision is improved.

[0160] 3. Mechanical arm trajectory planning advantage:

[0161] Dynamic trajectory planning: The system can dynamically adjust the trajectory based on real-time data, target position and path obstacles to optimize the working path of the mechanical arm and avoid unnecessary waiting time and collision risk.

[0162] Efficient path generation: Advanced path planning algorithms (such as A-star algorithm, RRT algorithm, etc.) are used to generate the shortest or optimized path, improving the efficiency of the robot arm.

[0163] Task flexibility: Through flexible trajectory planning, the robot arm can perform various tasks (such as feeding, sorting, organizing, etc.), with strong versatility and adaptability.

[0164] 4、Efficiency improvement advantage:

[0165] Automation improves production efficiency: The efficient cooperation of the robot arm perception and sensing system makes the entire operation process more automated, reducing the need for manual intervention and improving work efficiency.

[0166] Multi-task parallelism: The system can handle multiple plug trays or tasks simultaneously, reducing the time waste of single-task operation and further improving overall efficiency.

[0167] Reducing waiting time: Real-time perception and feedback mechanism reduces the waiting time of the robot arm in operation, improving the overall response speed and operation rate of the system.

[0168] The perception and sensing device of the plug tray provided by the present application is described below, and the perception and sensing device of the plug tray described below can be mutually corresponding with the perception and sensing method of the plug tray described above.

[0169] As shown in Figure 9 In one embodiment, a perception and sensing device of a plug tray includes an acquisition module 910, an output module 920, and a determination module 930.

[0170] The acquisition module 910 is configured to acquire a complete point cloud structure to be grasped based on remote RGBD prior region estimation and mechanical arm RGBD camera close-range multi-view shooting.

[0171] The output module 920 is configured to output an optimal grasping pose according to feature extraction and grasping pose simulation.

[0172] The determination module 930 is configured to determine the optimal storage point position of the plug tray and path planning based on spatial search and reinforcement learning path planning.

[0173] In this embodiment, the acquisition module 910 is specifically configured to:

[0174] Acquire remote RGBD information and perform denoising and smoothing processing on the remote RGBD information;

[0175] Coarsely divide the region according to the size of a single plug tray platform, extract a segmentation map, and calibrate a suspected center point;

[0176] Obtain images taken at multiple angles in a suspected center point area, which includes an outer edge and an outer hole disc long edge area;

[0177] Multi-view reconstruction of point cloud data to generate a complete 3D point cloud;

[0178] Based on the Nerf neural network, the position, radius, color and opacity information of the scene point are rendered as Gaussian field parameters, and a fine 3D structure is obtained.

[0179] In this embodiment, remote RGBD information is obtained, specifically for:

[0180] Obtain leaf image data of different growth stages of the plant;

[0181] Label the key point information of the connecting part between seedlings and the center point of the crop plant in the leaf image;

[0182] Perform enhancement processing such as rotation, flipping, scaling and color adjustment on the leaf image data.

[0183] In this embodiment, the suspected center point is calibrated, specifically for:

[0184] Obtain a scene image of dense plants;

[0185] Based on the attention mechanism, extract a scene image of a single plant from the dense plants;

[0186] Input the scene image of a single plant into the YOLOv8-Pose model to output the position information of the grasping key points, including the two-dimensional pixel coordinates of the plant and the confidence score.

[0187] In this embodiment, the suspected center point is calibrated, specifically for:

[0188] Input the scene image with missing leaf data into a semi-supervised learning segmentation model to output the annotation information of the scene image, and the semi-supervised learning segmentation model is trained with a small amount of labeled data and a large amount of unlabeled data.

[0189] In this embodiment, the output module 920 is specifically configured to:

[0190] Based on geometric and semantic constraints, select key frames and perform weighted calculation combined with a time decay coefficient to optimize camera pose and scene parameters;

[0191] A feature extraction network is used to analyze the point cloud region;

[0192] Simulate the position and attitude of the end of different mechanical arms, calculate the grasping and lifting quality score under each attitude, and output the optimal grasping attitude.

[0193] In the embodiment, the determining module 930 is specifically configured to:

[0194] 3D space information of the shelf and the new culture table is acquired, and an idle area is screened out according to geometric constraints;

[0195] The idle area is divided by using a K-D tree algorithm, and an optimal storage point position of the plug is determined by combining a particle swarm optimization algorithm;

[0196] Based on an A-star algorithm, the shortest path in a static environment is calculated;

[0197] Based on monocular depth estimation and multi-view fusion depth completion, 3D positioning technology of the plant is determined to determine point cloud information of the target seedling and the crop;

[0198] According to a self-adaptive path planning system based on a deep Q network, the path planning is dynamically adjusted.

[0199] The perception and sensing device of the plug uses a feature extraction network based on an attention mechanism for cultivation plants that grow densely and block the seedling body, and image sample collection under different light conditions, different shooting angles and different backgrounds to improve the accuracy of semantic segmentation. To solve the problem of high cost of instance segmentation training data annotation, the semi-supervised idea is used to solve the problem of too little plant image annotation data in the existing seedling and seedling cultivation scene.

[0200] The self-supervised monocular depth estimation technology is introduced to complete the coarse positioning of the target crop and the seedling. Then, the RGBD information fusion depth completion and multi-view fusion three-dimensional semantic reconstruction based on deep learning are performed to perform three-dimensional registration of the target and enhance the accuracy of target positioning. According to the feature extraction of the 3D positioning segmentation point cloud result, the most suitable grasping pose is generated end to end to improve the success rate of accurate seedling transplanting and moving.

[0201] Figure 10 An example of an entity structure diagram of an electronic device is shown, which can be a smart terminal, and its internal structure diagram can be as shown in Figure 10 The electronic device includes a processor, a memory and a network interface connected by a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the electronic device is used to communicate with external terminals through network connection. The computer program is executed by the processor to implement the perception and sensing method of the plug, and the method comprises:

[0202] Based on remote RGBD prior region estimation and mechanical arm RGBD camera close-range multi-view shooting, complete point cloud structure to be grabbed is acquired;

[0203] According to feature extraction and grabbing posture simulation, an optimal grabbing posture is outputted;

[0204] Based on space search and reinforcement learning path planning, an optimal storage point position of the plug tray and path planning are determined.

[0205] Those skilled in the art can understand that, Figure 10 The skilled in the art can understand that,

[0206] In another aspect, the present application also provides a computer storage medium storing a computer program, and the computer program is executed by a processor to realize the perception and sensing method of the plug tray, and the method comprises:

[0207] Based on remote RGBD prior region estimation and mechanical arm RGBD camera close-range multi-view shooting, complete point cloud structure to be grabbed is acquired;

[0208] According to feature extraction and grabbing posture simulation, an optimal grabbing posture is outputted;

[0209] Based on space search and reinforcement learning path planning, an optimal storage point position of the plug tray and path planning are determined.

[0210] In another aspect, the present application also provides a computer storage medium storing a computer program, and the computer program is executed by a processor to realize the perception and sensing method of the plug tray, and the method comprises:

[0211] Based on remote RGBD prior region estimation and mechanical arm RGBD camera close-range multi-view shooting, complete point cloud structure to be grabbed is acquired;

[0212] According to feature extraction and grabbing posture simulation, an optimal grabbing posture is outputted;

[0213] Based on space search and reinforcement learning path planning, an optimal storage point position of the plug tray and path planning are determined.

[0214] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiment methods. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory.

[0215] By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0216] The technical features of the above embodiments can be combined in any way. In order to make the description simple, not all possible combinations of the technical features in the above embodiments are described, but as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0217] The above embodiments only express several embodiments of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the present application. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. A method for sensing and transmitting information about acupoints, characterized in that, The method includes: Based on remote RGBD prior region estimation and close-range multi-view shooting by the robotic arm's RGBD camera, the complete point cloud structure to be grasped is obtained. Based on feature extraction and grasping posture simulation, the optimal grasping posture is output. Based on spatial search and reinforcement learning path planning, the optimal storage location and path planning of the acupuncture plate are determined. The process of obtaining the complete point cloud structure to be grasped based on remote RGBD prior region estimation and close-range multi-view shooting by the robotic arm's RGBD camera includes: Acquire remote RGBD information and perform noise reduction and smoothing processing on the remote RGBD information; The area is roughly divided according to the size of a single acupuncture plate, and the segmentation map is extracted and the suspected center point is marked. Acquire images taken from multiple angles in the suspected center point area, which includes the outer edge and the long side of the acupoint plate near the outer edge; Reconstruct point cloud data from multiple perspectives to generate a complete 3D point cloud; Based on the Nerf neural network, the position, radius, color and opacity of scene points are used as Gaussian field parameters for rendering, and a detailed 3D structure is obtained. The acquisition of remote RGBD information includes: Acquire leaf image data at different growth stages of the plant; Mark key information such as the connection points between seedlings and the center point of crop plants in the leaf images; Enhancement processing of leaf image data, including rotation, flipping, scaling, and color adjustment; The identification of suspected center points includes: The scene image with missing leaf data is input into a semi-supervised learning segmentation model, which outputs the annotation information of the scene image. The semi-supervised learning segmentation model is trained with a small amount of labeled data and a large amount of unlabeled data.

2. The method for sensing and transmitting acupoints according to claim 1, characterized in that, The identification of suspected center points includes: Acquire scene images of densely planted vegetation; Based on the attention mechanism, scene images of individual plants are extracted from densely packed plants; The scene image of the single plant is input into the YOLOv8-Pose model, which outputs the location information of the key points, including the two-dimensional pixel coordinates and confidence score of the plant.

3. The method for sensing and transmitting acupoints according to any one of claims 1 to 2, characterized in that, The step of outputting the optimal grasping posture based on feature extraction and grasping posture simulation includes: Based on geometric and semantic constraints, key frames are selected and weighted calculations are performed using time decay coefficients to optimize camera pose and scene parameters. A feature extraction network is employed, and point cloud regions are analyzed. Simulate the position and posture of different robotic arm ends, calculate the grasping and lifting quality score for each posture, and output the optimal grasping posture.

4. The method for sensing and transmitting acupoints according to claim 3, characterized in that, The method of determining the optimal storage location and path planning for the acupuncture plate based on spatial search and reinforcement learning includes: Obtain 3D spatial information of the shelves and new incubation stations, and filter out the available areas based on geometric constraints; The KD-tree algorithm is used to divide the free area, and the particle swarm optimization algorithm is combined to determine the optimal storage location of the pit plate; Based on the A* algorithm, calculate the shortest path in a static environment; Plant 3D localization technology based on monocular depth estimation and multi-view fusion depth completion determines the point cloud information of target seedlings and crops. The path planning is dynamically adjusted based on the adaptive path planning system of deep Q-networks.

5. A sensing and sensing device for acupoints, characterized in that, include: The acquisition module is used to acquire the complete point cloud structure to be grasped based on remote RGBD prior region estimation and close-range multi-view shooting by the robotic arm's RGBD camera. The output module is used to output the optimal grasping posture based on feature extraction and grasping posture simulation. The determination module is used to determine the optimal storage location and path planning of the pit plate based on spatial search and reinforcement learning path planning; The process of obtaining the complete point cloud structure to be grasped based on remote RGBD prior region estimation and close-range multi-view shooting by the robotic arm's RGBD camera includes: Acquire remote RGBD information and perform noise reduction and smoothing processing on the remote RGBD information; The area is roughly divided according to the size of a single acupuncture plate, and the segmentation map is extracted and the suspected center point is marked. Acquire images taken from multiple angles in the suspected center point area, which includes the outer edge and the long side of the acupoint plate near the outer edge; Reconstruct point cloud data from multiple perspectives to generate a complete 3D point cloud; Based on the Nerf neural network, the position, radius, color and opacity of scene points are used as Gaussian field parameters for rendering, and a detailed 3D structure is obtained. The acquisition of remote RGBD information includes: Acquire leaf image data at different growth stages of the plant; Mark key information such as the connection points between seedlings and the center point of crop plants in the leaf images; Enhancement processing of leaf image data, including rotation, flipping, scaling, and color adjustment; The identification of suspected center points includes: The scene image with missing leaf data is input into a semi-supervised learning segmentation model, which outputs the annotation information of the scene image. The semi-supervised learning segmentation model is trained with a small amount of labeled data and a large amount of unlabeled data.

6. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the sensing and perception method for the acupoint plate as described in any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the sensing and perception method for the acupoint plate as described in any one of claims 1 to 4.

Images