Airport cart state recognition and mechanical arm guided grabbing method and system based on multi-modal perception

Abstract

This invention relates to the field of robotic arm control technology, and more particularly to a method and system for airport cart state recognition and robotic arm guided grasping based on multimodal perception. The method includes: acquiring multimodal perception data; inputting the multimodal perception data into a convolutional neural network and outputting candidate cart boxes; calculating the structural saliency of candidate carts; constructing a candidate cart set and initial state description; selecting several candidate carts for target determination; reading the front-end dominance level; calculating the reachability of the robotic arm and the target priority value; confirming the candidate target carts; extracting candidate edge segments and calculating their scores; determining the candidate contact area; generating the pre-contact pose of the end effector; calculating the operability index; correcting the candidate contact area; obtaining the confirmed contact area and operable state conclusion; solving for the grasping pose and execution intensity coefficient of the end effector; and completing the pull-out action based on the execution intensity coefficient.

Description

Technical Field

[0001] This invention relates to the field of robotic arm control technology, and more particularly to a method and system for airport trolley status recognition and robotic arm guided grasping based on multimodal perception. Background Technology

[0002] Airport baggage trolleys, as frequently used facilities during passenger travel, have long suffered from practical problems such as scattered distribution, reliance on manual retrieval, and unstable dispatch efficiency. With the continuous expansion of large airport terminal areas, increasing passenger density, and growing demands for refined services, the use of service robots with autonomous movement, environmental perception, and robotic arm capabilities for the automatic retrieval, organization, and redeployment of trolleys has become an important direction for airport operations automation. Existing related technologies typically use visual inspection, laser ranging, chassis navigation, and robotic arm grasping as their basic framework. These technologies identify the trolley's position using cameras or LiDAR, and then the robot approaches the target to perform gripping, pulling, or moving operations. Such solutions can complete basic tasks under test conditions of regular placement, minimal obstruction, and individual trolley parking. However, in the actual operation environment of an airport, the status of trolleys is often far more complex than in the experimental environment. Multiple trolleys are often parked in a nested arrangement, and there are also situations where handles and baskets block each other, the distance between the front and rear trolleys is unstable, the trolley is tilted relative to the robot, the wheel system has inconsistent turning angles, and the frame is partially stuck. At the same time, it may also be affected by passengers temporarily retrieving trolleys, luggage being placed, cleaning equipment passing by, and dynamic obstacles in the passageway. Even if the robot can detect the trolley, it is difficult to accurately determine which trolley is more suitable as the current task, which part of the trolley is more suitable for gripper contact, and whether the contact area can support the trolley to separate from the nested queue under real force conditions.

[0003] The main shortcoming of existing technologies is not just insufficient single-frame detection accuracy, but that their perception results usually remain at the level of target location or target category. They lack candidate target screening, contact area generation, and pre-grabbing operability verification processes for nested airport trolley structures. This makes it easy for robots to directly enter the grasping process based on a one-time recognition result, resulting in engineering problems such as misselecting the middle trolley, selecting unstable contact points, the trolley not moving after clamping, current rising but the trolley not separating, and deviation in the pulling direction causing the entire trolley to move in unison.

[0004] These issues can lead to single-task failures, repetitive positioning, robotic arm retraction, and decreased operational efficiency. They may also affect airport passageway order and the stability of continuous equipment operation. Therefore, automated airport trolley retrieval is not simply a matter of "identifying and grabbing trolleys." Rather, it requires building upon existing multimodal perception and robotic grasping technologies to establish a continuous operational chain from candidate trolley identification, target trolley determination, contact area selection, operability verification to grasping execution. This ensures that the robot can confirm the target trolley is indeed accessible, contactable, separable, and capable of being grasped before the actual grasping. Summary of the Invention

[0005] To address the aforementioned issues, this invention provides a method and system for airport trolley status recognition and robotic arm-guided grasping based on multimodal perception.

[0006] To achieve the above objectives, this invention proposes a method for airport cart status recognition and robotic arm-guided grasping based on multimodal perception, comprising: Acquire multimodal sensing data, input the multimodal sensing data into a convolutional neural network and output a candidate cart box, extract sortable state variables from the candidate cart boxes, calculate the structural saliency of the candidate carts based on the state variables, and construct a candidate cart set and an initial state description based on the state variables and structural saliency. Based on the structural saliency, several candidate carts are selected from the candidate cart set to enter the target determination. For each candidate cart entering the target determination, the front-end dominance of the candidate cart in the cart queue is read, and the reachability of the robotic arm is calculated according to the installation position of the robotic arm and the preset workspace boundary of the robotic arm. The target priority value of the candidate cart is calculated based on the front-end dominance and the reachability of the robotic arm. The candidate target cart is confirmed based on the target priority value, and the candidate edge segment corresponding to the candidate target cart is extracted. The score of the candidate edge segment is calculated, and the candidate contact area is determined based on the score. Based on the spatial position and edge direction of the candidate contact area, a pre-contact pose of the end effector is generated, and the robotic arm is driven to move to the position of the candidate contact area. After reaching the pre-contact pose, a trial action is performed and the response is collected. Based on the response, the operability index is calculated, and the candidate contact area is corrected using the actual contact point position and displacement direction to obtain the confirmed contact area and operability status conclusion. Based on the aforementioned operability conclusion, the grasping pose of the end effector is obtained by inverse kinematics solution in combination with the current robotic arm posture. After the gripping is completed based on the grasping pose, the execution strength coefficient of the robotic arm is calculated according to the operability index, and the pull-out action is completed based on the execution strength coefficient.

[0007] In some embodiments, the state variables include normalized spatial location, exposure ratio, and structural integrity markers. Extracting sortable state variables from the trolley candidate box specifically includes: For each candidate cart frame, the depth values ​​of multiple sampling points are read in the corresponding depth map. After removing the sampling values ​​that do not meet the conditions, the normalized distance value of the candidate cart frame is obtained. Edge extraction is performed within the candidate box of the trolley, and the exposed structure of the edge is determined by combining the depth continuity in the depth map. The exposed ratio is calculated based on the exposed structure. The existence of continuous boundaries of the candidate cart frame is checked using LiDAR point cloud. When continuous boundaries exist, a structural integrity marker is assigned.

[0008] In some embodiments, the candidate cart set includes the normalized spatial location, exposure ratio, and structural integrity marker corresponding to each candidate cart, and the initial state description includes the structural saliency and ranking result corresponding to each candidate cart.

[0009] In some embodiments, the step of correcting the candidate contact area using the actual contact point location and displacement direction specifically includes: When contact is established, record the actual spatial point where the end effector contacts the trolley, and use the actual spatial point as the new actual contact point position. When the probing action is performed, if the displacement direction is consistent with the main direction of the candidate contact area, the original direction is maintained as the grasping direction. If there is a deviation, the displacement direction is corrected according to the actual displacement vector.

[0010] In some embodiments, the clamping based on the grasping pose specifically includes: The robotic arm moves to the contact area according to the grasping posture, and approaches the candidate contact area along the normal direction in a gradually decelerating manner, so that the end effector establishes stable contact with the trolley; After contact is established, the gripper performs a closing action according to the allowable gripping range in the candidate contact area, and judges the effective gripping status through gripper drive stroke feedback; When the opening and closing distance remains within the allowable clamping range and the motor current increases, clamping is complete.

[0011] In some embodiments, the convolutional neural network includes a convolutional feature extraction layer, a multi-scale feature fusion layer, and a detection output layer. The convolutional feature extraction layer is used to extract structural features, the multi-scale feature fusion layer is used to concatenate shallow edge information with deep overall contour information, and the detection output layer is used to output cart candidate boxes and cart confidence scores.

[0012] In some embodiments, the structural saliency is calculated and generated by normalizing the proximity of candidate trolleys, the exposure ratio, the structural integrity marker, and a preset weight.

[0013] In some embodiments, the target priority value is calculated and generated using structural salience, front-end dominance, robotic arm reachability, structural integrity markers, and a preset scaling factor.

[0014] In some embodiments, the score of the candidate edge segment is calculated and generated by the exposure ratio, the normalized variation of the depth value within the candidate edge segment range, and the proximity direction matching degree of the candidate edge segment.

[0015] To achieve the above objectives, another aspect of the present invention proposes an airport cart status recognition and robotic arm-guided grasping system based on multimodal perception, comprising: The cart screening module is used to acquire multimodal perception data, input the multimodal perception data into a convolutional neural network and output a cart candidate box, extract sortable state variables from the cart candidate box, calculate the structural saliency of the candidate carts based on the state variables, and construct a candidate cart set and an initial state description based on the state variables and structural saliency. The candidate contact area generation module is used to select several candidate carts from the candidate cart set based on the structural saliency to enter the target determination. For each candidate cart entering the target determination, the module reads the front-end dominance of the candidate cart in the cart queue, calculates the reachability of the robotic arm based on the installation position of the robotic arm and the preset workspace boundary of the robotic arm, calculates the target priority value of the candidate cart based on the front-end dominance and the reachability of the robotic arm, confirms the candidate target cart based on the target priority value, extracts the candidate edge segment corresponding to the candidate target cart, calculates the score of the candidate edge segment, and determines the candidate contact area based on the score. The grasping verification module is used to generate a pre-contact pose of the end effector based on the spatial position and edge direction of the candidate contact area, drive the robotic arm to move to the position of the candidate contact area, perform a trial action after reaching the pre-contact pose, and collect the response amount. Based on the response amount, the operability index is calculated, and the candidate contact area is corrected using the actual contact point position and displacement direction to obtain the confirmed contact area and operability status conclusion. The gripping execution module is used to obtain the gripping pose of the end effector by inverse kinematics solution based on the operability conclusion and the current posture of the robotic arm. After gripping is completed based on the gripping pose, the execution intensity coefficient of the robotic arm is calculated according to the operability index, and the pull-out action is completed based on the execution intensity coefficient.

[0016] The beneficial effects of this invention are as follows: This application first uses color images, depth maps, and LiDAR point clouds to form a candidate cart set, and organizes each candidate cart into an initial state description including spatial location, exposed proportion, structural integrity, and structural salience, so that the multimodal perception results can reflect the structural differences in the nested queue of airport carts. Then, based on this state description, candidate target carts are determined by combining queue front features and robotic arm reachability features, and further candidate contact areas are generated based on exposed structure, depth stability, and gripper approach direction, so that the system obtains a clear work object and contact position. Then, the robotic arm performs controlled short-range probing actions at the candidate contact areas, and uses end displacement, motor current changes, and response direction consistency to determine whether the target cart has an operable state that can be separated and stably contacted under real force conditions, and forms an operable state conclusion and a confirmed contact area accordingly. Finally, based on the conclusion and the confirmed contact area, grasping guidance information for robotic arm approach, gripping, and pulling out is generated, and the grasping execution intensity is adjusted according to operability and contact stability to complete the actual grasping and preliminary separation actions.

[0017] Through the above improvements, the present invention will see the trolley advance to determine the operable trolley, and further advance to perform grasping according to the verified contact area. This allows the trolley state recognition result to directly serve the generation of robotic arm movements and grasping execution, overcoming the problem of the disconnect between target detection, contact selection, physical separability judgment and grasping control in the prior art. Therefore, it is more suitable for the continuous operation requirements of trolley nesting, partial occlusion and high grasping failure cost in real airport scenarios. Attached Figure Description

[0018] Figure 1 This is a flowchart of the airport cart status recognition and robotic arm guided grasping method based on multimodal perception in a specific embodiment of the present invention; Figure 2 This is a block diagram of an airport cart status recognition and robotic arm-guided grasping system based on multimodal perception, as described in a specific embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] refer to Figure 1 As shown, one embodiment of this application proposes a method for airport cart status recognition and robotic arm-guided grasping based on multimodal perception, including: S1: Acquire multimodal sensing data, input the multimodal sensing data into a convolutional neural network and output candidate cart boxes, extract sortable state variables from the candidate cart boxes, calculate the structural saliency of the candidate carts based on the state variables, and construct a candidate cart set and an initial state description based on the state variables and structural saliency, specifically including: This step serves as the initial step of the entire solution, receiving multimodal perception data collected in real-time by the robot within the airport's operational area. Specifically, this includes color images captured by the head and chest cameras, depth maps output from the RealSense D435 and aligned with the color images, and planar point clouds output from the vehicle's LiDAR. During actual operation, as the robot approaches the cart recovery point, multiple nested carts, pillars, suitcases, and passing personnel may simultaneously appear in the camera feed. The depth map provides the depth distribution of each candidate object relative to the front of the robot, while the LiDAR point cloud provides the lower contour and obstacle boundary information. The system inputs the color images into a convolutional neural network for airport cart detection. This network employs a three-segment structure: the first segment is a five-layer convolutional feature extraction layer, where each layer uses local convolution to extract structural features such as straight bounding boxes, handle curves, basket edges, and wheel frame edges; the second segment is a multi-scale feature fusion layer, which stitches together shallow edge information with deep overall contour information to account for both the large contours of nearby carts and the small targets of carts at a distance; the third segment is the detection output layer, which outputs the candidate cart bounding boxes and the cart confidence scores. The network's training samples are derived from images of trolleys in airport or simulated airport environments. The sample annotations include the trolley's bounding box and category. During runtime, the network outputs several candidate trolley bounding boxes. Then, for each candidate bounding box, depth values ​​from multiple sampling points in the corresponding depth map are read. The effective depths of the candidate bounding box's center region, handle region, and basket edge region are prioritized. After removing obviously missing or abrupt depth points, the normalized distance value of the candidate trolley is obtained. Airport trolleys often have local cutouts and metallic reflections, making single-point depth measurements prone to fluctuation. Multi-point sampling provides a more stable positional description of the candidate trolleys.

[0021] The system further extracts the exposed structure of the carts within each candidate box. Specifically, it first extracts edges within the candidate box and then uses depth continuity data from the depth map to determine if edges belong to the same cart structure. Continuous handle edges, basket edges, and side frame edges that are not interrupted by objects in front are counted as exposed structures; edges interrupted by other carts are not counted. This yields the exposure ratio, ranging from 0 to 1. The lead cart at the front of the tandem typically has more complete handles, basket edges, and side frames, resulting in a higher exposure ratio; while middle carts nested within the lead cart may also be detected, their handles or basket edges are obscured, significantly reducing their exposure ratio. The system then uses LiDAR point clouds to check for continuous contours in the lower part of the candidate region: if the region forms a continuous boundary in the point cloud, it is marked as valid for structural integrity; if the point cloud is sparse, broken, or inconsistent with the candidate box position, the candidate target is marked as structurally incomplete or a low-confidence target. In this way, each candidate cart is organized into a candidate object that includes markers of spatial location, exposure ratio, and structural integrity, rather than simply retaining the detection box in the image.

[0022] To combine distance, exposure, and structural integrity into a single, orderable state variable, this step employs a structural saliency improved from the classic weighted scoring method. Weighted scoring is a linear weighted evaluation method in mathematics, the original form of which involves multiplying multiple unidirectional evaluation indicators by their weights and then summing them. This application makes two modifications specifically for the airport cart scenario: first, the distance indicator, which prioritizes smaller distances, is replaced with a proximity indicator, avoiding directly adding distance to the exposure ratio; second, a structural integrity marker is introduced, ensuring that severely occluded or discontinuous point cloud candidates are not prioritized simply because they are close. The specific calculation is as follows: ; in, Indicates the first The structural saliency of each candidate cart is calculated in this step and used to rank the candidate carts. Indicates the first The normalized proximity of each candidate cart is obtained by converting the distance value after sampling the depth map. The value ranges from 0 to 1, and the larger the value, the more suitable it is as the current task object. Indicates the first The exposed proportion of each candidate cart is obtained by the proportion of the continuously visible cart edge within the candidate box, and the value ranges from 0 to 1; Indicates the first The structural integrity markers of each candidate trolley are determined by the continuity of the lidar point cloud; if continuous, the marker is 1, and if discontinuous, it is 0. , , The preset weights are derived from experience set during the system debugging phase for airport cart scenarios, and the sum of the three is 1. All terms in the formula are normalized dimensionless quantities, therefore they can be added together; if the original distance is involved in the calculation, it should first be converted to normalized proximity. The ratio must not be directly added to the exposed ratio. In a typical operation at an airport cart recovery point, the robot detected three candidate carts. Candidate cart A had a normalized proximity of 0.70, an exposed ratio of 0.85, and a structural integrity label of 1; candidate cart B had a normalized proximity of 0.90, an exposed ratio of 0.30, and a structural integrity label of 1; and candidate cart C had a normalized proximity of 0.60, an exposed ratio of 0.50, and a structural integrity label of 0. (The last sentence appears to be incomplete and possibly refers to a different calculation method.) , , The structural significance of candidate stroller A is 0.30×0.70+0.50×0.85+0.20×1=0.835; the structural significance of candidate stroller B is 0.30×0.90+0.50×0.30+0.20×1=0.620. The structural significance of candidate cart C is 0.30×0.60+0.50×0.50+0.20×0=0.430. Although candidate cart B is closer to the robot, its exposed proportion is lower, indicating that it is more likely to be an intermediate cart in a nested queue or an occluded cart; candidate cart A has a complete exposed structure and continuous point cloud, which better meets the engineering requirement of prioritizing the first cart in airport cart recovery.

[0023] After completing the above processing, this step outputs a set of candidate carts. and initial state description Candidate stroller collection It consists of multiple candidate carts, each containing normalized spatial location, exposed proportion, and structural integrity markers; initial state description. Record the structural saliency of each candidate cart. And its sorting results. These will be used in subsequent steps. and The candidate target carts and candidate contact areas are identified. This step transforms the multimodal perception results into a state description oriented towards the nested structure of airport carts, enabling subsequent processing to continue using the same set of candidate objects.

[0024] S2: Based on the structural saliency, select several candidate carts from the candidate cart set to enter the target determination. For each candidate cart entering the target determination, read the front-end dominance of the candidate cart in the cart queue, and calculate the reachability of the robotic arm based on the installation position of the robotic arm and the preset workspace boundary of the robotic arm. Calculate the target priority value of the candidate cart based on the front-end dominance and the reachability of the robotic arm. Confirm the candidate target cart based on the target priority value, extract the candidate edge segment corresponding to the candidate target cart, calculate the score of the candidate edge segment, and determine the candidate contact area based on the score, specifically including: This step follows the candidate cart set output by S1. and initial state description Candidate target carts are determined from the same set of candidate objects. And generate candidate contact areas on the candidate target cart. In S1, The normalized spatial location and exposed proportion of each candidate stroller have been recorded. and structural integrity markers , The structural saliency of each candidate cart has been recorded. And the sorting results. This step further transforms this state information into targets and areas that the robotic arm can subsequently execute: first from the candidate cart set... Select the most suitable candidate target cart to enter the verification process And then On the exposed structure, select a suitable candidate contact area for the two-finger gripper to approach. In airport trolley retrieval operations, trolleys are often nested in a series. Trolleys that are visually obvious may not be at the front of the queue, and trolleys that are close to each other may not be within the comfortable reach range of the robotic arm. Therefore, this step, based on the structural salience in S1, further introduces features of the front of the queue and features that the robotic arm can reach, so that the target selection is further transformed from "structurally obvious" to "suitable for performing subsequent operations".

[0025] The system first follows Structural saliency in right The candidate carts are sorted in descending order, and the top-ranked candidate carts are selected for target determination. For each candidate cart entering the determination, the system reads its... The normalized spatial position of the candidate cart is obtained by projecting this position onto the robot's current forward direction, thus determining the degree of front-end dominance of the candidate cart in the cart queue. In practice, the robot's current direction of travel is determined by the chassis odometer and navigation controller. The normalized spatial position of the candidate trolleys is derived from the fusion of the depth map and radar data in S1. The system projects the position vector of the candidate trolleys relative to the robot's coordinate system onto the direction of travel and normalizes it based on the foremost and rearmost positions within the same candidate set, resulting in a value ranging from 0 to 1. The higher the value, the more closely the cart matches the characteristics of the lead cart in an airport cart series. Simultaneously, the system calculates the reachability of the robotic arm based on its installation location and the preset workspace boundaries. The specific method involves reading the normalized spatial position of the candidate trolley relative to the robotic arm base, determining whether it is within the stable working range of the robotic arm, and normalizing it based on its deviation from the center of the workspace, resulting in a value ranging from 0 to 1. This approach corresponds to a situation in actual engineering: although a trolley located at the edge of the robotic arm's workspace is visible, the robotic arm may need to approach an unusual posture or extend significantly, resulting in poor stability during subsequent grasping; a trolley located in the middle of the workspace is more suitable as the object to be manipulated.

[0026] The target selection employs a priority value derived from the mathematical concepts of linear weighted evaluation and gating. The original form of linear weighted evaluation involves proportionally weighting and summing multiple indices to express the combined impact of multiple factors on the same target. The original idea of ​​gating is to suppress the overall evaluation result when a necessary condition is not met. This application modifies these approaches to suit specific scenarios: it retains the structural salience obtained in S1. As a baseline candidate quality, the degree of front-end dominance will be considered. And the extent that robotic arms can reach Weighted job adaptation factors are generated and structural integrity markers are used. As a gating term, candidate carts with discontinuous point cloud structures are excluded from subsequent validation. The specific calculation is as follows: ; in, Indicates the first The target priority value of each candidate cart is calculated in this step and is used to select from the candidate cart set. Candidate target trolleys were identified in the middle. ; Indicates the output of S1 of the first... The structural saliency of each candidate cart is derived from the initial state description. ; Indicates the first The degree of front-end dominance of each candidate trolley is calculated from the projection of the normalized spatial position of the candidate trolley along the robot's forward direction, with a value ranging from 0 to 1. Indicates the first The reachability of the robotic arm for each candidate trolley is calculated by the normalized relationship between the position of the candidate trolley relative to the robotic arm base and the preset workspace of the robotic arm, with a value range of 0 to 1. The structural integrity flag output by S1 is determined by the continuity of the lidar point cloud; a value of 1 indicates continuity and a value of 0 indicates breakage. and This represents a preset proportional coefficient used to adjust the influence of the queue front-end features and the reachability features of the robotic arm; the sum of the two is 1. In the formula... , , , , , All are normalized proportions or labeled quantities, left side It is also a normalized evaluation value, and the calculation relationship remains consistent. This formula has a progressive relationship with the structural salience of S1: S1 solves "which cart has a more obvious structure", and this step further solves "which cart with an obvious structure is more suitable as the current robotic arm operation target".

[0027] During an operation at an airport cart return point, S1 received three candidate carts. The structural saliences recorded in the data are as follows: Candidate cart A The value is 0.835, which is the value of candidate stroller B. The value is 0.620, which is the value of candidate trolley C. The value is 0.430. Based on the normalized projection results of the candidate trolleys along the robot's forward direction, the system obtains the front-end dominance of the three trolleys as 0.90, 0.55, and 0.40, respectively. Based on the workspace relationship of the robotic arm, the system obtains the reachability of the three trolleys as 0.80, 0.95, and 0.60, respectively, and the structural integrity labels as 1, 1, and 0, respectively. (The last sentence appears to be incomplete and possibly refers to a different system or approach.) , Then the target priority value of candidate cart A is The target priority value of candidate cart B is The target priority value of candidate cart C is Therefore, candidate cart A was selected as the candidate target cart. The calculation results reflect the actual selection logic in airport cart operations: although candidate cart B has a higher reachability with the robotic arm, its queue front features and structural saliency are insufficient; candidate cart C is suppressed by gating terms because its structural integrity is marked as 0; candidate cart A has high structural saliency, queue front features, and stable reachability, making it more suitable for subsequent operability verification.

[0028] Identify candidate target carts Then, the system generates candidate contact areas within the image candidate area corresponding to the trolley. System read exist The proportion of inheritance in China The system normalizes spatial location and structural integrity markers and extracts continuous edge segments within the candidate regions of the trolley's image. These continuous edge segments are categorized into candidate segments for the handle beam, the top edge of the basket, and the side frame, based on direction and location. The two-finger gripper of the airport trolley is better suited for contacting the handle beam or the top edge of the basket, as these areas typically have high exposure, continuous edges, and stable contact directions, allowing the robotic arm to approach and grip the trolley laterally or obliquely. The system reads multiple sampling points from the corresponding depth map for each candidate edge segment and calculates the normalized depth variation within that edge segment's range. The smaller the depth variation, the smoother and less obstructed the corresponding spatial surface of the edge segment. The system also calculates the approach direction matching degree based on the angle between the edge segment's direction and the robotic arm's gripper's preset approach direction. The more suitable the angle is for the closing direction of the two-finger gripper, the better. The higher the level, the better. Therefore, the selection of the contact area is not only based on "visibility", but also on "structural stability" and "reasonable approach direction of the grippers".

[0029] The contact area score was derived using a product-type comprehensive evaluation method. A common application of product-type evaluation is to express the common constraints between multiple necessary conditions; when any condition is significantly insufficient, the overall score decreases. This application applies this method to the selection of contact areas for airport carts, incorporating the exposed proportion of candidate target carts, the depth stability of candidate edge segments, and the matching degree of the robotic arm's approach direction into the score. The specific calculation is as follows: ; in, Indicates candidate target cart Upper The contact area score of each candidate edge segment is used to select candidate contact areas. ; Indicates candidate target cart The exposed proportion is derived from the exposed proportion of the corresponding candidate cart in S1, and follows... From the candidate cart collection Inheritance; Indicates the first The normalized variation of depth values ​​within a candidate edge segment is obtained by normalizing the maximum variation of the depth sampling point corresponding to that edge segment, and the value range is from 0 to 1. Indicates the first The proximity direction matching degree of each candidate edge segment is obtained by normalizing the angle between the edge segment's direction and the robot arm's gripper's preset proximity direction, with a value ranging from 0 to 1. Where... , and All are normalized proportions, left side This is also a normalized score, maintaining a consistent calculation relationship. This formula aligns with the target priority value. The logical relationship is as follows: First, identify the target of the operation at the candidate trolley level. , Then in Determining candidate contact regions at local edge levels These two correspond to "which car to choose" and "which section of structure to contact," respectively.

[0030] On candidate trolley A, the system extracted three candidate edge segments: the handle crossbeam segment, the top edge of the basket segment, and the vertical segment of the side frame. The exposed proportion of A... The score is 0.85; the depth variation of the handle beam section is 0.08, and the proximity direction matching degree is 0.90, so the contact area score for this section is... The depth variation range of the upper edge of the basket is 0.15, and the approach direction matching degree is 0.75. Therefore, the score of the contact area of ​​this section is... The depth variation of the vertical segment of the side frame is 0.10, and the proximity direction matching degree is 0.45. Therefore, the score of the contact area of ​​this segment is... The system selects the handle beam segment with the highest score as the candidate contact area. The center point, edge direction, and two endpoints of the edge segment are transformed from image coordinates to robot coordinates, forming a spatial region that can be directly used for subsequent verification of the robotic arm. This candidate contact area is expressed as a narrow band region, including the center position, direction, and spatial range, which is adapted to the gripping width, positioning error, and slight attitude deviation of the trolley's metal frame in actual operation of the two-finger gripper.

[0031] This step ultimately outputs candidate target carts. and candidate contact area Candidate target cart Collection of candidate trolleys Medium target priority value The highest-ranking candidate stroller is determined, and that candidate stroller is inherited. Normalized spatial location, exposure ratio, and structural integrity markers; candidate contact areas Candidate target trolley Upper contact area score The highest candidate edge segment is determined, including the region's center position, orientation, and spatial extent in the robot coordinate system. Subsequent steps are based on... and To perform operability verification on the input, the robotic arm performs verification actions continuously around the same target and the same contact area.

[0032] S3: Based on the spatial position and edge direction of the candidate contact area, generate a pre-contact pose for the end effector, drive the robotic arm to move to the position of the candidate contact area, perform a trial action after reaching the pre-contact pose, and collect the response data. Calculate the operability index based on the response data, and correct the candidate contact area using the actual contact point position and displacement direction to obtain a confirmed contact area and operability status conclusion, specifically including: This step uses the candidate target cart output by S2. and candidate contact area Using this as the sole input, a controlled short-range interactive motion of the robotic arm is used to quantitatively analyze the response of the cart under real force conditions, thereby determining whether the cart is in a graspable state and precisely correcting the contact area. S2 has determined the most suitable cart for operation based on structural saliency, queue front features, and robotic arm reachability features. Candidate contact areas were given on its exposed structure. This step introduces physical interaction verification on this basis, which further transforms the judgment of "visually suitable for grasping" into the judgment of "mechanically separable", thereby avoiding nested carts or stuck carts from entering the grasping stage.

[0033] The robotic arm first based on Given the spatial center position and edge direction, the pre-contact pose of the end effector is generated, and the end effector moves to a position a preset normalized distance from the contact area. This pre-contact pose is obtained through inverse kinematics of the robotic arm, with the input being... The spatial coordinates and direction vectors are used to output the angles of each joint. The robotic arm then slowly advances along the normal direction until the grippers make slight contact with the cart structure. At this point, the contact is confirmed by the continuous change in the joint motor current. After contact is confirmed, the robotic arm performs a short-range probing motion along the main direction, which is directly inherited from the edge direction of the contact area in S2, such as the direction of the handle beam or the top edge of the basket. The probing motion is position-controlled, given a normalized displacement command. This value is determined during the system debugging phase and remains consistent in each trial to ensure comparability between different carts.

[0034] During the trial operation, the system simultaneously collects three response quantities: the actual displacement change of the end effector. The change in joint motor current is calculated by the joint encoder of the robotic arm through forward kinematics. Real-time data acquisition is achieved through the driver interface; and the change in the angle between the actual displacement direction and the main direction of the contact area is analyzed by comparing the end displacement vector with... The direction vector is obtained and normalized into a direction consistency index. These three types of responses correspond to the trolley's displacement capability, resistance level, and release direction characteristics under force, respectively, and together reflect whether the trolley is in a separable state. In classic impedance response analysis, the system's mobility is usually judged by the "relationship between displacement response and applied force." This step builds upon this by indirectly representing the applied force using changes in motor current and introducing a directional consistency term to make the indicators more closely resemble the actual scenario of nested trolleys in airports.

[0035] Based on the above ideas, construct actionable indicators. Its expression is: ; in, Indicates candidate target cart In the contact area Operational indicators at the location; The normalized displacement command representing the robotic arm's probing motion is preset by the control system; This represents the normalized actual displacement change of the end effector when performing the action, calculated from the joint encoder data; This represents the normalized change in the joint motor current during the trial process, measured by the driver. This represents the maximum allowable normalized current variation of the system, obtained from the equipment safety parameter calibration. This indicates the consistency of displacement direction, obtained by normalizing the angle between the actual displacement direction and the main direction of the contact area, with a value ranging from 0 to 1. All the above quantities have been normalized. (Left side) It is also a normalized comprehensive index. This formula originates from the evaluation form of the "displacement-force" relationship in impedance response. Based on this, a direction consistency term is added, and a product form is used to ensure that the three conditions of sufficient displacement response, low resistance, and reasonable direction are simultaneously met. A high value was achieved.

[0036] In practical applications, this index can be calculated using specific numerical values. For example, a normalized displacement command can be set. The actual displacement was measured during a certain test. Current change The maximum current variation calibrated by the system Consistency in direction Then the operability index is In another case, if the actual displacement Current change Consistency in direction ,but In the first scenario, the cart experiences significant displacement with relatively low resistance, and the direction aligns with the contact area, indicating the cart can be separated from the queue. In the second scenario, the cart displacement is minimal while resistance is high, suggesting nesting or jamming. The system will... The threshold was compared with a preset threshold, which was determined during the debugging phase using multiple sets of trolley experimental data. The candidate target cart is determined when the value is higher than the threshold. It is in a crawlable state; otherwise, it is determined to be in an uncrawlable state.

[0037] In calculation At the same time, the system uses the actual contact point location and displacement direction to determine the candidate contact area. Make corrections to obtain the confirmed contact area. The specific procedure is as follows: when a light contact is established, record the actual spatial point of contact between the gripper and the cart, and use this point as a new candidate contact center; when the trial action is performed, if the actual displacement direction is... If the main direction is consistent, the original direction is maintained as the grasping direction; if there is a slight deviation, the direction is slightly corrected based on the actual displacement vector. This correction process is still based on the output of S2. It simply makes fine adjustments to the physical contact conditions so that subsequent grasping actions are based on the actual contact state rather than pure visual estimation.

[0038] This step ultimately outputs two results: an operational status conclusion. and confirmed contact areas .in, Includes candidate target carts The results of the judgment on whether the crawlable conditions are met and the corresponding operability indicators. The judgment result is determined by Obtained by comparison with a preset threshold; This represents the contact area confirmed after actual contact and probing actions, including the updated contact center position, contact direction, allowable gripping range, local depth stability information, and response direction consistency information. Through this step, the candidate target cart and contact area obtained in S2 based on multimodal perception and geometric relationships are further verified as operable objects and operation positions under real mechanical conditions, providing a direct basis for subsequent grasping actions.

[0039] S4: Based on the aforementioned operability conclusion, the grasping pose of the end effector is obtained through inverse kinematics solution in combination with the current robotic arm posture. After gripping based on the grasping pose, the execution intensity coefficient of the robotic arm is calculated according to the operability index. The pull-out action is then completed based on the execution intensity coefficient, specifically including: This step concludes with the operational status output by S3. and confirmed contact areas As input, the physically verified cart state is directly converted into a grasping action that the robotic arm can perform. In S3, Candidate target carts have already been included. The results of the judgment on whether the crawlable conditions are met and the corresponding operability indicators. , It already includes the actual contact center location, contact direction, allowable clamping range, local depth stability information, and response direction consistency information. This step revolves around... and A grasping guidance process is constructed so that the robotic arm's movements are directly executed based on the verified contact states.

[0040] Before executing the capture, the system first reads The determination result in the middle; when the determination result indicates that the candidate target cart When the grasping conditions are met, the robotic arm controller reads... The contact center point, contact direction, and allowable gripping range are combined with the current robot arm posture to obtain the grasping pose of the end effector through inverse kinematics. The inverse kinematics solution is based on the mapping relationship between the end effector pose and joint space in robotics. Its inputs are the position and direction vectors of the contact point in the robot coordinate system, and the outputs are the angles of each joint of the robot arm. The robot arm moves to the vicinity of the contact area according to the solution results and approaches along the normal direction with gradual deceleration, establishing stable contact between the gripper and the trolley structure. After contact is established, the gripper... The system performs a closing action within the allowable clamping range and determines whether effective clamping has been achieved through feedback of the gripper drive stroke. When the opening and closing distance remains within the allowable clamping range and the motor current shows a stable increase, the clamping is considered complete.

[0041] After clamping is complete, the trolley pulling-out stage begins. This stage is based on... Operational indicators carried in The robotic arm's actuation intensity is adjusted to overcome nested resistance without generating excessive impact. The control system employs proportional control from classical control theory, correlating the actuation intensity with system state variables. A contact area stability factor is then introduced to correct the actuation intensity. This contact area stability factor is derived from... The recorded local depth stability information, gripper stroke maintenance within the allowable gripping range, and response direction consistency information are used to reflect the reliability of the gripper's contact with the cart structure. Based on these factors, a gripping execution strength coefficient is constructed: ; in, This represents the gripping execution intensity coefficient, used to adjust the speed and displacement amplitude of the robotic arm during the pull-out phase; This represents the operational state conclusion output by S3. The operability index carried in the vehicle reflects the displacement response, resistance level, and consistency of response direction of the trolley under stress conditions. The contact area stability factor is determined by the confirmed contact area. The formula is derived from a combination of local depth stability information, gripper stroke maintenance, and response direction consistency information recorded in the database, with a value range of 0 to 1. This formula originates from the basic form of proportional control, where "output is proportional to state quantity," and is further improved by introducing... Constraining local contact stability ensures that even with high overall operability, local instability in the contact area reduces execution strength. All terms in the formula are normalized quantities; therefore, the left side... It remains a dimensionless scale value and can be directly used to scale the preset maximum speed or displacement of the robotic arm.

[0042] In actual implementation, for example, S3 obtains a certain cart Contact area stability factor ,but If the system's preset maximum normalized pull-out speed is 1.00, then the robotic arm's actual execution speed is 0.52; if the other trolley's... and ,but The robotic arm will then perform the pull-out action at a higher speed. During the pull-out process, the robotic arm moves along the main direction, which is inherited from the modified contact direction in S3, ensuring that the trolley separates from the queue along the path of least resistance. During the pull-out process, the end-effector displacement is continuously monitored by the joint encoder. The pull-out stops when the displacement reaches the preset release distance. This release distance is set by experience based on the trolley nesting depth during the equipment commissioning phase to ensure that the trolley is completely removed from the original queue.

[0043] After completing the pull-out action, the robotic arm maintains its gripping state and proceeds to the subsequent handling or retrieval process. The entire gripping execution process is handled by... Provides spatial location and orientation information, by and The intensity of execution is determined and translated into specific mechanical movements through inverse kinematics and joint control, achieving a closed loop from "state recognition" to "actual grasping." The computational relationships in this process are related to the operability index of S3. Formation of logical continuity: S3 through This step determines whether the conditions for crawling are met. The judgment result is mapped to a specific execution intensity, thereby ensuring that the system can adopt appropriate operation strategies in different cart states.

[0044] refer to Figure 2 As shown, to achieve the above objectives, another aspect of this application proposes an airport trolley state recognition and robotic arm-guided grasping system based on multimodal perception, including: The cart screening module is used to acquire multimodal perception data, input the multimodal perception data into a convolutional neural network and output a cart candidate box, extract sortable state variables from the cart candidate box, calculate the structural saliency of the candidate carts based on the state variables, and construct a candidate cart set and an initial state description based on the state variables and structural saliency. The candidate contact area generation module is used to select several candidate carts from the candidate cart set based on the structural saliency to enter the target determination. For each candidate cart entering the target determination, the module reads the front-end dominance of the candidate cart in the cart queue, calculates the reachability of the robotic arm based on the installation position of the robotic arm and the preset workspace boundary of the robotic arm, calculates the target priority value of the candidate cart based on the front-end dominance and the reachability of the robotic arm, confirms the candidate target cart based on the target priority value, extracts the candidate edge segment corresponding to the candidate target cart, calculates the score of the candidate edge segment, and determines the candidate contact area based on the score. The grasping verification module is used to generate a pre-contact pose of the end effector based on the spatial position and edge direction of the candidate contact area, drive the robotic arm to move to the position of the candidate contact area, perform a trial action after reaching the pre-contact pose, and collect the response amount. Based on the response amount, the operability index is calculated, and the candidate contact area is corrected using the actual contact point position and displacement direction to obtain the confirmed contact area and operability status conclusion. The gripping execution module is used to obtain the gripping pose of the end effector by inverse kinematics solution based on the operability conclusion and the current posture of the robotic arm. After gripping is completed based on the gripping pose, the execution intensity coefficient of the robotic arm is calculated according to the operability index, and the pull-out action is completed based on the execution intensity coefficient.

[0045] The above embodiments are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A method for airport trolley status recognition and robotic arm-guided grasping based on multimodal perception, characterized in that, include: Acquire multimodal sensing data, input the multimodal sensing data into a convolutional neural network and output a candidate cart box, extract sortable state variables from the candidate cart boxes, calculate the structural saliency of the candidate carts based on the state variables, and construct a candidate cart set and an initial state description based on the state variables and structural saliency. Based on the structural saliency, multiple candidate carts are selected from the candidate cart set to enter the target determination. For each candidate cart entering the target determination, the front-end dominance of the candidate cart in the cart queue is read, and the reachability of the robotic arm is calculated according to the installation position of the robotic arm and the preset workspace boundary of the robotic arm. The target priority value of the candidate cart is calculated based on the front-end dominance and the reachability of the robotic arm. The candidate target cart is confirmed based on the target priority value, and the candidate edge segment corresponding to the candidate target cart is extracted. The score of the candidate edge segment is calculated, and the candidate contact area is determined based on the score. Based on the spatial position and edge direction of the candidate contact area, a pre-contact pose of the end effector is generated, and the robotic arm is driven to move to the position of the candidate contact area. After reaching the pre-contact pose, a trial action is performed and the response is collected. Based on the response, the operability index is calculated, and the candidate contact area is corrected using the actual contact point position and displacement direction to obtain the confirmed contact area and operability status conclusion. Based on the operational state conclusion, the grasping pose of the end effector is obtained by inverse kinematics solution in combination with the current robot arm posture. After the gripping is completed based on the grasping pose, the execution intensity coefficient of the robot arm is calculated according to the operational index, and the pull-out action is completed based on the execution intensity coefficient.

2. The airport trolley state recognition and robotic arm guided grasping method based on multimodal perception according to claim 1, characterized in that, The candidate cart set includes the normalized spatial location, exposure ratio, and structural integrity marker for each candidate cart, and the initial state description includes the structural saliency and ranking result for each candidate cart.

3. The airport trolley state recognition and robotic arm guided grasping method based on multimodal perception according to claim 1, characterized in that, The convolutional neural network includes a convolutional feature extraction layer, a multi-scale feature fusion layer, and a detection output layer. The convolutional feature extraction layer is used to extract structural features, the multi-scale feature fusion layer is used to concatenate shallow edge information with deep overall contour information, and the detection output layer is used to output cart candidate boxes and cart confidence scores.

4. An airport trolley status recognition and robotic arm guided grasping system based on multimodal perception, characterized in that: include: The cart screening module is used to acquire multimodal perception data, input the multimodal perception data into a convolutional neural network and output a cart candidate box, extract sortable state variables from the cart candidate box, calculate the structural saliency of the candidate carts based on the state variables, and construct a candidate cart set and an initial state description based on the state variables and structural saliency. The candidate contact area generation module is used to select multiple candidate carts from the candidate cart set based on the structural saliency to enter the target determination. For each candidate cart entering the target determination, the module reads the front-end dominance of the candidate cart in the cart queue, calculates the reachability of the robotic arm based on the installation position of the robotic arm and the preset workspace boundary of the robotic arm, calculates the target priority value of the candidate cart based on the front-end dominance and the reachability of the robotic arm, confirms the candidate target cart based on the target priority value, extracts the candidate edge segment corresponding to the candidate target cart, calculates the score of the candidate edge segment, and determines the candidate contact area based on the score. The grasping verification module is used to generate a pre-contact pose of the end effector based on the spatial position and edge direction of the candidate contact area, drive the robotic arm to move to the position of the candidate contact area, perform a trial action after reaching the pre-contact pose, and collect the response amount. Based on the response amount, the operability index is calculated, and the candidate contact area is corrected using the actual contact point position and displacement direction to obtain the confirmed contact area and operability status conclusion. The gripping execution module is used to obtain the gripping pose of the end effector by inverse kinematics solution based on the operability state conclusion and the current robot arm posture. After gripping is completed based on the gripping pose, the execution intensity coefficient of the robot arm is calculated according to the operability index, and the pull-out action is completed based on the execution intensity coefficient.

Images