A luggage empty basket identification and grabbing method and device

Abstract

The application discloses a kind of luggage empty basket identification method and device, the method includes collecting luggage conveyor belt image and surrounding personnel movement situation, according to surrounding personnel movement situation, motion trajectory and behavior prediction are carried out to surrounding environment personnel, based on the prediction result, the motion trajectory of grabbing manipulator is calculated, including based on the optimal motion trajectory obtained by establishing multi-target trajectory planning, the grabbing sequence of multiple grabbing targets is determined and algorithm optimization is carried out.The application realizes efficient luggage empty basket grabbing, can simultaneously recycle multiple empty baskets and based on the motion trajectory and action of passenger, the state of luggage basket is predicted in advance, ensures the operation stability and safety of empty basket recycling manipulator under the condition of considering optimal time and energy consumption, for the path mode that already has in historical data, path planning can be made more quickly and simply.

Description

Technical Field

[0001] This invention relates to the field of luggage empty basket identification and grasping, and in particular to a method and apparatus for luggage empty basket identification and grasping. Background Technology

[0002] With the continuous increase in air passenger traffic and the ongoing expansion of modern hub airport terminals, the efficient handling capacity of baggage systems faces severe challenges. For non-standardized baggage such as soft-sided bags and irregularly shaped backpacks, these items are properly loaded into baggage baskets during check-in. This ensures the stability of baggage transport on high-speed conveyor lines and effectively prevents flexible bags from becoming tangled during transit.

[0003] Existing technologies include automatic empty baggage basket grabbing systems, which use vision cameras to capture images of empty baskets and transmit them in real time to a collaborative robot, which then grabs the empty basket. However, the grabbing robot in existing technologies can only grab one basket at a time, which is inefficient. Furthermore, it cannot achieve intelligent planning and obstacle avoidance, nor can it predict passengers' baggage retrieval actions in advance, resulting in low overall efficiency and poor performance. Summary of the Invention

[0004] (a) Technical problems to be solved

[0005] To address the aforementioned technical problems, this invention provides a method and apparatus for identifying and grabbing empty luggage baskets.

[0006] (II) Technical Solution

[0007] To solve the aforementioned technical problems and achieve the invention's objective, the present invention is implemented through the following technical solution:

[0008] A method for identifying and grabbing empty luggage baskets includes the following steps:

[0009] S1: Image acquisition and preprocessing, including capturing images of the baggage conveyor belt and the movement of people in the surrounding area via a camera;

[0010] S2: Luggage basket recognition and location tracking;

[0011] The location of the luggage basket in the image is identified based on its specific shape, and the position of the luggage basket is tracked by combining the running speed of the conveyor belt.

[0012] S3: Based on the movement of people in the surrounding environment, predict the movement trajectory and behavior of people in the surrounding environment. If it is identified that the person intends to pick up luggage, the time when the luggage basket will be empty and the moving position will be obtained in the future based on the speed of their movement and the speed of the conveyor belt.

[0013] S4: Calculate the motion trajectory of the gripping robot based on the prediction results, including obtaining an optimal motion trajectory based on the establishment of multi-target trajectory planning, and determining the gripping order of multiple gripping targets;

[0014] S5: Optimization of motion trajectory planning algorithm;

[0015] Furthermore, step S3 also includes processing the image captured by the camera based on the YOLOV5 target detection algorithm, extracting pedestrians from the background area of ​​the image, adding a 1×1 convolution module between the YOLOV5 and CSPBlock recognition models, and using max pooling and average pooling in parallel in the spatial pyramid pooling module and adding the results of the two.

[0016] Furthermore, the prediction of the movement trajectory of people in the surrounding environment includes prediction based on the pedestrian's historical trajectory sequence and pedestrian direction.

[0017] Furthermore, the action recognition model based on graph neural networks in the behavior prediction achieves action data modeling and classification by constructing a dynamic relational topology.

[0018] Furthermore, the objective function for the multi-objective trajectory planning is as follows:

[0019] L = w1L1 + w2L2 + w3L3 + ε

[0020] L1 is the time objective function; L2 is the energy consumption objective function; L3 is the risk level objective function; ε is the penalty factor; w1, w2, and w3 are the weighting factors of the three respectively.

[0021] Furthermore, the L3 risk level objective function is determined based on the height of the robotic arm when it moves after gripping the empty basket and when it rotates, the distance from surrounding personnel, and motion errors.

[0022]

[0023] r1 is the distance risk weight during linear motion; dis1 is the minimum distance between the robotic arm and surrounding personnel during linear motion; σ1 is the distance sensitivity coefficient during linear motion; h1 is the end-effector height of the robotic arm during linear motion; δ1 is the height adjustment coefficient during linear motion; r2 is the distance risk weight during rotational motion; dis2 is the minimum distance between the robotic arm and surrounding personnel during rotational motion; σ2 is the distance sensitivity coefficient during rotational motion; h2 is the end-effector height of the robotic arm during rotation; δ2 is the height adjustment coefficient during rotation; r3 is the motion error risk weight; x ac x represents the actual location; de To set the location.

[0024] Furthermore, ε includes acceleration and jerk penalty. Changes in speed and acceleration during the movement of the robotic arm can cause mechanical vibration and wear. This invention introduces acceleration and jerk.

[0025] Where k a k j These are parameters adjusted based on the rigidity of the robotic arm and the dynamics of the task; For acceleration, Let n1 and n2 be the acceleration and the number of times the acceleration changes, respectively.

[0026] Furthermore, step S5 includes constructing path patterns, similarity matching, and path pattern selection; wherein, it includes obtaining a set of data from historical data, such as the location of the empty basket to be processed with a complete motion trajectory and the predicted empty time, taking the robot arm starting from the stacking point and returning to the stacking point with an empty load as a complete motion trajectory, clustering the number of empty baskets collected according to a complete motion trajectory and their relative positions, constructing several path patterns, performing similarity matching on the data to be processed, and for the matching cases, using the existing motion trajectories in the historical database for planning.

[0027] The present invention also provides a luggage empty basket identification and grabbing device, which includes: an image acquisition and preprocessing module, which is used to acquire images of the luggage conveyor belt and the movement of people in the surrounding area through a camera;

[0028] The luggage basket recognition and position tracking module is used to identify the position of the luggage basket in the image based on the specific shape of the luggage basket, and to track the position of the luggage basket in combination with the running speed of the conveyor belt;

[0029] The module for predicting the movement of people in the surrounding environment is used to predict the movement trajectory and behavior of people in the surrounding environment.

[0030] The robotic arm motion trajectory calculation module is used to obtain an optimal motion trajectory based on the establishment of multi-target trajectory planning and to determine the grasping order of multiple grasping targets.

[0031] The motion trajectory planning algorithm optimization module is used to optimize the motion trajectory planning algorithm for lightweighting based on path pattern recognition.

[0032] In addition, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing program instructions for a method of identifying and grabbing empty luggage baskets. The program instructions for identifying and grabbing empty luggage baskets can be executed by one or more processors to implement the steps of the method of identifying and grabbing empty luggage baskets as described above.

[0033] (III) Beneficial Effects

[0034] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0035] (1) This invention achieves efficient empty baggage basket grabbing, and recovers multiple empty baskets in one process from the origin to the origin. It also predicts the state of the baggage basket in advance based on the passenger's movement trajectory and actions, which makes it easier for the grabbing robot arm to reach the designated position in advance and improves efficiency.

[0036] (2) Based on the improved path planning method, the present invention intelligently plans the grasping motion trajectory and action of the robotic arm, combining the planning of mechanical energy trajectory with time, energy consumption and risk level, and incorporating a penalty factor, ensuring the operational stability and safety of the empty basket recycling robotic arm while considering optimal time and energy consumption.

[0037] (3) The present invention also optimizes the motion trajectory planning algorithm. By matching the constructed path pattern, the existing planned trajectory is selected, which reduces the complexity of path calculation. For path patterns that already exist in historical data, path planning can be made more quickly and easily. Attached Figure Description

[0038] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0039] Figure 1 This is a schematic flowchart of a method for identifying and grabbing empty luggage baskets according to an embodiment of this application. Detailed Implementation

[0040] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0041] The following specific examples illustrate the implementation of this disclosure. Those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. This disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0042] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this disclosure. The drawings only show the components related to this disclosure and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0043] See Figure 1 A method for identifying and grabbing empty luggage baskets includes the following steps:

[0044] S1: Image acquisition and preprocessing, including capturing images of the baggage conveyor belt and the movement of people in the surrounding area via a camera;

[0045] S2: Luggage basket recognition and location tracking;

[0046] The location of the luggage basket in the image is identified based on its specific shape, and the position of the luggage basket is tracked by combining the running speed of the conveyor belt.

[0047] S3: Based on the movement of people in the surrounding environment, predict their movement trajectories and behaviors; including the following steps:

[0048] S31: Individual extraction of surrounding people; Based on the YOLOv5 target detection algorithm, the images captured by the camera are processed to extract pedestrians from the background area of ​​the image;

[0049] Because identification in complex environments like airports is susceptible to interference from the surrounding environment, especially human-shaped billboards, which can lead to misidentification, this invention improves the target detection algorithm to further enhance the recognition of people in the surrounding area. A 1×1 convolutional module is added between the YOLOv5 and CSPBlock recognition models. In the spatial pyramid pooling module, a strategy of parallel max pooling and average pooling is employed, and the results are summed to replace the traditional single pooling operation. This design better preserves feature information at different levels, enhancing the expressive power of the features. Subsequently, through the improved feature pyramid structure, the feature map output by CSPBlock is added to the feature map output by the feature pyramid, effectively avoiding the loss of shallow semantic information and improving the overall performance of the model.

[0050] S32: Motion Trajectory Prediction

[0051] Let the foot position of the i-th person at time t be represented as...

[0052]

[0053] in, Here are the foot position coordinates; the foot position sequence is...

[0054] In areas near the baggage conveyor belt with baggage baskets, it indicates whether the person will pick up the baggage or not. This invention adds pedestrian direction as input to the historical trajectory sequence of people, i.e., learning the distribution p(FP). f |FP obs ,OP obs ), where FP f For the predicted pedestrian movement trajectory, OR obs For pedestrians.

[0055] Optionally, the present invention uses an LSTM network to obtain the trajectory of personnel movement at each future time.

[0056] S33: Behavior prediction, predicting whether people in the vicinity will pick up luggage from the luggage basket; specifically achieved through the following steps:

[0057] Action recognition algorithms based on multi-feature fusion graph neural networks can model and classify the action data of surrounding people by fusing multiple feature information and using graph neural network models. Assume there are N samples and M action categories, with each sample having K features, including joint angles, body posture, and movement speed.

[0058] The graph neural network-based action recognition model achieves data modeling and classification by constructing a dynamic relational topology. This architecture employs dynamic relationship modeling and feature propagation mechanisms between nodes, utilizing the message-passing paradigm of graph structures to capture the spatiotemporal correlation characteristics in action sequences. In the feature extraction stage, hierarchical graph convolutional operations learn local joint motion patterns and global limb collaboration rules respectively, enhancing the expressive power of key action features through adaptive edge weight adjustment. The network deeply integrates node attributes (coordinates, velocity) and edge attributes (joint distance, motion phase), employing a multi-head attention mechanism to achieve multimodal feature interaction. Finally, a fully connected classification layer maps the high-order graph representation to the action label space, completing the end-to-end recognition task. This framework, through hierarchical feature aggregation and context-aware mechanisms of graph structures, fully exploits the topological correlations in action data, demonstrating excellent classification robustness and generalization ability in complex scenarios.

[0059] S34: Predict the time when the corresponding luggage basket will be empty and its location will be moved based on the person's intention to pick up luggage.

[0060] If it is determined that the person intends to pick up luggage, the time when the luggage basket will be empty and its location will be determined in the future based on the speed of their movement and the speed of the conveyor belt.

[0061] S4: Calculate the motion trajectory of the grasping robotic arm based on the prediction results.

[0062] This invention enables the simultaneous retrieval of multiple empty baskets in a single process, starting from and returning to the origin. Based on multi-target trajectory planning, an optimal motion trajectory is obtained, determining the grasping order of multiple targets. The robotic arm rotates according to the position and angle of the next empty basket to be grasped, overlapping and grasping the already grasped baskets with the next basket at the same angle. The process includes the following steps:

[0063] S41: Constructing the motion model of the robotic arm

[0064] This invention establishes a robotic arm motion model based on an improved DH modeling method. The origin of the link coordinate system is set at the junction of the link with the previous link, and the Z-axis of the coordinate system is set as the junction line between the previous link and the next link. The homogeneous transformation matrix between adjacent coordinate systems is obtained according to the improved DH parameter method. The improved DH parameter coordinate system, by setting the origin at the beginning of the link, not only eliminates the ambiguity in parameter interpretation caused by coordinate system drift, but also significantly simplifies the derivation process of the homogeneous transformation matrix between adjacent coordinate systems.

[0065] S42: Establish a multi-objective trajectory planning model

[0066] S421: Model Building

[0067] This invention is based on the improved Diikstra algorithm for trajectory planning. Based on the future time when the luggage basket will be empty and the moving position obtained in step S3, the mechanical energy trajectory planning combines time, energy consumption, and risk level, and incorporates a penalty factor. Under the consideration of optimal time and energy consumption, the stability and safety of the empty basket recovery robot arm are guaranteed.

[0068] S422: Determine the objective function

[0069] L = w1L1 + w2L2 + w3L3 + ε

[0070] L1 is the time objective function; L2 is the energy consumption objective function; L3 is the risk level objective function; ε is the penalty factor; w1, w2, and w3 are the weighting factors of the three respectively;

[0071] in,

[0072] t i The time taken for the i-th sub-path movement, where n is the number of sub-paths contained in the entire movement path;

[0073]

[0074] p iP represents the power of the linear motion of the i-th discretized sub-path; the motion from 1 to n1 is unloaded, and the motion from n1+1 to n is loaded with an empty basket; k represents the number of robotic arm rotations and end effector actions; P represents the power of the linear motion of the i-th discretized sub-path. j This represents the power consumption for the corresponding robotic arm rotation and end effector actions.

[0075] The L3 risk level objective function is determined based on the movement of the robotic arm after it grips an empty basket, the height of the robotic arm during rotation, the distance from surrounding personnel, and motion errors.

[0076]

[0077] r1 is the distance risk weight during linear motion; dis1 is the minimum distance between the robotic arm and surrounding personnel during linear motion; σ1 is the distance sensitivity coefficient during linear motion; h1 is the end-effector height of the robotic arm during linear motion; δ1 is the height adjustment coefficient during linear motion; r2 is the distance risk weight during rotational motion; dis2 is the minimum distance between the robotic arm and surrounding personnel during rotational motion; σ2 is the distance sensitivity coefficient during rotational motion; h2 is the end-effector height of the robotic arm during rotation; δ2 is the height adjustment coefficient during rotation; r3 is the motion error risk weight; x ac x represents the actual location; de To set the location.

[0078] Among them, the risk weight is higher for rotational motion than for linear motion;

[0079] ε includes acceleration and jerk penalty. Changes in speed and acceleration during the movement of the robotic arm can cause mechanical vibration and wear. This invention introduces acceleration and jerk.

[0080] Where k a k j These are parameters adjusted based on the rigidity of the robotic arm and the dynamics of the task; For acceleration, Let n1 and n2 be the acceleration and the number of times the acceleration changes, respectively.

[0081] S423: Perform dynamic trajectory planning;

[0082] Dynamic programming is used to plan the motion trajectory of the robotic arm. Specifically, the motion trajectory and behavior of people around the robotic arm are predicted based on real-time video, and the planned trajectory of the robotic arm is matched. If a collision between the planned trajectory of the robotic arm and the behavior of people around the robotic arm is detected, the motion trajectory of the robotic arm is replanned.

[0083] S5: Optimization of motion trajectory planning algorithm;

[0084] Since the above-mentioned motion trajectory planning algorithm has high time complexity and requires a lot of computing resources, and the environment and personnel movement patterns at the baggage claim area are relatively fixed, this invention optimizes the motion trajectory planning algorithm based on path pattern recognition, specifically including constructing path patterns, similarity matching, and path pattern selection.

[0085] S51: Construct path patterns; Obtain a set of data from historical data, including the location of empty baskets to be processed and the predicted empty time, with each complete motion trajectory as a unit. Consider the complete motion trajectory as the robotic arm starting from the stack and returning to the stack with an empty basket. Cluster the number of empty baskets collected according to a complete motion trajectory and their relative positions to construct R path patterns.

[0086] Optionally, the clustering method includes using random forest for clustering.

[0087] S52: Similarity matching; Match the number of empty baskets and their relative positions in the data to be processed with the R path patterns in the previous step. If the matching degree is greater than the set threshold, the two are considered to belong to the same path pattern.

[0088] S53: For matching scenarios, use existing motion trajectories from the historical database for planning.

[0089] This reduces the complexity of path calculations, allowing for faster and simpler path planning for existing path patterns in historical data.

[0090] In this embodiment, efficient empty baggage basket retrieval is achieved. The baggage basket's state is predicted in advance based on the passenger's movement trajectory and actions, facilitating the robotic arm's early arrival at the designated position and improving efficiency. An improved path planning method intelligently plans the robotic arm's grasping trajectory and actions, incorporating time, energy consumption, and risk level considerations into the mechanical trajectory planning, along with a penalty factor. This ensures the stability and safety of the empty baggage basket retrieval robotic arm while considering optimal time and energy consumption. Furthermore, the motion trajectory planning algorithm has been optimized. By matching constructed path patterns and selecting existing planned trajectories, the complexity of path calculations is reduced, allowing for faster and simpler path planning based on existing path patterns in historical data.

[0091] This invention also proposes a luggage empty basket identification and grasping device, comprising:

[0092] The image acquisition and preprocessing module is used to acquire images of the baggage conveyor belt and the movement of people in the surrounding area via a camera;

[0093] The luggage basket recognition and position tracking module is used to identify the position of the luggage basket in the image based on the specific shape of the luggage basket, and to track the position of the luggage basket in combination with the running speed of the conveyor belt;

[0094] The module for predicting the movement of people in the surrounding environment is used to predict the movement trajectory and behavior of people in the surrounding environment.

[0095] The robotic arm motion trajectory calculation module is used to obtain an optimal motion trajectory based on the establishment of multi-target trajectory planning and to determine the grasping order of multiple grasping targets.

[0096] The motion trajectory planning algorithm optimization module is used to optimize the motion trajectory planning algorithm for lightweighting based on path pattern recognition.

[0097] Furthermore, this embodiment of the invention also proposes a computer-readable storage medium storing program instructions for a method of identifying and grabbing empty luggage baskets. These program instructions can be executed by one or more processors to implement the steps of the method of identifying and grabbing empty luggage baskets as described above.

[0098] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A method for identifying empty containers of baggage, characterized in that, Comprising the following steps: S1: image acquisition and preprocessing, including acquiring luggage conveyor belt images and surrounding personnel movement conditions through a camera; S2: luggage basket identification and position tracking; Identify the position of the luggage basket in the image based on its specific shape, and track the position of the luggage basket in combination with the running speed of the conveyor belt; S3: According to the movement of the surrounding personnel, the movement trajectory and behavior of the surrounding personnel are predicted, and if it is identified that the personnel have the intention to pick up the luggage, the emptying time and moving position of the luggage basket at the future time are obtained according to the action speed combined with the moving speed of the conveyor belt; The image collected by the camera is processed based on the YoloV5 target detection algorithm, and the pedestrians are extracted from the image background area. A 1x1 convolution module is added between the YoloV5 and CSPBlock of the identification model, and maximum pooling and average pooling are used in parallel in the spatial pyramid pooling module, and the results of the two are added; The action recognition model based on graph neural network in the behavior prediction realizes the modeling and classification of action data by constructing a dynamic relationship topology structure; S4: Calculate the movement trajectory of the grabbing manipulator based on the prediction result, including obtaining an optimal movement trajectory based on the establishment of multi-target trajectory planning, and determining the grabbing sequence of multiple grabbing targets; The objective function of the multi-target trajectory planning is as follows: is a time objective function; is an energy consumption objective function; is a risk level objective function; is a penalty factor; , , are weight factors of the three, respectively; the risk level objective function is determined based on the height and distance from the surrounding people and movement failure when the mechanical arm moves after holding the empty basket and when the mechanical arm rotates. is the distance risk weight when moving in a straight line; is the minimum distance between the robot arm and the surrounding people when moving in a straight line; is the distance sensitivity coefficient when moving in a straight line; is the height of the end of the robot arm when moving in a straight line; is the height adjustment coefficient when moving in a straight line; is the distance risk weight when rotating; is the minimum distance between the robot arm and the surrounding people when rotating; is the distance sensitivity coefficient when rotating; is the height of the end of the robot arm when rotating; is the height adjustment coefficient when rotating; is the risk weight of movement failure; is the actual position; is the set position; the includes acceleration and jerk penalties. The changes in speed and acceleration during the movement of the robot arm will cause mechanical vibration and wear. The present application introduces acceleration and jerk. ; wherein , are parameters adjusted according to the rigidity of the robot arm and the dynamics of the task, respectively; is the acceleration, is the jerk, and are the number of acceleration and jerk changes, respectively; S5: Motion trajectory planning algorithm optimization.

2. The empty-bag identification method according to claim 1, wherein The motion trajectory prediction of the surrounding personnel includes prediction based on the historical trajectory sequence of the pedestrian and the direction of the pedestrian.

3. The luggage empty basket identification method according to claim 1, characterized by, The step S5 includes constructing path patterns, similarity matching, and path pattern selection; wherein, a set of data of the position of the empty basket and the predicted emptying time in the historical data is obtained, a complete movement trajectory is formed from the start of the emptying of the manipulator from the pile to the recycling of the empty basket to the pile, the number of empty baskets collected according to a complete movement trajectory and the relative position are clustered, and a plurality of path patterns are constructed. Similarity matching is performed on the to-be-processed data, and the existing movement trajectory in the historical database is used for planning for the matched case.

4. An apparatus based on the luggage empty basket identification method according to any one of claims 1-3, comprising: An image acquisition and preprocessing module for acquiring luggage conveyor belt images and surrounding personnel movement conditions through a camera; A luggage basket identification and position tracking module for identifying the position of the luggage basket in the image based on its specific shape, and tracking the position of the luggage basket in combination with the running speed of the conveyor belt; A surrounding personnel movement condition prediction module for predicting the movement trajectory and behavior of the surrounding personnel; A grabbing manipulator movement trajectory calculation module for obtaining an optimal movement trajectory based on the establishment of multi-target trajectory planning, and determining the grabbing sequence of multiple grabbing targets; A motion trajectory planning algorithm optimization module for lightweight optimization of the motion trajectory planning algorithm based on path pattern recognition.

5. A computer readable storage medium, characterized in that, The computer readable storage medium stores program instructions of the luggage empty basket recognition method, and the program instructions of the luggage empty basket recognition method can be executed by one or more processors to implement the steps of the luggage empty basket recognition method in any one of claims 1-3.

Images