Automatic stringing method and system based on double-robot arm cooperative control and force perception feedback

Abstract

This invention provides an automated skewer-making method and system based on dual-robotic arm collaborative control and force sensing feedback, comprising: a first robotic arm grasping bamboo skewers and a second robotic arm grasping raw materials; performing basic positioning of the first robotic arm grasping bamboo skewers and the second robotic arm grasping raw materials to meet preset requirements, and identifying the position information of the bamboo skewers; calculating the alignment points of the first robotic arm and the second robotic arm for skewer-making based on the identified position information of the bamboo skewers; performing skewer-making with the robotic arms based on the alignment points of the first robotic arm and the second robotic arm, and determining whether skewer-making is successful through force sensing to achieve closed-loop stable skewer-making; controlling the robotic arm holding the skewered bamboo skewers to reach a preset position, and resetting the force sensing of the robotic arm; when the force sensing of the robotic arm changes, controlling the gripper of the robotic arm to release a certain width; when the human hand is removed, controlling the robotic arm to return to the preset position.

Description

Technical Field

[0001] This invention relates to the fields of intelligent manufacturing and food processing automation technology, specifically to an automated threading method and system based on dual robotic arm collaborative control and force sensing feedback. Background Technology

[0002] In food skewer processing, manual skewering is inefficient, costly, and produces inconsistent results. Existing mechanical skewering equipment often relies on fixed trajectories, making it difficult to handle changes in the posture of the skewers and ingredients. Inaccurate positioning and alignment deviations frequently lead to skewering failures or ingredient damage, failing to meet the demands for high-precision and stable production. Therefore, there is an urgent need for an automated skewering technology that can achieve precise positioning and intelligent alignment. Furthermore, during the entire food retrieval process, the robotic arm's gripper release action relies entirely on user input: when a user retrieves food, they must send a release signal to the robotic arm by triggering a preset button, touch sensor, or voice command. Only after receiving this signal will the robotic arm release the gripper. This control logic, dependent on user input, introduces significant operational redundancy in the food retrieval process. Users must perform additional input operations beyond the food retrieval action, increasing the complexity of the process and potentially causing interruptions or delayed gripper release due to input delays or errors, thus impacting efficiency and user experience.

[0003] Patent examination CN112167684A (application number: 202010926884.9) discloses a skewering device, particularly an automatic skewering device for processing block food, including a support platform, a support frame, a material frame, a first feeding mechanism, a second feeding mechanism, and a skewering mechanism. The support platform is rectangular, and four support frames are fixedly installed around the lower part of the support platform. The material frame is fixedly installed on the upper left front side of the support platform, and the first feeding mechanism is fixedly installed on the upper right side of the support platform. Summary of the Invention

[0004] In view of the deficiencies in the existing technology, the purpose of this invention is to provide an automated threading method and system based on dual robotic arm collaborative control and force sensing feedback.

[0005] An automated threading method based on dual-robotic arm collaborative control and force sensing feedback, provided by the present invention, includes:

[0006] Step S1: The first robotic arm grabs the bamboo skewers, and the second robotic arm grabs the raw materials;

[0007] Step S2: Perform basic positioning of the first robotic arm for grasping bamboo skewers and the second robotic arm for grasping raw materials to meet preset requirements; under the basic positioning of the first robotic arm for grasping bamboo skewers and the second robotic arm for grasping raw materials to meet preset requirements, identify the position information of bamboo skewers; wherein, the spatial position of the tip of the bamboo skewer and the angle information between the target bamboo skewer and the horizontal line;

[0008] Step S3: Calculate the alignment points of the first and second robotic arms based on the identified bamboo skewer position information;

[0009] Step S4: Based on the alignment of the first and second robotic arms at the threading points, the robotic arms perform threading, and determine whether the threading is successful through force control sensing to achieve closed-loop stable threading;

[0010] Step S5: Control the robotic arm holding the skewered bamboo sticks to reach the preset position, and clear the force sensing of the robotic arm.

[0011] Step S6: When the force sensing of the robotic arm changes, control the gripper of the robotic arm to loosen by a certain width;

[0012] Step S7: When the human hand is removed, control the robotic arm to return to the preset position.

[0013] Preferably, step S1 includes:

[0014] Step S1.1: Obtain the fixed position information of the target bamboo stick, control the gripper of the first robotic arm to grab the target bamboo stick based on the fixed position information of the target bamboo stick, and determine whether the target bamboo stick has been successfully grabbed by force control sensing of the gripper and real-time feedback of the gripping width.

[0015] Step S1.2: Identify the center location of the target raw material and the distance between the target raw material and adjacent raw material locations using AI;

[0016] Step S1.3: Use the identified center position of the target material as the center position for the gripper of the second robotic arm; obtain the gripping posture of the second robotic arm gripper based on the distance between the original position of the target and the adjacent material positions, so that the second robotic arm can avoid interference between the gripper and the material to the greatest extent.

[0017] Step S1.4: Based on the gripping posture of the second robotic arm gripper, and combined with the width of the target material, obtain the opening width of the second robotic arm gripper;

[0018] Step S1.5: Grab the target raw material based on the gripping posture and opening width of the second robotic arm gripper.

[0019] Preferably, step S2 includes:

[0020] Step S2.1: Move the first robotic arm that grabs bamboo skewers and the second robotic arm that grabs raw materials to a preset range directly above the raw material cooking pot, and make the center point of the raw material located on the central axis of the bamboo skewers, thus completing the basic positioning of the first robotic arm that grabs bamboo skewers and the second robotic arm that grabs raw materials to meet the preset requirements.

[0021] Step S2.2: Use AI to identify the spatial position of the tip of the bamboo stick that the first robotic arm grasps and the angle between the bamboo stick and the horizontal line.

[0022] Preferably, step S3 includes:

[0023] Step S3.1: In the Cartesian coordinate system of the first robotic arm, the spatial position of the tip of the bamboo skewer is obtained by AI recognition, and the starting point of stringing is obtained based on the spatial position of the tip of the bamboo skewer.

[0024] Step S3.2: Calculate the skewering end point by recognizing the tilt angle of the bamboo skewers and the order of the meatballs to be skewered using AI.

[0025] Preferably, step S4 includes:

[0026] The first robotic arm that is currently grasping bamboo skewers and the second robotic arm that is grasping raw materials move relative to each other along the central axis of the bamboo skewers based on the starting point and ending point of skewering to implement the skewering of the robotic arms;

[0027] During the process from the start point to the end point of the stringing process, the second robotic arm that grabs the raw materials obtains the sensing force through the force control sensor of the first robotic arm. When the sensing force is always less than or equal to the preset value, it is determined that the current stringing has failed; when the sensing force is greater than the preset value, it is determined that the current stringing has been completed.

[0028] Preferably, step S5 includes: obtaining the force of each mechanical joint based on the torque sensor integrated in each joint of the robotic arm; and obtaining the end-effector force by back-calculating the obtained force of each mechanical joint through a dynamic model.

[0029] Preferably, step S7 includes: setting up a detection device in a preset food collection area to detect whether a hand enters; when a hand is detected, triggering an image acquisition device; acquiring image information of the food collection area in real time through the image acquisition device; performing hand recognition based on the image information; when no hand information is detected in the image information, and the robotic arm gripper releases a certain width of preset time, it is considered that the hand has been removed, and the robotic arm is controlled to return to the preset position.

[0030] An automated threading system based on dual-robotic arm collaborative control and force sensing feedback, provided by the present invention, includes:

[0031] Module M1: The first robotic arm grabs bamboo skewers, and the second robotic arm grabs raw materials;

[0032] Module M2: Performs basic positioning of the first robotic arm for grasping bamboo skewers and the second robotic arm for grasping raw materials to meet preset requirements; under the basic positioning of the first robotic arm for grasping bamboo skewers and the second robotic arm for grasping raw materials to meet preset requirements, identifies the position information of bamboo skewers; wherein, the spatial position of the tip of the bamboo skewer and the angle information between the target bamboo skewer and the horizontal line;

[0033] Module M3: Calculates the alignment points of the first and second robotic arms based on the identified bamboo skewer position information;

[0034] Module M4: Based on the alignment of the first and second robotic arms at the threading points, the robotic arms perform threading and determine whether the threading is successful through force control sensing, thus achieving closed-loop stable threading.

[0035] Module M5: Controls the robotic arm holding the skewered bamboo sticks to reach the preset position and resets the force sensing of the robotic arm to zero.

[0036] Module M6: When the force sensing of the robotic arm changes, it controls the gripper of the robotic arm to loosen by a certain width;

[0037] Module M7: When the human hand is removed, the robotic arm is controlled to return to the preset position.

[0038] Preferably, the module M1 includes:

[0039] Module M1.1: Obtain the fixed position information of the target bamboo stick, control the gripper of the first robotic arm to grab the target bamboo stick based on the fixed position information of the target bamboo stick, and determine whether the target bamboo stick has been successfully grabbed by force control sensing of the gripper and real-time feedback of the gripping width.

[0040] Module M1.2: Uses AI to identify the center location of the target raw material and the distance between the target raw material and adjacent raw material locations;

[0041] Module M1.3: The center position of the identified target material is used as the center position for the gripper of the second robotic arm; the gripping posture of the second robotic arm gripper is obtained based on the distance between the original position of the target and the adjacent material positions, so that the second robotic arm can avoid interference between the gripper and the material to the greatest extent.

[0042] Module M1.4: Based on the gripping posture of the second robotic arm gripper, and combined with the width of the target material, obtain the opening width of the second robotic arm gripper;

[0043] Module M1.5: Based on the gripping posture and opening width of the second robotic arm's gripper, the target raw material is gripped;

[0044] The module M2 includes:

[0045] Module M2.1: Move the first robotic arm that grabs bamboo skewers and the second robotic arm that grabs raw materials to a preset range directly above the raw material cooking pot, and make the center point of the raw material located on the central axis of the bamboo skewers, thus completing the basic positioning of the first robotic arm that grabs bamboo skewers and the second robotic arm that grabs raw materials to meet the preset requirements.

[0046] Module M2.2: Uses AI to identify the spatial position of the tip of the bamboo skewer grasped by the first robotic arm and the angle between the bamboo skewer and the horizontal line.

[0047] Preferably, the module M3 includes:

[0048] Module M3.1: In the Cartesian coordinate system of the first robotic arm, the spatial position of the tip of the bamboo stick is obtained through AI recognition, and the starting point of stringing is obtained based on the spatial position of the tip of the bamboo stick;

[0049] Module M3.2: Calculates the end point of skewering by recognizing the tilt angle of the bamboo skewers and the order of the meatballs to be skewered using AI;

[0050] The module M4 includes:

[0051] The first robotic arm that is currently grasping bamboo skewers and the second robotic arm that is grasping raw materials move relative to each other along the central axis of the bamboo skewers based on the starting point and ending point of skewering to implement the skewering of the robotic arms;

[0052] During the process from the start point to the end point of stringing, the second robotic arm that grabs the raw materials obtains the sensing force through the force control sensor of the first robotic arm. When the sensing force is always less than or equal to the preset value, it is determined that the current stringing has failed; when the sensing force is greater than the preset value, it is determined that the current stringing has been completed.

[0053] The module M5 includes: obtaining the force of each mechanical joint based on the torque sensor integrated in each joint of the robotic arm; and obtaining the end force by back-calculating the force of each mechanical joint through a dynamic model.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] 1. This invention uses AI to identify the spatial position, tilt angle, and raw material information of the bamboo skewer tip, calculates the alignment point for threading, and then uses force control sensing to achieve closed-loop control. This effectively solves the problems of inaccurate positioning and alignment deviation in traditional mechanical threading, significantly reduces the threading failure rate and raw material damage rate, and ensures the consistency of product quality.

[0056] 2. This invention uses robotic arms for automated operation, replacing traditional manual operation. This not only reduces the intensity of manual labor, but also enables continuous and stable batch production, greatly improving the production efficiency of food skewering and making it suitable for large-scale processing needs.

[0057] 3. This invention uses AI to accurately identify information such as the center position of the raw material and the distance between adjacent raw materials, providing the robotic arm with a suitable gripping posture and minimizing interference. At the same time, force control sensing can determine in real time whether the gripping and skewering are successful, enabling the robotic arm to operate stably when faced with changes in the posture of raw materials and bamboo skewers, and adapting to the skewering needs of different types and specifications of raw materials.

[0058] 4. This invention achieves a safe and stable human-computer interaction process for food retrieval by employing precise force control sensing of the robotic arm and real-time width feedback of the end gripper.

[0059] 5. This invention utilizes the force sensing capability of the robotic arm, eliminating the need for users to input additional commands. The release of the gripper is triggered simply by the force change generated when a person picks up a bamboo skewer, thus eliminating the need for users to actively operate the command input device. This simplifies the food retrieval process, making food retrieval more natural and efficient for users, and reducing the inconvenience caused by redundant operations.

[0060] 6. The robotic arm integrates torque sensors in each joint, and uses a dynamic model to back-calculate the force and torque of the end effector TCP. After reaching the food retrieval position, the force is reset to zero. It can accurately sense changes in force in any direction, ensuring that the gripper releases in time when the hand comes into contact with it, avoiding squeezing or scratching the hand due to continuous gripping by the gripper, and reducing safety risks during the interaction process.

[0061] 7. Combining the detection device and image acquisition device of the preset food pick-up area, the image acquisition is triggered when a hand is detected to enter. The image recognition technology monitors in real time whether the hand has been removed. At the same time, the preset time condition after the gripper releases a certain width ensures the accuracy of the hand removal judgment. This avoids the problem of the robotic arm removing too early or too late due to relying solely on a fixed time delay. It not only prevents interference with the user's food pick-up, but also improves the working efficiency of the robotic arm. Attached Figure Description

[0062] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0063] Figure 1 This is a flowchart of an automated threading method based on dual-arm collaborative control and force sensing feedback.

[0064] Figure 2 This is a flowchart of a precise threading method for a robotic arm based on AI recognition and force control perception.

[0065] Figure 3 This is a flowchart of the interactive control method for a robotic arm to pick up food.

[0066] Figure 4 This is a flowchart of an automatic threading control method for a robotic arm based on machine vision and real-time communication. Detailed Implementation

[0067] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0068] Example 1

[0069] An automated threading method based on dual-robotic arm collaborative control and force sensing feedback is provided by the present invention, such as... Figure 1 As shown, it includes:

[0070] Step S1: The first robotic arm grabs the bamboo skewers, and the second robotic arm grabs the raw materials;

[0071] Step S2: Perform basic positioning of the first robotic arm for grasping bamboo skewers and the second robotic arm for grasping raw materials to meet preset requirements; under the basic positioning of the first robotic arm for grasping bamboo skewers and the second robotic arm for grasping raw materials to meet preset requirements, identify the position information of bamboo skewers; wherein, the spatial position of the tip of the bamboo skewer and the angle information between the target bamboo skewer and the horizontal line;

[0072] Step S3: Calculate the alignment points of the first and second robotic arms based on the identified bamboo skewer position information;

[0073] Step S4: Based on the alignment of the first and second robotic arms at the threading points, the robotic arms perform threading, and determine whether the threading is successful through force control sensing to achieve closed-loop stable threading;

[0074] Step S5: Control the robotic arm holding the skewered bamboo sticks to reach the preset position, and clear the force sensing of the robotic arm.

[0075] Step S6: When the force sensing of the robotic arm changes, control the gripper of the robotic arm to loosen by a certain width;

[0076] Step S7: When the human hand is removed, control the robotic arm to return to the preset position.

[0077] Specifically, step S1 includes:

[0078] Step S1.1: Obtain the fixed position information of the target bamboo stick, control the gripper of the first robotic arm to grab the target bamboo stick based on the fixed position information of the target bamboo stick, and determine whether the target bamboo stick has been successfully grabbed by force control sensing of the gripper and real-time feedback of the gripping width.

[0079] Step S1.2: Identify the center location of the target raw material and the distance between the target raw material and adjacent raw material locations using AI;

[0080] Step S1.3: Use the identified center position of the target material as the center position for the gripper of the second robotic arm; obtain the gripping posture of the second robotic arm gripper based on the distance between the original position of the target and the adjacent material positions, so that the second robotic arm can avoid interference between the gripper and the material to the greatest extent.

[0081] Step S1.4: Based on the gripping posture of the second robotic arm gripper, and combined with the width of the target material, obtain the opening width of the second robotic arm gripper;

[0082] Step S1.5: Grab the target raw material based on the gripping posture and opening width of the second robotic arm gripper.

[0083] Specifically, step S2 includes:

[0084] Step S2.1: Move the first robotic arm that grabs bamboo skewers and the second robotic arm that grabs raw materials to a preset range directly above the raw material cooking pot, and make the center point of the raw material located on the central axis of the bamboo skewers, thus completing the basic positioning of the first robotic arm that grabs bamboo skewers and the second robotic arm that grabs raw materials to meet the preset requirements.

[0085] Step S2.2: Use AI to identify the spatial position of the tip of the bamboo stick that the first robotic arm grasps and the angle between the bamboo stick and the horizontal line.

[0086] Specifically, step S3 includes:

[0087] Step S3.1: In the Cartesian coordinate system of the first robotic arm, the spatial position of the tip of the bamboo skewer is obtained by AI recognition, and the starting point of stringing is obtained based on the spatial position of the tip of the bamboo skewer.

[0088] Step S3.2: Calculate the skewering end point by recognizing the tilt angle of the bamboo skewers and the order of the meatballs to be skewered using AI.

[0089] Specifically, step S4 includes:

[0090] The first robotic arm that is currently grasping bamboo skewers and the second robotic arm that is grasping raw materials move relative to each other along the central axis of the bamboo skewers based on the starting point and ending point of skewering to implement the skewering of the robotic arms;

[0091] During the process from the start point to the end point of the stringing process, the second robotic arm that grabs the raw materials obtains the sensing force through the force control sensor of the first robotic arm. When the sensing force is always less than or equal to the preset value, it is determined that the current stringing has failed; when the sensing force is greater than the preset value, it is determined that the current stringing has been completed.

[0092] Specifically, step S5 includes: obtaining the force of each mechanical joint based on the torque sensor integrated in each joint of the robotic arm; and obtaining the end force by back-calculating the force of each mechanical joint through a dynamic model.

[0093] Specifically, step S7 includes: setting up a detection device in a preset food collection area to detect whether a hand enters; when a hand is detected, triggering an image acquisition device; acquiring image information of the food collection area in real time through the image acquisition device; performing hand recognition based on the image information; when no hand information is detected in the image information, and the robotic arm gripper releases a certain width of preset time, it is considered that the hand has been removed, and the robotic arm is controlled to return to the preset position.

[0094] The present invention also provides an automated threading system based on dual-arm collaborative control and force sensing feedback. The automated threading system based on dual-arm collaborative control and force sensing feedback can be implemented by executing the process steps of the automated threading method based on dual-arm collaborative control and force sensing feedback. That is, those skilled in the art can understand the automated threading method based on dual-arm collaborative control and force sensing feedback as a preferred embodiment of the automated threading system based on dual-arm collaborative control and force sensing feedback.

[0095] Example 2

[0096] Example 2 is a preferred example of Example 1.

[0097] The equipment automatically heats the meatballs in the cooking pot. Once the temperature reaches a certain degree Celsius and is maintained for a period of time, the equipment determines that cooking is complete. Heating stops, and the status of the ordering page on the client's mobile app is updated simultaneously.

[0098] Customers access the ordering page on the app, where the equipment displays the current inventory of each product in the pot. Customers select and place their order, allowing them to customize the type, quantity, and skewering order of the meatballs. After ordering, the equipment begins skewering the meatballs according to the customer's request, using an automated skewering method based on dual robotic arm collaborative control and force sensing feedback. At this point, the ordering app displays a loading animation and a progress bar showing the equipment's movement.

[0099] Based on the order and quantity of the food ordered, as well as the type of meatballs ordered, the robot fulfills the customer's order. After the meatballs are skewered, the robotic arm delivers the meatballs to the customer; at this time, the app page displays that the meatballs are ready to be picked up.

[0100] The user removes the finished product from the robotic arm. Based on the force control recognition, the robotic arm and other devices return to their original positions after the item has been removed. At this time, the app updates the status synchronously, the food order process is completed, and the user returns to the ordering page.

[0101] Example 3

[0102] Example 3 is a preferred example of Example 1.

[0103] According to the present invention, a precise threading method for a robotic arm based on AI recognition and force control perception is provided, such as... Figure 2 As shown, the process includes: achieving basic positioning based on gripper grasping, and then using AI to identify the characteristics of raw materials and bamboo skewers, performing complex algorithm calculations to simulate human hand alignment and stringing.

[0104] The precise threading method for robotic arms based on AI recognition and force control perception includes:

[0105] Step 201: The left robotic arm grabs the bamboo skewers, and the right robotic arm grabs the raw materials;

[0106] Specifically, step 201 includes:

[0107] Step 2011: Obtain the fixed position information of the target bamboo stick, and control the gripper of the left robotic arm to grasp the target bamboo stick based on the fixed position information of the target bamboo stick. In this embodiment, the left robotic arm can grasp the bamboo stick based on its fixed position. Then, determine whether the target bamboo stick has been successfully grasped by the force control sensing of the gripper and the real-time feedback of the gripping width.

[0108] Step 2012: Identify the center location of the target raw material and the distance between the target raw material and adjacent raw material locations using AI;

[0109] Step 2013: Use the identified center position of the target material as the center position for gripping by the right robotic arm's gripper; obtain the gripping posture of the right robotic arm's gripper based on the distance between the original position of the target and the adjacent material positions, so that the right robotic arm can avoid interference between the gripper and the material to the greatest extent possible.

[0110] Step 2014: Based on the gripping posture of the right robotic arm gripper, and combined with the width of the target material, obtain the opening width of the right robotic arm gripper to achieve precise gripping and avoidance of interference with the material.

[0111] Step 2015: Grasp the target raw material based on the gripping posture and opening width of the right robotic arm's gripper.

[0112] Step 202: Perform basic positioning of the left robotic arm for grasping bamboo skewers and the right robotic arm for grasping raw materials to meet preset requirements; under the basic positioning of the left robotic arm for grasping bamboo skewers and the right robotic arm for grasping raw materials to meet preset requirements, identify the position information of bamboo skewers; wherein, the spatial position of the tip of the bamboo skewer and the angle information between the target bamboo skewer and the horizontal line.

[0113] Specifically, step 202 includes:

[0114] Step 2021: Move the left robotic arm that grabs the bamboo skewer and the right robotic arm that grabs the raw material to the preset range directly above the raw material cooking pot, and make the center point of the raw material located on the central axis of the bamboo skewer, thus completing the basic positioning of the left robotic arm that grabs the bamboo skewer and the right robotic arm that grabs the raw material to meet the preset requirements.

[0115] Step 2022: Use AI to identify the spatial position of the tip of the bamboo skewer grasped by the left robotic arm and the angle between the bamboo skewer and the horizontal line.

[0116] Step 203: Calculate the alignment points for the left and right robotic arms based on the identified bamboo skewer position information;

[0117] Specifically, step 203 includes:

[0118] Step 2031: Under the Cartesian coordinate system of the left robotic arm, the spatial position of the tip of the bamboo skewer is obtained through AI recognition, and the starting point of stringing is obtained based on the spatial position of the tip of the bamboo skewer.

[0119] Step 2032: Calculate the skewering end point by recognizing the tilt angle of the bamboo skewers and the order of the meatballs to be skewered using AI.

[0120] Step 204: Based on the alignment of the left and right robotic arms at the threading points, the robotic arms perform threading, and determine whether the threading is successful through force control sensing to achieve closed-loop stable threading.

[0121] Specifically, step 204 includes: controlling the left robotic arm currently gripping the bamboo skewer and the right robotic arm gripping the raw material to move relative to each other along the central axis of the bamboo skewer based on the starting point and ending point of the stringing process to implement the skewering process.

[0122] During the process from the start point to the end point of the stringing process, the second robotic arm that grabs the raw materials obtains the sensing force through the force control sensor of the first robotic arm. When the sensing force is always less than or equal to the preset value, it is determined that the current stringing has failed; when the sensing force is greater than the preset value, it is determined that the current stringing has been completed.

[0123] The present invention also provides a precise threading system for a robotic arm based on AI recognition and force control perception. The precise threading system for a robotic arm based on AI recognition and force control perception can be implemented by executing the process steps of the precise threading method for a robotic arm based on AI recognition and force control perception. That is, those skilled in the art can understand the precise threading method for a robotic arm based on AI recognition and force control perception as a preferred embodiment of the precise threading system for a robotic arm based on AI recognition and force control perception.

[0124] Example 4

[0125] Example 4 is a preferred example of Example 1.

[0126] According to the present invention, a robotic arm food retrieval interactive control method is provided, such as... Figure 3 As shown, it includes: obtaining force in any direction based on the force sensing of the robotic arm, obtaining force changes based on the sensed force in any direction, and controlling the gripper of the robotic arm to release based on the force changes, thereby realizing interactive control of the robotic arm to pick up food.

[0127] Specifically, the method of acquiring force in any direction based on the force sensing of the robotic arm, acquiring force changes based on the sensed force in any direction, and controlling the release of the gripper of the robotic arm based on the force changes to achieve interactive control of the robotic arm for food retrieval includes:

[0128] Step 301: Control the robotic arm holding the skewered bamboo sticks to reach the preset position and clear the force sensing of the robotic arm.

[0129] Step 302: When the force sensing of the robotic arm changes, control the gripper of the robotic arm to loosen by a certain width;

[0130] Step 303: When the human hand is removed, control the robotic arm to return to the preset position.

[0131] Specifically, step 301 includes: obtaining the force of each mechanical joint based on the torque sensor integrated in each joint of the robotic arm; and obtaining the end force by back-calculating the force of the end effector based on the obtained force of each mechanical joint through a dynamic model.

[0132] Specifically, step 303 includes: setting up a detection device in a preset food collection area to detect whether a hand enters; when a hand is detected, triggering an image acquisition device; acquiring image information of the food collection area in real time through the image acquisition device; performing hand recognition based on the image information; when no hand information is detected in the image information, and the robotic arm gripper releases a certain width of preset time, it is considered that the hand has been removed, and the robotic arm is controlled to return to the preset position.

[0133] The present invention also provides a robotic arm food retrieval interaction control system, which can be implemented by executing the process steps of the robotic arm food retrieval interaction control method. That is, those skilled in the art can understand the robotic arm food retrieval interaction control method as a preferred embodiment of the robotic arm food retrieval interaction control system.

[0134] Example 5

[0135] Example 5 is a preferred example of Example 1.

[0136] The present invention provides an automatic threading control method for a robotic arm based on machine vision and real-time communication, such as... Figure 4 As shown, it includes:

[0137] Step 401: The overall scheduling module controls the image acquisition module to acquire food images and bamboo skewer images, and then the recognition module is used to identify the target food information and target bamboo skewer information;

[0138] Step 402: Based on the real-time communication between the overall scheduling module and the robotic arm, dynamically acquire the motion state of the robotic arm;

[0139] Step 403: Based on the identified target ingredient information and target bamboo skewer information, as well as the dynamically acquired motion state of the robotic arm, control the robotic arm to perform preset actions according to the preset execution process, thereby realizing automatic skewering by the robotic arm.

[0140] Specifically, during the real-time communication between the overall scheduling module and the robotic arm, the hardware information of the robotic arm is dynamically acquired. Based on the dynamically acquired hardware information, the status of the robotic arm is monitored. When the status of the robotic arm is abnormal, an abnormal warning is issued. The hardware information includes whether there is an emergency stop and whether there is an error message.

[0141] The real-time communication between the overall scheduling module and the robotic arm includes: the overall scheduling module communicating with the robotic arm in real time via the ModbusTCP protocol.

[0142] Specifically, step 401 includes:

[0143] Step 4011: When the main scheduling module receives the automatic stringing instruction, it controls the filter lifting module to rise until the liquid level sensor module determines that the filter lifting module is above the liquid level.

[0144] Step 4012: Control the image acquisition module through the overall scheduling module to acquire the food images and bamboo skewer images in the target area of ​​the current filter lifting module;

[0145] Step 4013: Based on the food image and bamboo skewer image, the AI ​​algorithm in the recognition module identifies the target food information and target bamboo skewer information; the target food includes: the food with the highest similarity to the food requested by the user; the target food information includes: the target food coordinates, the length, width, and angle of the target food; the target bamboo skewer information includes: the bamboo skewer at the target location within the target area; the target bamboo skewer information includes: the target bamboo skewer coordinates, length, and angle information. Based on the target food information and target bamboo skewer information, the first and second robotic arms in the robotic arm are controlled to grip the target food and target bamboo skewer respectively.

[0146] Specifically, step 4011 includes: the main scheduling module establishing communication with the filter lifting module and the liquid level sensor module through the USB to RS485 module; controlling the filter lifting module to rise and fall through the main scheduling module, and determining whether the filter lifting module is above or below the liquid level through the liquid level sensor module; and ensuring the stability and safety of the equipment operation through the liquid level sensor module.

[0147] Specifically, the method further includes: storing basic information of ingredients and bamboo skewers in a Redis database; updating the basic information of ingredients and bamboo skewers in the Redis database when the robotic arm automatically completes the skewering; storing basic information of meatballs in real time and accurately controlling inventory deduction, effectively improving data processing efficiency and accuracy;

[0148] The basic information about the ingredients includes: the remaining quantity of different ingredients, the location information of the remaining bamboo skewers, and the remaining quantity of bamboo skewers.

[0149] The remaining quantity of different ingredients is displayed through the display module.

[0150] Specifically, the method further includes: controlling the cooking equipment through the overall scheduling module, obtaining the status information of the cooking equipment in real time, and pushing it to the client through the WebSocket communication protocol.

[0151] In this embodiment, during each cycle of the threading process, the overall control module communicates in real time with the robotic arm, image acquisition module, and hardware devices to precisely schedule the execution of each module at each key node, ensuring that each link performs its own function, does not interfere with each other, and operates in a coordinated and efficient manner.

[0152] Specifically, step 403 includes:

[0153] Step 4031: The left robotic arm grabs the bamboo skewer, and the right robotic arm grabs the raw material;

[0154] Specifically, step 4031 includes:

[0155] Step 40311: Obtain the fixed position information of the target bamboo stick, and control the gripper of the left robotic arm to grasp the target bamboo stick based on the fixed position information of the target bamboo stick. In this embodiment, the left robotic arm can grasp the bamboo stick based on its fixed position. Then, determine whether the target bamboo stick has been successfully grasped by the force control sensing of the gripper and the real-time feedback of the gripping width.

[0156] Step 40312: Identify the center location of the target raw material and the distance between the target raw material and adjacent raw material locations using AI;

[0157] Step 40313: Use the identified center position of the target material as the center position for gripping by the right robotic arm's gripper; obtain the gripping posture of the right robotic arm's gripper based on the distance between the original position of the target and the adjacent material positions, so that the right robotic arm can avoid interference between the gripper and the material to the greatest extent.

[0158] Step 40314: Based on the gripping posture of the right robotic arm gripper, and combined with the width of the target material, obtain the opening width of the right robotic arm gripper to achieve precise gripping and avoidance of interference with the material.

[0159] Step 40315: Grasp the target raw material based on the gripping posture and opening width of the right robotic arm's gripper.

[0160] Step 4032: Perform basic positioning of the left robotic arm for grasping bamboo skewers and the right robotic arm for grasping raw materials to meet preset requirements; under the basic positioning of the left robotic arm for grasping bamboo skewers and the right robotic arm for grasping raw materials to meet preset requirements, identify the position information of bamboo skewers; wherein, the spatial position of the tip of the bamboo skewer and the angle information between the target bamboo skewer and the horizontal line.

[0161] Specifically, step 4032 includes:

[0162] Step 40321: Move the left robotic arm that grabs the bamboo skewer and the right robotic arm that grabs the raw material to a preset range directly above the raw material cooking pot, and make the center point of the raw material located on the central axis of the bamboo skewer, thus completing the basic positioning of the left robotic arm that grabs the bamboo skewer and the right robotic arm that grabs the raw material to meet the preset requirements.

[0163] Step 40322: Use AI to identify the spatial position of the tip of the bamboo skewer grasped by the left robotic arm and the angle between the bamboo skewer and the horizontal line.

[0164] Step 4033: Calculate the alignment points for the left and right robotic arms based on the identified bamboo skewer position information;

[0165] Specifically, step 4033 includes:

[0166] Step 40331: Under the Cartesian coordinate system of the left robotic arm, the spatial position of the tip of the bamboo skewer is obtained through AI recognition, and the starting point of stringing is obtained based on the spatial position of the tip of the bamboo skewer.

[0167] Step 40332: Calculate the skewering end point by recognizing the tilt angle of the bamboo skewers and the order of the meatballs to be skewered using AI.

[0168] Step 4034: Based on the alignment of the left and right robotic arms at the threading points, the robotic arms perform threading, and determine whether the threading is successful through force control sensing to achieve closed-loop stable threading.

[0169] Specifically, step 4034 includes: controlling the left robotic arm currently gripping the bamboo skewer and the right robotic arm gripping the raw material to move relative to each other along the central axis of the bamboo skewer based on the starting point and ending point of the stringing process to implement the stringing process.

[0170] During the process from the start point to the end point of the stringing process, the second robotic arm that grabs the raw materials obtains the sensing force through the force control sensor of the first robotic arm. When the sensing force is always less than or equal to the preset value, it is determined that the current stringing has failed; when the sensing force is greater than the preset value, it is determined that the current stringing has been completed.

[0171] The present invention also provides an automatic threading control system for a robotic arm based on machine vision and real-time communication. The automatic threading control system for a robotic arm based on machine vision and real-time communication can be implemented by executing the process steps of the automatic threading control method for a robotic arm based on machine vision and real-time communication. That is, those skilled in the art can understand the automatic threading control method for a robotic arm based on machine vision and real-time communication as a preferred embodiment of the automatic threading control system for a robotic arm based on machine vision and real-time communication.

[0172] Those skilled in the art will understand that, in addition to implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.

[0173] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. An automated threading method based on dual-robotic arm collaborative control and force sensing feedback, characterized in that, include: Step S1: The first robotic arm grabs the bamboo skewers, and the second robotic arm grabs the raw materials; Step S2: Perform basic positioning of the first robotic arm for grasping bamboo skewers and the second robotic arm for grasping raw materials to meet preset requirements; under the basic positioning of the first robotic arm for grasping bamboo skewers and the second robotic arm for grasping raw materials to meet preset requirements, identify the position information of bamboo skewers; wherein, the spatial position of the tip of the bamboo skewer and the angle information between the target bamboo skewer and the horizontal line; Step S3: Calculate the alignment points of the first and second robotic arms based on the identified bamboo skewer position information; Step S4: Based on the alignment of the first and second robotic arms at the threading points, the robotic arms perform threading, and determine whether the threading is successful through force control sensing to achieve closed-loop stable threading; Step S5: Control the robotic arm holding the skewered bamboo sticks to reach the preset position, and clear the force sensing of the robotic arm. Step S6: When the force sensing of the robotic arm changes, control the gripper of the robotic arm to loosen by a certain width; Step S7: When the human hand is removed, control the robotic arm to return to the preset position; Step S1 includes: Step S1.1: Obtain the fixed position information of the target bamboo stick, control the gripper of the first robotic arm to grab the target bamboo stick based on the fixed position information of the target bamboo stick, and determine whether the target bamboo stick has been successfully grabbed by force control sensing of the gripper and real-time feedback of the gripping width. Step S1.2: Identify the center location of the target raw material and the distance between the target raw material and adjacent raw material locations using AI; Step S1.3: Use the identified center position of the target material as the center position of the gripper of the second robotic arm; obtain the gripping posture of the gripper of the second robotic arm based on the distance between the target material position and the adjacent material positions, so that the second robotic arm can avoid interference between the gripper and the material to the greatest extent. Step S1.4: Based on the gripping posture of the second robotic arm gripper, and combined with the width of the target material, obtain the opening width of the second robotic arm gripper; Step S1.5: Grasp the target raw material based on the gripping posture and opening width of the second robotic arm's gripper; Step S4 includes: The first robotic arm that is currently grasping bamboo skewers and the second robotic arm that is grasping raw materials move relative to each other along the central axis of the bamboo skewers based on the starting point and ending point of skewering to implement the skewering of the robotic arms; During the process from the start point to the end point of stringing, the second robotic arm that grabs the raw materials obtains the sensing force through the force control sensor of the first robotic arm. When the sensing force is always less than or equal to the preset value, it is determined that the current stringing has failed; when the sensing force is greater than the preset value, it is determined that the current stringing has been completed. Step S7 includes: setting up a detection device in a preset food collection area to detect whether a hand enters; when a hand is detected, triggering an image acquisition device; acquiring image information of the food collection area in real time through the image acquisition device; performing hand recognition based on the image information; when no hand information is detected in the image information, and the robotic arm gripper releases a certain width of preset time, it is considered that the hand has been removed, and the robotic arm is controlled to return to the preset position.

2. The automated threading method based on dual-robotic arm collaborative control and force sensing feedback as described in claim 1, characterized in that, Step S2 includes: Step S2.1: Move the first robotic arm that grabs bamboo skewers and the second robotic arm that grabs raw materials to a preset range directly above the raw material cooking pot, and make the center point of the raw material located on the central axis of the bamboo skewers, thus completing the basic positioning of the first robotic arm that grabs bamboo skewers and the second robotic arm that grabs raw materials to meet the preset requirements. Step S2.2: Use AI to identify the spatial position of the tip of the bamboo stick that the first robotic arm grasps and the angle between the bamboo stick and the horizontal line.

3. The automated threading method based on dual-robotic arm collaborative control and force sensing feedback according to claim 1, characterized in that, Step S3 includes: Step S3.1: In the Cartesian coordinate system of the first robotic arm, the spatial position of the tip of the bamboo skewer is obtained by AI recognition, and the starting point of stringing is obtained based on the spatial position of the tip of the bamboo skewer. Step S3.2: Calculate the skewering end point by recognizing the tilt angle of the bamboo skewers and the order of the meatballs to be skewered using AI.

4. The automated threading method based on dual-robotic arm collaborative control and force sensing feedback according to claim 1, characterized in that, Step S5 includes: obtaining the force of each mechanical joint based on the torque sensor integrated in each joint of the robotic arm; and obtaining the end force by back-calculating the force of the end effector based on the obtained force of each mechanical joint through a dynamic model.

5. An automated threading system based on dual-robotic arm collaborative control and force sensing feedback, characterized in that, include: Module M1: The first robotic arm grabs bamboo skewers, and the second robotic arm grabs raw materials; Module M2: Performs basic positioning of the first robotic arm for grasping bamboo skewers and the second robotic arm for grasping raw materials to meet preset requirements; under the basic positioning of the first robotic arm for grasping bamboo skewers and the second robotic arm for grasping raw materials to meet preset requirements, identifies the position information of bamboo skewers; wherein, the spatial position of the tip of the bamboo skewer and the angle information between the target bamboo skewer and the horizontal line; Module M3: Calculates the alignment points of the first and second robotic arms based on the identified bamboo skewer position information; Module M4: Based on the alignment of the first and second robotic arms at the threading points, the robotic arms perform threading and determine whether the threading is successful through force control sensing, thus achieving closed-loop stable threading. Module M5: Controls the robotic arm holding the skewered bamboo sticks to reach the preset position and resets the force sensing of the robotic arm to zero. Module M6: When the force sensing of the robotic arm changes, it controls the gripper of the robotic arm to loosen by a certain width; Module M7: When the human hand is removed, the robotic arm is controlled to return to the preset position; The module M1 includes: Module M1.1: Obtain the fixed position information of the target bamboo stick, control the gripper of the first robotic arm to grab the target bamboo stick based on the fixed position information of the target bamboo stick, and determine whether the target bamboo stick has been successfully grabbed by force control sensing of the gripper and real-time feedback of the gripping width. Module M1.2: Uses AI to identify the center location of the target raw material and the distance between the target raw material and adjacent raw material locations; Module M1.3: The center position of the identified target material is used as the center position for the gripper of the second robotic arm; the gripping posture of the second robotic arm gripper is obtained based on the distance between the target material position and the adjacent material positions, so that the second robotic arm can avoid interference between the gripper and the material to the greatest extent. Module M1.4: Based on the gripping posture of the second robotic arm gripper, and combined with the width of the target material, obtain the opening width of the second robotic arm gripper; Module M1.5: Based on the gripping posture and opening width of the second robotic arm's gripper, the target raw material is gripped; The module M4 includes: The first robotic arm that is currently grasping bamboo skewers and the second robotic arm that is grasping raw materials move relative to each other along the central axis of the bamboo skewers based on the starting point and ending point of skewering to implement the skewering of the robotic arms; During the process from the start point to the end point of stringing, the second robotic arm that grabs the raw materials obtains the sensing force through the force control sensor of the first robotic arm. When the sensing force is always less than or equal to the preset value, it is determined that the current stringing has failed; when the sensing force is greater than the preset value, it is determined that the current stringing has been completed. The module M7 includes: setting a detection device in a preset food pick-up area to detect whether a hand enters; when a hand is detected, triggering an image acquisition device; acquiring image information of the food pick-up area in real time through the image acquisition device; performing hand recognition based on the image information; when no hand information is detected in the image information, and the robotic arm gripper releases a certain width of preset time, it is considered that the hand has been removed, and the robotic arm is controlled to return to the preset position.

6. The automated threading system based on dual robotic arm collaborative control and force sensing feedback according to claim 5, characterized in that, The module M2 includes: Module M2.1: Move the first robotic arm that grabs bamboo skewers and the second robotic arm that grabs raw materials to a preset range directly above the raw material cooking pot, and make the center point of the raw material located on the central axis of the bamboo skewers, thus completing the basic positioning of the first robotic arm that grabs bamboo skewers and the second robotic arm that grabs raw materials to meet the preset requirements. Module M2.2: Uses AI to identify the spatial position of the tip of the bamboo skewer grasped by the first robotic arm and the angle between the bamboo skewer and the horizontal line.

7. The automated threading system based on dual robotic arm collaborative control and force sensing feedback according to claim 5, characterized in that, The module M3 includes: Module M3.1: In the Cartesian coordinate system of the first robotic arm, the spatial position of the tip of the bamboo stick is obtained through AI recognition, and the starting point of stringing is obtained based on the spatial position of the tip of the bamboo stick; Module M3.2: Calculates the end point of skewering by recognizing the tilt angle of the bamboo skewers and the order of the meatballs to be skewered using AI; The module M5 includes: obtaining the force of each mechanical joint based on the torque sensor integrated in each joint of the robotic arm; and obtaining the end force by back-calculating the force of each mechanical joint through a dynamic model.

Images