A method for unpacking and feeding of an industrial robot supporting natural language instructions

Abstract

The application provides an industrial robot unpacking and feeding method supporting natural language instructions, and the method comprises the following steps: S1, receiving the unstructured natural language demand information about the unpacking and feeding task input by a user through a natural language interaction interface, wherein the demand information comprises at least one of the material type, the packaging specification, the feeding rate and the production line environment layout; S2, performing semantic analysis on the natural language demand information by using a large language model fine-tuned by professional knowledge in the field of robot unpacking and feeding, and performing intention recognition and key information extraction; if it is identified that the demand information is missing or ambiguous, generating natural language prompt information for follow-up questioning and returning to the user through the interaction interface, and repeating the step until a complete and clear task parameter set is obtained.

Description

Technical Field

[0001] This invention belongs to the field of industrial robots, and specifically relates to a method for unpacking and loading materials for industrial robots that supports natural language commands. Background Technology

[0002] Currently, unpacking and loading is a common step in industrial production, typically requiring robotic systems to grasp and unpack the bags and transport the materials to the production equipment. However, the configuration process for traditional robotic unpacking and loading systems is extremely complex. Operators need professional robot programming knowledge and a thorough understanding of unpacking and loading processes to complete tasks such as setting system parameters, planning paths, and choreographing actions. For example, based on the size, material, and weight of the packaging bags, as well as the equipment layout on the production line, operators need to manually adjust parameters such as the robot's gripping force, unpacking position, and loading speed. The entire process is time-consuming and labor-intensive, and is prone to system malfunctions due to human error.

[0003] Meanwhile, different production scenarios have significantly different requirements for unpacking and loading systems. When it is necessary to change the material type or adjust the production rhythm, operators need to perform complex reconfigurations, resulting in poor adaptability and severely impacting production efficiency. Furthermore, traditional systems often rely on fixed commands or complex interfaces for interaction, which is not intuitive or convenient, further increasing the user's learning curve. These issues have greatly limited the widespread adoption and application of robotic unpacking and loading systems in small and medium-sized enterprises. Summary of the Invention

[0004] This invention proposes an industrial robot unpacking and loading method that supports natural language commands, solving the technical problems of industrial robot unpacking and loading systems being highly dependent on professional programmers for configuration and operation, having poor flexibility, being unable to intelligently understand and respond to unstructured natural language commands, and lacking autonomous decision-making ability under abnormal working conditions.

[0005] The technical solution of the present invention is implemented as follows: a method for unpacking and loading materials using an industrial robot that supports natural language commands, the method comprising the following steps: S1. Receive unstructured natural language requirement information about the unpacking and loading task from the user through a natural language interaction interface. The requirement information includes at least one of the following: material type, packaging specifications, loading rate, and production line layout. S2. Using a large language model fine-tuned with expertise in the field of robot unpacking and loading, semantic analysis of the natural language requirement information is performed to perform intent recognition and key information extraction. If the requirement information is found to be missing or ambiguous, natural language prompts for follow-up questions are generated and returned to the user through the interactive interface. This step is repeated until a complete and clear set of task parameters is obtained. S3. In response to obtaining the complete set of task parameters, the pre-built domain knowledge base is invoked for matching and logical reasoning to generate the workstation configuration basis; the domain knowledge base includes robot parameter library, gripper adaptation library, material property library and unpacking process library. S4. Based on the configuration criteria, automatically select hardware configuration from multiple preset configuration templates and plan the workstation layout. At the same time, automatically generate the corresponding robot motion trajectory program, motion control instructions and system operating parameters to form a complete workstation configuration scheme. The motion control instructions include gripping force, unpacking tool angle and cutting depth. S5. Convert the motion trajectory program, motion control instructions and operating parameters in the workstation configuration scheme into control signals that can be recognized by the underlying actuator, and drive the robot, conveyor belt and unpacking tool to perform unpacking and loading operations in a coordinated manner. S6. Monitor the operating status of the actuator in real time. When an abnormality is detected, feed the abnormal status information back to the large language model. The large language model generates anomaly handling instructions, which include sending alarm prompts to the user through a natural language interaction interface or automatically adjusting the parameters of the workstation configuration scheme.

[0006] Existing industrial robot unpacking and loading systems generally suffer from insufficient automation and intelligence. The core problem lies in the heavy reliance on human experience and offline programming for system configuration, instruction generation, and anomaly handling. Specifically, current technology lacks a deep understanding of natural language instructions. Operators must use strictly formatted technical terminology or input parameters through complex software interfaces. Any ambiguity or missing information in the description of requirements will cause the system to fail to execute. Non-professional users find it difficult to operate directly, resulting in low human-machine interaction efficiency.

[0007] The system configuration process is rigid and isolated. Selecting robot models, end effectors, planning motion trajectories, and setting process parameters (such as gripping force and cutting depth) rely on engineers manually consulting manuals, making decisions based on experience, and writing code. This is not only time-consuming and labor-intensive, but also makes it difficult to guarantee the overall optimality of the configuration scheme. In particular, when material characteristics or production cycle changes, a lot of manual adjustments are required, resulting in poor system adaptability.

[0008] Existing system knowledge bases (such as equipment and material databases) are often merely static reference data, failing to deeply integrate with automatic configuration logic. This hinders rule-based automatic reasoning and decision-making, resulting in low knowledge application efficiency. Finally, existing systems suffer from weak monitoring and anomaly handling capabilities. When anomalies such as fetching failures or material blockages occur during operation, they typically only trigger simple alarms or shutdowns, lacking intelligent diagnostic capabilities to identify the root causes of the anomalies. Furthermore, they cannot autonomously generate effective recovery or adjustment strategies, heavily relying on manual intervention and hindering production continuity.

[0009] The technical challenges addressed by this invention are as follows: First, how to enable machines to truly "understand" unstructured natural language instructions rich in context and potentially ambiguous information, and to possess the ability to proactively complete key information through multi-round interactions. The core of this is constructing a large-scale language model that has been fine-tuned for a specific domain and possesses both language understanding and domain knowledge reasoning capabilities. Second, how to automatically match and logically reason with the parsed user requirements against a structured, multi-dimensional domain knowledge base, thereby generating accurate equipment selection, layout planning, and process parameter configuration bases without human intervention, achieving automatic conversion from "requirements" to "configuration schemes." Third, how to automatically compile high-level configuration schemes into low-level robot motion control instructions and system control parameters that can directly drive actuators. This is a complex process involving the integration of kinematics, dynamics calculations, and process knowledge. Fourth, how to construct a closed-loop control mechanism of "perception-decision-execution-feedback" so that the system can perceive the operating status in real time and use the analytical reasoning capabilities of large language models to diagnose and make decisions on anomalies, and finally generate coping strategies including natural language prompts and automatic parameter adjustment, thereby realizing the system's autonomy and resilience under abnormal operating conditions.

[0010] As a preferred implementation, in step S2, semantic parsing of natural language demand information specifically includes: using a finely tuned large-scale language model to perform deep semantic understanding and contextual analysis on the input text, identifying the domain operation category corresponding to the user's intent, and extracting key operation parameters from the statement based on predefined entity extraction rules; when generating natural language prompts for follow-up questions, the model dynamically constructs a multi-turn dialogue process based on the logical correlation of missing parameters, and constrains the professionalism and accuracy of the generated content through a domain knowledge base.

[0011] As a preferred implementation, the deep semantic understanding and contextual association analysis is based on a domain-adaptive semantic parsing algorithm, which focuses on material characteristic descriptions, packaging specifications, and process requirements through an attention mechanism. The entity extraction rules adopt a sequence labeling model based on a combination of bidirectional long short-term memory networks and conditional random fields, specifically for identifying and normalizing compound noun phrases in the unpacking and loading domain. The multi-turn dialogue process adopts a state machine-based dialogue management strategy, which dynamically generates a follow-up question sequence based on the logical dependencies between currently confirmed parameters and missing parameters. The generation of each follow-up question node is verified by the equipment parameter constraints and process feasibility rules in the domain knowledge base.

[0012] In a preferred embodiment, the sequence labeling model uses a pre-trained language model based on the Transformer architecture as a feature extractor, and its output layer is connected to the CRF layer to jointly decode the optimal label sequence; the state machine node in the dialogue management strategy includes a parameter completeness verification module, which dynamically evaluates the completeness of the current dialogue state by querying the equipment compatibility rules and process parameter constraints in the domain knowledge base.

[0013] As a preferred implementation, in step S3, the domain knowledge base is organized using a knowledge graph structure, in which the robot parameter library, gripper adaptation library, material characteristic library, and unpacking process library are interconnected in the form of entity-relationship-attribute triples; the logical reasoning process is implemented based on a rule engine and graph retrieval algorithm, and by matching the physical characteristics and mechanical performance constraints of the materials, the equipment combination and operation parameters that meet the process requirements are inferred.

[0014] As a preferred embodiment, in step S4, the robot motion trajectory program includes: calculating the optimal path of each joint of the robotic arm based on the workstation layout planning results and material physical properties through kinematic algorithms, and introducing a collision detection algorithm to avoid interference with surrounding equipment; the generation of the motion control command integrates material morphological features and gripper mechanical model, and dynamically adapts to gripping strategies and tool operating parameters.

[0015] As a preferred implementation, the anomaly monitoring in step S6 includes capturing the position and orientation of the actuator and the material flow pattern in real time through a visual sensor and comparing them with the expected state in the digital twin system; when the deviation exceeds the threshold, the anomaly handling process is triggered, and the large language model performs root cause analysis based on the historical fault case library and generates adaptive adjustment instructions that include equipment parameter reconfiguration and motion trajectory optimization.

[0016] In a preferred embodiment, the visual sensor is a multimodal sensing unit that integrates a high-resolution industrial camera and a 3D point cloud scanning device. It fuses 2D texture information and 3D geometric data through a spatiotemporal synchronous registration mechanism. The digital twin system constructs a dynamic behavior pattern comparison benchmark by mapping the pose deviation and material flow morphology changes of the physical entity and the virtual model in real time. When a statistically significant deviation is detected between the multidimensional perception data and the virtual expected state, an abnormal diagnosis process is triggered. The large language model performs fault root cause reasoning based on multi-source data correlation analysis and historical fault case library, and generates a composite adjustment command that includes equipment parameter reconfiguration, online optimization of motion trajectory, and fault self-recovery strategy.

[0017] After adopting the above technical solution, the beneficial effects of the present invention are: Lowering the barrier to entry: Through natural language interaction, users do not need professional knowledge of robot programming and unpacking processes. They can complete the system configuration simply by expressing their needs in everyday language, which greatly reduces the professional requirements for operators and makes the system easy for more non-professionals to use.

[0018] Simplified configuration process: Based on large model technology, the system can automatically understand user needs and complete the hardware configuration, program generation and parameter setting of the workstation, eliminating the tedious manual operations in the traditional method and significantly improving configuration efficiency, reducing it from several hours or even days to a few minutes.

[0019] Improve adaptability and flexibility: The system can automatically generate corresponding configuration schemes according to the needs of different users (different materials, different outputs, different production line layouts, etc.). When the needs change, the system can be quickly reconfigured by simply modifying the requirements through natural language interaction, adapting to a variety of production scenarios.

[0020] Enhanced user experience: Natural language interaction is more intuitive and convenient, conforms to human communication habits, and the system can proactively ask follow-up questions to clarify needs, reducing the user's operational burden and improving the user experience.

[0021] Enhanced system reliability: Through logical reasoning of large models and knowledge-based configuration, human error is reduced. At the same time, combined with real-time monitoring and feedback mechanisms, abnormal situations can be handled in a timely manner, improving the reliability of system operation.

[0022] By introducing a collaborative working mechanism between a domain-fine-tuned large-scale language model and a structured knowledge base, significant benefits have been achieved. First, it greatly reduces the technical barrier to system operation and configuration. Users do not need professional robot programming or process knowledge; they can drive the system to generate and deploy a complete solution simply by describing their needs in natural language. This achieves excellent human-computer interaction and operational efficiency, shortening system deployment and changeover adjustment time. Second, it significantly improves the intelligence and automation level of system configuration. The deep semantic understanding and multi-turn interaction capabilities of the large-scale language model ensure accurate capture of user intent and completeness of requirement information. Combined with rule reasoning and matching from domain knowledge bases (such as equipment, material, and process libraries), the system can automatically output optimized and reliable hardware configurations, layout schemes, and detailed motion and control instructions. This completely replaces the traditional model that relies on manual programming and trial and error by engineers, ensuring the scientific rigor and optimality of the solution, while also significantly improving the flexibility to respond to changing requirements. Third, it enables efficient knowledge reuse and continuous optimization of system performance. The structured domain knowledge base accumulates valuable expert experience and historical data. Through automatic system invocation and reasoning, professional knowledge can be standardized and applied at scale. Simultaneously, the continuous accumulation of operational data and historical solutions can feed back into and optimize the knowledge base and models, forming a virtuous cycle of self-improvement. Finally, it enhances the system's reliability and ability to autonomously handle anomalies. Through real-time status monitoring and digital twin comparison, problems can be detected promptly. Utilizing the powerful analytical and reasoning capabilities of large-scale language models, root cause analysis of anomalies can be performed, generating intelligent recovery strategies including parameter adjustment and trajectory replanning, or providing users with precise natural language alerts. This significantly reduces unplanned downtime, improves the overall equipment efficiency (OEE) and adaptability of the production line, and provides key technical support for realizing unmanned intelligent factories. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] Example: like Figure 1 As shown, an industrial robot unpacking and loading method supporting natural language commands is disclosed, the method comprising the following steps: S1. Receive unstructured natural language requirement information about the unpacking and loading task from the user through a natural language interaction interface. The requirement information includes at least one of the following: material type, packaging specifications, loading rate, and production line layout. S2. Using a large language model fine-tuned with expertise in the field of robot unpacking and loading, semantic analysis of the natural language requirement information is performed to perform intent recognition and key information extraction. If the requirement information is found to be missing or ambiguous, natural language prompts for follow-up questions are generated and returned to the user through the interactive interface. This step is repeated until a complete and clear set of task parameters is obtained. S3. In response to obtaining the complete set of task parameters, the pre-built domain knowledge base is invoked for matching and logical reasoning to generate the workstation configuration basis; the domain knowledge base includes robot parameter library, gripper adaptation library, material property library and unpacking process library. S4. Based on the configuration criteria, automatically select hardware configuration from multiple preset configuration templates and plan the workstation layout. At the same time, automatically generate the corresponding robot motion trajectory program, motion control instructions and system operating parameters to form a complete workstation configuration scheme. The motion control instructions include gripping force, unpacking tool angle and cutting depth. S5. Convert the motion trajectory program, motion control instructions and operating parameters in the workstation configuration scheme into control signals that can be recognized by the underlying actuator, and drive the robot, conveyor belt and unpacking tool to perform unpacking and loading operations in a coordinated manner. S6. Monitor the operating status of the actuator in real time. When an abnormality is detected, feed the abnormal status information back to the large language model. The large language model generates anomaly handling instructions, which include sending alarm prompts to the user through a natural language interaction interface or automatically adjusting the parameters of the workstation configuration scheme.

[0027] This application protects a method for unpacking and loading industrial robots that supports natural language commands. In a typical implementation scenario, its working principle and workflow can be detailed as follows: The user, through a natural language interface integrated into the industrial control terminal, submits a specific unpacking and loading task request in voice or text form, such as, "Please configure a loading system for processing 25 kg bags of polypropylene granules, requiring 15 bags per hour. The production line entrance is located directly in front of the control panel." This unstructured natural language input is captured by the system and immediately transmitted to the core processing unit—a large-scale language model fine-tuned with extensive domain-specific expertise in robot unpacking and loading. This model first performs deep semantic parsing and contextual analysis on the input statement, performs domain-specific intent recognition, determines that the user's goal is "system configuration" and "loading task execution," and simultaneously extracts key operational parameter entities, such as material type (polypropylene granules), packaging specifications (25 kg bags), loading rate (15 bags per hour), and relative position information (directly in front of the control panel). Given the potential for implicit or missing information in natural language descriptions, the model performs completeness checks based on its internally constructed domain knowledge logic graph. If key parameters, such as the specific dimensions of the packaging bag or the material flow properties, are not clearly defined, the model dynamically generates context-sensitive natural language follow-up questions, such as "Please provide the standard packaging bag length and width dimensions of the bagged polypropylene so that we can accurately fit the handle opening and closing range for you." This information is then fed back to the user through the same interactive interface, initiating multiple rounds of dialogue until all necessary parameters are clarified and completed, ultimately forming a complete, structured, and unambiguous set of task parameters.

[0028] Subsequently, in response to the complete parameter set, the system initiates the automatic configuration generation process. The large language model serves as the intelligent reasoning hub, invoking a pre-built domain knowledge base based on a knowledge graph architecture. This knowledge base deeply integrates robot parameter libraries, gripper adaptation libraries, material characteristic libraries, and unpacking process libraries. Efficient association and cross-modal retrieval between these libraries are achieved through entity-relationship-attribute triples. The model executes logical reasoning and matching based on rules and graph traversal: First, based on the material characteristics of "polypropylene granules" (such as particle shape, friction coefficient, and possible electrostatic adsorption properties), the material characteristic library is queried to determine the required unpacking process category (vibrationless cutting is recommended to avoid contamination) and the constraints on the feeding speed. Then, combining the packaging specifications and feeding rate requirements, robot models that meet the requirements in terms of load capacity, working range, and repeatability are selected from the robot parameter library. Simultaneously, based on the material characteristics and packaging bag material (such as woven bags), gripper types with anti-slip textures and adaptive gripping force are recommended from the gripper adaptation library, and a rotary cutting tool and its corresponding cutting angle and speed control parameters are matched from the unpacking process library. All reasoning results are aggregated to generate detailed workstation configuration data.

[0029] Upon receiving this information, the workstation configuration module activates its built-in configuration algorithm and template library. It first performs automatic selection and layout planning of hardware resources: based on the relative position on the production line (directly in front of the control panel), it calculates and determines the optimal base position of the robot, the direction and length of the conveyor belt, ensuring no interference within the working area and the shortest logistics path. Next, the core automatic programming process begins: based on the kinematic model and dynamic parameters of the selected robot, the motion planning algorithm automatically calculates a series of continuous path points for the robotic arm to grasp, transport, unpack, and dump materials, and introduces a collision detection algorithm to ensure the safety of the trajectory in the virtual environment; the synchronously generated motion control commands deeply integrate the previous reasoning results. For example, to adapt to a 25 kg load and the surface characteristics of woven bags, the grasping force is set within a threshold range that ensures reliable grasping without easily causing packaging deformation, and the cutting depth and rotation speed of the rotary cutting tool are precisely set to avoid damaging the inner material; system operating parameters such as conveyor belt speed and the timing sequence of robot collaboration are also generated, thus compiling a complete workstation configuration scheme that can be directly deployed.

[0030] After user confirmation, the control module takes over the process. It compiles the abstract trajectory program and control commands from the solution into specific control signals (such as EtherCAT or Profinet messages) recognizable by the robot controller, frequency converter, and tool driver via the underlying driver interface. These signals are then sent to their respective actuators, driving the robot to move precisely along the predetermined trajectory, the conveyor belt to run synchronously at a set speed, and the unpacking tool to perform cutting operations at designated positions. The entire system then enters a collaborative operation state.

[0031] Throughout the physical execution process, the system's state monitoring unit operates continuously. Through integrated high-resolution vision sensors, force sensors, and encoder feedback, it collects real-time pose data, torque information, and material flow status of each actuator. This real-time data is continuously compared with a virtual expected operating model built based on digital twin technology. Once an abnormal deviation is detected, such as the vision system identifying a grasping pose shift leading to grasping failure, or the force sensor detecting an abnormal increase in cutting resistance, this abnormal state information is immediately encapsulated and fed back to the large-scale language model. The large-scale language model then acts as a fault diagnosis engine, integrating current abnormal characteristics, historical operating logs, and fault case libraries from the domain knowledge base to perform multi-source information correlation and root cause reasoning analysis. Based on the analysis results, the model generates intelligent anomaly handling instructions: if it is a non-fatal error, such as a slight deviation in the posture of the material bag, the model can automatically generate adjustment parameters (such as fine-tuning the coordinates of the robot's gripping point or gripping force) and send them directly to the actuator to attempt self-recovery; if a complex fault requires human intervention, it sends precise alarm prompts to the user through a natural language interaction interface, such as "Warning: The bagged material at workstation 3 is not pouring smoothly, suspected of being blocked. It is recommended to check on-site and clean the guide channel." This forms a complete closed-loop control from intelligent perception, analysis and decision-making to automatic execution or manual assistance, significantly improving the system's autonomy and reliability.

[0032] In a specific application, a small building materials factory needs to configure a robotic unpacking and feeding system to handle bagged lime powder.

[0033] User interaction process: The user says to the natural language interaction module via voice: "I want to install a bag unpacking system to process 25 kg bags of lime powder. The bags are made of kraft paper. The process is about 15 bags per hour. Our robot is model A and should be placed in the right corner." Natural Language Interaction Module: Converts speech into text and transmits it to the large model processing module.

[0034] Large Model Processing Module: Parses the user requirements: The material is 25 kg kraft paper bags of lime powder, with a production capacity of 15 bags / hour, using the existing Model A robot, located in the right corner. Calls the data storage and retrieval module to retrieve parameters such as the Model A robot's load capacity and working radius, as well as the unpacking characteristics of the kraft paper bags of lime powder (dust prevention is necessary; the blade selection should avoid excessive dust generation). Confirms the requirements are complete and outputs the configuration basis to the workstation configuration module.

[0035] Workstation Configuration Module: Based on the configuration criteria, and considering the lime powder and the kraft paper bags, a dustproof gripper and round-headed unpacking cutter are selected. According to the parameters and position of Robot Model A, the motion trajectory is planned to ensure the robot can grab the bags from the picking point, move them to the unpacking position, and then deliver the material to the designated production line hopper. The gripping force is set to 50N (to avoid tearing the bags or failing to grip them securely), the unpacking speed is moderate to reduce dust, and the feeding rhythm matches a production rate of 15 bags / hour. The corresponding control program and parameters are generated.

[0036] Natural Language Interaction Module: Provides the configuration plan to the user: "The configuration plan is as follows: Using your existing Model A robot, equipped with a dustproof gripper and round-headed blades, with a gripping force of 50N, it can process 15 bags of 25kg kraft paper bags of lime powder per hour. The robot will complete the gripping, unpacking, and loading according to the planned trajectory. Confirm?" After user confirmation: The workstation configuration module stores the plan and sends it to the execution control module.

[0037] The execution control module drives the Model A robot, dustproof gripper, round-headed cutter, etc., to work according to the plan. During operation, a slight shift in the bag position was detected during one gripping operation. This was fed back to the large model processing module, which then prompted the user through the natural language interaction module: "A shift in the bag position has been detected. The gripping position has been automatically adjusted. The system is operating normally." In this embodiment, users do not need professional knowledge and can complete the system configuration through simple natural language interaction. The configuration process only takes 5 minutes. The system runs stably and meets production needs, fully demonstrating the advantages of the present invention.

[0038] In the preferred embodiment of step S2, the process of semantically parsing the natural language demand information is manifested as an intelligent interactive understanding stage that deeply integrates domain knowledge. Specifically, when a user gives an instruction such as "I need a system for unpacking bagged starch, with high capacity, and the production line is on the right," the finely tuned large-scale language model first initiates a deep semantic understanding and contextual analysis mechanism. The model does not perform simple keyword matching, but uses its internally constructed domain semantic network to identify "bagged starch" as a lightweight, dusty powder, interpret "high capacity" as a non-quantitative requirement for system cycle time and feeding rate, and understand "right" as a relative position description from the operator's perspective in the absence of an absolute coordinate system. This process involves accurately identifying the user's intent, categorizing it into the core operation category of "system configuration," and further refining it into the subcategory of "powder unpacking." Based on predefined entity extraction rules, the model extracts core operation parameters from the seemingly short statement: target material (starch), packaging form (bagged), performance expectation (high capacity), and layout intention (right side). However, the initial instructions contained significant information gaps, such as a lack of clear packaging weight, bag dimensions, precise production capacity figures, and specific distance information.

[0039] At this point, the model dynamically constructs a structured multi-turn dialogue process based on the logical relationships between the missing parameters. It first determines that material characteristics (starch) and packaging specifications (weight, dimensions) are the most fundamental constraints for equipment selection (such as robot load and gripper size), thus generating the follow-up question: "What are the standard packaging weight and length, width, and height dimensions of bagged starch?" After the user adds "25 kg per bag, with approximate length, width, and height," the model realizes that the vague expression "high capacity" needs to be quantified for precise cycle time calculation and equipment capacity verification. Therefore, it initiates a second follow-up question: "To achieve your desired high capacity, how many bags per hour do you aim to process?" Throughout the interaction, the model does not ask questions in isolation; each follow-up question is strictly constrained by the domain knowledge base to ensure professionalism and accuracy. For example, after the user provides the packaging dimensions, the model will immediately perform an implicit comparison with the grasping range of standard grippers in the knowledge base. If an abnormality is found (such as being too long or too thin), the model will proactively suggest possible adaptation problems or recommend special gripper options in subsequent dialogues, thereby guiding the dialogue to a more professional and in-depth level. This ensures that the final set of task parameters is not only complete but also has undergone preliminary technical feasibility verification, laying a solid and reliable foundation for subsequent automatic configuration.

[0040] In a preferred embodiment of step S3, the domain knowledge base is organized using a knowledge graph structure, which is the core infrastructure for realizing intelligent reasoning. In this knowledge graph, each entity (such as a specific model of robot, a gripper, a material, or a process) in the robot parameter library, gripper adaptation library, material property library, and unpacking process library is assigned a unique identifier and forms a densely interconnected network through "entity-relationship-attribute" triples. For example, a "six-axis industrial robot" entity has attributes such as "rated load," "working radius," and "repeatability"; it is connected to a series of "gripper" entities through the "can be mounted" relationship; a "vacuum suction cup gripper" entity is associated with the "flat-surface bagged material" entity through the "suitable for" relationship; the "bagged starch" material entity not only has physical and chemical attributes such as "density," "angle of repose," and "dust-proneness," but is also connected to the "rotary blade cutting and unpacking" process entity through the "recommended process" relationship, and this process entity itself has attributes such as "tool speed" and "cutting angle."

[0041] The logical reasoning process is implemented on this massive knowledge graph through the collaboration of a rule engine and a graph retrieval algorithm. Continuing with the aforementioned scenario of bagged starch, the rule engine first receives the complete parameter set from S2. It initiates the reasoning chain: First, based on the characteristics of the material "starch" (powdered, easily dusty), it retrieves and identifies "closed-loop rotary cutting and unpacking" as the optimal process from the process library, which effectively suppresses dust diffusion. Next, based on the packaging weight and dimensions, it calculates the total mass and gripping volume. Using this as a constraint, it traverses all "robot" entities in the graph that can be connected to the "vacuum suction cup gripper" through the "can be mounted" relationship, filtering out a candidate set of robot models that meet the requirements for rated load and working radius. Simultaneously, the rule engine verifies the synergy of the equipment combination: for example, whether the end effector interface of the selected robot is compatible with the mounting interface of the target gripper, and whether the suction cup material of the selected gripper has a good history of airtightness compatibility with the surface material of the starch packaging bag (such as a woven bag). Ultimately, the reasoning process outputs a highly compatible configuration: recommending the use of a six-axis articulated robot with a specific load range, equipping it with a dustproof vacuum suction cup gripper suitable for powder packaging, employing a closed rotary cutting and unpacking process unit, and initially setting a series of operational parameter ranges. This entire process is not a simple query, but a complex calculation based on constraint satisfaction and optimal matching, ensuring the technical feasibility and process rationality of the recommended solution.

[0042] In the preferred embodiment of step S4, the automatic generation of robot motion trajectory programs and control instructions is a crucial bridge connecting virtual configuration and physical execution. This process begins after the workstation layout planning is completed, at which point the system obtains a relatively accurate three-dimensional spatial relationship between the robot, conveyor belt, unpacking station, and production line interface. Based on this layout and the physical characteristics of the materials (such as the size, weight, and fragility of starch bags), the motion planning algorithm begins its work. It first determines a series of critical path points (Via Points) and goal points based on the task sequence (grabbing, moving, unpacking, and dumping). Subsequently, using kinematic algorithms, it inversely calculates the displacement, velocity, and acceleration curves of each joint of the robotic arm along the path from the starting point to the end point, aiming to generate an optimal path that is efficient in time and smooth in motion. A crucial element is the introduction of a collision detection algorithm, which continuously detects whether the moving robot model geometrically interferes with other static equipment (such as guardrails) or dynamic equipment (such as running conveyor belts) within the workstation in the three-dimensional virtual environment. For bagged starch scenarios, the planned trajectory must ensure that the robot moves smoothly in a narrow space while holding a large bag, and that the rapidly rotating unpacking blades maintain a safe distance from the robot's own arm, bag debris, and dust collection device inlet during operation.

[0043] Meanwhile, the generation of motion control commands is a process that deeply integrates perception and models. The generation of gripping commands is not based on a fixed value, but rather integrates the morphological characteristics of the material (the rigidity and surface texture of the starch bag) with the mechanical model of the gripper (the relationship between the flow rate and negative pressure of the vacuum generator). The system dynamically adapts the gripping strategy: for example, to avoid positional deviation caused by the bag sinking due to its softness during suction gripping, the command includes a "pre-tightening" step, which involves applying a small negative pressure to tighten the surface of the packaging bag before lifting to the rated negative pressure. For the unpacking cutter's operating parameters, the system also dynamically sets the cutter's rotation speed, cutting depth, and feed rate based on material characteristics (powders are susceptible to breakage from excessive cutting) and recommendations from the process knowledge base. For example, for starch bags, a higher rotation speed and a shallower cutting depth are selected to achieve quick and clean opening of the packaging while minimizing disturbance to the material inside the bag and wear on the cutter. Ultimately, the results of all these algorithms are compiled into brand-specific language code (such as KRL or URScript) and IO control instructions that the robot controller can directly parse and execute, forming a highly customized, out-of-the-box workstation configuration.

[0044] In the preferred embodiment of step S6, the anomaly monitoring and handling constructs an intelligent closed loop that enables the system to be resilient. The core of this mechanism lies in the continuous real-time capture of the actuator's pose and material flow pattern using visual sensors. In a typical scenario of processing bagged starch, a high-resolution industrial camera continuously monitors the deviation between the actual position of the robot's end effector and the preset target position, ensuring accurate gripping. Simultaneously, it also monitors the unpacking station, determining whether the bags are correctly delivered to the cutting position and whether the cut material is smoothly poured into the next feed inlet. All this real-time acquired multi-dimensional sensing data is synchronously transmitted to a digital twin system. This system maintains a virtual model that is a complete mirror image of the physical entity and runs a time-series animation of the expected ideal working state.

[0045] In a digital twin environment, the real-time pose data of physical entities and the material flow pattern (such as the pouring flow line of powder) are continuously compared with the virtual expected state at millisecond levels. This comparison is not a simple numerical comparison, but a dynamic behavior analysis based on pattern recognition. When the deviation exceeds a preset safety threshold—for example, the vision system detects a continuous deviation between the actual position of the robot's end effector and the expected trajectory after a certain grasping action, or detects continuous accumulation of starch at the pouring port instead of smooth flow—the system immediately triggers an anomaly handling process. At this point, a large language model intervenes as the brain, receiving descriptions of abnormal features from the perception layer (such as "the gripper deviates from the expected position by millimeters in the Y-axis direction" or "the flow rate at the discharge port has slowed down by tens of percent"), and calling upon a historical failure case library for root cause analysis. By comparing historical data, the model may infer that the current deviation is due to a leak in one of the vacuum suction cups, resulting in insufficient gripping force, which in turn causes the bag to slip during accelerated movement; while the poor discharge may be due to an abnormal increase in material moisture, leading to slight adhesion and blockage. Based on this diagnosis, the adaptive adjustment instructions generated by the large language model are complex: for grasping misalignment, the instructions may include immediately fine-tuning the position of subsequent grasping points to compensate, and issuing instructions to slightly increase the power of the vacuum generator to maintain effective grasping force; for material blockage, the instructions may include controlling the robot to perform a slight vibration to shake off the material adhering to the wall, and simultaneously sending a natural language alert to the user: "Warning: Abnormal flow rate detected at the discharge port, suspected slight material agglomeration, preliminary handling has been attempted, it is recommended to observe the subsequent situation and check the material storage humidity." This achieves automated closed-loop management from perceiving abnormalities to analysis and decision-making, and then to executing recovery.

[0046] The vision sensor has been upgraded to a highly integrated multimodal sensing unit, which elevates the system's state perception capabilities to unprecedented dimensions and precision. This unit typically consists of a high-resolution 2D industrial camera and a 3D point cloud scanning device (such as structured light or laser profilometer), and uses a precise spatiotemporal synchronous registration mechanism to perform pixel-level fusion of captured 2D texture information (color, contrast) and 3D geometric data (depth, contour). In the scenario of unpacking bagged starch, as the robot grasps the bag and moves it to the unpacking station, this multimodal sensing unit simultaneously scans the target. The 2D camera provides rich surface texture information, which can be used to accurately identify the location of seams, printed marks, or soiled areas on the packaging bag; while the 3D scanner simultaneously acquires a precise 3D point cloud model of the bag, calculating its actual dimensions, volume, and precise pose relative to the robot (including spatial coordinates and rotation angle) in real time. This fusion of multi-source information enables the system to precisely locate the optimal entry point of the unpacking tool, and even if the bag deforms due to its softness after grasping, the system can adaptively adjust.

[0047] All this enriched multidimensional sensory data is continuously fed into the digital twin system. This twin is no longer a static geometric model, but a dynamic system capable of mapping the pose changes of physical entities and the morphology of material flow (such as a three-dimensional particle cloud formed by spilled starch) in real time. It constructs an extremely accurate and dynamic benchmark for behavioral pattern comparison. When a statistically significant deviation is detected between the fused multidimensional sensory data and the virtual expected state—for example, a 3D point cloud showing that the shape of the bag after grasping is consistently different from the standard model, or 2D texture analysis revealing an abnormally large amount of dust escaping from the cut—a high-level anomaly diagnosis process is triggered. A large language model then plays the role of a high-level diagnostic expert, performing correlation analysis on this massive amount of multi-source data: cross-modal matching and inference between the bag deformation data, real-time readings of the grasping force, and visual characteristics of dust escape, with records in a historical fault case library. It might diagnose that the root cause is "the abnormal flexibility of the packaging bag material in the current batch, resulting in excessive deformation under standard grasping force, which in turn affects the cutting positioning accuracy and causes dust escape due to the bag's violent rebound." Based on this in-depth diagnostics, the model generates highly precise composite adjustment commands: these include reconfiguration of equipment parameters (dynamically reducing the vacuum gripping force to a new value that allows for stable gripping without excessive deformation), online optimization of the motion trajectory (replanning a smoother movement trajectory to reduce deformation caused by inertia and fine-tuning the cutting path to compensate for deformation), and a fault self-recovery strategy (controlling the dust removal device to perform a powerful pulse cleaning to remove escaped dust after this unpacking is completed). This series of operations fully demonstrates the system's profound understanding, proactive adaptation, and self-recovery capabilities under complex and abnormal operating conditions.

[0048] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for unpacking and loading materials using an industrial robot that supports natural language commands, characterized in that, The method includes the following steps: S1. Receive unstructured natural language requirement information about the unpacking and loading task from the user through a natural language interaction interface. The requirement information includes at least one of the following: material type, packaging specifications, loading rate, and production line layout. S2. Using a large language model fine-tuned with expertise in the field of robot unpacking and loading, semantic analysis of the natural language requirement information is performed to perform intent recognition and key information extraction. If the requirement information is found to be missing or ambiguous, natural language prompts for follow-up questions are generated and returned to the user through the interactive interface. This step is repeated until a complete and clear set of task parameters is obtained. S3. In response to obtaining the complete set of task parameters, the pre-built domain knowledge base is invoked for matching and logical reasoning to generate the workstation configuration basis; the domain knowledge base includes robot parameter library, gripper adaptation library, material property library and unpacking process library. S4. Based on the configuration criteria, automatically select hardware configuration from multiple preset configuration templates and plan the workstation layout. At the same time, automatically generate the corresponding robot motion trajectory program, motion control instructions and system operating parameters to form a complete workstation configuration scheme. The motion control instructions include gripping force, unpacking tool angle and cutting depth. S5. Convert the motion trajectory program, motion control instructions and operating parameters in the workstation configuration scheme into control signals that can be recognized by the underlying actuator, and drive the robot, conveyor belt and unpacking tool to perform unpacking and loading operations in a coordinated manner. S6. Monitor the operating status of the actuator in real time. When an abnormality is detected, feed the abnormal status information back to the large language model. The large language model generates anomaly handling instructions, which include sending alarm prompts to the user through a natural language interaction interface or automatically adjusting the parameters of the workstation configuration scheme.

2. The method for unpacking and loading materials using an industrial robot supporting natural language commands as described in claim 1, characterized in that: In step S2, semantic parsing of natural language demand information specifically includes: using a finely tuned large-scale language model to perform deep semantic understanding and contextual analysis on the input text, identifying the domain operation category corresponding to the user's intent, and extracting key operation parameters from the statement based on predefined entity extraction rules; when generating natural language prompts for follow-up questions, the model dynamically constructs a multi-turn dialogue process based on the logical correlation of missing parameters, and constrains the professionalism and accuracy of the generated content through a domain knowledge base.

3. The method for unpacking and loading materials using an industrial robot supporting natural language commands as described in claim 2, characterized in that: The deep semantic understanding and contextual association analysis are based on a domain-adaptive semantic parsing algorithm, which focuses on material characteristic descriptions, packaging specifications, and process requirements through an attention mechanism. The entity extraction rules adopt a sequence labeling model based on a combination of bidirectional long short-term memory networks and conditional random fields, specifically for identifying and normalizing compound noun phrases in the unpacking and loading domain. The multi-turn dialogue process adopts a state machine-based dialogue management strategy, which dynamically generates a follow-up question sequence based on the logical dependencies between currently confirmed parameters and missing parameters. The generation of each follow-up question node is verified by the equipment parameter constraints and process feasibility rules in the domain knowledge base.

4. The method for unpacking and loading materials using an industrial robot supporting natural language commands as described in claim 3, characterized in that: The sequence labeling model uses a pre-trained language model based on the Transformer architecture as a feature extractor, and its output layer is connected to the CRF layer to jointly decode the optimal label sequence. The state machine node in the dialogue management strategy includes a parameter completeness verification module. The parameter completeness verification module dynamically evaluates the completeness of the current dialogue state by querying the equipment compatibility rules and process parameter constraints in the domain knowledge base.

5. The method for unpacking and loading materials using an industrial robot supporting natural language commands as described in claim 1, characterized in that: In step S3, the domain knowledge base is organized using a knowledge graph structure, in which the robot parameter library, gripper adaptation library, material property library, and unpacking process library are interconnected in the form of entity-relationship-attribute triples; the logical reasoning process is implemented based on a rule engine and graph retrieval algorithm, and by matching the physical properties and mechanical performance constraints of the materials, the equipment combination and operation parameters that meet the process requirements are inferred.

6. The method for unpacking and loading materials using an industrial robot supporting natural language commands as described in claim 1, characterized in that: In step S4, the robot motion trajectory program includes: calculating the optimal path of each joint of the robotic arm based on the workstation layout planning results and material physical properties, and introducing a collision detection algorithm to avoid interference with surrounding equipment; the generation of the motion control command integrates material morphological features and gripper mechanical model, and dynamically adapts to gripping strategy and tool operating parameters.

7. The method for unpacking and loading materials using an industrial robot supporting natural language commands as described in claim 1, characterized in that: In step S6, the operation of anomaly monitoring includes capturing the position and state of the actuator and the material flow pattern in real time through a visual sensor and comparing them with the expected state in the digital twin system. When the deviation exceeds the threshold, the abnormal handling process is triggered. The large language model performs root cause analysis based on the historical fault case library and generates adaptive adjustment instructions that include equipment parameter reconfiguration and motion trajectory optimization.

8. The method for unpacking and loading materials using an industrial robot supporting natural language commands as described in claim 7, characterized in that: The visual sensor is a multimodal sensing unit that integrates a high-resolution industrial camera and a 3D point cloud scanning device, and fuses 2D texture information and 3D geometric data through a spatiotemporal synchronous registration mechanism. The digital twin system constructs a dynamic behavior pattern comparison benchmark by mapping the pose deviation and material flow morphology changes of the physical entity and the virtual model in real time. When a statistically significant deviation is detected between the multi-dimensional perception data and the virtual expected state, an abnormal diagnosis process is triggered. The large language model performs root cause reasoning of the fault based on multi-source data correlation analysis and historical fault case library, and generates a composite adjustment instruction that includes equipment parameter reconfiguration, online optimization of motion trajectory and fault self-recovery strategy.

Images