Method and system for intelligent judgment of robot arm state based on multi-modal large language model

Abstract

The application discloses a mechanical arm state intelligent judgment method and system based on a multimodal large language model, and relates to the technical field of industrial state data processing. The method comprises the following steps: obtaining a task structure body; building a multimodal template prompt set; loading work area images and mechanical arm body state data, combining the multimodal template prompt set, and compiling multimodal large language model input; calling the multimodal large language model, processing the multimodal large language model input, and outputting a mechanical arm motion state judgment result; searching for historical benchmark motion states under the restriction of the task structure body, the work area images and the mechanical arm model; calculating the similarity between the historical benchmark motion states and the recommended motion states of the mechanical arm as reasoning confidence; when the reasoning confidence is greater than or equal to a confidence threshold value, mapping the mechanical arm motion state judgment result and the task structure body into an action primitive sequence to generate a control instruction. The application effectively improves the success rate and reliability of mechanical arm operation under complex instruction driving.

Description

Technical Field

[0001] This invention relates to the field of industrial status data processing technology, specifically to a method and system for intelligent judgment of robotic arm status based on a multimodal large language model. Background Technology

[0002] With the development of multimodal large language model technology, the field of robotic arm control has begun to explore the integration of visual and natural language understanding capabilities to improve the naturalness of human-computer interaction and task adaptability. In existing technologies, robotic arms mostly rely on pre-programmed process scripts or taught trajectories to perform tasks. Although they are stable and controllable in structured environments, they are difficult to directly understand and respond to complex unstructured natural language instructions.

[0003] However, existing technical solutions have obvious limitations: during the motion process, there is a lack of interpretable state consistency verification, making it difficult to cope with execution deviations caused by factors such as occlusion and target movement; in the positioning and operation phases, they often rely only on simple position errors or grasp point detection, making it difficult to comprehensively judge the end-effector posture, operation direction matching, and environmental interference risks; at the same time, there is a lack of a unified verification and safety gating mechanism throughout the entire process from instruction understanding to action execution, resulting in task execution disconnect, low fault tolerance, and safety risks due to the uncertainty of the multimodal large language model output, leading to insufficient success rate and reliability of robotic arm operations driven by complex instructions. Summary of the Invention

[0004] This invention provides a method and system for intelligent judgment of the state of a robotic arm based on a multimodal large language model, aiming to solve the technical problem of insufficient success rate and reliability of robotic arm operation driven by complex instructions in the prior art.

[0005] In view of the above problems, the present invention provides a method and system for intelligent judgment of the state of a robotic arm based on a multimodal large language model.

[0006] In a first aspect, the present invention provides a method for intelligently determining the state of a robotic arm based on a multimodal large language model, including:

[0007] Obtain a task structure, wherein the task structure includes a target object, a target region, an operation type, and constraints;

[0008] A multi-template prompt set is built for the task structure, wherein the multi-template prompt set includes a motion phase state verification template, an arrival phase position verification template, an attitude verification template, and an operation phase execution verification template;

[0009] When performing tasks, the system loads images of the work area and the status data of the robotic arm, and combines these with the multi-template prompt set to arrange the input of a multimodal large language model.

[0010] A multimodal large language model is invoked to process the input of the multimodal large language model and output the motion state determination result of the robotic arm, wherein the motion state determination result of the robotic arm includes the recommended motion state of the robotic arm;

[0011] Using the task structure, the working area image, and the robot arm model as constraints, retrieve the historical baseline motion state of the robot arm with the same constraints;

[0012] The similarity between the historical baseline motion state of the same-constraint robotic arm and the recommended motion state of the robotic arm is calculated as the inference confidence.

[0013] When the inference confidence is greater than or equal to the confidence threshold, the robotic arm motion state determination result and the task structure are mapped into a sequence of action primitives to generate control commands.

[0014] Secondly, this invention provides a robotic arm state intelligent judgment system based on a multimodal large language model, comprising:

[0015] The task structure acquisition module is used to obtain the task structure, wherein the task structure includes the target object, the target region, the operation type, and the constraints.

[0016] A multi-template prompt set building module is used to build a multi-template prompt set for the task structure, wherein the multi-template prompt set includes a motion phase state verification template, an arrival phase position verification template, an attitude verification template, and an operation phase execution verification template.

[0017] The multimodal input orchestration module is used to load the working area image and the robot arm body status data when performing work, and combine them with the multi-template prompt set to orchestrate the multimodal large language model input;

[0018] The output state determination module is used to call the multimodal large language model, process the input of the multimodal large language model, and output the motion state determination result of the robotic arm, wherein the motion state determination result of the robotic arm includes the recommended motion state of the robotic arm.

[0019] The historical baseline retrieval module is used to retrieve the historical baseline motion state of the robotic arm with the same restrictions as the task structure, the working area image and the robotic arm model.

[0020] The inference confidence calculation module is used to calculate the similarity between the historical baseline motion state of the same-constraint robotic arm and the recommended motion state of the robotic arm as the inference confidence.

[0021] The control instruction generation module is used to map the robotic arm motion state determination result and the task structure into a sequence of action primitives when the inference confidence is greater than or equal to the confidence threshold, and generate control instructions.

[0022] One or more technical solutions provided in this invention have at least the following technical effects or advantages:

[0023] This invention provides a method and system for intelligent judgment of robotic arm status based on a multimodal large language model. By decomposing the task into multiple stages and designing corresponding dedicated prompt templates, combining the fusion perception and reasoning capabilities of the multimodal large language model for real-time images and ontology data, and introducing a historical experience database for confidence verification and gating decision-making, it achieves phased and interpretable intelligent judgment of the robotic arm's motion state. Ultimately, it achieves the control goal of safe, reliable, and closed-loop execution of the robotic arm from natural language commands, effectively improving the success rate and reliability of robotic arm operations driven by complex commands. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 A flowchart illustrating the intelligent state judgment method for a robotic arm based on a multimodal large language model provided in an embodiment of the present invention;

[0026] Figure 2 A schematic diagram of the structure of the intelligent state judgment system for a robotic arm based on a multimodal large language model provided in an embodiment of the present invention;

[0027] The components represented by each number in the attached diagram are explained below:

[0028] Task structure acquisition module 11, multi-template hint set construction module 12, multimodal input orchestration module 13, output status determination module 14, historical benchmark retrieval module 15, inference confidence calculation module 16, and control command generation module 17. Detailed Implementation

[0029] This invention provides a method and system for intelligent judgment of the state of a robotic arm based on a multimodal large language model, which is used to address the technical problem of insufficient success rate and reliability of robotic arm operation driven by complex instructions in the prior art.

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0031] It should be noted that the terms "comprising" and "having" are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or modules that are not explicitly listed or that are inherent to these processes, methods, products, or devices.

[0032] Example 1, as Figure 1 As shown, this invention provides a method for intelligent judgment of the state of a robotic arm based on a multimodal large language model, the method comprising:

[0033] S100: Obtain the task structure, wherein the task structure includes the target object, the target region, the operation type, and the constraints.

[0034] In this embodiment of the invention, a task structure is obtained, wherein the task structure includes a target object, a target area, an operation type, and constraints. In practical scenarios such as warehouse sorting and assembly, users typically issue tasks via natural language commands. Natural language commands are characterized by their unstructured nature and flexible expression. In existing technologies, the parsing of natural language commands by robotic arms largely relies on simple rule matching, making it difficult to accurately extract elements such as the target object and operation type. Furthermore, it is susceptible to parsing deviations due to ambiguous expressions and scene differences, leading to subsequent operation failures. In addition, traditional parsing methods lack a unified structured output standard, failing to provide standardized input for subsequent multi-template prompt set construction and phased state determination. Therefore, it is necessary to train a specialized task structure parsing model to transform unstructured natural language commands into structured task structures. Simultaneously, by utilizing binary encoding and the Hamming distance loss function, parsing speed and accuracy are balanced, laying the foundation for closed-loop control throughout the entire process.

[0035] Step S100 in the method provided in this embodiment of the invention includes:

[0036] The process of obtaining the task structure includes, before:

[0037] Collect a set of historical input statements from users and a set of labels for identifying task structures, and set them as the user intent recognition training dataset. In the set of labels for identifying task structures, any label for identifying a task structure is composed of binary codes for the target object description entity, binary codes for the target region description entity, binary codes for the operation type description entity, and binary codes for the constraint condition description entity.

[0038] A task structure label mapping layer is built and connected to the output layer of the Transformer architecture to obtain the task structure parsing model training architecture. The output task of the Transformer architecture is to identify the user's historical input statement and output the task structure description entity.

[0039] A training loss function is constructed based on Hamming distance, wherein the training loss function is positively correlated with the Hamming distance;

[0040] The task structure parsing model is trained using the user's historical input statement set as input and the label set of the task structure identifier as supervision. After convergence, the task structure label mapping layer is deleted to obtain the task structure parsing model.

[0041] The task structure is obtained by processing user input statements using the task structure parsing model.

[0042] First, a set of historical user input statements and a set of labels identifying task structures are collected and designated as the user intent recognition training dataset. Each label in the task structure label set consists of binary codes representing the target object, target region, operation type, and constraint conditions. The historical user input statement set refers to the collection of natural language instructions given by the user to operate the robotic arm in past scenarios, serving as the input sample source for training the task structure parsing model. The task structure label set is a set of structured labels corresponding one-to-one with historical input statements. Each label consists of four types of binary codes used to annotate the task elements in the statement. Binary encoding converts the descriptive entities of task elements into fixed-length 0-1 strings for rapid calculation and matching.

[0043] Specifically, historical user input statements are collected from multiple scenarios, covering operation types such as grabbing, placing, pushing, pulling, and rotating, forming a set of historical user input statements. For each historical input statement, four types of descriptive entities are extracted: target object, target area, operation type, and constraint conditions. A binary encoding rule is formulated for each type of descriptive entity, such as using 8-bit binary, with different entities assigned a unique code. The codes of the four types of entities are combined to form the label corresponding to the historical input statement, forming a label set that identifies the task structure. The set of historical user input statements and the label set that identifies the task structure are combined to form a user intent recognition training dataset.

[0044] For example, a total of 10,000 user intent recognition training data were collected. One user's historical input statement was: "Grab the blue plastic part on the third shelf and place it on the left assembly table. The height during the translation process should not be less than 50cm." The extracted descriptive entities are: target object = blue plastic part, target area = left assembly table, operation type = grab + place, and constraint condition = translation height ≥ 50cm. The binary encoding rules are as follows: target object: blue plastic part = 00101100, target area: left assembly table = 01010011, operation type: grab + place = 10011000, constraint condition: translation height ≥ 50cm = 11000101. The corresponding label is: 00101100010100111001100011000101, with a total length of 32 bits, concatenated in the order of target object, target area, operation type, and constraint condition.

[0045] Secondly, a task structure label mapping layer is constructed and connected to the output layer of the Transformer architecture to obtain the task structure parsing model training architecture. The output task of the Transformer architecture is to identify the task structure description entities from the user's historical input statements. The task structure label mapping layer is a dedicated layer connected to the output of the Transformer architecture, used to convert the task description entities identified by the Transformer into binary encoded labels consistent with the label set format. The Transformer architecture is a natural language processing model based on a self-attention mechanism. Its function is to extract semantic features from input statements and identify task description entities such as target objects and operation types. Task structure description entities refer to the specific descriptions of the target objects, target regions, operation types, and constraints that constitute the task structure, parsed from the user's statements.

[0046] For example, a pre-trained Transformer model such as BERT-base is selected as the basic architecture, with the following specific configuration: the number of encoder layers is set to 6, the number of heads in the multi-head attention mechanism of each layer is 12, the number of neurons in the hidden layer is 768, the number of neurons in the intermediate layer of the Feed-Forward network is 3072, the activation function is GELU (Gaussian error linear unit), and the dropout probability is set to 0.1 to balance the model's fitting ability and generalization ability; the task structure label mapping layer is a fully connected layer, with the input dimension consistent with the output dimension of the Transformer architecture, and the output dimension is 32-dimensional, corresponding to the total length of the four types of entity binary codes concatenated, and the activation function is Sigmoid to ensure that the output value is mapped to the [0,1] interval, which is suitable for the probability prediction requirements of binary labels.

[0047] Furthermore, a training loss function is constructed based on the Hamming distance, wherein the training loss function is positively correlated with the Hamming distance. The Hamming distance D refers to the number of bits that differ at corresponding positions in two binary strings of equal length, used to measure the degree of difference between two binary labels. The training loss function is used in model training to calculate the difference between the predicted result and the true label, and serves as the basis for optimizing model parameters. The training loss function is constructed such that the loss value is positively correlated with the Hamming distance between the predicted label and the true label of the task structure parsing model. For example, the loss value... , where α is the scaling factor, P is the predicted label, and T is the true label; by leveraging the fast binary calculation and accurate difference measurement of Hamming distance, the loss calculation speed during model training is accelerated, while accurately reflecting the label matching bias.

[0048] Subsequently, using the user's historical input statement set as input and the label set identifying the task structure as supervision, the task structure parsing model training architecture is trained based on the training loss function. After convergence, the task structure label mapping layer is deleted, obtaining the task structure parsing model. Model convergence refers to the loss function value continuously decreasing and stabilizing during training, at which point the model's prediction accuracy reaches its optimal state. Deleting the label mapping layer means that after training, the label mapping layer has fulfilled its purpose; deleting this layer preserves the Transformer architecture's ability to identify and structure the task description entities, simplifying the model structure.

[0049] Specifically, the user intent recognition training dataset is first preprocessed using BPE segmentation, fixed-length padding and truncation, and word embedding vector transformation. It is then divided into training and validation sets in a 7:3 ratio. The preprocessed training set is input into the model training architecture in batches, using a 32-bit binary label set identifying the task structure as a supervision signal. Based on a loss function combining Hamming distance and cross-entropy, the AdamW optimizer iteratively optimizes the model parameters. After each training epoch, the model performance is evaluated using the validation set. The task structure parsing model is considered converged when the validation set loss value fluctuates less than 0.001 and below 0.05 for 10 consecutive epochs, the overall parsing accuracy is ≥95%, the constraint parsing accuracy is ≥93%, and the difference between the training and validation set accuracy is <3%. The task structure parsing model is then removed from the converged model, retaining the core functionality of the Transformer architecture to obtain the final task structure parsing model.

[0050] For example, 10,000 historical statements of robotic arm operations in multiple scenarios and their corresponding 32-bit binary tags were preprocessed by word segmentation and vectorization, and divided into a 7:3 ratio to obtain 7,000 training sets and 3,000 validation sets. Training process: AdamW was used as the optimizer for 20 iterations. The performance was evaluated using the validation set in each iteration. The model loss value continued to decrease and the parsing accuracy steadily improved. Convergence condition: After 20 iterations, the validation set loss value stabilized at 0.04 and the overall parsing accuracy stabilized at 96%, satisfying all convergence conditions. The task structure label mapping layer was then removed to obtain the trained task structure parsing model.

[0051] Finally, the task structure parsing model is used to process user input statements to obtain the task structure. User input statements refer to the natural language instructions given by the user for robotic arm operations in practical applications, serving as the real-time input to the task structure parsing model. The task structure refers to the output of the task structure parsing model, which is structured data containing four elements: target object, target region, operation type, and constraints, providing standardized input for subsequent steps. The model receives natural language instructions from the user, in plain text strings with no fixed syntax requirements; these instructions are then input into the trained task structure parsing model; the model performs semantic understanding and core element extraction on the instructions, outputting a structured task structure in key-value pairs or JSON format, with fixed fields: obj, roi, op, and constraints.

[0052] For example, the user inputs the natural language command "pick up the red box on the left and put it on the right tray"; the task structure parsing model extracts the target object "red box", the target area "right tray", the operation type "pick up and place", and the constraint condition "no special constraints" from the command; the output task structure is: red box, right tray, pick up and place, no special constraints.

[0053] In this embodiment of the invention, through a process of dataset construction, model architecture building, loss function design, model training, and real-time parsing, accurate and rapid conversion of unstructured natural language instructions into structured task structures is achieved. The model based on the Transformer architecture possesses powerful semantic understanding capabilities. Combining binary encoding and the Hamming distance loss function ensures both parsing speed and accuracy. The output standardized task structure provides a unified input basis for subsequent S200 multi-template prompt set construction and phased state determination, solving the problems of ambiguous instruction understanding and inconsistent input formats in existing technologies. Simultaneously, the task structure parsing model adapts to various operation types such as grasping, pushing, pulling, and placing, meeting the instruction parsing needs of different scenarios and providing reliable instruction understanding support for the entire closed-loop control of the robotic arm.

[0054] S200: Build a multi-template prompt set for the task structure, wherein the multi-template prompt set includes a motion phase state verification template, an arrival phase position verification template, an attitude verification template, and an operation phase execution verification template.

[0055] In this embodiment of the invention, a multi-template prompt set is built for the task structure. This multi-template prompt set includes a motion phase state verification template, a positioning phase position verification template, a posture verification template, and an operation phase execution verification template. During the robotic arm's task execution, the verification objectives differ significantly across stages: the motion phase requires attention to trajectory and speed compliance; the positioning phase requires focusing on positional deviations; the posture phase requires verifying the joint and operating surface matching degree; and the operation phase requires monitoring the gripper's execution status. In existing technologies, multimodal large language models (MLMs) often input undifferentiated mixed information, leading to blurred focus and significant interference from irrelevant information during model judgment. This makes it difficult to accurately respond to the verification requirements of each stage, easily resulting in judgment bias or low efficiency. Furthermore, the lack of structured prompt templates leads to inconsistent output formats from MLMs, making them unsuitable for subsequent state verification and stage switching logic. Therefore, based on the core elements of the task structure, a targeted multi-template prompt set needs to be constructed according to the execution stage, clearly defining the input information type and structure of each template. This allows the MLM to focus on key parameters at different stages, improving the accuracy and efficiency of state judgment.

[0056] Step S200 in the method provided in this embodiment of the invention includes:

[0057] The motion phase status verification template consists of a working area image, a robotic arm motion parameter table, a task structure summary, and the judgment result of the previous cycle.

[0058] The positioning verification template for the arrival stage consists of a working area image, a robotic arm position parameter table, a task structure summary, and the judgment result of the previous cycle.

[0059] The attitude verification template consists of a working area image, a robotic arm joint parameter table, a task structure summary, and the judgment result of the previous cycle.

[0060] The verification template for the operation phase consists of a working area image, a gripper status parameter table, a task structure summary, and the judgment result of the previous cycle.

[0061] First, a motion phase status verification template is constructed. This template consists of a working area image, a robotic arm motion parameter table, a task structure summary, and the previous cycle's judgment result. The motion phase status verification template is a dedicated prompt template designed for the robotic arm's movement towards the target area, used to verify trajectory compliance, motion parameter rationality, and constraint fulfillment. The robotic arm motion parameters are a set of key dynamic parameters during the robotic arm's movement, focusing on movement-related indicators to determine whether movement is along the correct path. The key points of motion phase verification are clearly defined: trajectory deviation, motion speed / acceleration, constraint fulfillment, and target object visibility; core motion parameters are selected: trajectory deviation, end effector speed, end effector acceleration, current height, and target object visibility, specifying the units and constraint thresholds for each parameter; the motion phase status verification template is formed by combining the working area image, robotic arm motion parameters, task structure summary, and previous cycle's judgment result in that order, with the motion parameters synchronized with the robotic arm's motion data in real time.

[0062] For example, the complete structure of the motion phase status verification template is as follows: Work area image: Real-time operation image of the shelf from the third layer to the left assembly station captured by an RGBD camera, with blue plastic parts clearly visible; Robotic arm motion parameters: Trajectory deviation is 3.2mm, constraint threshold ≤ 5mm; End-effector movement speed is 0.2m / s, constraint threshold ≤ 0.3m / s; End-effector acceleration is 0.1m / s². 2 The constraint threshold is ≤0.5m / s. 2 The current height is 65cm, which meets the core constraint requirement of translation height ≥ 50cm; the target object visibility is 98%, and the constraint threshold is ≥ 80%; Task structure summary: Target object: blue plastic part; Operation type: grab + place; Target area: left assembly table; Constraint condition: translation height ≥ 50cm; Previous cycle judgment result: stage not started, status normal, confidence level 1.0, judgment reason is no abnormality in the initial state.

[0063] Secondly, a position verification template for the arrival stage is established. This template consists of a working area image, a robotic arm position parameter table, a task structure summary, and the judgment result from the previous cycle. This template is specifically designed for the robotic arm approaching the target area and preparing for attitude adjustment, verifying whether the positional relationship between the end effector and the target area meets the operational prerequisites. The robotic arm position parameters are a collection of position-related parameters of the end effector, focusing on static position indicators such as position coordinates and distance deviation to support position compliance judgment. The verification focus for the arrival stage is clearly defined: the three-dimensional coordinates of the end effector, the straight-line distance to the target object, the positioning accuracy within the target area, and obstacle interference. Core position parameters are selected: the XYZ coordinates of the end effector, the distance between the end effector and the target object, the positioning accuracy of the target area, and the distance to obstacles. The units, target values, and deviation thresholds for each parameter are clearly defined. The template is formed by combining the working area image, robotic arm position parameters, task structure summary, and the judgment result from the previous cycle in that order. The position parameters are collected in real-time by the robotic arm's body status acquisition unit.

[0064] For example, the complete structure of the position verification template for the arrival stage is as follows: Work area image: a close-up image of the third shelf where the blue plastic part is located, with the end effector approaching the target object; Robotic arm position parameters: end effector X coordinate is 1250mm, target value is 1248mm, deviation threshold ±5mm; end effector Y coordinate is 860mm, target value is 862mm, deviation threshold ±5mm; end effector Z coordinate is 620mm, target value is 620mm, deviation threshold ±3mm; distance between end effector and target object is 15mm, requirement ≤20mm; target area positioning accuracy is 2.1mm, requirement ≤3mm; obstacle distance is 35mm, requirement ≥20mm; Task structure summary: target object: blue plastic part; operation type: gripping + placement; target area: left assembly table; constraint condition: translation height ≥50cm; previous cycle judgment result: stage not started, status normal, confidence level 1.0, judgment reason: no abnormality in the initial state.

[0065] Next, an attitude verification template is constructed. This template consists of a working area image, a robotic arm joint parameter table, a task structure summary, and the previous cycle's judgment results. The attitude verification template is a dedicated template designed for adjusting the end effector's attitude to adapt to the operation phase, used to verify the operational adaptability of the end effector's attitude to the target object. The robotic arm joint parameters are a collection of parameters such as the joint angles and attitude matrix of the robotic arm, used to accurately characterize the spatial attitude of the end effector and support attitude adaptability judgment. The focus of the attitude phase verification is clearly defined: the angles of each joint, the end effector attitude matrix, the alignment of the gripper opening direction with the target object's main axis, and the fit of the operating surface; core joint parameters are selected: the angles of the core joints, the target angle, the deviation threshold, as well as the end effector roll / pitch / yaw angles, and the alignment of the gripper with the target, with the units and acceptance standards for each parameter clearly defined; the attitude verification template is formed by combining the working area image, robotic arm joint parameters, task structure summary, and the previous cycle's judgment results in that order, with the joint parameters being collected in real time by the robotic arm's body state acquisition unit.

[0066] For example, the complete structure of the attitude verification template is as follows: Work area image: Real-time image of the end effector aligned with the blue plastic part, with the gripper opening direction parallel to the long side of the part; The robotic arm is a 6-DOF robotic arm, with the following joint parameters: Joint 1 angle is 35.2°, target angle is 35.0°, deviation threshold is ±2°; Joint 2 angle is 120.5°, target angle is 121.0°, deviation threshold is ±2°; ...; Joint 6 angle is 15.3°, target angle is 15.0°, deviation threshold is ±1°; End effector roll angle is 0.5°, pitch angle is 1.2°, yaw angle is 3.1°; The alignment between the gripper and the target object is 96%; Task structure summary: Target object: Blue plastic part; Operation type: Grab + Place; Target area: Left assembly table; Constraints: Translation height ≥ 50cm; Previous cycle judgment result: Stage is in position, status is normal, confidence level is 0.96, the judgment reason is that the end effector has entered the target area, the position deviation meets the requirements, and it can be switched to attitude verification.

[0067] Finally, an operation phase execution verification template is constructed. This template consists of a work area image, a gripper status parameter table, a task structure summary, and the judgment results from the previous cycle. This template is specifically designed for the robotic arm's gripping / placement operations, verifying the execution effectiveness and gripper status. Gripper status parameters are a collection of operating status parameters of the robotic arm's end effector gripper, focusing on gripper indicators directly related to operation execution to support the judgment of operation effectiveness. The key verification points for the operation phase are clearly defined: gripper opening degree, gripping force, gripping stability, and operation execution progress; core gripper status parameters are selected: gripper opening degree, gripping force, gripping duration, and slip detection results, specifying the units, target values, and acceptable ranges for each parameter; the operation phase execution verification template is formed by combining the work area image, gripper status parameters, task structure summary, and the judgment results from the previous cycle in that order. Gripper parameters are collected in real-time by gripper status sensors.

[0068] For example, the complete structure of the operation phase execution verification template is as follows: Work area image: Real-time image of the gripper holding a blue plastic part, with no displacement or detachment of the part; Gripper status parameters: Gripper opening degree is 28mm, target value is 30mm, acceptable range is 25-32mm; Gripping force is 8.5N, target value is 8-10N, and the acceptable range is consistent with the target value; Gripping duration is 3s, requiring ≥2s; Slip detection result is no slippage, and the acceptable standard is no slippage; Task structure summary: Target object: Blue plastic part; Operation type: Grip + Place; Target area: Left assembly table; Constraints: Translation height ≥50cm; Previous cycle judgment result: The stage is the posture stage, the state is normal, the confidence level is 0.98, and the judgment reason is that the end posture is aligned with the target, there is no risk of interference, and the gripping operation can be performed.

[0069] In this embodiment of the invention, a multi-template prompt set covering the entire process is constructed to achieve precise adaptation to the verification requirements of different execution stages. Each template focuses on the key parameters of the corresponding stage, eliminating irrelevant information interference, allowing the multimodal large language model to quickly lock onto the verification focus, improving the accuracy and efficiency of state determination. The standardized template structure ensures the structured and consistent input of the multimodal large language model, avoiding chaotic output formats and providing a reliable input foundation for subsequent staged state determination and stage switching. At the same time, the templates are deeply bound to the task structure, and the parameters are synchronized with the device status in real time, ensuring the relevance and real-time nature of the verification, providing key support for the closed-loop control of the robotic arm throughout the entire process.

[0070] S300: When performing work, load the work area image and the robot arm body status data, and combine them with the multi-template prompt set to arrange the multimodal large language model input.

[0071] In this embodiment of the invention, when performing work, the working area image and the robotic arm's body state data are loaded, and combined with the multi-template prompt set, a multimodal large language model input is arranged. The robotic arm's body state data refers to real-time operational data collected by the robotic arm's built-in sensors, covering core parameters such as joint angles, end-effector pose, gripper state, and movement speed, reflecting the robotic arm's current operating state. The multimodal large language model input refers to model input data composed of images and structured data combined in a preset format, possessing both visual information and numerical parameter features, adapting to the multimodal understanding capabilities of the multimodal large language model.

[0072] First, the real-time working area image and the robot arm's status data are synchronously loaded through the multimodal perception module, with a sampling frequency of 100Hz, and stored in a structured format according to parameter name, unit, and current value. Second, based on the current execution stage of the robot arm, the corresponding exclusive verification template is matched from the multi-template prompt set. Finally, the input is arranged in a fixed order of working area image, structured ontology status data, task structure summary, and previous cycle judgment result. The image data is embedded in tensor form, and the structured data is presented in a standardized text description to ensure that the input format is compatible with the multimodal large language model interface and that there is no information redundancy or missing information.

[0073] For example, taking the motion phase of "grabbing the blue plastic part on the third shelf and placing it on the left assembly table" as an example: the loaded work area image is a real-time dynamic picture of the shelf to the assembly table captured by an RGBD camera, clearly showing the position of the blue plastic part, the absence of obvious obstacles, and the motion trajectory of the robotic arm's end effector; the loaded robotic arm body status data includes structured parameters such as the real-time angles of the 6 joints, the end effector movement speed of 0.2m / s, the current height of 65cm, and the visibility of the target object of 98%; combined with the matched motion phase status verification template, the multimodal large language model input is arranged in a preset order: first, the work area image tensor is embedded, and then the robotic arm body status data, task structure summary, and the judgment result of the previous cycle are presented in text form in sequence, forming logically coherent and standardized multimodal large language model input data.

[0074] In this embodiment of the invention, standardized multimodal input orchestration solves the problems of messy model input information, inconsistent formats, and disconnect between visual and numerical data in existing technologies. Synchronously loaded real-time image and ontology state data provide the multimodal large language model with complete scene background and device operating status information, avoiding the bias caused by single data types. Targeted orchestration using templates specific to the current stage ensures that input information focuses on the core verification indicators of that stage, reducing interference from irrelevant information and improving the understanding efficiency and judgment accuracy of the multimodal large language model. Simultaneously, the standardized input format adapts to the processing logic of the multimodal large language model, ensuring that the model can quickly parse and associate different types of data, providing a clear and complete input foundation for subsequent staged state judgments, and guaranteeing the consistency and reliability of the entire judgment process.

[0075] S400: Call the multimodal large language model, process the input of the multimodal large language model, and output the motion state determination result of the robotic arm, wherein the motion state determination result of the robotic arm includes the recommended motion state of the robotic arm.

[0076] In this embodiment of the invention, a multimodal large language model is invoked, its input is processed, and a robotic arm motion state determination result is output. This result includes a recommended motion state for the robotic arm. The determination targets differ significantly across different stages of the robotic arm's execution. In existing technologies, multimodal large language models are often invoked indiscriminately, failing to match input and determination logic to stage characteristics. This results in poor model output focus, excessive interference from irrelevant information, and inconsistent decision result formats across different stages, making it difficult to directly support subsequent stage switching and action gating. Furthermore, the lack of a clear recommended motion state output leads to a lack of clear basis for subsequent control command generation, easily causing discontinuities in action transitions. Therefore, it is necessary to invoke the multimodal large language model step-by-step according to the execution stage, combining it with the corresponding stage's dedicated template input to output structured, stage-adapted determination results and recommended motion states, ensuring both accuracy and process continuity.

[0077] Step S400 in the method provided in this embodiment of the invention includes:

[0078] When the work is in motion, the multimodal large language model is invoked to process the motion phase state verification template input and obtain the motion phase robotic arm trajectory determination result.

[0079] When the work is in place, the multimodal large language model is invoked to process the in-place stage position verification template input and obtain the in-place stage end position determination result;

[0080] When the work is in the posture verification stage, the multimodal large language model is called to process the posture verification template input and obtain the robot arm joint judgment result in the posture verification stage.

[0081] When the work is in the operation phase, the multimodal large language model is invoked to process the operation phase execution verification template input and obtain the operation phase gripper state determination result.

[0082] The determination result of the robotic arm's motion state is one of the following: the determination result of the robotic arm trajectory during the motion phase, the determination result of the end position during the positioning phase, the determination result of the robotic arm joint during the posture verification phase, and the determination result of the gripper state during the operation phase.

[0083] First, when the operation is in the motion phase, the multimodal large language model is invoked to process the motion phase state verification template input, obtaining the motion phase trajectory determination result of the robotic arm. The motion phase trajectory determination result is structured determination information output during the robotic arm's movement process, including phase identifier, state label, confidence level, determination reason, and recommended motion state, used to clarify whether the trajectory is compliant and the direction of subsequent actions. The recommended motion state refers to the subsequent motion suggestions given by the multimodal large language model based on the current state, such as continue, slow down, or replan, providing a direct basis for the generation of control commands.

[0084] Specifically, the current execution stage of the robotic arm is identified as the motion stage. The motion stage state verification template constructed by S200 and the corresponding multimodal input data arranged by S300 are automatically matched. A pre-trained multimodal large language model, such as an improved model based on LLaVA, is called. The inference temperature is set to 0.1, and the output format is constrained to be structured text, including stage, state, confidence, reason, and recommended motion state. The multimodal large language model performs joint analysis on the input working area image, robotic arm motion parameters, and other information to determine whether the trajectory deviation, speed, height, etc., meet the constraints, and generates the motion stage trajectory determination result.

[0085] For example, based on the motion phase scenario of "grabbing the blue plastic part on the third shelf and placing it on the left assembly table": the multimodal input corresponding to the motion phase status verification template is called: the work area image shows that the part is clearly visible, the robot arm motion parameters are trajectory deviation of 3.2mm, end-effector speed of 0.2m / s, current height of 65cm, etc.; the output motion phase trajectory judgment result is: the phase is the motion phase, the status is normal, the confidence level is 0.97, the judgment reason is that the trajectory deviation is within the constraint threshold of ≤5mm, the end-effector speed does not exceed the safety threshold of 0.3m / s, the current height meets the constraint condition of ≥50cm, and the target object visibility is 98% as required; the recommended motion state is to continue moving at a constant speed along the current trajectory, and keep the end-effector height not less than 50cm.

[0086] Secondly, when the work is in the positioning stage, the multimodal large language model is invoked to process the positioning verification template input for the positioning stage, and the positioning stage end position determination result is obtained. The positioning stage end position determination result is a structured determination information output when the robotic arm approaches the target area, focusing on the positional relationship between the end and the target object, positioning accuracy, and obstacle conditions, supporting the decision on whether to enter the attitude verification stage.

[0087] Specifically, when the motion phase determination result suggests switching to the positioning determination or the distance between the robotic arm end effector and the target area is less than a preset trigger threshold, the current phase is identified as the positioning phase; the positioning phase position verification template constructed by S200 and the corresponding multimodal input data arranged by S300 are matched, and the multimodal large language model is called to keep the inference parameters consistent; the multimodal large language model combines the working area image and position parameters to determine whether the positioning condition is met, and outputs the determination result containing the recommended motion state.

[0088] For example, continuing the above task's arrival stage scenario: The multimodal input corresponding to the arrival stage position verification template is called: the working area image shows the end point is close to the blue plastic part, the position parameters are end point X coordinate 1250mm, Y coordinate 860mm, Z coordinate 620mm, all deviations from the target coordinates are within the threshold, the distance between the end point and the part is 15mm, the positioning accuracy is 2.1mm, and the obstacle distance is 35mm; the output of the arrival stage end point position determination result is: the stage is the arrival stage, the state is normal, the confidence level is 0.96, the determination reason is that the end point's three-dimensional coordinate deviation meets the threshold requirement of ±5mm / ±3mm, the distance to the target object is 15mm, which meets the arrival standard of ≤20mm, the positioning accuracy is 2.1mm≤3mm, and there is no risk of obstacle interference; the recommended motion state is to stop moving and switch to the attitude verification stage to prepare for adjusting the end point attitude.

[0089] Secondly, when the operation is in the posture verification stage, the multimodal large language model is invoked to process the posture verification template input and obtain the joint judgment results of the robotic arm in the posture verification stage. The joint adaptation judgment results in the posture verification stage are structured judgment information output when the robotic arm adjusts its end effector posture, verifying the operational adaptability of the joint angles, end effector posture and the target object, and determining whether the conditions for performing the operation are met.

[0090] Specifically, after receiving the recommended motion state for switching from the arrival stage to the posture verification stage, the current stage is identified as the posture verification stage; the posture verification template constructed by S200 and the corresponding multimodal input data arranged by S300 are matched, and the multimodal large language model is called; the multimodal large language model jointly analyzes information such as joint angle deviation, alignment between gripper and target object, and operating surface fit, determines whether the posture is suitable, and outputs the result containing the recommended motion state.

[0091] For example, continuing the attitude verification stage scenario of the above task: calling the multimodal input corresponding to the attitude verification template: the working area image shows that the opening direction of the gripper is parallel to the long side of the blue plastic part, the joint parameters are that the deviations of the 6 core joint angles from the target angle are all within the threshold, the end effector roll angle is 0.5°, the pitch angle is 1.2°, the yaw angle is 3.1°, and the alignment is 96%; output the joint adaptation judgment result of the attitude verification stage: the stage is the attitude verification stage, the state is normal, the confidence level is 0.98, the judgment reason is that the deviation of each joint angle does not exceed the threshold of ±1°-±2°, the end effector attitude parameters meet the operation requirements, the alignment of the gripper and the part is 96%≥90% of the qualified standard, and there is no risk of interference; the recommended motion state is to maintain the current attitude, start the gripper opening and closing preparation, and execute the grasping operation.

[0092] Furthermore, when the operation is in the operation phase, the multimodal large language model is invoked to process the operation phase execution verification template input and obtain the gripper state determination result for the operation phase. The gripper state determination result for the operation phase refers to the structured judgment information output by the robotic arm when performing operations such as grasping and placing, which monitors the gripper's operating status and operation execution effect, and supports decisions on whether to continue, fine-tune, or terminate the operation.

[0093] Specifically, after receiving the recommended motion state for the operation to be performed during the posture verification stage, the current stage is identified as the operation stage; the operation stage execution verification template constructed by S200 and the corresponding multimodal input data arranged by S300 are matched, and the multimodal large language model is called; the multimodal large language model parses the gripper opening degree, gripping force, gripping stability and the state of the target object in the image, determines whether the operation is successful, and outputs the result containing the recommended motion state.

[0094] For example, continuing the above task's grasping operation phase scenario: The multimodal input corresponding to the operation phase execution verification template is called: the work area image shows the gripper holding the blue plastic part without offset or detachment; the gripper status parameters are: opening degree 28mm, clamping force 8.5N, clamping duration 3s, and no slippage detection; the output is the gripper status judgment result for the operation phase: the phase is the operation phase, the status is normal, the confidence level is 0.99, the judgment reason is that the gripper opening degree of 28mm is within the acceptable range of 25-32mm, the clamping force of 8.5N meets the target requirement of 8-10N, the clamping duration of 3s≥2s, and there is no slippage; the recommended motion state is to maintain stable clamping force and move to the left assembly table according to the preset trajectory, with a height not less than 50cm during the movement.

[0095] In this embodiment of the invention, the precise determination of the robotic arm's entire process state is achieved through a phased template matching and targeted model invocation design. Targeted input information based on the core requirements of each stage allows the multimodal large language model to focus on key verification points, reducing interference from irrelevant information and effectively lowering the risk of misjudgment. The standardized structured output not only unifies the result format but also provides a clear basis for subsequent stage switching and action primitive generation, solving the problem of the disconnect between judgment results and control logic in existing technologies. Simultaneously, the phased invocation mode adapts to the real-time execution rhythm of the robotic arm, meeting real-time control requirements.

[0096] S500: Using the task structure, the working area image, and the robot arm model as constraints, retrieve the historical baseline motion state of the robot arm that is subject to the same constraints.

[0097] In this embodiment of the invention, the historical baseline motion states of the robotic arm are retrieved based on the constraints of the task structure, the working area image, and the robotic arm model. While the recommended motion states output by the multimodal large language model are based on real-time data, they exhibit a degree of randomness and uncertainty, and lack historical scenario reference support. Directly using them for control may lead to operational deviations. In existing technologies, historical motion data is often indiscriminately accessed without precise matching to the current task, robotic arm model, and working environment, resulting in low reference value. Furthermore, historical data contains discrete abnormal parameters that interfere with the accuracy of the baseline state and affect the reliability of subsequent confidence verification. Therefore, a comprehensive process involving multi-dimensional constraint retrieval, image similarity filtering, and discrete parameter removal is needed to extract baseline motion states highly adapted to the current scenario from historical data, providing accurate and reliable reference for the confidence verification of recommended motion states.

[0098] Step S500 in the method provided in this embodiment of the invention includes:

[0099] Using the task structure and the robotic arm model as primary constraints, an anomaly sample retrieval is performed to obtain a set of historical motion states of robotic arms with the same primary constraints. Each historical motion state of a robotic arm with the same primary constraints in the set of historical motion states of robotic arms with the same primary constraints has an associated stored working area recording image.

[0100] Based on the working area image, the image similarity set is compared by traversing the working area image set to obtain the image similarity set;

[0101] Based on the image similarity set, a second-level set of historical motion states of the same-restricted robotic arm with image similarity greater than or equal to the image similarity threshold is selected from the first-level set of historical motion states of the same-restricted robotic arm.

[0102] Delete the discrete parameters at each moment in the historical motion state set of the second-level same-constraint robotic arm, and construct the historical reference motion state of the same-constraint robotic arm using the remaining parameters.

[0103] First, using the task structure and the robotic arm model as primary constraints, anomaly-free sample retrieval is performed to obtain a set of historical motion states of robotic arms with the same primary constraints. Each historical motion state of a robotic arm with the same primary constraints in this set has an associated stored image of its working area. Primary constraints refer to using both the task structure and the robotic arm model as dual filtering conditions to ensure that the retrieved historical data is consistent with the current task and equipment type. Anomaly-free samples are those historical motion states that do not contain abnormal records such as collisions, slippage, or execution failures, and whose parameters are all within safe threshold ranges. The set of historical motion states of robotic arms with the same primary constraints is the collection of historical motion states obtained after primary constraint filtering; each sample is associated with a corresponding image of its working area for subsequent environmental matching.

[0104] Specifically, the task structure and the model of the robotic arm used in the current task are extracted; the historical motion state database of the robotic arm is accessed, and the filtering conditions are set: the task elements are consistent with the current task structure, the robotic arm model is the same, and there are no abnormal records in the motion process; the search operation is performed to collect all historical motion states that meet the conditions, forming a set of historical motion states of the robotic arm with the same first-level constraint, and at the same time, the working area record images corresponding to each sample are stored in association.

[0105] For example, the current task structure is as follows: target object: blue plastic part, operation type: gripping + placement, constraint condition: translation height ≥ 50cm, robotic arm model is CR5. After accessing the database, all historical motion states that performed gripping and placement operations on the blue plastic part, with a constraint height ≥ 50cm, using a CR5 robotic arm, and without collision / slippage anomalies are filtered out, totaling 120 groups; each group of states is associated with a corresponding work area recorded image, ultimately forming a set of historical motion states of the robotic arm with the same first-level constraint.

[0106] Secondly, based on the work area image, the work area recorded image set is traversed to perform image similarity comparison, obtaining an image similarity set. The work area recorded image set refers to the set of historical work environment images associated with all samples in the first-level restricted historical motion state set. Image similarity comparison refers to extracting image feature vectors, calculating the similarity between the current work area image and historical image records, filtering out samples with consistent environments, and ensuring that the baseline state is adapted to the current scene. The image similarity set is the set of similarity values ​​between all historical image records and the current work area image, with values ​​ranging from [0,1]. The closer the value is to 1, the higher the similarity.

[0107] Specifically, the current working area image loaded in S300 is extracted, and a 2048-dimensional feature vector, denoted as V_current, is extracted using the ResNet-50 model. The working area image set is traversed, and the same feature extraction operation is performed on each historical image to obtain the corresponding 2048-dimensional feature vector V_history. The cosine similarity algorithm is used to calculate the similarity between V_current and V_history. The similarity value of each historical image is recorded to form an image similarity set.

[0108] For example, the current working area image feature vector V_current is [0.12, 0.35, ..., 0.28], with a total of 2048 dimensions. Select an image from the historical image set, whose feature vector V_history is [0.11, 0.36, ..., 0.27]. The similarity is calculated using the cosine similarity algorithm to be approximately 0.81, meaning that the similarity between this historical image and the current image is 81%. After traversing all 120 historical images, an image similarity set {0.81, 0.79, 0.92, ..., 0.65} is formed.

[0109] Furthermore, based on the image similarity set, a secondary set of historical motion states of the same-constraint robotic arm is selected from the primary set of historical motion states of the same-constraint robotic arm, where the image similarity is greater than or equal to an image similarity threshold. The image similarity threshold is a preset screening threshold, such as 80%, used to determine the degree of adaptation between historical images and the current environment; samples higher than or equal to this threshold are considered to be environmentally consistent. The secondary set of historical motion states of the same-constraint robotic arm refers to the set of historical motion states selected from the primary set that meet the image similarity standard and are highly adapted to the current environment.

[0110] For example, an image similarity threshold of 80% is set, which can be adjusted according to scenario requirements. The image similarity set is traversed, and the historical motion states corresponding to each similarity value in the primary set are matched. Samples with a similarity value ≥ 80% are retained, while samples with insufficient similarity are removed, forming a secondary set of historical motion states for the same-constraint robotic arm. For instance, if the image similarity set contains 95 samples with a similarity ≥ 80%, the remaining 25 samples with a similarity below 80% are removed. The final result is a secondary set of historical motion states for the same-constraint robotic arm containing 95 samples, where the historical environment corresponding to all samples is highly consistent with the current working area.

[0111] Then, each discrete parameter at any given moment in the historical motion state set of the second-level confined robotic arm is deleted. Using the remaining parameters, the historical baseline motion state of the confined robotic arm is constructed. Discrete parameters at any given moment refer to anomalous parameters in the historical motion state set of the second-level confined robotic arm at the same instant, which deviate significantly from the parameters of most samples and interfere with the accuracy of the baseline state. The historical baseline motion state of the confined robotic arm refers to a standardized and referable sequence of historical motion states formed by integrating the effective parameters from the historical motion state set of the second-level confined robotic arm according to a time series after removing discrete parameters.

[0112] Specifically, deleting the discrete parameters at each moment in the historical motion state set of the secondary-level same-restriction robotic arm includes:

[0113] From the historical motion state set of the second-level same-limited robotic arm, extract the historical motion state set at time i, where i represents the time of the historical motion state of the robotic arm, the initial value of i is equal to 1, and i is an integer.

[0114] The pairwise state deviations of the historical motion state set at time i are calculated to obtain the set of historical motion state deviation coefficients;

[0115] Based on the set of historical motion state deviation coefficients, outlier historical motion states in the set of historical motion states at time i are deleted to obtain the set of concentrated historical motion states at time i.

[0116] Based on the error tolerance threshold of the deviation coefficient, the historical motion state set at time i is traversed to build the historical baseline motion state at time i.

[0117] When i equals the total number of time points, the historical reference motion state at time point i is connected sequentially with the historical reference motion state at the previous i time points, and the output is the historical reference motion state of the same-limited robotic arm.

[0118] When i is less than the total number of time points, increment i by one and execute the loop.

[0119] First, from the historical motion state set of the secondary confined robotic arm, the historical motion state set at time i is extracted, where i represents the time of the robotic arm's historical motion state, and i is initially equal to 1 and is an integer. The historical motion state set at time i refers to the set of motion state parameters of all samples in the historical motion state set of the secondary confined robotic arm at the same time node i. Initialize i=1, where i is an integer, i=1 corresponds to the 1st ms, i=2 corresponds to the 2nd ms, representing the current processing time; from the historical motion state set of the secondary confined robotic arm, extract the motion state parameters of all samples at time i, such as joint angle, end effector velocity, height, etc., to form the historical motion state set at time i. For example, the historical motion state set of the secondary confined robotic arm contains 95 sets of samples, with a total of 1000 times. When i=50, extract the motion state parameters of the 95 sets of samples at 50ms, including joint 1 angle, end effector velocity, and current height, to form the historical motion state set at time 50.

[0120] Secondly, the pairwise state deviations of the historical motion state set at the i-th time are calculated to obtain the historical motion state deviation coefficient set.

[0121] Specifically, pairwise state deviation calculations are performed on the historical motion state set at time i to obtain a set of historical motion state deviation coefficients, including:

[0122] Extract the first historical motion state and the second historical motion state from the historical motion state set at time i;

[0123] The type parameter of the first historical motion state is encoded using multi-valued encoding, and combined with the quantization parameter, a first state array is constructed.

[0124] The type parameter of the second historical motion state is encoded using multi-valued encoding, and combined with the quantization parameter, a second state array is constructed.

[0125] The same attribute deviation is calculated for the first state array and the second state array, and the deviation of each attribute is normalized to obtain the state deviation array;

[0126] Using the state deviation of the state deviation array as the distance parameter for each dimension, perform Euclidean distance calculation to obtain the first historical motion state deviation coefficient, and add it to the historical motion state deviation coefficient set.

[0127] First, extract the first historical motion state and the second historical motion state from the historical motion state set at time i. The first historical motion state and the second historical motion state are two different sets of samples randomly selected from the historical motion state set at time i. From the historical motion state set at time i, arbitrarily select two sets of samples, denoted as the first historical motion state and the second historical motion state, respectively.

[0128] For example, from the historical motion state set at time 50, sample A and sample B are selected: Sample A is denoted as the first historical motion state: Type parameter: Operation phase: Motion phase, Constraint type: Height constraint; Quantization parameter: Joint 1 angle: 35.2°, End-effector velocity: 0.2m / s, Current height: 65cm; Sample B is denoted as the second historical motion state: Type parameter: Operation phase: Motion phase, Constraint type: Height constraint; Quantization parameter: Joint 1 angle: 35.5°, End-effector velocity: 0.21m / s, Current height: 64.9cm.

[0129] Secondly, the type parameters of the first historical motion state are multi-valued and encoded, and combined with the quantization parameters to construct a first state array. Multi-valued encoding refers to the encoding method of converting type parameters into fixed values. For example, two types are encoded as 1 and 0; three types are encoded as 0, -1, and 1, and so on. The type parameters of the first historical motion state are multi-valued and encoded. For example, if the operation stage has only one type, the motion stage, it is encoded as 0; if the constraint type has only one type, the height constraint, it is encoded as 0. The quantization parameters of the first historical motion state are extracted and the original values ​​are retained. The first state array is formed by combining the type parameter encoding and the quantization parameters in the order of type parameter encoding and quantization parameters. For example, if the type parameter encoding of sample A is [0,0] and the quantization parameter is [35.2,0.2,65], the first state array is [0,0,35.2,0.2,65].

[0130] Furthermore, the type parameter of the second historical motion state is multi-valued and encoded, and combined with the quantization parameter to construct a second state array. Using the above multi-valued encoding rules, the type parameter of the second historical motion state is encoded; the quantization parameter of the second historical motion state is extracted, retaining its original value; and combined in the same order as the first state array to form the second state array. For example, if the type parameter of sample B is encoded as [0,0] and the quantization parameter is [35.5,0.21,64.9], the second state array is [0,0,35.5,0.21,64.9].

[0131] Then, the same-attribute deviation is calculated for the first state array and the second state array, and the deviation of each attribute is normalized to obtain the state deviation array. The same-attribute deviation refers to the difference between the corresponding position parameters in the first state array and the second state array, which is used to measure the difference of a single attribute. The normalization calculation means converting the attribute deviation into a value in the range [0,1] to eliminate the influence of different parameter units. The same-attribute deviation is obtained by subtracting the first state array and the second state array element by element; for each attribute, the maximum possible deviation value of the attribute is set, such as a maximum deviation of 5° for joint angle and a maximum deviation of 0.5m / s for speed; each attribute deviation is divided by the corresponding maximum possible deviation value to obtain the normalized state deviation array.

[0132] For example, the same attribute deviation calculation between sample A and sample B is as follows: Type parameter deviation: 0-0=0, 0-0=0; Quantization parameter deviation: 35.5-35.2=0.3°, 0.21-0.2=0.01m / s, 65-64.9=0.1cm; Set the maximum possible deviation: Type parameter 0, joint angle 5°, speed 0.5m / s, height 5cm; Normalized state deviation array = [0,0,0.3 / 5=0.06,0.01 / 0.5=0.02,0.1 / 5=0.02].

[0133] Subsequently, using the state deviations in the state deviation array as distance parameters for each dimension, Euclidean distance calculation is performed to obtain the first historical motion state deviation coefficient, which is then added to the historical motion state deviation coefficient set. Euclidean distance refers to the square root of the sum of the squares of the distance parameters in each dimension of the state deviation array, used to comprehensively measure the overall deviation between the two sets of states. The historical motion state deviation coefficient is the result of the Euclidean distance calculation; the smaller the value, the closer the two sets of states are. Each element in the state deviation array is used as a dimension distance parameter; the Euclidean distance formula is executed; the calculated deviation coefficient is added to the historical motion state deviation coefficient set. For example, based on the state deviation array [0,0,0.06,0.02,0.02], the deviation coefficient is calculated to be approximately 0.066; 0.066 is added to the historical motion state deviation coefficient set, and the above steps are repeated to complete the historical motion state deviation calculation for all pairwise samples at time i, resulting in a complete historical motion state deviation coefficient set.

[0134] Further, based on the historical motion state deviation coefficient set, outlier historical motion states in the historical motion state set at time i are removed to obtain the centralized historical motion state set at time i. Outlier historical motion states refer to abnormal states with deviation coefficients significantly higher than those of the majority of samples, whose parameters do not conform to the overall trend. The mean μ and standard deviation σ of the historical motion state deviation coefficient set are calculated; an outlier determination threshold of μ+2σ is set, and states with deviation coefficients higher than this threshold are determined to be outlier states; from the historical motion state set at time i, all outlier historical motion states are removed to obtain the centralized historical motion state set at time i.

[0135] For example, at time 50, the mean μ of the deviation coefficient set is 0.07, the standard deviation σ is 0.02, and the outlier determination threshold is 0.07 + 2 × 0.02 = 0.11. The deviation coefficient of a certain sample is 0.13, which is higher than the outlier determination threshold of 0.11, and it is determined to be an outlier. After deleting the outlier sample, there are 92 samples remaining in the historical motion state set at time 50.

[0136] Then, based on the deviation coefficient tolerance threshold, the historical motion state set at time i is traversed to construct the historical baseline motion state at time i. The deviation coefficient tolerance threshold is a preset allowable deviation threshold, such as 0.05, used to filter valid parameters in the historical motion state set at time i. The historical baseline motion state at time i refers to the average of the valid parameters in the historical motion state set at time i, serving as the standard benchmark for that time. The deviation coefficient tolerance threshold is set to 0.05; the historical motion state set at time i is traversed, retaining sample parameters with a deviation coefficient ≤ tolerance threshold; the average of the retained parameters is taken according to their attributes to obtain the historical baseline motion state at time i.

[0137] For example, in the 92 samples of the historical motion state set at time 50, the deviation coefficient of 88 samples is ≤0.05: Calculate the mean values ​​of the parameters of the 88 samples: mean value of joint 1 angle ≈35.2°, mean value of end velocity ≈0.2m / s, mean value of current height ≈64.9cm; historical reference motion state at time 50: operation stage: motion stage, constraint type: height constraint, joint 1 angle: 35.2°, end velocity: 0.2m / s, current height: 64.9cm.

[0138] Furthermore, when i equals the total number of time points, the historical reference motion state at time point i is sequentially connected with the historical reference motion states at the previous i time points, and the result is output as the historical reference motion state of the same-constraint robotic arm. If i equals the total number of time points, the historical reference motion state at time point i is sequentially connected with the historical reference motion states at the previous i-1 time points, forming a complete historical reference motion state of the same-constraint robotic arm.

[0139] Furthermore, when i is less than the total number of time points, i is incremented by one, and the loop is executed. If i is less than the total number of time points, i is set to i+1, and the loop is repeated until all time points have been processed.

[0140] For example, the total number of time points is 1000. When i=1000: the historical reference motion state at time point 1000 has been constructed; connect all the reference motion states from time point 1ms to time point 1000ms in sequence to form a complete historical reference motion state sequence, which is the final historical reference motion state of the same-constraint robotic arm.

[0141] In this embodiment of the invention, a first-level restriction retrieval based on the task structure and robotic arm model is used to achieve accurate matching between historical data and the current task and equipment type. By using cosine similarity comparison of the working area image, the historical scene is made highly consistent with the current working environment. Through pairwise deviation calculation and outlier state screening, the benchmark parameters are denoised and purified. Through time-by-time parameter averaging and time series concatenation, the benchmark motion state is standardized and coherently constructed. Finally, a structured and high-purity historical benchmark motion state that is highly adapted to the current scene is formed, providing a reliable reference for subsequent reasoning confidence calculation. This effectively solves the problems of low reference value of historical data and insufficient accuracy of benchmark state.

[0142] S600: Calculate the similarity between the historical baseline motion state of the same-constraint robotic arm and the recommended motion state of the robotic arm as the inference confidence level.

[0143] In this embodiment of the invention, the similarity between the historical baseline motion state of the robotic arm and the recommended motion state of the robotic arm is calculated as the inference confidence score. The reliability of the recommended motion state needs to be verified by comparison with a highly compatible historical baseline motion state. However, significant differences in the number of moments can render the comparison meaningless; for example, if the recommended state has only 500 moments and the baseline state has 1000 moments, the stages cannot be matched. In existing technologies, similarity calculations often ignore the matching of the number of moments or directly use the overall parameter mean for comparison, which is difficult to accurately reflect the consistency of the motion state moment by moment and is prone to misjudgment of confidence scores. Therefore, it is necessary to first verify the matching of the number of moments, then compare parameters moment by moment, use a deviation tolerance threshold to define whether a match has occurred, and finally calculate the inference confidence score by the proportion of qualifying moments to ensure that the confidence score truly reflects the reliability of the recommended motion state.

[0144] Step S600 in the method provided in this embodiment of the invention includes:

[0145] When the difference between the total number of moments in the first moment of the recommended motion state of the robotic arm and the total number of moments in the second moment of the historical baseline motion state of the same restricted robotic arm is greater than or equal to the difference threshold, the inference confidence of the recommended motion state of the robotic arm is set to 0.

[0146] Otherwise, extract the historical reference motion state at time i from the historical reference motion state of the same restricted robotic arm;

[0147] Extract the recommended motion state of the robotic arm at time i;

[0148] The proportion of the number of motion states that match the recommended motion state at time i and the historical baseline motion state at time i is set as the similarity at time i.

[0149] When i equals the total number of the second time point, the proportion of time point similarity greater than or equal to the time point similarity threshold is set as the inference confidence.

[0150] Otherwise, increment i and execute the comparison loop.

[0151] First, when the difference between the total number of first moments of the recommended motion state of the robotic arm and the total number of second moments of the same historical baseline motion state of the robotic arm is greater than or equal to a difference threshold, the inference confidence of the recommended motion state is set to 0. The total number of first moments refers to the total number of time nodes included in the recommended motion state, such as the total number of moments from startup to completion of the suggested operation. The total number of second moments refers to the total number of time nodes included in the same historical baseline motion state of the robotic arm. The difference threshold is a preset allowable difference in the number of moments, used to determine the temporal dimension matching of the two sets of states. The total number of first moments of the recommended motion state of the robotic arm is extracted and denoted as N1; the total number of second moments of the same historical baseline motion state of the robotic arm is extracted and denoted as N2; the difference in the number of moments between the two is calculated: ΔN = |N1 - N2|; a difference threshold is set. If ΔN ≥ the difference threshold, the inference confidence is directly set to 0; if ΔN < the difference threshold, the subsequent time-by-time comparison process begins.

[0152] For example, continuing the task of grasping the blue plastic part: the total number of the first moment of the recommended motion state of the robotic arm is N1 = 990 moments, or 990ms; the total number of the second moment of the same historical baseline motion state is N2 = 1000 moments, or 1000ms; the difference in the number of moments is ΔN = |990-1000| = 10 moments, and the threshold for the difference in the number of moments is set to 50 moments; since 10 < 50, the number matching requirement is met, and the comparison is carried out moment by moment.

[0153] First, the historical reference motion state at time i is extracted from the historical reference motion state of the same-constraint manipulator. The historical reference motion state at time i refers to the standardized set of motion state parameters at the i-th time node in the historical reference motion state of the same-constraint manipulator, covering parameters such as joint angle, end effector velocity, and current height. From the constructed historical reference motion state sequence of the same-constraint manipulator, all motion state parameters corresponding to time i are located and extracted to ensure that the parameter types are consistent with the recommended motion state parameters to be compared later. For example, when i=50, the parameters corresponding to 50ms are extracted from the historical reference motion state sequence: joint 1 angle 35.2°, end effector velocity 0.2m / s, and current height 64.9cm.

[0154] Secondly, the recommended motion state at time i is extracted from the recommended motion state of the robotic arm. The recommended motion state at time i refers to the set of motion state parameters in the recommended motion state of the robotic arm that corresponds to the historical benchmark motion state at time i, and the parameter types correspond one-to-one with the historical benchmark parameters. The motion state parameters at time i are matched from the recommended motion state sequence of the robotic arm output by the multimodal large language model to ensure that the extracted time point is completely aligned with the historical benchmark at time i. For example, when i=50, the parameters corresponding to 50ms are extracted from the recommended motion state sequence: joint 1 angle 35.4°, end effector velocity 0.21m / s, current height 65.2cm.

[0155] Next, the proportion of the recommended motion state at time i that matches the historical baseline motion state at time i is calculated and set as the similarity at time i. The number of matched motion states refers to the number of parameters in the recommended motion state at time i whose parameter deviation from the corresponding historical baseline motion state is within a preset deviation tolerance threshold. The proportion of matched motion states is equal to the ratio of the number of matched motion states to the total number of parameters in the motion state at time i. The similarity at time i refers to the degree of matching between the recommended motion state at time i and the historical baseline motion state, as represented by the proportion of matched motion states.

[0156] Specifically, the parameter deviation between the recommended motion state at time i and the historical baseline motion state at time i is calculated for each parameter, and the absolute value of the deviation is taken. If the absolute value of the deviation of a certain parameter is less than or equal to the deviation tolerance threshold corresponding to that parameter, the parameter is determined to be a hit motion state parameter. The number of hit motion state parameters at time i and the total number of motion state parameters at that time are counted. The proportion of hit motion state is obtained by dividing the number of hit motion states by the total number of parameters, and this proportion is set as the similarity at time i.

[0157] For example, when i=50, the total number of motion state parameters at time i is 3: joint 1 angle, end velocity, and current height. The preset deviation tolerance thresholds are joint angle ±0.3°, end velocity ±0.02m / s, and current height ±0.5cm. The parameter-by-parameter deviation calculation results are: joint 1 angle deviation 0.2°, end velocity deviation 0.01m / s, and current height deviation 0.3cm, all within the tolerance thresholds, and the number of hit motion states is 3. The percentage of hit motion states = 3 / 3 = 1, and the similarity at time 50 is set to 100%.

[0158] Furthermore, when i equals the total number of the second time points, the proportion of time point similarities greater than or equal to the time point similarity threshold is calculated and set as the inference confidence. Otherwise, i is incremented by one, and the comparison loop is executed. The total number of the second time points: This is the same as the total number of time nodes included in the historical baseline motion state of the robotic arm, and serves as the termination node for time-by-time comparison. The time point similarity threshold is a preset minimum threshold for determining a single time point similarity as a qualified state, used to filter moments with acceptable matching. The inference confidence refers to the overall matching degree between the recommended motion state and the historical baseline motion state, characterized by the proportion of qualified moments among all times; it is a quantitative indicator of the reliability of the recommended motion state. Determining if the current value of i equals the total number of the second time points: If i equals the total number of the second time points, the similarity of all times is iterated, and the number of qualified moments greater than or equal to the time point similarity threshold is counted. The number of qualified moments is divided by the total number of the second time points to obtain the proportion, which is set as the inference confidence. If i is less than the total number of the second time points, the value of i is incremented by 1, and the time-by-time comparison loop is continued.

[0159] For example, the preset similarity threshold for time moments is 80%, and the total number of second time moments is 1000. If the current i=1000, which is equal to the total number of second time moments, the similarity results of time moments 1 to 1000 are iterated, and it is found that there are 985 time moments with a similarity ≥80%, and the number of time moments that meet the standard is 985. The inference confidence = 985 / 1000 = 98.5%, and this value is set as the final inference confidence. If the current i=50, which is less than the total number of second time moments 1000, i is incremented by 1 to 51, and then the historical baseline and recommended motion state of time moment 51 are extracted and the subsequent comparison is completed.

[0160] In this embodiment of the invention, accurate calculation of inference confidence is achieved through time-quantity matching verification, time-by-time parameter comparison, and statistical analysis of the proportion of qualified times. First, invalid comparisons with severe time mismatches are filtered out using a quantity difference threshold to avoid confidence distortion. Then, a customized deviation tolerance threshold is used to define whether a parameter hits, taking into account both parameter characteristics and actual operational errors. Finally, qualified times are selected using a time similarity threshold to comprehensively reflect the overall matching degree. This solves the problems of neglecting time matching and having ambiguous parameter deviation tolerance in existing technologies. The output inference confidence can realistically and quantitatively reflect the reliability of the robotic arm's recommended motion state, providing a precise basis for confidence gating of subsequent control command generation.

[0161] S700: When the inference confidence is greater than or equal to the confidence threshold, the robotic arm motion state determination result and the task structure are mapped into an action primitive sequence to generate control instructions.

[0162] In this embodiment of the invention, when the inference confidence level is greater than or equal to a confidence threshold, the robotic arm motion state determination result and the task structure are mapped to an action primitive sequence to generate control instructions. The confidence threshold is a preset inference confidence qualification standard, a gating indicator for determining whether the recommended motion state of the robotic arm has execution value. Only when the confidence threshold is reached will the determination result be converted into an execution instruction. The action primitive sequence is an ordered set of basic action units recognizable by the robotic arm, arranged according to execution logic. It serves as an intermediate carrier connecting the high-level motion state determination result and the low-level control instructions, with each action primitive corresponding to a single basic operation of the robotic arm. The control instructions are standardized instructions generated based on the action primitive sequence, which can be directly parsed and executed by the robotic arm controller, and are fully compatible with the communication protocol and execution logic of the robotic arm model used.

[0163] Specifically, the inference confidence score calculated by S600 is first retrieved and compared with a preset confidence threshold. If the inference confidence score is less than the confidence threshold, this step is terminated and no execution instructions are generated. If the inference confidence score is greater than or equal to the confidence threshold, the subsequent mapping and generation process begins. Based on the robot arm's device model and motion control protocol, a mapping rule is established between the robot arm's motion state determination results and action primitives. Combining the target object, operation type, and constraints in the task structure, the determination results and the task structure are jointly mapped into a sequence of action primitives that conforms to the execution logic. According to the underlying control specifications of the robot arm, the action primitive sequence is parsed and transformed into standardized control instructions, ensuring that the format and parameters of the instructions are consistent with the receiving requirements of the robot arm controller, thus completing the generation of the final control instructions.

[0164] For example, the inference confidence level calculated by S600 is 98.5%, which is ≥90% confidence threshold, satisfying the instruction generation conditions. The current robotic arm motion state determination result is: normal operation phase, it is recommended to maintain stable gripping force and move towards the left assembly table according to the preset trajectory, with a height not less than 50 cm during the movement. Combining the requirements of the task structure, the CR5 robotic arm's motion logic is mapped into a sequence of motion primitives: maintain gripping force of 8-10N, move the end effector at a uniform speed of 0.2m / s, maintain end effector height ≥50cm, decelerate to 0.05m / s when approaching the left assembly table, adjust the gripper opening to 30mm after reaching the target position to release the part, and reset the gripper to the initial opening after release. According to the CR5 robotic arm's underlying control protocol, the above sequence of motion primitives is converted into directly executable control instructions. The instructions include the parameter thresholds, execution timing, and triggering conditions for each action, completing the final generation of control instructions.

[0165] In this embodiment of the invention, this step serves as the final implementation stage of the entire robotic arm motion state determination and control process. By gating through a confidence threshold, it filters out the generation of instructions for recommending motion states with low reliability, thus avoiding risks such as robotic arm operation errors, equipment collisions, and task failures caused by state determination deviations from the source, thereby improving the safety and stability of robotic arm operations. Simultaneously, through intermediate mapping of motion primitive sequences, it achieves precise connection between the high-level abstract motion state determination results and the low-level executable robotic arm control instructions, effectively solving the problems of disconnect between determination results and execution logic, and the lack of clear basis for instruction generation in existing technologies. The final generated control instructions not only strictly conform to the requirements and constraints of the task structure but also are highly compatible with the hardware characteristics and control protocol of the robotic arm model, ensuring that the robotic arm can complete grasping and placement operations in a compliant and accurate manner. This achieves closed-loop control throughout the entire process from task reception and state determination to instruction execution, improving the task execution accuracy and process continuity of the robotic arm in complex work scenarios.

[0166] Through the specific implementation methods described above, the embodiments of the present invention achieve the following technical effects:

[0167] This invention provides a method and system for intelligent judgment of robotic arm status based on a multimodal large language model. By parsing structured task structures from natural language instructions, a phased verification template is constructed. Relying on the multimodal large language model, a recommended motion state for the robotic arm is output in stages. Then, historical benchmark motion states are retrieved to calculate the inference confidence. After confidence gating, control commands are generated through mapping. The entire process achieves accurate judgment of the robotic arm's motion state and reliable generation of control commands. The phased design adapts to the verification needs of each operational stage, historical benchmark matching improves the reliability of the judgment, and confidence gating avoids the risk of operational errors from the source. Ultimately, a closed-loop control is achieved from task reception to command execution, improving the accuracy, safety, and process continuity of robotic arm operations.

[0168] Example 2, as Figure 2 As shown, this invention provides an intelligent system for judging the state of a robotic arm based on a multimodal large language model. The system includes:

[0169] The task structure acquisition module 11 is used to acquire a task structure, wherein the task structure includes a target object, a target region, an operation type, and constraints.

[0170] The multi-template prompt set building module 12 is used to build a multi-template prompt set for the task structure, wherein the multi-template prompt set includes a motion phase state verification template, an arrival phase position verification template, an attitude verification template, and an operation phase execution verification template.

[0171] The multimodal input orchestration module 13 is used to load the working area image and the robot arm body status data when performing work, and to orchestrate the multimodal large language model input in combination with the multi-template prompt set.

[0172] The output state determination module 14 is used to call the multimodal large language model, process the input of the multimodal large language model, and output the motion state determination result of the robotic arm, wherein the motion state determination result of the robotic arm includes the recommended motion state of the robotic arm.

[0173] The historical baseline retrieval module 15 is used to retrieve the historical baseline motion state of the robotic arm with the same restrictions as the task structure, the working area image and the robotic arm model.

[0174] The inference confidence calculation module 16 is used to calculate the similarity between the historical baseline motion state of the same-limit robotic arm and the recommended motion state of the robotic arm as the inference confidence.

[0175] The control instruction generation module 17 is used to map the robotic arm motion state determination result and the task structure into an action primitive sequence and generate control instructions when the inference confidence is greater than or equal to the confidence threshold.

[0176] In one embodiment, the task structure acquisition module 11 is further configured to:

[0177] The process of obtaining the task structure includes, before:

[0178] Collect a set of historical input statements from users and a set of labels for identifying task structures, and set them as the user intent recognition training dataset. In the set of labels for identifying task structures, any label for identifying a task structure is composed of binary codes for the target object description entity, binary codes for the target region description entity, binary codes for the operation type description entity, and binary codes for the constraint condition description entity.

[0179] A task structure label mapping layer is built and connected to the output layer of the Transformer architecture to obtain the task structure parsing model training architecture. The output task of the Transformer architecture is to identify the user's historical input statement and output the task structure description entity.

[0180] A training loss function is constructed based on Hamming distance, wherein the training loss function is positively correlated with the Hamming distance;

[0181] The task structure parsing model is trained using the user's historical input statement set as input and the label set of the task structure identifier as supervision. After convergence, the task structure label mapping layer is deleted to obtain the task structure parsing model.

[0182] The task structure is obtained by processing user input statements using the task structure parsing model.

[0183] In one embodiment, the multi-template prompt set building module 12 is also used for:

[0184] The motion phase status verification template consists of a working area image, a robotic arm motion parameter table, a task structure summary, and the judgment result of the previous cycle.

[0185] The positioning verification template for the arrival stage consists of a working area image, a robotic arm position parameter table, a task structure summary, and the judgment result of the previous cycle.

[0186] The attitude verification template consists of a working area image, a robotic arm joint parameter table, a task structure summary, and the judgment result of the previous cycle.

[0187] The verification template for the operation phase consists of a working area image, a gripper status parameter table, a task structure summary, and the judgment result of the previous cycle.

[0188] In one embodiment, the output state determination module 14 is further configured to:

[0189] When the work is in motion, the multimodal large language model is invoked to process the motion phase state verification template input and obtain the motion phase robotic arm trajectory determination result.

[0190] When the work is in place, the multimodal large language model is invoked to process the in-place stage position verification template input and obtain the in-place stage end position determination result;

[0191] When the work is in the posture verification stage, the multimodal large language model is called to process the posture verification template input and obtain the robot arm joint judgment result in the posture verification stage.

[0192] When the work is in the operation phase, the multimodal large language model is invoked to process the operation phase execution verification template input and obtain the operation phase gripper state determination result.

[0193] The determination result of the robotic arm's motion state is one of the following: the determination result of the robotic arm trajectory during the motion phase, the determination result of the end position during the positioning phase, the determination result of the robotic arm joint during the posture verification phase, and the determination result of the gripper state during the operation phase.

[0194] In one embodiment, the historical benchmark retrieval module 15 is further configured to:

[0195] Using the task structure and the robotic arm model as primary constraints, an anomaly sample retrieval is performed to obtain a set of historical motion states of robotic arms with the same primary constraints. Each historical motion state of a robotic arm with the same primary constraints in the set of historical motion states of robotic arms with the same primary constraints has an associated stored working area recording image.

[0196] Based on the working area image, the image similarity set is compared by traversing the working area image set to obtain the image similarity set;

[0197] Based on the image similarity set, a second-level set of historical motion states of the same-restricted robotic arm with image similarity greater than or equal to the image similarity threshold is selected from the first-level set of historical motion states of the same-restricted robotic arm.

[0198] Delete the discrete parameters at each moment in the historical motion state set of the second-level same-constraint robotic arm, and construct the historical reference motion state of the same-constraint robotic arm using the remaining parameters.

[0199] Specifically, deleting the discrete parameters at each moment in the historical motion state set of the secondary-level same-restriction robotic arm includes:

[0200] From the historical motion state set of the second-level same-limited robotic arm, extract the historical motion state set at time i, where i represents the time of the historical motion state of the robotic arm, the initial value of i is equal to 1, and i is an integer.

[0201] The pairwise state deviations of the historical motion state set at time i are calculated to obtain the set of historical motion state deviation coefficients;

[0202] Based on the set of historical motion state deviation coefficients, outlier historical motion states in the set of historical motion states at time i are deleted to obtain the set of concentrated historical motion states at time i.

[0203] Based on the error tolerance threshold of the deviation coefficient, the historical motion state set at time i is traversed to build the historical baseline motion state at time i.

[0204] When i equals the total number of time points, the historical reference motion state at time point i is connected sequentially with the historical reference motion state at the previous i time points, and the output is the historical reference motion state of the same-limited robotic arm.

[0205] When i is less than the total number of time points, increment i by one and execute the loop.

[0206] Specifically, pairwise state deviation calculations are performed on the historical motion state set at time i to obtain a set of historical motion state deviation coefficients, including:

[0207] Extract the first historical motion state and the second historical motion state from the historical motion state set at time i;

[0208] The type parameter of the first historical motion state is encoded using multi-valued encoding, and combined with the quantization parameter, a first state array is constructed.

[0209] The type parameter of the second historical motion state is encoded using multi-valued encoding, and combined with the quantization parameter, a second state array is constructed.

[0210] The same attribute deviation is calculated for the first state array and the second state array, and the deviation of each attribute is normalized to obtain the state deviation array;

[0211] Using the state deviation of the state deviation array as the distance parameter for each dimension, perform Euclidean distance calculation to obtain the first historical motion state deviation coefficient, and add it to the historical motion state deviation coefficient set.

[0212] In one embodiment, the inference confidence calculation module 16 is further configured to:

[0213] When the difference between the total number of moments in the first moment of the recommended motion state of the robotic arm and the total number of moments in the second moment of the historical baseline motion state of the same restricted robotic arm is greater than or equal to the difference threshold, the inference confidence of the recommended motion state of the robotic arm is set to 0.

[0214] Otherwise, extract the historical reference motion state at time i from the historical reference motion state of the same restricted robotic arm;

[0215] Extract the recommended motion state of the robotic arm at time i;

[0216] The proportion of the number of motion states that match the recommended motion state at time i and the historical baseline motion state at time i is set as the similarity at time i.

[0217] When i equals the total number of the second time point, the proportion of time point similarity greater than or equal to the time point similarity threshold is set as the inference confidence.

[0218] Otherwise, increment i and execute the comparison loop.

[0219] It should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for intelligent judgment of robotic arm state based on a multimodal large language model, characterized in that, The method includes: Obtain a task structure, wherein the task structure includes a target object, a target region, an operation type, and constraints; Obtain the task structure, which previously included: Collect a set of historical input statements from users and a set of labels for identifying task structures, and set them as the user intent recognition training dataset. In the set of labels for identifying task structures, any label for identifying a task structure is composed of binary codes for the target object description entity, binary codes for the target region description entity, binary codes for the operation type description entity, and binary codes for the constraint condition description entity. A task structure label mapping layer is built and connected to the output layer of the Transformer architecture to obtain the task structure parsing model training architecture. The output task of the Transformer architecture is to identify the user's historical input statement and output the task structure description entity. A training loss function is constructed based on Hamming distance, wherein the training loss function is positively correlated with the Hamming distance; The task structure parsing model is trained using the user's historical input statement set as input and the label set of the task structure identifier as supervision. After convergence, the task structure label mapping layer is deleted to obtain the task structure parsing model. The task structure is obtained by processing user input statements using the task structure parsing model. A multi-template prompt set is built for the task structure, wherein the multi-template prompt set includes a motion phase state verification template, an arrival phase position verification template, an attitude verification template, and an operation phase execution verification template; When performing tasks, the system loads images of the work area and the status data of the robotic arm, and combines these with the multi-template prompt set to arrange the input of a multimodal large language model. A multimodal large language model is invoked to process the input of the multimodal large language model and output the motion state determination result of the robotic arm, wherein the motion state determination result of the robotic arm includes the recommended motion state of the robotic arm; Using the task structure, the working area image, and the robot arm model as constraints, retrieve the historical baseline motion state of the robot arm with the same constraints; The similarity between the historical baseline motion state of the same-constraint robotic arm and the recommended motion state of the robotic arm is calculated as the inference confidence. When the inference confidence is greater than or equal to the confidence threshold, the robotic arm motion state determination result and the task structure are mapped into a sequence of action primitives to generate control commands.

2. The intelligent judgment method for the state of a robotic arm based on a multimodal large language model as described in claim 1, characterized in that, include: The motion phase status verification template consists of a working area image, a robotic arm motion parameter table, a task structure summary, and the judgment result of the previous cycle. The positioning verification template for the arrival stage consists of a working area image, a robotic arm position parameter table, a task structure summary, and the judgment result of the previous cycle. The attitude verification template consists of a working area image, a robotic arm joint parameter table, a task structure summary, and the judgment result of the previous cycle. The verification template for the operation phase consists of a working area image, a gripper status parameter table, a task structure summary, and the judgment result of the previous cycle.

3. The intelligent judgment method for the state of a robotic arm based on a multimodal large language model as described in claim 1, characterized in that, A multimodal large language model is invoked to process the input of the multimodal large language model and output the motion state determination result of the robotic arm. The multimodal large language model input includes motion phase state verification template input, positioning phase position verification template input, posture verification template input, and operation phase execution verification template input, including: When the work is in motion, the multimodal large language model is invoked to process the motion phase state verification template input and obtain the motion phase robotic arm trajectory determination result. When the work is in place, the multimodal large language model is invoked to process the in-place stage position verification template input and obtain the in-place stage end position determination result; When the work is in the posture verification stage, the multimodal large language model is called to process the posture verification template input and obtain the robot arm joint judgment result in the posture verification stage. When the work is in the operation phase, the multimodal large language model is invoked to process the operation phase execution verification template input and obtain the operation phase gripper state determination result. The determination result of the robotic arm's motion state is one of the following: the determination result of the robotic arm trajectory during the motion phase, the determination result of the end position during the positioning phase, the determination result of the robotic arm joint during the posture verification phase, and the determination result of the gripper state during the operation phase.

4. The intelligent judgment method for the state of a robotic arm based on a multimodal large language model as described in claim 1, characterized in that, Using the task structure, the work area image, and the robotic arm model as constraints, retrieve the historical baseline motion state of the robotic arm that is subject to the constraints, including: Using the task structure and the robotic arm model as primary constraints, an anomaly sample retrieval is performed to obtain a set of historical motion states of robotic arms with the same primary constraints. Each historical motion state of a robotic arm with the same primary constraints in the set of historical motion states of robotic arms with the same primary constraints has an associated stored working area recording image. Based on the working area image, the image similarity set is compared by traversing the working area image set to obtain the image similarity set; Based on the image similarity set, a second-level set of historical motion states of the same-restricted robotic arm with image similarity greater than or equal to the image similarity threshold is selected from the first-level set of historical motion states of the same-restricted robotic arm. Delete the discrete parameters at each moment in the historical motion state set of the second-level same-constraint robotic arm, and construct the historical reference motion state of the same-constraint robotic arm using the remaining parameters.

5. The intelligent judgment method for the state of a robotic arm based on a multimodal large language model as described in claim 4, characterized in that, Delete the discrete parameters at each instant in the historical motion state set of the secondary-level confined robotic arm, including: From the historical motion state set of the second-level same-limited robotic arm, extract the historical motion state set at time i, where i represents the time of the historical motion state of the robotic arm, the initial value of i is equal to 1, and i is an integer. The pairwise state deviations of the historical motion state set at time i are calculated to obtain the set of historical motion state deviation coefficients; Based on the set of historical motion state deviation coefficients, outlier historical motion states in the set of historical motion states at time i are deleted to obtain the set of concentrated historical motion states at time i. Based on the error tolerance threshold of the deviation coefficient, the historical motion state set at time i is traversed to build the historical baseline motion state at time i. When i equals the total number of time points, the historical reference motion state at time point i is connected sequentially with the historical reference motion state at the previous i time points, and the output is the historical reference motion state of the same-limited robotic arm. When i is less than the total number of time points, increment i by one and execute the loop.

6. The intelligent judgment method for the state of a robotic arm based on a multimodal large language model as described in claim 5, characterized in that, The pairwise state deviations of the historical motion state set at time i are calculated to obtain the historical motion state deviation coefficient set, including: Extract the first historical motion state and the second historical motion state from the historical motion state set at time i; The type parameter of the first historical motion state is encoded using multi-valued encoding, and combined with the quantization parameter, a first state array is constructed. The type parameter of the second historical motion state is encoded using multi-valued encoding, and combined with the quantization parameter, a second state array is constructed. The same attribute deviation is calculated for the first state array and the second state array, and the deviation of each attribute is normalized to obtain the state deviation array; Using the state deviation of the state deviation array as the distance parameter for each dimension, perform Euclidean distance calculation to obtain the first historical motion state deviation coefficient, and add it to the historical motion state deviation coefficient set.

7. The intelligent state judgment method for a robotic arm based on a multimodal large language model as described in claim 5, characterized in that, The similarity between the historical baseline motion state of the same-constraint robotic arm and the recommended motion state of the robotic arm is calculated as the inference confidence score, including: When the difference between the total number of moments in the first moment of the recommended motion state of the robotic arm and the total number of moments in the second moment of the historical baseline motion state of the same restricted robotic arm is greater than or equal to the difference threshold, the inference confidence of the recommended motion state of the robotic arm is set to 0. Otherwise, extract the historical reference motion state at time i from the historical reference motion state of the same restricted robotic arm; Extract the recommended motion state of the robotic arm at time i; The proportion of the number of motion states that match the recommended motion state at time i and the historical baseline motion state at time i is set as the similarity at time i. When i equals the total number of the second time point, the proportion of time point similarity greater than or equal to the time point similarity threshold is set as the inference confidence. Otherwise, increment i and execute the comparison loop.

8. A robotic arm state intelligent judgment system based on a multimodal large language model, characterized in that, The system is used to implement the intelligent state determination method for a robotic arm based on a multimodal large language model as described in any one of claims 1-7, the system comprising: The task structure acquisition module is used to obtain the task structure, wherein the task structure includes the target object, the target region, the operation type, and the constraints. A multi-template prompt set building module is used to build a multi-template prompt set for the task structure, wherein the multi-template prompt set includes a motion phase state verification template, an arrival phase position verification template, an attitude verification template, and an operation phase execution verification template. The multimodal input orchestration module is used to load the working area image and the robot arm body status data when performing work, and combine them with the multi-template prompt set to orchestrate the multimodal large language model input; The output state determination module is used to call the multimodal large language model, process the input of the multimodal large language model, and output the motion state determination result of the robotic arm, wherein the motion state determination result of the robotic arm includes the recommended motion state of the robotic arm. The historical baseline retrieval module is used to retrieve the historical baseline motion state of the robotic arm with the same restrictions as the task structure, the working area image and the robotic arm model. The inference confidence calculation module is used to calculate the similarity between the historical baseline motion state of the same-constraint robotic arm and the recommended motion state of the robotic arm as the inference confidence. The control instruction generation module is used to map the robotic arm motion state determination result and the task structure into a sequence of action primitives and generate control instructions when the inference confidence is greater than or equal to the confidence threshold.

Images