Human-computer interaction method and system for real-time task adjustment input and visual feedback based on VLA
By generating high-level semantic action units using VLA models and combining them with AR/3D spatial visualization technology, the lack of transparency and controllability in existing systems is solved, enabling efficient and secure human-computer collaborative task editing and execution.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHENZHEN MOYING TECH CO LTD
- Filing Date
- 2026-01-30
- Publication Date
- 2026-05-08
AI Technical Summary
Existing human-computer interaction systems based on large models lack transparency and controllability. Users cannot preview or modify the task execution process, it is difficult to judge the accuracy of robot operations, the editing threshold is high and the efficiency is low, there is a lack of spatial context feedback, and deep collaboration cannot be achieved.
A high-level semantic action unit sequence is generated through a VLA model, and a draggable and editable graphical task flowchart is constructed. Combined with AR/3D spatial visualization technology, three-dimensional alignment mapping is performed, supporting drag-and-drop editing and collision detection, and providing real-time feedback on execution status.
It improves the transparency and controllability of human-computer interaction, lowers the editing threshold, enhances the accuracy and security of task execution, reduces the risk of collisions, and achieves efficient human-computer collaborative decision-making.
Smart Images
Figure CN121996232A_ABST
Abstract
Description
Technical Field
[0001] This invention proposes a human-computer interaction method and system based on VLA for real-time task adjustment input and visual feedback, belonging to the interdisciplinary fields of human-computer interaction, embodied intelligence and industrial robot control. Background Technology
[0002] In the field of human-computer interaction, with the continuous development of robotics technology, how to achieve efficient, transparent, and controllable human-computer collaboration has become a key issue. Current human-computer interaction systems based on large models have revealed many limitations in practical applications, making it difficult to meet the human-computer interaction needs of modern complex scenarios.
[0003] Traditional systems often operate in a black-box manner. After a user issues a natural language command, the system directly generates and executes a sequence of actions. The user cannot preview, modify, or understand the intermediate decision-making logic, much like facing a mysterious black box, lacking control over the task execution process. Furthermore, the lack of spatial context feedback and the failure to align the task plan with the real physical environment in three-dimensional space makes it difficult for users to judge the accuracy of the robot's operations. For example, they cannot determine whether the robot has grasped the correct object or whether the path planning will result in collisions with surrounding objects.
[0004] Furthermore, when tasks require modification, the editing threshold is high, typically requiring the re-description of instructions or the writing of code. It does not support graphical, drag-and-drop low-code adjustments, making the operation complex and inefficient. Moreover, VLA outputs are mostly raw joint trajectories or API call sequences, not deconstructed into high-level semantic action units, leading to editing difficulties. Existing public demonstrations are mostly limited to one-way interactive videos or fixed UI buttons for selecting preset tasks, failing to convert VLA-generated results into editable graphical task flowcharts with overlaid AR spatial annotations. Essentially, they remain "smarter voice remote controls," unable to achieve deep collaboration through "human-computer co-editing and collaborative decision-making." Therefore, a new human-computer interaction method is urgently needed. Summary of the Invention
[0005] This invention provides a human-computer interaction method and system based on VLA for real-time task adjustment input and visual feedback, in order to solve the problems mentioned in the background art above:
[0006] The present invention proposes a human-computer interaction method based on VLA for real-time task adjustment input and visual feedback, the method comprising:
[0007] S1. Receive natural language instructions input by the user, generate robot task plan using VLA model, parse it into high-level semantic action unit sequence, generate structured task logic data, construct a drag-and-drop editable graphical task flowchart, and generate initial task flowchart data.
[0008] S2. Using AR / 3D spatial visualization technology, the initial task flowchart data is aligned and mapped with the real physical environment in three dimensions to generate spatially visualized task flowchart data. AR spatial annotations are then overlaid to generate visualized task flowchart data with spatial annotations.
[0009] S3. Based on the user's drag-and-drop editing operation on the spatially labeled visual task flowchart data, generate task flowchart modification instruction data, parse it, convert it into corresponding structured task logic adjustment data, update the initial task flowchart data, and generate optimized task flowchart data.
[0010] S4. Extract the action parameter data required for task execution from the optimized task flowchart data; perform collision detection analysis between the action parameter data and the real physical environment to generate collision detection result data, and further revise the optimized task flowchart data to generate the final confirmed task flowchart data.
[0011] S5. Based on the final confirmed task flowchart data, generate robot execution instruction data; send it to the robot control system to drive the robot to execute safely according to the optimized and confirmed task logic, while providing real-time feedback on execution status data, dynamically updating the execution progress of the task flowchart in the AR / 3D spatial visualization interface, and generating real-time execution feedback visualization data.
[0012] The present invention proposes a system for implementing the human-computer interaction method for real-time task adjustment input and visual feedback based on VLA as described above, the system comprising:
[0013] Command receiving module: Receives natural language commands input by the user, generates robot task plans using the VLA model, parses them into high-level semantic action unit sequences, generates structured task logic data, constructs a drag-and-drop editable graphical task flowchart, and generates initial task flowchart data.
[0014] Spatial alignment module: Using AR / 3D spatial visualization technology, the initial task flowchart data is aligned and mapped with the real physical environment in three dimensions to generate spatially visualized task flowchart data; AR spatial annotations are superimposed to generate visualized task flowchart data with spatial annotations.
[0015] Data update module: Based on the user's drag-and-drop editing operation on the spatially labeled visual task flowchart data, generate task flowchart modification instruction data, parse it, convert it into corresponding structured task logic adjustment data, update the initial task flowchart data, and generate optimized task flowchart data.
[0016] Detection and Analysis Module: Extracts action parameter data required for task execution from the optimized task flowchart data; performs collision detection analysis on the action parameter data and the real physical environment to generate collision detection result data; and further corrects the optimized task flowchart data to generate the final confirmed task flowchart data.
[0017] Feedback Execution Module: Based on the final confirmed task flowchart data, it generates robot execution instruction data; sends it to the robot control system to drive the robot to execute safely according to the optimized and confirmed task logic, and at the same time provides real-time feedback on execution status data, dynamically updates the execution progress of the task flowchart in the AR / 3D spatial visualization interface, and generates real-time execution feedback visualization data.
[0018] The beneficial effects of this invention are as follows: By transforming the robot task plan generated by VLA into a draggable and editable graphical flowchart, and presenting it with AR / 3D spatial visualization, the transparency and controllability of human-computer interaction are significantly improved. Users can intuitively correct and optimize the task logic before execution, reducing the uncertainty caused by black-box execution and enhancing the accuracy and safety of task execution. Simultaneously, the combination of graphical editing and AR spatial annotation reduces the tedious operation of re-describing instructions or writing code, lowering the editing threshold and improving task correction efficiency. Furthermore, this method avoids collision problems caused by the lack of spatial context feedback in traditional systems, reducing unnecessary losses and risks. It enables efficient human-computer collaborative decision-making and allows users to deeply participate in the task editing and optimization process, bringing a new and more practical human-computer collaboration paradigm to the field of human-computer interaction, significantly improving the overall interactive experience and practicality. Attached Figure Description
[0019] Figure 1 This is a diagram illustrating the steps of the method described in this invention;
[0020] Figure 2 This is a system module diagram of the present invention. Detailed Implementation
[0021] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.
[0022] One embodiment of the present invention, such as Figure 1 As shown, a human-computer interaction method based on VLA for real-time task adjustment input and visual feedback includes:
[0023] S1. Receive natural language commands input by the user, generate a robot task plan using the VLA model, and automatically parse the task plan into a high-level semantic action unit sequence to generate structured task logic data; based on the structured task logic data, construct a drag-and-drop editable graphical task flowchart to generate initial task flowchart data.
[0024] S2. Using AR / 3D spatial visualization technology, the initial task flowchart data is aligned and mapped with the real physical environment in three dimensions to generate spatially visualized task flowchart data; AR spatial annotations are superimposed on the spatially visualized task flowchart data to generate visualized task flowchart data with spatial annotations, which allows users to intuitively judge the matching between the task execution intention and the physical environment.
[0025] S3. Based on the user's drag-and-drop editing operation on the spatially labeled visual task flowchart data, generate task flowchart modification instruction data; parse the task flowchart modification instruction data and convert it into corresponding structured task logic adjustment data; update the initial task flowchart data based on the structured task logic adjustment data to generate optimized task flowchart data.
[0026] S4. Extract the action parameter data required for task execution from the optimized task flowchart data; perform collision detection analysis between the action parameter data and the real physical environment to generate collision detection result data; and further revise the optimized task flowchart data based on the collision detection result data to generate the final confirmed task flowchart data.
[0027] S5. Based on the final confirmed task flowchart data, generate robot execution instruction data; send the robot execution instruction data to the robot control system to drive the robot to execute safely according to the optimized and confirmed task logic, while providing real-time feedback on execution status data; based on the execution status data, dynamically update the execution progress of the task flowchart in the AR / 3D spatial visualization interface, and generate real-time execution feedback visualization data.
[0028] The working principle and effects of the above technical solution are as follows: The VLA model parses natural language to generate task flows and transforms them into visual graphics, lowering the operational threshold for non-professional users and avoiding task logic deviations caused by complex command inputs. AR 3D spatial alignment and annotation enhance the intuitiveness of matching task intent with the physical environment, reducing user misjudgments of task feasibility. Drag-and-drop editing improves the flexibility of flow optimization, avoiding the problem of fixed flows being difficult to adapt to complex scenarios. Collision detection corrects motion parameters, reducing the risk of collisions during robot execution and preventing equipment damage or task interruption. Real-time feedback on execution status forms a closed-loop interaction, allowing users to accurately grasp progress and quickly intervene in anomalies, improving the safety and efficiency of task execution and reducing the escalation of faults due to opaque status.
[0029] In one embodiment of the present invention, S1 includes:
[0030] S11. Receive natural language instructions input by the user. The natural language instructions include information such as task objectives, scenario constraints, and execution preferences. Perform intent recognition and context association analysis through the multimodal semantic understanding module of the VLA model to generate original task description data. The original task description data includes the core elements of the task.
[0031] S12. Based on the original task description data, the task planning engine of the VLA model is called, and the robot task plan is generated by combining the preset action unit library and logic rule library. The robot task plan covers the execution steps, action priority, and resource requirements, and generates structured task plan data.
[0032] S13. Based on the structured task plan data, using natural language and action unit mapping algorithms, the high-level task logic is automatically parsed into a high-level semantic action unit sequence. The high-level semantic action unit sequence can be decomposed and combined, clarifying the dependencies and temporal order between actions, and generating structured task logic data.
[0033] S14. Based on structured task logic data, call the graphical rendering engine to build a visual task flowchart framework. The visual task flowchart framework supports node dragging, logic rearrangement, parameter editing, defines the display rules and interaction logic of action nodes, judgment nodes, and branch nodes, and generates initial task flowchart prototype data.
[0034] S15. Perform syntax verification and logical integrity checks on the initial task flowchart prototype data, automatically correct problems such as action unit conflicts, timing contradictions, and missing parameters, optimize node layout and connection relationships, and generate initial task flowchart data.
[0035] The working principle and effects of the above technical solution are as follows: By using a VLA model for multimodal semantic understanding and analysis of natural language instructions, core elements such as task objectives and scenario constraints are accurately captured, reducing misunderstandings of instruction intent and preventing subsequent task planning from deviating from requirements due to misinterpretation of intent. Task plans are generated by combining an action unit library and a logic rule library, improving the rationality and completeness of task planning and reducing omissions or resource mismatches. High-level task logic is decomposed into composable sequences of semantic action units, enhancing the adjustability of task logic and avoiding tedious refactoring during subsequent process optimization. A visual flowchart framework presents task logic intuitively, lowering the understanding threshold for non-professional users and reducing reliance on professional operational skills. Syntax validation and logic correction automatically identify and resolve conflicts and contradictions, preventing hidden problems in the initial flowchart from affecting subsequent execution and improving the reliability of initial task flow data.
[0036] In one embodiment of the present invention, S2 includes:
[0037] S21. Obtain initial task flowchart data, collect spatial information of the real physical environment through AR spatial positioning module, the spatial information includes three-dimensional coordinates, scene structure, obstacle distribution, and operable area, and generate high-precision physical environment spatial model data.
[0038] S22. Based on the physical environment spatial model data and the initial task flowchart data, use the three-dimensional spatial alignment algorithm to perform coordinate system one, logical position matching, and scale calibration to realize the virtual-real fusion mapping between the task flowchart and the real physical environment, and generate the basic data for spatial visualization task flowchart.
[0039] S23. On the basic data of the spatial visualization task flowchart, AR spatial annotation information is superimposed. The AR spatial annotation information includes the boundary of the action execution area, the parameter threshold range, environmental constraint prompts and safety warning areas. The display style and interaction trigger conditions of the annotation are defined to generate visualization task flowchart data with preliminary spatial annotation.
[0040] S24. Through user perspective tracking and scene adaptation algorithms, adjust the transparency, size, and position of AR spatial annotations to ensure that the annotation information is clearly visible under different viewing angles and does not obscure key scene elements, and generate visual task flowchart data with accurate spatial annotations.
[0041] S25. Project the visualized task flowchart data with precise spatial annotations to the AR / 3D display device in real time. Users can zoom, rotate, and pan the data using gestures, voice, etc., so that users can intuitively judge the matching degree between the task execution intention and the physical environment and generate scenario-based interactive preview data.
[0042] The working principle and effects of the above technical solution are as follows: AR spatial positioning collects information such as the 3D coordinates and scene structure of the real environment, generating high-precision spatial model data. This provides a precise foundation for subsequent virtual-real fusion, avoiding task-environment matching deviations caused by environmental information distortion. 3D spatial alignment achieves virtual-real fusion of the task flowchart and the physical environment, allowing task logic to be intuitively combined with the real scene, reducing the difficulty for users to judge the matching situation. Overlaying AR spatial annotations supplements key information such as action boundary constraint prompts, reducing the possibility of users overlooking environmental constraints. Viewpoint tracking and adaptive adjustment of annotation status ensure that information is clear and unobstructed from different angles, avoiding misinterpretation of information due to changes in viewpoint. Projecting onto AR devices and supporting gesture and voice interaction enhances operational flexibility, allowing users to fully observe matching details, avoiding blind spots caused by fixed viewing methods, and further improving the accuracy of task execution intent matching with the physical environment.
[0043] In one embodiment of the present invention, step S22 includes:
[0044] Acquire physical environment spatial model data and initial task flowchart data, standardize the format of the two types of data, and generate dual-source data in a unified format;
[0045] Based on dual-source data in a unified format, a three-dimensional coordinate transformation algorithm is used to perform coordinate system unification calculations to generate dual-source data with aligned coordinate systems.
[0046] Based on the coordinate-aligned dual-source data, we perform association matching operations between task nodes and physical environment locations to generate location matching mapping data.
[0047] Based on the location matching mapping data, a scale ratio calibration operation is performed to eliminate the scale difference between the two types of data and generate scale-calibrated virtual-real fusion data.
[0048] Based on the scale-calibrated virtual-real fusion data, a fusion integrity verification operation is performed to correct matching deviations and generate basic data for the spatial visualization task flowchart.
[0049] The working principle and effects of the above technical solution are as follows: By standardizing the format of the physical environment spatial model data and the initial task flowchart data, the format barriers between the two types of data are eliminated, avoiding the problem of subsequent calculation anomalies caused by inconsistent formats. Three-dimensional coordinate transformation is used to achieve coordinate system unification, eliminating matching misalignment caused by different coordinate systems and improving the alignment accuracy of the dual-source data. Associating and matching task nodes with physical environment locations is performed, ensuring that task logic accurately corresponds to real spatial locations and reducing location mapping deviations. Scale calibration is performed to eliminate scale differences, avoiding scale distortion after virtual-real fusion and enhancing the realism of the fused data. Fusion integrity verification corrects matching deviations, further improving the accuracy of virtual-real fusion, generating reliable basic data for spatial visualization task flowcharts, providing solid support for subsequent AR annotation and user observation and judgment, and reducing problems in subsequent stages caused by deviations in basic data.
[0050] In one embodiment of the present invention, S3 includes:
[0051] S31. The interactive perception module captures the user's operations on the visualized task flowchart data with precise spatial annotations in real time. The operations include drag-and-drop editing, node addition and deletion, logical branch adjustment, and parameter modification. The operation information is recorded, including operation type, operation object, and adjustment range. The task flowchart modification operation data is generated.
[0052] S32. Based on the task flowchart, modify the operation data and use the operation instruction parsing engine to transform the user's visual operation into a structured task logic adjustment instruction to determine the core requirements, which include adding / deleting action units, changing the timing relationship, adjusting constraints, and updating parameters, and generate task flowchart modification instruction data.
[0053] S33. Modify instruction data according to the task flowchart, call the task logic update algorithm to make targeted adjustments to the action unit sequence, logical dependency relationship and parameter configuration information in the initial task flowchart data, and generate temporary optimized task flowchart data.
[0054] S34. Perform logical conflict detection and consistency verification on the temporary optimized task flowchart data, identify and correct problems such as action unit compatibility issues, timing breaks, and constraint contradictions that arise after the adjustment, and generate optimized task flowchart data.
[0055] S35. Synchronize the optimized task flowchart data to the AR / 3D visualization interface, refresh the flowchart display status in real time, allow users to confirm the adjustment effect a second time, and generate adjustment confirmation feedback data.
[0056] The working principle and effects of the above technical solution are as follows: The interactive perception module captures various user editing operations in real time and records detailed information, accurately capturing the user's adjustment intentions and avoiding adjustment deviations caused by missing operation information. Based on the parsing of operation data, it transforms the data into structured adjustment instructions, locating core needs, reducing instruction interpretation errors, and ensuring subsequent adjustments better meet user expectations. Algorithms are invoked to specifically adjust action unit sequences, logical dependencies, and other information, improving the accuracy of adjustments and reducing redundancy caused by irrelevant modifications. Logical conflict detection and consistency verification are performed on temporary optimization data to correct compatibility issues, timing breaks, and other potential problems, preventing hidden logical vulnerabilities in the adjusted task flow from affecting subsequent execution. The optimization data is synchronized to a visual interface and allows for secondary confirmation, enabling users to intuitively grasp the adjustment effects and preventing improper adjustments from proceeding to the next stage without verification, further improving the reliability of the task flow diagram optimization.
[0057] In one embodiment of the present invention, S33 includes:
[0058] Obtain task flowchart modification instruction data and initial task flowchart data, perform association mapping processing on the two types of data, and generate the associated task data to be adjusted.
[0059] Based on the associated task data to be adjusted, the action unit sequence adjustment requirements are extracted, the action unit sequence rearrangement operation is performed, and the adjusted action unit sequence data is generated.
[0060] Based on the adjusted action unit sequence data, the logical dependency adjustment requirements are analyzed, logical dependency link reconstruction operation is performed, and reconstructed logical dependency data is generated.
[0061] Based on the reconstructed logical dependency data, match the parameter configuration adjustment rules, perform parameter update operations, and generate intermediate flowchart data with updated parameters.
[0062] Based on the intermediate data of the flowchart after parameter updates, data integration and calculation are performed to correct data correlation deviations and generate temporary optimized task flowchart data.
[0063] The working principle and effects of the above technical solution are as follows: By associating and mapping the task flowchart modification instruction data with the initial task flowchart data, the adjustment requirements are precisely matched with the original task data, eliminating the disconnect between the two types of data and avoiding modification deviations caused by not finding the corresponding association during adjustment. Based on the associated data, the action unit sequence adjustment requirements are extracted and rearranged, making the action sequence adjustment more aligned with the user's instruction intent and reducing logical misalignments caused by sequence chaos. Logical dependency adjustment requirements are parsed and the chain is reconstructed to ensure the smooth and coherent dependency relationships between actions after adjustment, avoiding task execution stuttering caused by dependency breaks. Update operations are performed by matching parameter configuration rules, improving the accuracy of parameter adjustments and reducing action execution anomalies caused by parameter mismatches. Intermediate data is integrated and association deviations are corrected to ensure the internal consistency of temporary optimization data, avoiding data contradictions affecting subsequent logical verification stages and laying the foundation for generating a reliable optimized task flowchart.
[0064] In one embodiment of the present invention, step S4 includes:
[0065] S41. Extract the key action parameters corresponding to each action unit from the optimized task flow chart data. The key action parameters include execution position coordinates, motion trajectory parameters, attitude control parameters, execution time window, and equipment resource requirements, and generate a standardized action parameter dataset.
[0066] S42. Associate the standardized motion parameter dataset with the physical environment space model data, and use the 3D collision detection engine to simulate the collision risk of the robot with environmental obstacles, its own joints, and other equipment during the robot's execution, and generate the original collision detection result data.
[0067] S43. Classify the original collision detection results data into risk levels (high / medium / low risk) and trace the cause of the collision, clarify the time node, location coordinates and trigger parameters of the collision, and generate collision detection analysis report data;
[0068] S44. Based on the collision detection analysis report data and combined with the motion parameter optimization algorithm, adjust the motion parameters that pose a collision risk. The adjustment of the motion parameters includes optimizing the motion trajectory, avoiding the collision area, adjusting the execution sequence, and modifying the attitude angle, and generating the corrected task flowchart data.
[0069] S45. Perform secondary collision detection and execution feasibility verification on the revised task flowchart data to ensure that there is no collision risk and that the task objectives, equipment performance, environmental constraints and other requirements are met, and generate the final confirmed task flowchart data.
[0070] The working principle and effects of the above technical solution are as follows: Standardized motion parameters are extracted to generate a unified dataset, ensuring parameter standardization and consistency, and avoiding deviations in subsequent collision detection caused by inconsistent parameter formats. Collision detection is performed by correlating the parameter data with the physical environment model, simulating and identifying potential collision risks during execution in advance, preventing collisions between the robot and obstacles, its own joints, or other equipment during actual operation, reducing equipment damage and task interruptions. Risk levels are classified based on the detection results, and key information is traced to accurately locate collision triggers, improving the targeting of parameter adjustments and avoiding ineffective operations caused by blind adjustments. Algorithm optimization of risky motion parameters further reduces the probability of collisions and enhances the safety of task execution. Secondary collision detection and feasibility verification further identify residual risks, ensuring that the task process conforms to the objectives and environmental constraints, preventing unqualified processes from entering the execution stage, and providing a solid guarantee for the robot to complete tasks safely and reliably.
[0071] In one embodiment of the present invention, S42 includes:
[0072] Acquire standardized motion parameter datasets and physical environment spatial model data, perform data association and fusion processing, and generate an associated and fused dataset;
[0073] Based on the associated fusion dataset, a collision detection scene model framework is constructed, and basic data for the scene model is generated.
[0074] Import motion-related parameters from standardized motion parameter data into the basic data of the scene model to generate motion trajectory simulation data;
[0075] Based on the environmental element data in the motion trajectory simulation data and the basic data of the scene model, a 3D collision detection operation is initiated to generate preliminary collision detection data.
[0076] Redundancy removal and validity verification are performed on the preliminary collision detection data, and the data is integrated to generate the original collision detection result data.
[0077] The working principle and effects of the above technical solution are as follows: By fusing standardized motion parameter datasets with physical environment spatial model data, motion parameters and environmental information are precisely matched, eliminating the disconnect between the two types of data and avoiding detection deviations caused by data mismatch during collision detection. A collision detection scene model framework is constructed based on the fused data, generating scene base data that closely matches the real environment, preventing scene model distortion from affecting the realism of collision simulation. Motion-related parameters are imported into the scene model to generate motion trajectory simulation data, ensuring the simulated trajectory closely matches the actual execution path, reducing the risk of missed collisions due to trajectory deviations. Three-dimensional collision detection calculations are initiated to capture preliminary collision information, proactively identifying potential collision hazards and preventing sudden collisions during actual robot execution. Redundancy removal and validity verification are performed on the preliminary detection data to filter out reliable detection information, preventing invalid data from interfering with subsequent risk analysis, improving the accuracy of the original collision detection results, and providing solid support for subsequent risk level classification and cause tracing.
[0078] In one embodiment of the present invention, S44 includes:
[0079] Acquire collision detection analysis report data and optimized task flowchart data, perform correlation matching processing on the two types of data, and generate correlated risk action parameter data;
[0080] Based on the associated risk action parameter data, the collision risk type and corresponding action parameter item are extracted to generate risk parameter classification data.
[0081] Based on risk parameter classification data, the corresponding action parameter optimization algorithm module is matched to generate algorithm matching configuration data;
[0082] The matching optimization algorithm module is invoked to perform adjustment operations on different types of risk action parameters, generating an adjusted action parameter set;
[0083] The adjusted action parameter set is integrated into the optimized task flowchart data, and a data consistency correction operation is performed to generate the corrected task flowchart data.
[0084] The working principle and effects of the above technical solution are as follows: By associating collision detection analysis report data with optimized task flow diagram data, risk information and task parameters are precisely matched, avoiding modification deviations caused by not finding corresponding associations during adjustments. Collision risk types and corresponding action parameter items are extracted to accurately locate the core parameters that need adjustment, reducing ineffective operations caused by blind adjustments. Matching corresponding action parameter optimization algorithm modules ensures that different types of risk parameters can receive suitable adjustment solutions, improving the targeting and accuracy of parameter adjustments. Adjustment calculations are performed on risk parameters separately, effectively reducing collision risks and enhancing the safety of task execution. The adjusted parameter set is integrated and data consistency is corrected to ensure the internal coherence of the corrected task flow diagram data, avoiding data contradictions affecting subsequent secondary collision detection, and laying the foundation for generating a reliable final task flow.
[0085] In one embodiment of the present invention, step S5 includes:
[0086] S51. Based on the final confirmed task flowchart data, the structured task logic is transformed into low-level execution instructions that can be recognized by the robot control system through the instruction generation engine. The low-level execution instructions include motion control instructions, parameter configuration instructions, safety assurance instructions and exception handling instructions, thereby generating robot execution instruction data.
[0087] S52. Through a high-speed communication protocol, including industrial Ethernet, 5G, etc., the robot execution command data is sent to the robot control system in real time, establishing a two-way data transmission channel for command issuance and status feedback, driving the robot to start execution according to the optimized and confirmed task logic, and generating command issuance confirmation data.
[0088] S53. During the robot's execution, the robot's execution status data is collected in real time through the robot's body sensors and environmental sensors. The execution status data includes position information, posture data, execution progress, equipment operating status, and abnormal alarm information, achieving millisecond-level data collection and uploading.
[0089] S54. Based on the execution status data, call the visualization feedback engine to dynamically update the execution node status (not executed / in execution / completed / abnormal), progress bar display, and abnormal warning indicators of the task flowchart in the AR / 3D spatial visualization interface, and simultaneously display the robot's real-time position and attitude in the physical environment, generating real-time execution feedback visualization data;
[0090] S55. Supports users to perform emergency intervention based on real-time execution feedback visualization data. The emergency intervention includes pausing the task, adjusting parameters, and terminating execution. After the intervention command is parsed, it is synchronized to the robot control system in real time, forming a closed-loop interaction of command issuance, status acquisition, visualization feedback and manual intervention, and generating interactive closed-loop human-machine interaction data.
[0091] The working principle and effects of the above technical solution are as follows: It transforms structured task logic into low-level execution instructions that the robot can recognize, covering all dimensions of instruction requirements such as motion control parameter configuration, avoiding execution obstacles caused by instruction incompatibility. Real-time instruction delivery is achieved through a high-speed communication protocol, establishing a bidirectional data transmission channel to reduce instruction transmission delays or loss, ensuring the robot smoothly starts and executes according to the optimized logic. Millisecond-level acquisition of execution status data comprehensively captures information such as position, posture, and progress, preventing the omission of abnormal situations. An AR visualization interface dynamically updates the execution node status and real-time position, allowing users to intuitively grasp task progress and reducing monitoring difficulty. It supports emergency intervention and forms a closed-loop interaction, enabling rapid response to sudden problems and preventing the escalation of faults. This ensures both robot execution safety and improves the flexibility and reliability of human-machine interaction, ensuring that the entire task execution process is controllable and adjustable.
[0092] In one embodiment of the present invention, S52 includes:
[0093] Acquire robot execution command data, identify available high-speed communication protocol types, perform protocol compatibility screening, and generate protocol matching configuration data;
[0094] Based on the protocol matching configuration data, the robot's execution command data is processed to convert the protocol format and generate standardized command data that adapts to the target protocol;
[0095] By using a matching high-speed communication protocol, a transmission request is sent to the robot control system to generate instruction transmission initiation data;
[0096] Receives instructions and response signals from the robot control system, establishes a two-way data transmission link between instruction issuance and status feedback, and generates link establishment completion data;
[0097] Based on the data indicating the completion of the link establishment, the robot control system is triggered to load standardized instruction data, which drives the robot to start task execution. The transmission status and start information are integrated to generate instruction issuance confirmation data.
[0098] The working principle and effects of the above technical solution are as follows: It identifies available high-speed communication protocols and performs compatibility screening to accurately match compatible transmission protocols, avoiding instruction transmission failures or delays caused by protocol incompatibility. It converts instruction data into standardized instructions through protocol format conversion, ensuring that the instructions can be accurately recognized by the robot control system and reducing parsing anomalies caused by format mismatches. It initiates a transmission request by matching the protocol, and establishes a bidirectional data transmission link after receiving the response signal, ensuring smooth interaction between instruction issuance and status feedback, and avoiding the problem of execution status not being able to be returned due to one-way transmission. Based on the established link, the data triggers the control system to load the instruction and drive its execution. It integrates the transmission status and startup information to generate issuance confirmation data, making the instruction issuance status clear and traceable, avoiding the chaos caused by starting execution without confirmation, and improving the reliability of instruction issuance and execution startup.
[0099] One embodiment of the present invention, such as Figure 2 As shown, a system for implementing the human-computer interaction method based on VLA for real-time task adjustment input and visual feedback as described above is provided, the system comprising:
[0100] Command receiving module: Receives natural language commands input by the user, generates a robot task plan using the VLA model, and automatically parses the task plan into a high-level semantic action unit sequence to generate structured task logic data; Based on the structured task logic data, it constructs a drag-and-drop editable graphical task flowchart to generate initial task flowchart data.
[0101] Spatial Alignment Module: Using AR / 3D spatial visualization technology, the initial task flowchart data is aligned and mapped with the real physical environment in three dimensions to generate spatially visualized task flowchart data; AR spatial annotations are superimposed on the spatially visualized task flowchart data to generate visualized task flowchart data with spatial annotations, which allows users to intuitively judge the matching between the task execution intention and the physical environment.
[0102] Data update module: Based on the user's drag-and-drop editing operations on the spatially labeled visual task flowchart data, generate task flowchart modification instruction data; parse the task flowchart modification instruction data and convert it into corresponding structured task logic adjustment data; update the initial task flowchart data based on the structured task logic adjustment data to generate optimized task flowchart data.
[0103] Detection and Analysis Module: Extracts action parameter data required for task execution from the optimized task flowchart data; performs collision detection analysis on the action parameter data and the real physical environment to generate collision detection result data; and further corrects the optimized task flowchart data based on the collision detection result data to generate the final confirmed task flowchart data.
[0104] Feedback Execution Module: Based on the final confirmed task flowchart data, it generates robot execution instruction data; sends the robot execution instruction data to the robot control system to drive the robot to execute safely according to the optimized and confirmed task logic, while providing real-time feedback on execution status data; based on the execution status data, it dynamically updates the execution progress of the task flowchart in the AR / 3D spatial visualization interface, generating real-time execution feedback visualization data.
[0105] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.
Claims
1. A human-computer interaction method based on VLA for real-time task adjustment input and visual feedback, characterized in that, The method includes: S1. Receive natural language instructions input by the user, generate robot task plan using VLA model, parse it into high-level semantic action unit sequence, generate structured task logic data, construct a drag-and-drop editable graphical task flowchart, and generate initial task flowchart data. S2. Using AR / 3D spatial visualization technology, the initial task flowchart data is aligned and mapped with the real physical environment in three dimensions to generate spatially visualized task flowchart data. AR spatial annotations are then overlaid to generate visualized task flowchart data with spatial annotations. S3. Based on the user's drag-and-drop editing operation on the spatially labeled visual task flowchart data, generate task flowchart modification instruction data, parse it, convert it into corresponding structured task logic adjustment data, update the initial task flowchart data, and generate optimized task flowchart data. S4. Extract the action parameter data required for task execution from the optimized task flowchart data; perform collision detection analysis between the action parameter data and the real physical environment to generate collision detection result data, and further revise the optimized task flowchart data to generate the final confirmed task flowchart data. S5. Based on the final confirmed task flowchart data, generate robot execution instruction data; send it to the robot control system to drive the robot to execute safely according to the optimized and confirmed task logic, while providing real-time feedback on execution status data, dynamically updating the execution progress of the task flowchart in the AR / 3D spatial visualization interface, and generating real-time execution feedback visualization data.
2. The human-computer interaction method based on VLA for real-time task adjustment input and visual feedback according to claim 1, characterized in that, S1 includes: S11. Receive natural language instructions input by the user, perform intent recognition and contextual analysis through the multimodal semantic understanding module of the VLA model, and generate raw task description data; S12. Based on the original task description data, call the VLA model's task planning engine, combine the preset action unit library and logic rule library to generate a robot task plan and generate structured task plan data. S13. Based on the structured task plan data, use natural language and action unit mapping algorithms to automatically parse the high-level task logic into a high-level semantic action unit sequence to generate structured task logic data. S14. Based on structured task logic data, call the graphical rendering engine to build a visual task flowchart framework, define the display rules and interaction logic of action nodes, judgment nodes, and branch nodes, and generate initial task flowchart prototype data. S15. Perform syntax validation and logical integrity checks on the initial task flowchart prototype data to generate the initial task flowchart data.
3. The human-computer interaction method based on VLA for real-time task adjustment input and visual feedback according to claim 1, characterized in that, The S2 includes: S21. Obtain initial task flowchart data, collect spatial information of the real physical environment through AR spatial positioning module, and generate high-precision physical environment spatial model data. S22. Based on the physical environment spatial model data and the initial task flowchart data, use the three-dimensional spatial alignment algorithm to perform coordinate system one, logical position matching, and scale calibration to generate basic data for spatial visualization task flowchart. S23. On the basic data of the spatial visualization task flowchart, overlay AR spatial annotation information, define the display style and interactive trigger conditions of the annotations, and generate visualization task flowchart data with preliminary spatial annotations. S24. Through user perspective tracking and scene adaptation algorithms, adjust the transparency, size, and position of AR spatial annotations to generate visual task flowchart data with accurate spatial annotations; S25. Project the visualized task flowchart data with precise spatial annotations to the AR / 3D display device in real time to generate scene-based interactive preview data.
4. The human-computer interaction method based on VLA for real-time task adjustment input and visual feedback according to claim 3, characterized in that, S22 includes: Acquire physical environment spatial model data and initial task flowchart data, standardize the format of the two types of data, and generate dual-source data in a unified format; Based on dual-source data in a unified format, a three-dimensional coordinate transformation algorithm is used to perform coordinate system unification calculations to generate dual-source data with aligned coordinate systems. Based on the coordinate-aligned dual-source data, we perform association matching operations between task nodes and physical environment locations to generate location matching mapping data. Based on the location matching mapping data, a scale ratio calibration operation is performed to eliminate the scale difference between the two types of data and generate scale-calibrated virtual-real fusion data. Based on the scale-calibrated virtual-real fusion data, a fusion integrity verification operation is performed to correct matching deviations and generate basic data for the spatial visualization task flowchart.
5. The human-computer interaction method based on VLA for real-time task adjustment input and visual feedback according to claim 1, characterized in that, The S3 includes: S31. The interactive perception module captures the user's operations on the visualized task flowchart data with precise spatial annotations in real time, records the operation information, and generates task flowchart modification operation data. S32. Based on the task flowchart, modify the operation data, use the operation instruction parsing engine to convert the user's visual operation into a structured task logic adjustment instruction, determine the core requirements, and generate task flowchart modification instruction data. S33. Call the task logic update algorithm to make targeted adjustments to the action unit sequence, logical dependency relationship and parameter configuration information in the initial task flowchart data, and generate temporary optimized task flowchart data; S34. Perform logical conflict detection and consistency verification on the temporary optimized task flowchart data, and generate optimized task flowchart data. S35. Synchronize the optimized task flowchart data to the AR / 3D visualization interface, refresh the flowchart display status in real time, allow users to confirm the adjustment effect a second time, and generate adjustment confirmation feedback data.
6. The human-computer interaction method based on VLA for real-time task adjustment input and visual feedback according to claim 5, characterized in that, S33 includes: Obtain task flowchart modification instruction data and initial task flowchart data, perform association mapping processing on the two types of data, and generate the associated task data to be adjusted. Based on the associated task data to be adjusted, the action unit sequence adjustment requirements are extracted, the action unit sequence rearrangement operation is performed, and the adjusted action unit sequence data is generated. Based on the adjusted action unit sequence data, the logical dependency adjustment requirements are analyzed, logical dependency link reconstruction operation is performed, and reconstructed logical dependency data is generated. Based on the reconstructed logical dependency data, match the parameter configuration adjustment rules, perform parameter update operations, and generate intermediate flowchart data with updated parameters. Based on the intermediate data of the flowchart after parameter updates, data integration and calculation are performed to correct data correlation deviations and generate temporary optimized task flowchart data.
7. The human-computer interaction method based on VLA for real-time task adjustment input and visual feedback according to claim 1, characterized in that, The S4 includes: S41. Extract the key action parameters corresponding to each action unit from the optimized task flow chart data and generate a standardized action parameter dataset. S42. Associate the standardized motion parameter dataset with the physical environment space model data, and use the 3D collision detection engine to simulate the collision risk of the robot with environmental obstacles, its own joints, and other equipment during the robot's execution, and generate the original collision detection result data. S43. Classify the risk level and trace the cause of collisions in the original collision detection results data, clarify the time node, location coordinates and triggering parameters of the collision, and generate collision detection analysis report data. S44. Based on the collision detection analysis report data and combined with the motion parameter optimization algorithm, adjust the motion parameters that pose a collision risk and generate the corrected task flow chart data. S45. Perform secondary collision detection and execution feasibility verification on the revised task flowchart data to generate the final confirmed task flowchart data.
8. The human-computer interaction method based on VLA for real-time task adjustment input and visual feedback according to claim 7, characterized in that, S42 includes: Acquire standardized motion parameter datasets and physical environment spatial model data, perform data association and fusion processing, and generate an associated and fused dataset; Based on the associated fusion dataset, a collision detection scene model framework is constructed, and basic data for the scene model is generated. Import motion-related parameters from standardized motion parameter data into the basic data of the scene model to generate motion trajectory simulation data; Based on the environmental element data in the motion trajectory simulation data and the basic data of the scene model, a 3D collision detection operation is initiated to generate preliminary collision detection data. Redundancy removal and validity verification are performed on the preliminary collision detection data, and the data is integrated to generate the original collision detection result data.
9. The human-computer interaction method based on VLA for real-time task adjustment input and visual feedback according to claim 1, characterized in that, The S5 includes: S51. Based on the final confirmed task flowchart data, the structured task logic is transformed into low-level execution instructions that can be recognized by the robot control system through the instruction generation engine, generating robot execution instruction data. S52. Through a high-speed communication protocol, the robot execution command data is sent to the robot control system in real time, establishing a two-way data transmission channel for command issuance and status feedback, driving the robot to start execution according to the optimized and confirmed task logic, and generating command issuance confirmation data. S53. During the robot's execution, the robot's execution status data is collected in real time through the robot's body sensors and environmental sensors, achieving millisecond-level data acquisition and uploading; S54. Based on the execution status data, call the visualization feedback engine to dynamically update the execution node status, progress bar display, and abnormal warning indicators of the task flowchart in the AR / 3D spatial visualization interface, and simultaneously display the robot's real-time position and attitude in the physical environment to generate real-time execution feedback visualization data. S55. Supports users to perform emergency intervention based on real-time execution feedback visualization data. After the intervention command is parsed, it is synchronized to the robot control system in real time to form a closed-loop interaction and generate interactive closed-loop human-machine interaction data.
10. A system for implementing the human-computer interaction method for real-time task adjustment input and visual feedback based on VLA as described in claim 1, characterized in that, The system includes: Command receiving module: Receives natural language commands input by the user, generates robot task plans using the VLA model, parses them into high-level semantic action unit sequences, generates structured task logic data, constructs a drag-and-drop editable graphical task flowchart, and generates initial task flowchart data. Spatial alignment module: Using AR / 3D spatial visualization technology, the initial task flowchart data is aligned and mapped with the real physical environment in three dimensions to generate spatially visualized task flowchart data; AR spatial annotations are superimposed to generate visualized task flowchart data with spatial annotations. Data update module: Based on the user's drag-and-drop editing operation on the spatially labeled visual task flowchart data, generate task flowchart modification instruction data, parse it, convert it into corresponding structured task logic adjustment data, update the initial task flowchart data, and generate optimized task flowchart data. Detection and Analysis Module: Extracts action parameter data required for task execution from the optimized task flowchart data; performs collision detection analysis on the action parameter data and the real physical environment to generate collision detection result data; and further corrects the optimized task flowchart data to generate the final confirmed task flowchart data. Feedback Execution Module: Based on the final confirmed task flowchart data, it generates robot execution instruction data; sends it to the robot control system to drive the robot to execute safely according to the optimized and confirmed task logic, and at the same time provides real-time feedback on execution status data, dynamically updates the execution progress of the task flowchart in the AR / 3D spatial visualization interface, and generates real-time execution feedback visualization data.