A robot multimodal interaction method and system based on a vision-language large model

Abstract

The present application relates to the technical field of robot intelligent interaction and multi-modal understanding, and particularly relates to a robot multi-modal interaction method and system based on a vision-language large model. The present application obtains multi-modal interaction data and constructs environment state information through a camera, a microphone and a body sensor; performs speech recognition on voice data and inputs vision data into a vision-language large model for multi-modal intention understanding analysis in combination with the vision data to obtain intention structured information; performs task planning analysis based on the multi-modal intention understanding analysis result to generate a task sequence and a planning constraint set; performs safety execution risk prediction based on the task planning analysis result, and generates an execution constraint configuration according to the risk prediction result; further performs execution control determination based on the execution constraint generation analysis result to output a control instruction set and a control mode, and realizes chained closed-loop processing from perception, understanding, planning, risk assessment to constraint generation and control determination.

Description

Technical Field

[0001] This invention relates to the field of neural network technology, and in particular to a method, system, and storage medium for multimodal robot interaction based on a large vision-language model. Background Technology

[0002] With the increasing application of service robots, warehouse handling robots, and companion robots in indoor and outdoor scenarios, human-computer interaction has gradually evolved from traditional buttons and remote controls to multimodal interaction methods such as voice, vision, and touch. In existing technologies, a common approach is to use a voice recognition module to convert user speech into text commands, then combine this with a preset command template or intent classification model to complete intent recognition; simultaneously, cameras are used for target detection, localization, and scene understanding to assist the robot in performing tasks such as grasping, navigation, and delivery. In more complex applications, large models or vision-language models are also integrated to enhance the understanding of natural language commands and scene semantics, thereby supporting more flexible interactive expressions.

[0003] However, existing multimodal interaction solutions typically suffer from the following shortcomings: First, there is a lack of a unified process for timestamping, timing alignment, and calibration of multi-source sensor data. Asynchrony between voice, vision, and ontology states can easily lead to errors in command object recognition, misjudgment of spatial relationships, and inconsistencies in action triggering timing, resulting in unstable interaction results. Second, intent understanding is often disconnected from task planning. The fusion of voice / text and visual scene information is insufficient, and there is a lack of structured extraction and consistency verification of task objects, action requirements, and constraints, easily leading to inconsistent interpretations of the same command in different scenarios. Third, existing technologies rely heavily on fixed rules or empirical thresholds for safe execution. Risk assessment and planning processes are insufficiently coupled, making it difficult to uniformly predict risks related to kinematics, dynamics, resource constraints, and environmental obstacles after task sequence generation, resulting in delayed safety strategies. Fourth, there is a lack of mechanisms to further transform risk prediction results into execution constraints, making it impossible to form constraint configurations that can be directly issued to the control layer. Common practices include internal processing by the controller or reliance on manual configuration, leading to poor system scalability and reusability. Fifth, when faced with changes in risk, the control layer often adopts a single stop or simple obstacle avoidance strategy, lacking control mode determination and instruction set selection based on constraint generation results, making it difficult to maintain mission continuity while ensuring safety.

[0004] Therefore, how to achieve a chain-like closed-loop processing for vision-language understanding, task planning, security risk prediction, execution constraint generation and control judgment based on multimodal data acquisition and alignment, so as to improve interaction consistency and enhance execution security, remains a problem that existing technologies need to solve. Summary of the Invention

[0005] To achieve the above objectives, the present invention adopts the following technical solution: To achieve the above objectives, the present invention adopts the following technical solution:

[0006] In a first aspect, the present invention provides a robot multimodal interaction method based on a large vision-language model, comprising the following steps: Step S1: Acquire multimodal interaction data and construct environmental state information through camera, microphone and body sensor; Step S2: Perform speech recognition on the speech data and combine it with visual data to input into the visual-language big data model for multimodal intent understanding analysis; Step S3: Perform task planning analysis based on the results of multimodal intent understanding analysis; Step S4: Predict safety execution risks based on task planning analysis results; Step S5: Analyze and generate execution constraints based on the results of the safety execution risk prediction. Step S6: Execution control determination is made by generating analysis results through execution constraints.

[0007] In one implementation of the present invention, step S1 includes the following specific contents: configuring a camera, microphone, and body sensor as multimodal data sources, and determining the sampling frequency, data format, and unified time reference corresponding to the camera, microphone, and body sensor; acquiring visual image or video frame data through the camera, acquiring speech waveform data through the microphone, and acquiring joint position, joint speed, posture, torque, and power status data through the body sensor; timestamping the visual data, speech data, and body status data, and performing time-series alignment of the visual data, speech data, and body status data based on the unified time reference, while simultaneously completing the camera intrinsic parameter calibration, extrinsic parameter calibration, and body sensor zero-bias calibration; and performing noise reduction processing and missing value completion on the aligned multimodal data. The system performs outlier removal and sampling rate consistency processing on visual and audio data. Environmental element information is extracted from the aligned visual data, including target object information, target spatial location and scene area information, as well as obstacle boundaries and passable area information. Robot body state information is constructed based on the aligned body state data, including pose information, motion state information, and actuator state information. The environmental element information and robot body state information are fused to form environmental state information in a unified coordinate system, and the environmental state information, including target object information, spatial position relationship information, obstacle information, and robot body state information, is output, thus constructing the environmental state information.

[0008] In one implementation of the present invention, the multimodal intent understanding and analysis in step S2 includes the following specific steps: S21. Acquire the voice data collected by the microphone and the visual data collected by the camera, and read the environmental state information output in step S1 as the context data source; perform endpoint detection on the voice data to determine the start and end intervals of the voice segment, and perform noise reduction and sampling rate consistency processing on the voice data. At the same time, align the voice segment and the visual frame with timestamps based on a unified time reference to obtain aligned voice data and aligned visual data. S22. The aligned speech data is processed by a speech recognition model to obtain text instructions, and the text instructions are processed by word segmentation, entity extraction and semantic normalization in sequence to generate text parsing information; the aligned visual data is processed by target detection and scene element extraction to generate scene element information, which includes target category, target location, target attributes and obstacle boundary information. S23. After obtaining the text parsing information and scene element information, construct the text parsing information, scene element information and environmental state information into a joint input according to the preset input format, and input the joint input into the visual-language large model; S24. Reason the joint input through the visual-language big model to obtain the intention candidate set and corresponding confidence level, and construct the intention candidate information by combining the intention candidate set and confidence level; filter the intention candidate information based on the confidence level threshold to obtain the intention recognition result, and extract the task object, action requirements and constraints based on the intention recognition result to generate structured intention information. S25. Perform consistency verification between the structured intent information, scene element information, and environmental state information, and output multimodal intent understanding analysis results based on the verification results. The multimodal intent understanding analysis results include intent labels, task objects, action requirements, constraints, confidence levels, and verification mark fields to obtain multimodal intent understanding analysis results.

[0009] In one implementation of the present invention, the task planning analysis in step S3 includes the following specific contents: S31. Obtain the multimodal intent understanding analysis results output in step S2, and extract intent labels, task objects, action requirements and constraints from the multimodal intent understanding analysis results as planning inputs; S32. Obtain the environmental state information and robot capability parameters output in step S1, and perform coordinate system consistency processing and unit standardization processing on the environmental elements in the task object and environmental state information to form planning input data with unified caliber; construct a planning constraint set based on intention tags, action requirements and constraints, the planning constraint set including safety distance constraints, speed upper limit constraints, torque upper limit constraints and resource constraints; after constructing the planning constraint set, obtain the skill library, action library or task template library, and generate candidate task sequences and corresponding key parameter sets based on task objects and action requirements, wherein the key parameter set is used to characterize the execution parameters and execution boundaries of each task step; S33. Perform feasibility screening on the candidate task sequences to eliminate candidate task sequences that do not meet the planning constraint set, environmental state information or robot capability parameters, and obtain the screened candidate task sequences. Perform cost evaluation on the screened candidate task sequences based on preset cost items, and perform weighted summation on the cost items based on preset weights to obtain the candidate score. S34. Based on the candidate scores, sort and select the candidate task sequences to obtain the preferred task sequences and corresponding key parameters. Encapsulate the preferred task sequences, key parameters, and constraints and output the task planning analysis results. The task planning analysis results include task sequences, key parameters, planning constraint sets, candidate scores, and execution order fields to obtain the task planning analysis results.

[0010] In one implementation of the present invention, the risk prediction in step S4 includes the following specific contents: S41. Obtain the task planning analysis results output in step S3, and extract the task sequence, key parameters, planning constraint set and execution order relationship from the task planning analysis results as risk prediction input; obtain the environmental state information and robot capability parameters output in step S1, and perform coordinate system consistency processing and unit standardization processing on the target objects and spatial position relationships involved in the task sequence to form risk prediction input data with unified caliber. S42. Construct a risk assessment input set based on task sequence and environmental state information. The risk assessment input set includes robot pose trajectory, target object position, obstacle boundary, passable area, and actuator state information. Generate a risk element candidate set based on the risk assessment input set. The risk element candidate set includes collision risk elements, boundary crossing risk elements, overload risk elements, and energy shortage risk elements. Perform feasibility screening on the risk element candidate set to eliminate risk element candidates that do not meet safety distance constraints, speed upper limit constraints, torque upper limit constraints, or resource constraints, thereby obtaining the screened risk element set.

[0011] S43. Calculate risk scores for the selected risk factor set based on preset cost items and impact coefficients, and weight and summarize the risk scores based on preset weights to obtain the risk score results; classify the risk scores into risk levels based on preset thresholds, and construct risk level results by combining risk levels with corresponding triggering conditions; encapsulate the risk score results, risk level results, and risk factor information, and output the safe execution risk prediction results, wherein the safe execution risk prediction results include risk level, risk score, risk factor information, triggering conditions, and corresponding task step fields, thus obtaining the safe execution risk prediction results output in this step.

[0012] In one implementation of the present invention, the constraint generation analysis in step S5 includes the following specific contents: S51. Obtain the safety execution risk prediction results output in step S4, and extract the risk level, risk score, risk element information, triggering conditions and corresponding task steps from the safety execution risk prediction results as constraint generation inputs; obtain the task planning analysis results output in step S3 and the environmental status information output in step S1, and perform consistency processing on the task step identifier, coordinate system and unit to form a unified constraint generation input data. S52. Construct a constraint generation input set based on the constraint generation input, the constraint generation input set including risk element type, risk level, key parameters of task sequence, obstacle information and robot body state information; generate an execution constraint candidate set based on the constraint rule base or constraint template base, the execution constraint candidate set including speed upper limit constraint, acceleration upper limit constraint, torque upper limit constraint, path no-entry area constraint and safety distance constraint; perform feasibility screening on the execution constraint candidate set to eliminate execution constraint candidates that do not meet the robot capability parameters, planning constraint set or constraint consistency requirements, to obtain the screened execution constraint candidate set; S53. Based on a preset threshold, the risk level and risk score are mapped to constraint strength to generate a constraint configuration candidate set and configure the candidate associated constraint parameter range and effective conditions for each constraint; the constraint configuration candidate set is evaluated and weighted based on a preset cost item and a preset weight to determine the constraint configuration result; the constraint configuration result is bound to the task step identifier and encapsulated as an execution constraint generation analysis result, wherein the execution constraint generation analysis result includes constraint type, constraint parameter range, effective conditions, corresponding task step and constraint strength fields, to obtain the output execution constraint generation analysis result of this step.

[0013] In one implementation of the present invention, the control determination in step S6 includes the following specific contents: S61. Obtain the execution constraint generation analysis results output in step S5, and extract the constraint type, constraint parameter range, effective conditions, corresponding task steps and constraint strength from the execution constraint generation analysis results as control judgment inputs; obtain the task planning analysis results output in step S3 and the environmental status information output in step S1, and perform consistency processing on the task step identifier, coordinate system and unit to form control judgment input data with unified caliber. S62. Construct a control decision input set based on the control decision input, the control decision input set including the current task step, execution constraint configuration, robot body state information and environmental element information; generate a control strategy candidate set based on the control strategy library, the control strategy candidate set including continue execution strategy, decelerate execution strategy, pause execution strategy, replanning trigger strategy and terminate execution strategy; perform feasibility screening on the control strategy candidate set to eliminate control strategy candidates that do not meet the kinematic constraints, dynamic constraints, planning constraint set or execution constraint consistency requirements, and obtain the screened control strategy candidate set; S63. Trigger the effective conditions based on a preset threshold, and generate a control configuration candidate set based on the trigger determination result; evaluate and weight the control configuration candidate set based on preset cost items and preset weights, determine the control configuration result, and generate a control instruction set; bind the control configuration result, the control instruction set, and the corresponding task steps and encapsulate them into an execution control determination result, wherein the execution control determination result includes the control mode, the control instruction set, the corresponding task steps, the effective conditions, and the constraint reference fields, and obtain the execution control determination result output by this step.

[0014] Secondly, the present invention also provides a robot multimodal interaction system based on a large vision-language model, including: The multimodal data acquisition and environmental status construction module is used to acquire multimodal interaction data and construct environmental status information through cameras, microphones and body sensors; The multimodal intent understanding and analysis module is configured to perform speech recognition on speech data and combine it with visual data to input into a visual-language big model for multimodal intent understanding and analysis, so as to generate multimodal intent understanding and analysis results; The task planning and analysis module is configured to perform task planning and analysis based on the multimodal intent understanding and analysis results to generate task planning and analysis results; The security execution risk prediction module is configured to perform security execution risk prediction based on the task planning analysis results, so as to generate security execution risk prediction results; The constraint generation analysis module is configured to perform execution constraint generation analysis based on the security execution risk prediction results, so as to generate execution constraint generation analysis results; The control determination module is configured to perform execution control determination based on the analysis results generated by the execution constraints, so as to generate execution control determination results.

[0015] Thirdly, the present invention provides an electronic device comprising: a processor and a memory, wherein the memory stores a computer program that can be called by the processor, and the processor executes a robot multimodal interaction method based on a vision-language large model by calling the computer program stored in the memory.

[0016] Fourthly, the present invention provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to execute a robot multimodal interaction method based on a large vision-language model.

[0017] Compared with the prior art, the present invention has the following advantages and beneficial effects: (1) Multimodal interaction data is acquired through cameras, microphones and body sensors, and the multimodal data is timestamped, time-series aligned and sensor calibrated to construct environmental state information, reduce object referencing deviation and spatial relationship misjudgment caused by the asynchrony of voice, vision and robot state, and improve the consistency and stability of multimodal interaction input.

[0018] (2) By constructing the text parsing information obtained from speech recognition and the scene element information and environmental element information extracted from visual data into a visual-language large model joint input according to the preset input format, the intention candidates and confidence scores are output and the filtering and structured extraction are completed. Then, the consistency check is performed to generate multimodal intention understanding analysis results, reducing the intention ambiguity and constraint omission caused by relying only on a single modality or fixed template, and improving the usability and feasibility of the intention understanding results.

[0019] (3) By performing task planning analysis based on the results of multimodal intent understanding analysis, a set of planning constraints is constructed by combining environmental state information and robot capability parameters. Candidate task sequences are generated and screened, and cost evaluation is performed to obtain task planning analysis results. This enables task sequences, key parameters and constraints to form a reusable planning expression, providing clear input for subsequent safety assessment and reducing the risk lag caused by the separation of planning and safety.

[0020] (4) By predicting the safety execution risk based on the task planning analysis results, and further generating the execution constraint configuration based on the safety execution risk prediction results, and finally making the execution control judgment based on the execution constraint generation analysis results, the chain of "risk prediction - constraint generation - control judgment" is realized, so that the risk assessment can be brought forward to the execution and transformed into constraints and control decisions that can be directly issued, thereby improving the safety controllability of the execution process and the consistency of abnormal handling. Attached Figure Description

[0021] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 The flowchart shows a robot multimodal interaction method based on a large vision-language model proposed in this invention. Figure 2 This is a schematic diagram of the speech recognition, visual data processing, and multimodal intent understanding and analysis process based on the visual-language big model in step S2 of the present invention. Figure 3 This is a schematic diagram of the safe execution risk prediction process based on task planning analysis results in step S4 of the present invention; Figure 4 This is a schematic diagram of the execution constraint generation and analysis process based on the prediction results of security execution risks in step S5 of the present invention; Figure 5 This is a schematic diagram of the module composition of a robot multimodal interaction system based on a vision-language large model proposed in this invention. Detailed Implementation

[0022] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention; however, the present invention may be implemented in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Furthermore, the term "an embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single embodiment or an embodiment selectively excluded from other embodiments. Example

[0023] like Figures 1 to 4 As shown, this embodiment provides a method for analyzing defects in power transmission foundation concrete based on acoustic signature signals, specifically including the following steps: Step S1: Acquire multimodal interaction data and construct environmental state information through camera, microphone and body sensor; In this embodiment, step S1 includes the following specific content: Configuring a visual acquisition device, a voice acquisition device, and a body state acquisition device as multimodal data sources, and determining the sampling frequency, data format, and unified time reference corresponding to the visual acquisition device, voice acquisition device, and body state acquisition device; wherein, the visual acquisition device is a color industrial camera or a depth camera, the voice acquisition device is an array microphone, and the body state acquisition device includes at least a joint encoder, an inertial measurement unit, and a torque sensor, and may include motor driver feedback and a battery management system; acquiring visual image or video frame data through the visual acquisition device, acquiring voice waveform data through the voice acquisition device, and acquiring joint position, joint speed, posture, torque, and battery status data through the body state acquisition device; timestamping the visual data, voice data, and body state data, and synchronizing them based on the unified time reference. The robot's body state data is time-aligned, and the intrinsic and extrinsic parameters of the visual acquisition device are calibrated simultaneously. The zero-bias calibration of the inertial measurement unit and the zero-point calibration of the torque sensor are also performed. The aligned multimodal data undergoes denoising, missing value completion, and outlier removal. The sampling rate of the visual and audio data is consistent. Environmental element information is extracted from the aligned visual data, including target object information, target spatial location and scene area information, and obstacle boundary and passable area information. Robot body state information is constructed based on the aligned body state data, including pose information, motion state information, and actuator state information. The environmental element information and robot body state information are fused to form environmental state information in a unified coordinate system, and this environmental state information, including target object information, spatial position relationship information, obstacle information, and robot body state information, is output, thus constructing the environmental state information.

[0024] Step S2: Perform speech recognition on the speech data and combine it with visual data to input into the visual-language big data model for multimodal intent understanding analysis; In this embodiment, it should be specifically explained that step S2 includes the following specific contents: S21. Acquire the voice data collected by the array microphone and the visual data collected by the color industrial camera or depth camera, and read the environmental state information output in step S1 as the context data source; perform voice activity detection on the voice data to determine the start and end intervals of the voice segments, and perform noise reduction, automatic gain control and sampling rate consistency processing on the voice data. The voice activity detection is implemented by an endpoint detection module or a voice activity detection module, and the noise reduction is implemented by a spectral subtraction noise reduction module, a Wiener filter noise reduction module or a neural network noise reduction module; at the same time, align the voice segments and visual frames with timestamps based on a unified time reference. The unified time reference is established through a time synchronization protocol or middleware timestamp mechanism to obtain aligned voice data and aligned visual data. S22. The aligned speech data is processed by a speech recognition model to obtain text instructions. The speech recognition model is an end-to-end speech recognition model or a streaming speech recognition model, and outputs word sequences and confidence information. The text instructions are processed sequentially by word segmentation, entity extraction, and semantic normalization. The entity extraction process outputs task object entities, location entities, and quantity entities. The semantic normalization process outputs unit normalization, synonym normalization, and referential resolution results, generating text parsing information. The aligned visual data is processed by target detection, instance segmentation, and scene element extraction. The target detection is implemented using a single-stage detection model or a two-stage detection model. The instance segmentation is implemented using an instance segmentation model, generating scene element information. The scene element information includes target category, target location, target attributes, and obstacle boundary information. S23. After obtaining the text parsing information and scene element information, construct the text parsing information, scene element information and environmental state information into a joint input according to a preset input format. The preset input format is a structured field template or a message body template. The joint input includes at least a task object field, an action requirement field, a constraint condition field, a scene element field and a robot body state field. Input the joint input into the vision-language large model and carry an alignment timestamp and coordinate system identifier in the input. S24. Reason the joint input through the visual-language big model to obtain the intention candidate set and corresponding confidence level, and construct the intention candidate information by combining the intention candidate set and confidence level; filter the intention candidate information based on the confidence level threshold to obtain the intention recognition result, and extract the task object, action requirements and constraints based on the intention recognition result. The constraints include spatial constraints, speed constraints and interaction constraints, and generate structured intention information. S25. Perform consistency verification between the structured intent information and the scene element information and environmental state information. The consistency verification includes object consistency verification, spatial relationship consistency verification and constraint condition satisfiability verification. Based on the verification results, output multimodal intent understanding analysis results. The multimodal intent understanding analysis results include intent labels, task objects, action requirements, constraints, confidence levels and verification mark fields to obtain multimodal intent understanding analysis results.

[0025] Step S3: Perform task planning analysis based on the results of multimodal intent understanding analysis; In this embodiment, it should be specifically explained that step S3 includes the following specific contents: S31. Obtain the multimodal intent understanding analysis results output in step S2, and extract intent tags, task objects, action requirements and constraints from the multimodal intent understanding analysis results as planning inputs, wherein the task objects include object identifier, object category and object spatial pose fields, and the constraints include spatial constraints, velocity constraints and interaction constraints fields. S32. Obtain the environmental state information and robot capability parameters output in step S1, and perform coordinate system unification and unit normalization processing on the environmental elements in the task object and environmental state information to form unified planning input data; wherein, the coordinate system unification processing is completed based on coordinate transformation tree or extrinsic parameter calibration matrix, and the robot capability parameters are read from robot description file and controller configuration parameters, and at least include kinematic model parameters, dynamic parameters, actuator limiting parameters, end effector parameters, and workspace boundary parameters; after forming unified planning input data, based on intent A planning constraint set is constructed using tags, action requirements, and constraints. This planning constraint set includes safety distance constraints, speed upper limit constraints, torque upper limit constraints, and resource constraints. After constructing the planning constraint set, a skill library, action library, or task template library is obtained. This skill library, action library, or task template library is stored in the form of behavior tree node sets, state machine template sets, or action primitive sets. Based on the task object and action requirements, a candidate task sequence and a corresponding set of key parameters are generated. The set of key parameters includes target pose parameters, path key point parameters, velocity and acceleration parameters, force and torque limiting parameters, and execution timing parameters. S33. Perform feasibility screening on the candidate task sequences to eliminate those that do not meet the planning constraint set, environmental state information, or robot capability parameters, thus obtaining the screened candidate task sequences. The feasibility screening is performed through an inverse kinematics solution module, a collision detection module, and a trajectory validity check module. The reachability, collision constraints, and actuator amplitude constraints of the candidate task sequences are verified for consistency based on the planning scenario data. The screened candidate task sequences are then evaluated based on preset cost items, and the cost items are weighted and summarized based on preset weights to obtain a candidate score. The preset cost items are read from the cost configuration table and include path length cost, time cost, energy consumption cost, and constraint violation penalty cost. S34. Based on the candidate scores, sort and select the candidate task sequences to obtain the preferred task sequences and corresponding key parameters. Encapsulate the preferred task sequences, key parameters, and constraints and output the task planning analysis results. The task planning analysis results are encapsulated in a structured data carrier, which is a message body or a file body, and includes task sequence fields, key parameter fields, planning constraint set fields, candidate score fields, and execution order relationship fields to obtain the task planning analysis results.

[0026] Step S4: Predict safety execution risks based on task planning analysis results; In this embodiment, it should be specifically explained that step S4 includes the following specific contents: In this embodiment, it should be specifically explained that step S4 includes the following specific contents: S41. Obtain the task planning analysis results output in step S3, and extract the task sequence, key parameters, planning constraint set, and execution order relationship from the task planning analysis results as risk prediction input; obtain the environmental state information and robot capability parameters output in step S1, and perform coordinate system consistency processing and unit normalization processing on the target objects and spatial position relationships involved in the task sequence to form risk prediction input data with unified caliber; wherein, the coordinate system consistency processing is completed based on the coordinate transformation tree or extrinsic parameter calibration matrix, the unit normalization processing is completed based on the unit conversion table, and the robot capability parameters are read from the robot controller parameters, robot description file, and actuator configuration parameters, and at least include joint limit parameters, velocity and acceleration limit parameters, torque limit parameters, load parameters, and remaining energy threshold parameters; S42. Construct a risk assessment input set based on task sequence and environmental state information. The risk assessment input set includes robot pose trajectory, target object position, obstacle boundaries, passable area, and actuator state information. The robot pose trajectory is generated based on key parameters and represented as a sequence of trajectory points. The obstacle boundaries and passable area are extracted from the environmental state information and represented as an occupied grid map, cost map, or geometric boundary set. The actuator state information includes joint state, end effector state, and actuator feedback state. Generate a risk element candidate set based on the risk assessment input set. The risk element candidate set includes collision risk elements, boundary crossing risk elements, overload risk elements, and energy shortage risk elements. The candidate set of risk factors undergoes feasibility screening to eliminate those that do not meet safety distance constraints, speed upper limit constraints, torque upper limit constraints, or resource constraints, resulting in a screened set of risk factors. This feasibility screening is accomplished through a collision detection module, a boundary check module, a load check module, and an energy check module. The collision detection module performs collision consistency verification between the robot model and obstacle boundaries based on planning scene data. The boundary check module performs out-of-bounds consistency verification on the pose trajectory based on workspace boundary parameters. The load check module performs overload consistency verification on the actuator state based on torque limit parameters and load parameters. The energy check module performs consistency verification on the energy requirements of the task sequence based on remaining energy threshold parameters. S43. Calculate risk scores for the filtered risk element set based on preset cost items and influence coefficients, and weight and summarize the risk scores based on preset weights to obtain the risk score result; wherein the preset cost items are read from the risk cost configuration table, the influence coefficients are read from the risk element weight table, and the preset weights are read from the risk summary weight table; classify the risk scores into risk levels based on preset thresholds, and construct a risk level result by combining the risk level with the corresponding triggering conditions, wherein the triggering conditions include at least a safe distance triggering condition, a speed limit triggering condition, a torque limit triggering condition, and an energy threshold triggering condition; encapsulate the risk score result, the risk level result, and the risk element information, and output a safe execution risk prediction result, wherein the safe execution risk prediction result includes a risk level, a risk score, risk element information, triggering conditions, and corresponding task step fields, thus obtaining the safe execution risk prediction result output in this step.

[0027] Step S5: Analyze and generate execution constraints based on the results of the safety execution risk prediction. In this embodiment, it should be specifically explained that step S5 includes the following specific contents: S51. Obtain the safety execution risk prediction result output in step S4, and extract the risk level, risk score, risk element information, triggering conditions, and corresponding task steps from the safety execution risk prediction result as constraint generation input; obtain the task planning analysis result output in step S3 and the environmental state information output in step S1, and perform consistency processing on the task step identifier, coordinate system, and units to form a unified constraint generation input data; wherein, the consistency processing of the task step identifier is completed through the task step index table, the consistency processing of the coordinate system is completed through the coordinate transformation relationship table or the external parameter calibration matrix, and the consistency processing of the units is completed through the unit conversion table; S52. Construct a constraint generation input set based on the unified constraint generation input data. The constraint generation input set includes risk element type, risk level, risk score, key parameters of task sequence, obstacle information, passable area information, and robot body state information. Read the constraint rule library or constraint template library, which is stored in the form of a structured configuration table and indexed according to constraint type. Generate an execution constraint candidate set based on the constraint rule library or constraint template library. The execution constraint candidate set includes speed upper limit constraint, acceleration upper limit constraint, torque upper limit constraint, path no-entry area constraint, and safety distance constraint, and associate each execution constraint candidate with a constraint. The constraint parameter range and constraint object fields are defined. Feasibility screening is performed on the candidate set of execution constraints to eliminate those that do not meet the robot capability parameters, planning constraint set, or constraint consistency requirements, resulting in a screened candidate set of execution constraints. The feasibility screening is accomplished through a parameter boundary verification component, a constraint conflict verification component, and a constraint scope verification component. The parameter boundary verification component verifies whether the constraint parameter range falls within the range of actuator limit parameters and load parameters. The constraint conflict verification component verifies the compatibility between constraints within the same task step. The constraint scope verification component verifies the consistency between the task step corresponding to the constraint and the environmental elements. S53. Based on a preset threshold, the risk level and risk score are mapped to constraint strength to generate a constraint configuration candidate set and configure the candidate associated constraint parameter range and effective conditions for each constraint; wherein, the constraint strength mapping is completed through a threshold mapping table, and the effective conditions include trigger condition type, trigger threshold, and trigger duration fields; the constraint configuration candidate set is evaluated and weighted based on a preset cost item and a preset weight to determine the constraint configuration result; wherein, the preset cost item is read from the constraint cost configuration table, and the preset weight is read from the constraint summary weight table; the constraint configuration result is bound to the task step identifier and encapsulated into an execution constraint generation analysis result, wherein the execution constraint generation analysis result includes constraint type, constraint parameter range, effective conditions, corresponding task step, and constraint strength fields, to obtain the output execution constraint generation analysis result of this step.

[0028] Step S6: Generate analysis results through execution constraints and make execution control determination. In this embodiment, it should be specifically explained that step S6 includes the following specific contents: S61. Obtain the execution constraint generation analysis results output in step S5, and extract the constraint type, constraint parameter range, effective conditions, corresponding task steps, and constraint strength from the execution constraint generation analysis results as control judgment inputs; obtain the task planning analysis results output in step S3 and the environmental status information output in step S1, and perform consistency processing on the task step identifier, coordinate system, and units to form control judgment input data with unified caliber; wherein, the consistency processing of the task step identifier is completed through the task step index table, the consistency processing of the coordinate system is completed through the coordinate transformation relationship table or the external parameter calibration matrix, and the consistency processing of the units is completed through the unit conversion table. S62. Construct a control decision input set based on the unified caliber control decision input data. The control decision input set includes the current task step, execution constraint configuration, robot body state information, and environmental element information. Read the control strategy library, which is stored in the form of control flow templates and indexed according to control modes. Generate a control strategy candidate set based on the control strategy library. The control strategy candidate set includes continue execution strategy, decelerate execution strategy, pause execution strategy, replanning trigger strategy, and terminate execution strategy, and associates control instruction type field and control parameter field for each control strategy candidate. Perform feasibility screening on the control strategy candidate set to eliminate control strategy candidates that do not meet the kinematic constraints, dynamic constraints, planning constraint set, or execution constraint consistency requirements, and obtain the screened control strategy candidate set. The feasibility screening is completed by a kinematic consistency verification component, a dynamic consistency verification component, and a constraint reference verification component. The constraint reference verification component is used to verify the integrity and consistency of the reference of the control strategy candidate to the execution constraint configuration. S63. Trigger the effective conditions based on a preset threshold, and generate a control configuration candidate set based on the trigger determination result; evaluate and weight the control configuration candidate set based on preset cost items and preset weights to determine the control configuration result and generate a control instruction set; wherein, the control instruction set includes trajectory following instructions, speed limit instructions, pause instructions, stop instructions and replanning request instructions, and associates each control instruction with a corresponding task step and effective condition field; bind the control configuration result, control instruction set and corresponding task steps and encapsulate them into an execution control determination result, wherein the execution control determination result includes control mode, control instruction set, corresponding task steps, effective conditions and constraint reference fields, and obtain the execution control determination result output by this step. Example

[0029] like Figure 5 As shown, this embodiment provides a robot multimodal interaction system based on a vision-language large model, which is implemented based on the robot multimodal interaction method based on a vision-language large model in Embodiment 1. It includes: a multimodal data acquisition and environmental state construction module, which is used to acquire multimodal interaction data and construct environmental state information through a camera, microphone and body sensor; The multimodal intent understanding and analysis module is configured to perform speech recognition on speech data and combine it with visual data to input into a visual-language big model for multimodal intent understanding and analysis, so as to generate multimodal intent understanding and analysis results; The task planning and analysis module is configured to perform task planning and analysis based on the multimodal intent understanding and analysis results to generate task planning and analysis results; The security execution risk prediction module is configured to perform security execution risk prediction based on the task planning analysis results, so as to generate security execution risk prediction results; The execution constraint generation and analysis module is configured to perform execution constraint generation and analysis based on the security execution risk prediction results, so as to generate execution constraint generation and analysis results; The execution control determination module is configured to perform execution control determination based on the analysis results generated by the execution constraints, so as to generate execution control determination results. The specific steps of each module in this embodiment are the same as the specific steps of the method embodiment in Embodiment 1, and will not be repeated here. Example

[0030] An electronic device according to an embodiment of the present invention includes a processor and a memory, wherein the memory stores a computer program that can be called by the processor, and the processor executes a robot multimodal interaction method based on a vision-language large model by calling the computer program stored in the memory. It should be noted that all computer programs for the robot multimodal interaction method based on a vision-language large model are implemented using the Python language. Example

[0031] This invention also provides a computer-readable storage medium storing a computer program. When the computer program is invoked and executed by a processor, it causes an electronic device to perform the multimodal robot interaction method based on a vision-language large model described in the foregoing method embodiments. The computer-readable storage medium can be any non-transitory storage medium capable of storing computer-executable instructions, and its type can be a magnetic storage medium, optical storage medium, semiconductor storage medium, or solid-state drive. The computer program instructions stored on the storage medium can be read and executed by a processor to realize the process and function of the foregoing method.

[0032] This invention also provides a computer program product comprising computer-executable instructions. When executed by a processor, these instructions cause an electronic device to perform a robot multimodal interaction method based on a vision-language large model as described in the foregoing method embodiments, thereby realizing the processes and functions corresponding to the foregoing method. The processes and functions can be implemented by software, hardware, firmware, or any combination thereof, and can be deployed on programmable devices such as computers, servers, network nodes, cloud platforms, or data centers. The computer program instructions can also be distributed, downloaded, or loaded via a network, and the network transmission method can be wired or wireless.

[0033] It should be noted that the above descriptions of computer-readable storage media, computer program products, and hardware / software implementations are only for illustrating the implementation forms of the present invention, and the related implementations should not exceed the protection scope of the present invention. For specific implementation details of the system, apparatus, unit, and process of the present invention, please refer to the descriptions in the foregoing method embodiments, which will not be repeated here.

[0034] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0035] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example.

[0036] Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0037] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A robot multimodal interaction method based on a large vision-language model, characterized in that, Includes the following steps: Step S1: Acquire multimodal interaction data and construct environmental state information through camera, microphone and body sensor; Step S2: Perform speech recognition on the speech data and combine it with visual data to input into the visual-language big data model for multimodal intent understanding analysis; Step S3: Perform task planning analysis based on the results of multimodal intent understanding analysis; Step S4: Predict safety execution risks based on task planning analysis results; Step S5: Analyze and generate execution constraints based on the results of the safety execution risk prediction. Step S6: Execution control determination is made by generating analysis results through execution constraints.

2. The robot multimodal interaction method based on a large vision-language model according to claim 1, characterized in that, The multimodal intent understanding and analysis includes the following specific steps: The system acquires visual data from the camera, voice data from the microphone, and status data from the body sensor, and performs time-series alignment of the visual data, voice data, and status data based on a preset timestamp to obtain aligned multimodal data. Speech recognition is performed on the speech data to obtain text commands, and word segmentation and semantic normalization are performed on the text commands to obtain text parsing information; Visual data is used to perform target detection and scene element extraction to obtain scene element information, and the scene element information is fused with state data to obtain environmental element information; Text parsing information, scene element information, and environmental element information are constructed as joint inputs to a visual-language big model according to a preset input format. The visual-language big model is used to infer the intent candidate set and obtain the confidence of each intent candidate. The intent candidate set is filtered based on confidence to obtain intent recognition results, and the task object, action requirements and constraints are extracted based on the intent recognition results to obtain structured intent information. The structured intent information and environmental element information are checked for consistency, and multimodal intent understanding and analysis results are generated based on the check results.

3. The robot multimodal interaction method based on a large vision-language model according to claim 2, characterized in that, The task planning and analysis includes the following specific steps: Obtain the results of multimodal intent understanding analysis and extract the task object, task objective, and constraints; Acquire environmental status information and robot capability parameters, and construct a planning constraint set through task objects, task objectives, and constraints; Acquire skill libraries, action libraries, or task template libraries, and generate candidate task sequences based on task objects and task objectives; Feasibility analysis is performed on candidate task sequences based on the set of planning constraints and combined with environmental state information and robot capability parameters to screen out candidate task sequences that do not meet kinematic constraints, dynamic constraints or resource constraints. The feasible candidate task sequences are evaluated for cost according to preset cost terms, and the cost terms are weighted and summed based on preset weights to obtain the candidate score, thereby optimizing the feasible candidate task sequences. The optimized task sequences, their key parameters, and constraints are summarized to generate task planning analysis results.

4. The robot multimodal interaction method based on a large vision-language model according to claim 1, characterized in that, The security execution risk prediction results include the following specific steps: Obtain the task planning and analysis results and extract the task sequence, key parameters, and execution constraints; Acquire environmental state information and robot capability parameters, and construct a risk assessment input set using environmental state information, robot capability parameters, and execution constraints; A risk factor candidate set is generated based on the risk assessment input set, and a risk prediction candidate set is constructed based on the risk factor candidate set; Feasibility screening of the risk prediction candidate set is performed based on safety distance constraints, speed upper limit constraints, and torque upper limit constraints to obtain a feasible candidate set; Risk scores are calculated for feasible candidate sets based on preset cost terms and impact coefficients, and the risk scores are weighted and summarized based on preset weights to obtain risk assessment results. The risk assessment results are mapped to risk levels and encapsulated to output the safe execution risk prediction results.

5. The robot multimodal interaction method based on a large vision-language model according to claim 1, characterized in that, The execution constraint generation analysis includes the following specific steps: Obtain the results of safety execution risk prediction and extract risk level, risk score and risk element information; Obtain task planning and analysis results and environmental status information, and construct constraint generation input set based on task planning and analysis results, environmental status information and risk factor information; Based on the constraint generation input set, an execution constraint candidate set is generated, and a constraint parameter range is associated with each execution constraint candidate; Based on the consistency requirements between robot capability parameters and execution constraints, a feasibility screening of the execution constraint candidate set is performed to obtain a feasible constraint candidate set. Based on a preset threshold, the risk level and risk score are mapped to the constraint strength, and a candidate set of constraint configurations is generated based on the mapping results. Based on preset cost terms and preset weights, the constraint configuration candidate set is evaluated and weighted to determine the constraint configuration result; The constraint configuration results are encapsulated and the results of the constraint generation analysis are output.

6. The robot multimodal interaction method based on a large vision-language model according to claim 1, characterized in that, The execution control determination includes the following specific steps: Obtain the execution constraint generation analysis results and extract constraint configuration, constraint parameter range and trigger condition information; Obtain task planning and analysis results and environmental status information, and construct a set of control and decision inputs based on task planning and analysis results, environmental status information and constraint configuration; A set of control strategy candidates is generated based on the control decision input set, and a set of control instructions is associated with each control strategy candidate. Based on the consistency requirements of kinematic constraints, dynamic constraints and execution constraints, the feasibility of the control strategy candidate set is screened to obtain a feasible control candidate set; Based on a preset threshold, the trigger condition information is mapped to a control mode, and a control configuration candidate set is generated based on the control mode mapping result. Based on preset cost terms and preset weights, the control configuration candidate set is evaluated and weighted to determine the control configuration result; The control configuration results are encapsulated and output as the execution control judgment result.

7. The robot multimodal interaction method based on a large vision-language model according to claim 1, characterized in that, Step S1 includes the following specific contents: Use a camera to collect visual images or video frame data, use a microphone to collect voice waveform data, and use a body sensor to collect joint position, joint speed, posture, torque and battery status data. The visual data, voice data, and body sensor data are timestamped, and the multimodal data are time-aligned based on a unified time reference. At the same time, the camera and body sensor are calibrated. Environmental element information is extracted based on aligned multimodal data, including target object information, spatial location relationship information, and obstacle information; Robot state information is constructed based on the aligned body state data, and the robot state information includes posture information, motion state information and actuator state information. Environmental element information is fused with robot state information to construct environmental state information.

8. A robot multimodal interaction system based on a vision-language large model, used to implement the robot multimodal interaction method based on a vision-language large model as described in any one of claims 1 to 6, characterized in that, The system includes: The multimodal data acquisition and environmental status construction module is used to acquire multimodal interaction data and construct environmental status information through cameras, microphones and body sensors; The multimodal intent understanding and analysis module is configured to perform speech recognition on speech data and combine it with visual data to input into a visual-language big model for multimodal intent understanding and analysis, so as to generate multimodal intent understanding and analysis results; The task planning and analysis module is configured to perform task planning and analysis based on the multimodal intent understanding and analysis results to generate task planning and analysis results; The security execution risk prediction module is configured to perform security execution risk prediction based on the task planning analysis results, so as to generate security execution risk prediction results; The execution constraint generation and analysis module is configured to perform execution constraint generation and analysis based on the security execution risk prediction results, so as to generate execution constraint generation and analysis results; The execution control determination module is configured to perform execution control determination based on the analysis results generated by the execution constraints, so as to generate execution control determination results.

9. An electronic device, comprising: A processor and a memory, wherein the memory stores a computer program that can be called by the processor; characterized in that the processor executes a robot multimodal interaction method based on a vision-language large model as described in any one of claims 1-7 by calling the computer program stored in the memory.

Images