Humanoid robot interaction control system fusing visual language action large model

By integrating a large-scale visual language and motion model into a humanoid robot interaction control system, real-time monitoring and dynamic adjustment of user feedback are achieved, solving the problem of low interaction efficiency in existing technologies and improving the robot's task accuracy and user satisfaction in complex environments.

CN122632666APending Publication Date: 2026-08-25PLANCK ROBOT TECHNOLOGY (SUZHOU) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202610790892.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-06-03
Publication Date
2026-08-25

AI Technical Summary

Technical Problem

Existing human-computer interaction control systems lack dynamic adjustment capabilities when handling multimodal commands and complex scenarios. They struggle to make reasonable choices among multiple reasonable objectives and fail to adjust behavior in real time using user feedback, resulting in low interaction efficiency and a lack of dialogue features.

Method used

The humanoid robot interaction control system, which integrates a large model of visual language and motion, achieves real-time monitoring and dynamic adjustment of user feedback through modules such as data acquisition, analysis and understanding, correlation assessment, decision generation, window period management and feedback evaluation. Combined with a high attention window period triggered by a high correlation threshold, the robot's behavior is dynamically corrected.

Benefits of technology

It improves the naturalness and fluency of interaction, increases the accuracy and fault tolerance of tasks in complex environments, and enhances the robot's social intelligence and collaborative adaptability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN122632666A_ABST
    Figure CN122632666A_ABST
Patent Text Reader

Abstract

The application discloses a humanoid robot interaction control system fusing visual language action large model, and relates to the field of intelligent interaction control, comprising: a data acquisition module, used for acquiring visual information of a target scene through a visual sensing unit, and receiving action instructions and language instructions of a user, and performing time-space alignment and fusion on multi-modal information through a preset fusion model; an analysis understanding module, used for analyzing the fused scene visual information based on a visual language large model, identifying a plurality of interactive objects and their attributes in the scene, and performing semantic understanding on the user language instructions to extract an interactive intention and a target; the application introduces a high attention window period and a dynamic feedback correction mechanism, when a high correlation degree option conflict is identified, the multi-modal monitoring of user micro expressions and supplementary instructions is actively suspended and started, real-time feedback data is used to drive decision reevaluation, so that the user's implied intention and preference are included in the decision.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of intelligent interactive control technology, specifically to a humanoid robot interactive control system that integrates a large model of visual language and motion. Background Technology

[0002] Humanoid robots are gradually moving from laboratories and specific industrial settings to more open environments such as homes, businesses, and public services. In this context, robots need to interact with humans naturally, efficiently, and safely, understanding and executing complex daily instructions. In recent years, large-scale visual language models have made breakthroughs in image understanding and semantic association, while motion generation and control technologies have also matured. Deeply integrating these two technologies to build intelligent agents capable of understanding instructions and performing actions has become a cutting-edge direction in the intersection of robotics and artificial intelligence.

[0003] Existing human-computer interaction control systems lack the ability to dynamically adjust based on real-time user feedback during execution when handling multimodal commands and complex scenarios. When multiple reasonable objectives exist in a scenario, the robot is prone to making arbitrary choices. If an error occurs, the task must be completely terminated and restarted, resulting in low interaction efficiency and rigidity. The fusion of multimodal information usually stops before decision-making, failing to take real-time feedback information such as user micro-expressions and accompanying gestures during the interaction as key inputs to re-evaluate its own behavior. This leads to a lack of genuine dialogue and negotiation characteristics in the interaction, and the correlation calculation with scene semantics and user intent is not close enough or dynamic enough, making it difficult to achieve action execution that conforms to common sense under ambiguous commands. Summary of the Invention

[0004] (a) Technical problems to be solved

[0005] In view of the above-mentioned shortcomings of the existing technology, the present invention provides a humanoid robot interactive control system that integrates visual language and large-scale action models, which can effectively solve the problems of the existing technology.

[0006] (II) Technical Solution

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] This invention discloses a humanoid robot interaction control system that integrates a large-scale visual-language-action model, comprising:

[0009] The data acquisition module is used to acquire visual information of the target scene through the visual sensing unit, and to receive the user's action commands and language commands. It also performs spatiotemporal alignment and fusion of multimodal information through a preset fusion model.

[0010] The parsing and understanding module is used to parse the fused scene visual information based on the visual language big model, identify multiple interactive objects and their attributes in the scene, and perform semantic understanding of user language commands to extract interaction intent and target;

[0011] The correlation assessment module, based on a pre-set action language model, combines the parsed scene objects, user action commands, and semantically understood interaction intentions to calculate and evaluate the overall correlation between each candidate interactive action that the robot can execute and the current scene and commands.

[0012] The decision generation module is used to select the candidate interaction action with the highest comprehensive relevance as the current main interaction action and generate the corresponding action control sequence. When generating the action control sequence, the kinematic and dynamic feasibility is verified by combining the robot's current joint angle, torque and balance state, and infeasible actions are filtered or adjusted.

[0013] The window period management module is used to determine the entry into a high attention window period when the overall correlation between at least two candidate interactive actions exceeds a preset threshold. During this time period, the main interactive action is suspended or delayed, and feedback monitoring is initiated.

[0014] The feedback evaluation module is used to monitor the user's facial expressions in real time through the visual sensing unit during the high attention window, evaluate the expression recognition and output the recognition evaluation coefficient, and detect whether the user generates any accompanying supplementary action or language commands.

[0015] The decision correction module is used to trigger a re-evaluation process during high-attention windows if the received approval evaluation coefficient is lower than the approval threshold or if supplementary action instructions and supplementary language instructions are received. The approval evaluation coefficient, supplementary action instructions, and supplementary language instructions are used as new inputs, combined with the current scenario information, to drive the relevance evaluation module to recalculate and sort the relevance of candidate interactive actions. In the re-evaluation process, supplementary instructions are regarded as refinements or corrections of the original instructions, and are used with higher weights to calculate the relevance together with the original instructions. After re-sorting, if the interactive action with the highest relevance changes, the main interactive action sent to the decision generation module is immediately updated.

[0016] Furthermore, the fusion model in step 1 includes an encoder network, a spatiotemporal attention alignment module, and a fusion network. Its construction and operation process includes:

[0017] Encoder Network: Visual encoder, action pose encoder and language encoder are constructed respectively to map RGB-D image sequences, user skeleton sequences and text commands to a high-dimensional feature space;

[0018] Spatiotemporal attention alignment module: Connects each encoder. It calculates the temporal alignment weight between visual feature sequences and action feature sequences, as well as the semantic spatial association weight between visual features, action features and language features, through a cross-attention mechanism, so as to realize the synchronization and alignment of multimodal information at the timestamp and semantic levels.

[0019] Fusion Network: Connects to the spatiotemporal attention alignment module, which receives aligned features from each modality and generates a joint feature representation that includes the scene, user actions, and command intent through feature concatenation, element-wise summation, and gating fusion mechanisms.

[0020] Furthermore, the decision generation module is interconnected with the execution monitoring module via a wireless network. The execution monitoring module is interconnected with the decision correction module via a wireless network. The execution monitoring module is used to finally execute the action control sequence corresponding to the highest correlation interaction action confirmed or corrected by the decision correction module, and to obtain feedback data of the action execution process in real time.

[0021] Furthermore, the visual perception unit in the data acquisition module is used to acquire RGB-D images of the scene containing depth information; identify the user's posture and gestures through skeleton tracking technology; convert the user's voice input into text commands; and associate and package image frames, motion skeleton data and text commands at the same timestamp to form a multimodal data frame.

[0022] Furthermore, the construction process of the large visual language model in the parsing and understanding module is as follows:

[0023] Collect and construct a large-scale multimodal pairing dataset, which includes a massive number of scene images and corresponding text annotations describing objects, attributes, spatial relationships and scene states in the images;

[0024] The model is based on a Transformer-based architecture, which includes a visual encoder and a language decoder. The visual encoder encodes the input image into a sequence of visual features, and the language decoder generates text descriptions or understands text instructions based on the visual feature sequences.

[0025] Design multi-task pre-training objectives to train the model;

[0026] After pre-training on general multimodal data, the model is fine-tuned using a dedicated dataset containing robot operation scenarios and instructions, so that the model can be adapted to understand the scene semantics and user instructions related to robot interaction.

[0027] Furthermore, the training tasks include: image-text contrast learning to narrow the feature distance between the matching image and the text; masked language modeling to predict masked text words based on context and visual information; and image-text matching to determine whether a given text description matches the image content.

[0028] Furthermore, the correlation evaluation module uses a pre-trained action language model with a set of scene objects, user action commands, and language command intentions as joint inputs, and outputs a correlation score for each candidate interaction action. The correlation score calculation integrates multiple dimensions such as action semantic matching degree, object functional applicability, and scene context rationality.

[0029] Furthermore, the preset threshold in the window period management module is dynamically adjusted based on the historical interaction success rate; the duration of the window period is set according to the complexity of the scenario, and it automatically ends after a clear user feedback signal is detected or a timeout occurs.

[0030] Furthermore, the process of evaluating facial expression acceptance in the feedback evaluation module includes:

[0031] During the high attention window, the user's facial image sequence is continuously acquired through the visual sensing unit;

[0032] The facial image sequence is input into a pre-trained facial expression analysis model. The model first locates and aligns facial key points, and then extracts temporal features including changes in facial features, muscle micro-movements, and overall expression patterns.

[0033] The extracted temporal features are compared with a predefined expected positive expression template corresponding to the current main interaction action. The expected positive expression template is pre-associated with positive feedback expression features such as smiling, nodding, and focused gaze according to the interaction context.

[0034] The facial expression analysis model calculates an expression acceptance evaluation coefficient based on the comparison results. This coefficient is a continuous value used to quantify the degree of matching between the user's expression and the expected positive expression. The aforementioned expression acceptance evaluation coefficient is sent as a key parameter to the decision correction module to trigger or weight subsequent re-evaluation processes.

[0035] Furthermore, the parsing and understanding module is interconnected with the data acquisition module and the correlation evaluation module via a wireless network; the window period management module is interconnected with the correlation evaluation module, the decision generation module, and the feedback evaluation module via a wireless network; and the feedback evaluation module is interconnected with the decision correction module via a wireless network.

[0036] (III) Beneficial Effects

[0037] Compared with the known prior art, the technical solution provided by this invention has the following beneficial effects:

[0038] 1. By introducing a high attention window and a real-time feedback evaluation mechanism, the traditional fixed pattern of robots executing instructions in one direction is broken. This allows the robot to pause at key decision points in action execution, actively observe the user's micro-expressions and immediate feedback, and dynamically correct its own behavior accordingly. This simulates the interactive mode in which humans use eye contact and facial expressions to make immediate confirmations and adjustments during collaboration, greatly improving the naturalness, fluency, and user satisfaction of the interaction, and enabling the robot to exhibit stronger social intelligence and collaborative adaptability.

[0039] 2. By integrating the visual language action model, the system can deeply understand the multi-dimensional relationship between scene object attributes, user command semantics and actionable actions, thereby making a more reasonable initial choice among multiple potential targets. When there is ambiguity in the initial choice, the system can use the supplementary information obtained during the window period to quickly re-evaluate and correct potential deviations in a timely manner, thereby improving the accuracy of completing tasks in unstructured, dynamically changing or interference-prone complex environments and the fault tolerance of the overall system. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0041] Figure 1 This is a schematic diagram of the framework of the present invention.

[0042] The numbers in the diagram represent: 1. Data acquisition module; 2. Analysis and understanding module; 3. Correlation assessment module; 4. Decision generation module; 5. Window period management module; 6. Feedback evaluation module; 7. Decision correction module; 8. Execution monitoring module. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0044] The present invention will be further described below with reference to embodiments.

[0045] This embodiment of the humanoid robot interaction control system integrates a large-scale visual language and action model, such as... Figure 1 As shown, it includes:

[0046] The data acquisition module 1 is used to acquire visual information of the target scene through the visual sensing unit and receive the user's action and voice commands. The visual perception unit is used to acquire RGB-D images of the scene containing depth information; recognize the user's posture and gestures through skeleton tracking technology; convert the user's voice input into text commands; and associate and package image frames, action skeleton data and text commands at the same timestamp to form a multimodal data frame.

[0047] Multimodal information is spatiotemporally aligned and fused using a pre-defined fusion model. The fusion model includes an encoder network, a spatiotemporal attention alignment module, and a fusion network. Its construction and operation process include:

[0048] Encoder Network: Visual encoder, action pose encoder and language encoder are constructed respectively to map RGB-D image sequences, user skeleton sequences and text commands to a high-dimensional feature space;

[0049] Spatiotemporal attention alignment module: Connects each encoder. It calculates the temporal alignment weight between visual feature sequences and action feature sequences, as well as the semantic spatial association weight between visual features, action features and language features, through a cross-attention mechanism, so as to realize the synchronization and alignment of multimodal information at the timestamp and semantic levels.

[0050] Fusion Network: Connects the spatiotemporal attention alignment module, which receives aligned features from each modality and generates a joint feature representation that includes scene, user actions and command intent through feature concatenation, element-wise summation and gating fusion mechanism;

[0051] The encoder network, spatiotemporal attention alignment module, and multimodal fusion network are jointly trained in an end-to-end manner. The training objective is to minimize the loss function based on the downstream interaction task, so that the model can learn to automatically achieve optimal spatiotemporal alignment and feature fusion.

[0052] Through multimodal fusion networks, the model can adaptively control the "flow" of information from each modality into the joint representation based on the current context. For example, in dim environments, the weight of verbal instructions may be increased; when user gestures are very clear, the weight of action features is greater. This context-aware deep fusion results in richer and more discriminative joint feature representations.

[0053] The parsing and understanding module 2 is used to parse the fused scene visual information based on the visual language big model, identify multiple interactive objects and their attributes in the scene, and perform semantic understanding of user language commands to extract interaction intent and goals; the construction process of the visual language big model is as follows:

[0054] Collect and construct a large-scale multimodal pairing dataset, which includes a massive number of scene images and corresponding text annotations describing objects, attributes, spatial relationships and scene states in the images;

[0055] The model is based on a Transformer-based architecture, which includes a visual encoder and a language decoder. The visual encoder encodes the input image into a sequence of visual features, and the language decoder generates text descriptions or understands text instructions based on the visual feature sequences.

[0056] Design multi-task pre-training objectives to train the model;

[0057] After pre-training on general multimodal data, the model is fine-tuned using a dedicated dataset containing robot operation scenarios and instructions, making the model adaptable to understanding scene semantics and user instructions related to robot interaction. The training tasks include: image-text contrast learning to narrow the feature distance between matching images and text; masked language modeling to predict masked text words based on context and visual information; and image-text matching to determine whether a given text description matches the image content.

[0058] By using end-to-end multimodal fusion and pre-training, we have achieved deep semantic understanding and association mapping of complex physical scenes and fuzzy human-machine commands. This overcomes the limitations of traditional methods, such as the fragmentation of vision, language and action planning modules and reliance on a large number of manual rules or fixed scripts. As a result, we have improved the intelligence and naturalness of the robot in understanding user intentions, making autonomous decisions and executing optimal interactive actions that fit the context in open and dynamic environments.

[0059] The correlation evaluation module 3, based on a pre-set action language model, combines the parsed scene objects, user action commands, and semantically understood interaction intentions to calculate and evaluate the comprehensive correlation between each candidate interactive action that the robot can execute and the current scene and commands. The correlation evaluation module 3 uses the pre-trained action language model with the scene object set, user action commands, and language command intentions as joint inputs, and outputs a correlation score for each candidate interactive action. The calculation of the correlation score integrates multiple dimensions such as action semantic matching degree, object functional applicability, and scene context rationality.

[0060] The decision generation module 4 is used to select the candidate interaction action with the highest comprehensive correlation as the current main interaction action and generate the corresponding action control sequence. When generating the action control sequence, the kinematic and dynamic feasibility is verified by combining the robot's current joint angle, torque and balance state, and infeasible actions are filtered or adjusted.

[0061] The window period management module 5 is used to determine the entry into a high attention window period when the overall correlation between at least two candidate interactive actions exceeds a preset threshold. During this time period, the main interactive action is suspended or delayed, and feedback monitoring is initiated. The preset threshold in the window period management module 5 is dynamically adjusted based on the historical interaction success rate. The duration of the window period is set according to the complexity of the scenario and automatically ends after a clear user feedback signal is detected or the timeout occurs.

[0062] Feedback evaluation module 6 is used to monitor the user's facial expressions in real time through a visual sensing unit during the high attention window, evaluate the expression acceptance, and output an acceptance evaluation coefficient. It also detects whether the user generates any accompanying supplementary action or verbal commands. The expression acceptance evaluation process in feedback evaluation module 6 includes:

[0063] During the high attention window, the user's facial image sequence is continuously acquired through the visual sensing unit;

[0064] The facial image sequence is input into a pre-trained facial expression analysis model. The model first locates and aligns facial key points, and then extracts temporal features including changes in facial features, muscle micro-movements, and overall expression patterns.

[0065] The extracted temporal features are compared with the predefined expected positive expression templates corresponding to the current main interaction action. The expected positive expression templates are pre-associated with positive feedback expression features such as smiling, nodding, and focused gaze based on the interaction context.

[0066] Based on the comparison results, the facial expression analysis model calculates the expression acceptance evaluation coefficient, which is a continuous value used to quantify the degree of matching between the user's expression and the expected positive expression. The above expression acceptance evaluation coefficient is sent as a key parameter to the decision correction module 7 to trigger or weight subsequent re-evaluation processes.

[0067] The decision correction module 7 is used to trigger a re-evaluation process if, during a high-attention window, the received approval evaluation coefficient is lower than the approval threshold or supplementary action instructions and supplementary language instructions are received. The approval evaluation coefficient, supplementary action instructions, and supplementary language instructions are used as new inputs, combined with the current scene information, to drive the relevance evaluation module 3 to recalculate and sort the relevance of candidate interactive actions. In the re-evaluation process, supplementary instructions are regarded as refinements or corrections of the original instructions, and are used with higher weights to calculate the relevance together with the original instructions. After re-sorting, if the interactive action with the highest relevance changes, the main interactive action sent to the decision generation module 4 is immediately updated.

[0068] The decision generation module 4 is connected to the execution monitoring module 8 via a wireless network. The execution monitoring module 8 is connected to the decision correction module 7 via a wireless network. The execution monitoring module 8 is used to finally execute the action control sequence corresponding to the highest correlation interaction action after being confirmed or corrected by the decision correction module 7, and to obtain feedback data of the action execution process in real time.

[0069] The parsing and understanding module 2 is connected to the data acquisition module 1 and the correlation evaluation module 3 via a wireless network. The window period management module 5 is connected to the correlation evaluation module 3, the decision generation module 4 and the feedback evaluation module 6 via a wireless network. The feedback evaluation module 6 is connected to the decision correction module 7 via a wireless network.

[0070] In summary, this invention deeply integrates visual, linguistic, and action modal information, and uses large model technology to achieve deep semantic understanding and correlation analysis of complex scenes and ambiguous instructions. It designs a high attention window period triggered by correlation degree threshold, enabling the robot to have the ability to actively perceive and verify key decision points, rather than blindly execute them, thereby enhancing its adaptability and reliability in dynamic environments.

[0071] This invention can dynamically adjust the interactive objects and actions based on the user's real-time facial expressions, language, and action feedback, realizing a context-adaptive interactive experience. From multimodal fusion and scene parsing to final action monitoring, it forms a complete technology chain, ensuring the intelligence of system decision-making and good scalability.

[0072] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A humanoid robot interactive control system integrating visual language and motion models, characterized in that, include: The data acquisition module is used to acquire visual information of the target scene through the visual sensing unit, and to receive the user's action commands and language commands. It also performs spatiotemporal alignment and fusion of multimodal information through a preset fusion model. The parsing and understanding module is used to parse the fused scene visual information based on the visual language big model, identify multiple interactive objects and their attributes in the scene, and perform semantic understanding of user language commands to extract interaction intent and target; The correlation assessment module, based on a pre-set action language model, combines the parsed scene objects, user action commands, and semantically understood interaction intentions to calculate and evaluate the overall correlation between each candidate interactive action that the robot can execute and the current scene and commands. The decision generation module is used to select the candidate interaction action with the highest comprehensive relevance as the current main interaction action and generate the corresponding action control sequence; The window period management module is used to determine the entry into a high attention window period when the overall correlation between at least two candidate interactive actions exceeds a preset threshold. During this time period, the main interactive action is suspended or delayed, and feedback monitoring is initiated. The feedback evaluation module is used to monitor the user's facial expressions in real time through the visual sensing unit during the high attention window, evaluate the expression recognition and output the recognition evaluation coefficient, and detect whether the user generates any accompanying supplementary action or language commands. The decision correction module is used to trigger a re-evaluation process if the received approval evaluation coefficient is lower than the approval threshold or if supplementary action instructions and supplementary language instructions are received during the high attention window period. The approval evaluation coefficient, supplementary action instructions, and supplementary language instructions are used as new inputs, combined with the current scene information, to drive the relevance evaluation module to recalculate and sort the relevance of candidate interaction actions.

2. The humanoid robot interactive control system integrating visual language action model according to claim 1, characterized in that, The fusion model in step 1 includes an encoder network, a spatiotemporal attention alignment module, and a fusion network. Its construction and operation process includes: Encoder Network: Visual encoder, action pose encoder and language encoder are constructed respectively to map RGB-D image sequences, user skeleton sequences and text commands to a high-dimensional feature space; Spatiotemporal attention alignment module: Connects each encoder, and through a cross-attention mechanism, it calculates the temporal alignment weight between visual feature sequences and action feature sequences, as well as the semantic space association weight between visual features, action features, and linguistic features; Fusion Network: Connects to the spatiotemporal attention alignment module, which receives aligned features from each modality and generates a joint feature representation that includes the scene, user actions, and command intent through feature concatenation, element-wise summation, and gating fusion mechanisms.

3. The humanoid robot interactive control system integrating visual language and motion models according to claim 1, characterized in that, The decision generation module is interconnected with the execution monitoring module via a wireless network. The execution monitoring module is interconnected with the decision correction module via a wireless network. The execution monitoring module is used to finally execute the action control sequence corresponding to the highest correlation interaction action after confirmation or correction by the decision correction module, and to obtain feedback data of the action execution process in real time.

4. The humanoid robot interactive control system integrating visual language and action models according to claim 1, characterized in that, The visual perception unit in the data acquisition module is used to acquire RGB-D images of the scene containing depth information; identify the user's posture and gestures through skeleton tracking technology; convert the user's voice input into text commands; and associate and package image frames, motion skeleton data and text commands at the same timestamp to form a multimodal data frame.

5. The humanoid robot interactive control system integrating visual language and action models according to claim 1, characterized in that, The construction process of the large visual language model in the parsing and understanding module is as follows: Collect and construct a large-scale multimodal pairing dataset, which includes a massive number of scene images and corresponding text annotations describing objects, attributes, spatial relationships and scene states in the images; The model is based on a Transformer architecture, which includes a visual encoder and a language decoder. Design multi-task pre-training objectives to train the model; After pre-training on general multimodal data, the model is fine-tuned using a dedicated dataset containing robot operation scenarios and instructions.

6. The humanoid robot interactive control system integrating visual language and motion models according to claim 5, characterized in that, The training tasks include: image-text contrast learning to narrow the feature distance between the matching image and the text; masked language modeling to predict masked text words based on context and visual information; and image-text matching to determine whether a given text description matches the image content.

7. The humanoid robot interactive control system integrating visual language action model according to claim 1, characterized in that, The correlation evaluation module uses a pre-trained action language model with a set of scene objects, user action commands, and language command intentions as joint inputs, and outputs a correlation score for each candidate interaction action.

8. The humanoid robot interactive control system integrating visual language action model according to claim 1, characterized in that, The preset threshold in the window period management module is dynamically adjusted based on the historical interaction success rate; the duration of the window period is set according to the complexity of the scenario, and it automatically ends after a clear user feedback signal is detected or the timeout occurs.

9. The humanoid robot interactive control system integrating visual language and motion models according to claim 1, characterized in that, The process of evaluating facial expression acceptance in the feedback evaluation module includes: During the high attention window, the user's facial image sequence is continuously acquired through the visual sensing unit; The facial image sequence is input into a pre-trained facial expression analysis model. The model first locates and aligns facial key points, and then extracts temporal features including changes in facial features, muscle micro-movements, and overall expression patterns. The extracted temporal features are compared with the predefined expected positive facial expression templates that correspond to the current main interaction action; The facial expression analysis model calculates the expression acceptance evaluation coefficient based on the comparison results.

10. The humanoid robot interactive control system integrating visual language action model according to claim 1, characterized in that, The parsing and understanding module, the data acquisition module, and the correlation evaluation module are interconnected via a wireless network. The window period management module, the correlation evaluation module, the decision generation module, and the feedback evaluation module are interconnected via a wireless network. The feedback evaluation module and the decision correction module are interconnected via a wireless network.