Control method, device, equipment, medium and program of intelligent robot with body

Abstract

The invention relates to the technical field of robot control, and provides a control method, device, equipment, medium and program for an intelligent robot with a body, and the method comprises the steps: receiving a task language instruction and environment visual observation data collected by the robot in real time; obtaining a corresponding original soft prompt vector according to the task language instruction; inputting the environment visual observation data and the task language instruction into a preset finite-state machine in the intelligent robot system with the body, executing task logic state verification, and outputting a prompt update judgment signal based on a verification result; when the prompt updating judgment signal indicates that updating is needed, a correction soft prompt vector is generated based on the environment visual observation data; and fusing the corrected soft prompt vector and the original soft prompt vector to generate an updated prompt. The method is used for improving the control adaptability and reliability of the intelligent robot with the body in a dynamic environment through a lightweight dynamic prompt mechanism, precise action generation and long-range task logic coherence are achieved, and full-model fine adjustment is avoided.

Description

Control methods, devices, equipment, media and programs for embodied intelligent robots Technical Field

[0001] This invention relates to the field of robot control technology, and in particular to a control method, apparatus, device, medium, and program for an embodied intelligent robot. Background Technology

[0002] In the field of embodied intelligent robot control, while Vision-Language-Action Model (VLA) can guide action generation through soft cues, existing technologies generally employ a static soft cue strategy—that is, a fixed cue vector is preset based on task instructions and remains unchanged throughout the inference process. This leads to environmental disturbances, such as changes in object displacement and lighting, causing the cue to mismatch with the current observation state, and the model's output action deviating from physical reality. To improve adaptability, some solutions attempt to fine-tune the VLA backbone parameters online, but the computational complexity is too great due to the large number of model parameters. The increased overhead makes it difficult to meet real-time control requirements, and continuous updates are prone to catastrophic forgetting, weakening the generalization ability of pre-trained knowledge. Simultaneously, existing feature extraction methods largely rely on single-stream architectures, deeply coupling the semantic information representing task logic with the spatial information representing geometric details. This makes it difficult to accurately verify task stage logic and to specifically correct spatial deviations. More critically, the control flow lacks an explicit coordination mechanism with the task state machine. When sudden environmental changes lead to stage misjudgments, the system cannot identify and block illegal state transitions, such as misjudging "grabbing" as "placement complete," resulting in long-term task logic breaks and error accumulation. These shortcomings collectively limit the VLA model from achieving both high reliability and high adaptability in robot autonomous control in open, unstructured environments.

[0003] Therefore, there is an urgent need for a control method, device, equipment, medium, and program for embodied intelligent robots to improve the above-mentioned problems. Summary of the Invention

[0004] This invention provides a control method, device, equipment, medium, and program for an embodied intelligent robot. This invention is used to improve the control adaptability and reliability of the embodied intelligent robot in dynamic environments through a lightweight dynamic prompting mechanism, realize precise motion generation and long-term task logic coherence, and avoid full model fine-tuning.

[0005] According to a first aspect of the present invention, a control method for an embodied intelligent robot is provided, applied to an embodied intelligent robot system. The system integrates a vision-language-action model. The method includes: receiving task language instructions and environmental visual observation data collected by the robot in real time; obtaining the corresponding original soft cue vector based on the task language instructions; inputting the environmental visual observation data and the task language instructions into a pre-set finite state machine in the embodied intelligent robot system, whereby the finite state machine performs task logic state verification, obtains the verification result, and outputs a cue update judgment signal based on the verification result; when the cue update judgment signal indicates that an update is needed, generating a corrected soft cue vector based on the environmental visual observation data; fusing the corrected soft cue vector with the original soft cue vector to generate an updated cue; and inputting the updated cue into the vision-language-action model, causing the vision-language-action model to output instructions to control the robot to perform actions.

[0006] In one implementation, environmental visual observation data and task language instructions are input into a pre-set finite state machine in the embodied intelligent robot system. This includes: extracting shallow spatial features from the environmental visual observation data, performing semantic parsing on the task language instructions to obtain deep semantic features, and inputting the shallow spatial features and deep semantic features into the pre-set finite state machine. The pre-set finite state machine is used to perform task logic state verification based on the shallow spatial features and deep semantic features and output a prompt update judgment signal.

[0007] In one implementation, the finite state machine is pre-configured with task state transition rules associated with task language instructions. The task state transition rules are constructed based on the expected state sequence of task execution and update the verified current task stage identifier. The finite state machine performs task logic state verification to obtain a verification result, including: parsing environmental visual observation data into candidate task stage identifiers; verifying whether the transition from the current task stage identifier to the candidate task stage identifier conforms to the task state transition rules and obtaining a verification result; if the verification result is successful, updating the candidate task stage identifier to the current task stage identifier; if the verification result is unsuccessful, blocking the current task state transition and maintaining the current task stage identifier unchanged.

[0008] In one implementation, generating a corrected soft cue vector based on environmental visual observation data includes: extracting spatial features from the environmental visual observation data to obtain current spatial features; inputting the current spatial features and standard reference features retrieved from a standard cue library into a difference generator; the difference generator is used to calculate the feature deviation between the current spatial features and the standard reference features, and to generate a corrected soft cue vector based on the feature deviation.

[0009] In one implementation, fusing the modified soft cue vector with the original soft cue vector to generate an updated cue includes: inputting the modified soft cue vector and the original soft cue vector into a gating subnetwork, and having the gating subnetwork output a first weight coefficient and a second weight coefficient; weighting the original soft cue vector based on the first weight coefficient, and weighting the modified soft cue vector based on the second weight coefficient; and adding the weighted original soft cue vector to the weighted modified soft cue vector to obtain the updated cue.

[0010] In one implementation, before generating the corrected soft cue vector, the difference generator is obtained through offline joint training. The offline training includes: acquiring expert demonstration data containing visual observation data, natural language text instructions, and corresponding expert real action sequences; extracting shallow spatial features and standard reference features of the current frame based on the visual observation data, and inputting them into the difference generator to generate the corrected soft cue vector; injecting dynamic control cue containing the corrected soft cue vector, natural language text instructions, and visual features of the current frame into the parameter-frozen action decoder to output the predicted action; using the expert real action sequence as a supervision signal, and backpropagating end-to-end to update the network parameters of the difference generator based on the action regression error between the predicted action and the expert real action.

[0011] In one implementation, offline training is performed while keeping the backbone network parameters of the vision-language-action model frozen.

[0012] In one implementation, after receiving task language instructions and environmental visual observation data collected in real time by the robot, the method further includes: extracting dual-stream features from the environmental visual observation data through the backbone network of the parameter-frozen vision-language-action model to obtain deep semantic features and shallow spatial features; wherein, the deep semantic features are used to represent the global task logic, and the shallow spatial features are used to represent local geometric details.

[0013] In one implementation, inputting environmental visual observation data and task language instructions into a pre-set finite state machine includes: inputting deep semantic features into a pre-trained state discriminator, whereby the state discriminator combines historical frame prediction results with the logic of the finite state machine to determine state transitions and outputs a current task stage identifier; and obtaining the corresponding original soft cue vector based on the task language instructions, including: retrieving a standard soft cue vector matching the current task stage identifier from a pre-set standard cue library as the original soft cue vector based on the current task stage identifier.

[0014] In one implementation, the standard cue library stores standard reference features associated with the identifiers of each task stage; generating the corrected soft cue vector based on environmental visual observation data includes: inputting shallow spatial features and standard reference features into a difference generator, which calculates the feature deviation and generates the corrected soft cue vector.

[0015] In one implementation, before receiving the task language instruction, the method further includes: collecting corresponding observation data under standard environmental conditions for each preset task stage of the target task; extracting dual-stream features from the observation data through the backbone network of the parameter-frozen vision-language-action model to obtain deep semantic features and shallow spatial features corresponding to each task stage; determining the deep semantic features as standard soft cue vectors and the shallow spatial features as standard reference features; and associating and storing the standard soft cue vectors, standard reference features, and corresponding task stage identifiers to form a standard cue library.

[0016] According to a second aspect of the present invention, a control device for an embodied intelligent robot is provided, used in any of the methods of the first aspect. The device includes: a receiving unit for receiving task language instructions and environmental visual observation data collected by the robot in real time; an acquisition unit for acquiring a corresponding original soft cue vector based on the task language instructions; a finite state machine for receiving the environmental visual observation data and the task language instructions, performing task logic state verification, and outputting a cue update judgment signal based on the verification result; a generation unit for generating a corrected soft cue vector based on the environmental visual observation data when the cue update judgment signal indicates that an update is needed; a fusion unit for fusing the corrected soft cue vector and the original soft cue vector to generate an updated cue; and an input unit for inputting the updated cue into a vision-language-action model, triggering the vision-language-action model to output instructions to control the robot to perform actions.

[0017] According to a third aspect of the present invention, an electronic device is provided, including a memory and a processor, wherein the memory is used to store a computer program executable by the processor; and the processor is used to execute the computer program in the memory to implement the method described above.

[0018] According to a fourth aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, enables the implementation of the above-described method.

[0019] According to a fifth aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method described above.

[0020] Compared with existing technologies, the advantages of this invention are as follows: This invention verifies the task logic state by inputting task language instructions and real-time environmental visual observation data into a pre-set finite state machine. This enables the system to intelligently determine whether a prompt update needs to be triggered based on the verification results, thereby avoiding decision redundancy and wasted computational resources caused by indiscriminate and frequent prompt adjustments. When verification confirms that an update is needed, the system dynamically generates a corrected soft prompt vector and merges it with the original soft prompt vector. This ensures that the prompt content input to the vision-language-action model retains the initial semantic intent of the task while accurately reflecting the real-time deviation between the current environmental state and the task stage, guiding the model to generate action instructions that conform to the physical environment. This mechanism, without altering the core structure and parameters of the vision-language-action model, achieves precise responses to environmental disturbances and task stage evolution solely through lightweight dynamic adaptation at the prompting level. It effectively overcomes the shortcomings of traditional static prompting schemes, which are prone to failure in dynamic scenarios, while avoiding the high computational cost and catastrophic forgetting risk associated with full model fine-tuning. The logic verification stage of the finite state machine further enhances the temporal coherence of task execution, implicitly constraining illegal state transitions and improving the reliability and robustness of control logic in long-term tasks. This provides an efficient and adaptable technical path for embodied intelligent robots to operate stably and autonomously in open and unstructured environments. Attached Figure Description

[0021] Figure 1 is a flowchart illustrating a control method for an embodied intelligent robot according to an exemplary embodiment.

[0022] Figure 2 is a block diagram illustrating a control device for an embodied intelligent robot according to another exemplary embodiment.

[0023] Figure 3 is a block diagram illustrating an electronic device according to an exemplary embodiment.

[0024] Explanation of the reference numerals in the figure:

[0025] 1. Receiving unit; 2. Acquisition unit; 3. Finite state machine; 4. Generation unit; 5. Fusion unit; 6. Input unit; 7. Control device for embodied intelligent robot; 900. Electronic device; 922. Processing component; 926. Power supply component; 932. Memory; 950. Network interface; 958. Input / output interface. Detailed Implementation

[0026] Unless otherwise defined, the technical or scientific terms used in this specification should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. Specific embodiments of the invention will be described below with reference to the accompanying drawings. It should be noted that, in order to provide a concise description, this specification cannot provide a detailed description of all features of the actual embodiments. Without departing from the spirit and scope of the invention, those skilled in the art can make modifications and substitutions to the embodiments of the invention, and the resulting embodiments are also within the protection scope of the invention.

[0027] As shown in Figure 1, the first embodiment of the present invention provides a control method for an embodied intelligent robot, which is applied to an embodied intelligent robot system. The embodied intelligent robot system integrates a vision-language-action model. The method includes the following steps S1-S6: S1, receiving task language instructions and environmental visual observation data collected by the robot in real time.

[0028] S2, obtain the corresponding raw soft prompt vector based on the task language instructions.

[0029] S3 inputs environmental visual observation data and task language instructions into the embodied intelligent robot system's pre-set finite state machine, which performs task logic state verification, obtains the verification result, and outputs a prompt update judgment signal based on the verification result.

[0030] S4, when the prompt update judgment signal indicates that an update is needed, generate a corrected soft prompt vector based on the environmental visual observation data.

[0031] S5 merges the corrected soft suggestion vector with the original soft suggestion vector to generate an updated suggestion.

[0032] S6. Input the updated prompts into the vision-language-motion model, so that the vision-language-motion model outputs instructions to control the robot to perform actions.

[0033] In one implementation, environmental visual observation data and task language instructions are input into a pre-set finite state machine in the embodied intelligent robot system. This includes: extracting shallow spatial features from the environmental visual observation data, performing semantic parsing on the task language instructions to obtain deep semantic features, and inputting the shallow spatial features and deep semantic features into the pre-set finite state machine. The pre-set finite state machine is used to perform task logic state verification based on the shallow spatial features and deep semantic features and output a prompt update judgment signal.

[0034] In some specific embodiments, the robot acquires the current frame's environmental image through a visual sensor and performs standardized preprocessing. The processed image is then input into the visual-language-action model backbone network with completely frozen parameters. A dual-stream feature extraction mechanism reuses this frozen backbone network, simultaneously extracting deep semantic features representing the global task logic and stage semantics from the deep coding layer of the network, and extracting shallow spatial features that preserve local geometric details such as object contours and relative positions from the shallow coding layer. Throughout the feature extraction process, all parameters of the backbone network are strictly kept frozen and do not participate in gradient updates. The resulting two types of features are output independently as decoupled data streams: deep semantic features are sent to a state discriminator for task stage determination and finite state machine verification, and shallow spatial features are sent to a difference generator to calculate the feature deviation between the current observation and the standard reference state, providing a decoupled feature basis for subsequent semantic state locking and spatial residual error correction.

[0035] In one implementation, the finite state machine is pre-configured with task state transition rules associated with task language instructions. The task state transition rules are constructed based on the expected state sequence of task execution and update the verified current task stage identifier. The finite state machine performs task logic state verification to obtain a verification result, including: parsing environmental visual observation data into candidate task stage identifiers; verifying whether the transition from the current task stage identifier to the candidate task stage identifier conforms to the task state transition rules and obtaining a verification result; if the verification result is successful, updating the candidate task stage identifier to the current task stage identifier; if the verification result is unsuccessful, blocking the current task state transition and maintaining the current task stage identifier unchanged.

[0036] In some specific embodiments, deep semantic features are input into a pre-trained state discriminator. The state discriminator fuses the semantic representation of the current frame with the state prediction sequence of historical frames, and performs collaborative judgment in conjunction with preset finite state machine (FSM) transition rules, outputting the legal stage ID of the current task. The FSM synchronously performs logical constraint verification. If an illegal state jump that violates the task's temporal logic is detected, the transition path is immediately blocked and a state rollback mechanism is triggered to ensure the temporal rigor and logical continuity of the task execution process. After obtaining the verified stage ID, the system synchronously retrieves the standard soft cue vector and standard reference feature uniquely associated with the stage ID from a preset standard cue library. The standard soft cue vector is used for subsequent cue fusion, and the standard reference feature serves as a benchmark reference for the environmental geometric state. Together, they constitute the dynamic control basis for stage awareness.

[0037] In some examples, typical scenarios of illegal state jumps include execution anomalies, such as objects accidentally slipping or being misplaced during the grasping process, causing the system to misjudge the task stage and attempt to perform state transitions that violate the preset timing logic. For example, jumping to the "placement" stage before completing the "grab confirmation" stage. The finite state machine identifies such logical conflicts caused by physical execution deviations by comparing the current observation with the historical state sequence, actively blocks illegal transition paths and triggers state rollback, ensuring that the task flow strictly follows the preset stage evolution rules, and effectively avoids long-term task logic breaks caused by single-frame misjudgments.

[0038] In one implementation, generating a corrected soft cue vector based on environmental visual observation data includes: extracting spatial features from the environmental visual observation data to obtain current spatial features; inputting the current spatial features and standard reference features retrieved from a standard cue library into a difference generator; the difference generator is used to calculate the feature deviation between the current spatial features and the standard reference features, and to generate a corrected soft cue vector based on the feature deviation.

[0039] In some specific embodiments, shallow spatial features and standard reference features are simultaneously input into a difference generator. The difference generator accurately quantifies the subtle deviations between the current observation state and the standard reference state at the local geometric level, such as object position offsets and posture changes, through feature space comparison. This deviation is then transformed into a correction soft cue vector via a lightweight mapping network. The correction soft cue vector is dedicated to fine-tuning the motion trajectory, specifically compensating for geometric mismatches caused by environmental disturbances, so that the subsequently generated motion commands closely match the actual state of the current physical scene. The entire process is completed under the premise of freezing the core parameters of the vision-language-motion model, which avoids the computational burden and knowledge forgetting risk of full model fine-tuning, and achieves efficient and targeted correction of spatial deviations, significantly improving the robot's motion accuracy and execution robustness in dynamic environments.

[0040] In one implementation, fusing the modified soft cue vector with the original soft cue vector to generate an updated cue includes: inputting the modified soft cue vector and the original soft cue vector into a gating subnetwork, and having the gating subnetwork output a first weight coefficient and a second weight coefficient; weighting the original soft cue vector based on the first weight coefficient, and weighting the modified soft cue vector based on the second weight coefficient; and adding the weighted original soft cue vector to the weighted modified soft cue vector to obtain the updated cue.

[0041] In some specific embodiments, standard soft cue vectors and modified soft cue vectors are input into a pre-defined gating subnetwork. The gating subnetwork dynamically generates adaptive weighting coefficients based on the semantics of the current task stage and the environmental spatial context, and performs weighted fusion of the two to generate a frame-level dynamic control cue that precisely matches the observation state of the current frame. Subsequently, the dynamic control cue is converted into a prefix token sequence and embedded as a guiding prefix into the starting position of the input sequence of the action decoder of the vision-language-action model. This mechanism enables the model to co-encode the task logic intent and real-time environmental deviations into the cue prefix while the backbone parameters are frozen throughout the process. This guides the action decoder to generate precise control instructions that meet the requirements of the task stage and are adapted to the current physical scene, effectively bridging the semantic gap between static cues and dynamic environments, and improving the robot's action adaptability and task execution reliability in open environments.

[0042] In other specific embodiments, the "updated cue" generated in step S5 is explicitly defined as a frame-level dynamic control cue. The action decoder uses this frame-level dynamic control cue as a prefix guide sequence, embeds it at the beginning of the input sequence, and integrates the visual observation features of the current frame with the historical action execution context, including historical action sequences and historical environmental observation features. It then uses an autoregressive approach to generate the robot's control action sequence step by step over time. In each prediction step, the decoder dynamically references the generated historical action sequence and the current environmental representation to ensure that the output action remains coherent under temporal logic and physical constraints. The generated discrete control action instructions are converted into robot joint control signals or end effector operation instructions through a standardized interface, driving the actuator to complete the action in real time. The entire process strictly relies on the parameter-frozen visual-language-action model backbone network, eliminating the need for online fine-tuning: on the one hand, it fully preserves the generalization ability of the pre-trained model, avoiding catastrophic forgetting; on the other hand, through the collaborative mechanism of "dynamic cue prefix guidance + autoregressive decoding," the action output accurately adapts to the current task stage and environmental state, effectively improving the adaptability, temporal rigor, and execution reliability of the robot's action planning in dynamic and unstructured scenarios.

[0043] In one implementation, before generating the corrected soft cue vector, the difference generator is obtained through offline joint training. The offline training includes: acquiring expert demonstration data containing visual observation data, natural language text instructions, and corresponding expert real action sequences; extracting shallow spatial features and standard reference features of the current frame based on the visual observation data, and inputting them into the difference generator to generate the corrected soft cue vector; injecting dynamic control cue containing the corrected soft cue vector, natural language text instructions, and visual features of the current frame into the parameter-frozen action decoder to output the predicted action; using the expert real action sequence as a supervision signal, and backpropagating end-to-end to update the network parameters of the difference generator based on the action regression error between the predicted action and the expert real action.

[0044] In one specific embodiment, the offline preparation phase employs a semantically sensitive automatic state segmentation mechanism: the visual sequence of the expert demonstration trajectory is input as a parameter to the frozen visual-language-action model backbone network, and deep semantic feature temporal sequences are extracted; a semantic change curve is constructed by calculating the cosine similarity of semantic features of adjacent frames, and the model's inherent sensitivity to key task events, such as sudden changes in object state or transitions in action stages, is utilized to make state switching points naturally appear as significant troughs on the curve; peak detection algorithms are used to accurately locate these troughs as task state switching boundaries, and the time segments between adjacent boundaries are defined as logically coherent task stages, achieving unsupervised stage segmentation that conforms to human cognitive logic and avoiding the oversensitivity of traditional geometric clustering methods to visual appearance changes.

[0045] In another specific embodiment, for each divided task stage, the system selects the shallow spatial features corresponding to the middle moment of the time window of that stage as the standard reference features. This strategy effectively avoids motion blur and transition noise interference during state transitions, based on the characteristic that the middle moment is in the temporally stable region. It should be clarified that the standard reference features are only used as dynamic anchor points for calculating spatial deviations in the online stage, such as the local coordinate origin, rather than the ultimate target state of motion control. The difference generator learns the mapping relationship between "deviation amount and motion correction amount", enabling the robot to continuously adjust its motion trajectory with this anchor point as a reference, smoothly traversing the entire task stage until completion, ensuring the temporal coherence and physical rationality of the control process.

[0046] In another specific embodiment, the standard soft suggestion vectors for each task stage in the standard suggestion library adopt a text semantic hot-start initialization strategy: a short natural language description, such as "grab the doorknob," is preset for each stage and input into the text encoder of the vision-language-action model to extract semantic embedding vectors as initial values. This method deeply couples the text-visual alignment prior established during the model pre-training stage, so that the initial suggestion vectors naturally have task semantic orientation, significantly improving the convergence efficiency and stability of subsequent joint training, while strengthening the semantic consistency between the suggestion vectors and the logic of the task stage, laying a semantic foundation for accurate suggestion retrieval in the online stage.

[0047] In some specific embodiments, offline training is performed while the backbone network parameters of the vision-language-action model are kept frozen.

[0048] In some examples, the parameters of the visual-language-action model backbone network are frozen throughout the offline training process, and only lightweight adaptation modules such as the difference generator, state discriminator, and gating sub-network are optimized. This design not only fully preserves the general visual-language representation capabilities obtained by the backbone network pre-training, effectively avoiding the risk of catastrophic forgetting, but also accurately focuses training resources on key aspects of task adaptation, significantly improving training efficiency and module convergence stability, while providing a consistent and reliable feature extraction benchmark for the online control phase.

[0049] Specifically, the first weighting coefficient Second weighting coefficient The generation employs a context-adaptive mechanism. In this embodiment, the prompt dynamic aggregation module includes a gating sub-network consisting of two fully connected layers. This gating sub-network receives the current semantic features ( The intermediate layer features of the difference generator are used as input, and the output is normalized weight coefficients. and During the model's training phase (i.e., the lightweight adapter training phase), this gated subnetwork is jointly trained end-to-end with the state discriminator and the difference generator.

[0050] The objective function for training is the Action MSE Loss: ;in: represents the mean squared error loss of action regression during the joint training phase of the model; Decoder represents the parameter-frozen language model decoder in the Vision-Language-Action (VLA) large model, which is used to receive multimodal sequences including visual features, language instructions and dynamic soft cues, and autoregressively output the robot's discretized action predictions. This indicates a frozen visual encoder (such as a Vision Transformer) on the current frame image. Extracted visual features; This represents the natural language text instruction corresponding to the current task, such as "grab the red square"; This refers to the standard soft cue vector retrieved from the standard cue library, which represents the global state of the current task phase. A correction soft cue vector containing local position fine-tuning information, generated for the spatial residual error correction module (Delta Generator, or DeltaGen for short); and The adaptive weight coefficients are dynamically output in real time by the gating sub-network and are used to control the fusion ratio of semantic logic inertia and spatial high-frequency error correction signal in the control loop. The ground truth sequence of actions corresponding to the current state in the expert demonstration data serves as a supervisory signal for network optimization. This represents the square of the L2 norm (Euclidean distance) between the predicted action vector and the expert's actual action vector.

[0051] In some examples, the error gradient is backpropagated to the gated subnetwork via gradient descent algorithms (such as AdamW). The model will automatically learn the optimal weight allocation strategy for different task stages: in the spatial error correction-dominated stage, such as "precision jacking": the model will automatically learn to output a larger... This allows for the introduction of high-frequency spatial difference signals to achieve real-time compensation for minute positional deviations; during the semantic logic-driven stage, such as "when moving over a large area: the model will output a larger value." Values ​​are used to maintain the smoothness and logical consistency of actions. This is because standard soft cueing ( As a static global feature, it possesses extremely high signal stability. (Increase) This is equivalent to introducing a "low-pass filter" and "logic inertia" mechanism into the control loop, which can effectively suppress high-frequency visual jitter caused by sensor noise or ambient light flicker, ensuring that the robot ignores unnecessary local disturbances during macroscopic movement and stably follows the main task path.

[0052] During online inference, the gated subnetwork calculates the corresponding value for the current frame in real time based on the trained parameters. and And based on this, generate the final prompt. : Finally, the generated The sequence is inserted at the very beginning of the text input sequence of the VLM action decoder.

[0053] In one implementation, after receiving task language instructions and environmental visual observation data collected in real time by the robot, the method further includes: extracting dual-stream features from the environmental visual observation data through the backbone network of the parameter-frozen vision-language-action model to obtain deep semantic features and shallow spatial features; wherein, the deep semantic features are used to represent the global task logic, and the shallow spatial features are used to represent local geometric details.

[0054] In some specific embodiments, the system performs real-time dual-stream decoupling processing on the environmental visual observation data acquired in real time: the visual-language-action model backbone network with parameters frozen throughout the process synchronously outputs deep semantic features and shallow spatial features, achieving functional decoupling at the source of feature generation; the parameter freezing mechanism effectively ensures the stability of the feature space across time steps, suppresses perceptual drift caused by environmental dynamic disturbances, and ensures that state discrimination and deviation calculation have a reliable temporal consistency basis; deep semantic features focus on carrying the global logic and stage semantics of the task, while shallow spatial features accurately depict local geometric structures and spatial relationships. The two types of features are directed as independent data streams to the state discriminator and the difference generator, respectively, avoiding the risk of misjudgment caused by semantic-spatial information coupling at the architectural level, providing high-fidelity, low-interference perceptual input for subsequent logic verification and spatial residual error correction, and significantly enhancing the system's perceptual robustness and control accuracy in open dynamic environments.

[0055] In one implementation, inputting environmental visual observation data and task language instructions into a pre-set finite state machine includes: inputting deep semantic features into a pre-trained state discriminator, whereby the state discriminator combines historical frame prediction results with the logic of the finite state machine to determine state transitions and outputs a current task stage identifier; and obtaining the corresponding original soft cue vector based on the task language instructions, including: retrieving a standard soft cue vector matching the current task stage identifier from a pre-set standard cue library as the original soft cue vector based on the current task stage identifier.

[0056] In some specific embodiments, the state discriminator parses the deep semantic features, outputs the probability distribution of the current frame belonging to each preset task stage, and selects the stage identifier with the highest confidence as the preliminary judgment result; the finite state machine performs temporal legality verification on the current transition path based on the preset state transition rule matrix and the legal state locked in the previous frame: if the transition conforms to the task logic rules, the current state identifier is updated; if an illegal jump that violates the preset temporal order is detected (such as crossing necessary intermediate stages), the transition path is immediately blocked and the legal state of the previous frame is forcibly maintained to ensure the logical rigor and anti-interference capability of the task execution process.

[0057] In another specific embodiment, the input processing of the difference generator adopts a feature channel dimension fusion strategy, which concatenates the shallow spatial features of the current frame with the standard reference features to construct a joint representation to explicitly highlight geometric deviations. Its internal lightweight mapping network focuses on parsing the subtle spatial offsets caused by environmental disturbances in the joint representation and encodes the offset information into a residual cue vector. This vector accurately represents the geometric difference between the current observation and the standard state, providing a directional and interpretable basis for fine-tuning the action trajectory, effectively bridging the spatial perception blind spot of the frozen backbone model in dynamic scenes.

[0058] In one implementation, the standard cue library stores standard reference features associated with the identifiers of each task stage; generating the corrected soft cue vector based on environmental visual observation data includes: inputting shallow spatial features and standard reference features into a difference generator, which calculates the feature deviation and generates the corrected soft cue vector.

[0059] In some specific embodiments, the standard prompt library is constructed by assigning a unique stage identifier to each task stage and initializing a learnable standard soft prompt vector bound to it. This initialization adopts a text semantic guidance strategy: based on the stage's core semantic preset, a concise and unambiguous natural language description is generated and converted into a semantic embedding vector by a visual-language model text encoder as the initial parameter. This standard soft prompt vector maintains fine-tunability in subsequent joint training. It obtains a high-quality semantic starting point by relying on the text-visual alignment prior of the pre-trained model, and adapts to the subtle differences in specific task scenarios through lightweight optimization. This allows the prompt vector to retain the semantic essence of the stage while having scene adaptability, laying a dual foundation for achieving high-precision and robust dynamic prompt retrieval and fusion in online stages.

[0060] In one implementation, before receiving the task language instruction, the method further includes: collecting corresponding observation data under standard environmental conditions for each preset task stage of the target task; extracting dual-stream features from the observation data through the backbone network of the parameter-frozen vision-language-action model to obtain deep semantic features and shallow spatial features corresponding to each task stage; determining the deep semantic features as standard soft cue vectors and the shallow spatial features as standard reference features; and associating and storing the standard soft cue vectors, standard reference features, and corresponding task stage identifiers to form a standard cue library.

[0061] In some specific embodiments, the construction of the standard cue library serves as an offline preparation step before system deployment. For each preset stage of the target task, representative observation data is collected in a standard scenario without environmental disturbance. The core network of the vision-language-action model is frozen using parameters identical to those in the online control stage. Dual-stream feature extraction is performed on the observation data. This architecture design ensures that offline construction and online inference are in a unified feature space, fundamentally guaranteeing feature comparability and system consistency. The extracted deep semantic features are established as the standard soft cue vector for that stage, and the shallow spatial features are established as the standard reference features. These two, along with the unique task stage identifier, form a structured triple and are stored in the standard cue library. This design makes the standard cue library not only a cue retrieval source but also a bridge connecting the task's logical semantics and spatial geometric benchmarks, providing a high-fidelity, low-noise reference system for accurate state determination and spatial deviation quantification in the online stage.

[0062] In another embodiment, the backbone architecture of the vision-language-action model can adopt a combined structure in which an independent vision encoding module and a language processing module are coupled through a cross-modal projection layer, and the two work together to complete multimodal alignment. This design retains the dual-stream feature decoupling capability while providing modular deployment flexibility for resource-constrained platforms. Furthermore, the backbone parameters can be selectively fine-tuned according to task requirements (such as only unfreezing the deep adaptation layer), which enhances scene adaptability and forms a complementary gain with the external state verification mechanism.

[0063] In another embodiment, deep semantic features and shallow spatial features are extracted independently through parallel external visual encoders: dedicated encoders focus on semantic logic parsing and geometric detail perception respectively, achieving physical-level feature decoupling; this bypass extraction architecture effectively avoids interference from feature coupling within a single backbone and is suitable for precision operation scenarios with stringent requirements for semantic-spatial separation accuracy.

[0064] In another embodiment, task phase division uses unsupervised temporal clustering or manual teaching temporal labeling to generate phase boundaries, and combines behavior trees or lightweight language models to construct dynamic logic controllers; behavior trees realize flexible planning of complex task flows through conditional nodes and backoff strategies, while lightweight language models perform logical reasoning verification based on natural language rule descriptions, significantly improving the system's adaptability and interpretability in unstructured task flows.

[0065] In another embodiment, the standard reference feature adopts the phase start frame feature or a time-varying reference based on the dynamic calculation of optical flow in continuous frames, so that the spatial deviation calculation has temporal continuity; the difference generator structure can be replaced by a multilayer perceptron, a lightweight visual transformer or a graph neural network, which models the spatial offset based on the feature topology relationship, and enhances the error correction capability for non-rigid deformation or topological change scenarios.

[0066] In another embodiment, dynamic cue injection employs a cross-attention mechanism to embed into the intermediate layer of the action decoder, or modulates the decoder activation state through a feature bias / scaling factor (FiLM); the cue fusion strategy uses a nonlinear neural network instead of linear weighting to learn the high-order interaction relationship between the original cue and the modified cue, making the fusion result more consistent with the complex context semantics and improving the fine-grained control accuracy of action generation.

[0067] In another embodiment, the environmental perception module is compatible with multi-source sensor inputs such as binocular vision, structured light, or lidar, and is uniformly sent to the dual-stream extraction process after feature alignment; the execution terminal can be extended to heterogeneous embodied platforms such as mobile chassis, humanoid robots, or drones, and the control commands are mapped into joint trajectories, path planning, or posture control signals through the platform adaptation layer.

[0068] It is worth noting that the system architecture of this embodiment includes a robot hardware layer, a frozen model layer, and a dynamic decision adaptation layer. The robot hardware layer includes a vision acquisition module and a robot actuator: the vision acquisition module uses an RGB-D camera, such as an Intel RealSense D435i, mounted at the robot's hand-eye or third-person perspective position, to acquire visual images of the workspace in real time; the robot actuator uses a multi-degree-of-freedom robotic arm, such as the Franka Emika Panda, and an end effector with two fingers, to receive and execute pose control commands and gripper opening and closing commands. The frozen VLM backbone network layer uses a pre-trained vision-language-action model as its base, and all parameters of its Vision Transformer encoder and language model decoder are kept completely frozen during inference. This layer includes a dual-stream feature extraction module, which uses the ViT encoder of the backbone network to extract features using a "hierarchical decoupling" strategy: deep semantic features are extracted from the [CLS]Token output of the last layer of ViT, which has a high degree of semantic abstraction and is used for task logic judgment; shallow spatial features are extracted from the middle layers of ViT, such as the PatchToken output of the 8th layer, which retains high-frequency geometric details such as object edges and textures and is used for spatial error correction. The dynamic decision adaptation layer, as the core training part of this embodiment, includes a semantic state locking module, a spatial residual error correction module, and a prompt dynamic aggregation module based on a gating network. Together, they constitute a lightweight external adaptation mechanism, realizing task logic verification, spatial deviation correction, and dynamic prompt fusion under the premise of frozen backbone network parameters.

[0069] In this embodiment of the home service robot, considering the unstructured characteristics of highly random distribution of items in the home environment and dynamic changes in lighting and occlusion, this embodiment uses a collaborative mechanism of semantic state locking and spatial residual error correction to enable the robot to identify stage deviations caused by environmental disturbances in real time when performing multi-step long-process tasks such as tidying and cooking. Under the constraints of a finite state machine, the robot can dynamically correct its motion trajectory, effectively avoiding task interruption or logical errors caused by single-frame perception noise, and significantly improving the continuous execution capability of service tasks and the reliability of user interaction in open home scenarios.

[0070] In this embodiment of flexible industrial assembly, addressing the deployment efficiency challenges posed by frequent production line changes, this embodiment leverages the rapid construction capabilities of the standard prompt library and the joint training mechanism of lightweight adaptation modules. This allows new assembly tasks to be adapted to the system based on only a few expert demonstrations. During the online phase, dynamic prompt fusion and state transition verification ensure that the assembly process maintains accuracy and logical rigor even under disturbances such as minor changes in workpiece pose and fixture switching. This achieves low-code production line reconstruction with "demonstration as deployment," significantly reducing the engineering debugging threshold and downtime costs in industrial settings.

[0071] In this embodiment of the medical care robot, to address the stringent requirements of both process timing and motion precision for operations such as drug administration and delivery, this embodiment deeply couples the logic blocking mechanism of a finite state machine with the ability to fine-tune spatial deviations: on the one hand, it enforces the verification of the timing legality of key steps such as "medication retrieval → verification → delivery" to prevent safety risks caused by illegal jumps; on the other hand, it performs centimeter-level dynamic correction of the end effector trajectory through standard reference feature anchoring and residual prompt generation, ensuring that the operation conforms to both medical standard procedures and meets the requirements of high-precision physical interaction, providing verifiable safety control guarantees for high-risk human-machine collaboration scenarios.

[0072] As shown in Figure 2, a control device for an embodied intelligent robot according to a second embodiment of the present invention is provided, used in any of the methods in the first aspect. The device includes: a receiving unit for receiving task language instructions and environmental visual observation data collected by the robot in real time; an acquisition unit for acquiring the corresponding original soft cue vector according to the task language instructions; a finite state machine for receiving environmental visual observation data and task language instructions, performing task logic state verification, and outputting a cue update judgment signal based on the verification result; a generation unit for generating a corrected soft cue vector based on the environmental visual observation data when the cue update judgment signal indicates that an update is needed; a fusion unit for fusing the corrected soft cue vector and the original soft cue vector to generate an updated cue; and an input unit for inputting the updated cue into a vision-language-action model, triggering the vision-language-action model to output instructions to control the robot to perform actions.

[0073] In one specific implementation, the receiving unit is implemented by the system communication interface module, configured to synchronously receive externally input task language instructions (preprocessed by the natural language parsing module) and environmental observation data streams output in real time by the robot's vision sensors, and distribute them to subsequent processing modules after being packaged in a standardized format.

[0074] The acquisition unit is implemented by the prompt retrieval logic module, which is configured to parse the semantic intent of the task language instruction, combine the current task stage identifier output by the finite state machine, match and output the standard soft prompt vector bound to the stage from the preset standard prompt library as the original soft prompt vector, and support a dual retrieval mechanism based on text semantic similarity or stage identifier index.

[0075] The finite state machine is implemented by a state verification engine, which integrates a pre-trained state discriminator and a task state transition rule base. The state discriminator performs stage probability parsing on deep semantic features, and the rule base performs time sequence legality verification based on historical locked states and current discrimination results. It implements logical blocking for illegal state transitions and outputs a binary prompt update judgment signal.

[0076] The generation unit is implemented by the bias correction engine. It is configured to call the pre-trained difference generator module after receiving the update judgment signal, perform feature alignment and bias quantization between the current shallow spatial features and the standard reference features, and generate a corrected soft cue vector representing the geometric perturbation of the environment. The consistency of the feature space is ensured throughout the process by relying on the frozen backbone network.

[0077] The fusion unit is implemented by a dynamic prompt fusion module, which is configured to input the original soft prompt vector and the modified soft prompt vector into a preset gated subnetwork. The gated subnetwork dynamically generates normalized weight coefficients based on real-time semantics and spatial context, performs weighted fusion, and outputs frame-level dynamic control prompts to ensure that the prompt content accurately adapts to the current task stage and environmental state.

[0078] The input unit is implemented by the prompt injection interface module, which is configured to convert the updated prompt into a prefix token sequence compatible with the visual-language-action model action decoder, embed it at the beginning of the decoder input layer, and trigger the model to generate discretized control instructions based on the prompt prefix, current visual features and historical action context through autoregression, and output them to the robot actuator through a standardized protocol.

[0079] A third embodiment of the present invention provides an electronic device, including a memory and a processor, wherein the memory is used to store a computer program executable by the processor; and the processor is used to execute the computer program in the memory to implement the method of any of the above embodiments.

[0080] Figure 3 is a block diagram illustrating an electronic device according to an exemplary embodiment. For example, the electronic device 900 may be provided as a server. Referring to Figure 3, the electronic device 900 includes a processing component 922, which further includes one or more processors, and memory resources represented by memory 932 for storing instructions executable by the processing component 922, such as application programs. The application programs stored in memory 932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 922 is configured to execute instructions to perform the methods described above.

[0081] Electronic device 900 may also include a power supply component 926 configured to perform power management of electronic device 900, a wired or wireless network interface 950 configured to connect electronic device 900 to a network, and an input / output interface 958. Electronic device 900 may operate on an operating system stored in memory 932, such as Windows Server™, MacOS X™, Unix™, Linux™, FreeBSD™, or similar.

[0082] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 932 including instructions, which can be executed by a processing component 922 of an electronic device 900 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device.

[0083] A fourth embodiment of the present invention provides a readable storage medium storing a program, which, when executed, implements the method of any of the above embodiments.

[0084] The fifth embodiment of the present invention provides a computer program product, including a computer program, which, when executed, implements the method of any of the above embodiments.

[0085] In this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The term "multiple" refers to two or more unless otherwise expressly defined.

[0086] The above description of the embodiments is intended to enable those skilled in the art to understand and apply the present invention. It will be apparent to those skilled in the art that various modifications can be made to these embodiments, and the general principles described herein can be applied to other embodiments without creative effort. Therefore, the present invention is not limited to the embodiments described herein, and any improvements and modifications made by those skilled in the art based on the disclosure of the present invention without departing from the scope and spirit of the invention are within the scope of the present invention.

Claims

1. A control method for an embodied intelligent robot, applied to an embodied intelligent robot system, wherein the system integrates a vision-language-motion model, characterized in that, The method includes: receiving task language instructions and real-time environmental visual observation data collected by the robot; obtaining the corresponding original soft cue vector based on the task language instructions; inputting the environmental visual observation data and the task language instructions into a pre-set finite state machine in the embodied intelligent robot system, whereby the finite state machine performs task logic state verification, obtains the verification result, and outputs a cue update judgment signal based on the verification result; when the cue update judgment signal indicates that an update is needed, generating a corrected soft cue vector based on the environmental visual observation data; fusing the corrected soft cue vector with the original soft cue vector to generate an updated cue; and inputting the updated cue into the vision-language-action model, causing the vision-language-action model to output instructions to control the robot to perform actions.

2. The method according to claim 1, characterized in that, The process of inputting the environmental visual observation data and the task language instructions into a pre-set finite state machine in the embodied intelligent robot system includes: extracting shallow spatial features from the environmental visual observation data, performing semantic parsing on the task language instructions to obtain deep semantic features, and inputting the shallow spatial features and the deep semantic features into the pre-set finite state machine; the pre-set finite state machine is used to perform task logic state verification based on the shallow spatial features and the deep semantic features and output a prompt update judgment signal.

3. The method according to claim 1, characterized in that, The finite state machine is pre-configured with task state transition rules associated with the task language instructions. The task state transition rules are constructed based on the expected state sequence of task execution and update the verified current task stage identifier. The task logic state verification is performed by the finite state machine to obtain the verification result, including: parsing the environmental visual observation data into candidate task stage identifiers; verifying whether the transition from the current task stage identifier to the candidate task stage identifier conforms to the task state transition rules, and obtaining the verification result; if the verification result is successful, updating the candidate task stage identifier to the current task stage identifier; if the verification result is unsuccessful, blocking the current task state transition and maintaining the current task stage identifier unchanged.

4. The method according to claim 1, characterized in that, Generating a corrected soft cue vector based on the environmental visual observation data includes: extracting spatial features from the environmental visual observation data to obtain current spatial features; inputting the current spatial features and standard reference features retrieved from a standard cue library into a difference generator; the difference generator is used to calculate the feature deviation between the current spatial features and the standard reference features, and generate the corrected soft cue vector based on the feature deviation.

5. The method according to claim 1, characterized in that, The process of fusing the corrected soft cue vector with the original soft cue vector to generate an updated cue includes: inputting the corrected soft cue vector and the original soft cue vector into a gating subnetwork, and having the gating subnetwork output a first weight coefficient and a second weight coefficient; weighting the original soft cue vector based on the first weight coefficient, and weighting the corrected soft cue vector based on the second weight coefficient; and adding the weighted original soft cue vector to the weighted corrected soft cue vector to obtain the updated cue.

6. The method according to claim 1 or 4, characterized in that, Before generating the corrected soft cue vector, the difference generator is trained offline. The offline training includes: acquiring expert demonstration data containing visual observation data, natural language text instructions, and corresponding expert real action sequences; extracting shallow spatial features and standard reference features of the current frame based on the visual observation data, and inputting them into the difference generator to generate the corrected soft cue vector; injecting the dynamic control cue containing the corrected soft cue vector, the natural language text instructions, and the visual features of the current frame into the parameter-frozen action decoder to output the predicted action; using the expert real action sequence as a supervision signal, and backpropagating end-to-end to update the network parameters of the difference generator based on the action regression error between the predicted action and the expert real action.

7. The method according to claim 6, characterized in that, The offline training is performed while keeping the backbone network parameters of the vision-language-action model frozen.

8. The method according to claim 2, characterized in that, After receiving task language instructions and real-time environmental visual observation data collected by the robot, the method further includes: extracting dual-stream features from the environmental visual observation data through the backbone network of the vision-language-action model with frozen parameters to obtain deep semantic features and shallow spatial features; wherein, the deep semantic features are used to represent global task logic, and the shallow spatial features are used to represent local geometric details.

9. The method according to claim 8, characterized in that, The process of inputting the environmental visual observation data and the task language instructions into the preset finite state machine includes: inputting the deep semantic features into a pre-trained state discriminator, whereby the state discriminator combines the prediction results of historical frames with the logic of the finite state machine to determine the state transition and output the current task stage identifier; the process of obtaining the corresponding original soft cue vector based on the task language instructions includes: retrieving a standard soft cue vector that matches the current task stage identifier from a preset standard cue library as the original soft cue vector.

10. The method according to claim 4 or 9, characterized in that, The standard prompt library stores standard reference features associated with the identifiers of each task stage; The step of generating the corrected soft cue vector based on the environmental visual observation data includes: inputting the shallow spatial features and the standard reference features into the difference generator, and having the difference generator calculate the feature deviation and generate the corrected soft cue vector.

11. The method according to claim 1, characterized in that, Before receiving the task language instructions, the method further includes: collecting corresponding observation data under standard environmental conditions for each preset task stage of the target task; extracting dual-stream features from the observation data through the backbone network of the parameter-frozen vision-language-action model to obtain deep semantic features and shallow spatial features corresponding to each task stage; determining the deep semantic features as the standard soft cue vector and the shallow spatial features as the standard reference features; and associating and storing the standard soft cue vector, the standard reference features, and the corresponding task stage identifiers to form the standard cue library.

12. A control device for an embodied intelligent robot, used in the method according to any one of claims 1 to 11, characterized in that, The device includes: a receiving unit for receiving task language instructions and real-time environmental visual observation data collected by the robot; an acquisition unit for acquiring the corresponding original soft cue vector based on the task language instructions; a finite state machine for receiving the environmental visual observation data and the task language instructions, performing task logic state verification, and outputting a cue update judgment signal based on the verification result; a generation unit for generating a corrected soft cue vector based on the environmental visual observation data when the cue update judgment signal indicates that an update is needed; a fusion unit for fusing the corrected soft cue vector with the original soft cue vector to generate an updated cue; and an input unit for inputting the updated cue into the vision-language-action model, triggering the vision-language-action model to output instructions to control the robot to perform actions.

13. An electronic device, characterized in that, The device includes a memory and a processor, the memory being used to store a computer program executable by the processor; the processor being used to execute the computer program in the memory to implement the method as described in any one of claims 1 to 11.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the executable computer program in the storage medium is executed by a processor, it can implement the method as described in any one of claims 1 to 11.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 11.

Images