An agent control method and apparatus, an electronic device, and a computer-readable medium

Abstract

The application discloses an agent control method and device, electronic equipment and a computer readable medium, and relates to the technical field of agent control. The method comprises the following steps: in response to an agent control task, obtaining current time step observation data comprising a current time step image and a current time step agent state, and a natural language task instruction, then performing multi-modal similarity retrieval to obtain a candidate sub-trajectory; performing trajectory-level alignment based on an agent state sequence according to the current time step agent state and the candidate sub-trajectory, selecting an optimal sub-trajectory from the candidate sub-trajectory according to the alignment cost required by the trajectory-level alignment; extracting a retrieval action sequence from the optimal sub-trajectory, performing cross-scene action adaptation based on the retrieval action sequence, a basic action sequence and the scene difference between a source scene and a target scene to obtain a final execution action of the current time step, so as to control the action of the agent at the current time step. The accuracy of controlling the agent is improved.

Description

Technical Field

[0001] This application relates to the field of intelligent agent control technology, and in particular to an intelligent agent control method, device, electronic device and computer-readable medium. Background Technology

[0002] Vision-Language-Action Models (VLAs) are widely used in robotic manipulation tasks. These models achieve end-to-end action generation by fusing visual information (RGB images), verbal commands, and robot state information. Existing methods typically rely on visual feature similarity matching, which may correspond to completely different action stages, leading to ambiguity in retrieval based solely on visual similarity. Directly reusing retrieved historical actions is problematic if the object positions, workbench layout, or camera perspective in the test scene differ from the source scene, rendering the original action trajectory unenforceable. Therefore, existing methods for agent control suffer from poor accuracy. Summary of the Invention

[0003] In view of this, embodiments of this application provide an intelligent agent control method, device, electronic device, and computer-readable medium, which can solve the technical problems of poor accuracy in existing intelligent agent control methods. These problems include the use of visual features for similarity matching, which may correspond to completely different action stages; ambiguity in retrieval based solely on visual similarity; and the inability to directly execute the original action trajectory if the object position, workbench layout, or camera view in the test scene differs from the source scene when directly reusing retrieved historical actions.

[0004] To achieve the above objectives, according to one aspect of the embodiments of this application, an agent control method is provided, comprising: in response to an agent control task, acquiring current time-step observation data and natural language task instructions, wherein the current time-step observation data includes a current time-step image and a current time-step agent state; based on the current time-step image and the natural language task instructions, performing multimodal similarity retrieval in a priori demonstration dataset to obtain candidate sub-trajectories; performing trajectory-level alignment based on the agent state sequence according to the current time-step agent state and the candidate sub-trajectories, and selecting the optimal sub-trajector from the candidate sub-trajectories according to the alignment cost required for trajectory-level alignment. The system extracts a retrieval action sequence from the optimal sub-trajectory. Based on the retrieval action sequence, the basic action sequence, and the scene differences between the source and target scenes, it performs cross-scene action adaptation to obtain the final execution action at the current time step. The basic action sequence is predicted by the basic action generation model based on continuous time step observation data and natural language task instructions. The continuous time step observation data ends with the current time step observation data. The retrieval action sequence and the basic action sequence have the same sequence length. The source scene corresponds to the optimal sub-trajectory, and the target scene corresponds to the agent control task. The system controls the agent's actions at the current time step based on the final execution action.

[0005] Optionally, based on the current time-step image and the natural language task instruction, multimodal similarity retrieval is performed in the prior demonstration dataset to obtain candidate sub-trajectories. This includes: extracting visual features from the continuous time-step observation images corresponding to the execution history window, wherein the continuous time-step observation images include the current time-step image, and the execution history window ends at the current time step; calculating visual similarity frame-by-frame between the execution history window and the candidate demonstration window based on the visual features, wherein the candidate demonstration window and the execution history window have the same window length, and the candidate demonstration window corresponds to the candidate demonstration trajectory in the prior demonstration dataset; extracting linguistic features from the natural language task instruction, and calculating linguistic similarity between the linguistic features and the linguistic features of the natural language task instruction corresponding to the candidate demonstration trajectory; for each candidate demonstration trajectory in the prior demonstration dataset, performing a weighted fusion calculation of the corresponding visual similarity and linguistic similarity to obtain a multimodal comprehensive similarity; and filtering the corresponding candidate demonstration trajectories based on the multimodal comprehensive similarity to obtain candidate sub-trajectories.

[0006] Optionally, after obtaining the multimodal comprehensive similarity, the agent control method further includes: when the largest multimodal comprehensive similarity in the multimodal comprehensive similarity is less than a preset similarity threshold, directly predicting the final execution action of the current time step based on the observation data of the current time step and the natural language task instructions through the basic action generation model; and controlling the agent's action at the current time step based on the final execution action predicted by the basic action generation model.

[0007] Optionally, candidate sub-trajectories are obtained by filtering the corresponding candidate demonstration trajectories based on the multimodal comprehensive similarity, including: when the largest multimodal comprehensive similarity in the multimodal comprehensive similarity is greater than or equal to a preset similarity threshold, a preset number of candidate demonstration trajectories corresponding to multimodal comprehensive similarities greater than or equal to the preset similarity threshold are selected from the candidate demonstration trajectories as candidate sub-trajectories.

[0008] Optionally, based on the agent state at the current time step and the candidate sub-trajectories, trajectory-level alignment based on the agent state sequence is performed. The optimal sub-trajectories are selected from the candidate sub-trajectories according to the alignment cost required for trajectory-level alignment. This includes: generating an agent state sequence matrix corresponding to the execution history window, where each row of the agent state sequence matrix corresponds to an agent state, and the last row of the agent state sequence matrix corresponds to the agent state at the current time step; constructing an agent state matching cost matrix based on the agent state sequence matrix and the agent state sequence matrix corresponding to the candidate demonstration window corresponding to the candidate sub-trajectories; and recursively obtaining the candidate sub-trajectories corresponding to the agent state sequence matrix corresponding to the execution history window with the minimum alignment cost by using a sub-sequence dynamic time warping algorithm based on the agent state matching cost matrix, and determining the candidate sub-trajectories corresponding to the minimum alignment cost as the optimal sub-trajectories.

[0009] Optionally, the scene difference is obtained as follows: First, the main-view visual features and wrist-view visual features of the observation images at consecutive time steps corresponding to the execution history window are extracted using a main-view visual encoder and a wrist-view visual encoder, respectively. Second, the state features of the agent state sequence matrix corresponding to the execution history window are extracted, where the execution history window corresponds to the trajectory segment ending at the current time step. Third, the main-view visual features, wrist-view visual features, and state features are fused using a fusion network to obtain the target scene vector. Fourth, the scene difference is obtained by calculating the difference between the target scene vector and the source scene vector using a scene difference modeling network, where the source scene vector is obtained based on the retrieval action sequence.

[0010] Optionally, cross-scene action adaptation is performed based on the retrieval action sequence, the basic action sequence, and the scene differences between the source and target scenes to obtain the final execution action for the current time step. This includes: encoding the retrieval action sequence and the basic action sequence to obtain the temporal context features of the retrieval action sequence and the basic action sequence, respectively; refining the temporal context features of the retrieval action sequence using a decoder with conditional memory based on the target scene to obtain retrieval action features; outputting retrieval candidate actions adapted to the target scene using a retrieval candidate action output head based on the retrieval action features; obtaining dimensional gating weights using a gating network based on the retrieval action features, the temporal context features of the basic action sequence, and the scene differences between the source and target scenes; and adapting the basic action of the current time step using an action adapter based on the dimensional gating weights and the retrieval condition residual to obtain the final execution action for the current time step. Here, the basic action of the current time step is the basic action of the last time step in the basic action sequence, and the retrieval condition residual is obtained based on the retrieval candidate actions and the basic action of the current time step.

[0011] Optionally, the action adapter is trained as follows: For each training scenario, a corresponding sample prior demonstration dataset is constructed, and training pairs are sampled from the sample prior demonstration dataset. The training pairs are sample trajectory pairs corresponding to the same sample natural language task instruction. The training scenario includes a sample source scenario and a sample target scenario. The sample real actions corresponding to the training pairs are obtained. Based on the sample trajectory pairs, sample real actions, and adapter optimization objectives, adapter training is performed to obtain the action adapter. The adapter optimization objective is obtained based on the final execution action prediction accuracy parameter, action time smoothness constraint parameter, loss weight, and gate weight supervision parameter. The final execution action prediction accuracy parameter and action time smoothness constraint parameter are obtained based on the sample final execution action and sample real action. The gate weight supervision parameter is obtained based on the sample retrieval candidate action, sample basic action, and sample real action. The sample final execution action is obtained from the sample trajectory pairs, and the sample retrieval candidate action and sample basic action are obtained based on the sample trajectory pairs.

[0012] In addition, this application also provides an intelligent agent control device, including: an acquisition unit configured to acquire current time step observation data and natural language task instructions in response to an intelligent agent control task, wherein the current time step observation data includes a current time step image and a current time step intelligent agent state; a multimodal similarity retrieval unit configured to perform multimodal similarity retrieval in a priori demonstration dataset based on the current time step image and the natural language task instructions to obtain candidate sub-trajectories; and a trajectory-level alignment unit configured to perform trajectory-level alignment based on the current time step intelligent agent state and candidate sub-trajectories, and select the optimal sub-trajectories from the candidate sub-trajectories according to the alignment cost required for trajectory-level alignment. The cross-scene action adaptation unit is configured to extract the retrieval action sequence from the optimal sub-trajectory, and perform cross-scene action adaptation based on the retrieval action sequence, the basic action sequence, and the scene differences between the source scene and the target scene to obtain the final execution action at the current time step. The basic action sequence is predicted by the basic action generation model based on continuous time step observation data and natural language task instructions. The continuous time step observation data ends with the current time step observation data. The retrieval action sequence and the basic action sequence have the same sequence length. The source scene corresponds to the optimal sub-trajectory, and the target scene corresponds to the agent control task. The agent control unit is configured to control the agent's actions at the current time step based on the final execution action.

[0013] Optionally, the multimodal similarity retrieval unit is further configured to: extract visual features of the continuous time-step observation images corresponding to the execution history window, wherein the continuous time-step observation images include the current time-step image, and the execution history window ends at the current time step; calculate visual similarity frame by frame between the execution history window and the candidate demonstration window based on the visual features, wherein the candidate demonstration window and the execution history window have the same window length, and the candidate demonstration window corresponds to the candidate demonstration trajectory in the prior demonstration dataset; extract linguistic features of the natural language task instructions, and calculate linguistic similarity between the linguistic features of the natural language task instructions corresponding to the candidate demonstration trajectories based on the linguistic features; for each candidate demonstration trajectory in the prior demonstration dataset, perform weighted fusion calculation on the corresponding visual similarity and linguistic similarity to obtain a multimodal comprehensive similarity; and filter the corresponding candidate demonstration trajectories based on the multimodal comprehensive similarity to obtain candidate sub-trajectories.

[0014] Optionally, the intelligent agent control device further includes a degradation processing unit, configured to: when the largest multimodal comprehensive similarity in the multimodal comprehensive similarity is less than a preset similarity threshold, directly predict the final execution action of the current time step based on the observation data of the current time step and the natural language task instructions through the basic action generation model; and control the action of the intelligent agent at the current time step based on the final execution action of the current time step predicted by the basic action generation model.

[0015] Optionally, the multimodal similarity retrieval unit is further configured to: when the largest multimodal comprehensive similarity in the multimodal comprehensive similarity is greater than or equal to a preset similarity threshold, select a preset number of candidate demonstration trajectories corresponding to multimodal comprehensive similarities greater than or equal to the preset similarity threshold from the candidate demonstration trajectories as candidate sub-trajectories.

[0016] Optionally, the trajectory-level alignment unit is further configured to: generate an agent state sequence matrix corresponding to the execution history window, wherein each row of the agent state sequence matrix corresponds to an agent state, and the last row of the agent state sequence matrix corresponds to the agent state at the current time step; construct an agent state matching cost matrix based on the agent state sequence matrix and the agent state sequence matrix corresponding to the candidate demonstration window of the candidate sub-trajectory; and, based on the agent state matching cost matrix, recursively obtain the candidate sub-trajectory corresponding to the one with the minimum alignment cost when aligned with the agent state sequence matrix corresponding to the execution history window by using a sub-sequence dynamic time warping algorithm, and determine the candidate sub-trajectory corresponding to the minimum alignment cost as the optimal sub-trajectory.

[0017] Optionally, the scene difference is obtained as follows: First, the main-view visual features and wrist-view visual features of the observation images at consecutive time steps corresponding to the execution history window are extracted using a main-view visual encoder and a wrist-view visual encoder, respectively. Second, the state features of the agent state sequence matrix corresponding to the execution history window are extracted, where the execution history window corresponds to the trajectory segment ending at the current time step. Third, the main-view visual features, wrist-view visual features, and state features are fused using a fusion network to obtain the target scene vector. Fourth, the scene difference is obtained by calculating the difference between the target scene vector and the source scene vector using a scene difference modeling network, where the source scene vector is obtained based on the retrieval action sequence.

[0018] Optionally, the cross-scene action adaptation unit is further configured to: encode the retrieval action sequence and the basic action sequence to obtain the temporal context features of the retrieval action sequence and the basic action sequence, respectively; refine and decode the temporal context features of the retrieval action sequence using a decoder with conditional memory based on the target scene to obtain retrieval action features; output retrieval candidate actions adapted to the target scene using a retrieval candidate action output head based on the retrieval action features; obtain dimensional gating weights using a gating network based on the retrieval action features, the temporal context features of the basic action sequence, and the scene differences between the source scene and the target scene; and adapt the basic action of the current time step using an action adapter based on the dimensional gating weights and the retrieval condition residual to obtain the final execution action of the current time step, wherein the basic action of the current time step is the basic action of the last time step in the basic action sequence, and the retrieval condition residual is obtained based on the retrieval candidate action and the basic action of the current time step.

[0019] Optionally, the action adapter is trained as follows: For each training scenario, a corresponding sample prior demonstration dataset is constructed, and training pairs are sampled from the sample prior demonstration dataset. The training pairs are sample trajectory pairs corresponding to the same sample natural language task instruction. The training scenario includes a sample source scenario and a sample target scenario. The sample real actions corresponding to the training pairs are obtained. Based on the sample trajectory pairs, sample real actions, and adapter optimization objectives, adapter training is performed to obtain the action adapter. The adapter optimization objective is obtained based on the final execution action prediction accuracy parameter, action time smoothness constraint parameter, loss weight, and gate weight supervision parameter. The final execution action prediction accuracy parameter and action time smoothness constraint parameter are obtained based on the sample final execution action and sample real action. The gate weight supervision parameter is obtained based on the sample retrieval candidate action, sample basic action, and sample real action. The sample final execution action is obtained from the sample trajectory pairs, and the sample retrieval candidate action and sample basic action are obtained based on the sample trajectory pairs.

[0020] In addition, this application also provides an intelligent agent control electronic device, including: one or more processors; and a storage device for storing one or more programs, which, when executed by one or more processors, enable the one or more processors to implement the intelligent agent control method as described above.

[0021] In addition, this application also provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the intelligent agent control method as described above.

[0022] To achieve the above objectives, according to another aspect of the embodiments of this application, a computer program product is provided.

[0023] A computer program product according to an embodiment of this application includes a computer program that, when executed by a processor, implements the intelligent agent control method provided in an embodiment of this application.

[0024] One embodiment of the above invention has the following advantages or beneficial effects: This application, in response to an agent control task, acquires current time-step observation data and natural language task instructions, wherein the current time-step observation data includes the current time-step image and the current time-step agent state; based on the current time-step image and natural language task instructions, multimodal similarity retrieval is performed in a priori demonstration dataset to obtain candidate sub-trajectories; trajectory retrieval is performed through visual-language joint similarity calculation, reducing visual ambiguity during trajectory retrieval; based on the current time-step agent state and candidate sub-trajectories, trajectory-level alignment based on the agent state sequence is performed; the optimal sub-trajectory is selected from the candidate sub-trajectories according to the alignment cost required for trajectory-level alignment; and motion layer is performed using the agent state sequence. Trajectory alignment effectively filters visually similar trajectories but with different motion stages, and accurately selects the optimal sub-trajectory based on alignment costs. Retrieved action sequences are extracted from the optimal sub-trajectories. Cross-scene action adaptation is performed based on the retrieved action sequences, basic action sequences, and scene differences between the source and target scenes to obtain the final execution action for the current time step. The basic action sequence is predicted by a basic action generation model based on continuous time-step observation data and natural language task instructions. The continuous time-step observation data ends with the current time-step observation data. The retrieved action sequence and the basic action sequence have the same sequence length. The source scene corresponds to the optimal sub-trajectory, and the target scene corresponds to the agent's control task. The agent's actions at the current time step are controlled based on the final execution action. Cross-scene action adaptation avoids directly reusing retrieved historical actions, improving the accuracy of agent control.

[0025] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0026] The accompanying drawings are provided to better understand this application and do not constitute an undue limitation thereof.

[0027] Figure 1 This is a schematic diagram of the main flow of an intelligent agent control method according to an embodiment of this application.

[0028] Figure 2 This is a schematic diagram of the main flow of an intelligent agent control method according to an embodiment of this application.

[0029] Figure 3 This is a schematic flowchart of the main process of an intelligent agent control method according to an embodiment of this application.

[0030] Figure 4 This is a schematic diagram of the main units of an intelligent agent control device according to an embodiment of this application.

[0031] Figure 5 This is an exemplary system architecture diagram in which embodiments of this application can be applied.

[0032] Figure 6 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers in the embodiments of this application. Detailed Implementation

[0033] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These embodiments should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description. It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solutions of this application comply with relevant national laws and regulations. It should also be noted that certain software, components, models, and other existing industry solutions may be mentioned in the embodiments of this application. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solutions of this application, and do not imply that the applicant has already used or necessarily used such solutions. The collection, gathering, updating, analysis, processing, use, transmission, and storage of user personal information involved in the technical solutions of this application all comply with relevant laws and regulations, are used for legal and reasonable purposes, do not violate public order and good morals, are not shared, disclosed, or sold outside of these legal uses, and are subject to supervision and management by regulatory authorities. Necessary measures should be taken to prevent unauthorized access to user personal information data, safeguard user personal information security and network security, and ensure that personnel authorized to access personal information data comply with relevant laws and regulations. Once this user personal information data is no longer needed, the risk should be minimized by restricting or even prohibiting data collection and / or deleting the data.

[0034] When used, including in certain relevant applications, data is deidentified to protect user privacy, for example by removing specific identifiers, controlling the amount or specificity of stored data, controlling how data is stored, and / or other methods.

[0035] The following is an explanation of some terms that may be used in this application.

[0036] VLA (Vision-Language-Action Model): This model integrates visual input, language commands, and robot state to output control actions.

[0037] CLIP: A contrastive learning model that maps images and text to the same feature space.

[0038] Faiss: An efficient vector similarity retrieval library.

[0039] DTW (Dynamic Time Warping): A dynamic time warping algorithm used to calculate the alignment distance between two time series.

[0040] S-DTW (Subsequence Dynamic Time Warping): Subsequence dynamic time warping is used to find the best subsequence match.

[0041] Transformer: An encoder is a sequence modeling network structure based on self-attention and cross-attention mechanisms.

[0042] LSTM (Long Short-Term Memory): A network used to model time-series dependencies.

[0043] Multimodal fusion: unifying the representation of different modal features such as vision, language, and robot state.

[0044] Zero-shot transfer: Performing a task directly without training on the new task.

[0045] Action Adapter: A lightweight network module used to map retrieved actions to the current scene.

[0046] Scene Difference Embedding: A feature vector representing the difference between the source scene and the target scene.

[0047] Figure 1 This is a schematic diagram of the main flow of an intelligent agent control method according to an embodiment of this application, as shown below. Figure 1 As shown, the intelligent agent control method mainly includes the following steps S101-S105.

[0048] Step S101: In response to the agent control task, acquire the current time step observation data and natural language task instructions, wherein the current time step observation data includes the current time step image and the current time step agent state.

[0049] In this embodiment, when the agent control method's execution entity (e.g., a server) receives the agent control task, it can acquire observation data (e.g., the current time step image) at the current time step (e.g., t). and the current time step agent state (where 1, ..., N are camera numbers) and natural language task instructions (e.g., please pour me a glass of water).

[0050] Step S102: Based on the current time step image and natural language task instructions, perform multimodal similarity retrieval in the prior demonstration dataset to obtain candidate sub-trajectories.

[0051] Specifically, based on the current time-step image and natural language task instructions, multimodal similarity retrieval is performed in the prior demonstration dataset to obtain candidate sub-trajectories, including: extracting the execution history window (e.g., The corresponding continuous time step (e.g., from the ) The time step to the (at each time step) observed images (e.g., The visual features of images observed at consecutive time steps (e.g., This includes the current time step image (e.g., ), Execution history window (corresponding to from the first The time step to the (Time steps) with the current time step t as the endpoint; based on visual features, calculate the visual similarity frame by frame between the execution history window and the candidate demonstration window, where the candidate demonstration window and the execution history window have the same window length, and the candidate demonstration window corresponds to the candidate demonstration trajectory in the prior demonstration dataset. For example, calculate the cosine similarity frame by frame between the execution history window and the candidate demonstration window, using the formula: .in, Indicates the first Visual similarity from the perspective of each camera; Indicates the length of the time window; Indicates the time offset within the window; and These represent the visual features of the current execution history window and the candidate demonstration window at the corresponding time step and under the corresponding camera, respectively; superscript This represents the vector transpose; the norm in the denominator represents the cosine similarity calculated after normalizing the feature vectors; linguistic features of the natural language task instructions are extracted, and the linguistic similarity between the linguistic features and the linguistic features of the natural language task instructions corresponding to the candidate demonstration trajectories is calculated based on these linguistic features. For example, the formula for calculating linguistic similarity is... .in, Indicates language similarity; This represents the feature vector corresponding to the extracted linguistic features of the current natural language task instruction; This represents the feature vector corresponding to the linguistic features of the natural language task instruction extracted from the candidate demonstration trajectory; superscript This represents the vector transpose; the norm in the denominator represents the cosine similarity calculated after normalizing the language features; for each candidate demonstration trajectory in the prior demonstration dataset, the corresponding visual similarity and language similarity are weighted and fused to obtain the multimodal comprehensive similarity. For example, the final multimodal weighted fusion calculation of the multimodal comprehensive similarity is given by the formula: .in, Indicates the final multimodal synthesis similarity; Indicates the number of cameras; Indicates the first Weights of visual similarity between cameras; Indicates the first Visual similarity from the perspective of each camera; Indicates language similarity weights; Indicates language similarity.

[0052] The weights satisfy: .in, Indicates the first Weights of visual similarity between cameras; The weights represent the language similarity; this constraint ensures that the sum of the weights of all visual similarities and the language similarities is 1, guaranteeing the scale stability of the weighted fusion result. Candidate demonstration trajectories are then selected based on the multimodal comprehensive similarity to obtain candidate sub-trajectories.

[0053] Specifically, after obtaining the multimodal comprehensive similarity, the agent control method further includes: when the largest multimodal comprehensive similarity in the multimodal comprehensive similarity is less than a preset similarity threshold, the final action to be executed at the current time step is directly predicted by the basic action generation model based on the observation data of the current time step and the natural language task instructions; and the agent's action at the current time step is controlled based on the final action to be executed at the current time step predicted by the basic action generation model.

[0054] For example, when the maximum multimodal integrated similarity is less than a preset similarity threshold... (For example, at 0.8), the execution of the basic action generation model is directly rolled back to ensure stability in low-confidence retrieval scenarios.

[0055] Specifically, candidate demonstration trajectories are screened based on multimodal comprehensive similarity to obtain candidate sub-trajectories, including: when the largest multimodal comprehensive similarity in the multimodal comprehensive similarity is greater than or equal to a preset similarity threshold, a preset number of candidate demonstration trajectories corresponding to multimodal comprehensive similarities greater than or equal to the preset similarity threshold are selected from the candidate demonstration trajectories as candidate sub-trajectories.

[0056] For example, when the maximum multimodal integrated similarity is greater than or equal to a preset similarity threshold... At that time, a preset number (e.g., Top-K) of candidate demonstration trajectories with a similarity threshold greater than or equal to the preset number are accurately selected. The candidate demonstration trajectory corresponding to the multimodal comprehensive similarity is used as the candidate sub-trajectory. When the maximum multimodal comprehensive similarity is greater than or equal to the preset similarity threshold, the candidate sub-trajectory is selected. When the search path is entered, the action adapter directly generates the adapted action (i.e., the final action is executed), and the basic action generation model does not participate in the online inference of this path.

[0057] Step S103: Based on the agent state at the current time step and the candidate sub-trajectories, perform trajectory-level alignment based on the agent state sequence, and select the optimal sub-trajectories from the candidate sub-trajectories according to the alignment cost required for trajectory-level alignment.

[0058] Specifically, based on the agent's state at the current time step and candidate sub-trajectories, trajectory-level alignment based on the agent's state sequence is performed. The optimal sub-trajector is selected from the candidate sub-trajectories based on the alignment cost required for trajectory-level alignment. This includes: obtaining the agent's historical state data corresponding to the agent's control task; and generating an execution history window based on this historical state data (e.g., ...). The corresponding agent state sequence matrix (e.g., In this matrix, each row of the agent state sequence corresponds to an agent state (or a robot state if the agent is a robot), and the last row of the agent state sequence matrix from top to bottom corresponds to the agent state at the current time step (e.g., ...). Based on the agent's state sequence matrix (e.g., ,in, This represents the dimension of the robot's state vector; (representing the real number space; n+1 is the window length of the execution history window) and the agent state sequence matrix corresponding to the candidate demonstration window of the candidate sub-trajectory (e.g., , Indicates the first The robot state sequence matrix corresponding to the candidate demonstration window of each candidate demonstration trajectory; This represents the window length of the candidate demonstration window corresponding to the candidate demonstration trajectory, for example, n+1; This represents the dimension of the robot's state vector; Represent the space of real numbers; (representing the candidate demonstration window number corresponding to the candidate demonstration trajectory), construct the agent state matching cost matrix (e.g., ,in, This indicates the robot state sequence matrix corresponding to the current execution history window. In the robot state sequence matrix corresponding to the candidate demonstration window of the candidate demonstration trajectory, the i-th state is... The cost of matching between states; This represents the first element of the robot state sequence matrix corresponding to the current execution history window. Each state is represented by a vector; Indicates the first The first candidate demonstration trajectory corresponds to the candidate demonstration window of the robot state sequence matrix. (Each state is represented by a vector; the subscript 2 denotes the Euclidean distance); based on the agent's state matching cost matrix, a subsequence dynamic time warping algorithm is used for recursion (the corresponding recursion formula, for example...). ,in, Indicates the position in the S-DTW dynamic programming table Alignment cost; This indicates the robot state sequence matrix corresponding to the current execution history window. In the robot state sequence matrix corresponding to the candidate demonstration window of the candidate demonstration trajectory, the i-th state is... The matching cost between states, i.e. the matching cost of the state at the current position; , , These represent the cumulative costs of the three transitional preceding states, respectively. This recursive formula represents the selection of the alignment path with the minimum cost, yielding the candidate sub-trajectory with the minimum alignment cost required to align with the agent's state sequence matrix corresponding to the execution history window (specifically, this can be based on the formula). We obtained, among which, , This represents the minimum alignment cost between the candidate demonstration window corresponding to the current candidate demonstration trajectory and the execution history window; Indicates the end position in the candidate demonstration window corresponding to the candidate demonstration trajectory; This indicates that the length is The execution history window is aligned to the candidate demonstration window corresponding to the candidate demonstration trajectory. The alignment cost at each position), accurately determining the candidate sub-trajectory corresponding to the minimum alignment cost as the optimal sub-trajectory, for example, the optimal sub-trajectory. ,in, This represents the observation data in the source demonstration scenario (i.e., the source scenario). This represents the actions in the source demonstration scene (i.e., the source scene). This indicates the length of the candidate sub-trajectory that was ultimately selected (i.e., the optimal sub-trajectory).

[0059] Step S104: Extract the retrieval action sequence from the optimal sub-trajectory. Perform cross-scene action adaptation based on the retrieval action sequence, the basic action sequence, and the scene differences between the source scene and the target scene to obtain the final execution action at the current time step. The basic action sequence is predicted by the basic action generation model based on continuous time step observation data and natural language task instructions. The continuous time step observation data ends with the current time step observation data. The retrieval action sequence and the basic action sequence have the same sequence length. The source scene corresponds to the optimal sub-trajectory, and the target scene corresponds to the agent control task.

[0060] For example, from the determined optimal sub-trajectory (e.g., ,in, This represents the source demonstration scenario (i.e., the source scenario, where t...). [1, The observation data in ]) This represents the source demonstration scenario (i.e., the source scenario, where t...). [1, The actions in ]) Extracting the length of the finally selected candidate sub-trajectory (i.e., the optimal sub-trajectory) from the length of the candidate sub-trajectory) is... The retrieval action sequence (e.g., Based on this retrieval action sequence (e.g., ), basic action sequences (e.g., ,in, This represents the basic action predicted by the basic action generation model in the current target scene, where t [1, L]) and the scene differences between the source scene and the target scene. Cross-scene action adaptation is performed by using gated residual fusion based on the retrieved action sequence, the basic action sequence, and the scene differences between the source and target scenes to obtain the final action to be executed at the current time step t. .

[0061] Specifically, the scene differences are obtained based on the following method: using a first-person perspective visual encoder respectively. and wrist-view visual encoder Extract the execution history window (e.g., The corresponding continuous time step observation images (e.g., ) the main perspective visual features (e.g., ) and wrist-view visual features (e.g., ); Extract execution history window (e.g., The state features of the corresponding agent state sequence matrix (e.g., , ,in, This represents a state-coded network; taking a robot as an example, Then it means from the first The time step to the A continuous sequence of robot states at each time step; (representing the hidden feature dimension), where the execution history window corresponds to the trajectory segment ending at the current time step t; through a fusion network (e.g., ) of the main viewpoint visual features (e.g., ), visual features from the wrist perspective (e.g., ) and state characteristics (e.g., ) are fused to obtain the target scene vector (e.g., ); calculate the target scene vector through a scene difference modeling network (e.g., ) and source scene vector (e.g., , and The calculation method is similar and will not be repeated here. The difference between these differences yields the scene differences (e.g., The source scene vector can be obtained based on the retrieval action sequence.

[0062] Step S105: Control the agent's actions at the current time step based on the final executed action.

[0063] The final control command for the current time step can be generated based on the final execution action, so as to accurately control the action of the agent (e.g., a robot) at the current time step, such as pouring water.

[0064] This embodiment responds to the agent control task by acquiring the current time step observation data and natural language task instructions. The current time step observation data includes the current time step image and the current time step agent state. Based on the current time step image and natural language task instructions, multimodal similarity retrieval is performed on the prior demonstration dataset to obtain candidate sub-trajectories. Trajectory retrieval is performed through visual-language joint similarity calculation, reducing visual ambiguity during trajectory retrieval. Based on the current time step agent state and candidate sub-trajectories, trajectory-level alignment based on the agent state sequence is performed. The optimal sub-trajector is selected from the candidate sub-trajectories based on the alignment cost required for trajectory-level alignment. Motion-level trajectory alignment is performed using the agent state sequence, effectively filtering... Trajectories that are visually similar but differ in their motion stages are identified, and the optimal sub-trajectory is accurately selected based on alignment costs. Retrieved action sequences are extracted from these optimal sub-trajectories. Cross-scene action adaptation is then performed based on the retrieved action sequences, the basic action sequences, and scene differences between the source and target scenes to obtain the final execution action for the current time step. The basic action sequences are predicted by a basic action generation model based on continuous time-step observation data and natural language task instructions. The continuous time-step observation data ends at the current time-step observation data. The retrieved action sequences and the basic action sequences have the same sequence length. The source scene corresponds to the optimal sub-trajectory, and the target scene corresponds to the agent's control task. The agent's actions at the current time step are controlled based on the final execution action. Cross-scene action adaptation avoids directly reusing retrieved historical actions, improving the accuracy of agent control.

[0065] Figure 2 This is a schematic flowchart of the main process of an intelligent agent control method according to an embodiment of this application, as follows: Figure 2 As shown, the intelligent agent control method mainly includes the following steps S201-S210.

[0066] Step S201: In response to the agent control task, acquire the current time step observation data and natural language task instructions, wherein the current time step observation data includes the current time step image (e.g., ) and the agent's state at the current time step (e.g., ).

[0067] Step S202: Based on the current time step image and natural language task instructions, perform multimodal similarity retrieval in the prior demonstration dataset to obtain candidate sub-trajectories.

[0068] Multimodal similarity retrieval can be, for example, a fusion of visual and linguistic similarity retrieval. By performing multimodal similarity retrieval on a prior demonstration dataset based on the current time step image and natural language task instructions, visual ambiguity during trajectory retrieval can be reduced, and candidate sub-trajectories can be accurately obtained.

[0069] Step S203: Based on the agent state at the current time step and the candidate sub-trajectories, perform trajectory-level alignment based on the agent state sequence, and select the optimal sub-trajectories from the candidate sub-trajectories according to the alignment cost required for trajectory-level alignment.

[0070] For example, an agent state matching cost matrix can be constructed based on the agent state at the current time step and candidate sub-trajectories. Trajectory-level alignment based on the agent state sequence can be performed based on this agent state matching cost matrix, and the candidate sub-trajectories corresponding to the minimum alignment cost required for trajectory-level alignment can be determined as the optimal sub-trajectories.

[0071] Step S204: Extract the retrieval action sequence from the optimal sub-trajectory.

[0072] For example, extracting a length of from the optimal sub-trajectory. The retrieval action sequence (e.g., ).

[0073] Step S205: Encode the retrieval action sequence and the basic action sequence to obtain the temporal context features of the retrieval action sequence and the temporal context features of the basic action sequence, respectively.

[0074] For example, encoding the retrieval action sequence and the basic action sequence separately yields the following results: .

[0075] in, This represents the temporal context features of the retrieved action sequence, where t-1 represents the previous time step of the current time step t, i.e., the (t-1)th time step. Indicates the retrieval action encoder; The temporal context features representing the basic action sequence are shown, where t-1 represents the previous time step of the current time step t, i.e., the (t-1)th time step. This represents the basic motion encoder; This represents the sequence of retrieval actions extracted from the optimal sub-trajectory (e.g., ) One search action; Represents the first action in the basic action sequence A basic movement; This represents the Long Short-Term Memory (LSTM) network.

[0076] Step S206: The temporal context features of the retrieved action sequence are refined and decoded by the decoder using conditional memory based on the target scene to obtain the retrieved action features.

[0077] Through a decoder (e.g., a Transformer decoder) to use conditional memory based on the target scene (e.g., ) for the temporal context features of the retrieved action sequence (e.g., Specifically, it can be ) Perform refined decoding to obtain the retrieval action features, for example, , specifically: , , .

[0078] in, This represents the sequence to be decoded, composed of the temporal context features of the retrieval action sequence; This represents the memory of the current target scene conditions, based on the visual features of the current main viewpoint. Visual features from the wrist perspective State characteristics Differences in scenarios composition.

[0079] Step S207: Output candidate actions adapted to the target scene by retrieving candidate action output heads based on the retrieved action features.

[0080] For example, this can be achieved by retrieving candidate action output headers (e.g., ) based on retrieval action features (e.g., Output the search candidate actions adapted to the target scene. , specifically: ,in, This indicates the output header for retrieving candidate actions; Indicates the action dimension. Represents the space of real numbers.

[0081] Step S208: The gating network is used to obtain the dimension-wise gating weights based on the retrieved action features, the temporal context features of the basic action sequence, and the scene differences between the source scene and the target scene.

[0082] Gated residual fusion can be performed using lightweight gating networks (e.g., ) to retrieve action features (e.g., ), and the temporal context features of the basic action sequence (e.g., ) and scene differences between the source scene and the target scene (e.g., ), to obtain the dimension-wise gate weights ,in, This indicates a feature concatenation operation. This represents the Sigmoid function. , Indicates the action dimension.

[0083] Step S209: The basic action of the current time step is adapted by the action adapter to perform stepwise correction based on the dimensional gating weight and the retrieval condition residual, so as to obtain the final execution action of the current time step. The basic action of the current time step is the basic action of the last time step in the basic action sequence, and the retrieval condition residual is obtained based on the retrieval candidate action and the basic action of the current time step.

[0084] For example, the final action to be executed at the current time step. It can be obtained through the following formula: Equivalent land That is, the basic action at the current time step through the action adapter. The superimposed weights are dimension-wise gated weights. Controlled search condition residuals ,in, This indicates element-wise multiplication.

[0085] Specifically, the action adapter is trained as follows: For each training scenario, a corresponding sample prior demonstration dataset is constructed, and training pairs are sampled from the sample prior demonstration dataset. Each training pair is a pair of sample trajectories corresponding to the same natural language task instruction. Each training scenario includes a sample source scenario and a sample target scenario. Real actions corresponding to the training pairs are obtained. For example, training the action adapter: For each training scenario... Construct a 10-shot sample prior demonstration dataset Training pairs are sampled from the prior demonstration dataset of this sample. For sample trajectory pairs of the same task (i.e., the same natural language task instruction), for example... The sampling time difference satisfies The aligned subsequences are trained; based on sample trajectory pairs, sample real actions, and adapter optimization objectives, adapter training is performed to obtain an action adapter; wherein, the adapter optimization objective is obtained based on the final execution action prediction accuracy parameter, action time smoothness constraint parameter, loss weight, and gate weight supervision parameter; the final execution action prediction accuracy parameter and action time smoothness constraint parameter are obtained based on the sample final execution action and sample real action; the gate weight supervision parameter is obtained based on the sample retrieval candidate action, sample basic action, and sample real action; the sample final execution action is obtained from the sample trajectory pairs; the sample retrieval candidate action and sample basic action are obtained based on the sample trajectory pairs. For example, the adapter optimization objective is set as follows: .in, These are parameters for the final action prediction accuracy, used to monitor the final action prediction accuracy. These are parameters constraining the smoothness of motion time; These are the gate weight supervision parameters, used to supervise the gate weights; and For loss weights. , .in, This indicates the final action performed by the sample, specifically, That is, the retrieval condition residuals controlled by gating are superimposed on the basic actions of the sample; This represents the actual actions of the samples in the target training trajectory; Represents the L1 norm; The predicted action difference is constrained to match the actual action difference to reduce action abrupt changes. Sampled actual actions are only used for supervised gating during the training phase. Let the normalized action be denoted as the underlined action variable, and the gating temperature be... ,but , .in, Indicates the target of gate control monitoring; , and These represent the normalized sample retrieval candidate actions, sample basic actions, and sample actual actions, respectively. This represents the gating temperature. The gating supervision objective encourages the gating to increase the weight of the retrieval residual when the candidate action of the sample retrieval is closer to the actual action of the sample, and to decrease the weight of the retrieval correction when the basic action of the sample is closer to the actual action of the sample (i.e., more reliable).

[0086] Step S210: Control the agent's actions at the current time step based on the final executed action.

[0087] The final control command for the current time step can be generated based on the final execution action, so as to accurately control the action of the agent (e.g., a robot or a robot dog) at the current time step.

[0088] Figure 3 This is a schematic flowchart of the main process of an intelligent agent control method according to an embodiment of this application. Figure 3 As shown, the executing agent responds to the intelligent agent control task. At the start of the task, it executes the following intelligent agent control loop: acquires the observation data of the current time step and the natural language task instructions, where the current time step is, for example, t, and the current time step observation data includes the current time step image. and the current time step agent state Based on the image at the current time step and the natural language task instructions, multimodal similarity retrieval is performed on the prior demonstration dataset to obtain candidate sub-trajectories. Trajectory-level alignment based on the agent state sequence is performed according to the current time step and the candidate sub-trajectories. The optimal sub-trajectory is selected from the candidate sub-trajectories based on the alignment cost required for trajectory-level alignment. A retrieval action sequence is extracted from the optimal sub-trajectory. Cross-scene action adaptation is performed based on the retrieval action sequence, the basic action sequence, and the scene differences between the source and target scenes to obtain the final execution action at the current time step t. The basic action sequence is predicted by the basic action generation model based on continuous time-step observation data and natural language task instructions. The continuous time-step observation data ends with the current time-step observation data. The retrieved action sequence has the same sequence length as the basic action sequence. The source scene corresponds to the optimal sub-trajectory, and the target scene corresponds to the agent control task. Based on the final executed action... Control the agent's actions at the current time step. Determine if the task is complete or the maximum number of steps has been reached. If so, the task ends; otherwise, continue acquiring observation data at the current time step and execute the agent control loop described above until the task is complete or the maximum number of steps has been reached, at which point the task ends.

[0089] In this application, the intelligent agent can be a robot or a robot dog. The embodiments of this application do not specifically limit the intelligent agent. Taking the intelligent agent as a robot as an example, the implementation of the intelligent agent control method of this application involves: freezing the basic vision-language-action model (Base VLA Policy), that is, the basic action generation model of this application; multimodal similarity retrieval module; trajectory-level reordering module based on S-DTW; cross-scene action adapter module (the only trainable part); action output module. The data flow involved is as follows: (1) the robot at the current time step (e.g., the 1st time step) The current time step observation data obtained at each time step) and natural language task instructions, among which, Indicates the first Each time step consists of RGB images captured by cameras numbered 1, ..., N. Representative at the (2) The multimodal similarity retrieval module retrieves Top-K candidate sub-trajectories in the prior demonstration dataset; (3) The trajectory-level reordering module based on S-DTW performs trajectory-level alignment based on the robot state sequence and selects the optimal sub-trajectories; (4) The retrieved action sequence is extracted from the optimal sub-trajectories; (5) Cross-scene action adaptation module: the action adapter performs cross-scene transfer transformation to achieve cross-scene action adaptation; (6) The adapted action (i.e. the final execution action) is input into the frozen VLA policy structure and the final control command is output to control the agent's action at the current time step. During the training phase, only the action adapter parameters are updated and all other modules are frozen; during the testing phase, no parameter updates are performed to achieve zero-sample transfer execution. This application achieves accurate robot action generation based on "retrieval enhancement + cross-scene action transfer". Without changing the parameters of the original large-scale pre-trained basic vision-language-action model (Base VLA Policy) (i.e. the basic action generation model of this application), cross-task and cross-scene generalization is achieved through external prior demonstration dataset retrieval and lightweight action adapter.

[0090] For example, let the prior demonstration dataset be: .in, This represents the prior demonstration dataset; Indicates the first One demonstration trajectory; Indicates the relationship with the first The language task instructions corresponding to each trajectory; Indicates the total number of demonstration trajectories; Indicates the trajectory number.

[0091] The trajectory is as follows: .in, This represents a complete robot execution trajectory; It can represent the first The observation data at each time step (i.e., the observation data at the current time step of this application); Indicates the first Robot control actions output or recorded at each time step; This indicates the total number of time steps contained in the trajectory; Indicates the time step index.

[0092] Observation: .in, Indicates the robot in the The observation data at each time step (i.e., the observation set, which is also the current time step observation data in this application); Indicates the first The time step is from the first RGB images captured by one camera; Indicates the number of cameras; Indicates the first Robot state at each time step; This indicates the camera's serial number.

[0093] The system involved in implementing the intelligent agent control method of this application includes a frozen, large-scale pre-trained base vision-language-action model (Base VLA Policy) (i.e., the base action generation model of this application), which contains a large-scale pre-trained base policy. Furthermore, it does not update parameters throughout the entire training and testing process.

[0094] To avoid mismatches caused by relying solely on visual similarity, this application employs a joint visual and linguistic similarity calculation. First, the CLIP encoder is used to extract visual and linguistic features separately: the visual feature vector corresponding to the visual features is as follows: ,in, Indicates the extracted first The time step, from the first Visual feature vectors corresponding to the visual features of RGB images captured by each camera; Indicates CLIP image encoder; Indicates the first The time step, from the first RGB images captured by one camera. Language feature vectors corresponding to language features: ,in, This represents the language feature vector corresponding to the language features extracted from the natural language task instructions; Indicates CLIP text encoder; This represents the input natural language task instructions.

[0095] For the execution history window: .in, Indicates the current number An execution history window with a time step as the endpoint; to Indicates from the first The time step to the Continuous observation data at each time step; This indicates the number of steps the execution history window has moved backward, and the window length of this execution history window is [value missing]. .

[0096] With candidate demonstration window: .in, This represents a candidate demonstration window (also a candidate segment) within a candidate demonstration trajectory. to This indicates that the candidate demonstration window starts from the first... The time step to the Continuous observation data at each time step; This indicates the time position index of the candidate demonstration window.

[0097] Calculate cosine similarity frame by frame: .in, Indicates the first Visual similarity from the perspective of each camera; Indicates the length of the time window; Indicates the time offset within the window; and These represent the visual features of the current execution history window and the candidate demonstration window at the corresponding time step and under the corresponding camera, respectively; superscript The denominator represents the transpose of a vector; the norm in the denominator represents the cosine similarity calculated after normalizing the eigenvectors.

[0098] Computing language similarity: .in, Indicates language similarity; This represents the feature vector corresponding to the extracted linguistic features of the current natural language task instruction; This represents the feature vector corresponding to the linguistic features of the natural language task instruction extracted from the candidate demonstration trajectory; superscript The vector transpose is represented; the norm in the denominator represents the cosine similarity calculated after normalizing the language features.

[0099] Finally, multimodal weighted fusion is used to calculate the multimodal comprehensive similarity: .in, Indicates the final multimodal synthesis similarity; Indicates the number of cameras; Indicates the first Weights of visual similarity between cameras; Indicates the first Visual similarity from the perspective of each camera; Indicates language similarity weights; Indicates language similarity.

[0100] The weights satisfy: .in, Indicates the first Weights of visual similarity between cameras; The weights represent the language similarity; this constraint means that the sum of the weights of all visual similarities and the weights of language similarities is 1, which is used to ensure the scale stability of the weighted fusion result.

[0101] A vector index library can be built using Faiss to select the Top-K sub-trajectories with the highest similarity as candidate sub-trajectories. When the maximum multimodal comprehensive similarity is less than a preset similarity threshold... When the similarity is 0.8 (e.g., 0.8), the execution of the basic action generation model is directly reverted to ensure stability in low-confidence retrieval scenarios; when the maximum multimodal comprehensive similarity is greater than or equal to the preset similarity threshold... When the search path is entered, the action adapter directly generates the adapted action (i.e., the final action is executed), and the basic action generation model does not participate in the online inference of this path.

[0102] Visually similar segments may belong to different motion stages, therefore, time-series alignment based on robot state is introduced. Let: , .in, This represents the robot state sequence matrix corresponding to the current execution history window, with each row corresponding to a robot state. Indicates the first The robot state sequence matrix corresponding to the candidate demonstration window of each candidate demonstration trajectory; This represents the window length of the candidate demonstration window corresponding to the candidate demonstration trajectory, for example, n+1; This represents the dimension of the robot's state vector; Represent the space of real numbers; This indicates the number of the candidate demonstration window corresponding to the candidate demonstration trajectory.

[0103] Construct the Euclidean distance cost matrix (i.e., the agent state matching cost matrix of this application): .in, This indicates the robot state sequence matrix corresponding to the current execution history window. In the robot state sequence matrix corresponding to the candidate demonstration window of the candidate demonstration trajectory, the i-th state is... The cost of matching between states; This represents the first element of the robot state sequence matrix corresponding to the current execution history window. Each state (represented by a vector); Indicates the first The first candidate demonstration trajectory corresponds to the candidate demonstration window of the robot state sequence matrix. There are 1 state (represented by a vector); the subscript 2 indicates the Euclidean distance.

[0104] The recursive formula is obtained by using the S-DTW algorithm. .in, Indicates the position in the S-DTW dynamic programming table Alignment cost; This indicates the robot state sequence matrix corresponding to the current execution history window. In the robot state sequence matrix corresponding to the candidate demonstration window of the candidate demonstration trajectory, the i-th state is... The matching cost between states, i.e. the matching cost of the state at the current position; , , These represent the cumulative costs of the three transitional preceding states. The recursive formula represents the selection of the alignment path with the minimum cost.

[0105] To obtain the minimum alignment cost: .in, This represents the minimum alignment cost between the candidate demonstration window corresponding to the current candidate demonstration trajectory and the execution history window; Indicates the end position in the candidate demonstration window corresponding to the candidate demonstration trajectory; This indicates that the length is The execution history window is aligned to the candidate demonstration window corresponding to the candidate demonstration trajectory. Alignment cost at each position.

[0106] Select the candidate example trajectory (which is also the candidate sub-trajectory here) with the minimum alignment cost. .in, This indicates the selected optimal candidate demonstration trajectory segment, i.e., the candidate demonstration window corresponding to the selected optimal candidate demonstration trajectory. Indicates the selection of alignment cost The smallest candidate demonstration trajectory segment (i.e., the candidate demonstration window corresponding to the candidate demonstration trajectory, i.e., the obtained demonstration window). This step ensures that the selected optimal sub-trajectory is consistent with the current task execution phase of the agent's control task in terms of motion pattern.

[0107] After multimodal retrieval and S-DTW reordering, a consistent demonstration window for the task execution phase is obtained. Unlike directly copying the demonstration actions, this application further queries the underlying freezing strategy. The output of the basic action generation model is used as the execution anchor point, and the gating residual correction is estimated by the retrieval trajectory and scene differences. This balances the original capabilities of the basic action generation model and the cross-scene transferability of the retrieval demonstration.

[0108] Let the finally selected candidate sub-trajectory (i.e., the optimal sub-trajectory) be... Extract the length of from it. retrieval action sequence Preferred The aim is to achieve an optimal balance between the motion jitter easily caused by single-step prediction and the sluggish response to environmental changes resulting from long-sequence prediction, thereby ensuring both the temporal continuity of motion execution and the real-time feedback capability of closed-loop control. Simultaneously, the underlying strategy for querying and freezing is also considered. This yields the basic action sequence corresponding to the current observation history. . This represents the observation data in the source demonstration scenario (i.e., the source scenario). This represents the actions in the source demonstration scene (i.e., the source scene). This represents the length of the candidate sub-trajectory that was ultimately selected (i.e., the optimal sub-trajectory). This represents the basic action predicted by the basic action generation model in the current target scene.

[0109] Multimodal scene encoding. Visual features are extracted from the execution history window observation data of the current target scene using both first-person and wrist-view visual encoders: .in, Indicates the first The main view visual features extracted from the main view camera images at each time step by the main view visual encoder; Indicates the first Wrist view visual features extracted from wrist view camera images at each time step using a wrist view visual encoder. and These represent the main-view visual encoder and the wrist-view visual encoder, respectively. Each visual encoder contains four convolutional blocks and outputs fixed-dimensional features using spatial downsampling and adaptive average pooling.

[0110] The robot's historical state is encoded using a two-layer MLP: .in, State characteristics representing the robot state corresponding to the execution history window; Represents a state-coded network; Indicates from the first The time step to the A continuous sequence of robot states at each time step; This indicates the hidden feature dimension.

[0111] Visual features (i.e., first-person view visual features + wrist view visual features) and state features are fused into a unified scene representation (i.e., a scene vector): .in, This represents the unified scene representation after fusion (i.e., scene vector). This represents a fusion network used to integrate multimodal features; Indicates the visual characteristics of the main perspective; Indicates visual characteristics from the wrist perspective; Indicates the robot's state characteristics; This indicates a feature stitching operation. It displays the current observation history and the selected demonstration window. The target scene vectors are obtained by performing feature fusion as described above using the same fusion module. and source scene vector .

[0112] Scene difference modeling. Given the current scene representation (i.e., the target scene vector) and the selected source demonstration scene representation (i.e., the source scene vector), the scene difference (i.e., the scene difference vector in this application) is calculated: .in, This represents the scene difference vector between the source scene and the current target scene; Represents a scene difference modeling network; This represents the current target scene representation (i.e., the target scene vector). This represents the source demonstration scene representation obtained from the retrieval (i.e., the source scene vector). This indicates a feature concatenation operation. The scene difference vector is used in subsequent retrieval action candidate generation and gated residual fusion.

[0113] This application encodes the retrieval action sequence and the basic action sequence separately. The retrieval action sequence provides transferable motion priors, while the basic action sequence provides execution anchors generated by the frozen basic action generation model (i.e., the basic VLA model) in the current target scene. The gating module determines the correction strength of the retrieval candidate action to the basic action dimension by dimension according to the relationship between the two and the scene differences.

[0114] Encode the retrieval action sequence and the basic action sequence separately: .in, This represents the temporal context features of the retrieved action sequence; Indicates the retrieval action encoder; Temporal context features representing basic action sequences; This represents the basic motion encoder; The sequence of retrieval actions extracted from the optimal retrieval trajectory (i.e., the optimal sub-trajectory of this application) represents the retrieval action sequence of the retrieval action sequence. One search action; Represents the first action in the basic action sequence A basic movement; This represents the Long Short-Term Memory (LSTM) network.

[0115] The Transformer decoder maps the sequence of retrieval actions to candidate retrieval actions in the current target scene: , , , .in, This represents the sequence to be decoded, composed of the temporal context features of the retrieval action sequence; This represents the memory of the current target scene conditions, based on the visual features of the current main viewpoint. Visual features from the wrist perspective State characteristics Differences in scenarios composition; This represents the retrieval action features refined by the Transformer decoder; This indicates the search candidate actions after adapting to the current target scenario; This indicates the output header for retrieving candidate actions; Indicates the action dimension.

[0116] Gated residual fusion: This application sets up a lightweight gating network to control the repair-by-repair strength of the candidate actions to the basic actions. , .in, Represents the dimension-wise gating weights; Represents the Sigmoid function; Indicates a gating network; This represents element-wise multiplication; Indicates the final action to be performed. Equivalently, That is, the basic action at the current time step through the action adapter. The superimposed weights are dimension-wise gated weights. Controlled search condition residuals .

[0117] Training for the motion adapter: For each training scenario Construct a 10-shot sample prior demonstration dataset Training pairs are sampled from the prior demonstration dataset of this sample. For sample trajectory pairs of the same task (i.e., the same natural language task instruction), for example... The sampling time difference satisfies The aligned subsequences are used for training. If the training scene and the test scene are the same, the same action adapter is used for both training and test retrieval. If the test scenario and the training scenario are different, for example... Then, another sample prior demonstration dataset is constructed based on sample data related to the target scene according to the same sampling protocol. And multimodal similarity retrieval is performed within it during testing.

[0118] The adapter optimization goal is set as follows: .in, These are parameters for the final action prediction accuracy, used to monitor the final action prediction accuracy. These are parameters constraining the smoothness of motion time; These are the gate weight supervision parameters, used to supervise the gate weights; and For loss weights. , .in, This indicates the final action performed by the sample, specifically, That is, the retrieval condition residuals controlled by gating are superimposed on the basic actions of the sample; This represents the actual actions of the samples in the target training trajectory; Represents the L1 norm; The predicted action difference is constrained to match the actual action difference to reduce action abrupt changes. Sampled actual actions are only used for supervised gating during the training phase. Let the normalized action be denoted as the underlined action variable, and the gating temperature be... ,but , .in, Indicates the target of gate control monitoring; , and These represent the normalized sample retrieval candidate actions, sample basic actions, and sample actual actions, respectively. This represents the gating temperature. The gating supervision objective encourages the gating to increase the weight of the retrieval residual when the candidate action of the sample retrieval is closer to the actual action of the sample, and to decrease the weight of the retrieval correction when the basic action of the sample is closer to the actual action of the sample (i.e., more reliable).

[0119] During the testing phase, real actions from samples are not used; gating weights are applied one dimension at a time. Completely Gated Network The retrieval action features are obtained by refining the temporal context features of the retrieval action sequence through a decoder based on the conditional memory of the target scene. Temporal context features of basic action sequences Scene differences between the source scene and the current target scene Prediction. Optimal settings. , And using AdamW, learning rate Training is performed using a cosine annealing strategy. During the testing phase for new tasks: real-time observation acquisition; retrieval + S-DTW reordering; cross-scene action adaptation; generation of the final action; and iterative execution. The entire process involves no parameter updates, achieving zero-sample transfer. This application forms a complete closed loop through "multimodal similarity retrieval + motion alignment reordering + explicit scene difference modeling + Transformer action transfer," significantly improving cross-scene generalization ability without changing the parameters of the original large model (i.e., the basic action generation model), achieving pluggable, low computational overhead, and deployment-friendly characteristics.

[0120] This application implements a pluggable architecture for retrieval enhancement that freezes the basic action generation model. Through a structural design of "completely freezing the basic action generation model + external retrieval module + lightweight action adapter" (i.e., "freezing the basic action generation model + external retrieval enhancement + cross-scene action transfer"), it achieves capability enhancement without updating the parameters of the basic action generation model during training and testing, relying solely on external retrieval and action adaptation. A non-parametric retrieval module provides task-related contextual information, and an independently trained action adapter completes cross-scene action mapping, thus avoiding modification of the internal parameters of the basic action generation model and decoupling capability enhancement from the original model's capabilities. A multimodal weighted similarity retrieval mechanism is implemented, using joint visual-language weighted similarity calculation and employing a time window alignment method to calculate the average similarity across multiple frames, rather than single-frame matching. Visual similarity is obtained by averaging the cosine similarity frame by frame within the time window. Linguistic semantic constraints are introduced to reduce visual ambiguity; a time window averaging mechanism is introduced to enhance temporal consistency; and a confidence threshold mechanism (the confidence threshold used, for example, a preset similarity threshold) is set to avoid interference from low-quality retrieval. A trajectory-level reordering mechanism based on robot state sequences is implemented. This mechanism not only relies on visual similarity but also utilizes robot state sequences for motion-level alignment, constructing a Euclidean distance cost matrix (i.e., the agent state matching cost matrix in this application). Subsequence dynamic time warping (S-DTW) is used, and the candidate sub-trajectory with the minimum alignment cost is selected as the optimal sub-trajectory. A cross-scene action transfer mechanism with explicit scene difference modeling is implemented, explicitly characterizing the differences between the source and target scenes and using them as conditional inputs for action transfer. An "LSTM encoding + Transformer encoder cross-attention mechanism" is employed to refine the retrieved action sequences, enabling each action vector to dynamically consider current visual features, scene differences, and temporal action context. This structure achieves conditional action transfer rather than a simple linear transformation. A three-stage closed-loop enhancement mechanism of retrieval + alignment + transfer is implemented: multimodal coarse retrieval; fine alignment and reordering of state sequences; and cross-scene action transfer. This forms a hierarchical filtering + adaptation closed-loop mechanism. In this application, the encoding of the retrieved action sequences and basic action sequences can also utilize DINO series visual encoders or ViT series pre-trained models. Multimodal similarity can be calculated using methods such as cosine similarity, dot product similarity, inverse Euclidean distance, learned similarity networks, or dual-tower structural similarity models. Motion-level alignment using robot states can also be achieved through standard DTW, Transformer-based sequence matching networks, or stage alignment methods based on Hidden Markov Models. Scene differences can be calculated using direct vector subtraction, dual-tower difference networks, conditional normalization modulation, or gated fusion mechanisms.The encoding during action refinement can also be achieved using a Cross-Attention module, a bidirectional LSTM, a diffusion model action generator, or a conditional MLP network. Furthermore, to realize the "freezing of the basic action generation model + external retrieval enhancement + cross-scene action transfer" proposed in this application, a single-stage joint ranking network can be used to replace "retrieval + re-ranking," and the scene difference can be directly input into the input of the basic action generation model, or the retrieval results can be used as prompts input into the basic action generation model.

[0121] Figure 4 This is a schematic diagram of the main units of an intelligent agent control device according to an embodiment of this application. Figure 4 As shown, the intelligent agent control device 400 includes an acquisition unit 401, a multimodal similarity retrieval unit 402, a trajectory-level alignment unit 403, a cross-scene action adaptation unit 404, and an intelligent agent control unit 405.

[0122] The acquisition unit 401 is configured to acquire the current time step observation data and natural language task instructions in response to the agent control task, wherein the current time step observation data includes the current time step image and the current time step agent state.

[0123] The multimodal similarity retrieval unit 402 is configured to perform multimodal similarity retrieval in the prior demonstration dataset based on the current time step image and natural language task instructions to obtain candidate subtrajectories.

[0124] The trajectory alignment unit 403 is configured to perform trajectory alignment based on the agent state sequence at the current time step and the candidate sub-trajectories, and select the optimal sub-trajectories from the candidate sub-trajectories according to the alignment cost required for trajectory alignment.

[0125] The cross-scene action adaptation unit 404 is configured to extract the retrieval action sequence from the optimal sub-trajectory, and perform cross-scene action adaptation based on the retrieval action sequence, the basic action sequence, and the scene differences between the source scene and the target scene to obtain the final execution action at the current time step. The basic action sequence is predicted by the basic action generation model based on continuous time step observation data and natural language task instructions. The continuous time step observation data ends with the current time step observation data. The retrieval action sequence and the basic action sequence have the same sequence length. The source scene corresponds to the optimal sub-trajectory, and the target scene corresponds to the agent control task.

[0126] The agent control unit 405 is configured to control the agent's actions at the current time step based on the final executed action.

[0127] In some embodiments, the multimodal similarity retrieval unit 402 is further configured to: extract visual features of the continuous time-step observation images corresponding to the execution history window, wherein the continuous time-step observation images include the current time-step image, and the execution history window ends at the current time step; calculate visual similarity frame by frame between the execution history window and the candidate demonstration window based on the visual features, wherein the candidate demonstration window and the execution history window have the same window length, and the candidate demonstration window corresponds to the candidate demonstration trajectory in the prior demonstration dataset; extract linguistic features of the natural language task instructions, and calculate linguistic similarity between the linguistic features of the natural language task instructions corresponding to the candidate demonstration trajectories based on the linguistic features; for each candidate demonstration trajectory in the prior demonstration dataset, perform weighted fusion calculation on the corresponding visual similarity and linguistic similarity to obtain a multimodal comprehensive similarity; and filter the corresponding candidate demonstration trajectories based on the multimodal comprehensive similarity to obtain candidate sub-trajectories.

[0128] In some embodiments, the intelligent agent control device further includes Figure 4 The downgrade processing unit (not shown) is configured to: when the largest multimodal comprehensive similarity in the multimodal comprehensive similarity is less than a preset similarity threshold, directly predict the final execution action of the current time step based on the observation data of the current time step and the natural language task instructions through the basic action generation model; and control the agent's action at the current time step based on the final execution action predicted by the basic action generation model.

[0129] In some embodiments, the multimodal similarity retrieval unit 402 is further configured to: when the largest multimodal comprehensive similarity in the multimodal comprehensive similarity is greater than or equal to a preset similarity threshold, select a preset number of candidate demonstration trajectories corresponding to multimodal comprehensive similarities greater than or equal to the preset similarity threshold from the candidate demonstration trajectories as candidate sub-trajectories.

[0130] In some embodiments, the trajectory-level alignment unit 403 is further configured to: generate an agent state sequence matrix corresponding to the execution history window, wherein each row of the agent state sequence matrix corresponds to an agent state, and the last row of the agent state sequence matrix corresponds to the agent state at the current time step; construct an agent state matching cost matrix based on the agent state sequence matrix and the agent state sequence matrix corresponding to the candidate demonstration window of the candidate sub-trajectory; and, based on the agent state matching cost matrix, recursively obtain the candidate sub-trajectory corresponding to the one with the minimum alignment cost when aligned with the agent state sequence matrix corresponding to the execution history window by using a sub-sequence dynamic time warping algorithm, and determine the candidate sub-trajectory corresponding to the minimum alignment cost as the optimal sub-trajectory.

[0131] In some embodiments, scene differences are obtained as follows: First, first-person view visual features and wrist-view visual features of the observation images at consecutive time steps corresponding to the execution history window are extracted using a first-person view visual encoder and a wrist-view visual encoder, respectively. Second, state features of the agent state sequence matrix corresponding to the execution history window are extracted, where the execution history window corresponds to the trajectory segment ending at the current time step. Third, the first-person view visual features, wrist-view visual features, and state features are fused using a fusion network to obtain the target scene vector. Fourth, the scene differences are obtained by calculating the difference between the target scene vector and the source scene vector using a scene difference modeling network, where the source scene vector is obtained based on the retrieval action sequence.

[0132] In some embodiments, the cross-scene action adaptation unit 404 is further configured to: encode the retrieval action sequence and the basic action sequence to obtain the temporal context features of the retrieval action sequence and the temporal context features of the basic action sequence, respectively; refine the temporal context features of the retrieval action sequence using a decoder with conditional memory based on the target scene to obtain retrieval action features; output retrieval candidate actions adapted to the target scene using a retrieval candidate action output head based on the retrieval action features; obtain dimensional gating weights using a gating network based on the retrieval action features, the temporal context features of the basic action sequence, and the scene differences between the source scene and the target scene; and adapt the basic action of the current time step using an action adapter based on the dimensional gating weights and the retrieval condition residual to obtain the final execution action of the current time step, wherein the basic action of the current time step is the basic action of the last time step in the basic action sequence, and the retrieval condition residual is obtained based on the retrieval candidate action and the basic action of the current time step.

[0133] In some embodiments, the action adapter is trained as follows: For each training scenario, a corresponding sample prior demonstration dataset is constructed, and training pairs are sampled from the sample prior demonstration dataset. Each training pair is a pair of sample trajectories corresponding to the same sample natural language task instruction. The training scenario includes a sample source scenario and a sample target scenario. Real sample actions corresponding to the training pairs are obtained. Based on the sample trajectory pairs, real sample actions, and the adapter optimization objective, adapter training is performed to obtain the action adapter. The adapter optimization objective is obtained based on the final execution action prediction accuracy parameter, action time smoothness constraint parameter, loss weight, and gate weight supervision parameter. The final execution action prediction accuracy parameter and action time smoothness constraint parameter are obtained based on the final execution action of the sample and the real sample action. The gate weight supervision parameter is obtained based on the sample retrieval candidate action, the sample basic action, and the real sample action. The final execution action of the sample is obtained from the sample trajectory pairs, and the sample retrieval candidate action and the sample basic action are obtained based on the sample trajectory pairs.

[0134] It should be noted that the intelligent agent control method and intelligent agent control device of this application are related in terms of specific implementation content, so repeated content will not be described again.

[0135] Figure 5 An exemplary system architecture 500 is shown that can be applied to the agent control method or agent control device of the present application embodiments.

[0136] like Figure 5 As shown, system architecture 500 may include terminal devices 501, 502, and 503, a network 504, and a server 505. Network 504 serves as the medium for providing communication links between terminal devices 501, 502, and 503 and server 505. Network 504 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0137] Users can use terminal devices 501, 502, and 503 to interact with server 505 via network 504 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 501, 502, and 503, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0138] Terminal devices 501, 502, and 503 can be various electronic devices with intelligent agent control processing screens and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0139] Server 505 can be a server providing various services, such as a background management server (for example only) supporting agent control tasks triggered by users using terminal devices 501, 502, and 503. The background management server can respond to agent control tasks by acquiring current time-step observation data and natural language task instructions. The current time-step observation data includes the current time-step image and the current time-step agent state. Based on the current time-step image and natural language task instructions, it performs multimodal similarity retrieval in the prior demonstration dataset to obtain candidate sub-trajectories. Trajectory retrieval is performed through visual-language joint similarity calculation to reduce visual ambiguity during trajectory retrieval. Based on the current time-step agent state and candidate sub-trajectories, trajectory-level alignment based on the agent state sequence is performed. The optimal sub-trajectory is selected from the candidate sub-trajectories based on the alignment cost required for trajectory-level alignment. Motion-level trajectory alignment is then performed using the agent state sequence, effectively... Trajectories that are visually similar but differ in their motion phases are filtered, and the optimal sub-trajectory is accurately selected based on alignment costs. Retrieved action sequences are extracted from these optimal sub-trajectories. Cross-scene action adaptation is performed based on the retrieved action sequences, the basic action sequences, and the scene differences between the source and target scenes to obtain the final execution action for the current time step. The basic action sequences are predicted by a basic action generation model based on continuous time-step observation data and natural language task instructions. The continuous time-step observation data ends with the current time-step observation data. The retrieved action sequences and the basic action sequences have the same sequence length. The source scene corresponds to the optimal sub-trajectory, and the target scene corresponds to the agent's control task. The agent's actions at the current time step are controlled based on the final execution action. Cross-scene action adaptation avoids directly reusing retrieved historical actions, improving the accuracy of agent control.

[0140] It should be noted that the intelligent agent control method provided in this application embodiment is generally executed by server 505, and correspondingly, the intelligent agent control device is generally set in server 505.

[0141] It should be understood that Figure 5 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0142] The following is for reference. Figure 6 It shows a schematic diagram of the structure of a computer system 600 suitable for implementing a terminal device according to the embodiments of this application. Figure 6 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0143] like Figure 6As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 602 or programs loaded from storage section 608 into random access memory (RAM) 603. The RAM 603 also stores various programs and data required for the operation of the computer system 600. The CPU 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0144] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.

[0145] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by central processing unit (CPU) 601, it performs the functions defined above in the system of this application.

[0146] It should be noted that the computer-readable medium shown in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. Computer-readable storage media can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0147] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0148] The units described in the embodiments of this application can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including an acquisition unit, a multimodal similarity retrieval unit, a trajectory-level alignment unit, a cross-scene action adaptation unit, and an intelligent agent control unit. The names of these units do not necessarily limit the specific unit itself.

[0149] In another aspect, this application also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs that, when executed by the device, cause the device to respond to an agent control task by acquiring current time-step observation data and natural language task instructions. The current time-step observation data includes a current time-step image and the current time-step agent state. Based on the current time-step image and the natural language task instructions, multimodal similarity retrieval is performed in a priori demonstration dataset to obtain candidate sub-trajectories. Trajectory retrieval is performed through visual-language joint similarity calculation to reduce visual ambiguity during trajectory retrieval. Based on the current time-step agent state and candidate sub-trajectories, trajectory-level alignment based on the agent state sequence is performed. The optimal sub-trajector is selected from the candidate sub-trajectories based on the alignment cost required for trajectory-level alignment. The system aligns motion-level trajectories with the body state sequence, effectively filtering visually similar trajectories but different motion stages, and accurately selects the optimal sub-trajectory based on the alignment cost. It then extracts a retrieved action sequence from the optimal sub-trajectory. Based on the retrieved action sequence, the basic action sequence, and scene differences between the source and target scenes, it performs cross-scene action adaptation to obtain the final execution action for the current time step. The basic action sequence is predicted by a basic action generation model based on continuous time-step observation data and natural language task instructions. The continuous time-step observation data ends at the current time-step observation data. The retrieved action sequence and the basic action sequence have the same sequence length. The source scene corresponds to the optimal sub-trajectory, and the target scene corresponds to the agent's control task. The final execution action controls the agent's actions at the current time step. Through cross-scene action adaptation, the system avoids directly reusing retrieved historical actions, improving the accuracy of agent control.

[0150] The computer program product of this application includes a computer program that, when executed by a processor, implements the intelligent agent control method in the embodiments of this application.

[0151] According to the technical solution of this application embodiment, trajectory retrieval is performed by visual-language joint similarity calculation to reduce visual ambiguity during trajectory retrieval; trajectory alignment at the motion level is performed by using the agent state sequence to effectively filter visually similar trajectories but different motion stages, and the optimal sub-trajectory is accurately selected based on the alignment cost; cross-scene action adaptation avoids directly reusing retrieved historical actions and improves the accuracy of controlling the agent.

[0152] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for controlling an intelligent agent, characterized in that, include: In response to the agent control task, the current time step observation data and natural language task instructions are acquired, wherein the current time step observation data includes the current time step image and the current time step agent state; Based on the current time step image and the natural language task instructions, multimodal similarity retrieval is performed in the prior demonstration dataset to obtain candidate sub-trajectories; Based on the agent state at the current time step and the candidate sub-trajectories, trajectory-level alignment based on the agent state sequence is performed, and the optimal sub-trajectories are selected from the candidate sub-trajectories according to the alignment cost required for trajectory-level alignment. The retrieval action sequence is extracted from the optimal sub-trajectory. Cross-scene action adaptation is performed based on the retrieval action sequence, the basic action sequence, and the scene differences between the source scene and the target scene to obtain the final execution action at the current time step. The basic action sequence is predicted by the basic action generation model based on continuous time step observation data and the natural language task instructions. The continuous time step observation data ends with the current time step observation data. The retrieval action sequence has the same sequence length as the basic action sequence. The source scene corresponds to the optimal sub-trajectory, and the target scene corresponds to the agent control task. The final executed action controls the agent's actions at the current time step.

2. The method according to claim 1, characterized in that, Based on the current time step image and the natural language task instructions, multimodal similarity retrieval is performed in the prior demonstration dataset to obtain candidate sub-trajectories, including: Extract visual features from the continuous time step observation images corresponding to the execution history window, wherein the continuous time step observation images include the current time step image, and the execution history window ends at the current time step; Based on the visual features, the visual similarity between the execution history window and the candidate demonstration window is calculated frame by frame, wherein the candidate demonstration window and the execution history window have the same window length, and the candidate demonstration window corresponds to the candidate demonstration trajectory in the prior demonstration dataset; Extract the linguistic features of the natural language task instruction, and calculate the linguistic similarity between the linguistic features of the natural language task instruction corresponding to the candidate demonstration trajectory and the linguistic features of the natural language task instruction. For each candidate demonstration trajectory in the prior demonstration dataset, the corresponding visual similarity and linguistic similarity are weighted and fused to obtain the multimodal comprehensive similarity. Based on the multimodal comprehensive similarity, the corresponding candidate demonstration trajectories are screened to obtain candidate sub-trajectories.

3. The method according to claim 2, characterized in that, After obtaining the multimodal integrated similarity, the method further includes: When the largest multimodal comprehensive similarity among the multimodal comprehensive similarities is less than a preset similarity threshold, the final execution action of the current time step is directly predicted by the basic action generation model based on the current time step observation data and the natural language task instructions. The final action to be executed at the current time step is predicted based on the basic action generation model, which controls the agent's actions at the current time step.

4. The method according to claim 2, characterized in that, The step of filtering corresponding candidate demonstration trajectories based on the multimodal comprehensive similarity to obtain candidate sub-trajectories includes: When the largest multimodal comprehensive similarity among the multimodal comprehensive similarities is greater than or equal to a preset similarity threshold, a preset number of candidate demonstration trajectories corresponding to multimodal comprehensive similarities greater than or equal to the preset similarity threshold are selected from the candidate demonstration trajectories as candidate sub-trajectories.

5. The method according to claim 2, characterized in that, The step of performing trajectory-level alignment based on the agent state sequence at the current time step and the candidate sub-trajectories, and selecting the optimal sub-trajectories from the candidate sub-trajectories according to the alignment cost required for trajectory-level alignment, includes: Generate an agent state sequence matrix corresponding to the execution history window, wherein each row of the agent state sequence matrix corresponds to an agent state, and the last row of the agent state sequence matrix corresponds to the agent state at the current time step; Based on the agent state sequence matrix and the agent state sequence matrix corresponding to the candidate demonstration window of the candidate sub-trajectory, an agent state matching cost matrix is ​​constructed. Based on the agent state matching cost matrix, the candidate sub-trajectory corresponding to the minimum alignment cost when aligning with the agent state sequence matrix corresponding to the execution history window is obtained by recursion through the sub-sequence dynamic time warping algorithm. The candidate sub-trajectory corresponding to the minimum alignment cost is determined as the optimal sub-trajectory.

6. The method according to claim 5, characterized in that, The scenario differences are obtained based on the following method: The main view visual features and wrist view visual features of the observation images corresponding to the continuous time steps of the execution history window are extracted using the main view visual encoder and the wrist view visual encoder, respectively. Extract the state features of the agent state sequence matrix corresponding to the execution history window, wherein the execution history window corresponds to the trajectory segment ending at the current time step; The target scene vector is obtained by fusing the main view visual features, the wrist view visual features, and the state features through a fusion network. The scene difference is obtained by calculating the difference between the target scene vector and the source scene vector through a scene difference modeling network, wherein the source scene vector is obtained based on the retrieval action sequence.

7. The method according to claim 1, characterized in that, The process of performing cross-scene action adaptation based on the retrieved action sequence, the basic action sequence, and the scene differences between the source and target scenes to obtain the final execution action at the current time step includes: The retrieval action sequence and the basic action sequence are encoded to obtain the temporal context features of the retrieval action sequence and the temporal context features of the basic action sequence, respectively. The temporal context features of the retrieval action sequence are refined and decoded using a decoder based on conditional memory of the target scene to obtain retrieval action features; By retrieving candidate action output headers, retrieved candidate actions adapted to the target scene are output based on the retrieved action features; A gating network is used to obtain dimension-wise gating weights based on the retrieval action features, the temporal context features of the basic action sequence, and the scene differences between the source scene and the target scene. The action adapter performs stepwise correction on the basic action of the current time step based on the dimension-wise gating weights and the retrieval condition residuals to obtain the final execution action of the current time step. The basic action of the current time step is the basic action of the last time step in the basic action sequence, and the retrieval condition residuals are obtained based on the retrieval candidate actions and the basic action of the current time step.

8. The method according to claim 7, characterized in that, The action adapter is trained in the following manner: For each training scenario, a corresponding sample prior demonstration dataset is constructed, and training pairs are sampled from the sample prior demonstration dataset. The training pairs are sample trajectory pairs corresponding to the same sample natural language task instruction. The training scenario includes a sample source scenario and a sample target scenario. The sample real actions corresponding to the training pairs are obtained. Based on the sample trajectory pairs, the actual actions of the samples, and the adapter optimization objective, adapter training is performed to obtain the action adapter; The adapter optimization objective is obtained based on the final execution action prediction accuracy parameter, the action time smoothness constraint parameter, the loss weight, and the gate weight supervision parameter. The final execution action prediction accuracy parameter and the action time smoothness constraint parameter are obtained based on the sample's final execution action and the sample's actual action. The gate weight supervision parameter is obtained based on the sample retrieval candidate action, the sample's basic action, and the sample's actual action. The sample's final execution action is obtained from the sample trajectory pair, and the sample retrieval candidate action and the sample's basic action are obtained based on the sample trajectory pair.

9. An intelligent agent control device, characterized in that, include: The acquisition unit is configured to acquire current time step observation data and natural language task instructions in response to an agent control task, wherein the current time step observation data includes the current time step image and the current time step agent state. The multimodal similarity retrieval unit is configured to perform multimodal similarity retrieval in the prior demonstration dataset based on the current time step image and the natural language task instruction to obtain candidate sub-trajectories; The trajectory-level alignment unit is configured to perform trajectory-level alignment based on the agent state sequence at the current time step and the candidate sub-trajectories, and select the optimal sub-trajectories from the candidate sub-trajectories according to the alignment cost required for trajectory-level alignment. A cross-scene action adaptation unit is configured to extract a retrieval action sequence from the optimal sub-trajectory, and perform cross-scene action adaptation based on the retrieval action sequence, the basic action sequence, and the scene differences between the source scene and the target scene to obtain the final execution action at the current time step. The basic action sequence is predicted by a basic action generation model based on continuous time step observation data and the natural language task instructions. The continuous time step observation data ends with the current time step observation data. The retrieval action sequence has the same sequence length as the basic action sequence. The source scene corresponds to the optimal sub-trajectory, and the target scene corresponds to the agent control task. The agent control unit is configured to control the agent's actions at the current time step based on the final executed action.

10. An intelligent agent-controlled electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-8.

11. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-8.

Images