Action instruction generation method and device, equipment and storage medium

Abstract

The invention discloses an action instruction generation method and device, equipment and a storage medium. The accuracy of model reasoning and the accuracy of action instruction generation can be improved. The method comprises the steps of performing feature processing on a natural language instruction, determining an instruction semantic feature, performing feature processing on an image frame sequence, and determining a current visual context feature; predicting the position of the target object based on the instruction semantic features and the current visual context features through a large language model, and outputting current position information corresponding to the target object; the current position information represents the direction and the proximity degree of the target object relative to the origin of coordinates in a sector where the perceptual view coordinate system is located; reasoning based on the current position information, the instruction semantic feature and the current visual context feature through a large language model to generate an action instruction; the action instruction is used for driving an execution mechanism to execute a target task indicated by the natural language instruction.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device and storage medium for generating action instructions. Background Technology

[0002] Currently, embodied visual tracking technologies typically employ end-to-end reasoning from perception to action mapping, or a hierarchical strategy to first infer high-level symbolic instructions, which are then used by the action planning model to generate specific action execution instructions by understanding these symbolic instructions. This black-box reasoning process or ambiguous symbolic instructions increases the uncertainty of model reasoning, thereby reducing the accuracy of model reasoning and the accuracy of the generated action instructions. Summary of the Invention

[0003] This application aims to provide a method, apparatus, device, and storage medium for generating action instructions, which can improve the accuracy of model reasoning and the accuracy of generating action instructions.

[0004] The technical solution of this application is implemented as follows: In a first aspect, this application provides a method for generating action instructions, the method comprising: Natural language instructions are processed to determine their semantic features, and image frame sequences are processed to determine the current visual context features. Using a large language model, the position of the target object is predicted based on the semantic features of the instruction and the current visual context features, and the current position information corresponding to the target object is output; the current position information represents the direction and proximity of the target object to the origin of the sector in the perceptual field coordinate system. Using the large language model, reasoning is performed based on the current location information, the semantic features of the instruction, and the current visual context features to generate action instructions; the action instructions are used to drive the actuator to perform the target task indicated by the natural language instructions.

[0005] Secondly, this application provides an action instruction generation device, the device comprising: The feature processing module is used to perform feature processing on natural language instructions to determine the semantic features of the instructions, and to perform feature processing on image frame sequences to determine the current visual context features. The position prediction module is used to predict the position of the target object based on the instruction semantic features and the current visual context features using a large language model, and output the current position information corresponding to the target object; the current position information represents the direction and proximity of the target object to the origin of the sector in the perceptual field coordinate system. The instruction generation module is used to generate action instructions by reasoning based on the current location information, the instruction semantic features, and the current visual context features through the large language model; the action instructions are used to drive the actuator to perform the target task indicated by the natural language instructions.

[0006] Thirdly, this application provides an electronic device, including a memory and a processor; wherein, The memory is used to store executable instructions; The processor is configured to implement the action instruction generation method provided in the embodiments of this application when executing executable instructions stored in the memory.

[0007] Fourthly, this application provides a computer-readable storage medium storing executable instructions for inducing a processor to execute and implement the action instruction generation method provided in the embodiments of this application.

[0008] Fifthly, embodiments of this application provide a computer program product, including a computer program or instructions, wherein when the computer program or instructions are executed by a processor, they implement the action instruction generation method provided in embodiments of this application.

[0009] This application provides an action command generation scheme, which performs feature processing on natural language commands and image frame sequences to determine the semantic features of the commands and the current visual context features; through a large language model, the position of the target object is predicted based on the semantic features of the commands and the current visual context features, and the current position information corresponding to the target object is output; the current position information represents the direction and proximity of the target object in the sector of the perceptual field coordinate system relative to the origin; through the large language model, inference is performed based on the current position information to generate action commands; the action commands are used to drive the actuator to perform the target task indicated by the natural language commands.

[0010] Through the above scheme, the current position information of the target object is explicitly output during the reasoning process of the large language model. This current position information is then used for subsequent action command reasoning, thereby decomposing the reasoning process into interpretable and observable intermediate steps, achieving complete and transparent thought chain reasoning. By using the direction and proximity of the target object's sector relative to the origin in the perceptual visual coordinate system to represent the current position information, consistent spatial position reasoning can be achieved across multiple viewpoints, improving the accuracy of reasoning. Furthermore, by explicitly predicting the target object's position information, an interpretable intermediate representation is established between low-level visual perception and high-level natural language commands, achieving tight coupling and collaborative reasoning of multimodal information, thereby improving the accuracy of model reasoning and the accuracy of generated action commands. Attached Figure Description

[0011] Figure 1 A schematic flowchart of an optional action instruction generation method provided in an embodiment of this application; Figure 2 A schematic flowchart of an optional action instruction generation method provided in an embodiment of this application; Figure 3 A schematic flowchart of an optional action instruction generation method provided in an embodiment of this application; Figure 4 A schematic diagram illustrating the confidence level and the current location information of the target object obtained by large language model inference in an embodiment of this application; Figure 5 This is a schematic diagram illustrating the action instruction generation process of an embodied intelligent model applied in a real-world scenario, based on an embodiment of this application. Figure 6 This is a schematic diagram of the structure of an action instruction generation device provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0013] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0014] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0015] Embodied Visual Tracking (EVT) is a fundamental core capability for intelligent robots to achieve real-world applications. It enables robots to continuously and stably lock onto moving targets while moving themselves, through "active observation." This capability is crucial in key scenarios such as companion robots, guidance robots, and public service assistants. It is not only a prerequisite for safe companionship, collaborative movement, and natural interaction, but also a fundamental guarantee for coping with dynamic and complex environments.

[0016] Currently, related technologies typically generate motion instructions for intelligent robots in two ways: one is to build an end-to-end vision-language-motion model, which directly generates following trajectories and their motion instructions by tightly coupling perception and planning, thereby reducing error accumulation in modular design; the other is to adopt a hierarchical strategy, using a large model for high-level task planning and instruction decomposition, with a dedicated module responsible for the underlying motion execution.

[0017] However, current related technologies have the following drawbacks: 1. Lack of explicit spatial reasoning ability: While end-to-end learning of the mapping from perception to action can effectively learn specific patterns, the model often lacks a clear understanding and reasoning process regarding the spatial relationship between the target and the agent. This lack of reasoning ability leads to a lack of transparency in the agent's decision-making process, making it difficult to analyze the source of errors (e.g., whether it is a perception error or a decision-making error). Furthermore, when encountering complex spatial configurations not covered by training data (such as a target suddenly making a right-angle turn or needing to choose the optimal following route from multiple paths), the rationality and generalization ability of its behavior are limited, thereby reducing the accuracy of the model's reasoning in generating action commands.

[0018] In a hierarchical strategy, the model may only generate high-level task sequences (e.g., in a following scenario, the instruction is broken down into "first find the target person, then maintain distance and follow"), while low-level actions (such as "how to identify and locate the person" and "how to plan a path to maintain a specific distance") are executed by a dedicated perception module and path planner. Although this paradigm also involves step decomposition, its high-level planning often lacks real-time perception feedback on the dynamic environment (e.g., it cannot handle situations where the target person suddenly turns or is obscured by crowds), and intermediate steps (such as the definition of "target person" and the precise measurement of "maintaining distance") rely on vague symbolic descriptions. These descriptions lack precise, quantitative definitions and require secondary interpretation and transformation by downstream modules, thus introducing uncertainty and error into the decision-making process.

[0019] 2. Lack of robust long-term target recognition mechanisms: The aforementioned technologies typically rely excessively on short-term visual matching, making it difficult to accurately re-identify targets after prolonged occlusion or disappearance from the field of vision. When the target reappears, the model easily confuses it with other visually similar distractors in the scene, leading to target switching or complete loss. This makes it difficult for the agent to achieve truly continuous following, significantly reducing its reliability in reasoning within real-world environments filled with dynamic obstacles (such as crowds). The agent is prone to losing target position, thus failing to generate accurate instructions to guide robot actions, greatly diminishing reliability.

[0020] This application provides a method, apparatus, device, and storage medium for generating action instructions. It can explicitly output the spatial location of a target object during model reasoning, achieving complete and transparent thought chain reasoning. Furthermore, it maintains the identity characteristics of the target object through a long-range time memory mechanism, enabling long-range following in complex environments, thus improving the reliability of model reasoning and the accuracy of generated action instructions. The action instruction generation method provided in this application is applied to electronic devices. The electronic devices in this application may include terminals and servers. In some embodiments, the terminal may include, but is not limited to, intelligent agent terminals with various motion modes and physical forms, such as robots, drones, vehicles, and mobile terminals. In some embodiments, the server may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, and big data and artificial intelligence platforms. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited in this application.

[0021] For example, when the electronic device is implemented as a terminal, the terminal can acquire natural language commands through its own configured interactive devices, such as a microphone or interactive screen, and acquire image frame sequences through its own configured image acquisition devices, such as a camera with at least one viewpoint. It then performs feature processing on the natural language commands and image frame sequences to determine the semantic features of the commands and the current visual context features. Using a large language model, it predicts the position of the target object based on the semantic features of the commands and the current visual context features, and outputs the current position information corresponding to the target object. Finally, using the large language model, it infers based on the current position information to generate action decision commands. When the electronic device is implemented as a server, the terminal can send the acquired natural language commands and image frame sequences to the server. The large language model configured on the server predicts the position of the target object based on the semantic features of the commands and the current visual context features, and outputs the current position information corresponding to the target object. Using the large language model, it infers based on the current position information to generate action decision commands. The specific choice depends on the actual situation, and this application does not impose specific limitations. The following explanation uses the case where the electronic device is implemented as an intelligent agent terminal as an example.

[0022] The action instruction generation method provided in this application embodiment can be as follows: Figure 1 As shown, including S101-S104, as follows: S101. Perform feature processing on natural language instructions to determine the semantic features of the instructions.

[0023] S102. Perform feature processing on the image frame sequence to determine the current visual context features.

[0024] This application applies to intelligent agents on various hardware platforms, such as mobile terminals like smartphones and tablets, robots with various motion modes and physical forms, drones, and intelligent transportation tools like vehicles, where natural language commands are processed through an artificial intelligence-based network model to achieve tasks such as navigation, target following, item finding, and home services.

[0025] In some embodiments, when applying this application to navigation or following tasks of robots (such as humanoid quadruped robots, wheeled / tracked robots, robot dogs, etc.), images or videos can be captured using image acquisition devices such as cameras deployed on the robot, thereby obtaining an image frame sequence. Human-computer interaction devices deployed on the robot, such as microphones or touchscreens, can receive natural language commands issued by the user in natural language. Based on the acquired natural language commands and the image frame sequence, intelligent navigation or following can be achieved using the methods of this application embodiment.

[0026] In this embodiment, natural language commands can be obtained through user voice or text input. These commands instruct the intelligent agent terminal on tasks to be performed, such as target following, navigation, item retrieval, and companionship services. The specific command selection depends on the actual situation and is not limited in this embodiment.

[0027] In this embodiment, natural language instructions can be feature-extracted to obtain semantic features corresponding to the natural language instructions, which serve as the instruction semantic features. For example, the natural language instructions can be segmented into words, mapped to embedding vectors, and the embedding vectors can be processed to obtain the semantic features corresponding to the natural language instructions. This embodiment does not impose specific limitations on the method of extracting instruction semantic features.

[0028] In this embodiment, the intelligent agent terminal may be configured with at least one image acquisition device, which continuously acquires image frames from at least one viewpoint. The image frame sequence includes image frames acquired at least at least one moment. Each moment corresponds to an image frame acquired from at least one viewpoint. For example, at least one image sensor is deployed at different locations on the robot, and each image sensor acquires video streams or images from its own viewpoint, thereby obtaining an image frame sequence acquired from at least one viewpoint at at least one moment. For example, at least one moment may include the current moment, or it may include the current moment and at least one historical moment. The image frame sequence may include image frames corresponding to at least one viewpoint at each moment obtained by sampling the video stream acquired by the image sensors at at least one viewpoint. In this embodiment, a moment may correspond to a time step in the model inference process, and the current moment may include the current time step of the model inference.

[0029] In this embodiment, feature extraction and processing are performed on image frames in an image frame sequence to determine the current visual context features. These current visual context features include visual context features extracted and generated from the current image frame and historical image frames in the image frame sequence. The visual context features are essentially a temporal enhancement representation, used to enhance the model's dynamic understanding of the current environment.

[0030] S103. Using a large language model, predict the position of the target object based on the semantic features of the instruction and the current visual context features, and output the current position information corresponding to the target object.

[0031] In this embodiment, the large language model can be trained on the hardware platform of the current intelligent agent terminal, or it can be trained on other types of hardware platform electronic devices and applied to the current intelligent agent terminal. Using the large language model, the semantic features of the instruction and the current visual context features are taken as input information to perform model inference and prediction on the position of the target object in the current environment, outputting the current position of the target object as the current position information.

[0032] In this embodiment, the current location information represents the direction and proximity of the target object within the sector of the perceptual field of view coordinate system relative to the origin. Unlike related technologies that generate lengthy text or bounding boxes of target objects, this embodiment establishes a perceptual field of view coordinate system for the perceptual field of view of intelligent agents such as robots, with the origin of the perceptual field of view coordinate system representing the intelligent agent terminal itself. This allows image frames from multiple perspectives to be unified into a single global perceptual field of view coordinate system for consistent spatial reasoning, avoiding problems such as ambiguity, overlap, and conflict caused by the target object appearing in multiple image frames from different perspectives.

[0033] This application embodiment divides the perceptual field of view into multiple sectors in a gridded manner, with each sector corresponding to unique position information in the perceptual field of view coordinate system. Since the origin of the intelligent agent terminal, i.e., the coordinate system's origin, is usually in a moving state, this application embodiment uses relative coordinates, i.e., the positional relationship of each sector grid relative to the origin, to represent the sector's position information. In some embodiments, the direction and proximity of the sector relative to the origin are used as the sector's position information. Thus, when performing position reasoning for a target object, the large language model can understand the target object to be detected, such as a target person or object, based on instruction semantic features. Combining this with the visual perception information in the perceptual field of view represented by the current visual context features, it infers the target object's position in the perceptual field of view coordinate system, determines the sector where the target object is located in the perceptual field of view coordinate system, and outputs the position information corresponding to that sector as the current position information.

[0034] S104. Using a large language model, reason based on current location information, instruction semantic features, and current visual context features to generate action instructions.

[0035] In this embodiment, the large language model explicitly outputs the current location information as an intermediate step in the reasoning process. That is, the current location information is explicitly output as an interpretable intermediate representation, rather than being hidden within the model. Then, the large language model is used again, taking the current location information as input and combining it with the instruction's semantic features and the current visual context features, to infer action instructions for the task indicated by the natural language instruction. Here, the action instructions are used to drive the actuator to perform the target task indicated by the natural language instruction. In some embodiments, the action instructions may include structured motion control parameters. The action instructions can be sent to an action decoding module, which decodes the action instructions into a series of executable path points.

[0036] In this embodiment, the actuator can also be referred to as a drive system or motion execution system, including a device or combination of devices that receives electrical control signals and converts them into physical motion. The actuator is used to execute action commands to achieve pose changes such as movement and turning of the robot, thereby performing tasks in the physical environment. For example, for a wheeled robot, the actuator may include a drive motor, a steering servo mechanism, and wheels connected to them. For a legged robot (such as a humanoid robot or a quadruped robot), the actuator may include servos or motors at multiple joints, coordinating and controlling the angles of these joints to achieve movements such as walking and running. For a tracked robot or other special-purpose robot, the actuator may include drive units that drive tracks, robotic arm joints, gimbals, etc., to achieve specific functional movements. The specific selection depends on the actual situation, and this embodiment does not limit the choice.

[0037] In some embodiments, motion commands are transmitted to a motion decoding module. This module, acting as a low-level controller, plans the motion trajectory based on parameters in the motion commands (such as target heading angle and velocity) and generates a series of path points along the trajectory. Then, through control signal conversion, the pose information of the path points is converted into low-level control signals for each drive unit (such as a motor or servo motor) in the actuator, such as motor pulse signals, speed, or position. These low-level control signals are sent to the corresponding actuators. The actuators respond to these signals, generating physical motion (e.g., wheels rotating at a specified speed, joints rotating at a specified angle), thereby driving the robot body to change pose in the environment and achieve the target task instructed by the natural language commands.

[0038] It should be noted that the action commands inferred from the current position information of the target object in this embodiment provide explicit motion constraints for the action decoding module compared to the abstract and vague symbolic descriptions in the hierarchical strategies of related technologies. For example, the symbolic descriptions in the hierarchical strategies of related technologies may be abstract task sequences such as "follow first, then stop," requiring further understanding and planning by the action decoding module. In contrast, the action commands in this embodiment provide explicit, direct, and quantifiable motion control parameters, such as "turn left," "adjust the body's orientation by a certain degree," and "move forward." The action decoding module only performs a deterministic transformation from the parameter space to the trajectory space, rather than semantic-based decision planning, thereby greatly reducing the uncertainty and error of the decision.

[0039] In this embodiment, the image frame sequence may include a real-time updated image stream or video stream. During the execution of natural language commands by the intelligent agent terminal, constantly changing environmental information is recorded through the real-time updated image stream or video stream. For each subsequent moment, the method described in this embodiment can be used to continuously update the target object's current position information in an explicit position prediction manner and generate new action commands based on the updated current position information. For example, in target following, the above method is implemented using a large language model. At each step, the target object's current position information is explicitly output, and action commands are generated based on the continuously updated current position information to adjust the intelligent agent terminal's actions. This is equivalent to mimicking the process of humans constantly reassessing the target object's position and adjusting their actions while moving.

[0040] Understandably, by employing the aforementioned method, the current position information of the target object is explicitly output during the reasoning process of the large language model. This current position information is then used for subsequent action command reasoning, thereby decomposing the reasoning process into interpretable and observable intermediate steps, achieving complete and transparent thought chain reasoning. Utilizing the direction and proximity of the target object's sector relative to the origin in the perceptual visual coordinate system to represent its current position information enables consistent spatial position reasoning across multiple viewpoints, improving reasoning accuracy. Furthermore, by explicitly predicting the target object's position information, an interpretable intermediate representation is established between low-level visual perception and high-level natural language commands, achieving tight coupling and collaborative reasoning of multimodal information, thus improving the accuracy of model reasoning and the accuracy of generated action commands.

[0041] In some embodiments, the aforementioned perceptual field coordinate system comprises multiple sectors, and the position information of each sector includes a unique position code corresponding to that sector; the position code represents the angle and distance of that sector relative to the origin; the aforementioned process of predicting the position of the target object based on instruction semantic features and current visual context features using a large language model, and outputting the current position information corresponding to the target object, includes: Using a large language model, predictions are made based on instruction semantic features and current visual context features to determine the target sector in multiple sectors; the target position code corresponding to the target sector is then output as the current position information.

[0042] In some embodiments, the field of view may include a perceptible circular viewpoint of the agent terminal, that is, a global viewpoint formed by the viewpoint of at least one image acquisition device on the agent terminal. A field of view coordinate system is established for the field of view, and the field of view is discretized into a structured grid of sectors, wherein each sector corresponds to an angle ( ) in the field of view coordinate system. ) and distance ( Polar coordinates are defined as follows. Here, the angle ( ) and distance ( The angle and distance refer to the angle and distance of a sector relative to the origin of the sensing field coordinate system (the intelligent agent terminal itself). The angle represents the direction of the sector relative to the origin, and the distance represents the degree of proximity of the sector to the origin. In this way, a sector in the sensing field can be uniquely identified through the quantified combination of angle and distance.

[0043] In this embodiment, the polar coordinates of the angle and distance corresponding to each sector can be encoded into a single location code, thereby forming a compact and unified spatial representation. For example, the location code may include a token or a vocabulary marker. Thus, a unified location code table (such as a vocabulary) can be determined based on the location codes corresponding to each sector in the perceptual field of view. The large language model then predicts the location of the target object based on the instruction semantic features and the current visual context features, infers the target sector where the target object is located among multiple sectors, and outputs the target location code (such as the token of the target sector) corresponding to the target sector in the location code table. This target location code directly corresponds to the polar coordinates of the sector where the target object is located. The target location code is then used as the current location information.

[0044] Understandably, by sectorizing and encoding the perceived field of view, a unified spatial representation is formed for images from multiple perspectives. This unified polar coordinate representation based on the agent ontology fundamentally avoids the problem of overlapping and conflicting bounding boxes of multiple target objects in the case of multiple camera settings, eliminates ambiguity and realizes consistent spatial reasoning across different views, thereby improving the accuracy of model reasoning.

[0045] In some embodiments, the current location information includes preset identification information; the preset identification information indicates that the location of the target object is uncertain.

[0046] In this embodiment of the application, to address the uncertainty in target object location prediction, preset identification information is defined to characterize situations where the target object's location is uncertain. For example, this information is specifically introduced into the vocabulary corresponding to the perceptual field of view. <invalid>The token serves as a pre-defined identifier, allowing the model to explicitly indicate whether a target object is occluded or outside the perceptual field of view. For example, the perceptual field of view of the intelligent agent terminal can be as follows: Figure 2 As shown, the perceptual field of view comprises multiple gridded sectors, each corresponding to a polar coordinate system. This polar coordinate system represents the sector's position relative to the agent's terminal, i.e., its angle and distance relative to the origin of the perceptual field of view coordinate system. The sector containing the target object within the perceptual field of view is called the target sector. Figure 2 The polar coordinates of the target sector in the perception field coordinate system can be represented as < , >. When the target object is occluded (e.g. Figure 2 (The target object shown is obscured by a wall), the large language model in the intelligent agent terminal can output... <invalid>The token is used to represent the location of the target object, which is uncertain.

[0047] Understandably, by defining predefined identifier information, large language models can explicitly output corresponding signals when the location of the target object cannot be determined, thereby enhancing the robustness of model inference in some challenging scenarios (such as scenarios with many obstructions or dense crowds).

[0048] In some embodiments, the current visual context features include: the visual feature sequence corresponding to the current moment and the recognition features of the target object corresponding to the current moment; the process of performing feature processing on the image frame sequence to determine the current visual context features can be as follows: Figure 3 As shown, it includes: S301. Extract features from the image frame sequence to determine the visual feature sequence.

[0049] In this embodiment of the application, feature extraction is performed on each image frame in the image frame sequence to determine the visual features corresponding to each image frame, thereby obtaining a visual feature sequence.

[0050] In some embodiments, the visual features corresponding to an image frame may include visual features at multiple resolutions. For example, a dual visual encoder can be used to process video streams from multiple cameras, extracting and fusing visual features to simultaneously capture fine-grained details and high-level semantic information; a grid pooling strategy is employed to generate feature representations at two resolutions based on the output of the dual visual encoder. For example, the first resolution visual features include high-resolution visual features to characterize the details of the current observation; the second resolution visual features include low-resolution visual features to summarize coarse-grained historical context.

[0051] S302. Based on a sliding window with a preset time span, determine the visual feature sequence corresponding to the current moment in the visual feature sequence according to the current visual feature corresponding to the current moment and at least one historical visual feature corresponding to at least one historical moment adjacent to the current moment.

[0052] In this embodiment, a sliding window mechanism can be used. Based on a sliding window with a preset time span, the current visual feature corresponding to the image frame at the current moment and at least one historical visual feature corresponding to the image frame at at least one historical moment adjacent to the current moment are selected from the visual feature sequence as the visual feature sequence corresponding to the current moment. For example, the visual features of the most recent 32 frames (with a preset time span of 32 frames) can be retained as the visual feature sequence corresponding to the current moment.

[0053] In this embodiment, the visual feature sequence corresponding to the current moment represents a short-term memory of environmental information, used to provide immediate context for reasoning in large language models. In some embodiments, the visual feature sequence corresponding to the current moment can be obtained based on the second resolution visual features corresponding to historical moments and the first resolution visual features corresponding to the current moment. In this way, the two resolution feature representations contained in the visual feature sequence corresponding to the current moment can be utilized to balance computational cost and historical context information.

[0054] S303. Obtain the recognition features of the target object at the previous moment.

[0055] To achieve a robust long-term target recognition mechanism, embodiments of this application maintain a recognition feature corresponding to the target object, which is used to maintain a stable and condensed target identity feature across time periods.

[0056] In S303, the recognition features of the target object at the previous moment before the current moment are obtained. It can be understood that the recognition features of the target object at the previous moment are obtained based on the visual features before the previous moment. The recognition features of the target object at the current moment obtained based on the recognition features of the target object at the previous moment will be used to maintain the recognition features of the target object at the next moment before the current moment.

[0057] S304. Based on the previous visual features corresponding to the previous time step and the previous position information predicted and output by the large language model at the previous time step, determine the candidate features corresponding to the target object.

[0058] In this embodiment, at the time preceding the current time, the large language model can predict and output the position information of the target object at the previous time using the methods described in S101-S104 above, and use this as the previous position information. Based on the previous position information of the target object, the visual features belonging to the target object can be located and identified in the previous visual features at the previous time, and used as candidate features corresponding to the target object.

[0059] For example, based on the previous position information corresponding to the target object, the object visual features belonging to the target object can be located and identified in the high-resolution visual features of the previous visual features.

[0060] It is understandable that the candidate features corresponding to the target object represent the identity features of the most recent target object.

[0061] S305. Based on the candidate features, update the recognition features corresponding to the previous time step to determine the recognition features corresponding to the current time step.

[0062] In this embodiment, the identification features corresponding to the previous moment represent the identity features of the target object in historical moments prior to the previous moment, i.e., the model's historical memory state of the target object. Based on the candidate features corresponding to the target object, the identification features corresponding to the previous moment are updated. For example, the candidate features and the identification features corresponding to the previous moment are weighted and fused, which can integrate the latest identity features of the target object into the historical memory identity features, update the visual state of the target object, and obtain the identification features corresponding to the target object at the current moment. In this way, the identity features of the target object can be maintained for a long time through the identification features corresponding to the target object at the current moment, and provided as enhanced information to the large language model for inference of the target object's location, thereby improving the accuracy of location prediction.

[0063] It should be noted that the process S302 described above, along with processes S303-S305 described above, can be executed in any order or in parallel. Figure 3 This is merely an example of an execution order; the specific order should be selected based on the actual situation, and this application does not limit the implementation.

[0064] In some embodiments, the visual feature sequence representing the current moment of short-term memory and the recognition features representing the current moment of long-term memory can be jointly projected onto the latent space of the large language model and concatenated with the instruction semantic features input to the large language model to obtain the input sequence for the large language model's inference. In this way, the large language model predicts the position of the target object based on the instruction semantic features, the visual feature sequence corresponding to the current moment, and the recognition features of the target object at the current moment, determines the target sector in the perceptual field coordinate system, and outputs the target position code corresponding to the target sector as the current position information.

[0065] It is understood that the embodiments of this application realize long-term and short-term dual memory management of target object and environmental information by using the visual feature sequence corresponding to the current time and the recognition features corresponding to the target object at the current time. By maintaining the recognition features corresponding to the target object, a robust representation of long-term memory that maintains the identity features of the target object changing over time is realized, which improves the reliability and accuracy of the large language model in detecting the target object. This allows the large language model to follow the target without losing it when the target object is occluded, thus achieving robust long-term following.

[0066] In some embodiments, the process of updating the identification features corresponding to the previous time step based on candidate features to determine the identification features corresponding to the current time step may include: Obtain a first confidence score and a second confidence score. The first confidence score is obtained through a large language model, inferred from the semantic features of the instruction and the visual context features corresponding to the previous time step. The second confidence score is obtained through a large language model, inferred from the semantic features of the instruction and the visual context features corresponding to at least one historical time step prior to the previous time step. Based on the first and second confidence scores, determine the first weight corresponding to the candidate feature and the second weight corresponding to the recognition feature corresponding to the previous time step. According to the first and second weights, perform weighted fusion of the candidate feature and the recognition feature corresponding to the previous time step to determine the recognition feature corresponding to the current time step.

[0067] In this embodiment of the application, the update frequency of the target object's recognition features can be adjusted using the confidence level obtained through large language model inference. For example, as shown... Figure 4 As shown, when the large language model predicts the location of a target object based on the semantic features of the instruction and the current visual context features at the current moment, it can output the target object's current location information along with its confidence score at that moment. The confidence score represents the credibility of the location information of the target object obtained through inference. The confidence score at the current moment can be calculated based on the probability distribution of the confidence scores at various historical moments.

[0068] Thus, in the previous time step of the current time step, the large language model can infer and output the confidence score of the target object in the previous time step based on the semantic features of the instruction and the visual context features corresponding to the previous time step, which serves as the first confidence score. For example, the first confidence score can be calculated using formula (1), as follows: Formula (1) in, P This represents the raw score output by the large language model when predicting the location of the target object. P It can be a length of A vector. Each element in the vector The unnormalized score of the i-th token in the corresponding positional encoding table (such as a vocabulary). This indicates that the softmax function is applied to the original vector. P Convert to a probability distribution. Representation based on the original vector P The entropy value is obtained by calculating the probability distribution; This means dividing the entropy value by the maximum possible entropy. The obtained normalized entropy, The value can be between 0 and 1. When it equals 0, it indicates that the probability distribution is perfectly concentrated, the certainty is highest, and the normalized entropy is 0. When When the value equals 1, it indicates that the probability distribution is completely uniform, the uncertainty is the highest, and the normalized entropy is 1. This represents the first confidence level, which is the confidence level of the large language model's output at the previous time T-1 before the current time T. The value ranges from 0 to 1. A value of 1 indicates that the model is completely determined (entropy is 0). An equal value of 0 indicates that the model is completely uncertain (entropy is 0). In other words, the first confidence level. The higher the value, the more reliable the location information of the target object output by the large language model in the previous time step.

[0069] Similarly, at each historical moment preceding the previous moment, the large language model can also infer and output the confidence score of the target object at that historical moment based on the instruction semantic features and the temporal visual context features corresponding to that historical moment, thereby obtaining at least one historical first confidence score corresponding to at least one historical moment preceding the previous moment. Then, based on the at least one historical first confidence score corresponding to at least one historical moment preceding the previous moment, a second confidence score can be obtained.

[0070] For example, the second confidence level can be obtained by averaging at least one historical first confidence level corresponding to at least one historical time point prior to the previous time point. As shown in formula (2), it is as follows: Formula (2) In formula (2), The first historical confidence level corresponds to historical time i. The calculation method of the first historical confidence level of historical time i can be referred to the description in formula (1), which will not be repeated here. The second confidence level represents the average of the first confidence level of at least one historical time point (from historical time point 1 to historical time point T-2) prior to the previous time point T-1. It can be understood that the second confidence level represents the historical average confidence level.

[0071] In some embodiments, the first weight and the second weight can be determined by the ratio of the first confidence level to the second confidence level. For example, the first weight can be determined according to formulas (3) and (4), as follows: Formula (3) in, The representative is based on the first confidence level. With second confidence level The calculated first weight is used in the model inference process at the current time T. Correspondingly, the second weight can be... .

[0072] In some embodiments, the process of weighted fusion of candidate features and recognition features corresponding to the previous time step according to the first weight and the second weight to determine the recognition features corresponding to the current time step can be as shown in formula (4), as follows: Formula (4) in, This represents the identification features of the target object at the previous time step. This represents the identification feature of the target object at the current moment. The initial value of the identification feature can be empty. These represent candidate features corresponding to the target object. It can be identified and determined from the high-resolution visual features of the previous time step based on the previous position information of the target object.

[0073] As can be seen, the first confidence level can be used to control whether to update the target object's identification features. For example, when the first confidence level is higher than a preset confidence threshold, such as... When the confidence level approaches 1, the target object's identification features are allowed to be updated based on candidate features. However, when the first confidence level is less than or equal to a preset confidence threshold, such as... When the value approaches 0, the recognition features corresponding to the previous time step are retained. .

[0074] Understandably, the weights of the most recent candidate features and the corresponding recognition features from the previous time step are calculated using confidence scores. When the first weight is large, the most recent candidate features of the target object can be fully incorporated into the model's long-term memory of the target object, updating the target's appearance. When the first weight approaches zero, representing low confidence or an invalid target, the impact of the most recent candidate features on long-term memory can be reduced, maintaining the model's effective memory state of the target object, effectively "freezing" the target's identity, and preventing erroneous updates introduced by occlusion or interference. This ensures that the model's memory state of the target object is only updated when the target object is reliably identified. In this way, even when faced with occlusion and visually similar interference, the model can always maintain the tracking of the correct target, greatly improving the accuracy of model inference.

[0075] In some embodiments, when the current location information includes preset identification information, the confidence level corresponding to the current time is set to a preset minimum value, and the confidence level corresponding to the current time is used as the first confidence level for the next time.

[0076] It is understandable that, given that the current location information includes preset identifier information, such as the output of a large language model... <invalid>The token indicates that the large language model is uncertain about the position of the target object. At this time, the confidence corresponding to the current time can be set as The confidence corresponding to the current time is forced to be the minimum value, for example, 0, so that when the confidence corresponding to the current time is used as the first confidence of the next time T+1, the effective recognition feature can be frozen, the reliable representation of the identity feature of the target object is preserved, and the accuracy of model inference is improved.

[0077] In some embodiments, before position prediction and action instruction inference are performed using the above large language model, the large language model can also be trained by the following method, as follows: The initial large language model is iteratively trained using the training sample and the preset loss function to obtain the large language model; wherein the preset loss function at least includes an inference loss; the inference loss represents the difference between the position information predicted by the large language model and the true position information in the training sample.

[0078] For example, the preset loss function can include a weighted loss function that combines trajectory planning loss, inference loss, and text prediction loss. Through the weighted loss function, the following three objectives can be optimized simultaneously during the training process of the initial large language model: through the trajectory planning loss, the error between the predicted path point and the true trajectory path point in the training sample is reduced; through the inference loss, the error between the predicted position information of the target object and the true position information of the target object in the training sample is reduced; through the text prediction loss, the basic language understanding ability of the model is improved.

[0079] It can be understood that through the setting of the loss function in the above training process, the model can be iteratively optimized according to the training result during the training process, and the accuracy of the large language model in predicting the position information of the target object can be improved.

[0080] In some embodiments, taking the target task as an example of a target following task, the target object can include a target object in a moving state; for example, the target object can include a user in a moving state, and the target following task can include a task of a robot following the user's movement. The action instruction can include an action sequence for controlling the movement of the robot. The process in S104 can be implemented by the following process, including: The large language model performs trajectory inference based on the current position information, instruction semantic features, and current visual context features, and generates at least one target trajectory point; the at least one target trajectory point is used to indicate a path of the target object in the moving state; each target trajectory point in the at least one target trajectory point comprises a moving direction and a moving position; and an action sequence corresponding to each target trajectory point is generated according to the moving direction and the moving position of each target trajectory point, so as to drive the execution mechanism to control the robot to move according to the at least one target trajectory point and follow the target object in the moving state.

[0081] The large language model performs trajectory inference based on the current position information, instruction semantic features, and current visual context features, and generates at least one target trajectory point; the at least one target trajectory point is used to indicate a path of the target object in the moving state; each target trajectory point in the at least one target trajectory point comprises a moving direction and a moving position; and an action sequence corresponding to each target trajectory point is generated according to the moving direction and the moving position of each target trajectory point, so as to drive the execution mechanism to control the robot to move according to the at least one target trajectory point and follow the target object in the moving state.

[0082] In some embodiments, the moving direction and the moving position of each target trajectory point in the at least one target trajectory point are converted into actions at a bottom layer, an action corresponding to each target trajectory point is generated, and thus an action sequence corresponding to the at least one target trajectory point is obtained. In this way, each action in the action sequence drives the execution mechanism of the robot, and the robot is controlled to move according to the at least one target trajectory point in sequence, thereby achieving following of the target object in the moving state.

[0083] It should be noted that after the action sequence corresponding to the at least one target trajectory point is generated according to the moving direction and the moving position of each target trajectory point, at the next moment of model processing, the image frame sequence can be updated based on the real-time image stream, the current visual context features are updated by performing feature processing on the updated image frame sequence, the position of the target object is predicted by the large language model based on the instruction semantic features and the updated current visual context features, the current position information corresponding to the target object is updated, the at least one target trajectory point and the action sequence are updated by the large language model based on the updated current position information, the instruction semantic features, and the updated current visual context features by performing trajectory inference and action generation in the above manner, and the execution mechanism of the robot is driven to continue moving by the updated action sequence, thereby achieving real-time and continuous following of the target object in the moving state, such as a target user.

[0084] In some embodiments, the embodiments of the present application provide a motion instruction generation process of an embodied intelligent model applied to an actual scene, as shown in Figure 5 The embodied intelligent model can include a visual encoder, a text encoder, a grid pooling layer, a visual-language-action model, and a motion decoding module. The text encoder is used to process the natural language instructions of the user to obtain text tokens (equivalent to instruction text features). The visual encoder is used to process the multi-view video stream (equivalent to the image frame sequence) to obtain the visual tokens corresponding to the current moment (equivalent to the visual feature sequence). The target object identity token is equivalent to the recognition feature of the target object at the current moment. The visual-language-action model is equivalent to the large language model described above. Through the visual-language-action model, the position of the target object is predicted according to the text token, the visual token corresponding to the current moment, and the target object identity token, and an inference token is output. The inference token includes the confidence of the current moment corresponding to the current position information of the target object, and is used to update the target object identity token corresponding to the next moment of the current moment. Through the visual-language-action model, the text token, the visual token corresponding to the current moment, the target object identity token, and the inference token are inferred again to generate a motion token (equivalent to a motion instruction), which is input into the motion decoding module to obtain the corresponding motion trajectory.

[0085] The above-mentioned embodied intelligent model architecture can be widely applied to various hardware platforms and task modes in scenes, and exemplary applications include: I. Outdoor navigation and following: An exemplary implementation includes deploying the model on a humanoid robot or a quadruped robot, combining natural language instructions (such as "follow me to the parking lot"), and using multi-view video perception and trajectory planning to achieve autonomous following. The model can process dynamic environments (crowds, obstacles) in real time, maintain stable following and obstacle avoidance capabilities in complex outdoor scenes, and exhibit strong generalization.

[0086] II. Smart home and intelligent companionship: An exemplary implementation includes deploying the model on a household service robot to achieve functions such as following and accompanying through natural language interaction, achieving intelligent assistance for the elderly and people with disabilities, and improving the convenience and safety of family life.

[0087] It can be understood that through the action instruction generation process of the above embodiment intelligent model, the spatial position of the target object can be explicitly inferred, transparent chain reasoning is realized, the reasoning speed is not greatly reduced, the identity characteristics of the target object are modeled, different identity information can be distinguished, the recognition and positioning ability of the model to the target object is effectively improved, further, through effective fusion of multi-view and time dynamics, the trajectory prediction accuracy is improved, and the accuracy of model reasoning and the accuracy of generated action instructions are improved.

[0088] The embodiment of the present application provides an action instruction generation device 1, as shown in the figure, comprising: Figure 6 a feature processing module 11, configured to perform feature processing on a natural language instruction, determine instruction semantic features, and perform feature processing on an image frame sequence to determine current visual context features; a position prediction module 12, configured to predict the position of a target object based on the instruction semantic features and the current visual context features through a large language model, and output current position information corresponding to the target object; the current position information represents the direction and proximity of the target object in a sector of a perception field coordinate system relative to a coordinate origin; an instruction generation module 13, configured to infer based on the current position information, the instruction semantic features and the current visual context features through the large language model, and generate an action instruction; the action instruction is used to drive an execution mechanism to execute a target task indicated by the natural language instruction.

[0089] In some embodiments, the perception field coordinate system includes a plurality of sectors, each sector corresponding to a unique position code; the position code represents the angle and distance of the sector relative to the coordinate origin; the position prediction module 12 is further configured to determine a target sector in which the target object is located in the plurality of sectors based on the instruction semantic features and the current visual context features through the large language model; and output the target position code corresponding to the target sector as the current position information.

[0090] ​In some embodiments, the current visual context feature includes a visual feature sequence corresponding to a current moment and an identification feature of the target object corresponding to the current moment; the feature processing module 11 is further configured to perform feature extraction on the image frame sequence to determine the visual feature sequence; according to a preset time span sliding window, in the visual feature sequence, according to a current visual feature corresponding to a current moment and at least one historical visual feature corresponding to at least one historical moment adjacent to the current moment, the visual feature sequence corresponding to the current moment is determined; an identification feature corresponding to a previous moment of the target object is obtained; according to a previous visual feature corresponding to the previous moment and previous position information predicted and output by the large language model at the previous moment, a candidate feature corresponding to the target object is determined; based on the candidate feature, the identification feature corresponding to the previous moment is updated to determine the identification feature corresponding to the current moment.

[0091] In some embodiments, the feature processing module 11 is further configured to determine a first confidence and a second confidence; the first confidence is obtained by the large language model based on the instruction semantic feature and the visual context feature corresponding to the previous moment; the second confidence is obtained by the large language model based on the instruction semantic feature and the visual context feature corresponding to at least one historical moment before the previous moment; the confidence represents the credibility of the position information of the target object. Based on the first confidence and the second confidence, a first weight corresponding to the candidate feature and a second weight corresponding to the identification feature corresponding to the previous moment are determined; according to the first weight and the second weight, the candidate feature and the identification feature corresponding to the previous moment are weighted and fused to determine the identification feature corresponding to the current moment.

[0092] In some embodiments, when the current position information includes preset identification information, the feature processing module 11 is further configured to set a confidence corresponding to the current moment to a preset minimum value, and the confidence corresponding to the current moment is used as a first confidence of a next moment; the preset identification information represents that the position of the target object is uncertain.

[0093] In some embodiments, the action instruction generation apparatus further includes a training module, the training module is configured to perform iterative training on an initial large language model by using a training sample and a preset loss function to obtain the large language model; wherein the preset loss function at least includes an inference loss; the inference loss represents a difference between the position information predicted by the large language model and the real position information in the training sample.

[0094] In some embodiments, the target task comprises a target following task; the target object comprises a target object in a moving state; the action instruction comprises a motion sequence for controlling the robot to move; the instruction generation module 13 is further configured to perform trajectory inference based on the current position information, the instruction semantic feature and the current visual context feature by the large language model to generate at least one target trajectory point; the at least one target trajectory point is used to indicate a path for moving to the target object in the moving state; each target trajectory point in the at least one target trajectory point comprises a moving direction and a moving position; and the action sequence corresponding to the at least one target trajectory point is generated according to the moving direction and the moving position of each target trajectory point, so as to drive the execution mechanism by the action sequence to control the robot to move according to the at least one target trajectory point to follow the target object in the moving state.

[0095] It should be noted that the description of the above device embodiments is similar to that of the above method embodiments, and has similar beneficial effects to the method embodiments. For technical details not disclosed in the device embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

[0096] The embodiments of the present application also provide an electronic device, Figure 7 An optional structural schematic diagram of the electronic device provided by the embodiments of the present application is shown in FIG. 3. As shown in FIG. 3, the electronic device 3 comprises a memory 32 and a processor 33. The memory 32 and the processor 33 are connected through a communication bus 34; the memory 32 is configured to store executable instructions; and the processor 33 is configured to execute the executable instructions stored in the memory 32 to implement the action instruction generation method provided by the embodiments of the present application. Figure 7

[0097] The embodiments of the present application provide a computer readable storage medium storing executable instructions, wherein the executable instructions are stored in the computer readable storage medium, and when the executable instructions are executed by the above processor, the above processor will execute the action instruction generation method provided by the embodiments of the present application.

[0098] In some embodiments, the computer readable storage medium can be a FRAM, a ROM, a PROM, an EPROM, an EEPROM, a flash memory, a magnetic surface memory, an optical disc, or a CD-ROM memory; or can be various devices comprising one or any combination of the above memories.

[0099] ​In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0100] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts within a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files storing one or more modules, subroutines, or code sections). As an example, executable instructions may be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0101] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0102] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0103] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0104] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0105] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.< / invalid> < / invalid> < / invalid>

Claims

1. A method for generating action instructions, characterized in that, include: Natural language instructions are processed to determine their semantic features, and image frame sequences are processed to determine the current visual context features. Using a large language model, the position of the target object is predicted based on the semantic features of the instruction and the current visual context features, and the current position information corresponding to the target object is output; the current position information represents the direction and proximity of the target object to the origin of the sector in the perceptual field coordinate system. Using the large language model, reasoning is performed based on the current location information, the semantic features of the instruction, and the current visual context features to generate action instructions; the action instructions are used to drive the actuator to perform the target task indicated by the natural language instructions.

2. The method according to claim 1, characterized in that, The perceptual field coordinate system comprises multiple sectors, each sector corresponding to a unique position code; the position code represents the angle and distance of the sector relative to the coordinate origin; the prediction of the target object's position based on the instruction semantic features and the current visual context features using a large language model, and the output of the target object's current position information, includes: Using the large language model, prediction is made based on the instruction semantic features and the current visual context features to determine the target sector in the multiple sectors where the target object is located. The target location code corresponding to the target sector is output as the current location information.

3. The method according to claim 1 or 2, characterized in that, The current visual context features include: the visual feature sequence corresponding to the current moment and the recognition features of the target object corresponding to the current moment; the feature processing of the image frame sequence to determine the current visual context features includes: Feature extraction is performed on the image frame sequence to determine the visual feature sequence; Based on a sliding window with a preset time span, the visual feature sequence corresponding to the current moment is determined in the visual feature sequence according to the current visual feature corresponding to the current moment and at least one historical visual feature corresponding to at least one historical moment adjacent to the current moment. Obtain the identification features of the target object at the previous time step; Based on the previous visual features corresponding to the previous time step, and the previous position information predicted and output by the large language model at the previous time step, the candidate features corresponding to the target object are determined. Based on the candidate features, the identification features corresponding to the previous time step are updated to determine the identification features corresponding to the current time step.

4. The method according to claim 3, characterized in that, The step of updating the identification features corresponding to the previous time step based on the candidate features to determine the identification features corresponding to the current time step includes: A first confidence level and a second confidence level are determined; the first confidence level is obtained by reasoning through the large language model based on the semantic features of the instruction and the visual context features corresponding to the previous moment; the second confidence level is obtained by reasoning through the large language model based on the semantic features of the instruction and the visual context features corresponding to at least one historical moment before the previous moment; the confidence level represents the credibility of the location information of the target object; Based on the first confidence level and the second confidence level, determine the first weight corresponding to the candidate feature and the second weight corresponding to the identification feature at the previous time step; Based on the first weight and the second weight, the candidate features and the recognition features corresponding to the previous time are weighted and fused to determine the recognition features corresponding to the current time.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: When the current location information includes preset identification information, the confidence level corresponding to the current time is set to a preset minimum value, and the confidence level corresponding to the current time is used as the first confidence level for the next time. The preset identification information indicates that the location of the target object is uncertain.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: The initial large language model is iteratively trained using training samples and a preset loss function to obtain the large language model. The preset loss function includes at least inference loss; the inference loss characterizes the difference between the location information predicted by the large language model and the actual location information in the training samples.

7. The method according to any one of claims 1 to 6, characterized in that, The target task includes: a target following task; the target object includes: a target object in a moving state; the action instructions include: an action sequence for controlling the robot's movement; the generation of action instructions through the large language model, based on the current position information, the semantic features of the instructions, and the current visual context features, includes: Using the large language model, trajectory reasoning is performed based on the current location information, the instruction semantic features, and the current visual context features to generate at least one target trajectory point; the at least one target trajectory point is used to indicate the path of movement to the target object in the moving state; each of the at least one target trajectory point includes a movement direction and a movement position; Based on the movement direction and position of each target trajectory point, an action sequence corresponding to the at least one target trajectory point is generated, so as to drive the actuator through the action sequence and control the robot to move along the target object in the movement state according to the at least one target trajectory point.

8. An action command generation device, characterized in that, The device includes: The feature processing module is used to perform feature processing on natural language instructions to determine the semantic features of the instructions, and to perform feature processing on image frame sequences to determine the current visual context features. The position prediction module is used to predict the position of the target object based on the instruction semantic features and the current visual context features using a large language model, and output the current position information corresponding to the target object; the current position information represents the direction and proximity of the target object to the origin of the sector in the perceptual field coordinate system. The instruction generation module is used to generate action instructions by reasoning based on the current location information, the instruction semantic features, and the current visual context features through the large language model; the action instructions are used to drive the actuator to perform the target task indicated by the natural language instructions.

9. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the method of any one of claims 1 to 7.

10. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, the method of any one of claims 1 to 7 is implemented.

Images