Action generation method and device based on continuous decision group, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, can be applied to business scenes of financial science and technology, medical treatment and health and the like, and discloses an action generation method and device based on a continuous decision group, equipment and a medium. And inputting the current action prediction into a multi-modal language model containing a plurality of continuous decision groups, sequentially activating the continuous decision groups to generate current action prediction, comparing the current action prediction with the previous action prediction, and judging whether the current action prediction and the previous action prediction meet an early leaving condition or not. And when an early exit condition is met, activating the subsequent continuous decision group and outputting the current action prediction as a final action instruction, and when the early exit condition is not met, continuously activating the next continuous decision group until all the continuous decision groups are activated, and outputting the action prediction generated by the last continuous decision group as a final action instruction. Dynamic reasoning is realized through the continuous decision group and the early leaving condition, redundant calculation is avoided, reasoning efficiency is improved, resource consumption and delay are reduced, and the method is suitable for a resource-constrained environment.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device and medium for generating actions based on continuous decision groups. Background Technology

[0002] In the current field of robot control, multimodal large language models (MLLMs) are widely used to understand complex instructions and generate corresponding actions. Representative models such as RoboFlamingo and RT-2, while having certain advantages in semantic understanding and cross-modal task execution, generally suffer from bottlenecks in efficiency and resource consumption. Existing technologies typically rely on activating all parameters of a deep large language model throughout the entire process, such as activating a complete model structure containing more than twenty layers, which consumes a lot of computing power even when processing simple tasks. This fixed computing mode leads to significant real-time latency and power consumption, severely restricting the application of robots in resource-constrained environments.

[0003] In the healthcare sector, rehabilitation assistive robots and remote diagnostic and treatment devices increasingly require real-time execution of action commands. For example, the complexity of tasks performed by robotic arms, such as assisting patient movement, delivering medication, or conducting rehabilitation training, varies significantly. However, existing methods consume the same computational resources regardless of the task's complexity, resulting in significant computational waste when handling most routine tasks. This inefficient computing model not only exacerbates device energy consumption but also makes it difficult to deploy these devices in embedded medical devices or mobile rehabilitation terminals, limiting their adoption in long-term care and high-frequency operation scenarios.

[0004] In the fintech sector, smart teller machines, business verification robots, and voice interaction terminals are increasingly incorporating multimodal understanding and control technologies to parse users' multimodal commands and execute financial transactions. In these applications, approximately 80% of tasks involve simple, fixed-format operations, such as file transfer, interface switching, or standard compliance checks. Existing models still incur high computational costs and consume significant amounts of GPU memory and processing resources in these low-complexity scenarios, making it difficult to deploy terminal devices at the edge and limiting the adoption by financial institutions in lightweight scenarios such as mobile terminals and self-service machines. Especially in scenarios requiring high-concurrency processing of user requests, the fixed computational overhead leads to response latency and excessive energy consumption, directly impacting service efficiency and user experience. Summary of the Invention

[0005] The main objective of this invention is to provide an action generation method, apparatus, device, and storage medium based on continuous decision groups. This invention aims to solve the technical problem that existing multimodal large language models use a fixed full-layer activation calculation mode when performing robot control tasks, resulting in a large amount of computing resources being wasted on redundant layers, causing high real-time latency, high energy consumption, and difficulty in adapting to differences in task complexity.

[0006] To achieve the above objectives, the present invention provides an action generation method based on continuous decision groups, comprising:

[0007] It receives multimodal input data, including visual input and language commands;

[0008] The multimodal input data is input into a multimodal language model containing multiple consecutive decision groups, each of which includes a partial model layer and an action prediction module;

[0009] The plurality of consecutive decision groups are activated sequentially. Each time a consecutive decision group is activated, the current action prediction is generated through the action prediction module of the consecutive decision group.

[0010] The current action prediction is compared with the previous action prediction, and the early departure condition is determined based on the comparison result.

[0011] If the early exit condition is met, the activation of subsequent continuous decision groups is terminated, and the current action prediction is output as the final action instruction.

[0012] If the early exit condition is not met, the next continuous decision group will be activated and the comparison and judgment steps will be repeated until all continuous decision groups are activated. The action prediction generated by the last continuous decision group will be output as the final action instruction.

[0013] Furthermore, to achieve the above objectives, the present invention provides an action generation device based on continuous decision groups, comprising:

[0014] The input parsing module is used to receive multimodal input data, including visual input and language commands;

[0015] A multimodal coding module is used to input the multimodal input data into a multimodal language model containing multiple consecutive decision groups, each of the consecutive decision groups containing a partial model layer and an action prediction module;

[0016] The decision group activation module is used to activate the plurality of consecutive decision groups in sequence. Each time a consecutive decision group is activated, the action prediction module of the consecutive decision group generates the current action prediction.

[0017] The prediction comparison module is used to compare the current action prediction with the previous action prediction, and determine whether the early termination condition is met based on the comparison result.

[0018] The early departure control module is used to terminate the activation of subsequent continuous decision groups if the early departure condition is met, and output the current action prediction as the final action instruction.

[0019] The result output module is used to continue activating the next continuous decision group and repeating the comparison and judgment steps if the early exit condition is not met, until all continuous decision groups are activated, and outputs the action prediction generated by the last continuous decision group as the final action instruction.

[0020] Furthermore, to achieve the above objectives, the present invention also provides a computer device, the computer device including a memory, a processor, and an action generation program based on a continuous decision group stored in the memory and executable on the processor, wherein when the action generation program based on a continuous decision group is executed by the processor, it implements the steps of the action generation method based on a continuous decision group as described above.

[0021] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing an action generation program based on a continuous decision group, wherein when the action generation program based on a continuous decision group is executed by a processor, the action generation program based on a continuous decision group implements the steps of the action generation method based on a continuous decision group as described above.

[0022] Beneficial Effects: This invention relates to the field of artificial intelligence technology and can be applied to business scenarios such as fintech and healthcare. It discloses a method, apparatus, device, and medium for action generation based on continuous decision groups, comprising: receiving multimodal input data of visual input and language commands; inputting the multimodal input data into a multimodal language model containing multiple continuous decision groups, each continuous decision group including a partial model layer and an action prediction module; sequentially activating multiple continuous decision groups and generating the current action prediction; comparing the current action prediction with the previous action prediction and determining whether an early termination condition is met; if the early termination condition is met, terminating the activation of subsequent continuous decision groups and outputting the current action prediction as the final action command; if the early termination condition is not met, continuing to activate the next continuous decision group and repeating the comparison and judgment until all continuous decision groups are activated; and outputting the action prediction generated by the last continuous decision group as the final action command. This invention achieves dynamic reasoning by setting multiple continuous decision groups in a multimodal language model and combining them with early termination conditions, enabling simple tasks to complete action prediction in the preceding decision group, while complex tasks generate action predictions step-by-step through more decision groups. This mechanism reduces redundant computation, improves inference efficiency, reduces computational resource consumption and real-time latency, and ensures deployability in resource-constrained environments. Attached Figure Description

[0023] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:

[0024] Figure 1 This is a schematic diagram of an application environment for an action generation method based on continuous decision groups according to an embodiment of the present invention;

[0025] Figure 2 This is a flowchart illustrating an embodiment of the action generation method based on continuous decision groups according to the present invention.

[0026] Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the action generation device based on continuous decision groups of the present invention;

[0027] Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0028] Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0029] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0030] The action generation method based on continuous decision groups provided in this invention can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network. The server can receive multimodal input data, including visual input and language commands, from the client. This multimodal input data is then input into a multimodal language model containing multiple consecutive decision groups. Each consecutive decision group includes a partial model layer and an action prediction module. Multiple consecutive decision groups are activated sequentially, generating the current action prediction. The current action prediction is compared with the previous action prediction to determine if an early termination condition is met. If the early termination condition is met, the activation of subsequent consecutive decision groups is terminated, and the current action prediction is output as the final action command. If the early termination condition is not met, the activation of the next consecutive decision group continues, and the comparison and judgment are repeated until all consecutive decision groups are activated. The action prediction generated by the last consecutive decision group is output as the final action command. This invention achieves dynamic reasoning by setting multiple consecutive decision groups in the multimodal language model and combining them with early termination conditions. This allows simple tasks to complete action prediction in the preceding decision group, while complex tasks generate action predictions step-by-step through more decision groups. This mechanism reduces redundant computation, improves reasoning efficiency, reduces computational resource consumption and real-time latency, and ensures deployability in resource-constrained environments. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will now be described in detail through specific embodiments.

[0031] Please see Figure 2 , Figure 2This is a flowchart illustrating an embodiment of the action generation method based on continuous decision groups provided by the present invention. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0032] like Figure 2 As shown, the action generation method based on continuous decision groups proposed in this invention includes the following steps:

[0033] S10, receiving multimodal input data including visual input and language commands;

[0034] In this embodiment, receiving multimodal input data, including visual input and language commands, means simultaneously acquiring image information of the environment and the user's verbal expression. These two types of information need to be synchronized in time and their formats must support unified processing by subsequent models. Visual input typically originates from image sensing devices, such as RGB cameras, depth cameras, or structured light sensors. These devices can convert the physical environment into digital visual data, with common data formats including pixel matrices or point clouds. Language commands can originate from voice input or text input. Voice input requires acquiring waveform data through an acoustic acquisition device, which is then converted into text using a speech recognition algorithm. Direct text input can be obtained through a keyboard, touchscreen, or remote communication interface. Visual input and language commands differ in data type, sampling frequency, and semantic content, thus requiring standardization through a preprocessing stage.

[0035] In the preprocessing of visual data, geometric transformation refers to eliminating distortions caused by the camera's perspective through matrix operations such as rotation, translation, and scaling, ensuring that the captured image remains consistent with the spatial relationships of real objects. Photometric calibration, on the other hand, improves the stability and robustness of image data through brightness equalization, color correction, and noise filtering, avoiding input fluctuations caused by changes in ambient light. The processing method for language commands depends on the input type. For speech commands, a recognition process combining acoustic modeling and language modeling is required to extract speech features and decode them into text. For text input, word segmentation, syntactic structure analysis, and intent extraction are necessary to obtain structured language expressions.

[0036] After standardized visual data and structured language instructions are generated, they need to be temporally aligned and synchronously packaged to ensure that the visual scene at the same moment corresponds to the user's language intent. This is usually achieved through a timestamp mechanism. The final multimodal input data is not just a simple concatenation, but a data pair with alignment information, including temporally matched visual frames and language instruction units.

[0037] In practical implementation, visual input can be obtained through different types of sensing devices, including fixed-position industrial cameras, mobile cameras mounted on mobile robots, and embedded cameras in head-mounted devices. In different application scenarios, the parameters of the geometric transformation need to be dynamically adjusted based on the camera's intrinsic and extrinsic parameters. Photometric calibration can be achieved through hardware-level automatic exposure control or software-level histogram equalization and gamma correction.

[0038] Different implementation methods can be chosen for processing language commands. For speech input, speech recognition based on end-to-end deep models can be used, or a combination of traditional acoustic and language models can be employed. For text input, rule-based parsing methods can be used, or large-scale pre-trained models can be used for semantic understanding and structured output.

[0039] For synchronizing multimodal input data, a buffer mechanism can be used to temporarily store visual frames and language commands, pairing them according to timestamps before packaging them together. Alternatively, a real-time streaming approach can be used, employing an event-triggered mechanism to retrieve and bind the latest visual frame upon receiving a new language command. For resource-constrained devices, compressed storage and lightweight alignment strategies can be used to reduce computational and storage burdens.

[0040] This embodiment ensures the multimodal consistency of input data by collecting, standardizing, and synchronizing visual input and language commands, avoiding interference from time drift or expression differences between different modalities on subsequent reasoning, thereby improving the accuracy and stability of action prediction.

[0041] S20, the multimodal input data is input into a multimodal language model containing multiple continuous decision groups, each of the continuous decision groups containing a partial model layer and an action prediction module;

[0042] In this embodiment, multimodal input data is fed into a multimodal language model containing multiple consecutive decision groups. This means that the input data not only includes standardized visual data and structured language instructions, but also needs to be uniformly received and fed into the model's input. A multimodal language model generally consists of a visual encoder, a language encoder, and a feature fusion layer. Its function is to process data from different modalities separately and achieve semantic alignment in the fusion layer. The representation obtained after merging visual and language features is then passed to subsequent computational units.

[0043] Multiple successive decision groups are the result of segmenting and reorganizing the hierarchical structure of a deep model. Traditional large models often contain dozens of deep network layers. In this structure, each successive decision group consists of several adjacent neural network layers, used for local processing of input features. The reason for this is to achieve dynamic control of computation through grouping, without having to activate all layers throughout the process, and to decide whether to continue computation step by step based on the complexity of the input features.

[0044] Each successive decision group comprises a partial model layer and an action prediction module. The partial model layer refers to the neural network units within the decision group that carry out feature transformations, such as multi-head attention layers, feedforward layers, or convolutional layers. These layers enhance the input features and output new representations. The action prediction module is a lightweight network attached to the end of the decision group. It typically consists of decoding and mapping layers, capable of transforming the output features of the current group into action predictions that can be compared and executed. The advantage of this modular design is that each group can output an action prediction simultaneously with feature computation, providing a foundation for early termination detection and dynamic computation.

[0045] In implementation, the division of the successive decision groups can vary. One approach is to divide the model's 24-layer network into 6 decision groups, each containing 4 layers of neural networks and an action prediction module. This allows for finer computational granularity while maintaining overall performance. Alternatively, a non-uniform partitioning method can be used, for example, where the early decision groups contain only 2 layers for quickly handling simple tasks, while later decision groups contain more layers to handle complex tasks.

[0046] The implementation of the motion prediction module can be diversified. It can employ a fully connected layer with softmax output to map multimodal features to discrete motion categories; alternatively, it can use regression output to directly map features to pose parameters and gripper opening / closing parameters. To adapt to the needs of different execution devices, a parameterized interface can be introduced into the motion prediction module, such as outputting numerical vectors that can be directly parsed by the robotic arm controller.

[0047] The format of the input multimodal data can also be flexibly configured. If the visual input is an RGB image and the language input is Chinese text, features can be extracted through a convolutional neural network of the visual encoder, the text can be parsed through a BERT structure of the language encoder, and then a joint representation can be generated using an interactive attention mechanism in the fusion layer. If the visual input is a depth map and the language input is English speech, features can be extracted separately through a point cloud network and a speech recognition model, and then mapped to a unified fusion space.

[0048] This embodiment avoids redundant computational overhead by inputting multimodal input data into a multimodal language model containing continuous decision groups, enabling adaptive computation to varying task complexities. Each decision group can independently output action predictions, providing a foundation for subsequent early termination mechanisms and dynamic reasoning. This structure effectively reduces computational load and latency while retaining reasoning capabilities for complex tasks, enhancing applicability in resource-constrained environments.

[0049] S30, the plurality of continuous decision groups are activated in sequence. Each time a continuous decision group is activated, the current action prediction is generated through the action prediction module of the continuous decision group.

[0050] In this embodiment, sequentially activating multiple consecutive decision groups means that when processing input data, the model does not run all network layers at once, but activates them one by one according to the pre-defined decision group order. Whenever a consecutive decision group is activated, it receives the input feature representation passed from the previous stage and performs forward propagation computation through a portion of the model layers in that group. These partial model layers may include multi-head attention layers, feedforward neural networks, or convolutional structures, and their role is to further enhance the semantics or model spatial relationships of the input features. After computation, the enhanced multimodal features are used as the output of that group.

[0051] The action prediction module is located at the output of the continuous decision group. Its function is to convert the enhanced multimodal features into action predictions. The conversion process includes two basic operations: first, a mapping layer compresses the high-dimensional features into a control-related low-dimensional space; second, a decoding layer converts these features into usable action parameters, such as pose information or gripper opening / closing parameters. The resulting value is the current action prediction. The design of the action prediction module ensures that each continuous decision group can provide a complete action output while completing its own calculations, providing input for subsequent comparison and judgment steps.

[0052] In practical implementation, different decision group structures can be adopted. For example, each decision group can be designed to contain a three-layer Transformer network and an action prediction module, so that each group has sufficient feature representation capabilities without causing excessive computational burden. Alternatively, a hierarchical design can be adopted, with the early decision groups containing only a few layers and simple prediction modules for quickly outputting preliminary predictions; the later decision groups contain more complex layers and multi-branch prediction structures to handle complex tasks.

[0053] The motion prediction module can employ a fully connected layer with a linear activation function to directly map enhanced multimodal features into numerical pose vectors. Alternatively, a convolutional decoder structure can be used to reconstruct temporal motion sequences for dynamic device control. When applied to robotic arm scenarios, the motion prediction module can also include an output constraint layer to ensure that the generated pose parameters conform to physical constraints, such as not exceeding the angle limits of the robotic arm joints.

[0054] To ensure the effectiveness of sequential activation, trigger conditions can be set for each consecutive decision group, such as activating the next group only when the input features pass validation. This can further reduce invalid computations. Simultaneously, data transfer between consecutive decision groups can employ a tensor caching mechanism, enabling efficient transfer of intermediate results within GPU memory or dedicated acceleration chips, avoiding redundant computations.

[0055] This embodiment achieves a phased and dynamic computation process by sequentially activating consecutive decision groups and generating action predictions for each group. Each stage has independent outputs that can be used for subsequent judgments. This structure avoids the waste of resources from fixed computation throughout the entire process, while ensuring that more refined actions can be calculated further, balancing efficiency and accuracy.

[0056] S40, compare the current action prediction with the previous action prediction, and determine whether the early exit condition is met based on the comparison result;

[0057] In this embodiment, comparing the current motion prediction with the previous motion prediction first requires clarifying how the current motion prediction is obtained. The current motion prediction is the result generated by the motion prediction module when sequentially activating consecutive decision groups. This result typically includes information on control dimensions such as robotic arm pose parameters, gripper opening and closing parameters, or motion path identifiers. To ensure the effectiveness of the comparison process, the most recently generated motion prediction needs to be extracted from the cumulative prediction sequence before comparison and used as the current motion prediction.

[0058] The acquisition of the previous action prediction depends on the historical records of the cumulative prediction sequence. When at least one action prediction exists in the cumulative prediction sequence before the current action prediction, the prediction stored immediately preceding the current action prediction can be retrieved as the previous action prediction. If no historical prediction exists, the early termination condition is directly determined not to be met, thus avoiding invalid calculations.

[0059] The comparison operation essentially involves calculating the differences between the current action prediction and the previous action prediction across various control dimensions. Control dimensions can be position coordinates and angle vectors in the pose, or the opening and closing degree of the gripper, or other motion parameters. Methods for calculating these differences can include Euclidean distance calculation, cosine similarity inverse quantization, or multidimensional absolute difference superposition; the specific method depends on the task requirements.

[0060] After the difference value is calculated, the action consistency threshold set by the current decision group needs to be called. This threshold represents the allowable range of prediction fluctuations, such as the deviation of the robotic arm's end effector not exceeding a certain number of millimeters, or the rotation angle not exceeding a preset radian. The difference value is compared with the threshold. When the difference value is less than or equal to the threshold, the early termination condition is met, indicating that the difference between the current prediction and the previous prediction has stabilized to an acceptable range. When the difference value is greater than the threshold, the early termination condition is not met, indicating that the prediction result is not yet stable, and it is necessary to continue activating the next consecutive decision group to obtain a more accurate prediction.

[0061] During implementation, comparison and judgment can be achieved in various ways. For example, in tasks involving precise movements, weighted Euclidean distance can be used to assign higher weights to key dimensions such as the position of the grasping point, ensuring that the judgment criteria better meet the actual needs of the task. Alternatively, a dynamic threshold strategy can be used to adjust the action consistency threshold in real time according to the complexity of the task environment. For example, a smaller threshold can be used in simpler tasks to trigger the early termination mechanism more quickly, while a larger threshold can be used in complex tasks to ensure accuracy.

[0062] When calculating the difference value, a time smoothing mechanism can be introduced to incorporate multiple historical predictions for reference, rather than relying solely on a single previous action prediction. This approach can further enhance the robustness of the judgment and avoid erroneous early termination decisions due to random fluctuations in a single prediction.

[0063] To improve operational efficiency, vectorized difference calculation and threshold comparison can be implemented on GPUs or dedicated inference chips, enabling comparison and judgment to be completed within milliseconds, thereby meeting the needs of real-time control.

[0064] This embodiment compares the current action prediction with the previous action prediction and determines whether the early termination condition is met based on the difference value. This allows for early termination of subsequent calculations when the action prediction result is stable, reducing redundant model layer activations and lowering computational overhead and energy consumption. Simultaneously, this mechanism ensures continued inference even when prediction results fluctuate significantly, preventing insufficient action execution accuracy and achieving a dynamic balance between efficiency and accuracy.

[0065] S50, if the early exit condition is met, the activation of the subsequent continuous decision group is terminated, and the current action prediction is output as the final action instruction.

[0066] In this embodiment, when the early termination condition is met, the activation process of subsequent successive decision groups needs to be stopped immediately. To achieve this, the system first generates a termination activation signal, which can interrupt the transmission of input feature representations in the control path, preventing subsequent unactivated successive decision groups from receiving new inputs. This termination mechanism can be implemented through a hardware control channel or a model runtime scheduler, for example, by cutting off the execution of subsequent computing nodes in the GPU computation graph, or by skipping unactivated parts of the model layers through dynamic scheduling logic.

[0067] After terminating the activation of subsequent consecutive decision groups, the system needs to extract the current action prediction corresponding to the early termination condition from the cumulative prediction sequence. This action prediction includes control parameters generated by the action prediction module, which may include dimensions such as spatial position, rotation angle, and gripper control signals. This prediction result is considered sufficiently stable and reliable, and therefore directly converted into the final action command.

[0068] To ensure that the instructions can be correctly recognized by the actual executing device, the extracted current motion prediction needs to be formatted. Formatting includes parameter unit conversion, numerical precision adjustment, and mapping of data structures to the actuator interface specification. For example, a robotic arm controller might require a homogeneous transformation matrix as input, while the prediction result consists of separate translation and rotation parameters; in this case, coordinate transformation and matrix construction are necessary.

[0069] Finally, the formatted instructions are sent as the final action instructions to the action execution interface. The action execution interface can be a driver module of the robotic arm controller or a general-purpose actuator protocol stack, such as the ROS control interface. The instructions are transmitted through this interface to the downstream hardware control unit, triggering the robotic arm or other execution devices to complete the actual action.

[0070] In terms of implementation, the termination and output process can be achieved through different hardware and software architectures. A dynamic graph control mechanism can be designed in the model inference engine to skip subsequent computational branches when a signal meeting the early termination condition is detected, reducing GPU computing resource consumption. Alternatively, a low-latency interrupt mechanism can be used in edge devices to directly block the input feature stream, preventing inactive consecutive decision groups from running, which is suitable for power-sensitive robotic platforms.

[0071] In the formatting process, a unified motion instruction standard can be used, such as industrial robot control standards (e.g., MoveIt or KUKA RSI interface), to map the prediction results. An adaptive mapping module can also be introduced to automatically adjust the output format according to the interface specifications of different actuators, enhancing the system's adaptability to heterogeneous hardware.

[0072] In the command transmission phase, real-time communication protocols, such as CAN bus or EtherCAT, can be used to achieve low-latency, high-reliability transmission of action commands. For remote control scenarios, encrypted network communication protocols can also be used to send commands to ensure data security.

[0073] This embodiment significantly reduces redundant calculations, lowers inference latency, and lowers computational resource consumption by terminating the activation of subsequent consecutive decision groups when the early termination condition is met and directly outputting the current action prediction as the final action instruction. Simultaneously, formatting ensures that the action prediction results can be correctly recognized and executed by the actual actuator, making the entire control process both efficient and stable. This mechanism can improve execution efficiency and response speed in robot control scenarios requiring real-time feedback and with limited resources.

[0074] S60, if the early exit condition is not met, continue to activate the next continuous decision group and repeat the comparison and judgment steps until all continuous decision groups are activated, and output the action prediction generated by the last continuous decision group as the final action instruction.

[0075] In this embodiment, when the early termination condition is not met, it indicates that the difference between the current action prediction and the previous action prediction still exceeds the action consistency threshold, and the system needs to continue calculation to obtain a more accurate output. At this point, the position index of the current continuous decision group in the decision group sequence is first determined. The position index can be implemented using a sequence counter; each time a decision group is activated, its index value is incremented by one, thus forming an ordered numbering system used to identify the processing order of the decision groups.

[0076] After the index is determined, the system needs to locate the next consecutive decision group to be activated. The location process depends on the structural mapping relationship of the decision group sequence, which can be implemented through array indexing or linked data structures to ensure that the next consecutive decision group to be processed can be accurately invoked.

[0077] After finding the next consecutive decision group, the current enhanced multimodal feature representation is passed to that decision group as the input feature representation. The passing process requires maintaining the consistency of the data in terms of numerical precision and structure. Tensor copying or shared memory mapping are usually used to reduce latency and ensure lossless connection between input and output.

[0078] After the next decision group receives the input feature representation, it needs to repeat the process of sequentially activating successive decision groups, generating the current action prediction, comparing it, and determining whether the early termination condition is met. Through this recursive execution mode, the system can dynamically determine whether further calculation is needed after processing each decision group.

[0079] If it is determined that all consecutive decision groups have been activated, it means that the early termination condition has not yet been met. In this case, the system retrieves the motion prediction generated by the last consecutive decision group from the cumulative prediction sequence. This motion prediction serves as the optimal output of the system after complete calculation. It then needs to be formatted to generate control commands that conform to the actuator interface specification. The formatting process maps the prediction results to a data structure that the actuator can recognize, such as the angle instruction set of a robotic arm or the clamping force parameters of a gripper. Finally, the generated final motion command is sent to the motion execution interface to trigger the execution device to complete the motion.

[0080] In terms of implementation, indexing and decision group management can be achieved through different structural designs. An array index structure can be used, where each decision group corresponds to an element in the array, and the index value corresponds one-to-one with the position; alternatively, a linked structure can be used, where each decision group stores a pointer to the next decision group, thereby achieving sequential traversal.

[0081] Regarding the method of passing feature representations, direct tensor copying can be used to ensure independence, or shared memory can be used to reduce memory overhead and latency. In a distributed computing environment, feature representations can also be passed between decision groups on different devices through a message queue mechanism to support large-scale model deployment.

[0082] In the final output stage, adaptation can be performed for different types of actuators. For example, for a robotic arm controller, the predicted results can be converted into joint space commands; for a UAV control system, they can be converted into control parameters for three-dimensional spatial position and attitude angles. An automatic adaptation module can also be designed to automatically adjust the format of the output commands according to the actuator interface specifications, thereby enhancing the system's versatility and portability.

[0083] This embodiment ensures sufficient computational results in complex tasks or delicate operation scenarios by continuing to activate subsequent consecutive decision groups and repeating the comparison and judgment when the early termination condition is not met. By introducing a position index and data transfer mechanism, the system avoids omissions or duplicate calculations while ensuring execution efficiency. Finally, the action prediction output after all consecutive decision groups have been completed guarantees the accuracy and stability of the results, thereby improving the reliability of task execution.

[0084] This invention relates to the field of artificial intelligence technology and can be applied to business scenarios such as fintech and healthcare. It discloses a method, apparatus, device, and medium for action generation based on continuous decision groups, comprising: receiving multimodal input data of visual input and language commands; inputting the multimodal input data into a multimodal language model containing multiple continuous decision groups, each continuous decision group including a partial model layer and an action prediction module; sequentially activating multiple continuous decision groups and generating the current action prediction; comparing the current action prediction with the previous action prediction and determining whether an early termination condition is met; if the early termination condition is met, terminating the activation of subsequent continuous decision groups and outputting the current action prediction as the final action command; if the early termination condition is not met, continuing to activate the next continuous decision group and repeating the comparison and judgment until all continuous decision groups are activated; and outputting the action prediction generated by the last continuous decision group as the final action command. This invention achieves dynamic reasoning by setting multiple continuous decision groups in a multimodal language model and combining them with early termination conditions, enabling simple tasks to complete action prediction in the preceding decision group, while complex tasks generate action predictions step-by-step through more decision groups. This mechanism reduces redundant computation, improves inference efficiency, reduces computational resource consumption and real-time latency, and ensures deployability in resource-constrained environments.

[0085] In one embodiment, step S10 above includes:

[0086] S101, captures the physical environment through an image sensing device to obtain raw visual data;

[0087] S102 receives voice or text input through the user interaction interface to obtain the original language command;

[0088] S103, Perform geometric transformation and photometric calibration on the original visual data to generate standardized visual data;

[0089] S104, Perform speech recognition or text parsing processing on the original language instructions to generate structured language instructions;

[0090] S105, synchronize the standardized visual data and the structured language instructions and package them into time-aligned multimodal input data.

[0091] In this embodiment, the goal of receiving multimodal input data, including visual input and language commands, is to provide a consistent, alignable, and reusable data baseline for subsequent dynamic inference, avoiding interference from temporal and scale deviations between different modalities in comparison and early termination judgment. Visual input and language commands differ in sampling methods, encoding forms, and temporal granularity, therefore a unified data representation must be established through four stages: acquisition, standardization, structuring, and alignment.

[0092] Image sensing devices acquire raw visual data from the physical environment. Sources can include industrial cameras, mobile platform cameras, structured light devices, ToF depth cameras, or multi-sensor fusion units. The output is a pixel matrix or point cloud along with timestamps, intrinsic parameters, extrinsic parameters, and exposure information. To ensure stable subsequent computation, fixed or adaptive frame rate control, exposure control, and noise reduction preprocessing are required. Interface types can include MIPI, USB, GigE, or on-chip bus. Caching strategies employ circular buffers or zero-copy shared memory to reduce latency. The metadata of the raw visual data needs to be bound to a clock source, which can be a standalone high-precision timer or network time synchronization, facilitating cross-modal consistency.

[0093] The user interface is used to obtain raw language commands. Voice input is captured as waveforms by a microphone array, and combined with endpoint detection, echo cancellation, noise suppression, and gain control to form a stable acoustic sequence, with timestamps and channel layout preserved. Text input can originate from a touchscreen, keyboard, mobile application, or remote API; the input event stream includes a session identifier and generation time. To ensure consistency across different input paths, encoding standardization and character normalization are required, including case unification, whitespace handling, and special character cleanup.

[0094] Geometric transformations are used to correct imaging geometric errors and establish spatial consistency. This includes distortion correction, perspective alignment, and scale unification, mapping pixel coordinates to a predefined coordinate system using camera intrinsics and extrinsic parameters. In multi-camera or mobile scenarios, a globally consistent field of view is obtained through extrinsic parameter calculation and coordinate stitching. Output resolution, aspect ratio, and region of interest are set according to downstream model requirements, and interpolation and clipping are performed on the GPU graphics pipeline or general-purpose parallel units to maintain throughput.

[0095] Photometric calibration is used to stabilize brightness and color distribution. White balance correction, exposure equalization, color space conversion, flicker suppression, and temporal noise reduction work together to reduce the impact of ambient light fluctuations on downstream features; dynamic range mapping and gamma shaping compress pixel values ​​to the target range, unify pixel format to a predetermined precision, and introduce linearization or HDR fusion when necessary to improve resolvable details. After completion, standardized visual data with stable geometric and photometric properties is obtained.

[0096] Speech recognition or text parsing is used to transform raw language instructions into structured expressions. The speech path extracts time-frequency representations from acoustic sequences, combines this with language modeling to transcribe the text, then performs punctuation restoration and sentence segmentation, outputting reliable text sequences and confidence distributions. The text path performs language recognition and word segmentation, and after unified encoding, completes word form normalization and entity extraction such as numbers, units, place names, and personal names. Both paths construct key fields such as action type, object, positional constraints, and temporal modifications at a unified semantic parsing layer, while preserving confidence and contextual clues to form structured language instructions.

[0097] Synchronization and packaging address multimodal time alignment and transmission consistency. The visual and linguistic timelines are matched based on monotonic timestamps, using strategies such as fixed time windows, nearest neighbor pairing, or latency-tolerant sliding windows. To address inconsistencies between speech segment lengths and video frame rates, buffer queues and playback windows are introduced to ensure stable coverage relationships between linguistic units and visual frame sets. After alignment, standardized visual data and structured language instructions are bound into time-aligned multimodal input data. Data containers record time ranges, source identifiers, spatial coordinate systems, and integrity checks. Memory layout employs compact packing or pointer references to balance bandwidth and multiplexing, with lightweight compression enabled when necessary to adapt to bandwidth-constrained links.

[0098] Anomaly and backoff mechanisms ensure robustness. When visual frames are missing, the nearest keyframe can be filled in and its availability marked. When speech recognition has low confidence, a restatement request can be triggered or the system can switch to the text channel. When time alignment exceeds limits, a delay or a downgraded wide-window alignment result can be output, while quality indicators are recorded in the data container to avoid misjudging stability in subsequent comparisons. The entire acquisition-to-packaging process remains traceable, facilitating the linkage of early termination thresholds and input quality during dynamic inference.

[0099] Through the aforementioned collection, standardization, structuring, and alignment processes, multimodal input data with temporal, spatial, and semantic consistency is ultimately generated, providing directly usable input for subsequent sequential processing and action prediction comparison in continuous decision groups. Geometric transformations and photometric calibration reduce visual variance, speech recognition and text parsing provide clear command intents and constraints, temporal alignment ensures comparability across modal comparisons, and buffering and fault-tolerance strategies maintain input continuity in asynchronous scenarios.

[0100] This embodiment establishes an integrated input chain encompassing acquisition, standardization, structuring, and temporal alignment. Visual and linguistic data converge at the geometric, photometric, and semantic levels, significantly reducing input variance and suppressing cross-modal temporal drift. Dynamic inference can thus execute phased outputs and comparisons of continuous decision groups on stable inputs. Early termination decisions are reliably triggered based on the same reference, reducing false triggers or delays caused by mismatches and noise. This allows for earlier cessation of deep computation in simple scenarios, reducing latency and computational overhead, while maintaining full utilization of high-precision input in complex scenarios. It enables adaptive computational allocation based on task complexity, improving real-time performance, energy efficiency, and reliability, and providing an operational data foundation for deployment on resource-constrained platforms.

[0101] In one embodiment, step S20 above includes:

[0102] S201, Load a pre-trained modal language model containing a visual encoder, a language encoder, and a feature fusion layer;

[0103] S202, the deep network layer of the modal language model is divided into multiple sequentially connected segments, and each segment constitutes a partial model layer of a continuous decision group;

[0104] S203, add a lightweight prediction network to the output of each continuous decision group to form the action prediction module of the continuous decision group;

[0105] S204, input the standardized visual data in the multimodal input data into the visual encoder to obtain a visual feature vector;

[0106] S205, the structured language instructions in the multimodal input data are input into the language encoder to obtain the language feature vector;

[0107] S206, Input the visual feature vector and the language feature vector into the feature fusion layer to generate an initial multimodal feature representation;

[0108] S207, the initial multimodal feature representation is input into a partial model layer of the first consecutive decision group, and the initial multimodal feature representation is processed sequentially through the multiple consecutive decision groups to generate action prediction.

[0109] In this embodiment, multimodal input data is fed into a multimodal language model containing multiple consecutive decision groups. First, three components are prepared: a visual encoder, a language encoder, and a feature fusion layer. The input receives standardized visual data and structured language instructions, which are then fed into the visual encoder and language encoder, respectively. The visual encoder originates from an image feature extraction network, such as a convolutional network or a visual Transformer. Its input is a pixel tensor with fixed or variable resolution, and its output is a visual feature vector. The dimension of this vector is consistent with the downstream fusion space or can be aligned via a linear mapping. The language encoder originates from a sequence modeling network, such as a context modeling structure based on self-attention. Its input is an index sequence of word segments or sub-word units, which, combined with position embedding and attention masks, outputs a language feature vector. The two feature paths then enter the feature fusion layer to complete cross-modal alignment. The fusion layer originates from a cross-modal interaction structure, such as interactive attention, gated concatenation, or two-stream interaction. It outputs an initial multimodal feature representation with a shape consistent with subsequent network conventions. The timestamp and alignment identifier maintain a one-to-one correspondence with the input, ensuring cross-modal temporal consistency.

[0110] Multiple consecutive decision groups are used to introduce a segmentation strategy in deep network graphs, breaking down long-link deep computations into several topologically adjacent consecutive segments. When a deep network layer is divided into multiple sequentially connected segments, the partition boundaries fall at the natural boundaries of residual blocks or attention blocks, avoiding the disruption of state dependencies across boundaries. Each segment contains several model layers responsible for performing nonlinear transformations and context enhancement on the input features. Continuous decision groups are connected through explicit tensor interfaces. The interface definitions use a unified data type and alignment format, and positional encoding, normalized statistics, and cached key-value pairs are passed sequentially between groups to maintain semantic continuity and numerical stability. To reduce memory usage, a lazy reuse strategy for intermediate activations can be introduced. Temporary tensors not used downstream are released immediately after leaving the current continuous decision group, and reconstructed as needed in subsequent stages through a checkpoint recalculation mechanism.

[0111] A lightweight prediction network is added to the output of each continuous decision group to form an action prediction module, achieving locally observable output. The action prediction module originates from the control output head and its structure can consist of a feature compression layer and an action mapping layer. The feature compression layer compresses the high-dimensional multimodal representation into the control-related space, while the action mapping layer transforms the compressed representation into a set of action parameters, such as the end-effector pose vector, gripper opening / closing amplitude, or discrete action labels. To improve adaptability to different execution objects, the action mapping layer can introduce normalization and coordinate system mapping units to unify the output to a predefined control coordinate system. To improve numerical stability, boundary clipping and smoothing units can be added before the output to suppress extreme values ​​and spike disturbances. The action prediction module and the continuous decision group have a one-to-one binding relationship, allowing a comparable current action prediction to be obtained from the current continuous decision group at any time, providing direct input for subsequent early termination judgment.

[0112] Before standardized visual data enters the visual encoder, geometric and photometric unification is performed to avoid unnecessary representational disturbances introduced by perspective distortion and color fluctuations. Geometric unification is achieved through intrinsic and extrinsic parameter mapping and perspective calibration, while photometric unification is achieved through white balance and dynamic range shaping. These processes ensure the comparability of visual feature vectors under different environmental conditions. Before structured language instructions enter the language encoder, character normalization, lexical merging, and entity extraction are performed to avoid the fragmentation of synonymous expressions affecting semantic discrimination. Language feature vectors are generated by context-sensitive embedding and can carry the dependencies between action verbs, object entities, and spatial modifiers.

[0113] After the initial multimodal feature representation is generated by the feature fusion layer, it is directly fed into a portion of the model layer of the first consecutive decision group for sequential processing. Sequential processing follows the ascending order of decision group indices. After any consecutive decision group completes its internal computation, the action prediction module provides the current action prediction and simultaneously derives the enhanced multimodal representation as input for the next consecutive decision group. To ensure high throughput in the inference chain, mechanisms such as tensor channel reuse, in-situ updates of layer normalized statistics, and attention cache reuse can be enabled during sequential processing to reduce resource consumption caused by repeated computation paths. If time alignment is performed on the multimodal input data in the upstream stage, no additional time resampling is introduced within the consecutive decision groups, ensuring that the time reference for action prediction is consistent with the input, thus guaranteeing time consistency across groups. To facilitate integration into heterogeneous hardware platforms, the interfaces between the visual encoder, language encoder, feature fusion layer, and consecutive decision groups use fixed signatures and well-defined tensor shapes. The inference engine can execute according to the graph or schedule as needed, ensuring the controllability and traceability of segmented execution and intermediate outputs.

[0114] The origins and examples of each term are as follows: Visual encoders originate from image feature extraction, exemplified by convolutional networks, visual Transformers, and multi-scale pyramid structures; language encoders originate from natural language sequence modeling, exemplified by self-attention-based context models and word embedding systems; feature fusion layers originate from cross-modal alignment and interaction, exemplified by interactive attention, gated concatenation, and bidirectional mutual information distillation; continuous decision groups originate from deep network segmented execution and intermediate exit design, exemplified by residual block segmentation, attention cluster segmentation, and hybrid segmentation; some model layers originate from functional units within deep networks, exemplified by multi-head attention, feedforward networks, normalization, and activation; action prediction modules originate from control output mapping, exemplified by multilayer perceptron mapping, distributed parameter regression, and discrete action classification. Standardized visual data and structured language instructions originate from unified processing of the preceding input stages; visual feature vectors and language feature vectors originate from the outputs of the two encoders; and the initial multimodal feature representation originates from the output of the fusion layer. Through the connection between the above components and intermediates, multimodal input data forms a self-consistent representation link after entering the model. The continuous decision group provides observable outputs at each stage, which not only meets the needs of subsequent comparison and early termination judgment, but also avoids the waste of resources caused by the one-time full-link operation of the entire network.

[0115] This embodiment achieves multimodal alignment through a visual encoder, a language encoder, and a feature fusion layer. The deep network is reorganized into multiple consecutive decision groups, with an action prediction module at the end of each group. This allows the initial multimodal feature representation to continuously generate observable outputs in the segmented process. Segmented processing yields two layers of derivation results. First, any stage can generate the current action prediction, and subsequent stages use this to perform comparisons and early termination checks, thus stopping deep computation when the output stabilizes. Delay and resource consumption adaptively converge with task difficulty. Second, in cases where early termination is not triggered, the process continues sequentially along the consecutive decision groups. The enhanced multimodal representation accumulates semantics and spatial constraints in a deeper representation space, achieving refined generation of complex actions. This simultaneously achieves both computational savings and preservation of expressive power, adapting to the real-time control requirements of resource-constrained platforms and maintaining high-precision action prediction in complex scenarios.

[0116] In one embodiment, step S30 above includes:

[0117] S301, starting with the first consecutive decision group, the initial multimodal feature representation is processed sequentially;

[0118] S302, For the currently activated continuous decision group, the input features are forward-propagated through a portion of the model layers of the currently activated continuous decision group to obtain an enhanced multimodal feature representation;

[0119] S303, the enhanced multimodal feature representation is input to the action prediction module of the currently activated continuous decision group. The action prediction module decodes and maps the enhanced multimodal feature representation to generate an action prediction containing pose and gripper control parameters as the current action prediction.

[0120] S304, Add the generated action prediction to the cumulative prediction sequence;

[0121] S305, the enhanced multimodal feature representation is used as the input feature of the next successive decision group to be activated.

[0122] In this embodiment, the goal of sequentially activating multiple consecutive decision groups is to process the initial multimodal feature representation in a segmented manner, so that each segment can produce an evaluable action output after completing its own transformation. Sequential processing of the initial multimodal feature representation means inputting the time-aligned joint representation sequentially, starting from the consecutive decision group with the smallest index, following a pre-determined inter-group topological order and tensor interface specification, ensuring that the tensor shape, coordinate system, timestamp, and normalization state remain consistent during cross-group transmission. The current consecutive decision group represents the segmented unit currently in an active state, derived from the structured decomposition of deep networks; examples can be residual clusters composed of several multi-head attention layers and feedforward layers, or convolutional clusters containing cross-channel interactions and spatial mapping. Forward propagation of the input features through a portion of the model layers in the current consecutive decision group is essentially a combination of linear and nonlinear transformation sequences, including tensor operations such as weight mapping, normalization, activation, and residual superposition; to stabilize the numerical range, layer normalization and scaling units can be placed at the beginning and end of each segment; to improve throughput, key-value caching and on-demand reuse of intermediate activations can be enabled on the device side. Enhanced multimodal feature representation refers to the joint representation after transformation within the current continuous decision group, which deepens its semantic association, spatial connection and task constraints; it originates from cross-channel interaction and context fusion within the segment; examples can be the concentration of attention weights on the target region, channel enhancement of action-related semantics, and spatial suppression of obstacle boundaries.

[0123] The enhanced multimodal feature representation is fed into the action prediction module of the current continuous decision group to directly generate an action output at the end of the segment. The action prediction module is derived from the design paradigm of the control mapping head and generally consists of a feature compression unit and a control mapping unit. The feature compression unit projects the high-dimensional representation onto a low-dimensional subspace related to control, and the control mapping unit then maps it into a set of action parameters. An action prediction containing pose and gripper control parameters is generated as the current action prediction. The pose can be a combination of position ternary vectors and attitude quaternary vectors or rotation vectors, and the gripper control parameters can be represented by scalars or vectors of opening and closing amplitude or target gripping force. To adapt to different actuators, a coordinate system module and a boundary clipping module can be added at the end of the mapping to ensure that the output meets the mechanically reachable domain and safety constraints. After the current action prediction is generated, it is appended to the cumulative prediction sequence. The cumulative prediction sequence is an ordered container maintained by time or group index, recording each segment-end output and its metadata (timestamp, confidence level, source group index). This container provides direct input for subsequent comparison and early termination judgment, and also supports backtracking checks and running records. The enhanced multimodal feature representation is used as the input feature of the next successive decision group to be activated. It requires maintaining accuracy consistency and a zero-copy memory priority strategy when passing across groups. Shared buffers and read-only views can be used to avoid duplicate copies. Alternatively, half-precision tensor transmission can be used in heterogeneous computing environments, and accuracy recovery can be performed at the entry point. When there is long sequence inference, lightweight compression and fast decompression units can be inserted between segments to limit the use of video memory and bandwidth.

[0124] The sources and scope of the above terms remain open: Continuous decision groups originate from the segmented execution concept of deep networks, examples include residual block cluster segmentation and attention cluster segmentation; some model layers originate from deep representation units, examples include multi-head attention, feedforward networks, depthwise separable convolutions, and gating units; action prediction modules originate from control head design, examples include multilayer perceptron mapping, hybrid classification regression heads, and distributed parameter regression heads; pose and gripper control parameters originate from commonly used motion description systems in robotics control, examples include six-dimensional representation of end-effector pose and force-related grip amplitude; cumulative prediction sequences originate from online caching and diagnostic needs, examples include circular queues, time-ordered lists, and tensor logs. Through sequential activation and end-of-segment observable output, the initial multimodal feature representation is progressively enhanced in a deeper representation space. The current action prediction can be generated and recorded in each segment, providing a continuous, traceable, and backtrackable basis for subsequent consistency comparisons and dynamic early termination; the enhanced multimodal feature representation provides more sufficient context for the next segment, enabling deeper segments to continue to perform more refined action inference when necessary.

[0125] This embodiment activates sequentially in segments and generates a current action prediction at the end of each segment. The action output forms a sequence that can be evaluated segment by segment. By combining the cumulative prediction sequence, early termination can be triggered at a stable point to avoid unnecessary deep computation. When the stable interval has not been reached, the enhanced multimodal feature representation is accurately passed to the next segment to continue to deepen the representation. This shortens the inference chain, reduces latency and resource consumption in simple cases, and retains in-depth inference capabilities in complex cases, forming an adaptive computation allocation and real-time response based on task difficulty.

[0126] In one embodiment, step S40 above includes:

[0127] S401, retrieve the latest generated action prediction from the cumulative prediction sequence as the current action prediction;

[0128] S401, determine whether other action predictions have been stored in the cumulative prediction sequence before the current action prediction;

[0129] S401, if other action predictions have been stored in the cumulative prediction sequence before the current action prediction, then the action prediction stored immediately before the current action prediction is retrieved as the previous action prediction.

[0130] S401, Determine the difference between the current action prediction and the previous action prediction in each control dimension;

[0131] S401, Obtain the action consistency threshold set for the current decision group;

[0132] S401, compare the difference value with the action consistency threshold;

[0133] S401, if the difference value is less than or equal to the action consistency threshold, it is determined that the early departure condition is met; if the difference value is greater than the action consistency threshold, it is determined that the early departure condition is not met.

[0134] S401, if no other action predictions are stored in the cumulative prediction sequence before the current action prediction, then it is determined that the early termination condition is not met.

[0135] In this embodiment, the goal of comparing the current action prediction with the previous action prediction is to determine whether the prediction sequence has stabilized to an acceptable range, thereby providing a reliable basis for triggering the early termination condition. First, the latest generated action prediction is retrieved from the cumulative prediction sequence, establishing a correspondence between the current action prediction and the time index to ensure a one-to-one match with the output of the upstream continuous decision group. The retrieval process uses timestamps or group indexes as keys, employing read-only views or zero-copy pointers to obtain data, avoiding unnecessary memory copying. Next, it checks whether other action predictions have been stored in the sequence before the current position. If no earlier entries exist, a determination that the early termination condition is not met is directly output, and the reason label is recorded as "lack of comparison benchmark." If earlier entries exist, the entry immediately preceding the current entry is located as the previous action prediction, ensuring adjacent pairing and avoiding crossing intermediate entries to prevent amplifying differences due to excessively large spans.

[0136] The determination of the difference value revolves around the control dimensions. These dimensions can include end-effector position, posture, gripper opening / closing parameters, path progress, or speed constraints. Each dimension is normalized according to a unified unit and dimensional benchmark to eliminate comparison biases caused by inconsistencies in dimensionality. The actual calculation employs a two-stage process: dimension-by-dimensional differencing and aggregation. In the difference-by-dimensional stage, a separate difference measure is generated for each dimension, while simultaneously recording the direction marker and confidence weight. In the aggregation stage, the differences of each dimension are weighted and synthesized within the weight space to obtain a single difference value for threshold comparison. To suppress the impact of occasional noise on the decision, outlier suppression and smoothing strategies are introduced. Outlier suppression applies robust statistics or out-of-threshold pruning within the dimension, while the smoothing strategy performs short-window fusion of adjacent differences over time, making the decision insensitive to instantaneous jitter. If the upstream provides input quality indicators, such as visual frame confidence or language parsing confidence, the weights of relevant dimensions can be dynamically adjusted in the aggregation stage, providing an adaptive buffer against input quality degradation in the difference value.

[0137] The action consistency threshold is derived from the current decision group's configuration or runtime adaptive strategy. Static configuration, based on task safety requirements and target accuracy, provides basic thresholds and dimension weights for different control dimensions. The adaptive strategy considers operational signals such as scenario complexity, residual prediction fluctuations, equipment load, and power consumption constraints, applying increment or decrement coefficients to the basic threshold to achieve a trade-off between real-time performance and accuracy. The actual threshold value is bound to the current decision group and time index during sequence retrieval, forming a set of traceable comparison parameters for easy subsequent auditing and parameter tuning.

[0138] The comparison between the difference value and the action consistency threshold follows two requirements: a single decision criterion and verifiable records. The single decision criterion ensures that only one Boolean result is generated at any given time, avoiding inconsistencies caused by multiple criteria. Verifiable records write metadata such as the difference value, threshold, dimension weights, input quality identifiers, and outlier pruning markers along with the decision result into the runtime log for offline playback and threshold retraining. If the comparison result meets the early termination condition, the result is immediately submitted to the downstream process along with the current action prediction; if the comparison result does not meet the early termination condition, a residual overview and a list of major out-of-limit dimensions are attached for scheduling reference when activating the next consecutive decision group.

[0139] Boundary cases require clearly defined handling strategies. When the cumulative prediction sequence has no previous action prediction, the system directly outputs that the early termination condition is not met and marks the first comparison in the sequence to ensure the process can continue. When the previous action prediction exists but contains missing dimensions, the comparison is only performed on the intersection dimension, and the missing dimension is included in the quality deduction to improve the conservatism of subsequent comparisons. When the current action prediction comes from an abnormal input quality range, the threshold can be automatically tightened or loosened according to the quality factor to avoid falsely triggering early termination or excessive delay. All boundary processing is completed without changing the terminology system and data flow direction, ensuring consistency with previous definitions and scalability for subsequent comparisons.

[0140] This embodiment achieves stable and traceable judgment results by using adjacent pairing in the cumulative prediction sequence, normalization and weighted aggregation in the control dimension, and threshold adaptation and anomaly suppression during runtime, comparing the current action prediction with the previous action prediction. Based on this judgment, early termination conditions are reliably triggered when prediction converges, deep computation is stopped in a timely manner when it is no longer necessary, and inference latency and resource consumption adaptively converge with task complexity. When prediction has not yet converged, the judgment provides excess dimension and residual information to support subsequent continuous decision groups to continue deepening computation, thereby achieving a controllable balance between efficiency and accuracy.

[0141] In one embodiment, step S50 above includes:

[0142] S501, if the early termination condition is met, a termination activation signal is generated to prevent the input feature representation from being sent to subsequent successive decision groups;

[0143] S502, retrieve the current action prediction corresponding to the early exit condition from the cumulative prediction sequence;

[0144] S503, the retrieved current action prediction is formatted to generate control instructions that conform to the actuator interface specification as the final action instructions;

[0145] S504, the final action instruction is sent to the action execution interface.

[0146] In this embodiment, when the early termination condition is met, the activation of subsequent continuous decision groups needs to be terminated immediately, and the current action prediction is safely and deterministically converted into the final action instruction output. To this end, a termination activation signal is generated first, and the signal path simultaneously covers the model runtime scheduler and the data path. After receiving the termination activation signal, the runtime scheduler marks subsequent continuous decision groups as skipped, cancels unsubmitted operator queues and asynchronous kernel calls; the data path sets a shielding flag at the tensor exchange layer to block the transmission of input feature representations to subsequent groups. To avoid race conditions in concurrent states, a monotonically increasing sequence number needs to be attached when generating the termination activation signal, and a comparison is performed on the scheduler side, accepting only signals with a sequence number not less than the current frame identifier; memory barriers and event synchronization points are set in the data path to ensure that the running kernel safely terminates before releasing intermediate buffers, avoiding dangling pointers and resource leaks.

[0147] Simultaneously with the termination process, the current action prediction corresponding to the early termination condition is retrieved from the cumulative prediction sequence. The retrieval key is a timestamp or decision group index, consistent with the early termination judgment record to ensure traceability. If the sequence contains multiple candidates at the same time, the entry with the highest confidence and complete dimensional coverage is selected according to a priority strategy; if missing dimensions exist, they are filled in with interpolation or the most recent valid value, and the source is marked in the metadata for subsequent auditing. The current action prediction typically contains multi-dimensional vectors such as pose and gripper control parameters. The units and coordinate system are based on the model's internal conventions and are not directly adapted to external execution devices, therefore, formatting is required.

[0148] The formatting process comprises four stages: unit conversion, coordinate system mapping, numerical trimming, and data layout serialization. Unit conversion unifies internal specifications to the length, angle, and mechanical units required by the actuator interface specification. Coordinate system mapping transforms the model reference frame to the actuator base frame or end effector frame based on the calibration matrix, adding tool compensation when necessary. Numerical trimming limits and smooths the pose and gripper control parameters based on the mechanically reachable domain, joint limits, and velocity / acceleration constraints, preventing instantaneous spikes. Data layout serialization organizes parameters into structures or message frames specified by the actuator interface specification, including checksums, timestamps, and action identifiers. To improve fault tolerance and portability, the formatting pipeline introduces a configurable mapping table, supporting one-click switching between different brands or protocol stacks, and loading and verifying it during startup.

[0149] The final action command is sent to the downstream control system via the action execution interface. The communication layer selects a real-time bus or low-latency network channel, such as EtherCAT, CAN, RS-485, UDP multicast, or ROS2 DDS, based on the scenario, and sets service quality parameters and timeout retransmission policies. Command transmission uses idempotent encapsulation, including a transaction number and acknowledgment mechanism; if no acknowledgment is received within the timeout window, a fast retry is initiated and an alarm is recorded; if consecutive failures occur, a rollback strategy is triggered, switching the execution target to a predetermined safe pose or requesting manual intervention. To ensure timing consistency, the action execution interface supports future execution tags, allowing alignment of execution at control cycle boundaries; to improve operational observability, the current action prediction, differences before and after formatting, pruning reasons, and communication status are synchronously written to the operation log for offline playback and parameter tuning. The entire link ensures mutual exclusion between termination and output: once the termination activation signal takes effect, subsequent continuous decision groups no longer receive input feature representations, ensuring the final action command has a unique source and consistent context.

[0150] This embodiment cuts off continuous computation and data transmission by terminating the activation signal, so subsequent continuous decision groups no longer consume computing power and memory. Relying on the accurate retrieval and formatting of the accumulated prediction sequence, the current action prediction is stably mapped into an executable final action instruction. With the help of the real-time communication, confirmation, and rollback mechanism of the action execution interface, the output is completed within the time and safety boundaries. Thus, on the one hand, deep computation is stopped immediately at the moment of prediction convergence, significantly reducing latency and resource overhead; on the other hand, unit conversion, coordinate system mapping, and numerical pruning ensure that the control parameters and actuator interface specifications are strictly consistent, reducing execution errors and anomaly risks, and improving the overall reliability and deployability of the operation.

[0151] In one embodiment, step S60 above includes:

[0152] S601, if the early exit condition is not met, determine the position index of the current continuous decision group in the decision group sequence;

[0153] S602, determine the next consecutive decision group to be activated based on the location index;

[0154] S603, pass the current enhanced multimodal feature representation to the next continuous decision group to be activated as the input feature representation;

[0155] S604, Repeat the steps of activating the continuous decision group, generating the current action prediction, comparing and determining whether the early departure condition is met;

[0156] S605, if it is determined that all consecutive decision groups have been activated, then retrieve the action prediction generated by the last consecutive decision group from the cumulative prediction sequence.

[0157] S606, the last retrieved motion prediction is formatted to generate a control instruction that conforms to the actuator interface specification as the final motion instruction;

[0158] S607, the final action instruction is sent to the action execution interface.

[0159] In this embodiment, when the comparison and judgment result does not meet the early termination condition, the calculation needs to continue in the decision group sequence. First, the position index is calculated based on the group identifier of the current continuous decision group and the global sequence table. The index generation adopts two mechanisms: monotonically increasing counting or static mapping table. Monotonically increasing counting avoids concurrent write conflicts through atomic addition and ring protection. The static mapping table is fixed along with the network topology during the initialization phase, and the query is O(1) access. After the index is determined, the next continuous decision group to be activated is located through the sequence mapping relationship. The mapping relationship can be direct access of array subscripts or linked pointers. When there is conditional branching or cross-device deployment, the mapping relationship is accompanied by device identifier, memory partition and priority label for the scheduler to perform affinity scheduling.

[0160] The current enhanced multimodal feature representation performs consistency checks and memory strategy selection before cross-group transmission. Consistency checks include shape verification, coordinate system label verification, normalized state verification, and timestamp monotonicity verification. Any anomalies are logged and the system reverts to the nearest valid intermediate representation according to the strategy. The memory strategy prioritizes zero-copy shared buffers, degenerating to local copying within the same device when necessary. Cross-device transmission reduces handshakes through fixed-bandwidth channels and batch packaging, and precision restoration and alignment are performed at the entry point. The next pending continuous decision group, after receiving input, continues forward propagation in a predetermined order, and the action prediction module produces a new current action prediction. The new prediction entry, along with the group index, timestamp, and confidence score, is written to the cumulative prediction sequence. Sequence writing employs a lock-free structure or fine-grained segmented locks to ensure throughput during high-frequency iterations while maintaining temporal consistency between entries.

[0161] The closed loop of sequential activation, prediction, comparison, and judgment is driven by a runtime scheduler. After each round of comparison, the scheduler reads the comparison results between the difference value and the consistency threshold. If the early termination condition is still not met, it advances to the next consecutive decision group based on the position index; if it has reached the end of the sequence, it triggers a full completion branch. To avoid invalid iterations, the scheduler carries a residual overview and major out-of-limit dimensions with each update as heuristic signals to guide kernel selection (e.g., enabling a kernel version with a larger receptive field or a deeper attention layer), achieving targeted deepening rather than blind advancement. To suppress jitter and livelock, a finite-length dwell window is introduced. If the residual decrease is insufficient for several consecutive rounds, it is allowed to directly enter the terminal convergence judgment, improving the overall convergence speed.

[0162] When all consecutive decision groups have been activated, the last entry in the cumulative prediction sequence is retrieved as the final candidate. The retrieval key is the combined key of the maximum group index and the latest timestamp. If there are concurrent writes from multiple sources, the first entry is selected based on confidence level and dimensional integrity. The retrieved action predictions enter the formatting pipeline: unit conversion maps internal measurements to the length, angle, and mechanical units required by the execution interface; coordinate system mapping uses a calibration matrix to convert from the model reference system to the actuator base system or end-effector tool system, and adds tool compensation and bias correction; boundary trimming saturates and smooths each dimension based on reachability, joint limits, and velocity / acceleration limits to avoid generating unexecutable or unsafe instructions; data layout serialization assembles the data into message frames or instruction structures according to the interface protocol, with transaction number, timestamp, and checksum. Upon completion, the action is sent via the action execution interface. The communication layer can be configured with a real-time bus (EtherCAT, CAN, RS-485) or a low-latency network channel (UDP, DDS), employing an acknowledgment and retry strategy to ensure reliable delivery. If a timeout or failure reaches a threshold, a fallback strategy is triggered to output a safe pose command and record an alarm. The entire link maintains a single-source output principle: the final action command is generated and sent only in one of two mutually exclusive scenarios: full completion or early fallback, avoiding duplication or conflict.

[0163] Example Explanation: In fintech businesses, users submit a task via voice command, such as requesting a smart robot to transfer funds across accounts, verify accounts, or perform compliance checks. The robot first captures raw visual data using a camera, such as QR codes on transaction documents, invoice images, or terminal interface images. Simultaneously, it receives natural language input via a microphone or input interface as raw verbal commands. This raw input is processed: the visual data undergoes geometric transformations and photometric calibration to ensure critical information such as amounts and account numbers remain accurate under varying lighting conditions; the verbal commands are converted into structured account information, amount parameters, and operation types through speech recognition or text parsing. Subsequently, the robot synchronizes the processed visual data and verbal commands in time, packaging them into time-aligned multimodal input data.

[0164] These multimodal input data are fed into a multimodal language model containing multiple consecutive decision groups. The model loads a visual encoder and a language encoder. The visual encoder extracts feature vectors such as amount and account number from the payment voucher, while the language encoder parses the user's transaction intent. These are then synthesized into an initial multimodal feature representation via a feature fusion layer. The deep network is divided into multiple consecutive decision groups, each containing a portion of the model layers and an action prediction module. The first decision group receives the initial feature representation and generates action prediction results, such as the direction of movement of a robot arm or the opening and closing amplitude of a gripper. These action predictions are directly mapped to the execution pose and confirmation identifier in financial operations.

[0165] The system sequentially activates each consecutive decision group. With each activation, the input features are processed through the group's model layer to generate an enhanced multimodal feature representation, which is then input into the action prediction module to generate a new action prediction. The robot adds these predictions one by one to the cumulative prediction sequence, while simultaneously using the enhanced features as input for the next decision group, ensuring that information is progressively enriched through layer-by-layer transmission. After each new prediction is generated, the system extracts the current prediction from the cumulative prediction sequence and compares it with the previous prediction, checking for consistency in fields such as amount, account, and authorization identifier. By calculating the differences in these dimensions and comparing them with a preset consistency threshold, if the difference is below or equal to the threshold, it indicates that the action has stabilized, and the system meets the early termination condition. It then directly terminates the operation of subsequent decision groups, formats the current prediction into a control command, and passes it to the actuator interface, thereby triggering the robot to complete the transfer confirmation or terminal operation.

[0166] If the discrepancy still exceeds the threshold, the robot will activate the next consecutive decision group. The system locates the next decision group to be processed using a location index, passes enhanced multimodal features as input, and repeats the action prediction and comparison process. Finally, when all decision groups have been activated, the last prediction result is extracted from the cumulative prediction sequence and converted into a control command conforming to the actuator interface specification, ensuring accurate execution even for complex transactions. The control command is sent to the action execution interface, driving the robot to complete operations on the financial terminal, such as screen confirmation, QR code scanning, or password input, ensuring the safe and controllable flow of funds.

[0167] In this way, robots in fintech scenarios can not only efficiently parse multimodal instructions but also dynamically adjust computing resource consumption, avoiding excessive waste of computing power on simple tasks while maintaining high-precision execution in complex tasks. Ultimately, this achieves a balance between reduced transaction processing latency, lower power consumption, and improved execution reliability, enabling financial robots to operate stably on resource-constrained device platforms and meet the business requirements of high-frequency trading and stringent compliance.

[0168] In the healthcare field, service robot systems designed for home rehabilitation and elderly care are being developed. Users can issue rehabilitation training requests via voice commands, such as instructing the robot to assist with grasping training or medication delivery. The robot first uses image sensors to capture real-time visual data of the home environment, including the desktop, medicine box, or training equipment. Simultaneously, it captures the user's natural language input via microphone, such as "Please put the medicine box on the table" or "Start hand grasping training." This raw visual data undergoes geometric transformation and photometric calibration to eliminate interference from lighting differences, generating standardized visual data. The language input is then processed through speech recognition and text parsing to generate structured language commands. Both types of information are simultaneously packaged into time-aligned multimodal input data, ensuring the robot maintains a consistent understanding of the environmental state and user needs during action reasoning.

[0169] The received multimodal input data is fed into a multimodal language model containing multiple consecutive decision groups. A pre-loaded visual encoder extracts geometric features of the medicine box location, user gestures, or rehabilitation equipment, while the language encoder parses the user's operational intent. Subsequently, an initial multimodal feature representation is generated through a feature fusion layer. The deep network is divided into multiple sequentially connected decision groups, each containing a portion of the model layers and an action prediction module. The first decision group receives the initial features and generates action predictions, such as arm movement paths or gripper opening and closing amplitudes. These predictions guide the robot on how to contact the target object.

[0170] The system sequentially activates multiple consecutive decision groups. Upon activation, the input features are forward-propagated through a portion of the model layers to obtain an enhanced multimodal feature representation, which is then passed to the action prediction module to generate new action predictions, such as robot arm pose adjustments or changes in grip strength. These generated action predictions are added to the cumulative prediction sequence one by one, while the enhanced features serve as input for the next decision group, ensuring continuous enrichment of information through layer-by-layer propagation. After each new action prediction is generated, the system retrieves it from the cumulative prediction sequence as the current action prediction and compares it with the immediately preceding action prediction, considering differences in parameters such as displacement distance, hand posture, and grip parameters. By calculating these differences in control dimensions and comparing them with a consistency threshold, if the difference is below or equal to the threshold, an early termination condition is met. The system immediately generates a termination activation signal, preventing subsequent decision groups from continuing, and formats the current action prediction into control instructions conforming to the actuator interface specification, passing them to the execution interface to drive the robot to grasp a medicine box or assist a patient in completing a rehabilitation movement.

[0171] If the comparison result shows that the difference value is still higher than the threshold, it indicates that the action prediction has not yet converged to a stable state. The system will determine the next continuous decision group to be activated through the position index, pass the enhanced multimodal features to the group, and repeat the cycle of action prediction and comparison judgment. When all continuous decision groups are activated, the system extracts the action prediction generated by the last continuous decision group from the cumulative prediction sequence, formats it into the final control command, and sends it to the action execution interface to ensure reliable output even in complex operations. Finally, the robot completes tasks such as delivering medicine, placing auxiliary training equipment, or executing continuous action sequences according to the control command.

[0172] In this healthcare scenario, the processing flow can dynamically adjust the decision depth according to the complexity of the task, avoiding the waste of computing resources in simple tasks, while maintaining high-precision reasoning and execution in complex tasks. This enables low-latency, low-power, and high-reliability robot operation in home rehabilitation and care environments, meeting the long-term rehabilitation training and daily assistance needs of patients.

[0173] This embodiment employs a position-index-driven sequential progression and a zero-copy-first tensor transfer mechanism to continue computationally with minimal overhead when early termination conditions are not met. Leveraging the ordered records of the cumulative prediction sequence and the scheduler's heuristic kernel selection, iterative iterations focus on the expression channels containing the over-limit dimensions, reducing unnecessary deep computations. After traversing to the end of the sequence, a formatted pipeline completes a one-time mapping of units, coordinates, and constraints, ensuring strict consistency between the final action instructions and the execution interface. This avoids redundant layer activation in simple cases and preserves deep expressive capabilities in complex cases, achieving adaptive convergence of latency and computational power consumption with task difficulty, while ensuring the executability and safety boundaries of the final output.

[0174] In one embodiment, an action generation device based on continuous decision groups is provided, which corresponds one-to-one with the action generation method based on continuous decision groups in the above embodiments. (Refer to...) Figure 3 , Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the action generation device based on continuous decision groups of the present invention. The module includes an input parsing module 10, a multimodal encoding module 20, a decision group activation module 30, a prediction comparison module 40, an early termination control module 50, and a result output module 60. Detailed descriptions of each functional module are as follows:

[0175] The input parsing module 10 is used to receive multimodal input data, including visual input and language commands;

[0176] The multimodal coding module 20 is used to input the multimodal input data into a multimodal language model containing multiple consecutive decision groups, each of the consecutive decision groups containing a partial model layer and an action prediction module;

[0177] The decision group activation module 30 is used to activate the plurality of consecutive decision groups in sequence. Each time a consecutive decision group is activated, the current action prediction is generated through the action prediction module of the consecutive decision group.

[0178] The prediction comparison module 40 is used to compare the current action prediction with the previous action prediction, and determine whether the early exit condition is met based on the comparison result.

[0179] The early departure control module 50 is used to terminate the activation of subsequent continuous decision groups if the early departure condition is met, and output the current action prediction as the final action instruction.

[0180] The result output module 60 is used to continue activating the next continuous decision group and repeating the comparison and judgment steps if the early exit condition is not met, until all continuous decision groups are activated, and outputs the action prediction generated by the last continuous decision group as the final action instruction.

[0181] In one embodiment, the input parsing module 10 is specifically used for:

[0182] The physical environment is captured by an image sensing device to obtain raw visual data;

[0183] Receive voice or text input through the user interaction interface to obtain raw language commands;

[0184] The original visual data is subjected to geometric transformation and photometric calibration to generate standardized visual data;

[0185] The original language instructions are processed by speech recognition or text parsing to generate structured language instructions;

[0186] The standardized visual data and the structured language instructions are synchronized and packaged into time-aligned multimodal input data.

[0187] In one embodiment, the multimodal coding module 20 is specifically used for:

[0188] Load a pre-trained modal language model that includes a visual encoder, a language encoder, and a feature fusion layer;

[0189] The deep network layer of the modal language model is divided into multiple sequentially connected segments, and each segment constitutes a partial model layer of a continuous decision group.

[0190] A lightweight prediction network is added to the output of each continuous decision group to form the action prediction module of the continuous decision group;

[0191] The standardized visual data from the multimodal input data is input into the visual encoder to obtain a visual feature vector;

[0192] The structured language instructions in the multimodal input data are input into the language encoder to obtain language feature vectors;

[0193] The visual feature vector and the language feature vector are input into the feature fusion layer to generate an initial multimodal feature representation;

[0194] The initial multimodal feature representation is input into a partial model layer of the first consecutive decision group, and the initial multimodal feature representation is then processed sequentially through the multiple consecutive decision groups to generate action predictions.

[0195] In one embodiment, the decision group activation module 30 is specifically used for:

[0196] The initial multimodal feature representations are processed sequentially, starting with the first consecutive decision group;

[0197] For the currently active continuous decision group, the input features are forward-propagated through a portion of the model layers of the currently active continuous decision group to obtain an enhanced multimodal feature representation;

[0198] The enhanced multimodal feature representation is input into the action prediction module of the currently activated continuous decision group. The action prediction module decodes and maps the enhanced multimodal feature representation to generate an action prediction that includes pose and gripper control parameters as the current action prediction.

[0199] Add the generated action predictions to the cumulative prediction sequence;

[0200] The enhanced multimodal features are used as input features for the next successive decision group to be activated.

[0201] In one embodiment, the prediction comparison module 40 is specifically used for:

[0202] Retrieve the most recently generated action prediction from the cumulative prediction sequence as the current action prediction;

[0203] Determine whether other action predictions have been stored in the cumulative prediction sequence before the current action prediction;

[0204] If other action predictions have been stored in the cumulative prediction sequence before the current action prediction, then the action prediction stored immediately before the current action prediction is retrieved as the previous action prediction.

[0205] Determine the differences between the current action prediction and the previous action prediction across each control dimension;

[0206] Obtain the action consistency threshold set for the current decision group;

[0207] Compare the difference value with the action consistency threshold;

[0208] If the difference value is less than or equal to the action consistency threshold, the early departure condition is determined to be met; if the difference value is greater than the action consistency threshold, the early departure condition is determined not to be met.

[0209] If no other action predictions are stored in the cumulative prediction sequence before the current action prediction, then the early termination condition is not met.

[0210] In one embodiment, the early departure control module 50 is specifically used for:

[0211] If the early termination condition is met, a termination activation signal is generated to prevent the transmission of input feature representations to subsequent successive decision groups;

[0212] Retrieve the current action prediction corresponding to the condition of early termination from the cumulative prediction sequence;

[0213] The retrieved current action prediction is formatted to generate control instructions that conform to the actuator interface specification as the final action instructions;

[0214] The final action command is sent to the action execution interface.

[0215] In one embodiment, the result output module 60 is specifically used for:

[0216] If the early exit condition is not met, then determine the position index of the current continuous decision group in the decision group sequence;

[0217] The next successive decision group to be activated is determined based on the location index;

[0218] The current enhanced multimodal feature representation is passed to the next successive decision group to be activated as the input feature representation;

[0219] Repeat the steps of activating the continuous decision group, generating the current action prediction, comparing and determining whether the early departure condition is met;

[0220] If it is determined that all consecutive decision groups have been activated, then retrieve the action prediction generated by the last consecutive decision group from the cumulative prediction sequence.

[0221] The last retrieved motion prediction is formatted to generate a control instruction that conforms to the actuator interface specification as the final motion instruction.

[0222] The final action command is sent to the action execution interface.

[0223] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When executed by the processor, the computer program implements the functions or steps of a server-side action generation method based on continuous decision groups.

[0224] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides decision-making and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements client-side functions or steps of an action generation method based on continuous decision groups.

[0225] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0226] It receives multimodal input data, including visual input and language commands;

[0227] The multimodal input data is input into a multimodal language model containing multiple consecutive decision groups, each of which includes a partial model layer and an action prediction module;

[0228] The plurality of consecutive decision groups are activated sequentially. Each time a consecutive decision group is activated, the current action prediction is generated through the action prediction module of the consecutive decision group.

[0229] The current action prediction is compared with the previous action prediction, and the early departure condition is determined based on the comparison result.

[0230] If the early exit condition is met, the activation of subsequent continuous decision groups is terminated, and the current action prediction is output as the final action instruction.

[0231] If the early exit condition is not met, the next continuous decision group will be activated and the comparison and judgment steps will be repeated until all continuous decision groups are activated. The action prediction generated by the last continuous decision group will be output as the final action instruction.

[0232] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0233] It receives multimodal input data, including visual input and language commands;

[0234] The multimodal input data is input into a multimodal language model containing multiple consecutive decision groups, each of which includes a partial model layer and an action prediction module;

[0235] The plurality of consecutive decision groups are activated sequentially. Each time a consecutive decision group is activated, the current action prediction is generated through the action prediction module of the consecutive decision group.

[0236] The current action prediction is compared with the previous action prediction, and the early departure condition is determined based on the comparison result.

[0237] If the early exit condition is met, the activation of subsequent continuous decision groups is terminated, and the current action prediction is output as the final action instruction.

[0238] If the early exit condition is not met, the next continuous decision group will be activated and the comparison and judgment steps will be repeated until all continuous decision groups are activated. The action prediction generated by the last continuous decision group will be output as the final action instruction.

[0239] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0240] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0241] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0242] It should be noted that if any software tools or components not belonging to this company appear in the embodiments of this application, they are merely illustrative examples and do not represent actual use. The embodiments described above are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for action generation based on sequential decision sets, characterized in that, The method comprises the following steps: receiving multi-modal input data comprising visual input and language instructions; inputting the multi-modal input data into a multi-modal language model comprising a plurality of sequential decision groups, each of the sequential decision groups comprising a partial model layer and an action prediction module; activating the plurality of sequential decision groups in sequence, and generating a current action prediction through the action prediction module of each of the sequential decision groups as the sequential decision group is activated; comparing the current action prediction with a previous action prediction, and determining whether an early termination condition is met based on the comparison result; if the early termination condition is met, terminating the activation of subsequent sequential decision groups, and outputting the current action prediction as a final action instruction; if the early termination condition is not met, continuing to activate a next sequential decision group and repeating the comparing and determining steps until all the sequential decision groups are activated, and outputting an action prediction generated by a last sequential decision group as the final action instruction.

2. The method of claim 1, wherein, receiving multi-modal input data comprising visual input and language instructions, comprising: capturing a physical environment through an image sensing device to obtain raw visual data; receiving voice or text input through a user interaction interface to obtain raw language instructions; performing geometric transformation and photometric calibration processing on the raw visual data to generate standardized visual data; performing speech recognition or text parsing processing on the raw language instructions to generate structured language instructions; synchronizing and packaging the standardized visual data and the structured language instructions into time-aligned multi-modal input data.

3. The method of claim 1, wherein, inputting the multi-modal input data into a multi-modal language model comprising a plurality of sequential decision groups, each of the sequential decision groups comprising a partial model layer and an action prediction module, comprising: loading a pre-trained modal language model comprising a visual encoder, a language encoder, and a feature fusion layer; dividing a deep network layer of the modal language model into a plurality of sequentially connected segments, each segment constituting a partial model layer of a sequential decision group; adding a lightweight prediction network at an output end of each sequential decision group to form an action prediction module of the sequential decision group; inputting standardized visual data in the multi-modal input data into the visual encoder to obtain a visual feature vector; inputting structured language instructions in the multi-modal input data into the language encoder to obtain a language feature vector; inputting the visual feature vector and the language feature vector into the feature fusion layer to generate an initial multi-modal feature representation; inputting the initial multi-modal feature representation into the partial model layer of a first sequential decision group to start sequentially processing the initial multi-modal feature representation through the plurality of sequential decision groups to generate an action prediction.

4. The method of claim 3, wherein, activating the plurality of sequential decision groups in sequence, and generating a current action prediction through the action prediction module of each of the sequential decision groups as the sequential decision group is activated, comprising: sequentially processing the initial multi-modal feature representation starting from the first sequential decision group; for a currently activated sequential decision group, performing forward propagation processing on input features through the partial model layer of the currently activated sequential decision group to obtain an enhanced multi-modal feature representation; inputting the enhanced multi-modal feature representation into an action prediction module of a currently activated continuous decision group, decoding and mapping the enhanced multi-modal feature representation by the action prediction module to generate an action prediction containing a pose and gripper control parameters as a current action prediction; adding the generated action prediction into a cumulative prediction sequence; inputting the enhanced multi-modal feature representation as an input feature of a next continuous decision group to be activated.

5. The method of claim 1, wherein, comparing the current action prediction with a previous action prediction, and determining whether an early termination condition is satisfied based on a comparison result, including: retrieving a latest generated action prediction from the cumulative prediction sequence as the current action prediction; determining whether other action predictions have been stored in the cumulative prediction sequence before the current action prediction; if other action predictions have been stored in the cumulative prediction sequence before the current action prediction, retrieving an action prediction stored immediately before the current action prediction as the previous action prediction; determining a difference value of the current action prediction and the previous action prediction in each control dimension; obtaining an action consistency threshold value set for the current decision group; comparing the difference value with the action consistency threshold value; if the difference value is less than or equal to the action consistency threshold value, determining that the early termination condition is satisfied; if the difference value is greater than the action consistency threshold value, determining that the early termination condition is not satisfied; if no other action prediction is stored in the cumulative prediction sequence before the current action prediction, determining that the early termination condition is not satisfied.

6. The method of claim 1, wherein, if the early termination condition is satisfied, terminating activation of subsequent continuous decision groups, and outputting the current action prediction as a final action instruction, including: if the early termination condition is satisfied, generating a termination activation signal to prevent sending of the input feature representation to the subsequent continuous decision group; retrieving the current action prediction corresponding to when the early termination condition is satisfied from the cumulative prediction sequence; formatting the retrieved current action prediction to generate a control instruction conforming to an actuator interface specification as the final action instruction; sending the final action instruction to an action execution interface.

7. The method of claim 1, wherein, if the early termination condition is not satisfied, continuing to activate the next continuous decision group and repeating the comparison and determination steps until all continuous decision groups are activated, and outputting an action prediction generated by a last continuous decision group as the final action instruction, including: if the early termination condition is not satisfied, determining a position index of the current continuous decision group in a decision group sequence; determining the next continuous decision group to be activated according to the position index; passing the current enhanced multi-modal feature representation to the next continuous decision group to be activated as the input feature representation; repeating the steps of sequentially activating the continuous decision groups, generating the current action prediction, comparing, and determining whether the early termination condition is satisfied; if it is determined that all continuous decision groups have been activated, retrieving an action prediction generated by the last continuous decision group from the cumulative prediction sequence; formatting the retrieved last action prediction to generate a control instruction conforming to an actuator interface specification as the final action instruction; sending the final action instruction to an action execution interface.

8. An action generating apparatus based on a continuous decision group, characterized by comprising: the continuous decision group based action generation apparatus includes: An input analysis module configured to receive multi-modal input data comprising visual input and language instructions; A multi-modal encoding module configured to input the multi-modal input data into a multi-modal language model comprising a plurality of sequential decision groups, each of the sequential decision groups comprising a partial model layer and an action prediction module; A decision group activation module configured to sequentially activate the plurality of sequential decision groups, and upon activation of each of the sequential decision groups, generate a current action prediction via the action prediction module of the sequential decision group; A prediction comparison module configured to compare the current action prediction with a previous action prediction, and determine whether an early exit condition is satisfied based on the comparison result; An early exit control module configured to, if the early exit condition is satisfied, terminate the activation of subsequent sequential decision groups, and output the current action prediction as a final action instruction; A result output module configured to, if the early exit condition is not satisfied, continue to activate a next sequential decision group and repeat the comparing and determining steps until all of the sequential decision groups are activated, and output an action prediction generated by a last sequential decision group as the final action instruction.

9. A computer device, comprising: The computer device comprises a memory, a processor, and a sequential decision group based action generation program stored in the memory and executable on the processor, and the sequential decision group based action generation program, when executed by the processor, implements the steps of the sequential decision group based action generation method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a sequential decision group based action generation program, and the sequential decision group based action generation program, when executed by the processor, implements the steps of the sequential decision group based action generation method according to any one of claims 1-7.

Images