Task processing method and device based on expert sub-path, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, can be applied to business scenes of financial science and technology, medical health and the like, and discloses a task processing method, device, equipment and medium based on an expert sub-path, comprising: setting a shared attention layer, an understanding expert sub-path, a control expert sub-path and a router module in a neural network, the method comprises the following steps: constructing a robot action trajectory data set, a visual text pair data set and a structured language template set, training based on the robot action trajectory data set and the structured language template set to obtain an intermediate training model, and training based on the intermediate training model and the visual text pair data set to generate a unified training model; and when target task input is received, the router module selects a corresponding expert sub-path, and a result is output through the unified training model. According to the method, staged training is combined with an expert mixing mechanism, catastrophic forgetting is avoided, semantic understanding and robot control keep stable coexistence in a unified model, and the accuracy of task execution is improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a task processing method, apparatus, device, and storage medium based on expert subpaths. Background Technology

[0002] In the fintech business, existing multimodal large models are mainly used for tasks such as customer risk analysis, compliance review, and multimodal interactive customer service. These models have made some progress in visual-language understanding tasks, and can handle operations such as document image recognition and text comparison. However, when financial business scenarios require the integration of robots or automated systems, such as automated document review equipment or process execution robots, existing models often perform poorly in performing control tasks. Especially after retraining for financial control scenarios, models are prone to "catastrophic forgetting," losing their original image-text semantic alignment capabilities, resulting in a significant decline in performance in areas such as customer document interpretation, compliance text review, or dialogue interaction, causing an imbalance in multimodal capabilities. In addition, control tasks and semantic understanding tasks interfere with each other in the parameter space, often causing models to be unable to maintain a high level of compliance language understanding or stable control performance in financial tasks.

[0003] In the healthcare field, existing large-scale multimodal models are commonly used for medical image interpretation, clinical record generation, and assisted diagnosis and answering. While these models show potential in image-language matching and medical dialogue generation, they exhibit significant shortcomings when integrated with medical robots. When used for medical robot control, such as rehabilitation training robots or surgical assistive arms, they easily lose their previous image-text understanding capabilities after retraining, failing to execute language-driven command interpretation and dialogue in complex scenarios. This "catastrophic forgetting" phenomenon makes it difficult for models to simultaneously handle medical image understanding and precise motion control. In terms of training strategies, existing methods often lack reasonable planning for the sequence of phased tasks, resulting in models that cannot stably master low-level motion control while struggling to maintain high-level semantic understanding, thus hindering the construction of a unified and reliable multimodal intelligent framework in healthcare scenarios. Summary of the Invention

[0004] The main objective of this invention is to provide a task processing method, apparatus, device, and storage medium based on expert subpaths, aiming to solve the technical problem that existing joint modeling of multimodal understanding and robot control is prone to catastrophic forgetting, and loses image and text understanding capabilities after gaining control capabilities.

[0005] To achieve the above objectives, the present invention provides a task processing method based on expert subpaths, comprising:

[0006] In the neural network, a shared attention layer, an expert sub-path understanding module, an expert sub-path control module, and a router module are set up.

[0007] Construct a robot motion trajectory dataset, a visual text pair dataset, and a structured language template set;

[0008] The first stage of training is performed based on the robot motion trajectory dataset and the structured language template set. The router module selects the control expert sub-path according to the task prompts to update the parameters and generate an intermediate training model.

[0009] The second stage of training is performed on the dataset based on the intermediate training model and the visual text. The router module selects the understanding expert sub-path according to the task prompts to update the parameters and generate a unified training model.

[0010] The router module receives the target task input, selects the understanding expert sub-path or the control expert sub-path based on the target task prompt in the target task input, and processes the target task input through the unified training model to generate the task processing result.

[0011] Furthermore, to achieve the above objectives, the present invention provides a task processing apparatus based on expert subpaths, comprising:

[0012] The expert hybrid mechanism module is used to set up shared attention layers, understand expert sub-paths, control expert sub-paths, and the router module in the neural network;

[0013] The training data construction module is used to build robot motion trajectory datasets, visual text pair datasets, and structured language template sets;

[0014] The control training module is used to perform the first stage of training based on the robot motion trajectory dataset and the structured language template set. The router module selects the control expert sub-path according to the task prompts to update the parameters and generate an intermediate training model.

[0015] The understanding training module is used to perform a second-stage training on the dataset based on the intermediate training model and the visual text. The router module selects the understanding expert sub-path according to the task prompts to update the parameters and generate a unified training model.

[0016] The task reasoning module is used to receive the target task input, and the router module selects the understanding expert sub-path or the control expert sub-path according to the target task prompt in the target task input, and processes the target task input through the unified training model to generate the task processing result.

[0017] Furthermore, to achieve the above objectives, the present invention also provides a computer device, the computer device including a memory, a processor, and an expert subpath-based task processing program stored in the memory and executable on the processor, wherein when the expert subpath-based task processing program is executed by the processor, it implements the steps of the expert subpath-based task processing method as described above.

[0018] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing an expert subpath-based task processing program, wherein the expert subpath-based task processing program, when executed by a processor, implements the steps of the expert subpath-based task processing method as described above.

[0019] Beneficial Effects: This invention relates to the field of artificial intelligence technology and can be applied to business scenarios such as fintech and healthcare. It discloses a task processing method, apparatus, device, and medium based on expert sub-paths, comprising: setting a shared attention layer, understanding expert sub-paths, controlling expert sub-paths, and a router module in a neural network; constructing a robot motion trajectory dataset, a visual text pair dataset, and a structured language template set; performing a first-stage training based on the robot motion trajectory dataset and the structured language template set; having the router module select the controlling expert sub-path based on task prompts to update parameters and generate an intermediate training model; performing a second-stage training based on the intermediate training model and the visual text pair dataset; having the router module select the understanding expert sub-path based on task prompts to update parameters and generate a unified training model; receiving target task input; having the router module select the understanding expert sub-path or the controlling expert sub-path based on the target task prompts in the target task input; and processing the target task input through the unified training model to generate task processing results. This invention introduces a shared attention layer, expert sub-path, and router module into a unified neural network, and adopts a phased training approach to enable the model to first acquire robot control capabilities and then enhance multimodal understanding capabilities. This avoids the problem of catastrophic forgetting during training and achieves the simultaneous maintenance of image and text understanding capabilities and robot control capabilities in a unified model, ensuring the stability and diversity of task execution. Attached Figure Description

[0020] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:

[0021] Figure 1 This is a schematic diagram of an application environment for a task processing method based on expert subpaths according to an embodiment of the present invention;

[0022] Figure 2 This is a flowchart illustrating an embodiment of the task processing method based on expert subpaths according to the present invention.

[0023] Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the task processing device based on expert subpaths of the present invention;

[0024] Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0025] Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0026] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0027] The expert subpath-based task processing method provided in this invention can be applied to, for example... Figure 1 In this application environment, the user terminal communicates with the server via a network. The server can use the user terminal to set up a shared attention layer, an understanding expert sub-path, a control expert sub-path, and a router module in the neural network. This allows for the construction of a robot motion trajectory dataset, a visual text pair dataset, and a structured language template set. The first stage of training is performed based on the robot motion trajectory dataset and the structured language template set. The router module then selects the control expert sub-path based on task prompts to update parameters and generate an intermediate training model. The second stage of training is performed based on the intermediate training model and the visual text pair dataset. The router module then selects the understanding expert sub-path based on task prompts to update parameters and generate a unified training model. Finally, the server receives the target task input and selects either the understanding expert sub-path or the control expert sub-path based on the target task prompts in the input. The unified training model then processes the target task input to generate the task processing result. This invention introduces a shared attention layer, expert sub-path, and router module into a unified neural network. By employing a phased training approach, the model first acquires robot control capabilities and then enhances its multimodal understanding capabilities. This avoids catastrophic forgetting during training and achieves simultaneous maintenance of image and text understanding capabilities and robot control capabilities within the unified model, ensuring the stability and versatility of task execution. The user end can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server end can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0028] Please see Figure 2 , Figure 2This is a flowchart illustrating an embodiment of the expert sub-path-based task processing method provided by the present invention. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0029] like Figure 2 As shown, the task processing method based on expert subpaths proposed in this invention includes the following steps:

[0030] S10, set up a shared attention layer, understand expert sub-paths, control expert sub-paths and router modules in the neural network;

[0031] In this embodiment, the introduction of a shared attention layer in the neural network is derived from research on cross-modal information fusion. This layer is typically implemented using self-attention or multi-head attention mechanisms, and can capture semantic dependencies between different modalities when the input consists of image and text features. Specifically, it involves spatial encoding of image features, serializing and embedding text features, and then calculating the interaction value between the two using a weighted correlation matrix, outputting a weighted fused cross-modal representation. In medical image analysis, for example, CT scans and descriptive text can be jointly encoded using a shared attention layer; similarly, in financial business report processing, table images and natural language descriptions can be aligned using this layer.

[0032] The expert sub-path concept originates from semantic parsing and natural language understanding. Its primary function is to handle reasoning and question-answering tasks related to image-text interaction. Its implementation typically involves a multi-layered Transformer network, containing a semantic alignment module and a relational reasoning unit, capable of outputting semantic reasoning results when the input is image features and text instructions. For example, in a medical image annotation task, given the input question "Does the image contain abnormal nodules?", this path can parse the region features in the image and combine them with the text question to generate the result; in financial document retrieval, given the input question "What is the total amount of assets in the report?", this path outputs the answer by parsing the correspondence between numerical values ​​and text.

[0033] The control expert sub-path is designed based on research findings in robot motion control modeling. This path primarily processes motion trajectories and state sequences. Its implementation typically involves recurrent neural networks or temporal Transformers to parse the state-motion mapping of time series and output corresponding control signals or predicted trajectories. In medical and healthcare assistive robots, this path can process motion data during patient rehabilitation training and output motion guidance; in financial office automation scenarios, this path can process sequences of virtual assistant commands and output execution actions that conform to operational logic.

[0034] The router module was introduced based on path selection research in hybrid expert networks. Its core function is to dynamically select between the understanding expert sub-path and the control expert sub-path based on task prompts. Its implementation includes a prompt parser, a classifier, and a path activation signal generator. The parser first encodes the prompts using word vectors and extracts keywords. The classifier categorizes the prompts according to the task categories defined during training. The path activation signal generator outputs the corresponding path selection instructions. In medical text and image interaction applications, when the prompt is "Explain the location of the tumor in the image," the router activates the understanding expert sub-path; when the prompt is "Control the robotic arm to complete the sampling action," it activates the control expert sub-path. In financial voice assistants, when the prompt is "Please analyze the income situation in the report," the router selects the understanding expert sub-path; when the prompt is "Generate a chart and export it," the router selects the control expert sub-path.

[0035] The four modules form an expert hybrid mechanism. The specific operation logic is that the shared attention layer is responsible for unifying the information base, the understanding expert sub-path and the control expert sub-path are responsible for task execution, and the router module provides a path allocation mechanism to ensure that the model can maintain both the ability to understand text and images and the ability to control output.

[0036] The implementation of the shared attention layer can be adjusted in different hardware environments. For example, on medical mobile devices with limited computing resources, low-rank decomposition and attention sparsity mechanisms can be used to reduce the amount of computation; in financial data center environments, multi-head attention parallelization can be used to accelerate cross-modal feature computation.

[0037] Understanding expert sub-paths allows for optimization based on different application needs. For example, in medical imaging tasks, adding a convolutional feature extraction layer can enhance the capture of local image features; in financial scenarios, adding a numerical parsing sub-module can strengthen the correspondence between table values ​​and text descriptions.

[0038] The control expert subpath can be modified for different motion control tasks. For example, in rehabilitation robots, electromyographic signal features can be introduced as additional inputs to enhance the stability of motion generation; in financial automation, the generated actions can be constrained by introducing rule bases and historical operation sequences.

[0039] The router module can be extended through different prompt parsing strategies. In medical applications, a prompt classifier based on domain knowledge graphs is used to improve the accuracy of task parsing, while in the financial field, an intent recognition model based on statistical learning is used to improve the ability to parse complex instructions.

[0040] Example description: In the healthcare business field, when patients use assistive robots in a home care environment, they can trigger the understanding expert sub-path by voice inputting "Check if there are nodules in the CT images I uploaded". The shared attention layer completes the fusion of image and text description and generates semantic understanding results. When inputting "Please control the robotic arm to deliver the medicine", the router module automatically switches to the control expert sub-path, enabling the robot to successfully complete the physical action.

[0041] In the fintech business, when staff process annual reports, they can input "Please analyze the table in the image and explain the asset and liability situation." The model will call the understanding expert sub-path to parse the table image and text descriptions, and generate a structured explanation. When inputting "Please generate an asset trend chart and export the file," the router module will activate the control expert sub-path, and the model will generate corresponding operation instructions and execute them in the system, realizing integrated support for data analysis and operation automation.

[0042] This embodiment achieves unified modeling of multimodal semantic understanding and robot control capabilities by setting up a shared attention layer, understanding expert sub-paths, control expert sub-paths, and a router module in the neural network. This design avoids the problem of mutual interference between semantic capabilities and control capabilities during previous training processes, and enables dynamic switching between multimodal tasks and action control tasks, thereby maintaining the integrity of cross-modal understanding capabilities and the stability of control task execution.

[0043] S20, construct a robot motion trajectory dataset, a visual text pair dataset, and a set of structured language templates;

[0044] In this embodiment, the robot motion trajectory dataset is designed for temporal modeling of states and actions. Data items include timestamps, environmental observations, robot joint or end-effector poses, control variables, execution feedback, and task context. Environmental observations can be provided by a combination of 3D position sensors, depth cameras, force / torque sensors, haptic patches, and inertial units. Joint or end-effector poses are expressed using a unified coordinate system and calibration matrix. Control variables cover velocity, torque, current, or end-effector velocity commands. Execution feedback records whether execution was successful, the error norm, and exception codes. To ensure the recognizability of subsequent learning, the trajectory is organized into segments based on state-action sequences. Each segment has clear start and end conditions, object identifiers, and scene identifiers, and retains metadata such as sampling frequency, device number, and calibration version.

[0045] The visual-text dataset focuses on the alignment between images and text. The image side provides the original image, region bounding boxes or segmentation masks, and lens parameters, while the text side provides natural language descriptions, question-and-answer pairs, instruction phrases, and entity vocabulary mappings, with additional object-level labels and spatial relationship labels as needed. A one-to-one or one-to-many pairing index is established between images and text, retaining scene IDs, object IDs, and timestamps to ensure temporal alignment with trajectory data. To improve cross-domain generalization, samples with varying lighting, viewpoints, materials, and backgrounds can be introduced and stored in a unified color space and resolution.

[0046] The structured language template set is used to anchor finite and stable task expressions into a rule-based language form. Template elements include action predicates, operation objects, target locations or attributes, constraints, and evaluation criteria; parameter bits use placeholders and are populated from an object library, location library, or value library during instantiation. Templates are represented in a machine-readable hierarchical structure, such as tree or graphical semantic slots, which can both generate task prompts and be bound to trajectory segments during training as supervision signals or routing criteria. Templates can be inherited and combined to form scalable subsets of expressions covering application areas such as handling, grasping, alignment, observation, counting, comparison, and interpretation.

[0047] A mapping relationship is established between the three types of data. Trajectory segments are aligned to corresponding image frames or frame groups using object IDs, scene IDs, and timestamps; template instances and trajectory segments are bound to each other using task type and object / target parameters. To avoid ambiguity, only a single main template label is retained for each segment, and auxiliary labels are allowed to record fine-grained constraints (such as "slow placement" and "avoid occlusion"). The complete data structure consists of two layers: data content and metadata. The content layer stores continuous trajectories, images, and text; the metadata layer stores calibration, sampling strategies, label confidence levels, quality review records, and access permissions.

[0048] Data acquisition and synchronization employ a unified time base strategy across multiple sensors. Hardware-triggered synchronization can be used to bind camera exposure, force / torque sampling, and joint readings to the same pulse; alternatively, software time synchronization can be used to correct timestamps from various sources to a unified time base and align trajectories and image sequences through dynamic time warping. During the trajectory acquisition phase, motion safety thresholds and soft joint limits are set, and the sampling frequency is automatically adjusted based on the task's dynamics, using a higher frequency for fast interactive tasks and a lower frequency for static positioning tasks to reduce redundancy. Trajectory preprocessing includes denoising, missing value interpolation, coordinate unification, and velocity / acceleration estimation; Kalman or Savitzky-Golay filtering is used to improve smoothness when necessary. Object pose is transformed to the world coordinate system using an extrinsic calibration matrix to ensure consistency across devices.

[0049] In the visual text pair construction process, images undergo distortion correction, color homogenization, and resolution normalization; object regions are generated using detection or segmentation algorithms and manually reviewed; text is processed using word segmentation, lexical reconstruction, and named entity recognition to extract objects, attributes, and relationships; for question-and-answer or instruction samples, a dictionary of question types and expected answer types is established to ensure type coverage and label consistency. Images and text are paired using shared key values ​​(scene ID, object ID) and nearest-neighbor timestamps. In the case of one-to-many pairs, the primary pair is selected using attention weights or heuristic rules, while alternative pairs are retained for data augmentation during training.

[0050] The construction of the structured language template set first defines the action predicate library and the relation predicate library, and then defines the object and target parameter spaces. Template combinations can be generated through rule scripts, or high-confidence templates can be manually written on a small scale and then parameterized and expanded. Template validation includes syntactic validity checks, parameter domain validity checks, and trajectory / image instantiation checks; automated unit testing verifies that the template can be instantiated into valid prompts and can be mapped back to trajectory statistical features (e.g., within the error threshold between the trajectory endpoint and the target position).

[0051] Mapping and annotation employ a two-stage process. The first stage uses an automatic aligner for coarse matching: aligning trajectory segments with image frames and template candidates based on timestamps, object re-identification, and action similarity. The second stage uses a human-machine collaborative review interface for refinement. The interface displays the segment trajectory, keyframes, and template parameters, allowing annotators to select the most suitable template instance and set the confidence level. After annotation, a consistency check is performed: verifying consistency between the template target and the trajectory's final state, object ID consistency, and spatial relationships consistency with image geometry.

[0052] Data storage and version management employ column-oriented binary format to store high-frequency time series (such as Parquet or RecordIO) and object-oriented storage to store images / masks. Text and templates are stored in JSON or Protobuf and the schema version is recorded. A manifest file is used to record the composition and checksum hash of each sample. A segmented view is prepared for subsequent training and inference, dividing the training, validation, and testing phases, and stratified sampling of trajectory segments according to task type and difficulty is provided. A sampling strategy file is provided to reproduce experiments. Privacy and compliance labels are provided to control the scope of access.

[0053] Quality control is implemented throughout the entire process. During data acquisition, sensor health and drift are monitored; during preprocessing, signal-to-noise ratio and frame drop rate are monitored; during annotation, consistency and cross-review pass rate are monitored; and during dataset release, sampling verification and model reproducibility experiments are conducted. In resource-constrained environments, frame rate downsampling and keyframe selection can be used to reduce storage and computation; in high-performance environments, complete frames can be retained and multi-view fusion can be enabled to improve the accuracy of spatial relationship annotation.

[0054] Example Description: In the healthcare field, trajectory data containing rehabilitation training movements is constructed, recording joint angles, end-effector trajectories, and force feedback. Simultaneously, images of the training scenario are acquired and text descriptions are written, such as "Raise your left arm to shoulder level and hold for three seconds." This is instantiated using a template as "Raise <limb> to <height> and hold for <duration>", and bound to the corresponding trajectory segment. During training and evaluation, the semantic description and the final state of the movement are strictly aligned, facilitating stable switching between semantic question answering and movement guidance, and preventing the model from losing its understanding of descriptive language after learning control.

[0055] In the fintech business, we construct operation trajectory data for office automation, recording the state of interface elements, mouse and keyboard operation sequences, and execution feedback. We simultaneously capture interface images and generate text descriptions, such as "Select the asset column in the balance sheet and export it as a CSV file." We instantiate templates as "Select <field> in and export it as <format>" and bind them to the corresponding operation fragments. This data organization method enables the model to achieve a stable mapping between understanding interface semantics and executing specific operations, reducing mutual interference between different tasks and improving the consistency and transferability of cross-modal understanding and control.

[0056] This embodiment constructs a unified dataset of robot motion trajectories, a dataset of visual text pairs, and a set of structured language templates, and establishes a verifiable three-way mapping relationship. During the training phase, it can separate the data sources for control learning and semantic learning, reducing mutual interference in parameter updates. Template instances are bound to trajectory segments, ensuring a one-to-one correspondence between language expressions and action results, which helps maintain semantic alignment capabilities during phased training and inference routing. Precise temporal and object alignment improves the reliability of cross-modal associations, providing high-quality supervision and routing basis for subsequent alignment calculations and expert path selection in the shared attention layer, thereby reducing performance degradation caused by catastrophic forgetting and task interference.

[0057] S30, the first stage of training is performed based on the robot motion trajectory dataset and the structured language template set. The router module selects the control expert sub-path according to the task prompts to update the parameters and generate an intermediate training model.

[0058] In this embodiment, the first stage of training uses a robot motion trajectory dataset and a structured language template set as joint inputs. The goal is to enable the control expert sub-path to learn the mapping from state to action without semantic interference. The robot motion trajectory dataset provides state-action sequences and execution feedback. The state includes multi-source observations and geometric quantities, the action includes joint commands or end-effector commands, and the execution feedback includes success and error metrics. The structured language template set has already established one-to-one or one-to-many mappings with trajectory segments during the acquisition phase. Template instances carry task type, operation object, target position, and constraint information. During training, a state-action sequence and its template instance are extracted from the mapping pair. Simultaneously, task prompts are derived from the template instances. The task prompts are encoded as short vectors and used as input to the router module. The router module calculates path selection weights for the task prompt vectors through several affine layers and nonlinear units. It generates binary or approximately binary control path activation signals by adjusting temperature and thresholds, and then sends the activation signals to the gate control unit, ensuring that only the control expert sub-path is in an active state. The shared attention layer provides the basic representation for cross-modal alignment. Parameters are frozen by default to isolate language perturbations to control learning, and can be fine-tuned at a small learning rate to absorb alignment information. The control expert sub-path receives the output and state features of the shared attention layer, outputting the next action or action distribution. Alignment time steps are replayed through teacher demonstration, using trajectory prediction loss as the primary supervision. Squared error or quantile regression losses are commonly used to cover deterministic and uncertain scenarios. To improve executability, smoothing regularization and safety constraint regularization can be added to limit acceleration spikes and out-of-bounds actions. During backpropagation, only the internal parameters of the control expert sub-path and a small number of layers that allow fine-tuning are unfrozen. The router module only participates in forward routing and is not updated, or is stably fine-tuned at an extremely low learning rate. Gradient masking ensures that the understanding expert sub-path and its proprietary parameters are not updated at all. The optimizer can employ adaptive moment estimation combined with cosine annealing or piecewise learning rates, enabling gradient pruning and mixed precision for stability and speedup. Each training cycle is structured according to task type and difficulty, starting with single-target alignment and grasping, followed by placement and obstacle avoidance with path constraints. The schedule is automatically generated from template fields. After the early stopping criterion is met, the parameters are solidified. The latest control expert sub-path parameters and the public layer parameters that are allowed to be fine-tuned are written into a unified structure definition, forming an intermediate training model that can be independently loaded and reproduced. At the same time, the data list version, hyperparameters, and check hash are recorded to ensure persistent consistency.

[0059] The router module can be configured as a hard router, directly outputting binary activation signals to activate the control expert sub-path and completely shielding the understanding path, suitable for action-driven training scenarios. Alternatively, it can be configured as a soft router, allowing minimal weight leakage to non-target paths in the early stages of training to obtain stable gradients, followed by convergence to single-path activation via temperature annealing. Shared attention layers can be frozen to maximize parameter isolation, or a layer-by-layer learning rate strategy can be enabled, unfreezing only one or two layers near the input to adapt to receptive domain shifts, while keeping layers near the output frozen to maintain semantic alignment. The output head of the control expert sub-path can use Gaussian mixture to predict joint and end-effector velocities, or a hierarchical output strategy can be used to predict pose first and then torque; both are jointly optimized using trajectory prediction loss and constraint regularization. To reduce the impact of distribution shifts, perturbation replay can be introduced, adding noise to the state-action sequence before replaying, forcing the output to remain stable to small perturbations through consistency loss. For long sequence dependencies, truncated backpropagation with an attached hidden state reset plan can be used, inserting keyframes near the split point to mitigate gradient forgetting. If the scenario involves multiple robots or tools, device identification slots can be added to the task prompts. The router module learns the path selection based on device awareness, and the control expert adapts to different mechanics parameters through conditional normalization within the sub-path. When training resources are limited, keyframe sampling and segment alignment can be used, retaining high-frequency samples only at velocity changes and contact points. When resources are sufficient, full-frequency sampling can be enabled, and multi-card data parallelism can be used. During gradient aggregation, different task types are re-weighted to avoid underlearning of long-tail tasks. In the parameter solidification stage, an inference graph can be output, and unused understanding side branches can be pruned to obtain a lighter intermediate training model; alternatively, the complete graph can be retained, and only the weights can be frozen to facilitate the addition of semantic capabilities in the second stage.

[0060] Example Description: In the healthcare business, home rehabilitation equipment collects the state movement sequence and corresponding template instances of upper limb training, such as raising the arm to a specified angle and holding it for a certain number of seconds. During training, the template text is generated into task prompts and input into the router. The control expert sub-path learns the mapping from shoulder, elbow, and wrist angles and muscle force feedback to joint commands. The loss function simultaneously constrains the arrival angle error and posture smoothness. After parameter solidification, an intermediate training model is obtained, which can stably output safe and executable joint trajectories under different users and different damping settings.

[0061] In the fintech business, the interface automation system records the operation sequence and template instances of report processing, such as selecting and exporting specific fields of the balance sheet. During training, task prompts are generated based on the template text to activate the control expert sub-path. The inputs are interface state codes and state variables such as cursor position, and the outputs are click and input action sequences. The trajectory prediction loss is combined with the interface state transition consistency constraint. The resulting intermediate training model can maintain stable operation in multi-resolution and multi-topic interfaces, laying the foundation for subsequent training that incorporates question answering and document understanding capabilities.

[0062] This embodiment uses task prompts to drive the router module to select the expert control sub-path and restricts parameter updates to the control side. During training, the gradient flow is constrained to the subspace required for action generation, and language-related parameters and the understanding-side structure remain stable, thereby mitigating catastrophic forgetting and task interference. A shared attention layer provides a consistent cross-modal basis under freeze or fine-tuning strategies. Trajectory supervision and regularization jointly ensure smooth and executable actions, ultimately forming an intermediate training model that retains transferable perceptual alignment and possesses robust control capabilities, providing a reliable starting point for subsequent multimodal understanding training.

[0063] S40, based on the intermediate training model and the visual text, perform the second stage of training on the dataset, and the router module selects the understanding expert sub-path according to the task prompt to update the parameters and generate a unified training model;

[0064] In this embodiment, the second-stage training uses the intermediate training model as the initial weights and the visual-text pair dataset as the supervision source. The goal is to enable the understanding expert sub-path to achieve robust image-text semantic alignment capabilities without compromising the already acquired robot control capabilities. After loading the intermediate training model, batches are constructed, with each sample containing an image tensor and paired text. Simultaneously, task prompts are generated from the text or structured labels and encoded as prompt vectors. Images are processed through a visual encoding stack to obtain multi-scale features, and text is processed through a text encoding stack to obtain contextual representations. These two features interact across modally within a shared attention layer to form a basic alignment representation. The prompt vectors are then fed into the router module, which generates path selection weights through feedforward mapping and normalization. Temperature annealing is then used to obtain the understanding path activation signal. The gating unit activates only the understanding expert sub-path based on this signal, keeping the expert sub-path and control-related output heads frozen or fine-tuned with a low learning rate. The understanding expert sub-path comprises a cross-modal fusion block and a semantic projection head. The fusion block performs multi-head interactions on the shared attention layer output to improve the alignment strength between local regions and phrases. The semantic projection head maps the image global embedding and text global embedding to the same contrast space and normalizes them. The loss is composed of a joint form of global alignment and local alignment. Global alignment is mainly based on contrastive learning, using a temperature-calibrated similarity matrix to calculate bidirectional information gain loss between image and text and text and image. Local alignment is based on the similarity assignment of regions and phrases for matching and calculates cross-entropy or smooth contrast loss. To enhance language generation capabilities, autoregressive text reconstruction or restricted mask reconstruction can be added in the same batch. The masked words are predicted by a cross-modal conditional decoding head and constrained by negative log-likelihood. To avoid backflow disruption to the control capabilities of the intermediate training model, a gradient masking strategy is introduced, limiting backpropagation to updating only the understanding expert sub-path and a subset of the shared attention layer that allows fine-tuning. At the same time, a low-amplitude update or exponential moving average is used to stabilize the allocation strategy of the router module. Training scheduling employs parallel curriculum design and difficult example mining. Initially, single-object, single-sentence description samples are used to improve basic alignment, then multi-object relationship and long-sentence inference samples are gradually added. Difficult examples are automatically sampled using the flatness index of similarity distribution. The optimizer uses a separation of group learning rate and weight decay. A high learning rate is used for the understanding path to quickly absorb semantic knowledge, while a low learning rate is used for the shared attention layer to maintain cross-stage representation consistency. The control path has a learning rate of zero or extremely low. Consistency constraints are introduced, applying distance constraints between the shared attention layer output of the intermediate training model and the current model output on the same input to reduce representation drift. Entropy regularization and temperature annealing curves are applied to the router decision sequence to ensure that the inference stage tends towards single-path activation. After the early stopping criterion is met, parameter solidification is performed. The updated understanding expert sub-path parameters, the fine-tuned shared attention layer parameters, and the original control parameters are combined and written into a unified structure definition and weight file to form a unified training model. Training configuration, data version, and verification hash are recorded to support reproducible experiments and deployment consistency.

[0065] Hard routing training can be employed, allowing the router to directly output near-binary understanding path activation signals to maximize parameter isolation on the control side. Alternatively, soft routing can be used, initially allowing a small amount of weight leakage to non-target paths to stabilize alignment learning, and then gradually reducing to single-path learning through temperature control. Shared attention layers can be fully frozen to ensure cross-stage representation stability, or only one or two layers closest to the fusion interface can be unfrozen, adapting to visual and textual domain offsets with an extremely low learning rate. Global alignment loss can use symmetric bidirectional contrastive learning combined with queued negative sample memory to broaden the field of view. Local alignment can be based on candidate region proposals or grid features, and matching strategies can employ Hungarian assignment or threshold-based soft assignment. The text reconstruction branch can choose a joint sampling strategy of mask ratio and span length to improve robustness to long sentences and complex relationships. To control computational overhead, image resolution adaptive scaling and text truncation strategies can be used within batches, with word-level truncation performed while maintaining the integrity of semantic keywords. For cross-domain data, a data augmentation process that combines style enhancement and semantic preservation can be introduced. For example, color dithering and small geometric perturbations can be applied to image channels, while semantically equivalent rewriting can be applied to text channels. Simultaneously, consistency loss constraints can be used to ensure the outputs are close before and after the augmentation process. If the deployment environment is limited, an adapter weight injection method can be enabled, inserting low-rank adaptation modules only into the understanding expert sub-path and shared attention layer for training. The adapters can then be merged back or saved separately for on-demand loading. When faster convergence is needed, a teacher alignment strategy can be used. This involves generating cross-modal attention priors using a frozen intermediate training model or producing soft labels using an external visual-language encoder. Temperature-softened knowledge distillation loss is then used to jointly optimize the model with the student model, without changing the control-side weights. The training plan can employ alternating mixed sampling, maintaining the ratio of visual-text pairs to consistency samples used for stable alignment within a configurable range. The learning rate and temperature can be automatically adjusted based on semantic retrieval and image-text matching metrics on the validation set. During the solidification phase, a pruned unified training model can be exported, pruning low-contribution channels and redundant attention heads in the understanding path, or all weights can be retained to support multi-task inference needs.

[0066] Example Explanation: In the healthcare field, home rehabilitation aids need to understand on-screen instructional images and textual key points to correct user movements. The second phase of training constructs paired data of motion instructional images and key text, generates task prompts to guide the router to activate the understanding expert sub-path, and the model learns the correspondence between visual semantic alignment and key action terms. Parameter updates are restricted to the understanding side while maintaining stability on the control side. The resulting unified trained model, after recognizing incorrect postures, can provide device pose adjustment suggestions in conjunction with the control capabilities developed in the first phase.

[0067] In the fintech business, intelligent reconciliation and report preparation require cross-modal understanding of interface screenshots and field descriptions. The second phase of training collects paired data of interface images and field descriptions, generates task prompts to drive the router to select the understanding expert sub-path, and learns the correspondence between regions and text fields in the image through global and local alignment. This unified training model can not only recognize the meaning of report fields during inference but also cooperate with existing automated control capabilities to complete click and data entry operations, thereby reducing human-machine collaboration latency and improving accuracy.

[0068] This embodiment performs targeted semantic alignment learning on the dataset using visual text based on an intermediate trained model. A router is used to activate only the understanding expert sub-path based on task prompts, restricting parameter updates to a semantically relevant subspace. Control-side weights remain stable, significantly mitigating catastrophic forgetting and task interference. Joint supervision of global and local alignment improves the accuracy of image-text correspondence, controlled fine-tuning of the shared attention layer maintains cross-stage representation continuity, and parameter solidification integrates control and semantic understanding capabilities into a unified training model, providing a low-latency and highly reliable inference foundation for subsequent switching between multimodal understanding and robot manipulation.

[0069] S50, receive the target task input, the router module selects the understanding expert sub-path or the control expert sub-path according to the target task prompt in the target task input, and processes the target task input through the unified training model to generate the task processing result.

[0070] In this embodiment, the target task input refers to the multimodal data set processed by the model during the inference phase, including image data or video frames, text commands or dialogue turn text, optional environmental state codes or sensor readings, originating from an online interactive client or batch processing queue. An example is a workbench image accompanied by a natural language command. The target task prompt refers to a prompting text fragment embedded in the target task input, used to guide the routing process and define the expected output format. Its source can be a user-explicitly entered control command or understanding question, or a summary of historical dialogues, such as "Please grab the blue cup" or "What is the object on the left in this picture?". The router module refers to the subsystem that receives the target task prompt and optional context embeddings and provides path selection signals. Internally, it includes an embedding layer, a matching network, and a normalized voting unit, outputting two types of signals: understanding expert sub-path activation or control expert sub-path activation. These signals are expressed as one-dimensional or two-dimensional gating vectors, such as a control gate with a value close to one and an understanding gate with a value close to zero. The understanding expert sub-path refers to the computational branch for visual language understanding tasks, including a cross-modal fusion layer and a semantic projection head, used to generate classification results, text answers, or structured annotations. The control expert subpath refers to the computational branch oriented towards action generation and control trajectory prediction, containing a state fusion layer and a trajectory decoder, used to generate end-effector pose sequences or discrete action sequences. The unified training model refers to the overall network that has completed two-stage training and has its weights fixed, containing a shared attention layer, two types of expert subpaths, a router module, and an output head, responsible for handling dual-task capabilities during inference. The task processing result refers to the consumable output formed by the unified training model after inferring the target task input on the activated expert subpaths, manifested as text answers, semantic labels, action control instructions, or hybrid structures, such as answering "The left side of the table is a red block" or returning a Cartesian trajectory of grabbing and placing. The specific processing flow is as follows: The target task input is received and preprocessed for normalization and security filtering. Word segmentation, sub-word encoding, and positional encoding are performed on the target task prompt. The encoding results are input into the router module to generate a path selection signal. According to the signal, trainable gating mechanisms that activate or control expert sub-paths and freeze inactive branches are used. Subsequently, the image and text are embedded in a shared attention layer to complete one or more interactions. The interaction output is fed into the activated expert sub-path to complete task-specific forward computation. The output header generates the task processing result and performs confidence verification and formatting as necessary. To reduce latency, a cached index of prompts to paths can be maintained within the router module, with repeated questions directly hitting the cached signal. To improve robustness, a low-amplitude temperature correction can be added after path selection to avoid boundary samples triggering erroneous branches. To preserve dual-capability consistency, consistency scores can be calculated simultaneously for text vectors and action vectors at the output stage to filter out untrusted outputs.

[0071] Hard routing can be implemented, where the router module directly outputs binary gating signals. When the understanding gate is set to one, only the understanding expert sub-path is activated; when the control gate is set to one, only the control expert sub-path is activated. The intermediate representations of inactive branches are not included in the computation to reduce inference overhead. Soft routing can also be implemented, initially allocating a small percentage of weights to non-target branches using a soft maximum allocation method with higher temperatures. Subsequently, annealing is used to lower the temperature to near the hard routing state for smooth transitions in uncertain scenarios. The shared attention layer in the unified training model can be set to read-only to improve stability, or one or two layers near the output can be opened for online fine-tuning with a very low learning rate to adapt to in-domain lighting and corpus style shifts. The image data input to the target task can use multi-scale pyramid inference to improve the recognition of small targets, or block attention can be used to control memory usage in high-resolution scenes. The encoding of the target task prompts can use context aggregation, concatenating the intent summary of the historical dialogue to the beginning of the prompts to enhance routing decisions, or a separate task type identifier prefix can be introduced to improve recognition accuracy. The generation of task processing results can employ pointer-generated hybrid decoding on the understanding expert sub-path, first locating image regions through alignment distribution and then generating natural language answers; on the control expert sub-path, a two-stage decoding approach can be used, first generating keyframe poses in the task space and then obtaining dense trajectories through temporal difference interpolation, with each keyframe accompanied by capture opening / closing and velocity upper limit markers. To support edge deployment, the unified training model can undergo channel pruning and attention head sparsification, with the router module and output head residing in memory, and the understanding and control expert sub-paths loaded on demand. To support high concurrency, batch-based hybrid scheduling can be used on the server side, merging requests with the same route into a unified tensor parallel segment, reducing throughput loss caused by cross-branch switching. To improve security, a result verification process can be added, with the understanding output filtering phantom text through a knowledge consistency discriminator, and the control output quickly filtering out unexecutable trajectories through kinematic reachability and collision detection; when verification fails, it can fall back to an alternative branch or request additional prompts to clarify the intent. Parameters and thresholds can be adaptively adjusted. The path selection temperature is fine-tuned based on the misrouting rate of the nearest window. The output confidence threshold is dynamically set according to the task type. The image resolution is automatically scaled according to the scene complexity to balance latency and accuracy.

[0072] Example Description: In the healthcare field, home rehabilitation equipment receives target task input as images of key points on the patient's body surface and the text instruction "Please determine whether the current knee flexion and extension meet the standard." The router module activates the understanding expert sub-path based on the target task prompt, and the unified training model completes the alignment of bone points and terminology in the shared attention layer. The output task processing result is "Meets the standard," "Does not meet the standard," and an explanation of the insufficient angle. When the target task prompt changes to "Please switch the training mode to low resistance and start ten exercises," the router module activates the control expert sub-path, and the unified training model generates a resistance adjustment and beat control trajectory, which the equipment executes accordingly.

[0073] In the fintech business, the intelligent reporting assistant receives a screenshot of a report and a text instruction to identify and summarize the third quarter's R&D expenses as input. The router module selects the understanding expert sub-path, and the unified training model extracts the header and cell ranges and generates task processing results with values ​​and fields aligned. When the target task prompt is "Please fill in the previous summary result in the budget system and submit", the router module selects the control expert sub-path, and the unified training model generates a click and input sequence based on the distribution of interface elements. The task processing result is returned as a record of completed operations and a verification screenshot.

[0074] This embodiment activates branches in the router module by using target task prompts during the inference phase, and completes computation only along the activated understanding expert sub-path or control expert sub-path within the unified training model. This leverages the cross-modal fusion advantages of the shared attention layer while avoiding interference and additional latency caused by irrelevant branches, thus maintaining stable performance in both language understanding and action control tasks. The path selection temperature and caching mechanisms reduce misrouting and redundant computation, while output verification and formatting improve the executability and usability of the results, achieving a low-latency, low-interference online inference process overall.

[0075] This invention relates to the field of artificial intelligence technology and can be applied to business scenarios such as fintech and healthcare. It discloses a task processing method, apparatus, device, and medium based on expert sub-paths, comprising: setting a shared attention layer, an understanding expert sub-path, a control expert sub-path, and a router module in a neural network; constructing a robot motion trajectory dataset, a visual text pair dataset, and a structured language template set; performing a first-stage training based on the robot motion trajectory dataset and the structured language template set; having the router module select the control expert sub-path based on task prompts to update parameters and generate an intermediate training model; performing a second-stage training based on the intermediate training model and the visual text pair dataset; having the router module select the understanding expert sub-path based on task prompts to update parameters and generate a unified training model; receiving target task input; having the router module select the understanding expert sub-path or the control expert sub-path based on the target task prompts in the target task input; and processing the target task input through the unified training model to generate a task processing result. This invention introduces a shared attention layer, expert sub-path, and router module into a unified neural network, and adopts a phased training approach to enable the model to first acquire robot control capabilities and then enhance multimodal understanding capabilities. This avoids the problem of catastrophic forgetting during training and achieves the simultaneous maintenance of image and text understanding capabilities and robot control capabilities in a unified model, ensuring the stability and diversity of task execution.

[0076] In one embodiment, step S10 includes:

[0077] S101, divide the feedforward network part of the neural network to form two parallel sub-paths, including the understanding expert sub-path and the control expert sub-path;

[0078] S102, Design a router module and establish a connection between the router module and the activation entry of the understanding expert sub-path and the activation entry of the control expert sub-path;

[0079] S103, Configure a shared attention layer for extracting cross-modal basic features between the understanding expert sub-path and the control expert sub-path;

[0080] S104, establish parameter mapping channels with the task output layer at the output ends of the understanding expert sub-path and the control expert sub-path, respectively.

[0081] In this embodiment, the feedforward network is divided into parallel understanding expert sub-paths and control expert sub-paths, with the division occurring in the feedforward computation section following the shared attention layer. The shared attention layer is responsible for integrating cross-modal basic features, taking image tokens and text tokens as inputs and outputting a unified-dimensional contextual representation. In implementation, the spatial tokens generated by the visual encoder and the semantic tokens generated by the text encoder are aligned to a consistent number of channels before entering a multi-head attention unit to complete key-value query interaction. The residuals and normalization are kept consistent with the backbone for stable distribution. The shared attention layer can originate from transformer self-attention or a cross-attention structure; its core is to ensure mutual conditionalization between vision and language, enabling the subsequent two sub-paths to receive isomorphic tensors. For example, with a batch size of N, a sequence length of L, and a total number of channels D, the shared attention layer yields a fused representation of shape N x L x D, serving as the common entry point for the two sub-paths.

[0082] The expert sub-path targets semantic understanding and includes a feedforward expansion unit, a semantic projection unit, and an optional alignment regularization unit. The feedforward expansion unit expands the channels to r multiplied by D and then shrinks them back to D through gating activation to enhance non-linear combination capabilities. The semantic projection unit maps the fused representation to the space required for answer distribution, classification labels, or structured annotations. The alignment regularization unit maintains image-text consistency during training and remains read-only during inference to reduce jitter. In terms of source, this sub-path addresses the representational needs of multimodal question answering and image-text inference; example results are text answers or region labels.

[0083] The control expert sub-path focuses on motion generation and trajectory prediction, internally comprising a state fusion unit, a motion decoding unit, and an optional dynamics constraint unit. The state fusion unit concatenates or weights the fused representation with available environmental state codes. The motion decoding unit outputs pose sequences or discrete control commands. The dynamics constraint unit prunes velocity and acceleration ranges during the generation phase to prevent out-of-bounds errors. Source-wise, this sub-path fulfills the representation requirements of the robot's control domain; example results include end effector keyframes and gripper opening / closing sequences.

[0084] The router module generates a path selection signal based on target-related text prompts. The signal type can be either hard-gating or soft-assignment. In implementation, the target text is segmented and embedded, concatenated with a compressed vector of historical dialogue intent, and then processed through one or more feedforward mapping layers to obtain a two-dimensional selection vector. After normalization, this vector is compared with a threshold to obtain an activation flag. The activation entry point is located at the beginning of two sub-paths. Gated multiplication or masking is used to shield the forward computation of inactive branches, ensuring that inference only traverses the selected path. To reduce the risk of false selection, a lightweight calibration branch can be maintained within the router module. Samples with confidence levels close to the boundary undergo a one-time temperature smoothing, and the activation flag is issued only after calibration.

[0085] The shared attention layer is placed strictly upstream of both sub-paths to ensure that both tasks share the same cross-modal basic features. To improve robustness, a lightweight residual adapter can be added to the output of the shared attention layer for fine-tuning the magnitude and bias in domain-shift scenarios. To ensure decoupling of the two sub-paths, normalized weights specific to each sub-path are introduced after the shared output, allowing the two statistical paths to be updated independently. The cross-modal basic features refer to the low- and mid-level semantic representations after the alignment of image regions and text fragments. These can originate from salient regions and keyword fragments aggregated by attention, such as a cluster of tokens spatially aligned between the cup and the blue text.

[0086] The parameter mapping channel is defined as an independent projection path from the output ends of the two sub-paths to the task output layer, serving to connect the representation space and output space in terms of dimension and semantics. The understanding channel connects to the language or classification output head, employing linear or sparsely gated linear units to reduce coupling; the control channel connects to the trajectory or action output head, using multi-branch regression with piecewise activation to cover the statistical distribution of different action segments. Structurally, the two channels do not share weights to avoid gradient cross-contamination; during execution, channel gating is driven by the selection signal from the router module, ensuring that only the channel matching the task effectively contributes to the task output layer.

[0087] The task output layer is a unified interface layer, compatible with three types of output: text generation, label prediction, and motion trajectory. Different headers are configured for different task types during implementation, but a unified call entry point and return format are maintained to facilitate service encapsulation and integration with downstream systems. To eliminate numerical discontinuities caused by path switching, minor temperature shaping and confidence recalibration can be added before the output layer to bring the distribution of different sub-paths back to a similar scale range.

[0088] The connection relationships are implemented following a clear data flow. Fusion means copying from the shared attention layer to the two sub-path entry points. Gating the active entry point determines that only one path is allowed to execute. After execution, the corresponding parameter mapping channel is opened, while non-corresponding channels remain unresponsive. The task output layer receives tensors from the activated channels and generates the final output based on the return type of the target task. To mitigate switching latency, the two sub-paths and two mapping channels can be resident in memory, while the intermediate cache of inactive paths is managed in low-priority storage. To improve throughput, batch encoding can be performed before the router module to merge requests with the same intent, reducing the number of hardware switches.

[0089] Initialization and numerical stabilization strategies are applied throughout the above units. The shared attention layer is initialized with a uniform distribution and frozen early in training. After the two sub-paths stabilize, a small number of layers are unfrozen for coordinated updates. The expansion factor, normalization parameter, and gating threshold in the sub-path are selected offline using a hyperparameter grid, allowing only minor adaptive adjustments online. The threshold of the router module is automatically corrected using a sliding window statistical method to adapt to the seasonal drift of the intent distribution. All tensor sizes, gating operators, and calling order are verified at compile time using static graphs or shape constraints to avoid dimension mismatches and runtime branch mismatches after deployment.

[0090] This embodiment isolates semantic understanding and action control into two computational paths through parallel partitioning and gated routing. Cross-modal basic features are uniformly aligned at the shared attention layer, avoiding redundant learning and mutual interference of upstream representations. During execution, the activation entry point and parameter mapping channel only allow selected paths; unselected paths do not participate in forward computation, thus reducing inference latency and memory overhead. The output layer connects to different task types via independent channels, ensuring that gradients and numerical distributions do not contaminate each other, improving training and deployment stability. The router module uses text prompts to drive selection, combined with confidence calibration to reduce the probability of misselection, making path switching more reliable. The overall structure maintains cross-modal alignment capabilities while preserving the continuity and executability of action trajectory generation, thereby mitigating the decay of semantic understanding capabilities after control learning and achieving low-interference and high-efficiency collaborative processing in online inference.

[0091] In one embodiment, step S20 above includes:

[0092] S201, Collect sensor data when the robot performs a task to form a robot motion trajectory dataset containing state and action sequences;

[0093] S202, collect image samples and corresponding text descriptions to form a visual text pair dataset;

[0094] S203, Create a set of structured language templates containing operational instructions and spatial location descriptions;

[0095] S204, Mark the mapping relationship between the motion trajectory and the set of structured language templates in the robot motion trajectory dataset.

[0096] In this embodiment, the construction of the robot motion trajectory dataset begins with sensor data acquisition. Data sources may include joint encoders, end-effector pose tracking modules, torque and force sensors, depth cameras, and inertial units. The acquisition clock is uniformly driven by a hardware time base, and a sampling frequency list and timestamp synchronization strategy are established to ensure that the joint angles, joint velocities, end-effector poses, gripper opening and closing states at the same moment correspond one-to-one with external observations on the time axis. Before the data is stored on the disk, hardware calibration and extrinsic parameter solving are performed, the coordinate system transformation between the robot arm base and the camera is recorded, and environmental variables such as temperature and power supply voltage are saved for subsequent drift compensation. The raw data undergoes denoising and jitter reduction. Sliding window mid-range filtering and a limiter are used to eliminate impulse interference. Abnormal segments are removed using Mahalanobis distance or confidence thresholds. The remaining data is segmented into state-action sequences based on task boundaries. State vectors include joint angles, joint velocities, six-dimensional end-effector pose, gripper opening / closing ratio, and observable environment indices. Action vectors can be implemented using one of three methods: joint space increment, Cartesian space increment, or discrete skill encoding. Time intervals within the sequences are standardized, and sequence lengths are automatically determined by start and end events or adaptively trimmed based on pose change thresholds. After cleaning and segmentation, a storage model for the robot motion trajectory dataset is established. A dual-table structure of sample and fragment tables is recommended. The sample table records sequence identifiers and metadata, while the fragment table stores compressed tensor fragments and corresponding timestamps, along with data version and acquisition batch labels for easy incremental updates and audit tracking.

[0097] The construction of the visual text dataset revolves around the consistency between image samples and text descriptions. Image samples can be static frames captured from real scenes, simulated renderings, or reconstructed from multiple perspectives. A unified resolution, color space, and distortion correction model are used, and automatic calibration records for exposure and white balance are established to control visual statistical distribution. Text descriptions can be sourced through manual annotation, semi-automatic generation, or weakly supervised mining. Text standardization includes unified Chinese word segmentation, standardization of numbers and units, alignment of synonym lists, and disabling of ambiguous terms. Long sentences are broken down into several concise propositions through dependency analysis, maintaining a pairing relationship between an image sample and one or more text descriptions. Difficult negative examples are introduced to improve discriminative ability; these can be constructed by different targets within the same scene, different attributes of the same target, or different relationships in similar layouts. The visual text dataset adheres to scene deduplication and object deduplication constraints in its segmentation. The training and validation sets avoid inter-frame leakage within the same sequence, and duplicate frames are removed using perceptual hashing and feature cosine similarity thresholding. The final output includes a lookup table containing image sample identifiers, text description identifiers, pairing confidence scores, and source labels, maintaining foreign key consistency with the content repositories of image samples and text descriptions.

[0098] The structured language template set unifies the expression of operational instructions and spatial location descriptions. Templates are organized in a slot-based manner. Operational instructions can include action predicate slots and object slots, while spatial location descriptions include reference object slots and orientation slots. Constraint slots are added when necessary for safety limits or sequence requirements. Slot values ​​can be sourced from controlled vocabularies, object detection dictionaries, and scene graphs. Lexical forms and units use a unified paradigm, and position units are standardized to the standard units of the scene coordinate system. Direction uses both discrete orientation sets and continuous angles to cover different expression habits. The structured language template set supports two instantiation paths: rule expansion and restricted generation. Rule expansion is based on combinatorial generation capabilities to cover common task expressions, while restricted generation uses a language model to sample from the controlled vocabulary to expand long-tail expressions. All instantiated templates undergo semantic consistency checks and slot integrity checks before being added to the database. The check rules include object existence, reference object visibility, and orientation validity to ensure that the template can be parsed and executed in subsequent binding stages.

[0099] The mapping relationship between action trajectories and the structured language template set is implemented using a binding table. Each state-action sequence corresponds to zero or one operation instruction template and zero or one spatial location description template, expressed as a one-to-one or one-to-zero constraint to reduce ambiguity. Mapping generation can employ three paths: Path 1 is direct binding based on recorded task metadata; Path 2 is automatic retrieval binding based on text inversion, i.e., retrieving matching operation instructions and spatial location descriptions from the structured language template set through the end trajectory and contact events of the sequence; Path 3 is manual review binding, used to cover high-complexity scenarios and boundary situations. Automatic retrieval uses dual feature vectors of trajectory keyframes and spatial relationships. Keyframes are extracted by curvature and velocity change detection algorithms, and spatial relationships are obtained by object detection and relative pose estimation. The matching score is a weighted combination of similarity; those exceeding a threshold enter the candidate set, and a lightweight voter selects the optimal template. The binding table records the confidence level, generation path, and review flag for each relationship pair, and provides a traceable version number for subsequent model training and error analysis. After mapping, the robot motion trajectory dataset and the visual text pair dataset are aligned in a unified representation space through template semantics, ensuring that subsequent training phases can be routed and supervised with consistent semantic anchors.

[0100] To improve overall data quality and usability, all three construction processes adhere to a consistent identification system and data protocol. All entities use unified coding rules, and the acquisition equipment, environmental conditions, and anonymous operator identifiers are mandatoryly recorded. Timestamps are standardized to the millisecond level, and random seeds and data augmentation strategies are explicitly recorded in the metadata. Image augmentation and motion perturbation employ a replayable pipeline to support traceability. At the data distribution level, a read-only cache and verification mechanism are established. Any modification must generate a new version through an incremental pipeline and automatically run an integrity check. The integrity check covers pair consistency, mapping coverage, and modality completeness. Batches that fail the check are not allowed to enter the training pipeline.

[0101] This embodiment constructs and aligns a unified robot motion trajectory dataset, a visual text pair dataset, and a structured language template set within the same namespace. State-action sequences are stably supplied in time-consistent tensor form. Image samples and text descriptions constrain semantic boundaries through pairing and difficult negative examples. Operation instructions and spatial location descriptions provide parsable semantic anchors using slotted templates. The mapping relationship between motion trajectories and templates is solidified using a backtrackable binding table. Thus, during training, consistent semantic supervision drives routing and parameter updates, reducing interference caused by semantic drift and data noise. During inference, the unified representation space improves the parsing accuracy and execution consistency of target task prompts, further mitigating the decline in image and text understanding capabilities after control learning, and providing a stable data foundation for capability inheritance and fusion in subsequent stages.

[0102] In one embodiment, step S30 above includes:

[0103] S301, extract state action sequence samples from the robot motion trajectory dataset, and bind the state action sequence samples to the corresponding templates in the structured language template set;

[0104] S302, Generate task prompts based on the bound structured language template;

[0105] S303, the router module parses the task prompt and generates a control expert subpath activation signal;

[0106] S304, the activation of the control expert subpath is triggered by the control expert subpath activation signal;

[0107] S305, The state action sequence sample is processed by the activated control expert sub-path to determine the trajectory prediction loss value;

[0108] S306, Update the motion control parameters inside the control expert sub-path based on the trajectory prediction loss value through backpropagation;

[0109] S307 saves the complete network architecture containing updated motion control parameters and generates an intermediate training model with robot control capabilities.

[0110] In this embodiment, training begins with joint sampling and pairing of the robot motion trajectory dataset and the structured language template set. State-motion sequence samples are retrieved from the trajectory library by task label and time period, and deduplicated. Uniform timestamps and equal-interval resampling ensure sequence alignment. State vectors include joint angles, joint angular velocities, end-effector pose, end-effector velocity, and gripper opening / closing ratio. Contact events and torque observations are added as needed to enhance observability. Motion vectors use either joint space increment or Cartesian increment encoding, automatically adapted based on the hardware control interface. Long sequences are pruned to fixed or variable length segments through keyframe detection and energy thresholding. These segments are normalized to a device-independent coordinate system, and their original scale information is recorded for denormalization reconstruction. During the sequence loading stage, a mask and effective step size index are constructed to ensure that incomplete segments do not affect the stability of the loss estimation.

[0111] The structured language template set provides bindable text instances. Templates are slot-based, including action predicate slots, object slots, reference slots, and orientation slots, with support for expansion to include quantity and safety constraint slots. The binding process is based on the joint similarity of trajectory end pose, path arc length, contact sequence, and object trajectory, binding state-action sequence samples one-to-one or one-to-zero to a single template or template pair. Similarity is obtained by weighting alignment score and semantic consistency score. The alignment score uses dynamic time warping to calculate the matching degree of path shape and rhythm, while the semantic consistency score uses object visibility and relative pose legality for judgment. Binding results are written to a binding table along with confidence level, matching path, and version number. Samples that fail to meet the confidence threshold are either manually reviewed or placed in a delayed training queue.

[0112] Task prompts are instantiated from bound templates. Template slots are filled with discrete tags from the scene vocabulary and object dictionary. Orientation fields are unified to a join expression of the scene coordinate system and discrete direction sets, and numerical slots undergo unit normalization. Prompts can be concatenated with context labels and device capability labels to form more stable control signals to the router input. After word segmentation and sub-wording, prompts are encoded into discrete identifier sequences, while paragraph masks and position indices are generated to prevent irrelevant prefixes from affecting attention allocation. To improve lexical robustness, synonym substitution and mild word order perturbations can be introduced, but the perturbation amplitude is limited by a semantic preservation threshold; instances exceeding the threshold will not enter the routing stage.

[0113] The router module is responsible for parsing the task prompts and generating activation signals for the control expert subpaths. The router front end uses an embedding layer and positional encoding to map the prompts into a vector sequence. After passing through one or two layers of lightweight attention or gated feedforward networks, it outputs an expert weight vector and a binary activation flag. The weight vector is temperature-scaled and top-one filtered to obtain discrete activations. The activation signals are written into a scheduling mask, and feedforward branches and gradient channels of non-target expert paths are masked. To reduce jitter, hysteresis and cold-start protection are introduced into the activation signals, maintaining consistent selection for several consecutive steps to prevent unstable gradients caused by high-frequency switching.

[0114] After the expert subpath is activated, forward computation and target generation are performed on the state-action sequence samples. The input first encodes the state vector into a latent space trajectory. The encoder can use gated recurrent units or temporal convolutional units, or a short-deep hybrid stacking approach to balance memory and parallelism. The latent space trajectory and the cue encoding are fused using a cross-modal gated fusion module, with the fusion weights controlled by the action predicate and object identifier in the cue. The decoder generates an action vector sequence or the next action distribution, aligning the output with the reference action vector in the same coordinate system. The trajectory prediction loss consists of multiple components, including the joint space mean square error, end-effector pose geodesic distance, velocity smoothing term, and contact consistency penalty. Component weights are adaptively adjusted using hyperparameters or an invariant risk minimization strategy. To stabilize the value, the loss is summed over an effective step-size mask and normalized by the length to avoid bias caused by different segment lengths.

[0115] Parameter updates are confined to the range of motion control parameters within the expert sub-path. The update process employs gradient pruning, mixed precision, and variable learning rate scheduling. The optimizer can be either a momentum adaptive class or a piecewise exponentially decaying momentum class. The shared attention layer, the understanding expert sub-path, and the router's main parameters are kept frozen or semi-frozen. In the semi-frozen state, only small adaptive channels for the normalized scale and bias are open to ensure controllable perturbations to existing graph representations. Backpropagation is performed along the active path in a single channel, while the weight gradients of non-active paths are set to zero. Masking is applied at the operator level to avoid invalid computations. After the update, weight constraints and spectral norm pruning are performed to limit the erosion of the feature space by the control head. To improve generalization, an exponential moving average copy can be maintained; this copy is used only for evaluation and snapshots.

[0116] At the end of each training round, the complete network architecture, including the updated motion control parameters, is persisted. The persisted content includes parameter tensors, computation graph definitions, vocabulary and coordinate system metadata, data normalization statistics, and key inference constants such as routing temperature. It also records the binding table summary hash and training recipe fingerprint to ensure reproducibility during replay. A snapshot is written to versioned storage and generates an intermediate training model identifier, which carries device adaptation information and control interface descriptions for direct loading during subsequent understanding task training stages. To ensure compatibility, the serialization format supports forward and backward reading; if parameter shape changes occur, a mapper is provided to handle shape rearrangement and default padding.

[0117] This embodiment achieves prompt-driven routing and constrained updates by jointly using trajectory samples and structured templates. This allows the control expert sub-paths to learn action mapping relationships consistent with language under stable semantic anchoring. Loss design and gradient masking strictly constrain the optimization scope to motion control parameters, avoiding destructive impacts on graph representation and routing behavior. The intermediate training model is solidified as a snapshot of the complete network and updated weights. Subsequent stages can add understanding capabilities without compromising the acquired control capabilities. This mitigates catastrophic forgetting and task interference, improves the determinism of prompt-based path selection and the consistency of control trajectories, and provides a reproducible capability and a safe iterative foundation for a unified training process.

[0118] In one embodiment, step S40 above includes:

[0119] S401, Load the intermediate training model with robot control capability;

[0120] S402, Extract image-text pair samples from the visual-text pair dataset;

[0121] S403, Generate multimodal task prompts based on the image text samples;

[0122] S404, The router module parses the multimodal task prompt and generates an expert path activation signal;

[0123] S405, the activation of the understanding expert sub-path is triggered by the understanding expert path activation signal;

[0124] S406, The image-text pair samples are processed by the activated understanding expert sub-path to determine the semantic alignment loss value;

[0125] S407, Update the semantic alignment parameters within the understanding expert sub-path based on the semantic alignment loss value through backpropagation;

[0126] S408 saves the complete network architecture, including the updated semantic alignment parameters, and generates a unified training model with multimodal understanding capabilities.

[0127] In this embodiment, training begins by loading an intermediate training model with robot control capabilities and performing structural consistency checks and weight mapping. The model file contains network topology, parameter tensors, vocabulary and coordinate system metadata, normalized statistics, and routing temperature, among other key inference constants. After loading, a freezing strategy is first applied to the weights of the control expert sub-paths, while the main parameters of the shared attention layer and router module are also frozen. Only the learnable channels within the understanding expert sub-paths used for semantic alignment are opened, and if necessary, only their normalized scale and bias are unlocked to limit the update magnitude. If there are differences in parameter shape between the intermediate training model and the current training environment, shape rearrangement and default padding are performed through a mapper to ensure that subsequent batch processing is fully aligned in graph structure and tensor dimensions.

[0128] Image-text pairs are extracted in batches from the visual text pair dataset. On the image side, resolution normalization, center or multi-scale cropping, and mild illumination perturbation and compression artifact suppression while preserving the color gamut are performed. These are then fed into a visual encoder to obtain hierarchical feature maps and image-level global embeddings. On the text side, word segmentation and subtotalization, vocabulary alignment, case and number normalization are performed, and paragraph masks and position indices are generated. These are then passed through a text encoder to obtain sequence-level and sentence-level embeddings. To improve the quality of negative samples, all negative samples are constructed within the batch, and cross-device queue caching is enabled to form more difficult comparison pairs. Cache entries carry timestamps and source identifiers for subsequent eviction. Sample metadata records object visibility, region candidates, and spatial relationship labels for alignment supervision and hard example sampling.

[0129] Multimodal task prompts are generated from image and text samples. On the image side, target region identifiers are selected from salient regions or a candidate bounding box pool. On the text side, action predicates, object references, and spatial relation predicates are extracted from the description. These three types of elements are fed into a prompt template generator to obtain structured prompts. Device capability labels and scene coordinate system identifiers are added as needed to constrain the semantic scope. The encoded prompts maintain vocabulary consistency with the embedded text and are provided to the router as guidance signals.

[0130] The router module receives multimodal task prompts and outputs activation signals for the understanding expert path. The prompts are embedded into an expert weight vector formed by lightweight attention and gated feedforward units. This weight vector undergoes temperature scaling and top-one filtering to obtain discrete activation selections, while simultaneously generating a hysteresis mask to maintain consistent activation across several training steps, avoiding high-frequency jitter. The activation signals are written into a scheduling mask to disable forward operators and gradient channels in non-target expert branches, ensuring that backpropagation only returns along the understanding expert sub-path.

[0131] The expert subpath performs forward and supervised alignment on image-text pairs in the active state. The global and region-level embeddings output by the visual branch are linearly projected onto the same semantic space as the text branch. The sentence-level and word-level embeddings output by the text branch, supported by read-only representations of a shared attention layer, complete cross-modal alignment attention queries and key-value retrieval. The fusion results are input into the semantic comparison head and the matching discriminator head. The semantic alignment loss is composed of multiple weighted factors, including image-to-text and text-to-image contrast loss, image-text matching binary cross-entropy, a term of reference consistency based on word-to-region attention, and a spatial relationship consistency penalty. The contrast loss uses a temperature-parameterized InfoNCE form, with both intra-batch and cross-device negative samples participating. The matching discriminator head samples hard negatives to improve the discrimination boundary. Referential consistency uses a word-to-region attention quality metric to constrain cross-modal binding. The spatial relationship term uses a threshold penalty defined by relative orientation and distance tolerance. The weights of each component can be adaptively adjusted with each training step, ensuring that early global alignment is prioritized while later fine-grained alignment is enhanced. The loss is aggregated on the effective mask and normalized according to the sample length and the negative sample size to avoid length bias and queue size bias.

[0132] During the backpropagation phase, only the semantic alignment parameters within the expert sub-path are updated. The optimizer employs a momentum-adaptive algorithm, combined with learning rate warm-up, cosine annealing, and hierarchical learning rates. The learning rate is higher for the region-level projection layer and alignment head, and lower for the deep semantic encoding layer. Gradient pruning and mixed precision are used to stabilize the values. Gradients in inactive branches are set to zero, keeping the parameters of the expert sub-path and shared attention layer frozen; router weights do not participate in updates. To constrain representation drift, an exponential moving average is introduced to evaluate the end-alignment metric, and spectral norm and L2 regularization constraints are applied after updates to prevent semantic space collapse. If long tails of categories or relationships appear, reweighted sampling and online hard example mining are used to balance gradient contributions.

[0133] After the training round is completed, persistence is performed, saving the complete network architecture including the updated semantic alignment parameters and generating a unified training model identifier. The persisted content covers the understanding expert sub-path weights, cross-modal projection matrices, alignment head parameters, temperature coefficients, negative sample queue state summaries, and loss weight trajectories, while preserving the control expert sub-paths and their normalized statistics of the intermediate training models. The serialization format supports versioned comparison and forward reading. If subsequent quantization or distillation is required at the inference end, an equivalent low-bit copy can be generated based on the current weight snapshot without changing the alignment geometry. The unified training model possesses both previously fixed control capabilities and newly learned graph-text semantic alignment capabilities upon loading. The router, driven by prompts, can stably select either understanding or control for single-path inference execution.

[0134] This embodiment, while keeping the control capabilities of the intermediate-trained model frozen, utilizes the semantic alignment loss value composed of image-text samples and multiple alignment supervisions to perform targeted updates only on the parameters within the understanding expert sub-path. The router, using activation signals generated by multimodal task prompts, restricts gradients and computations to the alignment channel, avoiding perturbations to the control channel and shared attention representations. Thus, without weakening the already acquired control performance, it introduces and reinforces image-text semantic alignment capabilities, unifies the training model to form a single-path inference behavior selectable by prompts, reduces inter-task interference and significantly mitigates catastrophic forgetting, and simultaneously covers the needs of multimodal understanding and robot manipulation with the same architecture during the inference phase, thereby reducing deployment and maintenance costs.

[0135] In one embodiment, step S403 includes:

[0136] S4031, The image data in the image-text pair sample is processed by a convolutional neural network to generate visual semantic alignment features;

[0137] S4032, parse the text description in the image text pair sample, and extract spatial relationship predicates and object references;

[0138] S4033, The visual semantic alignment features, spatial relationship predicates and object pointer expression input prompt template generator are used to generate scene description statements;

[0139] S4034, Add a task type identifier prefix before the scenario description statement to form a multimodal task prompt.

[0140] In this embodiment, generating task prompts is a crucial step in achieving image-text information fusion during multimodal training. First, a convolutional neural network (CNN) processes the image data from the image-text pair samples. The CNN uses convolutional kernels to slide across the spatial dimension to extract local features, thereby capturing edges, textures, region shapes, and higher-order object semantics in the image. After multiple convolutional and pooling operations, a multi-scale feature map representing the target image is obtained and projected onto a unified semantic space in a fully connected layer or a global average pooling layer to generate visual semantic alignment features. This feature not only preserves the local geometric information of the target object but also enhances global contextual relevance through hierarchical encoding, enabling it to form a comparable vector space with the linguistic semantic representation.

[0141] In the text processing section, the image text is parsed to extract the text descriptions from the samples, including spatial relation predicates and object references. Spatial relation predicates are derived from words in natural language that describe the positional relationships of objects, such as "on the left," "above," and "contained in," and are identified and extracted using syntactic dependency trees or sequence labeling models. Object references refer to the specific objects or entity names appearing in the text, such as "red cube" and "metal can." During extraction, named entity recognition and reference resolution methods can be combined to ensure that objects in the language correspond to regions or targets in the image. The parsed predicates and references are then encoded and matched with visual semantic alignment features, becoming semantic primitives for cross-modal fusion.

[0142] Subsequently, visual semantic alignment features, spatial relation predicates, and object referential expressions are input into a cue template generator. The cue template generator is essentially a structured language construction device that receives input features from different modalities and generates natural language sentences through placeholder matching and semantic filling. For example, when the visual semantic alignment features represent a "red cube" in an image, the spatial relation predicate obtained from text parsing is "left," and the object referential expression is "box," the cue template generator can construct a scene description statement such as "move the red cube to the left of the box." The template generator typically incorporates multiple language patterns, covering different grammatical forms such as action, position, comparison, and state descriptions, to enhance the diversity and expressiveness of the generated cue statements.

[0143] After obtaining the scene description statement, a task type identifier prefix is ​​added before it. This prefix is ​​derived from the task classification system during training, such as control tasks, comprehension tasks, and question-answering tasks. By attaching an identifier before the prompt, the expert path that the model should activate when processing that prompt can be clearly defined, thus ensuring correct execution of different task types under a unified architecture. The task type identifier can take the form of a label, symbol, or phrase, such as "[Control]" or "[Comprehension]". These are embedded and recognized by the router, directly influencing the selection of expert sub-paths.

[0144] Through the above process, the multimodal task prompts not only integrate the visual semantic alignment features of images with the logical relationships in text descriptions, but also achieve the unification and instruction of language expression through structured templates and task prefixes. This can efficiently guide the neural network to select appropriate expert sub-paths for processing during subsequent training or inference.

[0145] This embodiment extracts visual semantic alignment features from images using a convolutional neural network and combines them with spatial relationship predicates and object identifiers obtained from text parsing. A prompt template generator constructs scene description statements, adding a task type identifier prefix to the front end. The resulting multimodal task prompts achieve deep fusion of visual and linguistic information. This mechanism ensures that different modalities are uniformly encoded while giving task inputs a clear, instructive form, facilitating correct parsing and assignment by the router module. Therefore, the trained model can maintain the stability of image and text understanding without losing robot control capabilities, reduce interference between different tasks, and flexibly handle multiple input types during the inference phase, improving the accuracy and robustness of task execution.

[0146] In one embodiment, step S50 above includes:

[0147] S501, Obtain target task input including image data, text instructions, and target task prompts;

[0148] S502, the router module parses the target task prompt and generates an expert path selection signal;

[0149] S503, activate the understanding expert sub-path or the control expert sub-path according to the expert path selection signal;

[0150] S504, The image data and text instructions are processed through the shared attention layer in the unified training model to generate cross-modal fusion features;

[0151] S505, the cross-modal fusion features are processed by activating the expert sub-path in the unified training model, and the task processing result is output through the task output layer of the unified training model.

[0152] In this embodiment, the target task input includes image data, text instructions, and target task prompts. During data access, the three types of input are aligned and morphologically standardized within the same time window. Image data is mapped into a visual vector sequence after size normalization, color space correction, noise suppression, and block slicing. Text instructions and target task prompts are mapped into a language vector sequence after word segmentation, sub-word encoding, position embedding, and paragraph identification. The two vector sequences are fed into a shared attention layer in a unified training model to first establish a "base" representation of cross-modal dependencies, ensuring that the same cross-modal basic representation can still be used when only a single path is activated. The shared attention layer adopts a combination structure of multi-head dot product attention and gated normalization to perform bidirectional interaction between visual and language vectors. In the forward path, semantic binding is completed by the interaction of queries from text and keys and values ​​from images, and language constraint backprojection is completed by the interaction of queries from images and keys and values ​​from text. Finally, cross-modal fusion features are output, and the feature dimensions are strictly aligned with the interface dimensions of subsequent expert sub-paths. To combat instruction noise and visual pseudo-correlation, entropy constraints and attention head sparsification are introduced, which suppress irrelevant channels during training and naturally set them to zero during inference, thereby improving the separability and routableness of fused features.

[0153] The router module reads only the language vector corresponding to the target task prompt and calculates the expert path selection signal. The selection signal is obtained through a two-stage calculation: first, a lightweight feedforward network generates two scalar scores, corresponding to the understanding expert sub-path and the control expert sub-path, respectively; then, a temperature-adjustable normalization function maps the scores to a single-channel binary mask. To ensure that only one path is activated, a maximum response constraint and a threshold suppression mechanism are introduced; when the responses are close, threshold suppression forces the retention of the side with the higher amplitude and clears the other side, resulting in a single-bit expert path selection signal. The selection signal is sent to a predefined activation entry, which corresponds one-to-one with two parallel sub-paths. Upon arrival, only the computation graph of the sub-path matched with it is opened, while the other sub-path paths remain closed, and computational resources and backward gradients are not occupied. The selection signal also serves as the gating input of the shared attention layer, used to shield attention channels bound to inactive paths, preventing the diffusion of fused features within unused paths.

[0154] When the understanding expert sub-path is activated, the cross-modal fusion features enter the semantic alignment stack of the understanding branch. This stack contains several layers of semantic mapping and alignment normalization units. First, the fusion features are constrained to the language principal component subspace using low-rank projection. Then, higher-order visual concepts are back-injected to the feature sites of the language sequence using cross-layer residuals. The final aggregated representation is fed into the task output layer of the unified training model, and the output form is a discrete probability distribution or structured labels, such as answer text, classification labels, reference point coordinates, region boxes, or relation triples. When the control expert sub-path is activated, the cross-modal fusion features enter the action generation stack of the control branch. This stack contains state reconstruction units and action mapping units. First, the fusion features are transformed into a concatenated representation of the environment state vector and the target intent vector. Then, trajectory parameters or control variables are obtained through temporal unfolding or one-step mapping, such as end pose, joint angle increments, opening and closing commands, or temporal action fragments. The action vectors are directly fed into the task output layer of the unified training model, where the output layer performs scale restoration, reachability pruning, and safety domain constraints, and outputs the task processing results.

[0155] The task output layer serves as a unified exit point, simultaneously supporting two expert sub-paths. Internally, it contains multiple output heads coupled to the expert path selection signal. When the understanding branch is activated, only the language-related output heads and their post-processing operators are enabled, while control-related output heads remain closed. When the control branch is activated, only the action-related output heads and safety constraint operators are enabled, while language-related output heads remain closed. A morphology-aware post-processing module is placed before the output layer. For text results, it performs temperature sampling or beam search; for action results, it performs speed and jerk constraints, collision cost approximation, and soft-limit pruning, ensuring that the task processing result not only matches the input instructions but also satisfies executable constraints. To improve robustness, a lightweight confidence evaluator is introduced to estimate the quality of the results. When the confidence level falls below a threshold, the output layer triggers a self-checking branch, requiring the shared attention layer to refocus on the key region and repeat the fast forward pass, thereby reducing erroneous outputs caused by occasional misrouting.

[0156] The online runtime sequence is as follows: The target task input is collected and calibrated, generating fused features; the router module parses the target task prompts and generates expert path selection signals; upon receiving the signal at the corresponding active entry point, another branch is closed to prevent parallel interference; the fused features are processed within the active branch, and intermediate results are output; the task output layer activates the matching output header based on the selection signal, completes formatting and security constraints, and outputs the task processing results. The entire process does not rely on external toolchains; all operations are completed in a closed loop within the unified training model. The shared attention layer and task output layer are structurally centered, while the router module and expert sub-paths are distributed on either side, ensuring the shortest possible routing path from input to output and its interpretability.

[0157] In single-camera scenarios, image pyramid inference can enhance near-far scale stability; in multi-camera scenarios, extrinsic calibration aligns multi-view features before the shared attention layer, reducing attention drift caused by cross-parallax. On the text side, cue normalization can be performed before the tokenizer, mapping colloquial phrases in the target task cue to standard instruction phrases, improving the clarity of the router module's discrimination boundaries; alternatively, a small intent classifier can be used for pre-filtering, directly amplifying the scores of corresponding experts when the intent is strongly correlated, and maintaining the original distribution when the intent is weakly correlated, thus avoiding false activations.

[0158] The two-stage computation of the router module can be adjusted according to different real-time targets. In low-latency scenarios, the scoring network is limited to one feedforward layer and one gating unit, and a higher temperature value is used to reduce numerical sharpness and reduce flipping caused by transient noise. In high-precision scenarios, a normalization and residual layer is added, which, combined with a lower temperature value, achieves a stronger single-peak response. A morphological consistency check is added after scoring to ensure that the attention centroid positions of the selected signal and the fused features match. The signal propagation path selection can adopt dual-channel redundancy of event stream and parameter gate. The event stream is used for triggering, and the parameter gate is used to shield the read and write paths of the weights of unselected branches. Both must succeed simultaneously for the branch to take effect, reducing occasional race conditions from an engineering perspective.

[0159] The shared attention layer can adaptively scale the number of heads and the hidden dimension according to the input length. When the upstream visual sequence is too long, low-response blocks are first removed using a learnable sparse mask before entering cross-modal interaction, reducing inference latency. Cross-modal fusion features can be connected to a lightweight compressor to compress the feature dimension to a predetermined interface dimension, ensuring alignment with the expert sub-path while reducing the computational density of the output layer. Understanding the internal structure of the expert sub-path allows for two microstructures: alignment-first or structure-first. Alignment-first enhances the directionality from language to vision in early layers, suitable for question answering and reference parsing; structure-first strengthens the structural reconstruction from vision to language in mid-to-late layers, suitable for scene description and relation extraction. Controlling the internal structure of the expert sub-path allows for two action generation strategies: direct mapping is suitable for static targets and short-term actions, while temporal unrolling is suitable for multi-stage operations and constrained trajectories. Temporal unrolling adds a mask to ensure the dynamic continuity between action segments.

[0160] The multi-head output of the task output layer is decoupled by function. The text-related output head includes a vocabulary projection and language post-processing unit, supporting three types of output: answer, description, and tag. The action-related output head includes a multi-resolution trajectory head and a safe pruning unit. The trajectory head outputs action sequences at different time resolutions, which are then unified by interpolation. The safe pruning unit corrects the sequence point-by-point using a reachability space model and velocity, acceleration, and jerk constraints. To adapt to different hardware, the output layer provides a two-level quantization switch: integer quantization and channel rearrangement are enabled at the edge to reduce bandwidth, while half-precision quantization is enabled at the data center, and more attention heads are retained to improve upper bound performance. During inference batch processing, to avoid cross-sample routing interference, samples can be grouped. Within a group, the attention layer is shared once forward, and within each branch, each branch forwards independently. The output layer then converges to the final result sample by sample, saving computational power and avoiding path crosstalk.

[0161] Example Description: In the healthcare field, a system for individual health monitoring and interactive assistance can be built. First, a shared attention layer, an understanding expert sub-path, a control expert sub-path, and a router module are established within the neural network, enabling the model to simultaneously process multimodal information and execute decision-making control. In this architecture, the understanding expert sub-path processes image and text data from wearable devices, such as dietary images and user text input, while the control expert sub-path generates operational instructions, such as device adjustments and exercise program execution. The shared attention layer ensures the fusion of image and language information, while the router module parses the input health task prompts to determine whether the system enters the understanding path or the control path.

[0162] To ensure the model's stability in real-world operation, a training dataset needs to be constructed first. This involves collecting human motion trajectories during exercise or rehabilitation training using wearable devices and environmental sensors, creating a robot motion trajectory dataset that includes states and action sequences. Simultaneously, image samples related to healthy eating and corresponding text descriptions are collected to form a visual-text pair dataset. Furthermore, structured language templates are designed, covering exercise guidance, dietary advice, and spatial expression, such as "raise your arm to shoulder height" or "put the fruit in the lunchbox," and a mapping relationship is established between these templates and the motion trajectory dataset.

[0163] In the first phase of training, the model is trained using a motion trajectory dataset and a language template set. The router module parses the health task prompts generated by the templates and selects the control expert sub-path for parameter updates. For example, given the template phrase "extend your left arm" and the corresponding motion trajectory data, the router module generates activation signals for the control path. The model predicts the extension trajectory using the control expert sub-path and updates the parameters using the trajectory prediction loss value as feedback, forming an intermediate model with basic motor control capabilities. This intermediate model can reproduce standardized movements in rehabilitation assistance scenarios and lays the foundation for subsequent multimodal understanding.

[0164] In the second training phase, the intermediate model is combined with a visual-text pair dataset. Inputting images and text descriptions of healthy eating, the model generates multimodal task prompts, such as "Identify high-calorie foods in the image." A router module parses the prompts, activates the understanding expert sub-path, and the model fuses the food image and text description through a shared attention layer, calculating the semantic alignment loss and updating the semantic parameters in the understanding sub-path. The resulting unified model retains the motion control capabilities of the first phase while also possessing multimodal understanding capabilities, such as the ability to identify dietary components and provide dietary suggestions based on user input.

[0165] During inference, the system receives the target task input, which may include images of food, text instructions for exercise, and task prompts. The router module parses the prompts, such as "Determine if the food on the plate is suitable for a diabetic patient," generates activation signals, and selects an understanding path. The model processes the image and text input through a shared attention layer and outputs multimodal analysis results. If the input prompt is "Perform five minutes of slow walking training," the router module selects a control path, and the model outputs a motion execution plan based on the control trajectory generated from the input. The unified training model generates cross-modal fusion features through a shared attention layer and completes the task through the activated sub-paths. Finally, the task output layer provides the results.

[0166] For example, in home health management, the system can automatically recognize user-uploaded photos of food plates and, combined with text descriptions, determine food composition and generate dietary recommendations. It can also control assistive devices to perform rehabilitation training movements based on pre-set language templates from doctors. In elderly care facilities, the system can receive voice or text prompts, understand and answer elderly residents' health inquiries, and control assistive robots to perform nursing or rehabilitation actions. This allows a single model to handle multimodal understanding and action control, solving the problem of models losing existing capabilities after acquiring new ones in previous training models, and effectively improving the reliability of the health management system in terms of interactivity and execution.

[0167] In the fintech business, a system for intelligent risk control and transaction assistance can be built. In its implementation, the neural network incorporates a shared attention layer, an understanding expert sub-path, a control expert sub-path, and a router module, forming a unified architecture that combines multimodal understanding and operational control. The understanding expert sub-path processes transaction text, customer behavior descriptions, and market information, while the control expert sub-path generates automated operation instructions, such as triggering risk warnings, portfolio adjustments, or account security verification. The shared attention layer ensures joint modeling of images, text, and structured financial data, and the router module selects the appropriate sub-path by parsing the input financial task prompts, thus enabling flexible task switching.

[0168] The construction of the dataset is a prerequisite for training. By collecting transaction logs, user operation sequences, and market state changes, a financial action trajectory dataset containing state-action mappings is generated; financial charts, market event texts, and transaction news are collected to form a visual text pair dataset; at the same time, a structured language template library is established to cover the expression of trading operations, risk management, and capital flows, such as "upgrade the account risk level by one level" or "adjust the investment portfolio to a low-risk allocation", and these templates are mapped to the transaction trajectory data.

[0169] In the first phase of training, the router module uses transaction trajectory data and language templates to parse the templates, generate task prompts, and selects the control expert sub-path for updates. The model receives input, such as "freeze abnormal accounts," and combines it with corresponding operation trajectory data to generate control path activation signals. It then calculates the trajectory to predict losses and updates control parameters. After training, the intermediate model has the ability to execute risk control commands and automated operations, such as quickly freezing accounts when users engage in abnormal operations.

[0170] In the second phase of training, the intermediate model is combined with a visual-text pair dataset, and the router module selects and updates the understanding sub-path based on the prompts. The model fuses financial charts and news text through a shared attention layer to generate multimodal task prompts, such as "Determine whether the market chart shows a downward trend." The system updates the parameters of the understanding sub-paths using semantic alignment loss, resulting in a unified training model. After training, the unified model can perform both financial risk control and handle complex market understanding and textual reasoning.

[0171] During runtime, the system receives target task input, such as market charts, trading text entered by investors, and task prompts. The router module parses the prompts; if the prompt is "Analyze market downside risk," the understanding sub-path is activated, and the model outputs the market analysis results after text-image fusion; if the prompt is "Convert high-risk accounts to low-risk mode," the control sub-path is activated, and the model generates and executes an operation plan based on trajectory data. Input is processed through a shared attention layer to obtain cross-modal fusion features, and the final result is output by the activated sub-path, achieving a unification of understanding and control.

[0172] In robo-advisory scenarios, the system can automatically identify the correlation between market charts and news text, providing multimodal comprehensive suggestions based on investor input, and directly generating operational instructions for portfolio adjustments. In anti-money laundering and risk control scenarios, the system can identify potential abnormal patterns in text and transaction images, and directly trigger account control operations. This unified model avoids the limitations of traditional systems that lose semantic understanding after training transaction controls, effectively solving the problems of catastrophic forgetting and task interference, making risk control and investment assistance more efficient and reliable.

[0173] This embodiment generates cross-modal fusion features at the inference stage using a shared attention layer through a unified training model. A router module compresses the target task prompts into expert path selection signals, activating only one channel from either the understanding expert sub-path or the control expert sub-path. Finally, the task output layer generates the result at a single exit point. This approach creates short routing paths and single-branch execution on the computation graph, significantly reducing inference latency and avoiding interference between parallel branches. In terms of representation, the shared attention layer provides a unified cross-modal basic representation. Activated paths undergo only task-specific transformations, preserving language understanding capabilities while stably outputting executable control variables, mitigating representation conflicts formed during training that become apparent during inference. In terms of engineering, selection signals and output head gating enable adaptive switching of the result format, unifying text and actions into the same output layer and post-processing pipeline, reducing system integration complexity and improving stability.

[0174] In one embodiment, an expert sub-path-based task processing apparatus is provided, which corresponds one-to-one with the expert sub-path-based task processing method described in the above embodiments. (Refer to...) Figure 3 , Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the task processing device based on expert subpaths of the present invention. The modules include an expert hybrid mechanism module 10, a training data construction module 20, a control training module 30, an understanding training module 40, and a task reasoning module 50. Detailed descriptions of each functional module are as follows:

[0175] The expert hybrid mechanism module 10 is used to set up a shared attention layer, understand expert sub-paths, control expert sub-paths and router modules in the neural network;

[0176] Training data construction module 20 is used to construct robot motion trajectory dataset, visual text pair dataset and structured language template set;

[0177] The control training module 30 is used to perform the first stage of training based on the robot motion trajectory dataset and the structured language template set. The router module selects the control expert sub-path according to the task prompts to update the parameters and generate an intermediate training model.

[0178] The training module 40 is used to perform a second-stage training on the dataset based on the intermediate training model and the visual text. The router module selects the understanding expert sub-path according to the task prompts to update the parameters and generate a unified training model.

[0179] The task reasoning module 50 is used to receive the target task input, and the router module selects the understanding expert sub-path or the control expert sub-path according to the target task prompt in the target task input, and processes the target task input through the unified training model to generate the task processing result.

[0180] In one embodiment, the expert hybrid mechanism module 10 is specifically used for:

[0181] The feedforward network portion of the neural network is divided into two parallel sub-paths: an understanding expert sub-path and a control expert sub-path.

[0182] Design a router module and establish a connection between the router module and the activation entry of the understanding expert sub-path and the activation entry of the control expert sub-path;

[0183] A shared attention layer for extracting cross-modal fundamental features is configured between the understanding expert sub-path and the control expert sub-path;

[0184] Parameter mapping channels with the task output layer are established at the output ends of the understanding expert sub-path and the control expert sub-path, respectively.

[0185] In one embodiment, the training data construction module 20 is specifically used for:

[0186] Collect sensor data when the robot performs a task to form a robot motion trajectory dataset containing state and action sequences;

[0187] Collect image samples and their corresponding text descriptions to form a visual-text pair dataset;

[0188] Create a collection of structured language templates containing operational instructions and spatial location descriptions;

[0189] The mapping relationship between the robot motion trajectory and the set of structured language templates is marked in the robot motion trajectory dataset.

[0190] In one embodiment, the control training module 30 is specifically used for:

[0191] Extract state action sequence samples from the robot motion trajectory dataset and bind the state action sequence samples to corresponding templates in the structured language template set;

[0192] Generate task prompts based on the bound structured language template;

[0193] The router module parses the task prompt and generates a control expert subpath activation signal;

[0194] The activation of the control expert subpath is triggered by the control expert subpath activation signal;

[0195] The state action sequence samples are processed by the activated control expert sub-path to determine the trajectory prediction loss value;

[0196] The motion control parameters within the control expert sub-path are updated by backpropagation based on the trajectory prediction loss value.

[0197] Save the complete network architecture containing the updated motion control parameters to generate an intermediate training model with robot control capabilities.

[0198] In one embodiment, the training module 40 is specifically used for:

[0199] Load the intermediate training model with robot control capabilities;

[0200] Extract image-text pair samples from the visual-text pair dataset;

[0201] Multimodal task prompts are generated based on the image text samples;

[0202] The router module parses the multimodal task prompt and generates an expert path activation signal.

[0203] The activation of the understanding expert sub-path is triggered by the understanding expert path activation signal;

[0204] The image-text pair samples are processed by the activated understanding expert sub-path to determine the semantic alignment loss value;

[0205] The semantic alignment parameters within the understanding expert sub-path are updated by backpropagation based on the semantic alignment loss value.

[0206] Save the complete network architecture containing the updated semantic alignment parameters to generate a unified training model with multimodal understanding capabilities.

[0207] In one embodiment, the training module 40 is specifically used for:

[0208] The image data in the image-text pair samples are processed by a convolutional neural network to generate visual semantic alignment features.

[0209] The image text pairs are analyzed to extract spatial relationship predicates and object references from the text descriptions in the samples.

[0210] The visual semantic alignment features, spatial relationship predicates, and object reference expression input prompt template generator are used to generate scene description statements.

[0211] Add a task type identifier prefix before the scenario description statement to form a multimodal task prompt.

[0212] In one embodiment, the task reasoning module 50 is specifically used for:

[0213] Acquire target task input, including image data, text instructions, and target task prompts;

[0214] The router module parses the target task prompt and generates an expert path selection signal;

[0215] Activate the understanding expert sub-path or the control expert sub-path according to the expert path selection signal;

[0216] The image data and text instructions are processed through the shared attention layer in the unified training model to generate cross-modal fusion features;

[0217] The cross-modal fusion features are processed by activating expert sub-paths in the unified training model, and the task processing results are output through the task output layer of the unified training model.

[0218] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used for communication with external user terminals via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a server-side task processing method based on expert subpaths.

[0219] In one embodiment, a computer device is provided, which may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements user-side functions or steps of a task processing method based on expert subpaths.

[0220] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0221] In the neural network, a shared attention layer, an expert sub-path understanding module, an expert sub-path control module, and a router module are set up.

[0222] Construct a robot motion trajectory dataset, a visual text pair dataset, and a structured language template set;

[0223] The first stage of training is performed based on the robot motion trajectory dataset and the structured language template set. The router module selects the control expert sub-path according to the task prompts to update the parameters and generate an intermediate training model.

[0224] The second stage of training is performed on the dataset based on the intermediate training model and the visual text. The router module selects the understanding expert sub-path according to the task prompts to update the parameters and generate a unified training model.

[0225] The router module receives the target task input, selects the understanding expert sub-path or the control expert sub-path based on the target task prompt in the target task input, and processes the target task input through the unified training model to generate the task processing result.

[0226] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0227] In the neural network, a shared attention layer, an expert sub-path understanding module, an expert sub-path control module, and a router module are set up.

[0228] Construct a robot motion trajectory dataset, a visual text pair dataset, and a structured language template set;

[0229] The first stage of training is performed based on the robot motion trajectory dataset and the structured language template set. The router module selects the control expert sub-path according to the task prompts to update the parameters and generate an intermediate training model.

[0230] The second stage of training is performed on the dataset based on the intermediate training model and the visual text. The router module selects the understanding expert sub-path according to the task prompts to update the parameters and generate a unified training model.

[0231] The router module receives the target task input, selects the understanding expert sub-path or the control expert sub-path based on the target task prompt in the target task input, and processes the target task input through the unified training model to generate the task processing result.

[0232] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and user side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0233] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0234] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0235] It should be noted that if any software tools or components not belonging to this company appear in the embodiments of this application, they are merely illustrative examples and do not represent actual use. The embodiments described above are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A task processing method based on expert sub-paths, characterized in that, Includes the following steps: In the neural network, a shared attention layer, an expert sub-path understanding module, an expert sub-path control module, and a router module are set up. Construct a robot motion trajectory dataset, a visual text pair dataset, and a structured language template set; The first stage of training is performed based on the robot motion trajectory dataset and the structured language template set. The router module selects the control expert sub-path according to the task prompts to update the parameters and generate an intermediate training model. The second stage of training is performed on the dataset based on the intermediate training model and the visual text. The router module selects the understanding expert sub-path according to the task prompts to update the parameters and generate a unified training model. The router module receives the target task input, selects the understanding expert sub-path or the control expert sub-path based on the target task prompt in the target task input, and processes the target task input through the unified training model to generate the task processing result.

2. The task processing method based on expert sub-paths as described in claim 1, characterized in that, The neural network includes a shared attention layer, an expert sub-path understanding module, an expert sub-path control module, and a router module, including: The feedforward network portion of the neural network is divided into two parallel sub-paths: an understanding expert sub-path and a control expert sub-path. Design a router module and establish a connection between the router module and the activation entry of the understanding expert sub-path and the activation entry of the control expert sub-path; A shared attention layer for extracting cross-modal fundamental features is configured between the understanding expert sub-path and the control expert sub-path; Parameter mapping channels with the task output layer are established at the output ends of the understanding expert sub-path and the control expert sub-path, respectively.

3. The task processing method based on expert sub-paths as described in claim 1, characterized in that, Construct a robot motion trajectory dataset, a visual text pair dataset, and a structured language template set, including: Collect sensor data when the robot performs a task to form a robot motion trajectory dataset containing state and action sequences; Collect image samples and their corresponding text descriptions to form a visual-text pair dataset; Create a collection of structured language templates containing operational instructions and spatial location descriptions; The mapping relationship between the robot motion trajectory and the set of structured language templates is marked in the robot motion trajectory dataset.

4. The task processing method based on expert sub-paths as described in claim 1, characterized in that, The first stage of training is performed based on the robot motion trajectory dataset and the structured language template set. The router module selects the control expert sub-path based on task prompts to update parameters, generating an intermediate training model, including: Extract state action sequence samples from the robot motion trajectory dataset and bind the state action sequence samples to corresponding templates in the structured language template set; Generate task prompts based on the bound structured language template; The router module parses the task prompt and generates a control expert subpath activation signal; The activation of the control expert subpath is triggered by the control expert subpath activation signal; The state action sequence samples are processed by the activated control expert sub-path to determine the trajectory prediction loss value; The motion control parameters within the control expert sub-path are updated by backpropagation based on the trajectory prediction loss value. Save the complete network architecture containing the updated motion control parameters to generate an intermediate training model with robot control capabilities.

5. The task processing method based on expert sub-paths as described in claim 1, characterized in that, Based on the intermediate training model and the visual text dataset, a second stage of training is performed. The router module selects the understanding expert sub-path based on task prompts to update parameters, generating a unified training model, including: Load the intermediate training model with robot control capabilities; Extract image-text pair samples from the visual-text pair dataset; Multimodal task prompts are generated based on the image text samples; The router module parses the multimodal task prompt and generates an expert path activation signal. The activation of the understanding expert sub-path is triggered by the understanding expert path activation signal; The image-text pair samples are processed by the activated understanding expert sub-path to determine the semantic alignment loss value; The semantic alignment parameters within the understanding expert sub-path are updated by backpropagation based on the semantic alignment loss value. Save the complete network architecture containing the updated semantic alignment parameters to generate a unified training model with multimodal understanding capabilities.

6. The task processing method based on expert sub-paths as described in claim 5, characterized in that, Based on the image text, multimodal task prompts are generated from the samples, including: The image data in the image-text pair samples are processed by a convolutional neural network to generate visual semantic alignment features. The image text pairs are analyzed to extract spatial relationship predicates and object references from the text descriptions in the samples. The visual semantic alignment features, spatial relationship predicates, and object reference expression input prompt template generator are used to generate scene description statements. Add a task type identifier prefix before the scenario description statement to form a multimodal task prompt.

7. The task processing method based on expert sub-paths as described in claim 1, characterized in that, Upon receiving the target task input, the router module selects either the understanding expert sub-path or the control expert sub-path based on the target task prompts in the input, and processes the target task input through the unified training model to generate task processing results, including: Acquire target task input, including image data, text instructions, and target task prompts; The router module parses the target task prompt and generates an expert path selection signal; Activate the understanding expert sub-path or the control expert sub-path according to the expert path selection signal; The image data and text instructions are processed through the shared attention layer in the unified training model to generate cross-modal fusion features; The cross-modal fusion features are processed by activating expert sub-paths in the unified training model, and the task processing results are output through the task output layer of the unified training model.

8. A task processing device based on expert subpaths, characterized in that, The expert subpath-based task processing device includes: The expert hybrid mechanism module is used to set up shared attention layers, understand expert sub-paths, control expert sub-paths, and the router module in the neural network; The training data construction module is used to build robot motion trajectory datasets, visual text pair datasets, and structured language template sets; The control training module is used to perform the first stage of training based on the robot motion trajectory dataset and the structured language template set. The router module selects the control expert sub-path according to the task prompts to update the parameters and generate an intermediate training model. The understanding training module is used to perform a second-stage training on the dataset based on the intermediate training model and the visual text. The router module selects the understanding expert sub-path according to the task prompts to update the parameters and generate a unified training model. The task reasoning module is used to receive the target task input, and the router module selects the understanding expert sub-path or the control expert sub-path according to the target task prompt in the target task input, and processes the target task input through the unified training model to generate the task processing result.

9. A computer device, characterized in that, The computer device includes a memory, a processor, and an expert subpath-based task processing program stored in the memory and executable on the processor, wherein the expert subpath-based task processing program, when executed by the processor, implements the steps of the expert subpath-based task processing method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a task processing program based on expert subpaths, which, when executed by a processor, implements the steps of the task processing method based on expert subpaths as described in any one of claims 1-7.

Images