Embodied intelligent multitasking method and device based on dynamic routing, and medium

Abstract

This invention discloses an embodied intelligence multi-task processing method, device, and medium based on dynamic routing, relating to the field of embodied intelligence technology. The method includes: acquiring an input image; inputting the input image into a feature extraction network to extract general visual features; performing routing decisions based on the general visual features to generate routing weights, the routing weights indicating the activated task cluster branches from multiple task cluster branches; inputting the general visual features into the activated task cluster branches for feature transformation according to the routing weights to obtain task cluster features; and inputting the task cluster features into a task-specific header corresponding to the activated task cluster branch to obtain the task processing result. This invention provides an effective solution for efficiently meeting the real-time multi-task processing needs of embodied intelligence machine vision under the condition of limited resources on edge devices.

Description

Technical Field

[0001] This invention relates to the field of embodied intelligence technology, and in particular to an embodied intelligence multitasking method, device and medium based on dynamic routing. Background Technology

[0002] In the field of embodied intelligence, intelligent agents such as robots rely on machine vision to perform various perception tasks in real time in dynamic environments, such as personnel detection, behavior recognition, and accurate estimation of object poses for physical interactions. The computational complexity of these tasks varies significantly and often needs to be processed in parallel on edge platforms where computing power and power consumption are limited.

[0003] In existing technologies, multi-task learning networks for machine vision generally employ a fixed computational graph structure. Regardless of the complexity of the input scene, the network executes complete forward inference across all preset branches. Even when only a single stationary object needs to be identified in an open scene, the network still activates all computational units designed to handle complex multi-person interactions or dynamic physical constraints. However, this fixed full-scale computation method results in a significant waste of computing power, substantial fluctuations in inference latency, and persistently high power consumption, severely weakening the real-time responsiveness of embodied agents in dynamic environments. Furthermore, for systems that require rapid mapping of visual perception results to physical models for real-time control, these latency and energy efficiency bottlenecks directly impact operational accuracy and safety. Summary of the Invention

[0004] This invention provides an embodied intelligent multitasking processing method, device, and medium based on dynamic routing. The technical problem it aims to solve is: how to provide an effective solution that can efficiently meet the real-time processing needs of embodied intelligent machine vision multitasking under the condition of limited resources of edge devices.

[0005] In a first aspect, the present invention provides an embodied intelligent multitasking method based on dynamic routing, comprising: Obtain the input image; The input image is fed into a feature extraction network to extract general visual features; Routing decisions are made based on the general visual features, and routing weights are generated. The routing weights are used to indicate the task cluster branches activated from multiple task cluster branches. Based on the routing weights, the general visual features are input into the activated task cluster branches for feature transformation to obtain task cluster features; The task cluster features are input into the task-specific header corresponding to the activated task cluster branch to obtain the task processing result.

[0006] Optionally, the feature extraction network is composed of multiple depthwise separable convolutional modules stacked together, each of the depthwise separable convolutional modules including a channel-wise convolutional layer, a channel attention mechanism module, and a pointwise convolutional layer; The step of inputting the input image into the feature extraction network to extract general visual features includes: The input image is subjected to standard convolution processing to obtain an initial feature map; The initial feature map is sequentially passed through multiple depthwise separable convolutional modules. In each depthwise separable convolutional module, spatial features are first extracted through a channel-wise convolutional layer, then adaptively weighted through a channel attention mechanism module, and finally fused through a pointwise convolutional layer. The feature map output by the last depthwise separable convolutional module is used as the general visual feature.

[0007] Optionally, the step of making routing decisions based on the general visual features and generating routing weights includes: Global average pooling is performed on the general visual features to obtain the feature vector; The feature vector is input into a gating network to generate raw scores corresponding to the number of branches in the task cluster; Based on the general visual features, the scene complexity coefficient is predicted, and the scene complexity coefficient is used as the temperature coefficient. Divide the original score by the temperature coefficient and input it into the softmax function to obtain the probability distribution of each task cluster branch; Perform the argmax operation on the probability distribution to obtain the routing weights in one-hot form.

[0008] Optionally, the feature extraction network, the task cluster branch, and the task-specific header are deployed on a target edge computing chip, which includes multiple heterogeneous hardware computing units and a shared on-chip cache. The step of inputting the general visual features into the activated task cluster branch for feature transformation according to the routing weight to obtain task cluster features includes: The general visual features are stored in the shared on-chip cache. The currently active task cluster branch set is determined based on the routing weight; The instruction streams of each task cluster branch in the activated task cluster branch set are respectively allocated to different heterogeneous hardware computing units; The heterogeneous hardware computing units read the general visual features from the shared on-chip cache, execute the instruction stream of the assigned task cluster branches in parallel, generate specific feature maps for each task cluster branch, and write the specific feature maps back to the shared on-chip cache. The specific feature maps of all active task cluster branches are read from the shared on-chip cache and channel splicing is performed to obtain the task cluster features.

[0009] Optionally, the step of reading the general visual features from the shared on-chip cache through each of the heterogeneous hardware computing units, executing the instruction stream of the allocated task cluster branches in parallel, and generating specific feature maps for each task cluster branch includes: For each activated task cluster branch, the dynamic fusion coefficient of multiple parallel sub-branches within the activated task cluster branch is determined based on the branch weight corresponding to the activated task cluster branch in the routing weight. The general visual features are input into the multiple parallel sub-branches, and the multiple parallel sub-branches have convolution kernels of different sizes to obtain multiple sub-branch output feature maps; After multiplying the output feature maps of the multiple sub-branches by their respective dynamic fusion coefficients, the elements are added together to obtain the specific feature map of the activated task cluster branch.

[0010] Optionally, the parameters of the feature extraction network, the task cluster branch, the task-specific header, and the gating network used for the routing decision are obtained through the following training steps: Obtain training images and their corresponding multi-task labels; The training images are input into the feature extraction network to obtain training general visual features; The trained general visual features are input into each task cluster branch to obtain a specific feature map for each task cluster branch. The general visual features for training are input into the gating network, and Gumbel noise is added before the softmax function is used to generate training soft route weights. The specific feature maps of each task cluster branch are weighted and aggregated according to the training soft route weights to obtain the training task cluster features, and the prediction results are obtained through the task-specific header. The task loss is calculated based on the prediction results and the multi-task labels, and the entropy regularization loss is calculated based on the training soft route weights. With the goal of minimizing the weighted sum of the task loss and the entropy regularization loss, the parameters of the feature extraction network, the task cluster branch, the task-specific head, and the gating network are updated.

[0011] Optionally, updating the parameters of the feature extraction network, the task cluster branch, the task-specific head, and the gating network with the objective of minimizing the weighted sum of the task loss and the entropy regularization loss includes: Obtain the pre-calibrated computational resource consumption values ​​for each task cluster branch; Calculate the expected computational resource consumption of the current training sample based on the training soft route weights and the computational resource consumption value; The deviation between the expected computational resource consumption and the preset target resource limit is calculated to obtain the resource loss; The total loss is obtained by adding the resource loss to the weighted sum of the task loss and the entropy regularization loss. With the goal of minimizing the total loss, the parameters of the feature extraction network, the task cluster branch, the task-specific head, and the gating network are updated.

[0012] Optionally, the step of inputting the task cluster features into the task-specific header corresponding to the activated task cluster branch to obtain the task processing result includes: The currently active task type is determined based on the routing weight; At least one task-specific header matching the task type is invoked from a pre-configured set of task headers, each task-specific header containing a dynamic convolutional kernel to generate a sub-network; The feature subset corresponding to at least one task-specific head in the task cluster features is input into the dynamic convolution kernel generation subnetwork to generate the convolution kernel parameters of the task-specific head; The feature subset is convolved using the generated convolution kernel parameters to obtain the task processing result.

[0013] Secondly, the present invention also provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0014] Thirdly, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the above-described method.

[0015] This invention provides an embodied intelligent multi-task processing method, device, and medium based on dynamic routing. After acquiring an input image, a feature extraction network extracts general visual features, and routing decisions are made based on these general visual features to generate routing weights. The routing weights are used to indicate the activated task cluster branches from multiple task cluster branches. Further, according to the routing weights, the general visual features are input only to the activated task cluster branches for feature transformation to obtain task cluster features. These task cluster features are then input to the corresponding task-specific header to obtain the task processing result. In the above processing, the routing decision dynamically determines the task cluster branches to be activated based on the content characteristics of the input image. For task cluster branches that are irrelevant to the current scene, their feature transformation operations are completely skipped, and the corresponding computing resources are not consumed. This allows the network to perform only the necessary branch calculations for simple scenes and activate more branches as needed for complex scenes, thereby keeping the average computational load of multi-task inference at a low level. While maintaining the processing accuracy of each task, it reduces inference latency and power consumption, enabling embodied intelligent devices deployed on edge platforms with limited computing power and power consumption to efficiently meet the real-time processing needs of multiple machine vision tasks in parallel. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating an embodied intelligent multitasking method based on dynamic routing, provided in an embodiment of the present invention; Figure 2 This is a schematic block diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0020] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0021] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0022] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0023] Please see Figure 1This invention provides a dynamic routing-based embodied intelligence multitasking method, executed by an edge computing platform deployed on embodied intelligent devices. Embodied intelligent devices are intelligent devices that possess physical entities, can perceive environmental states through sensors, and exert influence on the physical world through actuators. Typical examples include intelligent robotic arms on industrial production lines, autonomous navigation and handling robots in warehousing and logistics, and quadrupedal robotic dogs or rotary-wing drones for outdoor inspections. These devices need to perform multiple machine vision tasks simultaneously during operation. Furthermore, machine vision refers to the technology that uses image sensors and computing devices to simulate human visual functions, extracting information from images or videos and making judgments. For example, in a grasping operation, the robotic arm needs to simultaneously detect and identify the operator, understand human actions, and estimate the six-degree-of-freedom pose of the target object. The six-degree-of-freedom pose refers to the position and orientation of an object in three-dimensional space, including three translational degrees of freedom and three rotational degrees of freedom. Furthermore, the aforementioned tasks differ significantly in their feature attributes. Person detection belongs to the semantic attribute category, focusing on target category and identity; object pose estimation belongs to the geometric attribute category, focusing on spatial location and orientation; and behavior recognition belongs to the dynamic analysis category, focusing on temporal motion patterns. Moreover, under the constraints of limited computing power and power consumption on edge devices, running multiple independent single-task models simultaneously can lead to exhaustion of computing resources and uncontrolled response latency. While traditional multi-task learning networks reduce some parameters by sharing underlying features, their computational graph structure is static. This means that regardless of the complexity of the input image scene, the network will execute complete forward inference across all preset branches. Even when only a single stationary object needs to be identified in an open scene, the network will still activate all computational units designed to handle complex multi-person interactions or dynamic physical constraints. This fixed full-scale computation method results in a significant waste of computing power, significant fluctuations in inference latency, and persistently high power consumption, severely weakening the real-time response capability of the embodied agent in dynamic environments. For systems that need to quickly map visual perception results to physical models for real-time control, these latency and energy efficiency bottlenecks directly affect operational accuracy and safety. Furthermore, to address the aforementioned issues, this embodiment introduces a dynamic routing mechanism into the network architecture. Its core concept is to use the content characteristics of the input image itself as the decision-making basis, and through a learnable routing decision network, adaptively determine which task cluster branches are needed for computation in the current scene. For simple scenes, unnecessary complex branches are automatically skipped, while for complex scenes, more analytical capabilities are activated. This transforms the computation mode of the multi-task network from "fixed full load" to "on-demand activation," significantly reducing the average inference computation load without sacrificing multi-task processing accuracy. Specifically, the method includes the following steps: S1, Obtain the input image.

[0024] In practice, an input image is acquired. The input image is current scene image data collected by a visual sensor mounted on the embodied intelligent device. The visual sensor can be an RGB camera, a depth camera, or an infrared camera; this invention is not specifically limited to these. The input image serves as the data starting point for the entire processing flow, providing raw visual information for all subsequent visual analysis tasks.

[0025] S2, input the input image into the feature extraction network to extract general visual features.

[0026] In specific implementation, the input image is fed into a feature extraction network to extract general visual features. The feature extraction network is a basic neural network used to extract feature maps with general representation capabilities from the pixels of the original image. Internally, it consists of multiple stacked convolutional layers, pooling layers, and activation function layers; this invention does not specifically limit the specific layers. Further, the input image undergoes layer-by-layer forward computation through the feature extraction network, outputting a high-dimensional feature map in the middle and later layers of the network, which is the general visual feature. Further, the general visual feature is a tensor representation of the input image after fusing and encoding multiple levels of visual information such as edges, textures, shapes, and semantics. A tensor refers to a multi-dimensional array data structure, typically containing three dimensions in image processing: channel dimension, height dimension, and width dimension. Further, the number of channels in the general visual feature is typically 256 or 512, and the spatial resolution is a fraction of the input image size, depending on the network structure design. Further, the extraction of the general visual feature is a fundamental step in the entire multi-task processing flow; subsequent routing decisions and task processing are all based on this general visual feature.

[0027] In some preferred embodiments, the feature extraction network is composed of multiple depthwise separable convolutional modules stacked together. Each depthwise separable convolutional module includes a channel-wise convolutional layer, a channel attention mechanism module, and a pointwise convolutional layer. The step of inputting the input image into the feature extraction network to extract general visual features includes: performing standard convolution processing on the input image to obtain an initial feature map; sequentially passing the initial feature map through multiple depthwise separable convolutional modules, where in each depthwise separable convolutional module, spatial features are first extracted through the channel-wise convolutional layer, then adaptively weighted through the channel attention mechanism module, and finally fused through the pointwise convolutional layer; the feature map output by the last depthwise separable convolutional module is used as the general visual feature.

[0028] In practical implementation, in the dynamic routing multi-task processing architecture, the feature extraction network is responsible for extracting general visual features from the original input image that can be used by all subsequent task cluster branches and routing decision networks. Its computational efficiency directly determines the basic latency and power consumption level of the entire inference process. Conventional standard convolutions have a significant multiplicative effect on computational cost in terms of channel number and spatial dimension. For lightweight models deployed on edge devices, excessively high basic feature extraction costs will compress the computational space saved by routing decisions. To significantly reduce the computational cost and parameter count of the feature extraction network while maintaining sufficient feature representation capabilities, this embodiment uses stacked depthwise separable convolutional modules and embeds a channel attention mechanism in each module. This allows the feature extraction network to output high-quality, multi-scale general visual representations with an extremely low computational budget, laying an efficient foundation for subsequent dynamic routing and branch computation. The specific implementation is as follows: The feature extraction network consists of multiple depthwise separable convolutional modules stacked together. Each depthwise separable convolutional module includes a channel-wise convolutional layer, a channel attention mechanism module, and a pointwise convolutional layer. Channel-wise convolution refers to spatial convolution performed on each channel of the input feature map using a separate two-dimensional convolutional kernel. The kernel only slides in the spatial dimension and does not involve information interaction between channels; its computational cost is a fraction of that of standard convolution. Furthermore, pointwise convolution refers to linearly combining the output of the channel-wise convolution using a 1x1 convolutional kernel, its function being to perform information fusion and dimensionality transformation along the channel dimension. Decomposing standard convolution into a cascade of channel-wise and pointwise convolutions is mathematically equivalent to the low-rank decomposition of standard convolution, but the computational cost can be reduced to a fraction of that of standard convolution. Furthermore, the channel attention mechanism module is a lightweight, plug-and-play structure whose function is to explicitly model the interdependencies between channels of the feature map, learn a channel-dimensional weight vector, enhance important feature channels, and suppress irrelevant feature channels. Furthermore, this adaptive recalibration enables general visual features to automatically focus on channels that are more informative for subsequent multi-tasks without significantly increasing computational burden, providing a higher-quality foundational representation for routing decisions. The workflow of the channel attention mechanism module is as follows: First, the input feature map is subjected to global average pooling in the spatial dimension to obtain a channel description vector. Global average pooling refers to the operation of taking the arithmetic mean of the values ​​at all spatial locations of each channel in the feature map. Then, the channel description vector is passed through a small bottleneck network consisting of two fully connected layers and a non-linear activation function. The bottleneck network is a network structure with intermediate layer dimensions smaller than the input and output dimensions, used to compress information and remove redundancy. Finally, channel attention weights are generated, where each element of the weight vector is a value between 0 and 1, representing the importance of the corresponding channel. Finally, the channel attention weights are multiplied element-wise by each channel of the original input feature map to obtain the recalibrated feature map. Recalibration refers to the operation of adjusting the response intensity of each channel in the feature map through a weighted approach.

[0029] Furthermore, the input image is fed into the feature extraction network to extract general visual features. Specifically, this includes performing standard convolution processing on the input image to obtain an initial feature map. Standard convolution processing is the first layer of the feature extraction network, using a conventional 3x3 convolution kernel to perform a convolution operation on the input image with a stride of 2, initially mapping the image from pixel space to feature space, while simultaneously reducing the spatial size of the feature map by half. The stride refers to the pixel interval by which the convolution kernel slides across the input feature map each time. Further, the initial feature map is sequentially passed through multiple depthwise separable convolution modules. In each depthwise separable convolution module, spatial feature extraction is first performed through a channel-wise convolutional layer, with each channel independently undergoing a 3x3 or 5x5 depthwise convolution. Then, a channel attention mechanism module adaptively weights the channels, explicitly enhancing the response of channels that are discriminative for the current task and suppressing irrelevant channels. Finally, channel fusion is performed through a pointwise convolutional layer, remapping the large number of channels output by the channel-wise convolution to an appropriate number of output channels through a linear combination of 1x1 convolution kernels. Furthermore, multiple depthwise separable convolutional modules are stacked sequentially from low to high layers. Low-layer modules extract low-level visual features such as edges and corners; mid-layer modules extract mid-level features such as texture and local shape; and high-layer modules extract high-level features such as semantics and global context. Edges refer to regions in an image where pixel brightness changes abruptly, and corners are the intersections of two edges. Furthermore, the feature map output by the last depthwise separable convolutional module is used as a general visual feature. This feature map integrates multi-scale, multi-layer visual information from the low to high layers, providing a rich and general visual representation foundation for subsequent routing decisions and various heterogeneous tasks.

[0030] This embodiment reduces the computational cost of the feature extraction network to a fraction of that of standard convolutions by replacing standard convolutions with depthwise separable convolutions. Under the limited computing power of edge devices, the inference time in the feature extraction stage is significantly reduced, freeing up more computational resources for subsequent task cluster branches and routing decision networks. Furthermore, a channel attention mechanism is embedded in each depthwise separable convolution module. Through lightweight inter-channel dependency modeling, adaptive recalibration of feature channels is achieved, enhancing the response strength of valuable feature channels in current multi-task visual analysis and suppressing interference from irrelevant or redundant channels. This results in higher discriminative power and information density of general visual features in the channel dimension, effectively improving the overall accuracy of subsequent multi-task processing without significantly increasing computational cost or parameter count. Moreover, the collaborative design of depthwise separable convolutions and channel attention simultaneously addresses the two mutually constraining design goals of operational efficiency and feature quality on edge devices, providing efficient and reliable basic feature support for the entire dynamic routing multi-task architecture.

[0031] S3, make routing decisions based on the general visual features and generate routing weights, which are used to indicate the task cluster branches activated from multiple task cluster branches.

[0032] In specific implementation, routing decisions are made based on general visual features to generate routing weights. Routing decision refers to the intelligent decision-making process that automatically determines which task cluster branches need to be activated for computation in the current scene based on the content characteristics of the input image. Further, the routing decision is executed by a routing decision network. This network receives the general visual features output by the feature extraction network as input, performs global information aggregation and feature mapping on them, and finally outputs a routing weight vector. Further, the routing weight vector is an ordered set of values ​​used to indicate the task cluster branches activated from multiple task cluster branches. Its dimension is equal to the preset number of task cluster branches. Each component in the vector corresponds to a task cluster branch, and the value of the component determines whether the corresponding task cluster branch is activated. Further, a task cluster branch is an independent feature transformation branch in the network responsible for extracting features related to a specific task attribute cluster. Each task cluster branch contains several convolutional layers, normalization layers, and activation function layers to further transform the input general visual features into discriminative features related to the attribute cluster represented by that branch. This invention does not specifically limit the specific features of this branch. In one implementation, multiple task clusters include branches for extracting geometric attribute features, branches for extracting semantic attribute features, and branches for extracting dynamic analysis features. Geometric attributes refer to visual attributes related to an object's spatial location, shape, and size; further, semantic attributes refer to visual attributes related to an object's category and identity; and further, dynamic analysis refers to visual attributes related to temporal changes and motion patterns. This clustering design allows tasks with large attribute differences to use independent feature transformation channels, fundamentally mitigating the negative transfer phenomenon caused by tightly coupling all heterogeneous tasks into a single feature space.

[0033] In some preferred embodiments, the step of making routing decisions based on the general visual features and generating routing weights includes: performing global average pooling on the general visual features to obtain a feature vector; inputting the feature vector into a gating network to generate an original score corresponding to the number of branches in the task cluster; predicting a scene complexity coefficient based on the general visual features and using the scene complexity coefficient as a temperature coefficient; dividing the original score by the temperature coefficient and inputting it into a softmax function to obtain a probability distribution for each task cluster branch; and performing an argmax operation on the probability distribution to obtain the one-hot routing weights.

[0034] In practical implementation, in a dynamic routing multi-task architecture, the quality of routing decisions directly determines the rationality of the choice of task cluster branches, thus affecting overall computational efficiency and task accuracy. Routing decisions face the dual challenges of non-differentiability of discrete decisions and the diversity of scene complexity. Discrete decision operations cannot propagate gradients during backpropagation, hindering end-to-end training; the softmax function with a fixed temperature coefficient cannot simultaneously adapt to the different deterministic requirements of simple and complex scenarios. Specifically, in simple scenarios, it is desirable to make routing decisions as deterministic as possible to avoid allocating computational resources to unnecessary branches; in complex scenarios, it is desirable to retain a certain degree of randomness during the training phase to explore different branch combinations, because there may be multiple reasonable branch activation schemes in complex scenarios, and overly sharp decisions will prematurely exclude potentially better paths. To solve the above problems, this embodiment adopts an adaptive temperature coefficient mechanism, dynamically binding the temperature coefficient to the scene complexity of the input image, and generating actually executable branch pruning through argmax operations, achieving scene adaptive adjustment in balancing training exploration and inference determinism. The softmax function is a mathematical function that maps any real-valued vector to a probability distribution, where each component of its output vector is non-negative and the sum of all components is 1. The temperature coefficient is a hyperparameter in the softmax function, used to control the sharpness of the output probability distribution. The argmax operation retrieves the index of the maximum value in a vector, and its implementation is as follows: First, global average pooling is performed on the general visual features to obtain a feature vector. Global average pooling is an operation that takes the arithmetic mean of the feature values ​​at all locations in the spatial dimension. It compresses a feature map with spatial dimensions into a one-dimensional feature vector, where each element represents the average activation intensity of the corresponding channel across the entire spatial range. Next, the feature vector is input into a gating network to generate raw scores corresponding to the number of task cluster branches. The gating network is a small, fully connected neural network whose input is the feature vector after global average pooling, and whose output is a raw score vector with the same dimension as the number of task cluster branches. Each component corresponds to a task cluster branch, and the magnitude of the component reflects the original tendency to activate that branch under the current input image features.

[0035] Furthermore, based on general visual features, a miniature network predicts the scene complexity coefficient, which is then used as the temperature coefficient. The miniature network refers to a small neural network configured in parallel with the gated network. This miniature network takes the general visual features as input and performs a non-linear mapping of these features through several fully connected layers, outputting a scalar value. This scalar value is then processed by a preset range constraint function and used as the scene complexity coefficient. The miniature network and the gated network share the same input but have independent parameters. They are used to generate the scene complexity coefficient and the original score, respectively, allowing the temperature coefficient to be dynamically adjusted based on the content characteristics of the input image, rather than using fixed global hyperparameters.

[0036] Furthermore, the scene complexity coefficient is a scalar value reflecting the complexity of the current input image content. The micro-network and the gated network work in parallel, both taking general visual features as input. After passing through a small fully connected sub-network, the output is a scalar value, which, after value range constraint processing, serves as the temperature coefficient. The general visual features themselves encode the richness information of the scene, such as the spatial distribution dispersion of feature responses and the mean activation intensity. The micro-network can implicitly learn the mapping relationship between these patterns and scene complexity. In simple scenes, the information contained in the general visual features is relatively sparse and singular, resulting in a smaller scene complexity coefficient and a lower temperature coefficient predicted by the micro-network. In complex scenes, the information contained in the general visual features is rich and diverse, leading to a larger scene complexity coefficient and a higher temperature coefficient predicted by the micro-network.

[0037] Furthermore, the original scores are divided by the temperature coefficient and then input into the softmax function to obtain the probability distribution of each task cluster branch. When the temperature coefficient is low, the output of the softmax function tends to be a sharp distribution, with the probability quality highly concentrated on a few branches with the highest scores; when the temperature coefficient is high, the output of the softmax function tends to be a flat distribution, with the probability quality relatively evenly distributed across multiple branches. The argmax operation is then performed on the probability distribution to obtain the one-hot routing weights. One-hot routing weights refer to an encoding form where only one position in the vector is 1, and all other positions are 0. The task cluster branch corresponding to the position with a value of 1 in the one-hot vector is the activated branch, and the branch corresponding to the position with a value of 0 is the skipped branch. Directly using argmax to obtain the one-hot weights during the inference phase completely eliminates the problem of all branches requiring partial computation due to soft weights.

[0038] This embodiment achieves dynamic binding between the temperature coefficient and the input content by adaptively predicting the scene complexity coefficient from general visual features and using it as the temperature coefficient, instead of using global static hyperparameters. In simple scenes such as empty corridors, the temperature coefficient automatically decreases, the softmax output tends to have a sharp distribution, and the routing decision has high determinism, ensuring that only the most necessary branches are activated to minimize computational cost. In complex scenes such as crowded places, the temperature coefficient automatically increases, and the softmax output tends to have a gentle distribution. During the training phase, it can retain a moderate amount of random exploration space, which is conducive to the optimization process converging to a better routing strategy. During inference, deterministic decisions are still obtained through argmax, taking into account the different needs of training exploration and inference determinism. Furthermore, a micro-network is used to learn the temperature coefficient directly from general visual features end-to-end, without the need to manually set fixed temperature values ​​or manually design temperature decay scheduling tables. This enables the confidence of the routing decision to automatically match the scene characteristics, improving the robustness of the routing mechanism to changes in open and dynamic environments. The one-hot routing weights obtained by argmax give the branch activation decision a clear binary nature. Skipped branches do not participate in the forward computation at all, thus minimizing the actual computational load.

[0039] S4. Based on the routing weights, the general visual features are input into the activated task cluster branches for feature transformation to obtain task cluster features.

[0040] In practice, based on the routing weights, general visual features are input into the activated task cluster branches for feature transformation to obtain task cluster features. Specifically, the routing weights output by the routing decision network are converted into a branch activation mask. The branch activation mask is a vector composed of logical values, where the task cluster branch corresponding to a bit with a logically true value is activated, and the task cluster branch corresponding to a bit with a logically false value is skipped. Further, the general visual features are only fed into the activated task cluster branches. Each activated task cluster branch independently performs feature transformation operations on the general visual features, extracting discriminative features related to the attribute cluster represented by that branch, and outputting the specific feature map of that branch. Feature transformation refers to the process of converting the input feature map into an output feature map with different numbers of channels or different semantic levels through operations such as convolution, normalization, and activation. Further, the specific feature maps of all activated task cluster branches are aggregated into a whole, i.e., the task cluster features. The aggregation operation can be a concatenation operation along the channel dimension, that is, connecting multiple feature maps end-to-end along the channel dimension to form a feature map with a larger number of channels. Furthermore, skipping inactive branches means that convolution calculations, normalization operations, and activation function operations in these branches are all omitted, thereby saving computation time and power consumption for the corresponding branches and achieving true on-demand computation, rather than simply reducing the number of multiplications through sparse weights.

[0041] In some preferred embodiments, the feature extraction network, the task cluster branches, and the task-specific headers are deployed on a target edge computing chip, which includes multiple heterogeneous hardware computing units and a shared on-chip cache. The step of inputting the general visual features into the activated task cluster branches for feature transformation according to the routing weights to obtain task cluster features includes: storing the general visual features in the shared on-chip cache; determining the currently active task cluster branch set according to the routing weights; allocating the instruction streams of each task cluster branch in the active task cluster branch set to different heterogeneous hardware computing units; reading the general visual features from the shared on-chip cache through each heterogeneous hardware computing unit, executing the instruction streams of the allocated task cluster branches in parallel, generating specific feature maps for each task cluster branch, and writing the specific feature maps back to the shared on-chip cache; reading the specific feature maps of all active task cluster branches from the shared on-chip cache and performing channel concatenation to obtain the task cluster features.

[0042] In practical implementation, embodied intelligent devices typically utilize edge AI chips as inference computing platforms. These chips integrate multiple heterogeneous computing units and shared high-speed caches, possessing instruction-level or task-level parallel processing capabilities. However, in dynamic routing architectures, the set of activated task cluster branches changes dynamically with input samples, and each branch has different computational load and memory access patterns. Without specialized task allocation and data synchronization design tailored to the chip's hardware characteristics, the parallel execution of multiple branches may fail to fully utilize the hardware's potential due to memory access conflicts, load imbalance, or excessive synchronization overhead. To address this issue, this embodiment utilizes the shared on-chip cache within the target edge computing chip as the data exchange hub for each computing unit. The broadcasting of general visual features and the collection of output feature maps from each branch are uniformly completed through this shared cache. Simultaneously, the instruction streams of different task cluster branches are distributed in parallel to multiple heterogeneous computing units for execution, achieving branch-level spatial parallel acceleration. This transforms the total time consumed when activating multiple branches from serial accumulation into the maximum time consumed by each branch. The target edge computing chip refers to an embedded processor chip specifically designed for edge AI inference. Heterogeneous hardware computing units refer to different types of processing units integrated within a chip, such as neural network processing units responsible for accelerating matrix multiplication and convolution, central processing unit cores responsible for general scalar operations, and digital signal processor cores responsible for vector operations. Different types of processing units have different focuses in terms of computing power, energy efficiency, and applicable computing types. Shared on-chip cache refers to a high-speed static random access memory (SRAM) region located inside the chip and accessed by multiple heterogeneous hardware computing units. Its access latency is much lower than that of off-chip dynamic random access memory (DRAM), and it can be used by various computing units for high-speed data exchange and temporary storage of intermediate results. Static random access memory (SRAM) is a semiconductor memory that can retain data without periodic refresh and has the characteristics of fast read and write speeds.

[0043] Furthermore, the feature extraction network, task cluster branches, and task-specific heads are deployed on the target edge computing chip, which contains multiple heterogeneous hardware computing units and a shared on-chip cache. The specific implementation of the step of inputting general visual features into the activated task cluster branches for feature transformation to obtain task cluster features based on routing weights is as follows: General visual features are stored in a shared on-chip cache. After the feature extraction network extracts general visual features on the main computing unit of the chip, it writes the output feature map tensor data into a specific address range allocated in the shared on-chip cache. The address range refers to a continuous range of memory addresses in the shared on-chip cache. The set of currently active task cluster branches is determined based on the routing weights. The routing weights output by the routing decision network are parsed. Parsing refers to the process of reading the positions with a value of 1 in the one-hot routing weight vector and converting them into the corresponding task cluster branch identifiers, obtaining a list of task cluster branches that need to be activated for the current sample. Further, the instruction streams of each task cluster branch in the set of active task cluster branches are allocated to different heterogeneous hardware computing units. The instruction stream refers to the sequence of machine instructions in the compiled computation graph corresponding to each task cluster branch. Machine instructions refer to binary instructions that the processor can directly recognize and execute. Further, the task allocation module distributes the instruction streams of different branches to different computing units. The allocation decision can include the current idle state of each computing unit, the optimal matching relationship between the computing unit and the branch computation type, and the estimated computational load of each branch. Understandably, when the number of activated task cluster branches exceeds the number of available heterogeneous computing units, the allocation module queues the excess instruction streams according to a first-come, first-served or priority strategy, executing them immediately upon release by the computing units, thus ensuring the reliability of multi-branch concurrent processing. Furthermore, each heterogeneous hardware computing unit reads general visual features from the shared on-chip cache, executes the instruction streams of its allocated task cluster branches in parallel, generates specific feature maps for each task cluster branch, and writes these specific feature maps back to the shared on-chip cache. Further, each computing unit independently reads the required data blocks from the storage address of the general visual features in the shared on-chip cache, completes the convolution, activation, and other computational operations for that branch, and writes the specific feature map output back to the corresponding output address range of the shared on-chip cache. Furthermore, since the computation processes of each branch are independent and have no data dependencies, multiple computing units can execute their respective instruction streams in complete parallel, and the total execution time is determined by the branch with the longest execution time, rather than the sum of the times of each branch. Furthermore, specific feature maps of all active task cluster branches are read from the shared on-chip cache and channel-wise concatenated to obtain the task cluster features. Further, after all computing units have completed the instruction stream execution of their respective branches and written back their specific feature maps, the main control unit sequentially reads the specific feature maps of each branch from the shared on-chip cache and performs a concatenation operation along the channel dimension to form a merged feature map containing the output channels of all active branches; this is the task cluster feature. Further, channel concatenation refers to the operation of connecting multiple feature maps with the same spatial size but different numbers of channels along the channel dimension; the number of channels in the concatenated feature map is equal to the sum of the number of channels in each input feature map.

[0044] This embodiment eliminates the high latency overhead of data exchange between different computing units via off-chip memory by uniformly storing general visual features and intermediate feature maps of each branch in a shared on-chip cache of the target edge computing chip. Parallel reading of general visual features and write-back of branch output feature maps by each computing unit are both completed in high-speed on-chip storage, reducing the proportion of data transfer time in the overall inference latency to an extremely low level. Furthermore, the parallel allocation and execution of task cluster branch instruction streams on different heterogeneous hardware computing units transforms the serial execution of multiple branches into spatially parallel computation. The total latency of task cluster feature transformation is reduced from the sum of the times of all activated branches to the maximum time of a single branch. In complex scenarios with multiple activated branches, the inference latency can be reduced to a fraction of that of serial execution, significantly improving the real-time response capability of embodied intelligent devices in multi-task processing scenarios. Furthermore, the rational use of heterogeneous hardware computing units allows different types of computing tasks to be assigned to the most suitable hardware engine for execution. For example, convolution-intensive branches are deployed on neural network processing units, while branches containing complex control logic are deployed on the core of the central processing unit. This further optimizes the overall energy efficiency ratio, enabling a limited power consumption budget to support more concurrent tasks.

[0045] In some preferred embodiments, the step of reading the general visual features from the shared on-chip cache through each of the heterogeneous hardware computing units, executing the instruction stream of the allocated task cluster branches in parallel, and generating specific feature maps for each task cluster branch includes: for each activated task cluster branch, determining the dynamic fusion coefficients of multiple parallel sub-branches within the activated task cluster branch according to the branch weights corresponding to the activated task cluster branch in the routing weights; inputting the general visual features into the multiple parallel sub-branches respectively, wherein the multiple parallel sub-branches have convolution kernels of different sizes to obtain multiple sub-branch output feature maps; multiplying the multiple sub-branch output feature maps by the corresponding dynamic fusion coefficients respectively, and then summing them element-wise to obtain the specific feature map of the activated task cluster branch.

[0046] In practical implementation, in a multi-task dynamic routing architecture, each activated task cluster branch typically uses a set of convolutional layers with a fixed structure to process general visual features when performing feature transformation. However, different input images exhibit significant differences in texture details, target scale, and scene spatial frequency, making it difficult for a fixed single-path feature transformation to adequately adapt to diverse scene characteristics. For example, detail analysis of large targets at close range requires a smaller receptive field to focus on local textures, while contour recognition of small targets at long distances requires a larger receptive field to capture the global context. The receptive field refers to the size of the region on the input image corresponding to a pixel on the output feature map of a certain layer in a convolutional neural network. To address the problem of insufficient adaptability of the single-path feature transformation, this embodiment designs a multi-scale parallel sub-branch structure within each task cluster branch, and dynamically controls the fusion ratio of each sub-branch using routing weights. This allows features of different scales to be adaptively combined within the same branch based on the strength of the routing signal, achieving fine-grained dynamic adjustment of feature aggregation within the branch. The specific implementation method is as follows: For each activated task cluster branch, the dynamic fusion coefficients of multiple parallel sub-branches within that activated task cluster branch are determined based on the branch weight corresponding to that activated task cluster branch in the routing weights. The branch weight is a scalar component in the routing weight vector that indicates the activation intensity of the task cluster branch. In the case of one-hot routing weights, the branch weight is 1, but during training or in soft routing scenarios, the branch weight can be a continuous value between 0 and 1. This branch weight not only determines whether the task cluster branch is activated but is also reused as an upper-level guidance signal to control the fusion ratio of multi-scale sub-branches within the branch. Furthermore, each task cluster branch has multiple pre-defined parallel sub-branches, with different sub-branches using convolutional kernels of different sizes. For example, one sub-branch uses a 3x3 convolutional kernel, another uses a 5x5 convolutional kernel, and yet another uses a 7x7 convolutional kernel. The convolutional kernel size refers to the width and height of the convolutional kernel in spatial dimensions, in pixels. Furthermore, different sized convolutional kernels provide different receptive field sizes; small convolutional kernels focus on local detail features, while large convolutional kernels capture global structural features. The dynamic fusion coefficients are a set of weighted coefficients equal to the number of sub-branches, generated by the branch weights through a small mapping function. This mapping function can be a simple linear mapping or a softmax-normalized mapping. Further, general visual features are input into multiple parallel sub-branches, resulting in multiple sub-branch output feature maps. The general visual features are simultaneously fed into all parallel sub-branches within the task cluster branch. Each sub-branch independently undergoes convolution, normalization, and activation operations, outputting its own sub-branch output feature map. Further, the multiple sub-branch output feature maps are multiplied by their corresponding dynamic fusion coefficients and then summed element-wise to obtain the specific feature map of the activated task cluster branch. Further, element-wise summation refers to the operation of adding the element values ​​at the same position on multiple feature maps with identical shapes. Further, since the dynamic fusion coefficients are derived from the branch weights, when the branch weights are high, the fusion coefficients tend to favor sub-branches with large convolution kernels, and when the branch weights are low, the fusion coefficients tend to favor sub-branches with small convolution kernels, achieving adaptive selection and fusion of multi-scale features within the branch.

[0047] This embodiment introduces parallel multi-scale convolutional sub-branches within each task cluster branch and dynamically generates the fusion coefficients of each sub-branch through branch weights. This allows the same task cluster branch to adaptively adjust its effective receptive field size and feature extraction mode when facing different inputs, enriching the feature representation capability for targets at different scales without increasing the total number of task cluster branches. Furthermore, in addition to controlling branch-level activation, branch weights are reused for multi-scale dynamic fusion within branches, extending the guidance signal for routing decisions from coarse-grained selection at the branch level to fine-grained feature aggregation at the sub-branch level. Under the global control of routing decisions, the entire network achieves multi-level adaptive computation from inter-branch to intra-branch, systematically enhancing the adaptability of the multi-task processing architecture to scene diversity and effectively solving the problem of limited representation capabilities within branches in fixed multi-branch networks.

[0048] S5, input the task cluster features into the task-specific header corresponding to the activated task cluster branch to obtain the task processing result.

[0049] In practice, task cluster features are input into the task-specific head corresponding to the activated task cluster branch to obtain the task processing result. The task-specific head is a lightweight neural network substructure responsible for mapping task cluster features to the final output of a specific task. It typically contains several fully connected or convolutional layers. A fully connected layer is a network layer where all neurons in adjacent layers have connection weights. Furthermore, each task cluster branch is associated with one or more task-specific heads. For example, a geometric attribute task cluster branch can be associated with an object pose estimation head and a depth estimation head, while a semantic attribute task cluster branch can be associated with a person detection head and an identity re-identification head. Identity re-identification refers to the task of matching and identifying the same person in a cross-camera scenario. Further, the task-specific head receives a subset of features related to its task from the task cluster features, processes them, and outputs the result corresponding to the task, such as target bounding box coordinates, object pose matrix, or behavior category label. Since only the task-specific head corresponding to the activated task cluster branch participates in the computation, tasks associated with unactivated branches do not produce output in this inference. Subsequent upper-layer applications can perform corresponding business logic processing based on the actual type of the output task result.

[0050] This embodiment introduces a dynamic routing decision mechanism between the feature extraction network and the task processing branches. This allows the network to adaptively select which task cluster branches to activate based on the complexity of the input image. For images in open scenes containing only simple targets, it skips the computation of complex branches such as geometric attributes and dynamic analysis. For images in crowded scenes containing multiple people and objects, it activates all or most branches for comprehensive analysis, achieving a flexible computing mode that allocates computing power on demand. Furthermore, since many simple scene input images do not require full computation of all task cluster branches, the average inference computation is significantly reduced compared to multi-task networks with fixed full computation. Inference latency and power consumption are correspondingly reduced, resulting in significant improvements in both battery life and real-time response for embodied intelligent devices. Furthermore, the task cluster branches are grouped according to attribute clusters such as geometric attributes, semantic attributes, and dynamic analysis. This design allows tasks with similar attributes to share specialized features extracted by the same branch, while tasks with large attribute differences use independent features extracted by different branches. This effectively suppresses negative transfer between heterogeneous tasks while sharing general visual features at the underlying level. Furthermore, negative transfer refers to the phenomenon in multi-task learning where the learning process of one task adversely affects the performance of other tasks. The inference accuracy of each task is improved compared to the hard-sharing scheme where all tasks share a single feature representation.

[0051] In some preferred embodiments, the step of inputting the task cluster features into a task-specific header corresponding to the activated task cluster branch to obtain a task processing result includes: determining the currently activated task type based on the routing weight; calling at least one task-specific header matching the task type from a pre-configured set of task headers, each task-specific header containing a dynamic convolutional kernel generation subnetwork; inputting a subset of features from the task cluster features corresponding to the at least one task-specific header into the dynamic convolutional kernel generation subnetwork to generate convolutional kernel parameters for the task-specific header; and performing convolution processing on the feature subset using the generated convolutional kernel parameters to obtain the task processing result.

[0052] In practical implementation, in a dynamic routing multi-task architecture, the same task cluster branch may serve multiple different types of downstream tasks. For example, the output features of the geometric attribute task cluster branch need to simultaneously support object pose estimation and depth estimation tasks, and these tasks have different preferences for the feature's region of interest and feature transformation method. If a convolutional kernel with fixed parameters is used to process all input features, the adaptability of the task head will be limited, especially when different task combinations cause changes in the statistical distribution of the input feature subset, the fixed parameters cannot be flexibly adjusted. In addition, different routing weights lead to differences in the task combinations activated for each inference, and the content characteristics of the feature subset input to the task-specific head under different task combinations may also change accordingly. If the task head parameters remain unchanged, it is difficult to fully adapt to such dynamic changes, becoming a bottleneck restricting overall performance. To solve the problem that the fixed parameters of the task head cannot adapt to dynamic input changes, this embodiment introduces a dynamic convolutional kernel generation subnetwork in each task-specific head, which predicts the convolutional kernel parameters required by the task head online from the input feature subset, so that the computational logic of the task head changes dynamically with the input content, thereby achieving end-to-end input adaptation from feature extraction to final output. The specific implementation method is as follows: The currently active task type is determined based on the routing weights. Since the routing weights indicate which task cluster branches are activated, and each task cluster branch is associated with a set of predefined task types, the specific tasks currently activated by the inference can be inferred from the routing weights. Further, at least one task-specific head matching the task type is invoked from a pre-configured set of task heads. The task head set is the total set of all available task-specific heads in the network, each with a unique identifier and a binding relationship with one or more task types. Further, each task-specific head contains a dynamic convolutional kernel generation subnetwork. The dynamic convolutional kernel generation subnetwork is a lightweight convolutional kernel parameter prediction network whose function is to generate online the convolutional kernel parameters used by the task-specific head during convolution operations based on the specific task cluster features of the current input, instead of using fixed convolutional kernels learned during training. The dynamic convolutional kernel generation subnetwork includes a global context aggregation layer and a fully connected mapping layer. The global context aggregation layer compresses and encodes the input feature subset in spatial dimensions to obtain a compact feature description vector. The fully connected mapping layer maps the feature description vector to the convolutional kernel weight parameters required for the task-specific head to perform convolution operations. Further, a feature subset corresponding to at least one task-specific head in the task cluster features is input into the dynamic convolutional kernel generation subnetwork to generate the convolutional kernel parameters for the task-specific head. Further, the feature subset is the set of channels extracted from the task cluster features by channel index, corresponding to the task cluster branch associated with the task-specific head. Further, the dynamic convolutional kernel generation subnetwork performs global context aggregation and feature mapping on the input feature subset. Global context aggregation refers to the operation of encoding the global spatial information of the feature map into a compact vector, outputting a set of tensor data, which, after shape reshaping, directly constitutes the weight parameters of the convolutional kernel. Further, shape reshaping refers to the operation of changing the tensor dimension arrangement without changing the total amount of data. Further, the generated convolutional kernel parameters are used to perform convolution processing on the feature subset to obtain the task processing result. In this way, for the same task header, different convolution kernels will be generated when faced with different inputs, thus achieving input adaptation in the computation process.

[0053] This embodiment introduces a dynamic convolutional kernel generation subnetwork into each task-specific head. This allows the specific convolutional kernel parameters used by the task head to process input features to be generated online by the input features themselves through the subnetwork, rather than using a fixed set of parameters determined during training. This achieves efficient task processing with feature adaptation. Furthermore, the dynamically generated convolutional kernel parameters are highly correlated with the content characteristics of the current input feature subset, enabling the task-specific head to adaptively adjust based on the activated task combination and the statistical characteristics of the feature subset within that combination during each inference. Compared to fixed convolutional kernels, this provides superior generalization ability and feature adaptability, making it particularly suitable for scenarios with frequently changing task combinations under dynamic routing architectures. Moreover, differences in routing weights not only affect the computational path selection at the task cluster branch level but also indirectly transmit the differences in the content of task cluster features to the generation of convolutional kernel parameters within the task-specific head. The entire network achieves deep adaptive computation of input content across the entire chain from general feature extraction to final task output, ensuring the network maintains high-efficiency task processing capabilities when facing diverse scenarios and task combinations.

[0054] In some preferred embodiments, the parameters of the feature extraction network, the task cluster branches, the task-specific head, and the gating network used for routing decisions are obtained through the following training steps: acquiring training images and corresponding multi-task labels; inputting the training images into the feature extraction network to obtain training general visual features; inputting the training general visual features into each task cluster branch to obtain specific feature maps for each task cluster branch; inputting the training general visual features into the gating network, adding Gumbel noise, and generating training soft routing weights via a softmax function; weighted aggregation of the specific feature maps of each task cluster branch according to the training soft routing weights to obtain training task cluster features, and obtaining prediction results through the task-specific head; calculating the task loss based on the prediction results and the multi-task labels, and calculating the entropy regularization loss based on the training soft routing weights; updating the parameters of the feature extraction network, the task cluster branches, the task-specific head, and the gating network with the goal of minimizing the weighted sum of the task loss and the entropy regularization loss.

[0055] In practical implementation, routing decisions in a dynamic routing architecture are inherently discrete, meaning each branch undergoes a binary choice of activation or deactivation. Discrete operations result in zero gradients in the standard backpropagation algorithm, making direct end-to-end training impossible. Backpropagation is the core algorithm for training neural networks, updating parameters by calculating the gradient of the loss function with respect to the parameters of each network layer and propagating the gradient back layer by layer along the network. Furthermore, if the routing gating network prematurely converges to a strategy that only activates one or two fixed branches during training, it will lose its ability to explore and adapt to different scenarios, causing the parameters of inactive branches to stop updating and their capabilities to degrade. When encountering complex scenarios that actually require these branches, it will be unable to respond properly. To address the dual problems of discrete non-differentiability and premature convergence leading to branch degradation, this embodiment introduces Gumbel noise and entropy regularization loss. During the training phase, discrete decisions are relaxed into differentiable soft decisions to open up gradient backpropagation paths. Simultaneously, entropy regularization encourages the gating network to output a low-entropy but non-degenerate decision distribution, ensuring that all branches continuously receive gradient updates throughout training and maintaining their feature extraction capability reserves. Gumbel noise refers to random noise sampled from the Gumbel distribution, an extreme value distribution often used for continuous relaxation of discrete sampling. Entropy regularization loss is a regularization term constructed based on information entropy. Information entropy is a measure of the uncertainty of a probability distribution; a larger entropy value indicates a more uniform distribution, while a smaller entropy value indicates a more concentrated distribution. Specifically, entropy regularization loss is a loss term constructed by calculating the information entropy based on the probability distribution of the trained soft-weights and aiming to minimize this information entropy. It is used to encourage routing decisions to tend towards determinism and avoid outputting ambiguous soft-weight distributions.

[0056] First, training images and their corresponding multi-task labels are obtained. Training images are a set of labeled sample images collected from the operating scenarios of embodied intelligent devices. Each sample is accompanied by labels for multiple tasks, such as bounding box coordinates for people, behavior category labels, and six-DOF pose labels for objects. Further, a bounding box refers to a rectangular box that selects the target object in the image, defined by its center coordinates and width and height. Next, the training images are input into a feature extraction network to obtain training general visual features. These general visual features are then input into each task cluster branch to obtain specific feature maps for each task cluster branch. Further, during the training phase, regardless of which branches are ultimately activated by the routing decision, all task cluster branches first undergo a complete forward computation to obtain the output feature maps of each branch for subsequent weighted aggregation, thus ensuring that each branch receives gradient updates on each batch of training data and maintains its representational ability. Finally, the training general visual features are input into a gating network, and after adding Gumbel noise, the training soft routing weights are generated via a softmax function. Furthermore, the structure of the gating network is the same as that of the inference stage. It takes the feature vector obtained after global average pooling of the trained general visual features as input and outputs the original score. Further, Gumbel noise is added to the original score, and then input into a softmax function with a temperature coefficient to generate the training soft routing weights. Further, the Gumbel-Softmax technique relaxes the non-differentiable discrete argmax operation into a differentiable continuous approximation operation by injecting controllable random noise into the discrete sampling process. The training soft routing weights are no longer one-hot vectors, but rather continuous probability vectors that are approximately one-hot but have small probability values ​​in each component. This vector is differentiable with respect to the gating network parameters, allowing the gradient to be successfully backpropagated through the routing decision stage. Further, the specific feature maps of each task cluster branch are weighted and aggregated according to the training soft routing weights to obtain the training task cluster features, and the prediction result is obtained through the task-specific head. Weighted aggregation refers to multiplying the specific feature maps of each task cluster branch by the corresponding scalar weights in the training soft routing weights, and then performing weighted concatenation or weighted summation along the channel dimension. Furthermore, task loss is calculated based on the prediction results and multi-task labels. Task loss is a scalar function that measures the difference between the model's prediction results and the true labels. Commonly used task losses include cross-entropy loss and mean squared error loss. Cross-entropy loss is a loss function used to measure the difference between two probability distributions, while mean squared error loss is the expected value of the square of the difference between the predicted value and the true value. Further, entropy regularization loss is calculated based on the trained soft-router weights. With the goal of minimizing the weighted sum of task loss and entropy regularization loss, the parameters of the feature extraction network, task cluster branches, task-specific heads, and gating network are updated.Furthermore, in the process of minimizing the total loss, the task loss drives the model to learn and produce correct multi-task inference results, while the entropy regularization loss drives the output of the gating network to tend towards low entropy, i.e. deterministic decisions, encouraging routing decisions to converge to explicit activation branch selections, and avoiding ambiguous soft weights that would prevent effective pruning of branch computations during actual deployment.

[0057] This embodiment relaxes discrete routing decisions into differentiable soft decisions using the Gumbel-Softmax technique during the training phase, establishing a full-link gradient backpropagation channel from task loss to gating network parameters. This allows the routing decision network to perform joint end-to-end optimization with the feature extraction network and task processing branches. Driven by the unified overall network optimization goal, all components collaboratively converge to a globally optimal operating point, avoiding the suboptimal optimization problems caused by phased training. Furthermore, the introduction of entropy regularization loss explicitly encourages deterministic and sparsity in routing decisions at the optimization goal level. During training, it gradually guides the network to learn to make clear branch activation choices for different scenarios, rather than remaining in a vague state of assigning similar weights to all branches. This ensures that the trained model can effectively reduce the actual computational cost during inference deployment through one-hot routing weights. Furthermore, during the training phase, all branches complete forward computation and are weighted and aggregated using soft weights, enabling each branch to participate in every round of gradient updates. Even if a branch has low weights on most samples, it will not degenerate due to a lack of training samples, thus maintaining the feature extraction capability reserves of all branches and making it possible to activate each branch as needed during routing decisions during inference.

[0058] In some preferred embodiments, updating the parameters of the feature extraction network, the task cluster branches, the task-specific head, and the gating network with the goal of minimizing the weighted sum of the task loss and the entropy regularization loss includes: obtaining the pre-calibrated computational resource consumption values ​​of each task cluster branch; calculating the expected computational resource consumption of the current training sample based on the training soft route weights and the computational resource consumption values; calculating the deviation between the expected computational resource consumption and the preset target resource limit to obtain the resource loss; adding the resource loss to the weighted sum of the task loss and the entropy regularization loss to obtain the total loss; and updating the parameters of the feature extraction network, the task cluster branches, the task-specific head, and the gating network with the goal of minimizing the total loss.

[0059] In practical implementation, embodied intelligent devices typically have strict real-time requirements in real-world deployment scenarios. For example, the total latency between visual perception and planning control must be kept within 30ms. Dynamic routing networks trained solely for task accuracy may tend to activate too many branches in complex scenarios to achieve high accuracy, leading to peak computational loads exceeding the device's real-time processing capabilities. Traditionally, this involves model pruning or manually setting routing thresholds after training, but this introduces additional hyperparameter tuning costs, and accuracy recovery after pruning requires additional fine-tuning steps. To address the disconnect between resource constraints during training and deployment, this embodiment introduces a penalty term in the training loss function to account for the deviation between expected computational resource consumption and target resource constraints. This directly encodes hardware real-time constraints into the optimization objective, enabling the routing strategy to internalize computational resource awareness during training and automatically learn to achieve the optimal trade-off between accuracy and real-time performance. The specific implementation method is as follows: First, the pre-calibrated computational resource consumption values ​​for each task cluster branch are obtained. Pre-calibration involves performing an independent computational cost analysis on each task cluster branch before training begins, including the number of floating-point operations or multiplication-accumulation operations required for that branch to complete one forward inference operation. Further, the computational resource consumption values ​​for each task cluster branch are stored as a one-dimensional lookup table, which is a data structure that takes an index as input and the corresponding value as output. The expected computational resource consumption of the current training sample is calculated based on the training soft route weights and the computational resource consumption values. Further, the calculation method involves multiplying the weight component corresponding to each branch in the training soft route weights by the computational resource consumption value of that branch and summing the results. The result is the expected number of floating-point operations for the current sample under the soft route decision, representing the average computational cost under the current routing strategy. Finally, the deviation between the expected computational resource consumption and the preset target resource limit is calculated to obtain the resource loss. The preset target resource limit is the upper limit of acceptable computing resource consumption per inference, calculated based on the real-time requirements of edge devices. It can be calculated based on the peak computing power of the target chip (e.g., floating-point operations per second) and the allowed single inference time budget, to determine the maximum allowed floating-point operation threshold per inference, which serves as the target resource limit. For example, a 30ms latency requirement can be converted to no more than a certain number of floating-point operations. Furthermore, the deviation can be calculated using mean squared error or the Huber loss function. The Huber loss is a robust loss function combining mean squared error and absolute error; it exhibits mean squared error when the deviation is small and absolute error when the deviation is large, providing better robustness to outliers. Further, the resource loss is added to the weighted sum of the task loss and entropy regularization loss to obtain the total loss. With the goal of minimizing the total loss, the parameters of the feature extraction network, task cluster branches, task-specific heads, and gating networks are updated. Furthermore, the existence of resource loss incentivizes the network to choose a lower expected computational cost while pursuing high accuracy. When the expected computational cost is lower than the target limit, the resource loss is small and has a negligible impact on the accuracy target. When the expected computational cost exceeds the target limit, the resource loss increases rapidly, forcing the routing decision to adjust towards a sparser direction.

[0060] This embodiment explicitly incorporates hardware real-time constraints into the total loss function of model training as a resource penalty. This allows routing decision optimization to move beyond solely focusing on task accuracy and instead employs a multi-objective trade-off between accuracy and computational cost. During training, the resource penalty term imposes a proportional penalty on decisions that activate computationally expensive branches, with the penalty proportional to the target deviation. During backpropagation, the network automatically learns to find the Pareto optimal activation strategy between accuracy gain and computational resource consumption. Furthermore, for simple samples, it automatically tends to activate low-computational branches to meet resource constraints, while for complex samples that truly require multi-branch collaboration for accurate processing, it allows for a slight exceedance of the target limit to ensure accuracy. This dynamic trade-off requires no manual adjustment afterward. Moreover, when the trained model is deployed to edge devices, its routing decision strategy, already shaped as resource-aware during training, can autonomously meet real-time requirements during inference without relying on additional post-processing pruning or online adjustments. This significantly simplifies the deployment process and improves the model's portability across different hardware platforms.

[0061] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0062] Please see Figure 2 , Figure 2 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a terminal or a server, wherein the server can be a standalone server or a server cluster composed of multiple servers.

[0063] The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0064] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, it enables the processor 502 to perform an embodied intelligent multitasking method based on dynamic routing.

[0065] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0066] The internal memory 504 provides an environment for the execution of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute an embodied intelligent multitasking method based on dynamic routing.

[0067] The network interface 505 is used for network communication with other devices. Those skilled in the art will understand that the above structure is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. A specific computer device 500 may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements.

[0068] The processor 502 is used to run a computer program 5032 stored in a memory to implement the steps of the embodied intelligent multitasking method based on dynamic routing provided in any of the above method embodiments.

[0069] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0070] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program may be stored in a storage medium, which is a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0071] Therefore, the present invention also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program. When executed by a processor, the computer program causes the processor to perform the steps of the embodied intelligent multitasking method based on dynamic routing provided in any of the above method embodiments.

[0072] The storage medium is a physical, non-transient storage medium, such as a USB flash drive, external hard drive, read-only memory (ROM), magnetic disk, or optical disk, or any other physical storage medium capable of storing program code. The computer-readable storage medium can be non-volatile or volatile.

[0073] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0074] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0075] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the device of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0076] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0077] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0078] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, this invention is also intended to include these modifications and variations as long as they fall within the scope of the claims of this invention and their equivalents.

[0079] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for embodied intelligent multitasking based on dynamic routing, characterized in that, include: Obtain the input image; The input image is fed into a feature extraction network to extract general visual features; Routing decisions are made based on the general visual features, and routing weights are generated. The routing weights are used to indicate the task cluster branches activated from multiple task cluster branches. Based on the routing weights, the general visual features are input into the activated task cluster branches for feature transformation to obtain task cluster features; The task cluster features are input into the task-specific header corresponding to the activated task cluster branch to obtain the task processing result.

2. The embodied intelligent multitasking method based on dynamic routing according to claim 1, characterized in that, The feature extraction network is composed of multiple depthwise separable convolutional modules stacked together. Each depthwise separable convolutional module includes a channel-wise convolutional layer, a channel attention mechanism module, and a pointwise convolutional layer. The step of inputting the input image into the feature extraction network to extract general visual features includes: The input image is subjected to standard convolution processing to obtain an initial feature map; The initial feature map is sequentially passed through multiple depthwise separable convolutional modules. In each depthwise separable convolutional module, spatial features are first extracted through a channel-wise convolutional layer, then adaptively weighted through a channel attention mechanism module, and finally fused through a pointwise convolutional layer. The feature map output by the last depthwise separable convolutional module is used as the general visual feature.

3. The embodied intelligent multitasking method based on dynamic routing according to claim 1, characterized in that, The step of making routing decisions based on the general visual features and generating routing weights includes: Global average pooling is performed on the general visual features to obtain the feature vector; The feature vector is input into a gating network to generate raw scores corresponding to the number of branches in the task cluster; Based on the general visual features, the scene complexity coefficient is predicted, and the scene complexity coefficient is used as the temperature coefficient. Divide the original score by the temperature coefficient and input it into the softmax function to obtain the probability distribution of each task cluster branch; Perform the argmax operation on the probability distribution to obtain the routing weights in one-hot form.

4. The embodied intelligent multitasking method based on dynamic routing according to claim 1, characterized in that, The feature extraction network, the task cluster branch, and the task-specific header are deployed on the target edge computing chip, which includes multiple heterogeneous hardware computing units and a shared on-chip cache. The step of inputting the general visual features into the activated task cluster branch for feature transformation according to the routing weight to obtain task cluster features includes: The general visual features are stored in the shared on-chip cache. The currently active task cluster branch set is determined based on the routing weight; The instruction streams of each task cluster branch in the activated task cluster branch set are respectively allocated to different heterogeneous hardware computing units; The heterogeneous hardware computing units read the general visual features from the shared on-chip cache, execute the instruction stream of the assigned task cluster branches in parallel, generate specific feature maps for each task cluster branch, and write the specific feature maps back to the shared on-chip cache. The specific feature maps of all active task cluster branches are read from the shared on-chip cache and channel splicing is performed to obtain the task cluster features.

5. The embodied intelligent multitasking method based on dynamic routing according to claim 4, characterized in that, The process of reading the general visual features from the shared on-chip cache through each of the heterogeneous hardware computing units, executing the instruction stream of the allocated task cluster branches in parallel, and generating specific feature maps for each task cluster branch includes: For each activated task cluster branch, the dynamic fusion coefficient of multiple parallel sub-branches within the activated task cluster branch is determined based on the branch weight corresponding to the activated task cluster branch in the routing weight. The general visual features are input into the multiple parallel sub-branches, and the multiple parallel sub-branches have convolution kernels of different sizes to obtain multiple sub-branch output feature maps; After multiplying the output feature maps of the multiple sub-branches by their respective dynamic fusion coefficients, the elements are added together to obtain the specific feature map of the activated task cluster branch.

6. The embodied intelligent multitasking method based on dynamic routing according to claim 1, characterized in that, The parameters of the feature extraction network, the task cluster branch, the task-specific header, and the gating network used for routing decisions are obtained through the following training steps: Obtain training images and their corresponding multi-task labels; The training images are input into the feature extraction network to obtain training general visual features; The trained general visual features are input into each task cluster branch to obtain a specific feature map for each task cluster branch. The general visual features for training are input into the gating network, and Gumbel noise is added before the softmax function is used to generate training soft route weights. The specific feature maps of each task cluster branch are weighted and aggregated according to the training soft route weights to obtain the training task cluster features, and the prediction results are obtained through the task-specific header. The task loss is calculated based on the prediction results and the multi-task labels, and the entropy regularization loss is calculated based on the training soft route weights. With the goal of minimizing the weighted sum of the task loss and the entropy regularization loss, the parameters of the feature extraction network, the task cluster branch, the task-specific head, and the gating network are updated.

7. The embodied intelligent multitasking method based on dynamic routing according to claim 6, characterized in that, The step of updating the parameters of the feature extraction network, the task cluster branch, the task-specific head, and the gating network with the objective of minimizing the weighted sum of the task loss and the entropy regularization loss includes: Obtain the pre-calibrated computational resource consumption values ​​for each task cluster branch; Calculate the expected computational resource consumption of the current training sample based on the training soft route weights and the computational resource consumption value; The deviation between the expected computational resource consumption and the preset target resource limit is calculated to obtain the resource loss; The total loss is obtained by adding the resource loss to the weighted sum of the task loss and the entropy regularization loss. With the goal of minimizing the total loss, the parameters of the feature extraction network, the task cluster branch, the task-specific head, and the gating network are updated.

8. The embodied intelligent multitasking method based on dynamic routing according to claim 1, characterized in that, The step of inputting the task cluster features into the task-specific header corresponding to the activated task cluster branch to obtain the task processing result includes: The currently active task type is determined based on the routing weight; At least one task-specific header matching the task type is invoked from a pre-configured set of task headers, each task-specific header containing a dynamic convolutional kernel to generate a sub-network; The feature subset corresponding to at least one task-specific head in the task cluster features is input into the dynamic convolution kernel generation subnetwork to generate the convolution kernel parameters of the task-specific head; The feature subset is convolved using the generated convolution kernel parameters to obtain the task processing result.

9. A computer device, characterized in that, The computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, can implement the method as described in any one of claims 1-8.

Images