Task scheduling method and electronic equipment

Abstract

The invention discloses a task scheduling method and electronic equipment. Belongs to the technical field of communication. The method comprises the steps that a graph data structure is generated based on entities in the multi-device ecological network and the incidence relation between the entities, and the entities comprise devices in the multi-device ecological network; extracting global features of the graph data structure through a graph neural network; obtaining multi-modal data of each device, and performing fusion processing on the multi-modal data based on the global features through the converter network to generate fusion features; identifying a user intention based on the fusion feature, and determining a target task and a target device in the multi-device ecological network based on the user intention; and sending the target task to the target device.

Description

Technical Field

[0001] This application relates to the field of communication technology, specifically to task scheduling methods and electronic devices. Background Technology

[0002] With the widespread adoption of the Internet of Things (IoT) and smart devices, users often simultaneously use mobile phones, wearable devices, extended reality devices, smart speakers, in-vehicle systems, and home appliances, forming a complex multi-device ecosystem. These devices continuously collect multimodal data, including visual, voice, text, motion, and environmental data.

[0003] In existing technologies, each device and the data it collects are independent of each other, requiring users to manually switch between different devices and configure tasks, which cannot provide users with intelligent and automated services. Summary of the Invention

[0004] The purpose of this application is to provide a task scheduling method and an electronic device that can provide users with intelligent and automated services, thereby improving the level of intelligent services in a multi-device ecosystem.

[0005] In a first aspect, embodiments of this application provide a task scheduling method, which includes: generating a graph data structure based on entities in a multi-device ecosystem network and the relationships between entities, wherein the entities include each device in the multi-device ecosystem network; extracting global features of the graph data structure through a graph neural network; acquiring multimodal data of each device; performing fusion processing on the multimodal data based on the global features through a converter network to generate fused features; identifying user intent based on the fused features; determining a target task and a target device in the multi-device ecosystem network based on the user intent; and sending the target task to the target device.

[0006] Secondly, embodiments of this application provide a task scheduling apparatus, comprising: a generation unit, configured to generate a graph data structure based on entities in a multi-device ecosystem network and the relationships between entities, wherein the entities include each device in the multi-device ecosystem network; an extraction unit, configured to extract global features of the graph data structure using a graph neural network; a fusion unit, configured to acquire multimodal data of each device, and perform fusion processing on the multimodal data based on the global features using a transformer network to generate fused features; a determination unit, configured to identify user intent based on the fused features, and determine a target task and a target device in the multi-device ecosystem network based on the user intent; and a sending unit, configured to send the target task to the target device.

[0007] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory stores programs or instructions executable on the processor, and the programs or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0008] Fourthly, embodiments of this application provide a readable storage medium on which a computer program is stored, and when executed by a processor, the computer program implements the steps of the method described in the first aspect above.

[0009] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method described in the first aspect.

[0010] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method described in the first aspect.

[0011] In this embodiment, a graph data structure is first generated based on entities and their relationships within a multi-device ecosystem network. Entities include all devices within the network. Then, global features of the graph data structure are extracted using a graph neural network. Next, multimodal data from each device is acquired, and a transformer network is used to fuse this data based on the global features, generating fused features. Then, user intent is identified based on these fused features, and the target task and target device within the multi-device ecosystem network are determined accordingly. Finally, the target task is sent to the target device. In this process, multimodal data is deeply fused by analyzing the relationships between devices. User intent is understood based on the multimodal fusion results, and tasks are automatically assigned to the most suitable device for execution. This achieves deep fusion of cross-device data, intelligent intent understanding, and automatic task generation and allocation. This enables the automatic generation and execution of optimal task scheduling schemes, providing users with intelligent and automated services without requiring manual switching and task configuration between different devices, thus improving the intelligent service level of the multi-device ecosystem. Attached Figure Description

[0012] Figure 1 This is a flowchart of the task scheduling method provided in the embodiments of this application; Figure 2 This is a schematic diagram of the processing procedure of the task scheduling method provided in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of the task scheduling device provided in the embodiments of this application; Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application; Figure 5 This is a schematic diagram of the hardware structure of an electronic device suitable for implementing the embodiments of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0014] The terms "first," "second," etc., used in this application's specification are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, in the specification, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects have an "or" relationship.

[0015] The task scheduling method and apparatus provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0016] Please refer to Figure 1 This document illustrates one of the flowcharts of a task scheduling method provided in an embodiment of this application. The task scheduling method provided in this application can be applied to electronic devices. For example, the aforementioned electronic device may be a smartphone, tablet computer, laptop computer, wearable device, etc.

[0017] The task scheduling method provided in this application includes the following steps: Step 101: Generate a graph data structure based on the entities in the multi-device ecosystem network and the relationships between them. The entities include each device in the multi-device ecosystem network.

[0018] In this embodiment, a multi-device ecosystem refers to a distributed computing system comprised of multiple heterogeneous smart terminal devices owned or controlled by a user and physically or logically related, interconnected via wired or wireless networks. Devices in a multi-device ecosystem may include, but are not limited to, mobile phones, wearable devices, extended reality devices, in-vehicle systems, smart speakers, vehicle systems, and smart home devices.

[0019] An entity refers to an element with independent identity and function in a multi-device ecosystem network, which may include, but is not limited to, the devices in the multi-device ecosystem network. The relationships between entities refer to the semantic type of directed or undirected edges connecting different entities, defining the nature of interactions between entities. These include, but are not limited to, physical proximity relationships, functional dependencies, and collaborative working relationships.

[0020] In this embodiment, entities in a multi-device ecosystem can be used as nodes. Edges between nodes are determined based on the relationships between entities, and a graph data structure is generated based on the nodes and edges. The graph data structure is a non-linear data structure containing nodes and edges between them, used to represent entities in a multi-device ecosystem network and the relationships between them. In practice, the graph data structure can be denoted as G=(V,E,R), where V represents a node, E represents a relationship, and R represents the relationship type.

[0021] Each node in the graph data structure represents an entity in a multi-device ecosystem. Nodes can record the entity's state information. For example, for a node corresponding to a certain device, its state information may include, but is not limited to, the device type, computing power, battery level, real-time load, network status, and other information.

[0022] In a graph data structure, edges between nodes represent relationships between entities. In a multi-device ecosystem network, if two devices are related, then an edge exists between their corresponding nodes. Depending on the type of relationship, the edge can have different semantic types.

[0023] By incorporating previously independent devices into a unified network for modeling, a structured foundation is provided for cross-device data association.

[0024] Step 102: Extract global features of the graph data structure using a graph neural network.

[0025] In this embodiment, a Graph Neural Network (GNN) is a deep learning model for processing graph data structures. It can propagate and aggregate information between nodes in a graph through a message passing mechanism, and can effectively learn node features and the topology of the graph.

[0026] In practice, the above graph data structure can be input into a graph neural network to obtain the global features of the graph data structure output by the graph neural network. Here, global features refer to the vector representations of the entire system topology and relational states obtained after encoding the entire graph data structure through the graph neural network. Global features can be used as contextual representations and input into subsequent transformer networks.

[0027] By extracting global features through graph neural networks, the relationships between devices can be encoded into machine-understandable vectors. Applying these vectors to the subsequent converter network processing allows the fusion process of multimodal data to take into account the relationships between devices, thereby making the generated fused features more comprehensive and accurate.

[0028] Step 103: Obtain multimodal data from each device, and use a converter network to fuse the multimodal data based on global features to generate fused features.

[0029] In this embodiment, multimodal data refers to different types of data collected by different devices in a multi-device ecosystem network, including but not limited to visual data, voice data, text data, motion sensing data, environmental sensing data, etc.

[0030] A transformer network is a deep neural network architecture based on a self-attention mechanism, adept at processing sequential data and modeling dependencies between elements within a sequence. A transformer network consists of multiple transformer layers. Each transformer layer includes a multi-head attention sublayer and a feedforward neural network sublayer. The multi-head attention sublayer performs attention computation based on the attention mechanism. The feedforward neural network sublayer performs a non-linear transformation on the features output by the multi-head attention sublayer.

[0031] In practice, multimodal data can be converted into encoded data and then input into a transformer network along with global features. The transformer network then fuses these features to obtain fused features. Fused features refer to a unified vector representation output after deep interaction between multimodal data and global features in the transformer network.

[0032] By using a converter network to fuse multimodal data based on global features, multimodal data fusion guided by global features is achieved. During the fusion process, the converter network fully considers the relationships between devices, thereby generating more comprehensive and accurate fusion features, improving the depth and accuracy of the converter network's understanding of the multi-device ecosystem.

[0033] Step 104: Identify user intent based on fusion features, and determine the target task and target device in the multi-device ecosystem network based on user intent.

[0034] In this embodiment, user intent refers to the user's goal or need in the current scenario, which can be inferred by analyzing fused features. For example, the fused features can be input into a classifier with user intent category classification function, and the user intent can be determined by the classifier.

[0035] The classifier described above can be pre-trained using machine learning methods to enable it to classify user intents. This classifier can employ a convolutional neural network. The classifier outputs a probability distribution of various user intents, and based on this distribution, the intent type with the highest probability can be identified as the user intent. The intent distribution can be represented as follows:

[0036] in, It is the probability distribution of user intent predicted by the classifier, which is a probability vector where each element represents the probability of a certain user intent; Softmax normalization exponential function; It is the weight matrix of the classifier; It is the bias term of the classifier; It is a feature of fusion.

[0037] After determining the user's intent, the target task to be executed can be identified based on that intent. Then, considering the device's capabilities, the target device within the multi-device ecosystem can be determined to perform this target task. The target task is the task that needs to be executed to fulfill the user's intent. The target device is the most suitable device within the multi-device ecosystem to execute this target task; specifically, it can be the device that best performs the target task in terms of latency, energy consumption, accuracy, and other metrics. The number of target tasks and target devices can be one or more, and is not limited here.

[0038] As an example, the user intent was identified as "start home fitness," with target tasks of "movement guidance video stream," "heart rate monitoring," and "background music playback." Based on device capabilities, the "movement guidance video stream" task can be assigned to the phone, the "heart rate monitoring" task to the watch, and the "background music playback" task to the smart speaker.

[0039] By using high-quality fusion features for user intent recognition, the accuracy of user intent recognition can be improved. Determining the target task and target device based on accurate intent understanding enables precise scene understanding and intelligent task scheduling.

[0040] Step 105: Send the target task to the target device.

[0041] In this embodiment, after identifying the target device and the target task, corresponding task instructions and configuration parameters can be sent to the target device via an inter-device communication protocol. Continuing the above example, instructions can be sent to a mobile phone to load fitness instruction videos, to a watch to activate real-time heart rate monitoring mode, and to a smart speaker to play fitness background music.

[0042] The following examples illustrate the embodiments of this application.

[0043] Scenario 1: A collaborative exercise scenario involving a mobile phone, watch, and earphones. In this scenario, the devices in the multi-device ecosystem include a watch, a mobile phone, and earphones. The watch can acquire the user's heart rate and posture data. The mobile phone can acquire image data through its camera, which reflects the user's current environment. The earphones can acquire audio data. After generating a graph data structure based on the multi-device ecosystem, the graph data structure can be input into a graph neural network to obtain the global features of the graph data structure. Then, the global features, along with the features or encodings of the aforementioned modal data, can be input into a transformer network to obtain fused features. Next, the fused features can be input into a classifier to determine that the user's intent is "to enable interval running guidance," and based on this user intent, the target task is determined to be "real-time posture estimation," which is then assigned to the watch or mobile phone with posture data acquisition capabilities.

[0044] Scenario 2: Mobile Phone-In-Vehicle System-Smart Glasses Travel Scenarios. In this scenario, the devices in the multi-device ecosystem include mobile phones, in-vehicle systems, and smart glasses. Mobile phones can acquire real-time location and acceleration data, reflecting the user's position and movement. In-vehicle systems can acquire image data from the vehicle's cameras, reflecting the user's environment. Smart glasses can acquire the user's voice and gesture data. After generating a graph data structure based on the multi-device ecosystem, this structure can be input into a graph neural network to obtain global features. Then, the global features, along with the features or encodings of the aforementioned modal data, can be input into a transformer network to obtain fused features. Next, the fused features can be input into a classifier to determine the user's intent as "reach destination + parking space navigation." Based on this intent, the target tasks are determined to be "navigation" and "voice broadcast," and the navigation task is assigned to the in-vehicle system with navigation capabilities, while the voice broadcast task is assigned to the smart glasses.

[0045] Scenario 3: Multi-device Home Scenario. In this scenario, the devices in the multi-device ecosystem include smart speakers, home cameras, smart air conditioners, and smart lights. Smart speakers can collect voice data, and home cameras can collect image data, which can indicate the user's room location. Smart air conditioners can provide temperature and status data. Smart lights can provide status data. After generating a graph data structure based on the multi-device ecosystem, the graph data structure can be input into a graph neural network to obtain global features. Then, the global features and the features or codes of the above modal data can be input into a converter network to obtain fused features. Next, the fused features can be input into a classifier to determine the user intent as "living room viewing mode," and based on this user intent, the target tasks are determined to be "playing video" and "adjusting lights." The task of playing video is then assigned to the smart TV, and the task of adjusting lights is assigned to the smart lights.

[0046] The method provided in the above embodiments of this application first generates a graph data structure based on entities in a multi-device ecosystem network and the relationships between entities. Entities include all devices in the multi-device ecosystem network. Then, global features of the graph data structure are extracted using a graph neural network. Next, multimodal data from each device is acquired, and a transformer network is used to fuse the multimodal data based on the global features to generate fused features. Then, user intent is identified based on the fused features, and the target task and target device in the multi-device ecosystem network are determined based on the user intent. Finally, the target task is sent to the target device. In the above process, multimodal data is deeply fused by analyzing the relationships between devices. User intent is understood based on the multimodal fusion results, and tasks are automatically assigned to the most suitable device for execution. This achieves deep fusion of cross-device data, intelligent intent understanding, and automatic task generation and allocation. This enables the automatic generation and execution of optimal task scheduling schemes, providing users with intelligent and automated services without requiring manual switching and task configuration between different devices, thus improving the intelligent service level of the multi-device ecosystem.

[0047] In some optional embodiments, entities in the multi-device ecosystem network may include, in addition to devices, users, current tasks, and environments. Here, a user refers to the user of the multi-device ecosystem network system. A current task is the task currently being performed by a device in the multi-device ecosystem network. The environment is the user's current surroundings, which may include, but is not limited to, time, location, and ambient light levels.

[0048] Based on this, step 101 above may further include the following steps: Step S11: Take each device, user, current task and environment in the multi-device ecosystem network as a node, and determine the edges between nodes based on at least one of the following: physical proximity relationship, functional dependency relationship, user interaction relationship, spatiotemporal association relationship and task allocation relationship.

[0049] Step S12: Generate a graph data structure based on the edges between nodes.

[0050] Specifically, each device, user, current task, and environment in the multi-device ecosystem network can be treated as a node, resulting in device nodes, user nodes, task nodes, and environment nodes. Device nodes can provide device status information, such as device type, computing power, battery level, real-time load, and network conditions. User nodes can provide user identity and preference information. Task nodes can provide task type and task processing status information. Environment nodes provide the scene context.

[0051] For any two nodes, if there is any kind of association between them, then there is an edge between them. The type of association between the two nodes is the edge type. Since two nodes can have one or more associations, the edges between the two nodes can be labeled with one or more types. All the nodes and edges between them can be combined to obtain a complete graph data structure.

[0052] The types of relationships may include, but are not limited to, at least one of the following: physical proximity, functional dependency, user interaction, spatiotemporal relationship, and task allocation. Physical proximity can be determined based on the connection status between devices. For example, a mobile phone and a watch connected via Bluetooth have a physical proximity relationship. Functional dependency refers to a relationship of mutual dependence in service provision or data processing, which can be determined based on a service dependency chain. For example, a smart TV's video playback function depends on the audio output capability of a sound system, thus the smart TV and the sound system have a functional dependency relationship. User interaction refers to the ongoing operational interaction between a user and a device, which can be determined through the user's operational status. For example, if a user is operating the playback interface of a smartphone, then the user and the smartphone have a user interaction relationship. Spatiotemporal relationship refers to the coexistence of entities in a specific time and space context. For example, if all devices and users are currently in a living room environment, then all devices and users have a spatiotemporal relationship. Task allocation refers to the assignment relationship between a current task and the executing device, which can be established based on task execution requirements. For example, if a music playback task is executed by a smart speaker, then the music playback task and the smart speaker have a task allocation relationship.

[0053] By introducing entities such as users, current tasks, and environment, a graph data structure containing multiple types of entities and diverse relationships can be constructed, thereby comprehensively representing the multi-device ecosystem and providing a sufficient information foundation for subsequent intelligent decision-making.

[0054] In some optional embodiments, step 102 above may further include the following steps: Step S21: Take each node in the graph data structure as the target node in turn, and determine the attention weight between the target node and the associated node for each type of association.

[0055] The type of association is the semantic type of the edges between nodes, such as physical proximity, functional dependency, user interaction, spatiotemporal association, and task allocation. For each type of association, the associated nodes of the target node are those connected to the target node via an edge whose semantic type is that type of association. For example, if a smart TV's video playback function depends on the audio output capability of a sound system, then the smart TV and the sound system have a functional dependency. If a user operates the smart TV, then the user and the smart TV have a user interaction relationship. For a functional dependency, the associated nodes of the smart TV node include the sound system node, but not the user. For a user interaction relationship, the associated nodes of the smart TV node include the user, but not the sound system.

[0056] Attention weights can be calculated internally within a graph neural network using an attention mechanism. Attention weights are used to measure the importance of associated nodes to the target node.

[0057] Specifically, for each type of association, the unnormalized attention weights can be calculated first:

[0058] in, The unnormalized attention weights between target node i and associated node j under the association relationship r; These are learnable attention parameters specific to the association r; For shared linear transformation weight matrix; Let i be the feature vector of the target node i; Let be the feature vector of the associated node j; || denotes the vector concatenation operation; LeakyReLU is the activation function.

[0059] Then, for each type of association, the unnormalized attention weights can be normalized to obtain normalized attention weights:

[0060] in, The normalized attention weights for target node i and associated node j under association relationship r; k belongs to , Let be the set of associated nodes of target node i under association relationship r; The unnormalized attention weights between target node i and associated node k under the association relationship r; This refers to the normalization process applied to the set of associated nodes.

[0061] Step S22: Aggregate the feature vectors of associated nodes based on attention weights to obtain the feature vector of the target node.

[0062] Specifically, for each type of association, the feature vectors of associated nodes under that association type can be aggregated based on attention weights to obtain the intermediate feature vector of the target node. Then, the intermediate feature vectors of the target node under different association types can be further aggregated to obtain the final feature vector of the target node. See the following formula:

[0063] in, Let R be the feature vector of the target node i; R is the set of all relation types; It is a non-linear activation function.

[0064] Step S23: Generate global features based on the feature vectors of each node in the graph data structure.

[0065] Specifically, the feature vectors of each node in the graph data structure can be summarized to obtain global features. Alternatively, the feature vectors of each node in the graph data structure can be further processed, such as through pooling, to obtain global features. No specific limitations are specified here.

[0066] By employing a multi-relationship graph attention mechanism to calculate the attention weights between target nodes and associated nodes under different relationship types, the importance of different relationships can be distinguished, enabling refined and adaptive modeling of complex dependencies between devices and providing accurate contextual representation for subsequent multimodal data fusion.

[0067] In some optional embodiments, based on the above embodiments, when the state of any device in the multi-device ecosystem changes, the feature vectors of the affected nodes in the graph data structure can be locally recalculated.

[0068] Device status may include, but is not limited to, device load, power level, network status, etc. Changes in device status can refer to changes in one or more of the above device statuses, such as a change in power level from normal to low, or a change in network status from connected to disconnected.

[0069] Affected nodes may include nodes in the graph data structure that correspond to devices whose states have changed, as well as nodes that are directly or indirectly related to that node and whose feature vectors need to be updated synchronously.

[0070] Local recomputation refers to updating only the feature vectors of the affected nodes in a graph data structure, rather than recalculating the feature vectors of all nodes. The update process of the feature vectors of the affected nodes can be represented as:

[0071] in, Let i be the state change of node i; It can be an additive or gated unit.

[0072] By implementing a local recalculation mechanism when the device status changes, the high latency and high energy consumption problems caused by repeated full-map calculations in dynamic environments are solved, enabling rapid response to device status fluctuations and energy efficiency optimization.

[0073] In some alternative embodiments, when the relationships between devices in a multi-device ecosystem change, the attention weights of the corresponding edges in the graph data structure can be updated using an exponential moving average method.

[0074] Changes in inter-device relationships refer to changes in the weight or existence of edges connecting device nodes, including at least one of the following: changes in physical distance, changes in functional dependency strength, changes in communication connection state, etc. The attention weight of the corresponding edge refers to the attention weight of the edge that corresponds to the changed relationship.

[0075] The exponential moving average method is a weighted average update method based on historical values. The process of updating the attention weights of corresponding edges in a graph data structure based on the exponential moving average method can be represented as:

[0076] in, It is the attention weight of edge (i,j); This is the attenuation factor, also known as the smoothing coefficient, with a value range of (0,1). The new observation weights are calculated based on the current interaction state.

[0077] By updating the association weights using the exponential moving average method, the technical problem of system instability caused by sudden changes in the strength of relationships between devices is solved, the smooth evolution of relationship weights is achieved, and the system robustness is improved.

[0078] Through the combined application of the above incremental update mechanisms, a continuously adaptive structural awareness capability is provided for a multi-device ecosystem.

[0079] In some optional embodiments, step 104 above, which involves fusing multimodal data based on global features using a transformer network to generate fused features, may further include the following steps: Step S31: Extract the features of each modality in the multimodal data.

[0080] Specifically, features can be extracted from various modal data using pre-trained deep learning models, such as convolutional neural networks. The features of each modal data can be a sequence of vectors representing the semantic information of that modality.

[0081] Step S32: Combine the global features with the features of the above modal data into an input sequence, and process the input sequence through a transformer network to generate fused features. The attention mechanism of the transformer model is controlled by a bias matrix generated based on the graph data structure.

[0082] Specifically, the transformer network comprises multiple transformer layers. Each transformer layer includes a multi-head attention sublayer and a feedforward neural network sublayer. The multi-head attention sublayer performs attention calculations based on an attention mechanism. The feedforward neural network sublayer performs nonlinear transformations on the features output by the multi-head attention sublayer. After combining the global features with the features of the modal data mentioned above into an input sequence, it can be input into the transformer network. After being processed sequentially by each transformer layer, operations such as pooling can be performed, and finally, fused features are output. The input sequence can be represented as:

[0083] Where H is the input sequence; g is the global feature output by the graph neural network; || denotes the vector concatenation operation; Represents the features of the i-th modal data; This represents the feature of the Mth modality data.

[0084] The attention mechanism transformer network is a core component that achieves weighted aggregation of the importance of sequence elements by calculating the relevance weights between queries, keys, and values. The bias matrix is ​​an additional matrix introduced in the transformer's attention calculation to inject structural information and guide the distribution of attention weights. Therefore, the multi-head attention sublayer in each transformer layer can perform attention calculations based on the bias matrix, as shown in the following formula:

[0085] in, To introduce the attention mechanism of the bias matrix; Q, K, and V are the query matrix, key matrix, and value matrix obtained by linear transformation of the input sequence, respectively. is the dimension of the key vector, used for scaling; B is the bias matrix generated based on the heterogeneous graph, used to inject structural information into the attention mechanism.

[0086] By extracting features from each modality and combining global and modal features into an input sequence, the problem of the separation between structural and content information in multimodal fusion is solved, achieving synchronous injection of structure and semantics and providing a unified information foundation for deep fusion. Furthermore, by introducing a bias matrix generated based on graph data structures into the attention mechanism of the transformer network, the problems of traditional attention mechanisms lacking structural guidance and having random fusion directions are addressed. This enables guided information interaction under the constraints of graph data structures, significantly improving the accuracy and robustness of fusion.

[0087] In some optional embodiments, the above bias matrix is ​​generated through the following steps: Step S41: Each node in the graph data structure is sequentially taken as the target node, and the attention weight between the target node and the associated nodes is determined for each type of association. This step can be found in step S21 of the embodiment, and will not be repeated here to avoid repetition.

[0088] Step S42: Input the attention weights, user intent context, and time information into the condition generator to obtain the bias matrix.

[0089] The user intent context represents the user intent predicted in the previous processing cycle. Specifically, it can be a vector representation of the user intent predicted in the previous processing cycle, serving as conditional information for the current processing cycle. The time information represents the timestamp or timecode of the current processing cycle or session phase. The condition generator is a learnable neural network module capable of dynamically generating a bias matrix based on the input conditional information. By inputting the attention weights, user intent context, and time information into the condition generator, the bias matrix can be obtained. The processing can be represented as follows:

[0090] Where B is the bias matrix; Φ is the condition generator; The normalized attention weights of target node i and associated node j under the association relationship r; t represents the user intent context; t represents time information.

[0091] By using attention weights as input to the condition generator, the lack of structured bias guidance in traditional converter networks is addressed, enabling the effective injection of device relationship topology into the attention mechanism and significantly improving the structural consistency of multimodal fusion. Introducing user intent context as a generation condition solves the technical problem of static bias matrices failing to adapt to dynamic user needs, achieving dynamic attention guidance based on semantic goals. By adding temporal information as a generation condition, the rigidity of attention distribution in long-term sessions is resolved, achieving temporal adaptation of the bias matrix and ensuring system stability during long-term operation.

[0092] In some optional embodiments, step 104 above, which determines the target task and the target device in the multi-device ecosystem network based on user intent, may further include the following steps: Step S51: The fused features are input as the system state into the reinforcement learning model. The reinforcement learning model outputs task scheduling information that maximizes the reward value. The reward value is determined by a reward function, which is generated based on a multi-objective optimization function and a penalty term. The optimization objective of the multi-objective optimization function is at least one of the following: latency objective, energy consumption objective, accuracy objective, and reliability objective.

[0093] Reinforcement learning is a machine learning method that learns optimal policies based on reward signals by having an agent interact with its environment. In practice, Double Deep Q-Network (DQN) can be used.

[0094] The system state is an observation representing the current state of the environment. Here, fused features can be used as input to the reinforcement learning model as the system state. The reinforcement learning model then outputs task scheduling information that maximizes the reward value. Task scheduling information refers to the actions output by the reinforcement learning agent, representing the task allocation scheme to devices.

[0095] A reward value is a scalar feedback signal used to evaluate the quality of an action in a reinforcement learning environment, and it can be determined by a reward function. The reward function is a function that defines the rules for calculating reward values ​​and serves as the optimization objective of the reinforcement learning model. The reward function can be generated based on a multi-objective optimization function and a penalty term. A multi-objective optimization function is a mathematical function that simultaneously considers multiple optimization objectives, which may include, but are not limited to, at least one of the following: latency objective, energy consumption objective, accuracy objective, and reliability objective. The penalty term is a negative reward component in the reward function used to constrain undesirable behavior.

[0096] The multi-objective optimization function can be expressed as:

[0097] Where L(x) is the delay target; E(x) is the energy consumption target; A(x) is the accuracy target; R(x) is the reliability target; min represents minimization.

[0098] The reward function can be expressed as:

[0099] in, For the reward function; , , , , All are weights, which can be preset as needed; Cost is a penalty, which may include, but is not limited to, at least one of the following: budget penalty, bandwidth penalty, latency penalty, etc.

[0100] Step S52: Based on the task scheduling information, determine the target task and target device.

[0101] Here, the task scheduling can be parsed to determine the target task and target device. Specifically, the task scheduling information can be an encoded action vector or action identifier, which can be decoded to obtain "task-device" pairs. The task in each "task-device" pair can be taken as the target task, and its corresponding device can be taken as the target device for executing the target task.

[0102] By using fused features as system state input for reinforcement learning, end-to-end optimization from environmental perception to scheduling decision-making is achieved, significantly improving the intelligence and adaptability of scheduling. By establishing a multi-objective optimization function, a comprehensive trade-off between latency, energy consumption, accuracy, and reliability is achieved, ensuring optimal system performance under multiple constraints. Introducing a penalty term into the reward function enables the effective embedding of constraints related to privacy budget, bandwidth, and latency, enhancing the system's practicality and reliability.

[0103] In some optional embodiments, step 105 above may further include the following steps: Step S61: Determine the migration score based on the expected gain and migration cost of migrating the target task from the original execution device to the target device.

[0104] Step S62: If the migration score is greater than the threshold, the target task is migrated from the original execution device to the target device.

[0105] The original execution device can be the device currently executing the target task.

[0106] Expected gain refers to the quantified value of the benefits expected to be gained in terms of computing speed, energy saving, and accuracy after migrating the target task from the original execution device to the target device. For example, it could be reduced latency, increased throughput, or reduced energy consumption.

[0107] Migration cost refers to the quantified value of the additional overhead incurred during the migration process, which may include, but is not limited to, state serialization cost, connection establishment cost, and state synchronization cost. For example, it can be the sum of task state serialization time, transmission time, and deserialization time.

[0108] The transfer score is a comprehensive evaluation value quantified based on expected gain and transfer cost, used to decide whether to perform task transfer. It is the result of calculating the expected gain as a function of various costs. In practice, a formula for calculating the transfer score can be pre-defined, and the variables in this formula can include expected gain and transfer cost. Substituting the calculated expected gain and transfer cost into the transfer score calculation formula yields the transfer score. For example, the transfer score calculation formula can be expressed as:

[0109] in, For transfer scores; In order, the values ​​are: reduction in delay, reduction in energy consumption, and improvement in accuracy; For the expected gain; For the cost of migration; All of these are weights, which can be preset as needed.

[0110] By comprehensively considering the expected gains and migration costs, problems such as system oscillations and performance degradation caused by improper migration are avoided, thus maximizing system-level benefits.

[0111] In some alternative embodiments, the graph neural network and the transformer network are supervised during training using a joint optimization objective function, which includes a first loss function, a second loss function, and a third loss function. The first loss function indicates the contrastive learning loss, the second loss function indicates the mutual information lower bound loss, and the third loss function indicates the structure regularization loss.

[0112] The joint optimization objective function can be expressed as:

[0113] in, To jointly optimize the objective function; The first loss function; This is the second loss function; This is the third loss function. All of these are weights, which can be preset as needed.

[0114] The first loss function can indicate the contrastive learning loss, specifically the Information Noise-Contrastive Estimation (InfoNCE) function, which learns feature representations by bringing positive sample pairs closer together and pushing negative sample pairs apart. Its expression is:

[0115] in, Features for matching positive sample pairs; is the negative sample feature; sim is the cosine similarity function; τ is the temperature hyperparameter used to control the distribution smoothness.

[0116] The second loss function can indicate a lower bound loss for mutual information. It maximizes the mutual information between feature representations of different modalities through a variational lower bound, encouraging the learning of shared semantic representations. Its expression is:

[0117] Where I(Z;X) is mutual information, which characterizes the degree of interdependence between variables Z and X; It is a variational function, parameterized by a neural network; Let X be the joint probability distribution of Z; The product of the marginal probability distributions of X and Z; E represents the expected value; It is a natural exponential function.

[0118] The third loss function can indicate the graph structure regularization loss, specifically a Laplacian regularization-based loss term, used to constrain the consistency between the model output and the graph structure. Its expression is:

[0119] in, For each element between node i and node j in the normalized adjacency matrix, it is a scalar weight. These are the attention statistics corresponding to nodes i and j, respectively, specifically an aggregation or summary of the attention weights associated with these nodes in the transformer model.

[0120] By introducing contrastive learning loss as the first loss function, the technical problem of semantic alignment of multimodal data is solved, and effective alignment of features of different modalities in the latent space is achieved, which significantly improves the accuracy of cross-modal retrieval and understanding.

[0121] By introducing the mutual information lower bound loss as the second loss function, the technical problem of insufficient extraction of shared semantic information in multimodal fusion is solved, the essential semantic association between modalities is maximized, and the robustness of the model to missing modalities is enhanced.

[0122] By introducing structural regularization loss as the third loss function, the technical problem of the lack of structure awareness in the converter attention mechanism is solved, the consistency constraint between attention distribution and graph topology is achieved, and the structural rationality of multimodal fusion is improved.

[0123] By jointly optimizing three related but complementary loss functions, the technical problem of model bias caused by single-objective training is solved, and the synergistic optimization of semantic alignment, information sharing and structural consistency is achieved, thus comprehensively improving model performance.

[0124] In summary, one or more of the above embodiments, Figure 2The diagram illustrates the processing flow of the task scheduling method. First, a graph data structure is generated based on entities and their relationships within the multi-device ecosystem network. Then, this graph data structure is input into a graph neural network to obtain global features output by the network. Next, multimodal data is converted into encoded data and input along with the global features into a transformer network. The transformer network fuses these features to obtain fused features. These fused features are then input into a classifier with user intent category classification capabilities to determine the user intent. Finally, the fused features are used as the system state input to a reinforcement learning model, which outputs task scheduling information. Task scheduling is then performed based on this information. By analyzing the relationships between devices to deeply fuse multimodal data, understanding user intent based on the fusion results, and automatically assigning tasks to the most suitable device, this method achieves deep fusion of cross-device data, intelligent intent understanding, and automatic task generation and allocation. This enables the automatic generation and execution of optimal task scheduling schemes, providing users with intelligent and automated services without requiring manual switching and task configuration between different devices, thus improving the intelligent service level of the multi-device ecosystem.

[0125] It should be noted that the task scheduling method provided in this application embodiment can be executed by a task scheduling device. This application embodiment uses the execution of the task scheduling method by a task scheduling device as an example to illustrate the task scheduling device provided in this application embodiment.

[0126] like Figure 3 As shown, the task scheduling device 300 of this embodiment includes: a generation unit 301, used to generate a graph data structure based on entities in a multi-device ecosystem network and the relationships between entities, wherein the entities include each device in the multi-device ecosystem network; an extraction unit 302, used to extract global features of the graph data structure through a graph neural network; a fusion unit 303, used to acquire multimodal data of each device, and perform fusion processing on the multimodal data based on the global features through a transformer network to generate fused features; a determination unit 304, used to identify user intent based on the fused features, and determine the target task and the target device in the multi-device ecosystem network based on the user intent; and a sending unit 305, used to send the target task to the target device.

[0127] In some optional implementations of this embodiment, the entities further include users, environment, and current task; the generation unit 301 is further configured to: take each device, user, environment, and current task in the multi-device ecosystem network as nodes, determine the edges between the nodes based on at least one of the physical proximity relationship, functional dependency relationship, user interaction relationship, spatiotemporal correlation relationship, and task allocation relationship between the nodes; and generate the graph data structure based on the nodes and the edges between the nodes. By introducing entities such as users, current task, and environment, a graph data structure containing multiple types of entities and diverse relationships can be constructed, thereby comprehensively representing the multi-device ecosystem and providing a sufficient information foundation for subsequent intelligent decision-making.

[0128] In some optional implementations of this embodiment, the extraction unit 302 is further configured to: sequentially take each node in the graph data structure as a target node, determine the attention weight between the target node and the associated node for each type of association; aggregate the feature vectors of the associated nodes based on the attention weights to obtain the feature vector of the target node; and generate the global feature based on the feature vectors of each node in the graph data structure. By employing a multi-relationship graph attention mechanism to calculate the attention weights between the target node and the associated node under different association types, the importance of different associations can be distinguished, achieving refined and adaptive modeling of complex dependencies between devices, and providing accurate contextual representation for subsequent multimodal data fusion.

[0129] In some optional implementations of this embodiment, the device further includes an update unit, configured to: locally recalculate the feature vectors of affected nodes in the graph data structure when the state of any device in the multi-device ecosystem changes; and update the attention weights of corresponding edges in the graph data structure using an exponential moving average device when the relationships between devices in the multi-device ecosystem change. Through the combined application of the above incremental update mechanisms, a continuously adaptive structural awareness capability is provided for the multi-device ecosystem.

[0130] In some optional implementations of this embodiment, the fusion unit 303 is further configured to: extract features from each modality of the multimodal data; combine the global features with the features of each modality of the data into an input sequence; process the input sequence through the transformer network to generate the fused features, wherein the attention mechanism of the transformer model is controlled by a bias matrix generated based on the graph data structure. Through the above process, the synchronous injection of structure and semantics is achieved, providing a unified information foundation for deep fusion; simultaneously, guided information interaction under the constraints of the graph data structure is realized, significantly improving the accuracy and robustness of the fusion.

[0131] In some optional implementations of this embodiment, the bias matrix is ​​generated through the following steps: Each node in the graph data structure is sequentially taken as a target node, and the attention weight between the target node and the associated node is determined for each type of association. The attention weight, user intent context, and time information are input into a condition generator to obtain the bias matrix, wherein the user intent context represents the user intent predicted in the previous processing cycle. Through the above process, the structural consistency of multimodal fusion is significantly improved, ensuring the stability of the system during long-term operation.

[0132] In some optional implementations of this embodiment, the determining unit 304 is further configured to: input the fused features as system state into a reinforcement learning model, and output task scheduling information that maximizes the reward value through the reinforcement learning model, wherein the reward value is determined by a reward function, the reward function being generated based on a multi-objective optimization function and a penalty term, and the optimization objective of the multi-objective optimization function being at least one of the following: latency objective, energy consumption objective, accuracy objective, and reliability objective; and determine the target task and the target device based on the task scheduling information. This ensures the optimal performance of the system under multiple constraints and improves the practicality and reliability of the system.

[0133] In some optional implementations of this embodiment, the sending unit 305 is further configured to: determine a migration score based on the expected gain and migration cost of migrating the target task from the original execution device to the target device; and, if the migration score is greater than a threshold, migrate the target task from the original execution device to the target device. By comprehensively considering the expected gain and migration cost, problems such as system oscillation and performance degradation caused by improper migration are avoided, thereby maximizing system-level benefits.

[0134] In some optional implementations of this embodiment, the graph neural network and the transformer network are supervised during training using a joint optimization objective function. This joint optimization objective function includes a first loss function, a second loss function, and a third loss function. The first loss function indicates the contrastive learning loss, the second loss function indicates the mutual information lower bound loss, and the third loss function indicates the structure regularization loss. By jointly optimizing three related but complementary loss functions, the model bias problem caused by single-objective training is solved, achieving synergistic optimization of semantic alignment, information sharing, and structural consistency, thus comprehensively improving model performance.

[0135] The apparatus provided in the above embodiments of this application deeply integrates multimodal data by analyzing the relationships between devices, understands user intent based on the multimodal fusion results, and automatically assigns tasks to the most suitable device for execution. This achieves deep fusion of cross-device data and intelligent intent understanding, as well as automatic generation and allocation of tasks. As a result, it can automatically generate and execute the optimal task scheduling scheme, providing users with intelligent and automated services without requiring users to manually switch and configure tasks between different devices, thus improving the level of intelligent services in the multi-device ecosystem.

[0136] The task scheduling device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0137] The task scheduling device in this application embodiment can be a device with an operating system. The operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system.

[0138] The task scheduling device provided in this application embodiment can achieve... Figure 1 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0139] Optionally, such as Figure 4 As shown, this application embodiment also provides an electronic device 400, including a processor 401 and a memory 402. The memory 402 stores a program or instructions that can run on the processor 401. When the program or instructions are executed by the processor 401, they implement the various steps of the above-described task scheduling method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0140] It should be noted that the electronic devices in the embodiments of this application include the aforementioned mobile electronic devices and non-mobile electronic devices.

[0141] Figure 5 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application. The electronic device 500 includes, but is not limited to, components such as: radio frequency unit 501, network module 502, audio output unit 503, input unit 504, sensor 505, display unit 506, user input unit 507, interface unit 508, memory 509, and processor 510.

[0142] Those skilled in the art will understand that the electronic device 500 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 510 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 5 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here. The processor 510 is configured to generate a graph data structure based on entities and their relationships within a multi-device ecosystem network, wherein the entities include each device in the multi-device ecosystem network; extract global features from the graph data structure using a graph neural network; acquire multimodal data from each device; perform fusion processing on the multimodal data based on the global features using a transformer network to generate fused features; identify user intent based on the fused features; determine the target task and the target device in the multi-device ecosystem network based on the user intent; and send the target task to the target device.

[0143] By analyzing the relationships between devices to deeply integrate multimodal data, and based on the results of multimodal fusion to understand user intent, tasks are automatically assigned to the most suitable devices for execution. This achieves deep integration of cross-device data, intelligent intent understanding, and automatic task generation and allocation, thereby automatically generating and executing optimal task scheduling schemes. It provides users with intelligent and automated services without requiring users to manually switch and configure tasks between different devices, thus improving the level of intelligent services in the multi-device ecosystem.

[0144] In some optional implementations of this embodiment, the entities further include users, environment, and current task; the processor 510 is further configured to treat each device, user, environment, and current task in the multi-device ecosystem network as nodes, and determine the edges between the nodes based on at least one of the following: physical proximity, functional dependency, user interaction, spatiotemporal correlation, and task allocation; and generate the graph data structure based on the nodes and the edges between them. By introducing entities such as users, current task, and environment, a graph data structure containing multiple types of entities and diverse relationships can be constructed, thereby comprehensively representing the multi-device ecosystem and providing a sufficient information foundation for subsequent intelligent decision-making.

[0145] In some optional implementations of this embodiment, the processor 510 is further configured to sequentially use each node in the graph data structure as a target node, determine the attention weight between the target node and the associated node for each type of association; aggregate the feature vectors of the associated nodes based on the attention weights to obtain the feature vector of the target node; and generate the global feature based on the feature vectors of each node in the graph data structure. By employing a multi-relationship graph attention mechanism to calculate the attention weights between the target node and the associated node under different association types, the importance of different associations can be distinguished, achieving refined and adaptive modeling of complex dependencies between devices, and providing accurate contextual representation for subsequent multimodal data fusion.

[0146] In some optional implementations of this embodiment, the processor 510 is further configured to: recalculate the feature vectors of affected nodes in the graph data structure when the state of any device in the multi-device ecosystem changes; and update the attention weights of corresponding edges in the graph data structure using an exponential moving average method when the relationships between devices in the multi-device ecosystem change. Through the combined application of the above incremental update mechanisms, a continuously adaptive structural awareness capability is provided for the multi-device ecosystem.

[0147] In some optional implementations of this embodiment, the processor 510 is further configured to extract features from each modality of the multimodal data; combine the global features with the features of each modality of the data into an input sequence; process the input sequence through the transformer network to generate the fused features; wherein the attention mechanism of the transformer model is controlled by a bias matrix generated based on the graph data structure. Through the above process, the synchronous injection of structure and semantics is achieved, providing a unified information foundation for deep fusion; simultaneously, guided information interaction under the constraints of the graph data structure is realized, significantly improving the accuracy and robustness of the fusion.

[0148] In some optional implementations of this embodiment, the bias matrix is ​​generated through the following steps: Each node in the graph data structure is sequentially taken as a target node, and the attention weight between the target node and the associated node is determined for each type of association. The attention weight, user intent context, and time information are input into a condition generator to obtain the bias matrix, wherein the user intent context represents the user intent predicted in the previous processing cycle. Through the above process, the structural consistency of multimodal fusion is significantly improved, ensuring the stability of the system during long-term operation.

[0149] In some optional implementations of this embodiment, the processor 510 is further configured to input the fused features as system state into a reinforcement learning model, and output task scheduling information that maximizes the reward value through the reinforcement learning model. The reward value is determined by a reward function, which is generated based on a multi-objective optimization function and a penalty term. The multi-objective optimization function has at least one of the following optimization objectives: latency objective, energy consumption objective, accuracy objective, and reliability objective. Based on the task scheduling information, the target task and the target device are determined. This ensures optimal system performance under multiple constraints and improves the system's practicality and reliability.

[0150] In some optional implementations of this embodiment, the processor 510 is further configured to determine a migration score based on the expected gain and migration cost of migrating the target task from the original execution device to the target device; and if the migration score is greater than a threshold, migrate the target task from the original execution device to the target device. By comprehensively considering the expected gain and migration cost, problems such as system oscillation and performance degradation caused by improper migration are avoided, thereby maximizing system-level benefits.

[0151] In some optional implementations of this embodiment, the graph neural network and the transformer network are supervised during training using a joint optimization objective function. This joint optimization objective function includes a first loss function, a second loss function, and a third loss function. The first loss function indicates the contrastive learning loss, the second loss function indicates the mutual information lower bound loss, and the third loss function indicates the structure regularization loss. By jointly optimizing three related but complementary loss functions, the model bias problem caused by single-objective training is solved, achieving synergistic optimization of semantic alignment, information sharing, and structural consistency, thus comprehensively improving model performance.

[0152] It should be understood that, in this embodiment, the input unit 504 may include a graphics processing unit (GPU) 5041 and a microphone 5042. The GPU 5041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 506 may include a display panel 5061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 507 includes at least one of a touch panel 5071 and other input devices 5072. The touch panel 5071 is also called a touch screen. The touch panel 5071 may include a touch detection device and a touch controller. Other input devices 5072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0153] The memory 509 can be used to store software programs and various data. The memory 509 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 509 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 509 in this embodiment includes, but is not limited to, these and any other suitable types of memory.

[0154] Processor 510 may include one or more processing units; optionally, processor 510 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 510.

[0155] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described task scheduling method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0156] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0157] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described task scheduling method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0158] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0159] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the task scheduling method embodiment described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0160] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0161] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0162] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A task scheduling method, characterized in that, The method includes: Based on the entities and relationships between entities in the multi-device ecosystem network, a graph data structure is generated, wherein the entities include each device in the multi-device ecosystem network; Global features of the graph data structure are extracted using a graph neural network. The multimodal data of each device is acquired, and the multimodal data is fused based on the global features through a converter network to generate fused features; User intent is identified based on the fused features, and the target task and target device in the multi-device ecosystem network are determined based on the user intent. Send the target task to the target device.

2. The method according to claim 1, characterized in that, The entities also include users, environment, and current task; the generation of a graph data structure based on entities in the multi-device ecosystem network and the relationships between entities includes: Each device, user, environment, and current task in the multi-device ecosystem network is taken as a node, and the edges between the nodes are determined based on at least one of the following: physical proximity, functional dependency, user interaction, spatiotemporal correlation, and task allocation. The graph data structure is generated based on the nodes and the edges between them.

3. The method according to claim 1, characterized in that, The step of extracting global features from the graph data structure using a graph neural network includes: Each node in the graph data structure is taken as a target node in turn, and the attention weight between the target node and the associated node is determined for each type of association. Based on the attention weights, the feature vectors of the associated nodes are aggregated to obtain the feature vector of the target node; The global feature is generated based on the feature vectors of each node in the graph data structure.

4. The method according to claim 3, characterized in that, The method further includes: In the event of a state change of any device in the multi-device ecosystem network, the feature vectors of the affected nodes in the graph data structure are locally recalculated. When the relationships between devices change in the multi-device ecosystem network, the attention weights of the corresponding edges in the graph data structure are updated using an exponential moving average method.

5. The method according to claim 1, characterized in that, The process of fusing the multimodal data based on the global features through a transformer network to generate fused features includes: Extract the features of each modality in the multimodal data; The global features are combined with the features of each modality data to form an input sequence. The input sequence is then processed by the transformer network to generate the fused features. The attention mechanism of the transformer model is controlled by a bias matrix generated based on the graph data structure.

6. The method according to claim 5, characterized in that, The bias matrix is ​​generated through the following steps: Each node in the graph data structure is taken as a target node in turn, and the attention weight between the target node and the associated node is determined for each type of association. The attention weights, user intent context, and time information are input into the condition generator to obtain the bias matrix, wherein the user intent context is used to represent the user intent predicted in the previous processing cycle.

7. The method according to claim 1, characterized in that, The process of determining the target task and the target device in the multi-device ecosystem network based on the user intent includes: The fused features are input as system state into a reinforcement learning model, and the reinforcement learning model outputs task scheduling information that maximizes the reward value. The reward value is determined by a reward function, which is generated based on a multi-objective optimization function and a penalty term. The optimization objective of the multi-objective optimization function is at least one of the following: latency objective, energy consumption objective, accuracy objective, and reliability objective. Based on the task scheduling information, the target task and the target device are determined.

8. The method according to claim 1, characterized in that, Sending the target task to the target device includes: A migration score is determined based on the expected gains and migration costs of migrating the target task from the original execution device to the target device; If the migration score is greater than the threshold, the target task will be migrated from the original execution device to the target device.

9. The method according to claim 1, characterized in that, The graph neural network and the transformer network are supervised by a joint optimization objective function during training. The joint optimization objective function includes a first loss function, a second loss function, and a third loss function. The first loss function indicates the contrastive learning loss, the second loss function indicates the mutual information lower bound loss, and the third loss function indicates the structure regularization loss.

10. A task scheduling device, characterized in that, The device includes: The generation unit is used to generate a graph data structure based on entities in the multi-device ecosystem network and the relationships between entities, wherein the entities include each device in the multi-device ecosystem network; The extraction unit is used to extract global features of the graph data structure through a graph neural network; The fusion unit is used to acquire multimodal data from each device, and to perform fusion processing on the multimodal data based on the global features through a converter network to generate fused features; The determining unit is used to identify user intent based on the fusion features, and to determine the target task and the target device in the multi-device ecosystem network based on the user intent; A sending unit is used to send the target task to the target device.

Images