CONTROL METHOD AND FINISH DEVICE

Abstract

A control procedure comprises: obtaining input data representing a user intent; in response to the input data, converting the input data into a target requirement, which includes a target action, a target capability, and a target execution device; in response to the conversion, determining, from a set of capabilities, a first capability that fulfills the target capability and a first execution device for the first capability as the target execution device; in response to the target requirement, responding to the target action with the first capability based on the first execution device to fulfill the user intent;In response to the fact that the first executing device is a target device among the electronic devices, transmit, via a communication channel to the target device, a call instruction to invoke the first capability and input contents provided to the first capability.

Description

REFERENCE TO RELATED REGISTRATION(S)

[0001] This application claims priority over Chinese patent application No. 2024113919018, filed on September 30, 2024, which is hereby incorporated in full by reference. TECHNICAL AREA

[0002] The present disclosure relates to the technical field of networking devices, in particular a control method and an end device. TECHNICAL BACKGROUND

[0003] In the current state of the art, an end device can use a functional module that provides the necessary capability when performing a specific action. However, there are scenarios in which the end device does not have a functional module with this capability, or the functional module cannot be used, resulting in low reliability in the execution of the action. OVERVIEW OF THE INVENTION

[0004] In one aspect, the present disclosure provides a control method. The method comprises: obtaining input data representing a user intent; in response to the input data, converting the input data into a target request comprising a target action, a target capability, and a target execution device; in response to the conversion, determining, from a set of capabilities, a first capability fulfilling the target capability and a first execution device for the first capability as the target execution device; wherein the set of capabilities comprises: a local capability set of an end device and a remote capability set of at least one electronic device fulfilling a connectivity requirement with the end device; in response to the target request, responding to the target action with the first capability based on the first execution device to fulfill the user intent;In response to the fact that the first executing device is a target device among the electronic devices, transmit, via a communication channel to the target device, a call instruction to invoke the first capability and input contents provided to the first capability.

[0005] In another aspect, the present disclosure provides an electronic device. The device comprises: a memory in which computer program instructions are stored; and a processor connected to the memory and configured to execute the computer program instructions: obtaining input data representing a user intent; in response to the input data, converting the input data into a target request comprising a target action, a target capability, and a target execution device; in response to the conversion, determining, from a set of capabilities, a first capability fulfilling the target capability and a first execution device for the first capability as the target execution device;wherein the capability set comprises: a local capability set of an end device and a remote capability set of at least one electronic device that satisfies a connectivity requirement with the end device; in response to the target request, responding to the target action with the first capability based on the first execution device to fulfill the user intent; in response to the fact that the first execution device is a target device among the electronic devices, transmitting, via a connectivity channel to the target device, a call instruction to invoke the first capability and input content provided to the first capability.

[0006] In another aspect, the present disclosure provides a non-volatile, computer-readable storage medium on which computer program instructions are stored, executable by at least one processor to perform: obtaining input data representing a user intent; in response to the input data, converting the input data into a target request comprising a target action, a target capability, and a target execution device; in response to the conversion, determining, from a set of capabilities, a first capability fulfilling the target capability and a first execution device for the first capability as the target execution device; wherein the set of capabilities comprises: a local set of capabilities of an end device and a remote set of capabilities of at least one electronic device fulfilling a connection requirement with the end device;In response to the target request, the first capability, based on the first execution device, will respond to the target action to fulfill the user intent; given that the first execution device is a target device among the electronic devices, it will transmit, via a communication channel to the target device, a call instruction to invoke the first capability and input content provided to the first capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] To more clearly illustrate the technical solutions of certain embodiments of the present disclosure, the drawings used in describing certain embodiments are briefly introduced below. The drawings described below depict only certain embodiments of the present disclosure. A person skilled in the art can derive other drawings from these drawings without any creative effort. Fig. 1 is a flowchart of a control procedure according to certain embodiments of the present disclosure; Fig. 2 is an example diagram of a network of devices according to certain embodiments of the present disclosure; Fig. Figure 3 is an example diagram of a mobile phone that uses a large language model on a server, according to certain embodiments of the present disclosure; Fig. Figure 4 illustrates an example of a mobile phone that uses a tablet to project its screen externally, according to certain embodiments of the present disclosure. Fig. Figure 5 illustrates an example of a mobile phone that uses a tablet to project its screen onto a television set, according to certain embodiments of the present disclosure. Fig. Figure 6 illustrates an example of a SC on a mobile phone that acquires capabilities of the mobile phone according to certain embodiments of the present disclosure. Fig. Figure 7 illustrates an example of a SC on a tablet that transmits capabilities to an SC on a mobile phone, according to certain embodiments of the present disclosure. Fig. Figure 8 illustrates an example of a tablet SC that transmits capability status update information to a mobile phone SC, according to certain embodiments of the present disclosure. Fig. Figure 9 illustrates a schematic diagram of the structure of an end device provided according to certain embodiments of the present disclosure. Fig. Figure 10 illustrates an example of establishing a network between a mobile phone, a server and a tablet via the SC according to certain embodiments of the present disclosure. Fig. Figure 11 illustrates an example of analyzing cross-device tasks and transferring locally unexecutable tasks to an executable device according to certain embodiments of the present disclosure. Fig. Figure 12 illustrates an example of converting spoken language into an executable task by invoking the speech-to-task capability in various devices in certain embodiments of the present disclosure. Fig. Figure 13 illustrates an interaction sequence diagram for implementing a aggregation of capabilities and interoperability via SCs in a device networking scenario in certain embodiments of the present disclosure. Fig. Figure 14 illustrates an interaction sequence diagram for the implementation of capability calls via SCs in a device networking scenario in certain embodiments of the present disclosure. DETAILED DESCRIPTION

[0008] With reference to the accompanying drawings, a description is provided below to illustrate several technical solutions in certain embodiments of the present disclosure. The described embodiments are only a subset of the embodiments of the present disclosure, not all embodiments. Other embodiments that a person skilled in the art can obtain without creative effort based on the embodiments described in the present disclosure are within the scope of protection of the present disclosure.

[0009] With reference to Fig. 1, which is a flowchart illustrating an implementation of a control method provided in certain embodiments of the present disclosure, the method can be applied to an end device that can meet connectivity requirements with at least one electronic device such as a mobile phone, a tablet, and the like. As, for example, in Fig. As shown in Figure 2, the terminal device can be a mobile phone, and the electronic device that meets the connectivity requirements with the terminal device can be another mobile phone, a tablet, a laptop, a television, or another device. The target functional module is installed on both the terminal device and the electronic device. The target functional module can be a hardware module or a software module. The software module can be a system module or a third-party application. The target functional module can have one or more capabilities. The target functional module can have one or more functional submodules, each functional submodule providing a capability. The connectivity requirement is a prerequisite for establishing a network connection between the terminal device and the electronic device.Specifically, the strong networking application SC (SmartConnect) is installed on the terminal device and the electronic device. The terminal device and the electronic device establish a connection channel through their respective SCs to achieve networking and meet the connection requirements. The technical solution according to this embodiment is primarily used to improve the reliability of the action execution.

[0010] In certain embodiments, the method may include at least one of the following steps:

[0011] Step 101: Obtaining input data.

[0012] The input data is used to represent a user intent.

[0013] In certain embodiments, the input data can be obtained through a target application, which may be an artificial intelligence assistant (AI assistant), for example, a voice assistant on a mobile phone. Voice assistants can receive input data, such as text or speech, that represents the user's intent.

[0014] 102: In response to receiving the input data, converting the input data into a target request which includes a target action, a target capability and a target execution device.

[0015] In certain embodiments, a requirements template may have been pre-configured. This requirements template contains three elements to be completed: target action, target capability, and target execution device. After receiving the input data, the input data is converted into a target requirement according to the requirements template. The target requirement comprises the target action, the target capability, and the target execution device.

[0016] If the terminal cannot analyze the input data, the target action in the target request is the action of analyzing the input data, which can be recorded as the first action. The target capability is the first capability capable of performing the first action. The target execution device is a device with the target capability, but the target execution device itself is not specified.

[0017] If the terminal device is capable of analyzing the input data, the target action in the target request is the action analyzed from the input data, which can be recorded as the first action, for example, a text-to-image action or a screen-sharing action. The target capability is the first capability that can execute the first action, and the target execution device is a device possessing the target capability, although the target execution device itself is not specified. In an implementation, the target capability can be a capability possessed by the terminal device or a capability possessed by a target device within an electronic device, and the target execution device can be either the terminal device or the target device within an electronic device, although the target execution device itself is not specified.

[0018] Step 103: In response to the conversion, a first capability in the capability set is determined to fulfill the target capability, and a first execution device of the first capability is determined as the target execution device.

[0019] The capability set comprises the local capability set of the terminal device and the remote capability set of any electronic device that meets the connectivity requirements with the terminal device. The local capability set includes capabilities possessed by the terminal device, and each capability corresponds to a capability description and status information. The capability description is in natural language and describes the function achieved by the corresponding capability. The capability status information can indicate whether the capability is available and the degree of function achieved by the capability. The remote capability set includes capabilities possessed by electronic devices that meet the connectivity requirements with the terminal device, and each capability corresponds to a capability description and status information.

[0020] After creating the target requirement, at least one target execution device remains uncertain. At this point, the capability needed to perform the first action can be determined from the local capability set of the terminal device or the remote capability set of any electronic device. This capability can be recorded as the first capability, and the first capability serves as the target capability in the target requirement. Furthermore, the execution device that can call the first capability to perform the first action is also determined. This can be recorded as the first execution device, and the first execution device serves as the target execution device in the target requirement. Therefore, after determining the first capability and the first execution device, the target action, the target capability, and the target execution device in the target requirement are complete.

[0021] The local capability set of an end device includes the capabilities of all functional target modules installed on the end device, while the remote capability set of an electronic device includes the capabilities of all functional target modules installed on the electronic device.

[0022] The target requirement includes, for example: analysis actions, analysis capabilities to perform analysis actions, and executing devices with analysis capabilities, such as servers.

[0023] In another example, the target requirement includes: a text-to-image action, a text-to-image capability to execute the text-to-image action, and an execution device with the text-to-image capability, for example, a local mobile phone.

[0024] In another example, the target requirement includes: a screen sharing action, a screen sharing capability to perform the screen sharing action, and an executing device with the screen sharing capability, for example, a tablet.

[0025] Step 104: In response to the target requirement, respond to the target action with the first capability based on the first execution device to fulfill the user's intent.

[0026] In response to the fact that the first executing device is a target device in an electronic device, the first executing device transmits a call instruction to call the first capability and input contents provided to the first capability via a communication channel to the target device.

[0027] In response to the fact that the first device is an end device, input content is provided to the first capability in response to the call instruction to call the first capability, so that the functional submodule that implements the first capability on the end device processes the input content to produce an output result.

[0028] The output result can be provided to the user as feedback information on the input data or simply recorded.

[0029] In one example, a mobile phone receives the input data "Transfer the image from the mobile phone to the television" from a voice assistant. The mobile phone cannot yet analyze the input data. At this point, the target action in the target request can be defined as the action of analyzing the input data. The analysis capability for performing the analysis action and the execution device with analysis capability, for example, a server, are then defined in the capability set. In response to this target request, the mobile phone sends a call instruction to invoke the analysis capability and the input content provided to the analysis capability over the communication channel between the mobile phone and the server. The input content is the input data "Transfer the image from the mobile phone to the television."Based on this, the server responds to the call instruction and invokes the analysis capability to process the received input. For example, the server inputs the data "Transfer the image from the mobile phone to the television" into a large language model with analysis capabilities, and the large language model outputs the analysis result, for example, the output result, and returns the analysis result to the mobile phone via the connection channel, as shown in [example]. Fig. Figure 3 shows that data analysis can be achieved on a mobile phone without any analytical capabilities.

[0030] In one example, a mobile phone receives input data from a voice assistant, such as "Create a picture of a dog with big eyes." In response to this input, the phone can analyze the input and determine that the target action in the target request is a text-to-image action. It then identifies the text-to-image capability and the device with that capability, such as the phone itself, that can perform the text-to-image action, from the capability set. In response to the target request, the phone inputs the content "Create a picture of a dog with big eyes" into the text-to-image model installed on the phone. The text-to-image model then outputs a picture of a dog with big eyes, which is the output result, and the picture of a dog with big eyes is displayed to the user.

[0031] In another example, the mobile phone receives the input data "Transfer the image from the mobile phone to the television" from the voice assistant. In response to this input data, the mobile phone analyzes it. At this point, it can be determined that the target action in the target request is the screen transfer action. The screen transfer capability for executing the screen transfer action and the executing device with screen transfer capability, for example, a tablet, are then determined from the capability set. In response to the target request, the mobile phone sends a call instruction to invoke the screen transfer capability and the transfer image provided to the screen transfer capability to the tablet via the communication channel between the mobile phone and the tablet. Based on this, the tablet responds to the call instruction and invokes the screen transfer capability.For example, the tablet transmits the received screen mirroring image through the installed screen mirroring application, as in . Fig. Figure 4 shows that, as a result, screen mirroring of the transmitted image can be achieved on a mobile phone without screen mirroring capability.

[0032] It is evident from the above technical solution that, in a control method used in certain embodiments of the present disclosure, when a target action is executed on an end device, the input data representing the user's intention is transformed into a target request that includes the target action. The first capability in the target request and the first execution device of the first capability are then determined from the set of capabilities, and the first capability is used to respond to the target action based on the first execution device. Here, the first capability can be a capability possessed by the end device or a capability possessed by an electronic device that has a connection to the end device. Therefore, the first execution device can be both an end device and a target device within the electronic device.Thus, if the terminal device cannot provide the first capability, the first execution device is the target device in the electronic device. In response to the first execution device being the target device, a call instruction and input content can then be sent to the target device via the connection channel. The target device then uses the call instruction to invoke the first capability to process the input content, thereby responding to the target action on the target device to fulfill the user's intent. In certain embodiments, even if the terminal device cannot execute the target action, the target action can still be executed via a target device that meets the connection requirements with the terminal device, thus improving the reliability of the target action's execution.

[0033] In one implementation, determining from the capability set that the first capability fulfills the target capability, and determining the first execution device of the first capability as the target execution device in 103 can be achieved as follows:

[0034] Based on prioritizing the local skill set over the remote skill set, determining from the skill set that the first skill fulfills the target skill, and determining the first execution device of the first skill as the target execution device.

[0035] In certain embodiments, the first capability fulfilling the target capability and the first execution device of the first capability can be determined from the local capability set as the target execution device; if the first capability fulfilling the target capability cannot be determined from the local capability set, then the first capability fulfilling the target capability and the first execution device of the first capability can be determined from the remote capability set as the target execution device.

[0036] In one scenario, the endpoint cannot analyze the input data. In this scenario, the description of the first action and the description of the first capability can be obtained in step 102 when converting the input data into the target requirement. The first action is the target action. The description of the first action is the description of the analysis action, and the description of the first capability is the description of the analysis capability.

[0037] In step 103, the terminal device, because it cannot analyze the input data, lacks the necessary analytical capability, and no analytical capability exists in the local capability set. At this point, based on the description of the first action and the description of the first capability, the analytical capability and the execution device with analytical capability can be determined from the remote capability set. Thus, the target requirement is met: analytical action, analytical capability to execute the analytical action, and an execution device with analytical capability, for example, a server.

[0038] In another scenario, the terminal is able to analyze the input data. In this scenario, the input data can be processed in step 102, when converting it into the target request, based on the terminal's natural language understanding model, to obtain the description of the first action and the description of the first capability.

[0039] In one example, the description of the first action could be the description of the text-to-image action, and the description of the first skill could be the description of the text-to-image skill. In another example, the description of the first action could be the description of the screen-sharing action, and the description of the first skill could be the description of the screen-sharing skill.

[0040] In step 103, the first capability that fulfills the target capability and the first execution device of the first capability from the local capability set can be determined as the target execution device; if no first capability that fulfills the target capability can be determined from the local capability set, then the first capability and the execution device with the first capability from the remote capability set are determined, thereby achieving the target requirement: the first action, the first capability to perform the first action, and the execution device with the first capability, for example, a mobile phone with text-to-image capabilities or a tablet with screen-transmitting capabilities.

[0041] In the above implementation, the execution of the first action depends not only on the first executing device, but also on other devices. For example, if the first action is screen transmission, the screen transmission depends on both the source device and the target device.

[0042] In certain embodiments, converting step 102 into a target requirement also includes obtaining the associated devices for the target execution device in order to perform the target action.

[0043] Therefore, after determining the first capability and the first execution device in step 103, the second execution device required by the first execution device to perform the first action can also be determined as an associated device. The resulting target requirement includes not only the first action, the first capability, and the first execution device, but also the second execution device associated with the first execution device to perform the first task.

[0044] In step 103, the second execution device can be determined from other devices connected to the first execution device.

[0045] In one example, the mobile phone uses a natural language understanding model to process the action "Transmit the image from the mobile phone to the television" in order to determine the description of the first action in the target requirement "Transmit Action" and the description of the first capability "Transmit the image from the mobile phone to the television." Based on this, in step 103, the first capability "Transmit the image from the mobile phone to the television" is determined from the remote capability set, and the first executing device capable of "Transmitting the image from the mobile phone to the television," for example, a tablet, is also determined. The second executing device, for example, "television" (TV), which can serve as the transmission target, can also be determined from the air conditioner, the washing machine, and the television that are connected to the tablet.Based on this, the mobile phone, in response to the target request, sends the transmission image and the call instruction to invoke the transmission capability to the tablet via the connection channel between the mobile phone and the tablet. The tablet then transmits the transmission image to the television according to the call instruction, as shown in [reference]. Fig. 5 is shown.

[0046] In an implementation, the skills in the local skill set can be acquired as follows: At least one capability of the terminal device is obtained from the terminal device's system registration database. The functional target module in the terminal device then registers its capabilities in the system registration database.

[0047] For example, the capabilities of each hardware and software module on a phone are registered in the system registry database when they are installed or launched. Furthermore, each hardware and software module on the phone updates its capabilities in the system registry database when it is upgraded or updated. Based on this, any functional target module on the phone, such as a service center or an AI application, can inherit the phone's capabilities from the system registry database. These capabilities constitute the local capability set.

[0048] In another implementation, the skills in the local skill set can be acquired as follows: Each functional target module on the end device can query any other functional target module about its declared capabilities. This allows each functional target module on the end device to acquire capabilities owned by the end device.

[0049] For example, any hardware module or software module on a mobile phone can query any other software module or hardware module about its declared capabilities, as in Fig. Figure 6 shows that any hardware or software module on a mobile phone, for example a smart controller or an AI application, can acquire capabilities of the mobile phone.

[0050] In one implementation, the skills in the remote skills set can be acquired through: Receiving the capabilities transmitted by each electronic device via the communication channel between the terminal device and the respective electronic device.

[0051] The capabilities transmitted by the electronic device are obtained by querying the capabilities of a functional target module in the electronic device or by obtaining the capabilities from the system registration database of the electronic device.

[0052] The process by which an electronic device acquires its own capabilities may be similar to the process by which a terminal device acquires its local set of capabilities; it will not be discussed further here. Based on this, any functional target module on an electronic device, for example a SC, can acquire all the capabilities of the electronic device.

[0053] An end device can receive the capabilities transmitted by the SC in the electronic device via a communication channel. The capabilities transmitted by the SC constitute the local capability set of the electronic device, which is the remote capability set for the end device.

[0054] In one example, a communication channel between the tablet and the mobile phone is established through their respective system controllers (SCs). Each hardware or software module on the tablet, for example the SC, can obtain the tablet's capabilities from the tablet's system registry database, or the SC can query the capabilities declared by another software or hardware module on the tablet, as shown in Fig. Figure 7 shows that any hardware or software module on the tablet, for example the SC, can acquire the tablet's capabilities. These capabilities are sent from the tablet's SC to the SC on the mobile phone via the connection channel. These capabilities constitute a remote capability set.

[0055] Any hardware or software module on the mobile phone, for example a SC, can obtain the mobile phone's capabilities from the mobile phone's system registry database, or the SC can query the capabilities declared by another software or hardware module on the mobile phone, as in Fig. Figure 7 shows that in this way, any hardware or software module on the mobile phone, for example a SC, can acquire the capabilities of the mobile phone. These capabilities are sent from the mobile phone's SC to the SC on the tablet via the connection channel. These capabilities constitute the remote capability set.

[0056] In certain embodiments, status update information for a capability in the remote capability set is obtained via the link channel, and the status information of the corresponding capability in the remote capability set is updated based on the status update information.

[0057] In certain embodiments, a functional target module on the electronic device queries the capabilities declared by another functional target module in real time or periodically. Thus, each functional target module, for example, the SC, can detect the status update information of each capability, such as capability upgrades or capability replacements. The electronic device can send the status update information of the changed capabilities to the terminal device via the communication channel. The terminal device can then update the status information of the corresponding capability in the remote capability set based on this status update information.

[0058] For example, when the SC on the server detects that a capability upgrade is available, it sends the capability's status update information to the mobile SC via the connection channel established with the mobile SC. The mobile SC can then update the status information of the corresponding capability in the remote capability set on the mobile phone according to the capability's status update information, as shown in Fig. 8 is shown.

[0059] When the SC on the mobile phone detects that it has a capability that needs upgrading, it sends the capability's status update information to the SC on the server via the connection channel established with the SC on the server. The SC on the server can then update the status information of the corresponding capability in the remote capability set on the server according to the capability's status update information, as described in Fig. 8 is shown.

[0060] In certain embodiments, step 104, in which the first executing device responds to the target action using the first capability, can be implemented as follows:

[0061] Because the first execution device is an end device, the input content can be sent to the first capability via the associated first invocation service, which invokes the first capability. The output result of the first capability, which processes the input content, can then be obtained via the first invocation service.

[0062] For example, any software or hardware module on a mobile phone can query the capabilities declared by another software or hardware module and connect to the calling service that invokes those capabilities. Based on this, if a software or hardware module on the mobile phone needs to invoke an initial capability, such as the text-to-image capability, it can input the input "Generate a picture of a dog with big eyes" into the text-to-image model installed on the mobile phone via the initial calling service, such as the service that invokes the text-to-image capability. The text-to-image model outputs the image of the dog with big eyes, which is the output result. Then, the image of the dog with big eyes is obtained by calling the text-to-image capability's calling service, and then the image of the dog with big eyes is displayed to the user.

[0063] In certain embodiments, step 104, in which the first executing device with the first capability responds to the target action, can be achieved as follows: In response to the fact that the first execution device is the target device, the call instruction and input contents can be sent to the target device via a connection channel, and then the output result returned by the target device can be received via the same connection channel. The input contents can be input data or input parameters that enable the first capability.

[0064] The target device sends input content to the first capability via the first calling service, which calls the first capability to which it is connected, and then obtains an output result via the first calling service, which is obtained by processing the input content by the first capability.

[0065] For example, any software or hardware module on a mobile phone can query the capabilities declared by another software or hardware module and connect to the invocation service that calls those capabilities. The mobile phone and the server can communicate with each other about capabilities via the communication channel. Based on this, when it is necessary for a software or hardware module on the mobile phone to invoke the first capability, for example, the analysis capability, the mobile phone sends an invocation instruction to call the analysis capability and the input content provided to the analysis capability to the server via the communication channel between the mobile phone and the server.The input content can be input data. Then, any module on the server that receives the call instruction and the input content can call the associated first call service, for example, the call service for invoking the analysis capability, to input the content "Transfer the image from the mobile phone to the television" into the large language model with analysis capability. The large language model outputs the analysis result, for example, the output result. The analysis result is then obtained by calling the call service for invoking the analysis capability, and the analysis result is returned to the mobile phone via the connection channel. In this way, data analysis is achieved on a mobile phone without analysis capability.

[0066] Fig. Figure 9 is a schematic diagram of the structure of an end device provided in certain embodiments of the present disclosure. The end device may comprise: a memory 901, which is set up to hold a computer program and To store data generated by the execution of the computer program; a 902 processor configured to run the computer program to implement: Obtaining input data that represents a user's intent; In response to the input data, converting the input data into a target requirement, which includes a target action, a target capability, and a target execution device; as a response to the conversion, determining, from a set of capabilities, an initial capability that fulfills the target capability, and an initial execution device of the initial capability as the target execution device; wherein the capability set comprises: the local capability set of the terminal device and the remote capability set of at least one electronic device that meets the connectivity requirements with the terminal device; wherein the first executing device responds to the target request using the first capability to perform the target action in order to fulfill the user's intention; wherein, in response to the fact that the first execution device is a target device in the electronic device, a call instruction to call the first capability and input contents provided to the first capability are transmitted to the target device via a link channel.

[0067] The processor on the end device can use at least one functional target module, such as a SC or an AI application.

[0068] As in Fig. As shown in Figure 10, a mobile phone, for example, has modules such as a voice assistant, a SC (Self-Containing Device), and a text-to-image model installed. The voice assistant is capable of compiling spoken language or text, the SC is capable of robust networking, and the text-to-image model has text-to-image capability. A SC and a screen-sharing application, which has screen-sharing capability, are installed on the tablet. A SC and a large-scale language model, capable of data analysis, are installed on the server. The mobile phone and the tablet establish a connection channel via their respective SCs, the mobile phone and the server establish a connection channel via their respective SCs, and the server and the tablet establish a connection channel via their respective SCs.The SC on the mobile phone can compile the local set of capabilities, the SC on the server can compile the local set of capabilities, and the SC on the tablet can compile the local set of capabilities. The mobile phone, the server, and the tablet communicate with each other via the connection channel established by the SC. The local capabilities of the server and the tablet are then transferred to the phone as its remote capabilities. The local capabilities of the phone and the tablet are then transferred to the server as its remote capabilities. The local capabilities of the server and the phone are then transferred to the tablet as its remote capabilities. The phone, the server, and the tablet can use their respective local capabilities, or they can use each other's remote capabilities via the connection channel.

[0069] Based on this, the mobile phone can obtain the input data "Transfer the image from the mobile phone to the television" from the voice assistant. In response to the input data, the mobile phone cannot yet analyze it. At this point, it can be determined that the target action in the target request is the action of analyzing the input data. Then, the analysis capability for executing the analysis action and the executing device with analysis capability, for example, the server, are specified in the capability set. In response to the target request, the mobile phone establishes a connection channel to the server via the SC (Self-Containing Device). A call instruction to invoke the analysis capability and the input content provided to the analysis capability are sent. The input content can be input data. Based on this, the server responds to the call instruction and invokes the analysis capability to process the received input content.For example, the server inputs the data "Transfer the image from the mobile phone to the television" into the large language model with analytical capabilities. The large language model outputs the analysis result, for example, the output result. The analysis result is returned to the mobile phone via the connection channel established by the SC. In this way, data analysis is achieved on the mobile phone without analytical capabilities.

[0070] In one example, a mobile phone receives input data from a voice assistant, such as "Create a picture of a dog with big eyes." In response to this input, the phone can analyze the input and determine that the target action in the target request is a text-to-image action. It then identifies the text-to-image capability and the device with that capability, such as the phone itself, that can perform the text-to-image action, from the capability set. In response to the target request, the phone inputs the content "Create a picture of a dog with big eyes" into the text-to-image model installed on the phone. The text-to-image model then outputs a picture of a dog with big eyes, which is the output result, and the picture of a dog with big eyes is displayed to the user.

[0071] In another example, the mobile phone receives the input data "Transfer the image from the mobile phone to the television" from the voice assistant. In response to this input data, the mobile phone analyzes it. At this point, it can be determined that the target action in the target request is the screen transmission action. The screen transmission capability for executing the screen transmission action and the executing device with screen transmission capability, for example, a tablet, are then specified in the capability set. In response to the target request, the mobile phone sends a call instruction to invoke the screen transmission capability and the transmission image provided to the screen transmission capability to the tablet via the connection channel established by the SC (Self-Containing Device). Based on this, the tablet responds to the call instruction and invokes the screen transmission capability.For example, the tablet transmits the received transmission image to the television, thus realizing the transmission output of the transmission image on a mobile phone without screen transmission capability.

[0072] It is evident from the above technical solution that, in an end device used in certain embodiments of the present disclosure, when a target action is executed on the end device, the input data representing the user's intention is transformed into a target request that includes the target action. The first capability in the target request and the first execution device of the first capability are then determined from the set of capabilities, and the first capability is used to respond to the target action based on the first execution device. Here, the first capability can be a capability possessed by the end device or a capability possessed by an electronic device that has a connection to the end device. Therefore, the first execution device can be either an end device or a target device within the electronic device.If the terminal device cannot provide the first capability, the first execution device is the target device in the electronic device. Because the first execution device is the target device, the call instruction and input content can be sent to the target device via the connection channel. This allows the target device to call the first capability through the call instruction to process the input content and thereby respond to the target action on the target device to fulfill the user's intention. As can be seen, even if the terminal device cannot execute the target action, the target action can still be executed by the target device, which meets the connection requirements with the terminal device, thus improving the reliability of the target action's execution.

[0073] Taking the device networking scenario as an example, the following illustrates the technical solution according to certain embodiments of the present disclosure: In the age of device networking, users are no longer limited to performing tasks on a single device. Therefore, when multiple devices are available, operations can be performed primarily on a single main device. However, instructions given must be parsed. If it is detected that a local personal computer (PC) is capable of processing them, the PC processes them. However, there may be scenarios where instructions given on the PC require processing on a phone or tablet. In one example, a user gives an instruction on the PC and types "Please help me mirror my phone screen to my tablet" into the PC input field.

[0074] In certain embodiments, the present disclosure provides the following solution: First, install SCs on PCs, phones, and tablets. SCs determine tasks based on the "From" and "On" parameters.

[0075] For example, a user types "Please enable screen mirroring from my phone to my tablet" on a PC. The PC analyzes the input and recognizes that the task is to mirror the screen from the phone to the tablet. The PC then locates the execution device. If none of the source / to devices are local, it looks for the task on a paired device. It then searches for the corresponding task on the paired device and executes it.

[0076] Fig. Figure 11 is an example diagram for analyzing cross-device tasks and transferring local non-executable tasks to executable devices. The PC performs task analysis on input data, such as text or speech input, related to the task. Upon obtaining the target requirement, it performs task coupling, i.e., invokes capabilities. If the PC cannot couple the task, it cannot execute it. If the PC can couple the task, and the device is either from or to the local PC, it is a local task and can be executed on the PC. If the PC can couple the task but is neither from nor to the local PC, it must couple the task device, and the coupled task device executes the task. The task can be transferred to the coupled task device via the Strong Networking application.

[0077] Taking the device networking scenario as an example, the following illustrates the technical solution according to certain embodiments of the present disclosure: In the age of device networking, even after the Service Center has completed device pairing and established a connection channel, users still rely on manual clicks and selections to access cross-device functions, thus failing to achieve truly automated networking. For example, to transfer a mobile application, a user may need to open the application list, swipe to find the application, and then click to open it.

[0078] In light of this, certain embodiments of the present disclosure provide the following solution: The SC in the mobile phone provides an input field that accepts voice or text input and converts natural language into action instructions, enabling tasks to be opened with a single touch. Furthermore, due to the SC's device coupling feature, the voice-to-action capability is no longer limited to triggering the device itself; any paired device with this capability can complete the task.

[0079] A user has an AI-enabled PC, an ordinary phone, and an ordinary tablet. "Ordinary" here means that it does not support AI capabilities or large language models (LLMs). Based on this, the user can issue a voice command on their ordinary phone: "Transfer my WeChat screen from my phone to my computer." Upon receiving this command, the phone dynamically queries connected devices and local capabilities to determine which devices support this analysis function. The command is then sent to the AI-enabled PC for analysis, resulting in the task and parameters, for example, transferring the screen from the phone to the PC. This enables automatic screen transfer in certain embodiments of the present disclosure.

[0080] As in Fig. Figure 12 shows an example diagram for converting spoken language into an executable task by invoking the speech-to-task capability across devices. The phone receives input data, such as text or voice input, related to the task. If the phone supports task analysis, it performs task analysis on the input data. If the PC supports task analysis, the phone sends the input data to the PC via a SC (Speaking Channel), which performs the task analysis. The phone can also request the PC to perform the task analysis. If the tablet supports task analysis, the phone sends the input data to the tablet via a SC, which performs the task analysis. The phone can also request the tablet to perform the task analysis.If the cloud server supports task analysis, the phone sends the input data via a cloud server (SC) to the cloud server, which then performs the task analysis. The phone can also request the cloud server to perform task analysis.

[0081] Fig. Figure 13 shows an interaction sequence diagram for the aggregation of capabilities and communication via SCs in a device networking scenario. In this diagram, AI applications and other functional modules exhibiting AI capabilities, such as text-to-image conversion, are installed on a laptop, while applications and other functional modules requiring text-to-image conversion are installed on a mobile phone. SCs are installed on both the laptop and the mobile phone. The process can be as follows:

[0082] First, each application installed on a laptop declares its own capabilities, such as the AI ​​graphics capability declared by an AI application or the strong networking capability declared by the SC. Each application on the laptop can query the capabilities declared by other applications and connect to the calling service for those capabilities. For example, the SC on the laptop can query the text-to-image capability declared by an AI application and connect to the calling service for that capability. As a result, each functional module on the laptop can have a local set of capabilities for the laptop.

[0083] Every application installed on a mobile phone declares its own capabilities, such as the AI ​​image generation capability declared by an AI application or the strong connectivity capability declared by the service provider (SC). Every application on a phone can query the capabilities declared by other applications and connect to the service that invokes those capabilities. For example, other applications on a phone can query the strong connectivity capability declared by the SC and connect to the service that invokes that capability. As a result, every functional module on a phone can have a local set of capabilities.

[0084] Secondly, the laptop's SC can manage the skills it has queried via the SmartConnect Skill Manager (MGR) and store them in its skill list; the mobile phone's SC can manage the skills it has queried via its MGR and store them in its skill list. Furthermore, the laptop and mobile phone can establish a private connection channel via their respective SCs and communicate with each other over this channel regarding the skills in their respective MGR skill lists. As a result, the MGR skill list of the laptop's SC includes the skills of all applications on the mobile phone—that is, the laptop's remote skill set—and the MGR skill list of the mobile phone includes the skills of all applications on the laptop, such as the AI ​​image generation skill—that is, the mobile phone's remote skill set.

[0085] Furthermore, the SCs on the laptop and phone can receive status updates from each other and update other local applications in real time regarding the other's capabilities. For example, if the laptop's AI imaging capability changes, the SC on the phone can obtain the new capabilities via the connection channel and update the application that was previously connected to the AI ​​imaging service.

[0086] Fig.Figure 14 is an interaction sequence diagram for implementing capability calls via SCs in a device networking scenario. The mobile phone application receives input data representing the need for AI image generation, for example, "Generate a picture of a dog," and first queries whether an AI image generation capability exists in the local capability set. If so, the input data is provided to the AI ​​image generation capability via the call service associated with the mobile phone application. After the AI ​​image generation capability has generated the image, the image is returned to the mobile phone application. If no AI image generation capability exists in the local capability set, then the execution device with an AI image generation capability, for example, a laptop, is queried in the remote capability set.Based on this, the call instruction pointing to the AI ​​image generation capability and the input "Generate an image with a dog" are sent to the SC on the mobile phone. The SC on the mobile phone sends the call instruction and the input to the AI ​​image generation capability via the connection channel to the SC on the laptop. The SC on the laptop calls the AI ​​image generation capability in the AI ​​application on the laptop by calling the service. For example, by calling the service, the call instruction and the input are passed to the AI ​​application, and the AI ​​application generates an image. The AI ​​application returns the generated image, i.e., the output result, to the SC on the laptop via the callback function. The SC on the laptop sends the output result to the SC on the mobile phone via the connection channel.The SC on the mobile phone returns the output result to the application on the mobile phone, thus realizing the AI ​​image generation capability of the laptop on the mobile phone application.

[0087] Several embodiments have been described, each focusing on the differences from other embodiments. Reference can be made to the common and similar parts of the various embodiments. Since the devices disclosed in certain embodiments correspond to the methods disclosed in certain other embodiments, their description is relatively simple, and for relevant parts, reference can be made to the method description.

[0088] The units and algorithm steps of the examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been described in the above description in terms of their functions. Whether these functions are performed in hardware or software depends on the implementation and the design constraints of the technical solution. The person skilled in the art can use various methods to implement the described functions; however, such an implementation should not be considered to be outside the scope of this disclosure.

[0089] Methods or algorithms described in connection with the embodiments disclosed herein can be implemented directly using hardware, a software module executed by a processor, or a combination of both. The software module can be stored in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable storage medium, a CD-ROM, or any other type of storage medium.

[0090] The above description of the disclosed embodiments is intended to enable the person skilled in the art to implement or use the present disclosure. Various modifications of these embodiments are obvious to the person skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not limited to the embodiments shown herein but is intended to correspond to the broadest scope compatible with the principles and novel features disclosed herein.

Claims

[1] Control procedures, including: Obtaining input data that represents a user intent; In response to the input data, converting the input data into a target requirement, which includes a target action, a target capability, and a target execution device; in response to the conversion, determining, from a set of capabilities, a first capability that fulfills the target capability, and a first execution device for the first capability as the target execution device; wherein the capability set comprises: a local capability set of an end device and a remote capability set of at least one electronic device that fulfills a connectivity requirement with the end device; In response to the target request, react to the target action with the first capability based on the first execution device to fulfill the user intent; In response to the fact that the first execution device is a target device among the electronic devices, transmit, via a communication channel to the target device, a call instruction to call the first capability and input contents provided to the first capability. [2] The method of claim 1, wherein determining the first capability and the first implementing device from the set of capabilities comprises: Prioritizing the local skill set over the remote skill set; and Determine, from the set of capabilities, the first capability that fulfills the target capability, and the first execution device for the first capability as the target execution device. [3] Method according to claim 1, wherein the conversion to the target requirement comprises: Obtaining a description of an initial action and a description of the initial ability. [4] Method according to claim 1, wherein the conversion to the target requirement comprises: Processing the input data based on a natural language understanding model to obtain a description of an initial action and a description of the initial capability. [5] Method according to claim 4, wherein the conversion to the target requirement comprises: Acquisition of associated devices of the target execution device for executing the target action. [6] The method of claim 1, wherein capabilities in the local capability set are obtained by: Obtain at least one capability possessed by the terminal from a system registration database of the terminal, wherein a functional target module in the terminal enters capabilities into the system registration database; or Using any functional target module in the terminal to query another functional target module about its declared capabilities. [7] The method of claim 1, wherein the capabilities in the remote capability set are acquired by: Receiving capabilities transmitted by the electronic device via the communication channel, wherein the capabilities transmitted by the electronic device are obtained by querying capabilities of a functional target module in the electronic device or by obtaining capabilities transmitted by the electronic device from a system registration database of the electronic device. [8] Method according to claim 7, further comprising: Obtaining status update information for a capability in the remote capability set via the communication channel; Updating status information of the corresponding capability in the remote capability set based on the status update information. [9] Method according to claim 1, wherein responding to the target action comprises the first capability: In response to the fact that the first execution device is the target device, the call instruction and the input contents are transmitted to the target device via the link channel, the target device transmitting the input contents to the first capability via a connected first call service that calls the first capability and obtaining an output result from the first capability that processes the input contents via the first call service; and Receiving the output result returned by the target device via the connection channel. [10] Electronic device comprising: a memory in which computer program instructions are stored; and a processor connected to the memory and configured to execute and carry out the computer program instructions: Obtaining input data that represents a user intent; In response to the input data, converting the input data into a target requirement, which includes a target action, a target capability, and a target execution device; in response to the conversion, determining, from a set of capabilities, a first capability that fulfills the target capability, and a first execution device for the first capability as the target execution device; wherein the capability set comprises: a local capability set of an end device and a remote capability set of at least one electronic device that fulfills a connectivity requirement with the end device; In response to the target request, react to the target action with the first capability based on the first execution device to fulfill the user intent; In response to the fact that the first execution device is a target device among the electronic devices, transmit, via a communication channel to the target device, a call instruction to call the first capability and input contents provided to the first capability. [11] Electronic device according to claim 10, wherein determining the first capability and the first implementing device from the set of capabilities comprises: Prioritizing the local skill set over the remote skill set; and Determine, from the set of capabilities, the first capability that fulfills the target capability, and the first execution device for the first capability as the target execution device. [12] Electronic device according to claim 10, comprising converting into the target requirement: Obtaining a description of an initial action and a description of the initial ability. [13] Electronic device according to claim 10, comprising converting into the target requirement: Processing the input data based on a natural language understanding model to obtain a description of an initial action and a description of the initial capability. [14] Electronic device according to claim 13, comprising converting into the target requirement: Acquisition of associated devices of the target execution device for executing the target action. [15] Electronic device according to claim 10, wherein capabilities in the local capability set are obtained by: Obtain at least one capability possessed by the terminal from a system registration database of the terminal, wherein a functional target module in the terminal enters capabilities into the system registration database; or Using any functional target module in the terminal to query another functional target module about its declared capabilities. [16] Electronic device according to claim 10, wherein capabilities in the remote capability set are obtained by: Receiving capabilities transmitted by the electronic device via the communication channel, wherein the capabilities transmitted by the electronic device are obtained by querying capabilities of a functional target module in the electronic device or by obtaining capabilities transmitted by the electronic device from a system registration database of the electronic device. [17] Electronic device according to claim 16, wherein the processor is further configured to perform: Obtaining status update information for a capability in the remote capability set via the communication channel; Updating status information of the corresponding capability in the remote capability set based on the status update information. [18] Electronic device according to claim 10, wherein responding to the target action comprises the first capability: In response to the fact that the first execution device is the target device, the call instruction and the input contents are transmitted to the target device via the link channel, the target device transmitting the input contents to the first capability via a connected first call service that calls the first capability and obtaining an output result from the first capability that processes the input contents via the first call service; and Receiving the output result returned by the target device via the connection channel. [19] Non-volatile, computer-readable storage medium on which computer program instructions are stored that can be executed by at least one processor to perform: Obtaining input data that represents a user intent; In response to the input data, converting the input data into a target requirement, which includes a target action, a target capability, and a target execution device; in response to the conversion, determining, from a set of capabilities, a first capability that fulfills the target capability, and a first execution device for the first capability as the target execution device; wherein the capability set comprises: a local capability set of an end device and a remote capability set of at least one electronic device that fulfills a connectivity requirement with the end device; In response to the target request, react to the target action with the first capability based on the first execution device to fulfill the user intent; In response to the fact that the first execution device is a target device among the electronic devices, transmit, via a communication channel to the target device, a call instruction to call the first capability and input contents provided to the first capability. [20] Non-volatile computer-readable storage medium according to claim 19, wherein determining the first capability and the first implementing device from the capability set comprises: Prioritizing the local skill set over the remote skill set; and Determine, from the set of capabilities, the first capability that fulfills the target capability, and the first execution device for the first capability as the target execution device.

Images