Control method and terminal equipment

Abstract

The invention discloses a control method and terminal equipment. The method comprises the steps of obtaining input data; the input data is used for representing user intention; in response to the input data, converting the input data into a target demand comprising a target action, a target capability and a target execution device; in response to the conversion, determining a first execution device with a first capability meeting a target capability and the first capability from a capability set as a target execution device; the capability set comprises a local capability set of the terminal device and a remote capability set of at least one electronic device meeting a connection condition with the terminal device; responding to a target demand, and responding to the target action with a first capability based on a first execution device so as to meet a user intention; and if the first execution equipment is target equipment in the electronic equipment, sending a calling instruction for calling the first capability and input content provided for the first capability through a connection channel with the target equipment.

Description

Technical Field

[0001] The present application relates to the technical field of device interconnection, and in particular to a control method and a terminal device. Background Art

[0002] Currently, when a terminal device performs a certain action, it usually needs to use a functional module that can provide corresponding capabilities. However, there are situations where the terminal device does not have the functional module with this capability or the functional module cannot be used, resulting in low reliability of the action being executed. Summary of the invention

[0003] In view of this, the present application provides a control method and a terminal device as follows:

[0004] A control method, comprising:

[0005] Obtaining input data; the input data is used to represent user intention;

[0006] In response to the input data, converting into target requirements including target actions, target capabilities, and target execution devices;

[0007] In response to the conversion, determining from the capability set a first capability that satisfies the target capability and a first execution device of the first capability as the target execution device;

[0008] The capability set includes: a local capability set of the terminal device and a remote capability set of at least one electronic device that meets the connection condition with the terminal device;

[0009] In response to the target requirement, based on the first execution device, responding to the target action with the first capability to meet the user intention;

[0010] If the first execution device is a target device in the electronic device, a calling instruction for calling the first capability and input content provided to the first capability are sent through a connection channel with the target device.

[0011] In the above method, preferably, determining from the capability set that a first capability satisfies the target capability and a first execution device of the first capability as the target execution device includes:

[0012] Based on the local capability set giving priority to the remote capability set, a first capability satisfying the target capability and a first execution device of the first capability are determined from the capability set as the target execution device.

[0013] In the above method, preferably, the conversion into target requirements including target actions, target capabilities and target execution devices includes:

[0014] An expression of a first action and an expression of a first capability are obtained.

[0015] In the above method, preferably, the conversion into target requirements including target actions, target capabilities and target execution devices includes:

[0016] Based on a natural language understanding model, the input data is processed to obtain an expression of the first action and an expression of the first capability.

[0017] In the above method, preferably, the conversion into target requirements including target actions, target capabilities and target execution devices further comprises:

[0018] Obtain an associated device for the target execution device to execute the target action.

[0019] In the above method, preferably, the capabilities in the local capability set are obtained in the following manner:

[0020] Obtaining at least one capability of the terminal device from a system registry of the terminal device; registering the capability of the target functional module in the terminal device to the system registry;

[0021] or,

[0022] Any target functional module in the terminal device is used to query any other target functional module for its declared capability.

[0023] In the above method, preferably, the capabilities in the remote capability set are obtained in the following manner:

[0024] Ability to receive transmission from the electronic device through the connection channel;

[0025] The transmission capability of the electronic device is obtained by querying the capability of any target functional module in the electronic device, or the transmission capability of the electronic device is obtained from a system registration table of the electronic device.

[0026] The above method, preferably, further comprises:

[0027] Obtaining, through the connection channel, status update information of any capability in the remote capability set;

[0028] According to the status update information, the status information of the corresponding capability in the remote capability set is updated.

[0029] The above method, preferably, based on the first execution device responding to the target action with the first capability, includes:

[0030] If the first execution device is the target device, sending the call instruction and the input content to the target device through the connection channel;

[0031] The target device sends the input content to the first capability through the first calling service that calls the first capability, and obtains the output result obtained by the first capability processing the input content through the first calling service.

[0032] The output result returned by the target device is received through the connection channel.

[0033] A terminal device, comprising:

[0034] A memory, used to store a computer program and data generated by the execution of the computer program;

[0035] A processor, configured to execute the computer program to implement:

[0036] Obtaining input data; the input data is used to represent user intention;

[0037] In response to the input data, converting into target requirements including target actions, target capabilities, and target execution devices;

[0038] In response to the conversion, determining from the capability set a first capability that satisfies the target capability and a first execution device of the first capability as the target execution device;

[0039] The capability set includes: a local capability set of the terminal device and respective remote capability sets of at least one electronic device that meets the connection condition with the terminal device;

[0040] In response to the target requirement, based on the first execution device, responding to the target action with the first capability to meet the user intention;

[0041] If the first execution device is a target device in the electronic device, a calling instruction for calling the first capability and input content provided to the first capability are sent through a connection channel with the target device. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0043] Figure 1 A flow chart of a control method provided in an embodiment of the present application;

[0044] Figure 2 This is an example diagram of the interconnection between devices in the embodiment of the present application;

[0045] Figure 3 This is an example diagram of a large language model on a server used by a mobile phone in an embodiment of the present application;

[0046] Figure 4 This is an example diagram of a mobile phone using a tablet device to project the screen externally in an embodiment of the present application;

[0047] Figure 5 This is an example diagram of a mobile phone using a tablet device to project the screen onto a TV in an embodiment of the present application;

[0048] Figure 6 This is an example diagram of the SC on the mobile phone obtaining the capabilities of the mobile phone in the embodiment of the present application;

[0049] Figure 7 This is an example diagram of the SC on the tablet sending capabilities to the SC on the mobile phone in an embodiment of the present application;

[0050] Figure 8 This is an example diagram of the SC on the tablet sending capability status update information to the SC on the mobile phone in an embodiment of the present application;

[0051] Fig. 9 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application;

[0052] Fig.10 This is an example diagram of establishing interconnection between a mobile phone, a server and a tablet device through a SC in an embodiment of the present application;

[0053] Fig.11 This is an example diagram of parsing cross-device tasks and transferring local non-executable tasks to executable devices in an embodiment of the present application;

[0054] Fig.12 This is an example diagram of converting speech into an executable task by invoking the speech-to-task capability across devices in an embodiment of the present application;

[0055] Fig.13 This application is applicable to the interaction sequence diagram of realizing capability collection and capability intercommunication through SC in the device interconnection scenario;

[0056] Fig.14 This application is applicable to the interaction sequence diagram of implementing capability calls through SC in device interconnection scenarios. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0058] refer to Figure 1 As shown in FIG. 1 , a flow chart of a control method provided in an embodiment of the present application is provided. The method can be applied to a terminal device that can meet the connection conditions with at least one electronic device, such as a mobile phone, a tablet device, etc. For example, Figure 2 As shown in , the terminal device can be a mobile phone, and the electronic device that meets the connection conditions with the terminal device can be another mobile phone, a tablet device, a notebook, a television and other devices. The target function module is deployed on both the terminal device and the electronic device. The target function module can be a hardware module or a software module. The software module can be a system module or a third-party application. The target function module can have one or more capabilities, and the target function module can include one or more functional sub-modules, each of which provides a capability. The connection condition is the condition for achieving interconnection between the terminal device and the electronic device. Specifically, the super interconnection application SC (SmartConnect) is deployed on the terminal device and the electronic device, and a connection channel is established between the terminal device and the electronic device through their respective SCs to achieve interconnection and meet the connection conditions. The technical solution in this embodiment is mainly used to improve the reliability of the execution of the action.

[0059] Specifically, the method in this embodiment may include the following steps:

[0060] Step 101: Obtain input data.

[0061] Among them, the input data is used to characterize the user's intention.

[0062] Specifically, the input data may be obtained through a target application, and the target application may be an artificial intelligence (AI) assistant, such as a voice assistant in a mobile phone, etc. The voice assistant may receive input data such as text or voice that represents the user's intention.

[0063] Step 102: In response to the input data, convert it into target requirements including target actions, target capabilities, and target execution devices.

[0064] Specifically, in this embodiment, a requirement template can be pre-set, and the requirement template contains three fill-in items: fill-in items corresponding to target action, target capability, and target execution device. After obtaining the input data, the target requirement is constructed in response to the input data according to the requirement template. The target requirement consists of target action, target capability, and target execution device.

[0065] It should be noted that if the terminal device cannot parse the input data, then the target action in the converted target requirement is the action of parsing the input data, which can be recorded as the first action, and the target capability is the first capability of the action that can execute the first action, and the target execution device is the device with the target capability, but the target execution device is not determined;

[0066] If the terminal device is able to parse the input data, then the target action in the converted target requirement is the action parsed from the input data, which can be recorded as the first action, such as the action of Wenshengtu or the action of screen projection, etc.; and the target capability is the first capability of the action that can execute the first action, and the target execution device is the device with the target capability, but the target execution device is not determined. In a specific implementation, the target capability can be the capability of the terminal device or the capability of the target device in the electronic device, and the target execution device can be the terminal device or the target device in the electronic device, but the target execution device is not determined.

[0067] Step 103: In response to the conversion, determine from the capability set that the first capability satisfies the target capability and the first execution device of the first capability is used as the target execution device.

[0068] Among them, the capability set includes: the local capability set of the terminal device and the respective remote capability set of each electronic device that meets the connection conditions with the terminal device. The local capability set contains all the capabilities of the terminal device, and each capability corresponds to a capability description and status information. The capability description conforms to natural language, and the capability description describes the function achieved by the corresponding capability through natural language description. The status information of the capability can indicate whether the capability is available and the degree of the function achieved by the capability. The remote capability set contains all the capabilities of each electronic device that meets the connection conditions with the terminal device, and each capability corresponds to a capability description and status information.

[0069] It should be noted that after the target requirement is converted, at least one target execution device in the target requirement is still uncertain. At this time, the capability required to execute the first action can be determined from the local capability set of the terminal device or the remote energy set of each electronic device, which can be recorded as the first capability. The first capability is used as the target capability in the target requirement. In addition, the execution device that can call the first capability to execute the first action is also determined, which can be recorded as the first execution device. The first execution device is used as the target execution device in the target requirement. Therefore, after determining the first capability and the first execution device, the target action, target capability, and target execution device in the target requirement are complete.

[0070] The local capability set of the terminal device is composed of the capabilities of each target functional module deployed on the terminal device, and the remote capability set of each electronic device is composed of the capabilities of each target functional module deployed on the electronic device.

[0071] For example, target requirements include: parsing actions, parsing capabilities for performing parsing actions, and execution devices with parsing capabilities, such as servers;

[0072] For another example, the target requirements include: Wenshengtu actions, Wenshengtu capabilities for executing Wenshengtu actions, and execution devices with Wenshengtu capabilities, such as local mobile phones;

[0073] For another example, the target requirements include: a screen projection action, a screen projection capability for executing the screen projection action, and an execution device with the screen projection capability, such as a tablet device.

[0074] Step 104: In response to the target requirement, based on the first execution device, respond to the target action with the first capability to meet the user's intention.

[0075] If the first execution device is a target device in the electronic device, a calling instruction for calling the first capability and input content provided to the first capability are sent through a connection channel with the target device.

[0076] If the first device is a terminal device, then in response to a call instruction to call the first capability, input content is provided to the first capability, so that a functional submodule implementing the first capability on the terminal device processes the input content to obtain an output result.

[0077] It should be noted that the output result can be provided to the user as feedback information of the input data, or only the output result can be recorded.

[0078] For example, a mobile phone obtains input data "cast the image on the mobile phone to the TV" through a voice assistant. In response to the input data, the mobile phone cannot parse the input data. At this time, the target action in the target requirement can be determined as the action to parse the input data. Then, the parsing capability for performing the parsing action and the execution device with parsing capability, such as a server, are determined in the capability set. In response to the target requirement, the mobile phone sends a call instruction for calling the parsing capability and input content provided to the parsing capability through a connection channel with the server. The input content is the input data "cast the image on the mobile phone to the TV". Based on this, the server responds to the call instruction and calls the parsing capability to process the received input content. For example, the server inputs the input data "cast the image on the mobile phone to the TV" into a large language model with parsing capability, and the large language model outputs the parsing result, i.e., the output result, and returns the parsing result to the mobile phone through the connection channel, such as Figure 3 As shown in , data analysis can be achieved on mobile phones without analysis capabilities.

[0079] For another example, the mobile phone obtains input data "generate an image of a dog with big eyes" through the voice assistant. In response to the input data, the mobile phone can parse the input data. At this time, it can be determined that the target action in the target requirement is the action of the Wensheng picture. Then, the Wensheng picture capability for executing the Wensheng picture action and the execution device with the Wensheng picture capability, such as the mobile phone itself, are determined in the capability set. In response to the target requirement, the mobile phone inputs the input content "generate an image of a dog with big eyes" into the Wensheng picture model deployed on the mobile phone, and the Wensheng picture model outputs the image of the "dog with big eyes", that is, the output result, and outputs the image of the "dog with big eyes" to the user.

[0080] For another example, the mobile phone obtains input data "cast the image on the mobile phone to the TV" through the voice assistant. In response to the input data, the mobile phone can parse the input data. At this time, it can be determined that the target action in the target requirement is the screen casting action. Then, the screen casting capability for executing the screen casting action and the execution device with screen casting capability, such as a tablet device, are determined in the capability set. In response to the target requirement, the mobile phone sends a call instruction for calling the screen casting capability and the cast image provided to the screen casting capability to the tablet device through the connection channel between the mobile phone and the tablet device. Based on this, the tablet device responds to the call instruction and calls the screen casting capability. For example, the tablet device casts the received cast image through the deployed screen casting application, such as Figure 4 As shown in , the screen projection output of the projection image can be achieved on a mobile phone without screen projection capability.

[0081] It can be seen from the above technical scheme that in a control method adopted by the embodiment of the present application, when executing a target action on a terminal device, the input data representing the user's intention is converted into a target requirement containing the target action, and the first capability in the target requirement and the first execution device of the first capability are determined from the capability set, and then the target action is responded to with the first capability based on the first execution device. The first capability here can be the capability of the terminal device, or the capability of the electronic device that satisfies the connection relationship with the terminal device. Thus, the first execution device can be both a terminal device and a target device in the electronic device. In this way, if the terminal device cannot provide the first capability, then the first execution device is the target device in the electronic device, and if the first execution device is the target device, then the call instruction and the input content can be sent through the connection channel with the target device, so that the target device can call the first capability through the call instruction to process the input content, thereby responding to the target action on the target device to meet the user's intention. It can be seen that in this embodiment, even if the terminal device cannot execute the target action, the target action can be executed by the target device that meets the connection conditions with the terminal device, thereby improving the reliability of the execution of the target action.

[0082] In one implementation, in step 103, when determining from the capability set that the first capability satisfies the target capability and the first execution device of the first capability is used as the target execution device, it can be implemented in the following manner:

[0083] Based on the local capability set prioritizing the remote capability set, a first capability satisfying the target capability and a first execution device of the first capability are determined from the capability set as the target execution device.

[0084] Specifically, in this embodiment, the first capability that satisfies the target capability and the first execution device of the first capability can be preferentially determined from the local capability set as the target execution device; if the first capability that satisfies the target capability cannot be determined from the local capability set, then the first capability that satisfies the target capability and the first execution device of the first capability can be determined from the remote capability set as the target execution device.

[0085] In one case, the terminal device cannot parse the input data. In this case, when converted into the target requirement in step 102, the expression of the first action and the expression of the first capability can be obtained. The first action is the target action. At this time, the expression of the first action is the expression of the parsing action, and the expression of the first capability is the expression of the parsing capability.

[0086] Based on this, in step 103, because the terminal device cannot parse the input data, the terminal device does not have the parsing capability, and there is no parsing capability in the local capability set. At this time, according to the expression of the first action and the expression of the first capability, the parsing capability and the execution device with the parsing capability can be determined from the remote capability set. Thus, the target requirements are obtained: parsing action, parsing capability for executing the parsing action, and an execution device with parsing capability, such as a server.

[0087] In another case, the terminal device is capable of parsing the input data. In this case, when converting into the target requirement in step 102, the input data can be processed based on the natural language understanding model of the terminal device to obtain the expression of the first action and the expression of the first capability.

[0088] For example, the expression of the first action is the expression of the Wenshengtu action, and the expression of the first ability is the expression of the Wenshengtu ability. For another example, the expression of the first action is recorded as the expression of the screen casting action, and the expression of the first ability is the expression of the screen casting ability.

[0089] Based on this, in step 103, the first capability that satisfies the target capability and the first execution device of the first capability can be preferentially determined from the local capability set as the target execution device; if the first capability that satisfies the target capability cannot be determined from the local capability set, then the first capability and the execution device with the first capability are determined from the remote capability set, thereby obtaining the target requirements: the first action, the first capability to execute the first action, and the execution device with the first capability, such as a mobile phone with text and image capability or a tablet device with screen projection capability.

[0090] Based on the above implementation, when the first action is executed, in addition to relying on the first execution device, it also relies on other devices. For example, the first action is a screen projection action, which relies on the source device of the screen projection and the destination device of the screen projection.

[0091] Based on this, when step 102 in this embodiment is converted into a target requirement, it also includes obtaining an associated device for the target execution device to execute the target action.

[0092] Therefore, after determining the first capability and the first execution device in step 103, the second execution device required by the first execution device to execute the first action can also be determined as an associated device. At this time, the obtained target requirement includes not only the first action, the first capability, and the first execution device, but also the second execution device associated with the first execution device to execute the first task.

[0093] Specifically, in step 103, the second execution device may be determined from other devices connected to the first execution device.

[0094] For example, the mobile phone uses a natural language understanding model to process the action "cast the image on the mobile phone to the TV" to determine the description of the first action in the target requirement "casting action" and the description of the first capability "casting the image on the mobile phone to the TV". Based on this, in step 103, the first capability of "casting the image on the mobile phone to the TV" is determined from the remote capability set, and the first execution device with "casting the image on the mobile phone to the TV" such as a tablet device is also determined. The second execution device that can serve as the screen casting destination such as "TV" can also be determined from the air conditioner, washing machine and TV connected to the tablet device. Based on this, in response to the target requirement, the mobile phone sends the cast image and the call instruction for calling the cast capability to the tablet device through the connection channel between the tablet device and the tablet device. The tablet device casts the cast image to the TV according to the call instruction. Figure 5 as shown in .

[0095] In one implementation, the capabilities in the local capability set may be obtained in the following manner:

[0096] At least one capability of the terminal device is obtained from the system registry of the terminal device, and the target function module in the terminal device registers its capability in the system registry.

[0097] For example, each hardware module and software module on the mobile phone will register its capabilities to the system registry when it is installed or started; further, each hardware module and software module on the mobile phone will update the updated capabilities to the system registry when the capabilities are upgraded or updated. Based on this, any target functional module on the mobile phone, such as SC or AI application, can obtain the capabilities of the mobile phone from the system registry, and these capabilities constitute the local capability set.

[0098] In another implementation, the capabilities in the local capability set may be obtained in the following manner:

[0099] Any target function module in the terminal device is used to query the declared capabilities of any other target function module. Thus, any target function module on the terminal device can obtain all capabilities possessed by the terminal device.

[0100] For example, any hardware module or software module on a mobile phone can query any other software module or hardware module for its declared capabilities, such as Figure 6 As shown in . Thus, any hardware module or software module on the mobile phone, such as SC or AI application, can obtain all the capabilities of the mobile phone.

[0101] In one implementation, the capabilities in the remote capability set may be obtained in the following manner:

[0102] The ability to receive transmission from each electronic device separately through the connection channel between the terminal device and each electronic device.

[0103] The transmission capability of the electronic device is obtained by querying the capability of any target functional module in the electronic device, or the transmission capability of the electronic device is obtained from a system registration table of the electronic device.

[0104] It should be noted that the way in which an electronic device obtains its own capabilities can refer to the way in which a terminal device obtains a local capability set, which will not be described in detail here. Based on this, any target functional module on an electronic device, such as a SC, can obtain all capabilities of the electronic device.

[0105] Specifically, the terminal device can receive the capabilities transmitted by the SC in the electronic device through the connection channel. The capabilities transmitted by the SC constitute the local capability set of the electronic device itself, which is a remote capability set for the terminal device.

[0106] For example, a tablet device and a mobile phone establish a connection channel through their respective SCs. Any hardware module or software module on the tablet device, such as the SC, can obtain the capabilities of the tablet device from the system registry of the tablet device, or the SC can query the capabilities declared by any other software module or hardware module on the tablet device, such as Figure 7 As shown in , any hardware module or software module such as SC on the tablet device can obtain the capabilities of the tablet device. These capabilities are sent by the SC of the tablet device to the SC on the mobile phone through the connection channel. These capabilities constitute a remote capability set.

[0107] Similarly, any hardware module or software module on the mobile phone, such as SC, can obtain the capabilities of the mobile phone from the system registry of the mobile phone, or SC can query the capabilities declared by any other software module or hardware module on the mobile phone, such as Figure 7 As shown in , any hardware module or software module on the mobile phone, such as SC, can obtain the capabilities of the mobile phone. These capabilities are sent by the SC of the mobile phone to the SC on the tablet device through the connection channel. These capabilities constitute a remote capability set.

[0108] Based on the above implementation, in this embodiment, the status update information of any capability in the remote capability set can also be obtained through the connection channel, and then the status information of the corresponding capability in the remote capability set is updated according to the status update information.

[0109] Specifically, any target functional module on the electronic device queries the capabilities declared by any other target functional module in real time or periodically. As a result, each target functional module, such as the SC, can detect the status update information of each capability, such as capability upgrade or capability replacement information. Then the electronic device can send the status update information of the capability with status change to the terminal device through the connection channel. The terminal device can update the status information of the corresponding capability in the remote capability set according to the status update information.

[0110] For example, when the SC on the server detects that a capability upgrade is available, it sends the status update information of the capability to the SC on the mobile phone through the connection channel established with the SC on the mobile phone. The SC on the mobile phone can update the status information of the corresponding capability in the remote capability set on the mobile phone according to the status update information of the capability, such as Figure 8 as shown in .

[0111] Similarly, when the SC on the mobile phone detects that it has the capability to be upgraded, it sends the status update information of the capability to the SC on the server through the connection channel established with the SC on the server. The SC on the server can update the status information of the corresponding capability in the remote capability set on the server according to the status update information of the capability, such as Figure 8 as shown in .

[0112] Based on the above implementation, in step 104, when the first execution device responds to the target action with the first capability, it can be implemented in the following manner:

[0113] If the first execution device is a terminal device, the input content can be sent to the first capability through the first calling service that calls the first capability; then, the output result obtained by the first capability processing the input content can be obtained through the first calling service.

[0114] For example, any software module or hardware module on a mobile phone can query the capabilities declared by any other software module or hardware module and connect to the calling service that calls these capabilities. Based on this, when any software module or hardware module on the mobile phone needs to call a first capability such as a Vincent image capability, the input content "generate an image of a big-eyed dog" can be input into the Vincent image model deployed on the mobile phone through a first calling service such as a calling service that calls the Vincent image capability. The Vincent image model outputs the image of the "big-eyed dog", that is, the output result, and then obtains the image of the "big-eyed dog" by calling the calling service of the Vincent image capability, and then outputs the image of the "big-eyed dog" to the user.

[0115] Based on the above implementation, in step 104, when the first execution device responds to the target action with the first capability, it can be implemented in the following manner:

[0116] If the first execution device is a target device, the call instruction and input content can be sent to the target device through the connection channel, and then the output result returned by the target device can be received through the connection channel. The input content can be input data, or the input content can be an input parameter that can realize the first capability.

[0117] The target device sends the input content to the first capability through the first calling service that calls the first capability to which it is connected; and then obtains the output result obtained by the first capability processing the input content through the first calling service.

[0118] For example, any software module or hardware module on a mobile phone can query the capabilities declared by any other software module or hardware module, and connect to the calling service that calls these capabilities. Any software module or hardware module on a mobile phone can query the capabilities declared by any other software module or hardware module, and connect to the calling service that calls these capabilities. The mobile phone and the server can achieve capability intercommunication through the connection channel. Based on this, when any software module or hardware module on the mobile phone needs to call the first capability such as the parsing capability, the mobile phone sends the calling instruction for calling the parsing capability and the input content provided to the parsing capability to the server through the connection channel between the mobile phone and the server. The input content can be input data. Afterwards, any module on the server that receives the calling instruction and the input content can call the first calling service connected, such as the calling service for calling the parsing capability, to input the input content "cast the image on the mobile phone to the TV" into the large language model with parsing capability, and the large language model outputs the parsing result, that is, the output result, and then obtains the parsing result by calling the calling service of the parsing capability, and returns the parsing result to the mobile phone through the connection channel, thereby realizing data parsing on a mobile phone without parsing capability.

[0119] refer to Fig. 9 , is a schematic diagram of the structure of a terminal device provided in an embodiment of the present application, and the terminal device may include the following structure:

[0120] Memory 901, used to store computer programs and data generated by the execution of the computer programs;

[0121] The processor 902 is configured to execute the computer program to implement:

[0122] Obtaining input data; the input data is used to represent user intention;

[0123] In response to the input data, converting into target requirements including target actions, target capabilities, and target execution devices;

[0124] In response to the conversion, determining from the capability set a first capability that satisfies the target capability and a first execution device of the first capability as the target execution device;

[0125] The capability set includes: a local capability set of the terminal device and respective remote capability sets of at least one electronic device that meets the connection condition with the terminal device;

[0126] In response to the target requirement, based on the first execution device, responding to the target action with the first capability to meet the user intention;

[0127] If the first execution device is a target device in the electronic device, a calling instruction for calling the first capability and input content provided to the first capability are sent through a connection channel with the target device.

[0128] Specifically, at least one target functional module, such as SC or AI application, can be deployed on the terminal device through the processor.

[0129] For example, Fig.10 As shown in , the mobile phone is deployed with modules such as voice assistant, SC, and text-based graph model. The voice assistant has the ability to collect voice or text, SC has the ability of super interconnection, and text-based graph model has text-based graph ability; SC and screen projection application are deployed on the tablet device, and the screen projection application has the ability of screen projection; SC and large language model are deployed on the server, and the large language model has the ability of data analysis. The mobile phone and the tablet device establish a connection channel through their respective SCs, the mobile phone and the server establish a connection channel through their respective SCs, and the server and the tablet device establish a connection channel through their respective SCs. The SC on the mobile phone can collect the local capability set, the SC on the server can collect the local capability set, and the SC on the tablet device can collect the local capability set. The mobile phone, the server, and the tablet device communicate their respective local capability sets through the connection channel established by SC. As a result, the local capability sets of the server and the tablet device are transmitted to the mobile phone as the remote capability set of the mobile phone; the local capability sets of the mobile phone and the tablet device are transmitted to the server as the remote capability set of the server; the local capability sets of the server and the mobile phone are transmitted to the tablet device as the remote capability set of the tablet device. Mobile phones, servers and tablet devices can use their own local capabilities, or they can use each other's remote capabilities through connection channels.

[0130] Based on this, the mobile phone can obtain the input data "cast the image on the mobile phone to the TV" through the voice assistant. In response to the input data, the mobile phone cannot parse the input data. At this time, the target action in the target requirement can be determined as the action of parsing the input data, and then the parsing capability for executing the parsing action and the execution device with parsing capability, such as the server, are determined in the capability set. In response to the target requirement, the mobile phone sends a call instruction for calling the parsing capability and the input content provided to the parsing capability through the connection channel established with the server through the SC. The input content can be input data. Based on this, the server responds to the call instruction and calls the parsing capability to process the received input content. For example, the server inputs the input data "cast the image on the mobile phone to the TV" into the large language model with parsing capability, and the large language model outputs the parsing result, that is, the output result, and returns the parsing result to the mobile phone through the connection channel established by the SC. In this way, data parsing is realized on the mobile phone without parsing capability.

[0131] For another example, the mobile phone obtains input data "generate an image of a dog with big eyes" through the voice assistant. In response to the input data, the mobile phone can parse the input data. At this time, it can be determined that the target action in the target requirement is the action of the Wensheng picture. Then, the Wensheng picture capability for executing the Wensheng picture action and the execution device with the Wensheng picture capability, such as the mobile phone itself, are determined in the capability set. In response to the target requirement, the mobile phone inputs the input content "generate an image of a dog with big eyes" into the Wensheng picture model deployed on the mobile phone, and the Wensheng picture model outputs the image of the "dog with big eyes", that is, the output result, and outputs the image of the "dog with big eyes" to the user.

[0132] For another example, the mobile phone obtains input data "cast the image on the mobile phone to the TV" through the voice assistant. In response to the input data, the mobile phone can parse the input data. At this time, it can be determined that the target action in the target requirement is the screen casting action, and then the screen casting capability for executing the screen casting action and the execution device with screen casting capability, such as a tablet device, are determined in the capability set. In response to the target requirement, the mobile phone sends a call instruction for calling the screen casting capability and the cast image provided to the screen casting capability to the tablet device through the connection channel established with the tablet device through the SC. Based on this, the tablet device responds to the call instruction and calls the screen casting capability. For example, the tablet device casts the received cast image to the TV, thereby realizing the screen casting output of the cast image on a mobile phone without the screen casting capability.

[0133] It can be seen from the above technical scheme that in a terminal device adopted in the embodiment of the present application, when executing a target action on the terminal device, the input data representing the user's intention is converted into a target requirement containing the target action, and the first capability in the target requirement and the first execution device of the first capability are determined from the capability set, and then the target action is responded to with the first capability based on the first execution device. The first capability here can be the capability of the terminal device, or the capability of the electronic device that satisfies the connection relationship with the terminal device. Thus, the first execution device can be both a terminal device and a target device in the electronic device. In this way, if the terminal device cannot provide the first capability, then the first execution device is the target device in the electronic device, and if the first execution device is the target device, then the call instruction and the input content can be sent through the connection channel with the target device, so that the target device can call the first capability through the call instruction to process the input content, thereby responding to the target action on the target device to meet the user's intention. It can be seen that in this embodiment, even if the terminal device cannot execute the target action, the target action can be executed by the target device that meets the connection conditions with the terminal device, thereby improving the reliability of the execution of the target action.

[0134] Taking the device interconnection scenario as an example, the technical solution of this application is illustrated below:

[0135] In the era of device interconnection, users are no longer limited to performing tasks on a single device. Therefore, when there are many devices, the main operation will be performed on a main device, but the instructions issued need to be parsed. If it is found that the local personal computer (PC) can handle it, then the PC will handle it. However, there may be a scenario where the instructions issued by the PC need to be processed by the phone and tablet. For example, the user issued a command on the PC, and the user entered "Please help me cast the screen on my phone to my tablet" in the input box on the PC.

[0136] In view of this, this application proposes the following solution:

[0137] First, SC is deployed on PC, phone and tablet. SC is determined based on the task and the parameters "From&To".

[0138] For example, a user enters the input data "Please help me turn on the screen projection from my phone to my tablet" on the PC. The PC parses the input data and finds that the task is to project the screen "from the phone to the tablet". At this time, the PC must first find the execution device. If neither From / To is the local device, it will search from the device paired with the local device, and then find the corresponding task on the matching device and execute the task.

[0139] Specifically, Fig.11 As shown in the figure, it is an example diagram for parsing cross-device tasks and transferring local non-executable tasks to executable devices. The PC performs task parsing on input data such as text input or voice input related to the task, and after obtaining the target requirements, performs task matching, that is, calling capabilities. If the PC cannot match the task, the PC cannot execute it; if the PC can match the task, where from or to is the local PC, it is a local task, and the task can be executed on the PC at this time; if the PC can match the task, but from or to is not the local PC, the PC needs to pair the task device, and the paired task device will execute the task. Specifically, the task can be transmitted to the paired task device through the super interconnected application.

[0140] Taking the device interconnection scenario as an example, the technical solution of this application is illustrated below:

[0141] In the era of device interconnection, after SC completes device pairing and establishes a connection channel, users still rely on manual clicks and manual selections to use cross-device functions, and cannot truly complete automated interconnection functions. For example, to complete the screen projection of a mobile app, users need to open the app list, and users can find the app by sliding, and then click to open the app.

[0142] In view of this, this application proposes the following solution:

[0143] The SC in the mobile phone provides an input box to accept voice or text input and convert natural language into action instructions, thereby completing the one-click opening task capability. At the same time, due to the device pairing characteristics of SC, the ability to convert voice into action instructions is no longer limited to triggering the ability of the local device. All paired devices with this ability can complete the task.

[0144] The user has an AI PC, an ordinary phone, and an ordinary tablet. The "ordinary" here means that it does not support AI capabilities and large language models LLM (Large Language Models). Based on this, the user triggers "help me cast the WeChat screen on my phone to my computer" through voice on an ordinary mobile phone. At this time, when the mobile phone receives this command, it will dynamically query the devices connected to it and the local capabilities to see which one can support this parsing function. Then this command is sent to the AI ​​PC to help with parsing, and the tasks and parameters are obtained, such as casting the screen from the phone to the PC. Based on this, automatic screen casting can be achieved in this embodiment.

[0145] like Fig.12As shown in the figure, it is an example diagram of converting speech into an executable task by calling the voice-to-task capability across devices. The mobile phone receives input data such as text input or voice input related to the task. If the mobile phone supports the task parsing capability, the mobile phone performs task parsing on the input data; if the PC supports the task parsing capability, the mobile phone sends the input data to the PC via SC, and the PC performs task parsing. In addition, the mobile phone can also prompt to perform task parsing on the PC; if the tablet supports the task parsing capability, the mobile phone sends the input data to the tablet via SC, and the tablet performs task parsing. In addition, the mobile phone can also prompt to perform task parsing on the tablet; if the cloud server supports the task parsing capability, the mobile phone sends the input data to the cloud server via SC, and the cloud server performs task parsing. In addition, the mobile phone can also prompt to perform task parsing on the cloud server.

[0146] like Fig.13 As shown in the figure, it is an interactive sequence diagram for realizing capability collection and capability intercommunication through SC in the device interconnection scenario. In the diagram, AI applications with AI capabilities such as Wenshengtu capabilities and other functional modules are deployed in the notebook, and applications and other functional modules that need to use Wenshengtu capabilities are deployed in the mobile phone. Both the notebook and the mobile phone are deployed with SC. The specific process is as follows:

[0147] First, each application deployed on the notebook declares its own capabilities, such as the AI ​​image generation capability declared by the AI ​​application, and the super interconnection capability declared by the SC; each application on the notebook can query the capabilities declared by other applications and connect to the call service of the queried capabilities. For example, the SC on the notebook can query the text image capability declared by the AI ​​application and connect to the call service of the text image capability. As a result, each functional module on the notebook can have a local capability set of the notebook.

[0148] Each application deployed on the mobile phone declares its own capabilities, such as the AI ​​image generation capability declared by the AI ​​application, and the super interconnection capability declared by the SC; each application on the mobile phone can query the capabilities declared by other applications and connect to the call service of the queried capabilities. For example, other applications on the mobile phone can query the super interconnection capability declared by the SC and connect to the call service of the super interconnection capability. Therefore, each functional module on the mobile phone can have a local capability set of the mobile phone.

[0149] Secondly, the SC of the notebook can manage the capabilities it has queried through the interconnection capability manager MGR (SmartConnect CapabilityManager) and save them to the capability list; the SC of the mobile phone can manage the capabilities it has queried through the MGR and save them to the capability list. Moreover, the notebook and the mobile phone can establish a private connection channel through their respective SCs, and communicate the capabilities in the capability lists of their respective MGRs to each other through the connection channel. Therefore, the capability list of the MGR of the SC on the notebook contains the capabilities of each application on the mobile phone, that is, the remote capability set of the notebook, and the capability list of the MGR of the SC on the mobile phone contains the capabilities of each application on the notebook, such as the AI ​​image generation capability, that is, the remote capability set of the mobile phone.

[0150] In addition, the SCs of the laptop and the phone can receive the status update information of the capabilities sent by the other party, and update the status information of the capabilities of the other party to other local applications in real time. For example, if the AI ​​image generation capability of the laptop changes, then through the connection channel, the SC on the phone can obtain the new AI image generation capability of the laptop and update it to the application that previously connected to the calling service of the AI ​​image generation capability.

[0151] like Fig.14 As shown in, it is an interactive timing diagram for implementing capability calls through SC in the device interconnection scenario. The application on the mobile phone receives input data representing the need for AI image generation, such as "generate a dog image", and then first queries whether there is AI image generation capability in the local capability set. If so, the input data is transmitted to the AI ​​image generation capability through the call service connected to the mobile phone application. After the AI ​​image generation capability generates the image, the image is returned to the mobile phone application; if there is no AI image generation capability in the local capability set, then the execution device with AI image generation capability, such as a notebook, is queried in the remote capability set. Based on this, the call instruction pointing to the AI ​​image generation capability and the input content "generate a dog image" are sent The SC on the mobile phone sends the calling instructions and input content of the AI ​​image generation capability to the SC on the notebook through the connection channel. The SC on the notebook calls the AI ​​image generation capability in the AI ​​application on the notebook by calling the service. For example, the calling instructions and input content are given to the AI ​​application by calling the service. The AI ​​application generates an image. The AI ​​application returns the generated image, that is, the output result, to the SC on the notebook through the callback function. The SC on the notebook sends the output result to the SC on the mobile phone through the connection channel. The SC on the mobile phone returns the output result to the application on the mobile phone, thereby realizing the AI ​​image generation capability on the notebook on the application on the mobile phone.

[0152] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0153] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0154] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0155] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A control method, comprising: Get input data; The input data is used to represent the user's intention; In response to the input data, converting into target requirements including target actions, target capabilities, and target execution devices; In response to the conversion, determining from the capability set a first capability that satisfies the target capability and a first execution device of the first capability as the target execution device; The capability set includes: a local capability set of the terminal device and a remote capability set of at least one electronic device that meets the connection condition with the terminal device; In response to the target requirement, based on the first execution device, responding to the target action with the first capability to meet the user intention; If the first execution device is a target device in the electronic device, a calling instruction for calling the first capability and input content provided to the first capability are sent through a connection channel with the target device.

2. The method according to claim 1, wherein determining from the capability set a first capability that satisfies the target capability and a first execution device of the first capability as the target execution device comprises: Based on the local capability set giving priority to the remote capability set, a first capability satisfying the target capability and a first execution device of the first capability are determined from the capability set as the target execution device.

3. The method according to claim 1 or 2, wherein the conversion into target requirements including target actions, target capabilities, and target execution devices comprises: An expression of a first action and an expression of a first capability are obtained.

4. The method according to claim 1 or 2, wherein the conversion into target requirements including target actions, target capabilities, and target execution devices comprises: Based on a natural language understanding model, the input data is processed to obtain an expression of the first action and an expression of the first capability.

5. The method according to claim 4, wherein the conversion into target requirements including target actions, target capabilities and target execution devices further comprises: Obtain an associated device for the target execution device to execute the target action.

6. The method according to claim 1, wherein the capabilities in the local capability set are obtained by: Obtaining at least one capability of the terminal device from a system registry of the terminal device; registering the capability of the target functional module in the terminal device to the system registry; or, Any target functional module in the terminal device is used to query any other target functional module for its declared capability.

7. The method according to claim 1, wherein the capabilities in the remote capability set are obtained by: Ability to receive transmission from the electronic device through the connection channel; in, The transmission capability of the electronic device is obtained by querying the capability of any target functional module in the electronic device, or the transmission capability of the electronic device is obtained from a system registration table of the electronic device.

8. The method according to claim 7, further comprising: Obtaining, through the connection channel, status update information of any capability in the remote capability set; According to the status update information, the status information of the corresponding capability in the remote capability set is updated.

9. The method according to claim 1, based on the first execution device responding to the target action with the first capability, comprising: If the first execution device is the target device, sending the call instruction and the input content to the target device through the connection channel; The target device sends the input content to the first capability through the connected first calling service for calling the first capability; and obtains the output result obtained by the first capability processing the input content through the first calling service; The output result returned by the target device is received through the connection channel.

10. A terminal device, comprising: A memory, used to store a computer program and data generated by the execution of the computer program; A processor, configured to execute the computer program to implement: Obtaining input data; the input data is used to represent user intention; In response to the input data, converting into target requirements including target actions, target capabilities, and target execution devices; In response to the conversion, determining from the capability set a first capability that satisfies the target capability and a first execution device of the first capability as the target execution device; The capability set includes: a local capability set of the terminal device and respective remote capability sets of at least one electronic device that meets the connection condition with the terminal device; In response to the target requirement, based on the first execution device, responding to the target action with the first capability to meet the user intention; If the first execution device is a target device in the electronic device, a calling instruction for calling the first capability and input content provided to the first capability are sent through a connection channel with the target device.

Images