Capability sharing method and device, electronic equipment and computer readable storage medium

Abstract

The invention discloses a capability sharing method and device, electronic equipment and a computer readable storage medium, and the method comprises the steps: determining a third-party artificial intelligence capability matched with a control type of a business control in response to a triggering operation for the business control in a current application; displaying a capability calling interface of the third-party artificial intelligence capability in a control interface of the service control; in response to a trigger operation on the capability calling interface, calling a third-party artificial intelligence capability to perform artificial intelligence processing on service data corresponding to the service control, and obtaining a processing result; and displaying the processing result in the current application. According to the method and the device, the service control without the artificial intelligence capability can also provide the artificial intelligence capability, so that the diversity of functions of the service control is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and particularly relates to a method, device, electronic device, and computer-readable storage medium for sharing capabilities. Background Art

[0002] Currently, applications on electronic devices such as mobile phones and tablet computers build graphical user interfaces through business controls for human-computer interaction with users. For example, the TextView control responsible for drawing text, the EditView control responsible for text editing, the ImageView control responsible for images, etc.

[0003] Generally, the current control functions rely on the self-development of application developers and can only provide limited functions. For example, the ImageView control usually provides functions such as image saving and image sharing, and the EditView control usually provides functions such as text cutting and text copying. In related technologies, there is a problem of single-function of controls. Summary of the Invention

[0004] Embodiments of this application provide a method, device, electronic device, and computer-readable storage medium for sharing capabilities, which can improve the diversity of business control functions.

[0005] In a first aspect, the method for sharing capabilities provided by this application includes:

[0006] Responding to a trigger operation on a business control in the current application, determining a third-party artificial intelligence capability that matches the control type of the business control;

[0007] Displaying an ability call interface of the third-party artificial intelligence capability in the control interface of the business control;

[0008] Responding to a trigger operation on the ability call interface, calling the third-party artificial intelligence capability to perform artificial intelligence processing on the business data corresponding to the business control to obtain a processing result;

[0009] Displaying the processing result in the current application.

[0010] In a second aspect, the ability sharing device provided by this application includes:

[0011] An ability determination module, configured to respond to a trigger operation on a business control in the current application and determine a third-party artificial intelligence capability that matches the control type of the business control;

[0012] An ability display module, configured to display an ability call interface of the third-party artificial intelligence capability in the control interface of the business control;

[0013] An ability invocation module, configured to, in response to a triggering operation on an ability invocation interface, invoke a third-party artificial intelligence ability to perform artificial intelligence processing on service data corresponding to a service control, so as to obtain a processing result;

[0014] A result display module, configured to display the processing result in the current application.

[0015] Thirdly, an electronic device provided in this application includes a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program in the memory to implement the steps in the ability sharing method provided in this application.

[0016] Fourthly, a computer-readable storage medium provided in this application stores a computer program, and this computer program is suitable for being loaded by a processor to implement the steps in the ability sharing method provided in this application.

[0017] In this application, in response to a triggering operation on a service control in the current application, a third-party artificial intelligence ability matching the control type of the service control is determined; an ability invocation interface of the third-party artificial intelligence ability is displayed in the control interface of the service control; in response to a triggering operation on the ability invocation interface, the third-party artificial intelligence ability is invoked to perform artificial intelligence processing on service data corresponding to the service control, so as to obtain a processing result; and the processing result is displayed in the current application. In this way, sharing of the third-party artificial intelligence ability can be achieved at the control level, enabling a service control that does not have artificial intelligence ability itself to also provide artificial intelligence ability, thereby enhancing the diversity of its functions. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in this embodiment, the following will briefly introduce the drawings required for the description of the embodiment. Obviously, the drawings in the following description are only some embodiments of this application. For those skilled in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0019] Figure 1 is a schematic diagram of a scenario of an ability sharing system provided by an embodiment of this application;

[0020] Figure 2 is a schematic flowchart of an ability sharing method provided by an embodiment of this application;

[0021] Figure 3 is an example diagram of configuring artificial intelligence ability for different service controls in an embodiment of this application;

[0022] Figure 4 is an example diagram of displaying a service control in an embodiment of this application;

[0023] Figure 5It is an example diagram for performing artificial intelligence processing in an embodiment of the present application;

[0024] Figure 6 It is an example diagram for presenting an abstract result in an embodiment of the present application;

[0025] Figure 7 It is a schematic diagram of the architectures of the AIPWMS component, AIPWM component, and AIPWS component provided in an embodiment of the present application;

[0026] Figure 8 It is an example diagram for triggering task decomposition in an embodiment of the present application;

[0027] Figure 9 It is an example diagram for presenting the execution progress of a cross-application task in an embodiment of the present application;

[0028] Figure 10 It is a schematic diagram of the structure of a capability sharing device provided in an embodiment of the present application;

[0029] Figure 11 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Detailed implementation manners

[0030] It should be noted that the principle of the present application is illustrated by being implemented in a suitable computing environment. The following description is based on the specific embodiments of the present application illustrated, and it should not be regarded as limiting other specific embodiments of the present application not detailed herein.

[0031] In the following description of the present application, reference is made to "some embodiments", which describe a subset of all possible embodiments. However, it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0032] In the following description of the present application, the terms "first / second / third" are only used to distinguish similar objects and do not represent a specific order for the objects. It can be understood that "first / second / third" can be interchanged in a specific order or sequence when allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0034] In order to improve the diversity of control functions, an embodiment of the present application provides a capability sharing method, a capability sharing device, an electronic device, and a computer-readable storage medium. Among them, the capability sharing method can be executed by the capability sharing device or by an electronic device integrated with the capability sharing device.

[0035] Next, the technical solutions in this embodiment will be clearly and completely described with reference to the accompanying drawings in this embodiment. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present application.

[0036] Please refer to Figure 1 , the present application also provides a capability sharing system, as Figure 1 shown. The capability sharing system includes an electronic device 100, and the capability sharing device provided by the present application is integrated in the electronic device 100. For example, the electronic device 100 can determine a third-party artificial intelligence capability that matches the control type of the service control in response to a trigger operation on the service control in the current application, display a capability call interface of the third-party artificial intelligence capability in the control interface of the service control, and in response to a trigger operation on the capability call interface, call the third-party artificial intelligence capability to perform artificial intelligence processing on the service data corresponding to the service control to obtain a processing result, and display the processing result in the current application.

[0037] Among them, the electronic device 100 can be any device configured with a processor and having processing capabilities, such as a mobile electronic device with a processor such as a smart phone, a tablet computer, a handheld computer, a notebook computer, a virtual reality device, an augmented reality device, or a mixed reality device, or a fixed electronic device with a processor such as a desktop computer, a television, a server, or an industrial device.

[0038] In addition, as Figure 1 shown, the capability sharing system may further include a data warehouse 200 for storing data during the capability sharing process. For example, the electronic device 100 stores the service data and the result data obtained after performing artificial intelligence processing on the service data in the data warehouse 200.

[0039] It should be noted that Figure 1 the scenario schematic diagram of the capability sharing system shown is only an example. The capability sharing system and scenario described in the embodiments of the present application are for more clearly explaining the technical solutions in the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art know that with the evolution of the capability sharing system and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are equally applicable to similar technical problems.

[0040] The following will be described in detail respectively. It should be noted that the serial numbers of the following embodiments do not limit the preferred order of the embodiments.

[0041] Please refer to Figure 2 , Figure 2 which is a schematic flowchart of the capability sharing method provided by the embodiments of the present application. As Figure 2 shown, the process of the capability sharing method provided by the present application is as follows:

[0042] In 110, in response to a trigger operation on a service control in the current application, a third-party artificial intelligence capability matching the control type of the service control is determined.

[0043] The current application refers to the application currently running in the foreground on the electronic device, which can be any application on the electronic device. For example, if a browser application is currently running in the foreground on the electronic device, then the browser application is the current application.

[0044] Artificial intelligence capability refers to the ability to perform artificial intelligence processing through a corresponding artificial intelligence model. The artificial intelligence model can be an artificial intelligence model with any function. For example, it can be a text summarization model for summarizing text, which provides text summarization ability; it can be a watermark processing model for removing watermarks from images, which provides watermark removal ability; it can also be a text conversion model for converting audio into text, which provides text conversion ability. As the name implies, the third-party artificial intelligence capability in the present application refers to the artificial intelligence capability not provided by the artificial intelligence model deployed by the current application, which can be the artificial intelligence capability provided by the artificial intelligence model deployed by other applications outside the current application, or the artificial intelligence capability provided by the artificial intelligence model locally deployed in the electronic device system.

[0045] Service controls (UI controls), also known as user interface controls, are used to build a graphical user interface and provide the ability to interact with the operation object. Common control types are shown in Table 1 below:

[0046]

[0047]

[0048] Table 1, Function Table of Control Types

[0049] It should be noted that in the embodiments of the present application, matching artificial intelligence capabilities are pre-configured for different control types, and a corresponding relationship between the control type and the artificial intelligence capability is established accordingly.

[0050] For example, please refer to Figure 3, in addition to basic functions such as copying, selecting all, and translating, text controls such as TextView controls can be configured with artificial intelligence capabilities such as associated search, adding sticky notes, explanation, and text summarization; for example, EditView controls can be configured with artificial intelligence capabilities such as associated input and automatic filling in addition to the basic functions of the input method; for example, Button controls can be added with artificial intelligence capabilities such as encapsulating as shortcut commands;

[0051] Image controls such as ImageView controls can be configured with artificial intelligence capabilities such as removing watermarks, optical character recognition, blurring, and extracting text in addition to basic functions such as saving to local, sharing, and favoriting;

[0052] List controls (not shown in the figure) can be configured with artificial intelligence capabilities such as intelligent recommendation, intelligent sorting, and intelligent grouping.

[0053] Progress controls (not shown in the figure) can be configured with artificial intelligence capabilities such as intelligent prediction, intelligent prompt, and intelligent feedback.

[0054] In the embodiments of the present application, if there is a trigger operation on a service control in the current application, then according to the pre-established correspondence between the control type and the artificial intelligence capabilities, determine the third-party artificial intelligence capabilities that match the control type of the triggered service control. For example, assuming that the triggered service control is a TextView control, then according to the pre-established correspondence between the control type and the artificial intelligence capabilities, it can be determined that the third-party artificial intelligence capabilities corresponding to its control type include voice output, associated search, adding sticky notes, explanation, text correction, text summarization, etc.

[0055] Among them, the embodiments of the present application do not specifically limit the trigger operation on the service control, which specifically depends on the actual configuration of the current application. For example, the trigger operation on the TextView control configured by a certain browser application is the selection operation on the text displayed on the interface of the browser application; for another example, the trigger operation on the ImageView control configured by a certain instant messaging application is the selection operation on the image displayed by the instant messaging application.

[0056] In 120, display the ability call interface of the third-party artificial intelligence capabilities in the control interface of the service control.

[0057] As described above, after determining the third-party artificial intelligence capabilities that match the control type of the triggered service control, display the control interface of the triggered service control. In addition to including the function interfaces of the basic functions provided by the service control itself, the control interface also includes the ability call interface for calling the third-party artificial intelligence capabilities. Here, there is no specific limitation on the display form of the ability call interface of the third-party artificial intelligence capabilities, which can be a pure text display form, a pure graphic display form, or a text combined with graphic display form, etc.

[0058] For example, please refer to Figure 4 , in response to the selection operation of the text in the browser application by the operation object, the control interface of the TextView control is displayed, as Figure 4 shown. The control interface includes function interfaces for basic functions such as cut, copy, select all, and share provided by the TextView control itself, and also includes ability call interfaces for third-party artificial intelligence capabilities such as voice output and text summarization.

[0059] In 130, in response to the trigger operation on the ability call interface, the third-party artificial intelligence ability is called to perform artificial intelligence processing on the service data corresponding to the service control, and a processing result is obtained.

[0060] It can be understood that if the operation object of the current application has a call requirement for the third-party artificial intelligence ability, then a trigger operation on the ability call interface can be input. On the other hand, if there is a trigger operation on the ability call interface displayed in the control interface, then in response to the trigger operation on the ability call interface, the corresponding third-party artificial intelligence ability of the triggered ability call interface is called to perform artificial intelligence processing on the service data corresponding to the triggered service control, and a processing result is obtained.

[0061] For example, please refer to Figure 5 , assuming that the operation object has a requirement for text summarization of the selected text, then the ability call interface "Text Summarization" can be triggered. Correspondingly, in response to the trigger operation on the ability call interface "Text Summarization", the text summarization model providing the text summarization ability is called to perform text summarization processing on the text selected by the operation object (i.e., the service data corresponding to the service control), and a summary result is obtained.

[0062] In 140, the processing result is displayed in the current application.

[0063] As described above, after performing artificial intelligence processing on the service data corresponding to the service control by calling the third-party artificial intelligence ability to obtain a processing result, the processing result is further displayed in the current application. In this way, the sharing of the third-party artificial intelligence ability can be realized at the control level, enabling the control that does not have the artificial intelligence ability itself to also obtain the artificial intelligence ability, thereby enhancing the diversity of its functions.

[0064] For example, please refer to Figure 6 , assuming that the service data corresponding to the service control is text, and the third-party artificial intelligence ability "Text Summarization" is called to perform text summarization processing to obtain a summary result, then the summary result is correspondingly displayed in the current application.

[0065] The following takes an electronic device using the Android system as an example to further illustrate the ability sharing method provided by this application.

[0066] Add AIPWM (AI-Powered Widgets Manager), AIPWMS (AI-Powered Widgets Manager Service), and AIPWS (AI-Powered Widgets Service) components to the system. Please refer to Figure 7 , where the AIPWM component is used for business widgets that share artificial intelligence capabilities, the AIPWMS component is used to interact with the system and receive capability call requests from the AIPWM component, and the AIPWS component is used to respond to capability call requests and perform corresponding artificial intelligence processing.

[0067] Among them, the code example of the AIPWM component is as follows:

[0068] public class AIPWM{

[0069] private List;

[0070] public void addwidget(Textview widget){

[0071] mwidgets.add(widget);

[0072] }

[0073] public void removewidget(TextView widget){

[0074] mwidgets.remove(widget);

[0075] }

[0076] public void updatewidgets(){

[0077] for(TextView widget:mwidgets);

[0078] }

[0079] }

[0080] The code example of the AIPWMS component is as follows:

[0081] public class AIPWMS extends Service{

[0082] private AIPWM mAIPWM;

[0083] private AIService mAIService;

[0084] @Override public int onStartCommand(Intent intent, int flags, int startId) { mAIPWM = new AIPWM();

[0085] mAIService = new AIService();

[0086] return super.onStartCommand(intent, flags, startId);

[0087] }

[0088] @Override public IBinder onBind(Intent intent) {

[0089] return null;

[0090] }

[0091] }

[0092] The code example of AIPWS is as follows:

[0093] public class AIservice {

[0094] public void processRequest(TextView widget) {

[0095] / / Perform corresponding artificial intelligence processing, such as text summarization, speech recognition, etc.

[0096] / / Return the result to the AIPWS component

[0097] }

[0098] The following further describes the process of capability sharing:

[0099] If there is a trigger operation for a business control in the current application, the AIPWM component sends a sharing request to the AIPWMS component. The AIPWM component determines the third-party artificial intelligence capabilities that match the control type of the business control and returns the information of the matched third-party artificial intelligence capabilities to the AIPWM component. According to the information of the third-party artificial intelligence capabilities received by the AIPWM component, a capability invocation interface of the third-party artificial intelligence capabilities is displayed in the control interface of the business control.

[0100] If there is a trigger operation for the capability invocation interface, the AIPWM component sends an invocation request to the AIPWMS component, and the AIPWMS component forwards the invocation request to the AIPWS component of the target application that provides the third-party artificial intelligence capabilities triggered by the capability invocation interface.

[0101] In response to the invocation request forwarded by the AIPWMS component, the AIPWS component of the target application obtains the business data corresponding to the business control from the current application according to the invocation request, performs artificial intelligence processing on the business data through the corresponding artificial intelligence model, obtains the processing result, and returns the processing result to the AIPWM component of the current application.

[0102] The AIPWM component displays the processing result on the corresponding container of the current application. For example, the processing result can be displayed on the container of the business control, or a new container can be created to display the processing result.

[0103] Optionally, in an embodiment, to enhance the compatibility and controllability of capability sharing, permission control is also performed on capability sharing. Among them, in response to a trigger operation for a business control in the current application, determining the third-party artificial intelligence capabilities that match the control type of the business control includes:

[0104] In response to a trigger operation for a business control in the current application, identify whether a sharing identifier is added to the control attributes of the business control;

[0105] In response to the sharing identifier being added to the control attributes of the business control, determine the third-party artificial intelligence capabilities that match the control type of the business control.

[0106] In the embodiment of the present application, the sharing identifier is used to represent whether the business control has the permission to share third-party artificial intelligence capabilities. The form of this sharing identifier is not specifically limited here and can be configured by those skilled in the art according to actual needs.

[0107] Exemplarily, in an embodiment of the present application, a flag bit "android:ai_powered" is added to the control attributes (or UI attributes) of a service control. The value range of the flag bit "android:ai_powered" can be true and false. When the value is true, it indicates that the corresponding service control has the permission to share third-party artificial intelligence capabilities. When the value is false, it indicates that the corresponding service control does not have the permission to share third-party artificial intelligence capabilities. This flag bit can be added to the control attributes of service controls at different levels:

[0108] For example, the above flag bit can be added to the control attributes of a single control level, such as a TextView control:

[0109]

[0110]

[0111] Among them, when the value of the flag bit is true, it indicates that the TextView control has the permission to share third-party artificial intelligence capabilities.

[0112] For another example, the above flag bit can be added to the control attributes of a regional control, such as a LinearLayout control:

[0113]

[0114] Among them, when the value of the flag bit is false, it indicates that the LinearLayout control does not have the permission to share third-party artificial intelligence capabilities.

[0115] For another example, the above flag bit can be added to the control attributes of a page-level control, such as an Activity control:

[0116]

[0117] Among them, when the value of the flag bit is true, it indicates that the TextView control has the permission to share third-party artificial intelligence capabilities.

[0118] In this way, by the above method of adding a flag bit to the control attributes, the application can specify whether controls at different levels need to share third-party artificial intelligence capabilities without adding or modifying any code, thus achieving the compatibility and controllability of capability sharing.

[0119] Correspondingly, in the embodiment of the present application, in response to a triggering operation on a service control in the current application, first identify whether the value of the android:ai_powered flag in the control attributes of the triggered service control is true. If so, it is determined that a sharing identifier is added to the control attributes of the service control (that is, the service control has the permission to share the third-party artificial intelligence capability). In response to the sharing identifier being added to the control attributes of the service control, determine the third-party artificial intelligence capability that matches the control type of the triggered service control.

[0120] Optionally, in one embodiment, to increase the flexibility of capability sharing, determining the third-party artificial intelligence capability that matches the control type of the service control includes:

[0121] Identify the sharing level of the sharing identifier added to the control attributes of the service control;

[0122] Determine the third-party artificial intelligence capability that matches the control type of the service control and meets the sharing level.

[0123] In the embodiment of the present application, the shared third-party artificial intelligence capabilities are classified to increase the flexibility of capability sharing. Among them, at least two sharing levels are predefined in the embodiment of the present application. For the same control type, the shareable third-party artificial intelligence capabilities corresponding to service controls with different sharing levels are different. For example, two sharing levels are defined as sharing level one and sharing level two. The shareable third-party artificial intelligence capabilities corresponding to sharing level one include capabilities A, B, and C, and the shareable third-party artificial intelligence capabilities corresponding to sharing level two include capabilities A, B, C, D, and E. That is to say, the higher sharing level includes all the shareable third-party artificial intelligence capabilities corresponding to the lower sharing level.

[0124] In addition, it should be noted that the embodiments of this application still use the android:ai_powered flag. The difference is that in the embodiments of this application, the value range of the android:ai_powered flag is no longer true and false, but 0 to X. Among them, when the value of the android:ai_powered flag is 0, it indicates that the corresponding service control does not have the permission to share the third-party artificial intelligence capability. X is a positive integer greater than 0, and its value depends on the number of defined sharing levels (for example, if the sharing level is defined as 5 levels, then X takes the value of 5; if the sharing level is defined as 3 levels, then X takes the value of 3). Correspondingly, when the value of the android:ai_powered flag is a positive integer less than or equal to X, it indicates that the corresponding service control has the permission to share the third-party artificial intelligence capability, and the sharing level is the positive integer value of the android:ai_powered flag.

[0125] Correspondingly, in the embodiments of this application, when determining the third-party artificial intelligence capabilities that match the control type of the triggered service control, first identify the sharing level of the sharing identifier added to the control attributes of the service control, and then determine the third-party artificial intelligence capabilities that meet its sharing level among the third-party artificial intelligence capabilities that match the control type of the service control for invocation.

[0126] Optionally, in one embodiment, to improve the efficiency of capability sharing, a capability invocation interface for third-party artificial intelligence capabilities is displayed in the control interface of the service control, including:

[0127] Evaluate the matching degree of the third-party artificial intelligence capabilities in the current business scenario;

[0128] Screen out the target third-party artificial intelligence capabilities whose matching degree reaches the degree threshold from the third-party artificial intelligence capabilities;

[0129] Display the capability invocation interface of the target third-party artificial intelligence capabilities in the control interface of the service control.

[0130] It can be understood that in practical applications, for a service control of a certain control type, there may be a large number of third-party artificial intelligence capabilities available for sharing. If the capability invocation interfaces of all the available third-party artificial intelligence capabilities are displayed in the control interface of the service control, it will take the operator a certain amount of time to find the capability invocation interface of the third-party artificial intelligence capability that needs to be invoked, and it may also affect the implementation of the original basic functions of the service control. Therefore, in the embodiments of this application, not all the capability invocation interfaces of the available third-party artificial intelligence capabilities are displayed, but only a part of the capability invocation interfaces of the third-party artificial intelligence capabilities are displayed.

[0131] Among them, in the embodiment of the present application, a preset matching degree evaluation strategy is first installed to evaluate the matching degree of the third-party artificial intelligence capabilities matched by the control type of the triggered service control in the current business scenario; then, from the third-party artificial intelligence capabilities matched by the control type of the triggered service control, the third-party artificial intelligence capabilities with a matching degree reaching the degree threshold (the degree threshold can be determined by those skilled in the art according to actual needs. For example, it can be configured as a fixed value, or its value can be dynamically determined according to the results of this evaluation, such as taking the median of all matching degrees obtained in this evaluation) are screened out and recorded as the target third-party artificial intelligence capabilities. There is no specific limitation on the configuration of the matching degree evaluation strategy here, and it can be configured by those skilled in the art according to actual needs. For example, the matching degree evaluation strategy can be configured as: according to the historical call information of the third-party artificial intelligence capabilities by the operation object and the scenario information of the current business scenario, the matching degree is evaluated through a pre-trained matching degree evaluation model.

[0132] As described above, after screening out the target third-party artificial intelligence capabilities with a matching degree reaching the degree threshold, the ability call interface of the target third-party artificial intelligence capabilities is displayed in the control interface of the service control. There is no specific limitation on the display method of the ability call interface of the target third-party artificial intelligence capabilities here, and it can be configured by those skilled in the art according to actual needs.

[0133] Exemplarily, the ability call interfaces of the target third-party artificial intelligence capabilities can be displayed in the control interface of the service control in order of the matching degree of the target third-party artificial intelligence capabilities from high to low. For example, the ability call interface of the target third-party artificial intelligence capabilities with a high matching degree can be displayed in a position that is more convenient for the operation object to trigger.

[0134] Optionally, in one embodiment, the ability sharing method adopted by the present application further includes:

[0135] Obtain the object intention of the operation object of the current application;

[0136] Determine the service control operation sequence for completing the object intention according to the content description information of the service control in the current application;

[0137] Operate the corresponding target service controls in the current application in sequence according to the service control operation sequence.

[0138] In the embodiments of the present application, during the process of an operating object operating on the current application, the object intention of the operating object is also obtained. Exemplarily, a query statement (query) input by the operating object can be obtained, and a prompt statement (prompt) for instructing a large language model to perform intention recognition on the query statement is generated. Then, the query statement and the prompt statement are input into the large language model, and intention recognition is performed through the large language model to correspondingly obtain the object purpose of the operating object. For example, assuming that the operating object inputs the query statement "Recharge my phone bill", the object intention of the operating object can be recognized as "Recharge mobile phone bill" through the large language model. In addition, the object intention of the operating object can also be obtained from the system.

[0139] Among them, the large language model refers to a deep learning model trained with a large amount of text data, which can generate natural language text or understand the meaning of language text. The large language model can process various natural language tasks, such as text classification, question answering, dialogue, etc.

[0140] After obtaining the object intention of the operating object, further according to the content description information (contentDescription) of the business control in the current application, task planning is carried out, and the planned tasks are disassembled into operations to be executed by the target business controls in the current application. According to the operation sequence of these target business controls, a business control operation sequence is obtained, denoted as the business control operation sequence. For example, the above task planning and task disassembling can be realized through the large language model. Among them, the content description information is default used to provide voice and text descriptions for the business controls to help assistive function devices (such as screen readers) better assist visually impaired users in obtaining and using application functions.

[0141] The following takes the Android system as an example to illustrate the acquisition method of the content description information:

[0142] (1) By means of redirection, guide the operating object to enable the accessibility service:

[0143] Intent intent = new Intent(Settings.ACTION_ACCESSIBILITY_SETTINGS);

[0144] (2) Create an accessibility service class, inheriting from AccessibilityService:

[0145] public class MyAccessibilityService extends AccessibilityService{

[0146] @Override

[0147] public void onAccessibilityEvent(AccessibilityEvent event){

[0148] }

[0149] @Override

[0150] public void onInterrupt(){

[0151] }

[0152] (3) Override the onAccessibilityEvent method to handle events. Use event.getSource() to obtain the source node AccessibilityNodeInfo of the event. Starting from this node, recursively traverse the entire view hierarchy to obtain a list of page elements:

[0153] @Override

[0154] public void onAccessibilityEvent(AccessibilityEvent event){

[0155] AccessibilityNodeInfo rootNode = event.getSource();

[0156] List<AccessibilityNodeInfo> elementsList = new ArrayList<>(); traverseViewHierarchy(rootNode, elementsList);

[0157] }

[0158] private void traverseViewHierarchy(AccessibilityNodeInfo rootNode, List<AccessibilityNodeInfo> elementsList){

[0159] if (rootNode == null) return;

[0160] elementsList.add(rootNode);

[0161] int childCount = rootNode.getChildCount();

[0162] for (int i = 0; i < childCount; i++) {

[0163] AccessibilityNodeInfo childNode = rootNode.getChild(i);

[0164] traverseViewHierarchy(childNode, elementsList);

[0165] }

[0166] }

[0167] In the above way, all elements (such as id, layout, text, contentDescription, etc.) of the current page of the current application can be obtained. After that, the getContentDescription() method can be used to obtain the content description information of the business control.

[0168] As above, after determining the operation sequence of the business control for completing the object intention, the corresponding target business control in the current application is operated in sequence according to the operation sequence of the business control, so as to complete the object intention of the operation object. For example, taking the Android system as an example, the View.performClick() method can be used to operate the target business control in the current application.

[0169] Optionally, in an embodiment, determining the operation sequence of the business control for completing the object intention according to the content description information of the business control in the current application includes:

[0170] Determining multiple candidate operation sequences of the business control for completing the object intention according to the content description information of the business control in the current application;

[0171] Performing a quality score on the multiple candidate operation sequences of the business control, and screening out the candidate operation sequence of the business control with the highest quality score from the multiple candidate operation sequences as the operation sequence of the business control.

[0172] In the embodiment of the present application, to ensure the completion quality of the object intention, when determining the operation sequence of the business control for completing the object intention, first, according to the content description information of the business control in the current application, multiple candidate operation sequences of the business control for completing the object intention are determined for selection, and then a quality score is performed on the multiple candidate operation sequences, and the candidate operation sequence of the business control with the highest quality score is screened out as the operation sequence of the business control.

[0173] The following takes the object intention of "recharging mobile phone bill" as an example for illustration:

[0174] First, based on the content of the currently applied page and the content description information of the business controls, identify the available recharge channels on the current page through a large language model.

[0175] Then, construct a tree structure through a large language model. Each recharge channel is used as a node, and there are different functional modules under each node. Under each module, there are recharge amounts and discount strengths as child nodes. For example, under recharge channel A, there is "My - Services - Phone Bill Recharge". Under phone bill recharge, there are child nodes such as 10 yuan, 20 yuan, 30 yuan, etc. Each child node corresponds to a discounted amount or discount.

[0176] Then, traverse this tree structure in a depth - first search manner through a large language model, calculate whether each leaf node can reach the target, the actual cost and the amount saved, so as to obtain the node paths that can reach the target. Among them, the nodes on the node path correspond to the business controls in the current application. Correspondingly, there are multiple node paths, that is, there are corresponding multiple candidate business control operation sequences.

[0177] Finally, use reinforcement learning (such as the Monte Carlo tree search algorithm) for multi - path screening: use the large language model to score the nodes in the node path, and finally select the node path with the highest score as the final decision path, and use the candidate business control operation sequence corresponding to this decision path as the business control operation sequence.

[0178] For example, assume that the operator wants to recharge 30 yuan of phone bill. Recharge channel A has a 5% discount, and recharge channel B has a 10% discount. Then the large language model will select the node path corresponding to recharge channel B as the final decision path.

[0179] Optionally, in one embodiment, before obtaining the object intention of the operator of the current application, it further includes:

[0180] Determine the business controls to be supplemented with incomplete content description information;

[0181] Obtain the content description information of the business controls to be supplemented, and configure the content description information for the business controls to be supplemented.

[0182] It should be noted that the accessibility service depends on the application providing the "android:contentDescription" attribute for the business control, that is, the content description information. However, the application often fails to be adapted, resulting in that accessibility services such as screen readers cannot provide services for the operator. In addition, due to the lack of content description information, the accessibility service cannot understand the function provided by the business control. Therefore, there is a need to supplement the content description information of the business control.

[0183] Correspondingly, in the embodiments of the present application, business controls with incomplete content description information are also identified. For example, through the page layout XML file, business controls with incomplete content description information can be determined, denoted as business controls to be supplemented. Then, the content description information of the business controls to be supplemented is obtained, and the content description information is configured for the business controls to be supplemented.

[0184] Optionally, in one embodiment, an optional method for obtaining content description information is provided. Among them, obtaining the content description information of the business controls to be supplemented includes:

[0185] Obtain the identity information of the business controls to be supplemented from the layout attributes of the business controls to be supplemented;

[0186] According to the identity information, use the large language model to describe the content of the business controls to be supplemented, and obtain the content description information of the business controls to be supplemented.

[0187] As an optional implementation manner, in the embodiments of the present application, the large language model is used to complete the content description information. Among them, for a business control to be supplemented, the identity information of the business control to be supplemented can be obtained from the layout attributes of the business control to be supplemented; then, according to the identity information, the large language model is used to describe the content of the business control to be supplemented, and the content description information of the business control to be supplemented is correspondingly obtained.

[0188] Exemplarily, first, the identity information of the business controls to be supplemented can be obtained from the layout attributes of the business controls to be supplemented corresponding to the page layout XML file, which can be any information that can represent the identity of the business controls to be supplemented, such as its ID information, picture path information, etc. For example, the ID information of a business control to be supplemented is obtained as "ic_add_user".

[0189] Then, according to the identity information of the business controls to be supplemented, a prompt statement is generated to instruct the large language model to describe the content of the business controls to be supplemented. For example, the generated prompt statement is "Please provide a concise content description for a button with the following action:{ic_add_user}", and the generated prompt statement is input into the large language model to obtain the content description information "Add user button" output by the large language model for the business controls to be supplemented.

[0190] Finally, configure the obtained content description information to the position of the business control to be supplemented in the page layout XML file. In this way, the content for supplementing the business control with incomplete content description information can be automatically generated, which can not only ensure the completion of the object intention in this application, but also improve the accessibility service quality of the device.

[0191] Optionally, in an embodiment, another optional method for obtaining content description information is provided. Among them, obtaining the content description information of the business control to be supplemented includes:

[0192] Obtain a screenshot of the application interface including the business control to be supplemented;

[0193] Input the application interface screenshot into a multimodal large model, and use the multimodal large model to describe the content of the business control to be supplemented to obtain the content description information of the business control to be supplemented.

[0194] As an optional implementation manner, in the embodiments of the present application, the image recognition ability and optical character recognition ability of the multimodal large model are used to complete the content description information of the business control to be supplemented.

[0195] It can be understood that different business controls play different roles in the user interface, and these business controls cooperate with each other to achieve human-computer interaction with the operation object. Therefore, for a business control, its purpose can be inferred based on its context. For example, assume there is an EditView control next to a Button class control, then this Button class control is probably used to submit the text content input in the EditView control. This is the principle of content description by the multimedia model in the embodiments of the present application. Correspondingly, for a business control to be supplemented, a screenshot of the application interface including the business control to be supplemented can be obtained, and then the application interface screenshot is input into the multimodal large model, and the multimodal large model is used to describe the content of the business control to be supplemented to obtain the content description information of the business control to be supplemented.

[0196] Optionally, in an embodiment, the ability sharing method provided by the present application further includes:

[0197] Determine the content to be recognized from the current application, and determine the cross-application task to be executed through the large language model according to the content to be recognized;

[0198] According to the application description information of different applications, disassemble the cross-application task into subtasks to be executed by different applications through the large language model;

[0199] Through the large language model, call different applications to execute the corresponding subtasks.

[0200] It should be noted that according to the different types of currently applied applications, the types of content to be recognized can also be different, which can be specifically configured by those skilled in the art according to actual needs. For example, for applications with a notification function, the notification content of these applications can be determined as the content to be recognized; for itinerary applications, the itinerary content therein can be determined as the content to be recognized; for note applications, the note content therein can be determined as the content to be recognized; for browser applications, the content in its favorites bar can be determined as the content to be recognized, etc.

[0201] In the embodiments of the present application, first, the content to be recognized is determined from the current application, and based on the content to be recognized, the cross-application tasks to be executed are determined through a large language model, where the cross-application tasks refer to tasks that need to be cooperatively executed by different applications.

[0202] Taking the Android system as an example, the method for obtaining the content to be recognized as notification content is described as follows:

[0203] (1) Obtain the NotificationManager object: Use the Context.getSystemService() method to obtain the NotificationManager object.

[0204] (2) Obtain the notification list: Use the NotificationManager.getActiveNotifications() method to obtain the current active notification list.

[0205] (3) Traverse the notification list: Traverse the notification list and use the Notification.Builder.getContentText() method to obtain the content text of each notification.

[0206] It can be understood that multiple notification contents may be obtained from one application. If all the notification contents are used as the content to be recognized, the execution efficiency will surely be affected. Therefore, important notification contents can be screened out as the content to be recognized. For example, prompt statements can be generated based on the notification contents to instruct the large language model to screen out important notification contents.

[0207] As described above, after determining the cross-application task to be executed, further according to the application description information of different applications, the cross-application task is disassembled into subtasks to be executed by different applications through a large language model. It should be noted that in the embodiments of the present application, the planning ability of the large language model is used to coordinate and arrange the execution actions between different applications (that is, disassemble the cross-application task into subtasks to be executed by different applications). It is worth noting that in this process, the dependency relationships between different applications need to be considered. Therefore, in the embodiments of the present application, according to the input requirements and output results of different applications, by calling the large voice model for analysis and processing, resources and time can be reasonably allocated to ensure that all dependencies can be effectively satisfied. In addition, the large voice model can predict possible new dependency relationships based on the existing information, so as to achieve a more refined scheduling effect.

[0208] Taking the Android system as an example, the method for obtaining application description information is described as follows:

[0209] (1) Obtain the PackageManager object: Use the Context.getPackageManager() method to obtain the PackageManager object.

[0210] (2) Obtain the list of installed applications: Use the PackageManager.getInstalledPackages() method to obtain the list of installed applications.

[0211] (3) Traverse the list of installed applications: Traverse the list of installed applications, and use the PackageManager.getApplicationInfo() method to obtain the application description information of each application, including package name, version number, icon, function description, etc.

[0212] For example, the application description information of a certain payment application obtained is as follows:

[0213] Application name: XX Payment

[0214] Package name: com.XX.android.phone.app

[0215] Version number: 10.2.50

[0216] Icon: The icon of XX Payment

[0217] Function description: XX Payment is a mobile payment application integrating payment, wealth management, and life services, supporting functions such as transfer, payment, and credit card repayment.

[0218] It is understandable that the application description information obtained through the above methods may not be detailed, and it can be expanded through a large language model. For example, prompt statements can be generated based on the application name of XX Payment, and the generated prompt statements can be input into the large language model. The large language model expands the application description information of XX Payment to obtain the expanded application description information as follows:

[0219] XX Payment is a mobile payment software developed by XX Developer, supporting various payment methods, including functions such as scan code payment, transfer, and credit card repayment. Users can use XX Payment for payments in various scenarios such as online shopping, offline consumption, and utility bill payment.

[0220] XX Payment also provides various financial management services, such as funds, fixed-term financial management, etc. Users can use their idle funds to purchase products such as funds and fixed-term financial management for investment.

[0221] In addition, XX Payment also provides various life services, such as movie tickets, train tickets, air tickets, hotel reservations, etc. Users can complete the reservation and payment of various life services through XX Payment.

[0222] In short, XX Payment is a powerful and convenient mobile payment software that provides users with various payment and financial management services, facilitating people's lives.

[0223] Finally, through the multi-modal model, different applications are called to execute corresponding subtasks.

[0224] Among them, for an application in a cross-application task, according to its corresponding subtask, the subtask is executed in combination with the content of its application interface.

[0225] For example, for the Android system, the application interface screenshot can be obtained through the MediaProjection API interface. Based on the disassembled subtask and the application interface screenshot, prompt statements are generated:

[0226] Imagine that you are a robot operating a mobile phone. Like how humans operate the mobile, you can tap an icon with your finger, or type some texts with the keyboard. You are asked to {sub-task}. Below is what you see on the mobile screen, predict your next move. If the action involves tapping an icon, describe the location as detailed as possible.

[0227] {Screenshot of the application interface}

[0228] Input the above prompt statements into the multimodal large model. The multimodal large model will output corresponding action instructions or prediction results according to the instructions of the above prompt statements. These action instructions or prediction results will interact with the corresponding execution environment, call the predefined execution strategy, and finally realize the automatic execution of the sub-task.

[0229] The following takes the cross-application task "Dinner Planning" as an example for illustration:

[0230] (1) Suppose the current application is a certain instant messaging application. The chat content of this instant messaging application can be determined as the content to be recognized. According to the determined content to be recognized, the cross-application task to be executed is determined as "Dinner Planning" through the large language model. In addition, an execution prompt for this cross-application task can be displayed, such as Figure 8 As shown, the execution prompt can be a button control for displaying the cross-application task.

[0231] (2) In response to the operation object triggering the "Dinner Planning" button control, the chat content of the instant messaging application is input into the large language model for planning. The cross-application task "Dinner Planning" is decomposed into sub-tasks such as selecting a gathering restaurant, adding a reminder, and sending a dinner invitation. These sub-tasks will be executed by calling different applications. For example, for the sub-task of selecting a gathering restaurant, a certain food and beverage application will be called to execute.

[0232] (3) Through the multimodal large model, different applications are called to execute the corresponding sub-tasks. Among them, during the execution of the sub-tasks, the multimodal large model will recognize the interface elements of the application and determine the specific execution method of the sub-tasks in combination with the preferences and historical operation behaviors of the operation object.

[0233] Optionally, in one embodiment, to facilitate the connection of the operation object to the execution progress of the cross-application task, the execution progress is also displayed. Specifically, the method for sharing capabilities provided in this application further includes:

[0234] Determine the execution progress of the cross-application task and display the execution progress of the cross-application task.

[0235] There is no specific limitation on the display method of the execution progress of the cross-application here, and those skilled in the art can select according to actual needs.

[0236] For example, please refer to Figure 9 , it is possible to superimpose and display the task execution interface on the application interface of the currently called application, and display the execution progress of the cross-application task by checking the completed subtasks in the task execution interface. As Figure 9 shown, for the cross-application task "Dinner Planning", the subtask of selecting a gathering restaurant has been completed, and the subtask of adding a reminder is currently being executed.

[0237] Optionally, in one embodiment, the operation object is provided with the ability to change the task execution. Specifically, the method for sharing capabilities provided in this application further includes:

[0238] In response to a change instruction for the currently executing subtask, change the execution method of the currently executing subtask according to the change instruction.

[0239] For example, the currently executing subtask is to select a gathering restaurant. The multimodal large model determines the execution method of this subtask as searching for "hot pot" according to the preference "hot pot" of the operation object, and selects a hot pot restaurant where the operation object's historical consumption times reach the preset number. Assuming that the operation object doesn't want to eat hot pot at this time but wants to eat Hakka cuisine, then "hot pot" entered in the search box of the currently called application can be modified to "Hakka cuisine", thereby inputting a change instruction for the currently executing subtask. In response to this change instruction, determine that the execution method of the subtask of selecting a gathering restaurant is to search for "Hakka cuisine", and select a Hakka cuisine restaurant with a recommendation degree reaching the degree threshold, and execute according to the changed execution method accordingly.

[0240] As can be seen from the above, the ability sharing solution provided by this application determines a third-party artificial intelligence ability that matches the control type of the business control in response to a trigger operation on the business control in the current application; displays an ability call interface of the third-party artificial intelligence ability in the control interface of the business control; in response to a trigger operation on the ability call interface, calls the third-party artificial intelligence ability to perform artificial intelligence processing on the business data corresponding to the business control to obtain a processing result; and displays the processing result in the current application. In this way, the sharing of third-party artificial intelligence abilities can be achieved at the control level, enabling business controls that do not have artificial intelligence capabilities themselves to also provide artificial intelligence capabilities, thereby enhancing the diversity of their functions.

[0241] To facilitate better implementation of the ability sharing method provided by the embodiments of this application, the embodiments of this application also provide an ability sharing device based on the above ability sharing method. The meanings of the terms are the same as those in the above ability sharing method, and for specific implementation details, please refer to the descriptions in the above method embodiments.

[0242] Please refer to Figure 10 , Figure 10 which is a schematic structural diagram of the ability sharing device provided by the embodiments of this application. The ability sharing device may include an ability determination module 310, an ability display module 320, an ability call module 330, and a result display module 340. Among them,

[0243] The ability determination module 310 is configured to determine a third-party artificial intelligence ability that matches the control type of the business control in response to a trigger operation on the business control in the current application;

[0244] The ability display module 320 is configured to display an ability call interface of the third-party artificial intelligence ability in the control interface of the business control;

[0245] The ability call module 330 is configured to call the third-party artificial intelligence ability to perform artificial intelligence processing on the business data corresponding to the business control to obtain a processing result in response to a trigger operation on the ability call interface;

[0246] The result display module 340 is configured to display the processing result in the current application.

[0247] Optionally, in one embodiment, the ability determination module 310 is configured to identify whether a sharing identifier is added to the control attributes of the business control in response to a trigger operation on the business control in the current application; and determine a third-party artificial intelligence ability that matches the control type of the business control in response to the sharing identifier being added to the control attributes of the business control.

[0248] Optionally, in one embodiment, the capability determination module 310 is configured to identify the sharing level of the shared identifier added to the control attributes of the service control; and determine a third-party artificial intelligence capability that matches the control type of the service control and conforms to the sharing level.

[0249] Optionally, in one embodiment, the capability display module 320 is configured to evaluate the matching degree of the third-party artificial intelligence capability in the current service scenario; screen out the target third-party artificial intelligence capabilities whose matching degree reaches the degree threshold from the third-party artificial intelligence capabilities; and display the capability call interface of the target third-party artificial intelligence capabilities in the control interface of the service control.

[0250] Optionally, in one embodiment, the capability sharing device provided by the present application further includes an intent execution module, configured to obtain the object intent of the operation object of the current application; determine the service control operation sequence for completing the object intent according to the content description information of the service control in the current application; and sequentially operate the corresponding target service controls in the current application according to the service control operation sequence.

[0251] Optionally, in one embodiment, the intent execution module is further configured to determine multiple candidate service control operation sequences for completing the object intent according to the content description information of the service control in the current application; perform quality scoring on the multiple candidate service control operation sequences, and screen out the candidate service control operation sequence with the highest quality score from the multiple candidate service control operation sequences as the service control operation sequence.

[0252] Optionally, in one embodiment, the intent execution module is further configured to determine the service control to be supplemented with incomplete content description information; obtain the content description information of the service control to be supplemented, and configure the content description information for the service control to be supplemented.

[0253] Optionally, in one embodiment, the intent execution module is configured to obtain the identity information of the service control to be supplemented from the layout attributes of the service control to be supplemented;

[0254] According to the identity information, use a large language model to describe the content of the service control to be supplemented, and obtain the content description information of the service control to be supplemented.

[0255] Optionally, in one embodiment, the intent execution module is configured to obtain a screenshot of the application interface including the service control to be supplemented; input the screenshot of the application interface into a multimodal large model, and use the multimodal large model to describe the content of the service control to be supplemented, and obtain the content description information of the service control to be supplemented.

[0256] Optionally, in one embodiment, the capability sharing device provided in the present application further includes a cross-application execution module, which is configured to determine the content to be recognized from the current application, and determine the cross-application task to be executed through a large language model according to the content to be recognized; and disassemble the cross-application task into subtasks to be executed by different applications through the large language model according to the application description information of different applications; and call different applications to execute the corresponding subtasks through a multimodal large model.

[0257] Optionally, in one embodiment, the cross-application execution module is further configured to determine the execution progress of the cross-application task and display the execution progress of the cross-application task.

[0258] Optionally, in one embodiment, the cross-application execution module is further configured to respond to a change instruction for the currently executed subtask and change the execution manner of the currently executed subtask according to the change instruction.

[0259] For the specific implementation of each of the above units / modules, reference may be made to the previous embodiments and will not be elaborated here.

[0260] An embodiment of the present application further provides an electronic device, including a memory and a processor, where the processor is configured to execute the steps in the capability sharing method provided in this embodiment by calling a computer program stored in the memory.

[0261] Please refer to Figure 11 , Figure 11 which is a schematic structural diagram of the electronic device provided in the embodiment of the present application.

[0262] The electronic device may include a processor 101 with one or more processing cores, a memory 102 with one or more computer-readable storage media, a power supply 103, an input unit 104, and other components. Those skilled in the art can understand that Figure 11 the structural diagram of the electronic device shown in

[0263] does not constitute a limitation to the electronic device, and it may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements. Among them:

[0264] The memory 102 can be used to store software programs and modules. The processor 101 executes various functional applications and data processing by running the software programs and modules stored in the memory 102. The memory 102 mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created according to the use of the electronic device. In addition, the memory 102 can include high-speed random access memory, and can also include non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 102 can also include a memory controller to provide the processor 101 with access to the memory 102.

[0265] The electronic device further includes a power supply 103 for powering each component. Optionally, the power supply 103 can be logically connected to the processor 101 through a power management system, so as to implement functions such as management of charging, discharging, and power consumption management through the power management system. The power supply 103 can also include any components such as one or more DC or AC power supplies, a recharge system, a power failure detection circuit, a power converter or inverter, and a power status indicator.

[0266] The electronic device may further include an input unit 104, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls.

[0267] Although not shown, the electronic device may further include a display unit, an image acquisition component, etc., which will not be elaborated here. Specifically, in this embodiment, the processor 101 in the electronic device will load the executable code corresponding to one or more computer programs into the memory 102 according to the following instructions, and the processor 101 will execute the steps in the ability sharing method provided in this application, such as:

[0268] In response to a trigger operation on a service control in the current application, determine a third-party artificial intelligence ability that matches the control type of the service control;

[0269] Display an ability call interface of the third-party artificial intelligence ability in the control interface of the service control;

[0270] In response to a trigger operation on the ability call interface, call the third-party artificial intelligence ability to perform artificial intelligence processing on the service data corresponding to the service control to obtain a processing result;

[0271] Display the processing result in the current application.

[0272] It should be noted that the electronic device provided in the embodiments of the present application and the capability sharing method in the above embodiments belong to the same concept. For the specific implementation process, please refer to the above relevant embodiments and will not be elaborated here.

[0273] The present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program stored thereon is executed by the processor of the electronic device provided in the embodiments of the present application, the processor of the electronic device executes the steps in the capability sharing method provided by the present application. Among them, the storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM), etc.

[0274] The above has introduced in detail a capability sharing method, apparatus, electronic device, and computer-readable storage medium provided by the present application. Specific examples are used in this article to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application; at the same time, for those skilled in the art, according to the idea of the present application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present application.

[0275] It should be noted that when the above embodiments of the present application are applied to specific products or technologies, and relevant user data is involved, user permission or consent needs to be obtained, and the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of relevant countries and regions.

Claims

1. A method for sharing capabilities, characterized in that, Including: In response to a trigger operation on a service control in the current application, determine a third-party artificial intelligence capability that matches the control type of the service control; Display a capability call interface of the third-party artificial intelligence capability in the control interface of the service control; In response to a trigger operation on the capability call interface, call the third-party artificial intelligence capability to perform artificial intelligence processing on the service data corresponding to the service control to obtain a processing result; Display the processing result in the current application.

2. The method for sharing capabilities according to claim 1, characterized in that, The step of, in response to a trigger operation on a service control in the current application, determining a third-party artificial intelligence capability that matches the control type of the service control includes: In response to a trigger operation on a service control in the current application, identify whether a sharing identifier is added to the control attributes of the service control; In response to the sharing identifier being added to the control attributes of the service control, determine a third-party artificial intelligence capability that matches the control type of the service control.

3. The method for sharing capabilities according to claim 2, characterized in that, The step of determining a third-party artificial intelligence capability that matches the control type of the service control includes: Identify the sharing level of the sharing identifier added to the control attributes of the service control; Determine a third-party artificial intelligence capability that matches the control type of the service control and meets the sharing level.

4. The method for sharing capabilities according to claim 3, characterized in that, The step of, in the control interface of the service control, displaying a capability call interface of the third-party artificial intelligence capability includes: Evaluate the matching degree of the third-party artificial intelligence capability in the current business scenario; Filter out target third-party artificial intelligence capabilities with a matching degree reaching a degree threshold from the third-party artificial intelligence capabilities; Display a capability call interface of the target third-party artificial intelligence capabilities in the control interface of the service control.

5. The method for sharing capabilities according to claim 1, characterized in that, The capability sharing method further includes: Obtain the object intention of the operation object of the current application; According to the content description information of the service control in the current application, determine a service control operation sequence for completing the object intention; According to the service control operation sequence, sequentially operate the corresponding target service controls in the current application.

6. The method for sharing capabilities according to claim 5, characterized in that, The step of, according to the content description information of the service control in the current application, determining a service control operation sequence for completing the object intention includes: According to the content description information of the service control in the current application, determine multiple candidate service control operation sequences for completing the object intention; Perform quality scoring on the multiple candidate service control operation sequences, and filter out the candidate service control operation sequence with the highest quality score from the multiple candidate service control operation sequences as the service control operation sequence.

7. The method for sharing capabilities according to claim 5, characterized in that, Before the step of obtaining the object intention of the operation object of the current application, it further includes: Determine a service control to be supplemented with incomplete content description information; Obtain the content description information of the service control to be supplemented, and configure the content description information for the service control to be supplemented.

8. The method for sharing capabilities according to claim 6, characterized in that, The step of obtaining the content description information of the service control to be supplemented includes: Obtain the identity information of the service control to be supplemented from the layout attributes of the service control to be supplemented; According to the identity information, use a large language model to describe the content of the business control to be supplemented, and obtain the content description information of the business control to be supplemented.

9. The method for sharing capabilities according to claim 6, characterized in that, The obtaining of the content description information of the business control to be supplemented includes: Obtain a screenshot of the application interface including the business control to be supplemented; Input the application interface screenshot into a multimodal large model, and use the multimodal large model to describe the content of the business control to be supplemented, so as to obtain the content description information of the business control to be supplemented.

10. The ability sharing method according to claim 1, wherein, The capability sharing method further includes: Determine the content to be recognized from the current application, and determine the cross-application task to be executed through a large language model according to the content to be recognized; According to the application description information of different applications, disassemble the cross-application task into subtasks to be executed by different applications through a large language model; Through a multimodal large model, call different applications to execute the corresponding subtasks.

11. The ability sharing method according to claim 10, wherein, The capability sharing method further includes: Determine the execution progress of the cross-application task and display the execution progress of the cross-application task.

12. The ability sharing method according to claim 10, wherein, The capability sharing method further includes: In response to a change instruction for the currently executed subtask, change the execution method of the currently executed subtask according to the change instruction.

13. An ability sharing device, wherein, It includes: A capability determination module, configured to determine a third-party artificial intelligence capability matching the control type of the business control in response to a trigger operation on the business control in the current application; A capability display module, configured to display a capability call interface of the third-party artificial intelligence capability in the control interface of the business control; A capability call module, configured to call the third-party artificial intelligence capability to perform artificial intelligence processing on the business data corresponding to the business control in response to a trigger operation on the capability call interface, and obtain a processing result; A result display module, configured to display the processing result in the current application.

14. An electronic device, wherein, It includes a memory and a processor, the memory stores a computer program, and the processor is configured to run the computer program in the memory to execute the steps in the capability sharing method according to any one of claims 1 to 12.

15. A computer-readable storage medium, wherein, The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute the steps in the capability sharing method according to any one of claims 1 to 12.

Images