Method for invoking client plug-in, device, medium, and program product

Abstract

Embodiments of the present invention provide a method for invoking a client plug-in, a device, a storage medium, and a computer program product. The method comprises: sending a target request to a target model, wherein the target request involves invoking a target client plug-in; receiving an invocation command for the target client plug-in from a long-connection service on the basis of a preset communication protocol, wherein the invocation command is generated on the basis of an invocation request received from the target model and the preset communication protocol; and executing the target client plug-in according to the invocation command. By means of the method in the embodiments of the present invention, a target model can be used to invoke a local client plug-in without listening, so as to intelligently execute corresponding functions of the client plug-in on the basis of a user request, thereby extending a capability boundary of the target model.

Description

Method, device, medium and program product for invoking an endpoint plugin TECHNICAL FIELD

[0001] The present disclosure relates generally to the field of computers, and more specifically to a method, an electronic device, a computer-readable storage medium, and a computer program product for invoking an endpoint plugin. BACKGROUND

[0002] With the rapid development and continuous breakthroughs in deep learning technology, many deep learning models have emerged. Deep learning models usually contain a large number of parameters and are capable of understanding the deep meaning of language through learning and generating coherent and logical text content, which has an impact on people's lives.

[0003] In order to enhance the functionality and flexibility of deep learning models, technical personnel have begun to associate deep learning models with various kinds of plugins, allowing users to expand the functionality of deep learning models according to needs, such as calling search engines, improving dialogue experiences, or integrating other external services, etc., so that deep learning models are more intelligent and easy to adapt to application requirements in different scenarios.

[0004] SUMMARY

[0005] According to example embodiments of the present disclosure, a method, an electronic device, a computer storage medium, and a computer program product for invoking an endpoint plugin are provided.

[0006] In a first aspect of the present disclosure, a method for invoking an endpoint plugin is provided, comprising: sending a target request to a target model, the target request involving invoking a target endpoint plugin; receiving a calling command for the target endpoint plugin from a long link service according to a preset communication protocol, the calling command being generated based on the calling request received from the target model and the preset communication protocol; and executing the target endpoint plugin according to the calling command.

[0007] In a second aspect of the present disclosure, a method for invoking an endpoint plugin is provided, comprising: receiving a target request, the target request involving invoking a target endpoint plugin; generating a calling request using a target model according to the target request, the calling request indicating invoking the target endpoint plugin; and sending the calling request to a long link service to generate a calling command according to the calling request using a preset communication protocol.

[0008] In a third aspect of the present disclosure, an electronic device is provided, comprising: at least one processing unit; at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, which when executed by the at least one processing unit causes the electronic device to perform the method described in the first aspect of the present disclosure.

[0009] In a fourth aspect of the present disclosure, there is provided a computer readable storage medium having stored thereon machine executable instructions, which when executed by a device, cause the device to perform the method described according to the first aspect of the present disclosure.

[0010] In a fifth aspect of the present disclosure, there is provided a computer program product comprising computer executable instructions, wherein the computer executable instructions implement the method described according to the first aspect of the present disclosure when executed by a processor.

[0011] The summary is provided to introduce a selection of concepts that are further described below in the detailed description. This summary is not intended to identify key or essential features of the disclosure, nor is it intended to limit the scope of the disclosure. Other features will be more fully described in the following detailed description, taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0012] FIG. 1 shows a schematic diagram of an example environment in which embodiments of the present disclosure can be implemented;

[0013] FIG. 2 shows a flowchart of a method for invoking a client-side plugin according to an embodiment of the present disclosure;

[0014] FIG. 3 shows a schematic diagram of a system 300 for invoking a client-side plugin according to an embodiment of the present disclosure;

[0015] FIG. 4 shows a schematic diagram of a method for invoking a client-side plugin according to an embodiment of the present disclosure;

[0016] FIG. 5 shows a schematic diagram of invoking in different interfaces according to an embodiment of the present disclosure;

[0017] FIG. 6 shows a schematic diagram of invoking in different interfaces according to an embodiment of the present disclosure;

[0018] FIG. 7A shows a schematic diagram of invoking a client-side plugin to collect data according to an embodiment of the present disclosure;

[0019] FIG. 7B shows a schematic diagram of a method for invoking a client-side plugin according to an embodiment of the present disclosure;

[0020] FIG. 8 shows a flowchart of a method for invoking a client-side plugin according to an embodiment of the present disclosure;

[0021] FIG. 9 shows a schematic block diagram of an example apparatus according to some embodiments of the present disclosure;

[0022] FIG. 10 shows a schematic block diagram of an example apparatus according to some embodiments of the present disclosure;

[0023] FIG. 11 illustrates a block diagram of an example device that can be used to implement embodiments of the present disclosure.

[0024] In all the drawings, like or similar reference numerals refer to like or similar elements. DETAILED DESCRIPTION

[0025] The names of the messages or information exchanged between the plurality of devices in the embodiments of the present disclosure are only for illustrative purposes, and are not intended to limit the scope of the messages or information. It can be understood that, before using the technical solutions disclosed in the embodiments of the present disclosure, the type of personal information involved in the present disclosure, the use range, the use scenario, etc. should be informed to the user and the authorization of the user should be obtained in a proper manner according to relevant laws and regulations.

[0026] For example, in response to receiving an active request of a user, a prompt information is sent to the user to explicitly prompt the user that the operation requested to be performed will require obtaining and using personal information of the user. Thus, the user can voluntarily choose whether to provide personal information to the software or hardware such as an electronic device, an application program, a server or a storage medium, etc. performing the operation of the technical solutions of the present disclosure according to the prompt information. As an optional but non-limiting implementation manner, in response to receiving an active request of a user, the manner of sending a prompt information to the user may, for example, be a pop-up window manner, and the prompt information may, for example, be presented in the form of text in the pop-up window. In addition, the pop-up window may, for example, carry a selection control for the user to select “agree” or “disagree” to provide personal information to the electronic device.

[0027] It can be understood that the above notification and obtaining of user authorization process is only illustrative, and does not limit the implementation manners of the present disclosure, and other manners meeting relevant laws and regulations can also be applied to the implementation manners of the present disclosure.

[0028] Embodiments of the present disclosure will be described in more detail below with reference to the drawings. Although certain embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms, and should not be interpreted as being limited to the embodiments set forth herein, but rather these embodiments are provided to more thoroughly and completely understand the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes, and are not intended to limit the scope of protection of the present disclosure.

[0029] In the description of embodiments of the present disclosure, the term “comprising” and similar terms are to be understood as open-ended, i.e., “including but not limited to”. The term “based on” is to be understood as “based at least in part on”. The term “one embodiment” or “the embodiment” is to be understood as “at least one embodiment”. The terms “a first”, “a second”, etc. can refer to different or identical objects, unless explicitly stated otherwise. Other explicit and implicit definitions can also be included below.

[0030] Currently, when a deep learning model (e.g., a large language model) calls a plug-in, it can usually only select a plug-in deployed and running on a server of an enterprise. This is because there is usually a listening service running in real time on such a server, which is set with a service interface and is dedicated to listening to requests for calling plug-ins on the server. This method cannot be applied to plug-ins on the client side, because there is no similar listening service running on the client side to listen to various requests in real time. The running of a plug-in (or application) on the client side requires the triggering of a user to run.

[0031] To this end, the present disclosure proposes a method for calling an end plug-in. The method uses a target model to generate a calling request according to a target request, and uses a long link and a preset communication protocol to replace a listening service, so that the target model can inform the client side of the calling request in real time, and the client side executes an end plug-in according to a calling command. Therefore, the method can call a local end plug-in using a target model without listening, so as to intelligently execute the corresponding function of the end plug-in according to the request of a user, and expand the capability boundary of the target model.

[0032] Embodiments of the present disclosure will be described in detail below with further reference to the accompanying drawings, in which FIG. 1 shows a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. The example environment 100 includes computing devices 110 and 120. The example environment 100 includes computing devices 110 and 120. The computing device 110 can be deployed with a target model (e.g., a large language model) that can generate various content after being trained. The computing device 120 is also shown in FIG. 1. In some embodiments, the computing device 120 communicates with the computing device 110 through a network 130. The network 130 can include a wired network, a wireless network, or a combination thereof, for providing communication between the computing device 120 and the computing device 110. In some embodiments, the computing device 120 can be connected to the computing device 110 through a data line, and the present disclosure does not limit the connection manner between the computing device 110 and the computing device 120.

[0033] The computing devices 110 and 120 can include, but are not limited to, personal computers, server computers, handheld or laptop devices, mobile devices (such as mobile phones, personal digital assistants (PDAs), media players, etc.), multi-processor systems, consumer electronics, wearable electronic devices, smart home devices, minicomputers, mainframe computers, edge computing devices, distributed computing systems including any of the above systems or devices, etc.

[0034] An application (e.g., a client program) installed in the computing device 120 can invoke the target model to execute a local application (i.e., an end plug-in). For example, in the system 100 of FIG. 1, the computing device 120 can communicate with the computing device 110 through the network 130, and send a target request 122 to the computing device 110, "please help me open "XX music player" and play song "AAA"! The target request 122 involves invoking the target end plug-in "XX music player" 140. The computing device 110 inputs the target request 122 into the target model. In some embodiments, the target model performs intent recognition on the target request 122 and identifies that the target request 122 involves invoking the target end plug-in "XX music player" 140 on the computing device 120. The target model generates an invocation request for the target end plug-in "XX music player" 140 through the content of the target request 122. In some embodiments, the invocation request is packaged into an invocation command by the long link service according to a preset communication protocol, and the invocation command is sent to the computing device 120 by the long link service. The long link service can be established in the network 130. In some embodiments, the computing device 120 receives the invocation command for the target end plug-in "XX music player" 140 from the long link service and parses the invocation command according to the preset communication protocol. After parsing the invocation command according to the preset communication protocol, the computing device 120 can generate an execution code according to the content of the invocation command, such as the function function to be invoked, etc. In some embodiments, the computing device 120 executes the target end plug-in "XX music player" 140 according to the parsed invocation command. As shown in FIG. 1, the current display interface of the computing device 120 is switched from the interface of the invoking client program to the interface of the target end plug-in "XX music player" 140, which means that the plug-in is started. Moreover, the computing device 120 is playing song AAA, which means that the function function about playing song AAA is executed. In some embodiments, the target model in the computing device 110 sends a feedback 124 to the computing device 120 after learning the execution result, "OK, "XX music player" is started and song AAA is played". In some embodiments, the feedback information can be presented in the form of voice playing. In some embodiments, the long link service and the target model can be located in different computing devices.

[0035] According to the method of the present disclosure, the target model is used to generate an invocation request according to a target request 122, and the long link and the preset communication protocol are used instead of the listening service, so that the target model can inform the client of the invocation request in real time, and the client executes the end plug-in according to the invocation command, thus the method can invoke the local end plug-in using the target model without listening, so as to intelligently execute the corresponding function of the end plug-in according to the user's request, and expand the capability boundary of the target model.

[0036] As shown in FIG. 1, in an environment 100, a network 130 can be utilized to transmit data between a computing device 110 and a computing device 120. The network 130 has a theoretical bandwidth, which refers to the maximum transmission speed supported by the network 130, representing the maximum amount of data that the network 130 can transmit in an ideal case, usually measured in bits per second (bps). For example, if the theoretical bandwidth of the network 130 is 100 Mbps, it means that in an ideal case it can transmit one hundred megabits of data per second. However, in reality, it can not reach the actual transmission speed of 100 Mbps due to other factors in the network (e.g., signal interference, bandwidth sharing, transmission delay, etc.).

[0037] As understood by those of ordinary skill in the art, the computing device 110 can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and big data and artificial intelligence platform, etc. The servers can be connected directly or indirectly through wired or wireless communication, which is not limited in the present application.

[0038] The computing device 120 can be any type of mobile computing device, including a mobile computer (e.g., a personal digital assistant (PDA), a laptop computer, a notebook computer, a tablet computer, a netbook, etc.), a mobile phone (e.g., a cellular phone, a smartphone, etc.), a wearable computing device (e.g., a smartwatch, a head-mounted device including smart glasses, etc.), or other types of mobile devices. In some embodiments, the computing device 120 can also be a stationary computing device, such as a desktop computer, a game console, a smart television, etc. It should be understood that the computing device 120 can be used to replace the computing device 110 to perform the above operations, or the computing device 110 and the computing device 120 can be used together to perform the above operations, as long as the computing device 120 has sufficient computing capability.

[0039] It should be understood that the architecture and functions in the example environment 100 are described for illustrative purposes only, without implying any limitation on the scope of the present disclosure. Embodiments of the present disclosure can also be applied to other environments with different structures and / or functions.

[0040] The processes according to embodiments of the present disclosure will be described in detail below with reference to FIGS. 2-7B. For ease of understanding, the specific data mentioned in the following description are exemplary and do not limit the protection scope of the present disclosure. It can be understood that the embodiments described below can also include additional actions not shown or can omit the actions shown, and the scope of the present disclosure is not limited in this respect.

[0041] FIG. 2 shows a flowchart of a method 200 for invoking an endpoint plugin according to certain embodiments of the present disclosure. At block 202, a target request is sent to a target model, the target request involves invoking a target endpoint plugin. The target model can be any type of target model, for example, it can be a large language model (LLM). The target model can be pre-trained using diverse samples, for example, it can use various sample libraries or datasets provided on the Internet. The target request can be any form of request. For example, the request can be “please help me play song AAA”, or it can be “please open music playing application A and play song AAA”. The endpoint plugin is a third-party application associated with the target model, the target model as the brain, the endpoint plugin is the ear, eye and hand of the large model, which can help the target model to enrich the capabilities of the large model and expand the application scenarios, and complete the tasks that the generation capability of the target model cannot complete.

[0042] At block 204, an invocation command for the target endpoint plugin is received from the long link service according to a preset communication protocol, the invocation command is generated based on the invocation request received from the target model and the preset communication protocol. The target model can perform intent analysis according to the target request 122 from the computing device 120 and identify the endpoint plugin that can be used. For example, the target model can identify from “please help me play song AAA” that the user can want to invoke the endpoint plugin A (i.e., the music playing application A) to play the song AAA. The target model can generate an invocation command, for example, the invocation command includes starting the endpoint plugin A, invoking the function of playing the song, etc. The long link service is a communication mode that keeps a long time connection state between the computing device 110 and the computing device 120. The long link is kept for a period of time after being established, rather than being closed immediately after completing a data interaction. This allows the computing device 110 and the computing device 120 to transmit data at any time without the need to reestablish the connection each time, thereby reducing the time overhead and resource consumption of connection establishment. The computing device 120 can parse the invocation command according to the preset communication protocol to understand the content of the invocation command, for example, which endpoint plugin needs to be invoked and which function function is used to invoke the endpoint plugin.

[0043] In this embodiment, the long link not only has the function of transmitting data, but also can link the endpoint plugin on the computing device 120 and control the endpoint plugin in combination with the preset communication protocol. The application or process executing the method of the present disclosure on the computing device 120 can invoke the endpoint plugin through the long link. At block 206, the target endpoint plugin is executed according to the invocation command. The computing device 120 can generate an execution code through the parameters contained in the invocation command, and execute the execution code in the computing device 120, so as to start the target endpoint plugin and execute the function called in the invocation command, etc.

[0044] According to the method of the present disclosure, the target model is used to generate a calling request according to a target request, and a long link and a preset communication protocol are used to replace the listening service, so that the target model can inform the client of the calling request in real time, and the client executes the end plug-in according to the calling command, and therefore the method can use the target model to call the local end plug-in without listening, so as to intelligently execute the corresponding function of the end plug-in according to the request of the user, and expand the capability boundary of the target model.

[0045] FIG. 3 shows a schematic diagram of a system 300 for calling an end plug-in according to an embodiment of the present disclosure. In this embodiment, the system 300 for calling an end plug-in can include a three-layer architecture, which is a management layer 310, a protocol layer 340 and an application layer 360. The management layer 310 includes two aspects of management, which are end plug-in registration 320 and end plug-in binding 330. The end plug-in registration 320 indicates that each end plug-in that can be called is pre-registered in the system, including but not limited to recording each registration information for the end plug-in. The registration information for the end plug-in includes the name and description of the end plug-in, the function function set for the end plug-in, and the parameter set for the function function set.

[0046] The function function in the function function set is a function required to execute the end plug-in, and these function functions are usually recorded in the code of the end plug-in. In order to enable the target model to know the operation mode of calling the end plug-in to implement a specific function, all the function functions that can be used can be extracted and registered in the system 300. Other registration information for the end plug-in can be similarly processed. As an example, for a music playing application, the start function function and the play function function in the music playing application can be extracted and registered in the system 300. In addition, the end plug-in version 324 and the end plug-in state 326 and the like can also be registered. The end plug-in version 324 is used to mark the version number of the current end plug-in, and the end plug-in state 326 indicates whether the current end plug-in is available.

[0047] In the management layer 310, an end plugin binding 330 can also be included. The end plugin binding 330 indicates the binding of the registered end plugin to the target model. It should be noted that the difference between registration and binding is that the registration information of the end plugin will be sent to the target model, which is used by the target model to generate a reasonable call request according to the registration information of the end plugin after intent recognition. While the end plugin binding refers to the call ability of the target model that can be known by the target model in the prompt engineering stage. For example, a keyword detection can be set in the prompt engineering stage, and when the target request appears to the keyword "M", the target model can know that it can call the end plugin corresponding to the keyword "M". The end plugin binding 330 can include an application binding 332 and a special model binding 334. The application binding 332 indicates the binding of the end plugin to the target model, and the special model binding 334 refers to the case where a special model is deployed locally, which can be regarded as the binding of the end plugin to the target model. In this case, the target model can call the ability of the special model, thereby expanding the ability boundary of the target model.

[0048] The long link management 350 includes a long link service 352, a call command push 354, an execution result push 356, and a communication protocol 358 for processing operations related to the long link. The long link service 352 is used to create or disconnect long link communication. The call command push 354 is used to package the call request generated by the target model to obtain a call command, and then send the call command to the other end of the long link communication. The execution result push 356 is used to send the result obtained by executing the code locally in the system 300 (such as whether the execution is successful or not, the specific data required to be called in the call command, etc.) to the other end of the long link communication. The communication protocol 358 is a communication rule agreed upon by the communication parties on the basis of long link communication, such as the role of each field.

[0049] In some embodiments, the long link service 352 in the long link management 350 is used to send the registration information of the target end plugin to the other end of the long link service 352 using the preset communication protocol 358. The communication protocol 358 is set on the long link service, and the content of the registration information represented by different fields can be executed in the communication protocol 358. For example, it can be agreed in advance that the name (or identifier) of the end plugin is filled in the first field, and the main purpose or function of the end plugin is described in the second field. As an example, in the case where the target end plugin is a music playing application, the start function function and the play function function are started, so that the target model knows that the function of one or more of the two function functions can be used to generate a call request.

[0050] At the application layer 360, there are a client 370 and a platform 380. The client 370 indicates a terminal device using the method of the embodiments of the present disclosure for use purpose, and the platform 380 indicates a terminal device using the method of the embodiments of the present disclosure for testing purpose. The platform 380 can include a development platform 382 where a user can develop a web application calling a target model, and if the user wants the web application to have the ability to call the client plug-in, the method of the client plug-in of the embodiments of the present disclosure can be applied.

[0051] The client 370 includes a target model application 372, a browser 374 and an application 376. The target model application 372 is an application developed for a target model, for example, a client application. It can provide a language user interface where a user can input a request (i.e. a target request). The language user interface refers to a conversational input interface provided with an input box, such as the interface displayed in the computing device 120 shown in FIG. 1, which includes the conversation contents 122 and 124. The browser 374 can also provide a service of calling a target model, for example, it can open an interface that can call a target model through a uniform resource locator, while providing a language user interface in the form of a web page where a user can input a request. The application 376 can be any application on the system 300, for example, it can be a music playing software. An interface of calling a target model can be set on the graphical user interface of the application 376. A user triggers the interface to jump to a language user interface and call a target model.

[0052] In this embodiment, various implementations are provided. For example, the application layer can be implemented as a separate target model application 372, can be implemented in a browser 374, can be implemented in a third-party application such as the application 376. Moreover, the embodiment of FIG. 3 indicates that the method of the client plug-in of the present disclosure can be used for use purpose, and can also be used for testing purpose, development purpose, etc. In addition, the embodiment of FIG. 3 provides a method of registering a client plug-in, which, together with a long link, plays a role of linking the client plug-in on the computing device 120, and expands the capability boundary of the target model.

[0053] Different types of applications are shown in the application layer in the FIG. 3 embodiment. Target model application 372 belongs to the native side, and browser 374 belongs to the web side. Generally, if the execution code is generated on the native side, the execution code can only be executed on the native side and call the application on the native side. If the execution code is generated on the web side, the execution code can only be executed on the web side and call the application on the web side. In order to enable the two types of applications to call each other, in some embodiments, if the execution code is located on the native side and the target side plug-in is on the web side, the target side plug-in is executed according to the execution code through a channel. In some embodiments, if the execution code is located on the web side and the target side plug-in is on the native side, the target side plug-in is executed according to the execution code through a channel. For example, the channel can be a Java Script Bridge (JSB) channel, through which the native side application can communicate with the application on the web side to enable execution of the execution code.

[0054] FIG. 4 shows a schematic diagram of a method for calling a side plug-in according to an embodiment of the present disclosure. In this embodiment, a user can operate on a client 410. The client 410 can have, for example, a target model application 372 installed thereon. The target model application 372 can present a language user interface 412 to the user. At 412, the user inputs data as a target request in the language user interface. The target model application 372 sends the request to a target model 430. At 432, the target model processes the user input. During the processing, the prompt word engineering can be completed for the target request, so that the target model 372 can know that it has the ability to call a side plug-in.

[0055] At 434, the target model 372 performs intent recognition 434 for the target request. If the target request does not involve calling a side plug-in, the target model 372 can generate a corresponding reply using known data, and then proceed to 420 to respond to the target request. If the target request involves calling a side plug-in, the target model 372 can receive the registration information for the side plug-in on the client 410 from the client 410 via a long link service 450 using a preset communication protocol, such as the information in the side plug-in registration 320 in FIG. 3.

[0056] At 442, the target model 372 matches the name (or identification) of the target plug-in corresponding to the target request from the registration information of the target plug-in. With the registration information of the target plug-in, the target model 372 generates a call request according to the target request. The call request is sent to the long link service 450. The long link service 450 encapsulates the call request into a call command according to the agreed communication protocol (i.e., the preset communication protocol), for example, sets the field values according to the call parameters in the call request. In some embodiments, the call command includes the name of the target plug-in, the function function of the request call for the target plug-in, and the parameters for the function function of the request call, which is at least part of the function function set of the target plug-in. Specifically, which function functions and how to set the parameters can depend on the specific target request and the registration information of the target plug-in. By designing the call command containing the above contents, the target model can achieve the call of the plug-in 440 on the client 410.

[0057] Via the long link communication, the long link service 450 sends the call command to the client 410 using the preset communication protocol. After the client 410 receives the call command for the target plug-in based on the long link service, the call command is parsed based on the preset communication protocol to obtain the name of the target plug-in, the function function of the request call, and the parameters. At 414, the execution code for the target plug-in is generated based on the name of the target plug-in, the function function of the request call, and the parameters. At 420, the target plug-in is executed according to the execution code for the target plug-in. The execution code is executed at 422. At 454, the execution result is sent to the long link service.

[0058] In some embodiments, the action performed when the target plug-in is executed has persistence, so a certain listening time needs to be set to listen to the execution result. The target model 430 can add parameters related to the listening time period in the call request, and the long link service 450 encapsulates them in the communication packet. In some embodiments, the client 410 parses the call command to obtain the listening time period. This embodiment also includes that the client 410 listens to the execution result of the target plug-in in the listening time period. Returning to 432, the long link service 450 sends the execution result of the target plug-in to the long link service 450 and inputs it into the target model 430, which determines the feedback for the target request according to the execution result and presents the feedback, for example, sends a notification to the client device that the target request has been completed. In this embodiment, the long link service management 424 and the long link service management 456 jointly maintain the long link communication in the long link service 450, including creating a long link communication, disconnecting a long link communication, sending and receiving data according to the preset communication protocol, etc.

[0059] In the embodiment of FIG. 4, the invocation of the endpoint plugin can also be triggered from the graphical user interface. At 418, a user can operate on the graphical user interface. At 416, the corresponding software code is executed proactively according to the user's operation on the application. For example, the user can right-click (or other invocation instruction) on the graphical user interface of the application, and the application executes the corresponding software behavior, such as popping up a list of options, which includes the invocation of the target segment plugin using the target model. If the user further selects the option, the operation content can be converted into a target request and sent to the long link service 450. At 454, the long link service 450 receives the endpoint response and proceeds to 432 to invoke the target model 430 to process the target request, thereby achieving the invocation of the target endpoint plugin.

[0060] In the embodiment shown in FIG. 4, the communication with the client 410 is implemented through the long link service 450, and the invocation of the endpoint plugin 440 by the target model 430 is implemented based on the long link service 450 and the endpoint plugin registration information, so that the target model can effectively and quickly use the local endpoint plugin without the need for listening, thereby being able to intelligently execute the corresponding function of the endpoint plugin according to the user's request.

[0061] FIG. 4 shows the process of invoking one endpoint plugin, but this should not limit the method of the present disclosure. How many endpoint plugins to invoke can be determined according to the target request. In some embodiments, the endpoint plugin registration 320 further includes registration information for a second endpoint plugin. In this implementation, a new invocation command is received by the computing device 120, which is determined by the target model according to the execution result of the previous target endpoint plugin and the endpoint plugin registration information for the target request. This embodiment further includes executing, by the computing device 120, the second endpoint plugin according to the new invocation command. This embodiment further includes obtaining, by the computing device 120, a second execution result for the second endpoint plugin. This embodiment further includes sending, by the computing device 120, the second execution result for the second endpoint plugin to the long link service. In this embodiment, the target model generates feedback based on the execution result of the previous target endpoint plugin and the second execution result for the second endpoint plugin.

[0062] As an example, the target request can be "It's time for lunch, I'm a little hungry, what's to eat nearby?" The target model can not only invoke an endpoint plugin for recording the client's location information to obtain the client's location information. It can also not rush to generate feedback, but continue to invoke a life class endpoint plugin to obtain the nearby food and the user's historical preferences based on the location information, and determine a restaurant or dish based on the nearby food and the user's historical preferences as the final feedback.

[0063] In addition, the present disclosure also relates to different interaction modes according to actual conditions when the target model application invokes the end plug-in. Generally, the target model application on the computing device 120 adopts a language user interface. FIG. 5 shows a schematic diagram of invocation in different interfaces according to an embodiment of the present disclosure. A user can interact with the target model in a language user interface 502. For example, a target request input by the user is “please help me open “XX music player” and play song “AAA” from the 2nd minute!” The end plug-in “XX music player” can be invoked according to the process disclosed in the foregoing embodiment, and if the execution code is successfully executed, the language user interface can be switched to a graphical user interface 504 (a play interface of the “XX music player”), and the song can be played from the 2nd minute.

[0064] Therefore, in this embodiment, the user can realize invocation of the graphical user interface based on the language user interface, and switching between the two interaction modes. In addition, the language user interface can also be invoked based on the graphical user interface.

[0065] FIG. 6 shows a schematic diagram of invocation in different interfaces according to an embodiment of the present disclosure. In this embodiment, the user is playing the song AAA by using the end plug-in “XX music player”. If the user wants to view the translated content of AAA, the user can right-click AAA to select the option “translate by using the local translation application”. After the user selects the option, the target model application can be triggered automatically. The target model application sends “translate AAA” as a target request to the target model. The target model identifies that the end plug-in does not need to be invoked in the intent recognition, and generates the translated content as feedback. The target model application receives the feedback based on the long link and presents the translated content in the language user interface. In this embodiment, the language user interface is invoked based on the graphical user interface.

[0066] In some embodiments, a floating component is arranged on the graphical user interface of the end plug-in. If the floating component is triggered, a language user interface is presented, and the language user interface is used to receive data of multiple modalities. If input data is received from the language user interface, the input data is taken as a target request. In this embodiment, the interface of the target model application can be integrated on the end plug-in to invoke the target model.

[0067] The method of the present disclosure can be used to invoke a variety of end plugins. In some embodiments, the end plugin can be a photo album application. In this embodiment, a user can input a target request "I went to the beach today and took a lot of interesting videos, help me make a video." The target model can then generate an invocation command that invokes the end plugin "photo album", and the computing device 120 receives the invocation command based on the long link service and the preset communication protocol and can generate an execution code according to the invocation command to execute the "photo album" to read the pictures and videos taken on the day from the "photo album" and return them as execution results to the target model, and the target model makes a video from the read pictures and videos and provides it as feedback to the computing device 120.

[0068] In some embodiments, the end plugin can be a document application. In this embodiment, a user can input a target request "write an article about CCC." The target model can then generate an invocation command that invokes the end plugin document application, and the computing device 120 receives the invocation command based on the long link service and the preset communication protocol and can generate an execution code according to the invocation command to execute the document application to create a new document. At the same time, the invocation command contains specific document content, so the computing device 120 can write the received document content into the created document during the execution process as part of the feedback, and return the success or failure of the execution to the target model as the execution result, and the target model generates another part of the feedback (e.g. "creation success") according to the execution result and provides it to the computing device 120.

[0069] FIG. 7A illustrates a schematic diagram of invoking an end plugin to collect data according to an embodiment of the present disclosure. In FIG. 7A, the communication between the client 710 and the target model 720 is mainly embodied, which is based on the long link service. The long link service part is omitted to show the interaction logic between the client 710 and the target model 720. In this embodiment, the user can interact with the target model in the language user interface. In this embodiment, the client 710 sends a request to the target model. FIG. 7B illustrates a schematic diagram of a method for invoking an end plugin according to an embodiment of the present disclosure, which can be, for example, “please tell me the schedule for next month” in FIG. 7B, and sends the registration information for the end plugin to the client. The target model 720 discovers that the schedule data in the end plugin “schedule application” needs to be invoked through intent recognition, then generates an invocation request and an invocation command and sends them to the client 710. The client 710 starts the end plugin “schedule application” according to the invocation command in combination with the function function of the end plugin “schedule application” registered in the system, and reads the schedule data therefrom. In this process, the user can not be aware of these operations. The client 710 performs the invocation successfully after listening to the schedule data within the listening time period specified in the invocation command, and sends the invocation success and the obtained schedule data as the execution result to the target model 720. The target model 720 processes the schedule data into a schedule summary and sends it to the client 710 as feedback. FIG. 7B shows the schedule summary, which includes two parts, namely, a text summary 730 and an image summary 730. The text summary 730 is “next month, 3rd to A city, 6th to B city, 10th to C city, the schedule is as follows”. The image summary 740 is a trajectory graph from A city to B city and then to C city.

[0070] FIG. 8 illustrates a flowchart of a method for invoking an end plugin according to an embodiment of the present disclosure. In block 802, a target request is received, the target request involving invoking a target end plugin. The target request can be received by the computing device 110 from the computing device 120 via the long link service. In block 804, an invocation request is generated according to the target request using a target model, the invocation request indicating invoking the target end plugin. For example, the target model can perform intent recognition on the target request, so as to know that the request needs to invoke an end plugin to complete the feedback. In block 806, the invocation request is sent to the long link service to generate an invocation command according to the invocation request using a preset communication protocol. The long link service can use the preset communication protocol to package the invocation request into the invocation command and send it to the computing device 120. In this embodiment, the target model can invoke the end plugin of the client without listening, so as to intelligently execute the corresponding function of the end plugin according to the request of the user, and expand the capability boundary of the target model.

[0071] FIG. 9 shows a schematic block diagram of an example apparatus 900 according to some embodiments of the present disclosure. The apparatus 900 can be implemented in software, hardware, or a combination of both. As shown in FIG. 9, the apparatus 900 includes a first sending module 910, a first receiving module 920, and a performing module 930.

[0072] In some embodiments, the first sending module 910 can send a target request to a target model, the target request involving invoking a target-side plug-in. The first receiving module 920 can receive, from a long link service, an invoking command for the target-side plug-in according to a preset communication protocol, the invoking command being generated based on the invoking request received from the target model and the preset communication protocol. The performing module 930 can execute the target-side plug-in according to the invoking command.

[0073] The apparatus 900 of FIG. 9 can be used to implement the processes described above in connection with FIGS. 1-7B, which will not be repeated here for brevity.

[0074] FIG. 10 shows a schematic block diagram of an example apparatus 1000 according to some embodiments of the present disclosure. The apparatus 1000 can be implemented in software, hardware, or a combination of both. As shown in FIG. 10, the apparatus 1000 includes a second receiving module 1010, a generating module 1020, and a second sending module 1030.

[0075] In some embodiments, the second receiving module 1010 can receive a target request, the target request involving invoking a target-side plug-in. The generating module 1020 can generate an invoking request using a target model according to the target request, the invoking request indicating invoking the target-side plug-in. The second sending module 1030 can send the invoking request to a long link service to generate an invoking command according to the invoking request using a preset communication protocol.

[0076] The apparatus 1000 of FIG. 10 can be used to implement the processes described above in connection with FIG. 8, which will not be repeated here for brevity.

[0077] The division of modules or units in the embodiments of the present disclosure is illustrative only, and is merely a logical function division, and another division manner can be used in actual implementation. In addition, each functional unit in the disclosed embodiments can be integrated into one unit, or can be physically separated, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware, or in the form of a software functional unit.

[0078] FIG. 11 illustrates a block diagram of an example device 1100 that can be used to implement embodiments of the present disclosure. It should be understood that the device 1100 illustrated in FIG. 11 is merely an example and should not be construed to limit the functionality and scope of the implementations described herein. For example, the device 1100 can correspond to the computing device 120 described herein in connection with FIG. 1, and can be used to perform the processes of FIGS. 1-7B described above.

[0079] As shown in FIG. 11, the device 1100 is in the form of a general-purpose computing device. Components of the computing device 1100 can include, but are not limited to, one or more processors or processing units 1110, a memory 1120, a storage device 1130, one or more communication units 1140, one or more input devices 1150, and one or more output devices 1160. The processing unit 1110 can be a real or virtual processor and is capable of executing various processing in accordance with programs stored in the memory 1120. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of the computing device 1100.

[0080] The computing device 1100 typically includes a plurality of computer storage media. Such media can be any available media that is accessible by the computing device 1100 and includes both volatile and non-volatile media, removable and non-removable media. The memory 1120 can be volatile memory (such as a register, cache, or random access memory (RAM)), non-volatile memory (such as read only memory (ROM), electrically erasable programmable read only memory (EEPROM), or flash memory), or some combination thereof. The storage device 1130 can be a removable or non-removable media and can include machine-readable media such as a flash drive, magnetic disks, or any other media that can be used to store information and / or data (e.g., training data for training) and that can be accessed by the computing device 1100.

[0081] The computing device 1100 can further include additional removable / non-removable, volatile / non-volatile storage devices. Although not shown in FIG. 11, a floppy disk drive for reading from or writing to a removable, non-removable, and volatile memory disk (e.g., a "floppy disk"), and an optical disk drive for reading from or writing to a removable, non-removable, and volatile memory disk (e.g., a "CD-ROM" or "DVD") can be provided. In these instances, each drive can be connected to the bus (not shown) by one or more data media interfaces. The storage device(s) 1120 can include a computer program product 1125 having one or more program modules configured to carry out the various methods or actions of the implementations of the present disclosure.

[0082] The communication unit 1140 enables communications with other computing devices over a communication medium. Additionally, the functionality of the components of the computing device 1100 can be implemented in a single computing cluster or multiple computer machines that are capable of communicating over a communication connection. Thus, the computing device 1100 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network nodes in the networking environment.

[0083] The input device(s) 1150 can be one or more input devices such as a mouse, a keyboard, a trackball, etc. The output device(s) 1160 can be one or more output devices such as a display, a speaker, a printer, etc. The computing device 1100 can also include, as desired, other input devices (not shown), such as a microphone, a joystick, etc., and other output devices (not shown), such as a speaker, a printer, etc. The computing device 1100 can further communicate with one or more external devices (not shown) such as a storage device, a display device, etc. through the communication unit 1140, with one or more devices that enable a user to interact with the computing device 1100, or with any devices (e.g., a network card, a modem, etc.) that enable the computing device 1100 to communicate with one or more other computing devices. Such communication can be enabled by an Input / Output (I / O) interface (not shown).

[0084] According to example implementations of the present disclosure, a computer readable storage medium is provided having computer executable instructions stored thereon, where the computer executable instructions are executed by a processor to implement the methods described above. According to example implementations of the present disclosure, a computer program product is also provided that is tangibly stored on a non-transitory computer readable medium and includes computer executable instructions, where the computer executable instructions are executed by a processor to implement the methods described above. According to example implementations of the present disclosure, a computer program is provided having computer executable instructions that, when executed by a processor, implement the methods described above.

[0085] The computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include a non-transitory computer readable storage medium that can be a computer- readable storage medium having no data, programs, program modules, e.g., instructions for operation, or digital content stored thereon or therein for a short time or not at all.

[0086] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include a non-transitory computer readable storage medium that can be a computer- readable storage medium having no data, programs, program modules, e.g., instructions for operation, or digital content stored thereon or therein for a short time or not at all.

[0087] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include a non-transitory computer readable storage medium that can be a computer- readable storage medium having no data, programs, program modules, e.g., instructions for operation, or digital content stored thereon or therein for a short time or not at all.

[0088] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable data processing apparatus or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0089] Having described various implementations of the disclosure above, the descriptions are not exhaustive and do not limit the disclosure to the disclosed implementations. Numerous modifications and adaptations thereof will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The scope of the disclosure includes all of the implementations described above and equivalents thereto. Language used in this specification should not be used to limit the scope of the claims.

Claims

1.A method for invoking a client-side plugin, comprising: sending a target request to a target model, the target request relating to invoking a target client-side plugin; receiving an invoking command for the target client-side plugin from a long link service according to a preset communication protocol, the invoking command being generated based on the invoking request received from the target model and the preset communication protocol; and executing the target client-side plugin according to the invoking command. 2.The method of claim 1, wherein the target client-side plugin is pre-registered, the registration information for the target client-side plugin including a name and a description of the target client-side plugin, a set of function functions for the target client-side plugin, and a set of parameters for the set of function functions, and the method further comprises: sending the registration information for the target client-side plugin to the long link service using the preset communication protocol. 3.The method of claim 2, wherein the invoking command includes the name of the target client-side plugin, a requested function function for the target client-side plugin, and a parameter for the requested function function, the requested function function being at least a part of the set of function functions for the target client-side plugin. 4.The method of claim 3, wherein executing the target client-side plugin according to the invoking command comprises: parsing the invoking command based on the preset communication protocol to obtain the name of the target client-side plugin, the requested function function, and the parameter; generating an execution code for the target client-side plugin based on the name of the target client-side plugin, the requested function function, and the parameter; and executing the target client-side plugin according to the execution code for the target client-side plugin. 5.The method of claim 4, wherein the invoking command further includes a listening time period, and parsing the invoking command based on the preset communication protocol further comprises: parsing the invoking command to obtain the listening time period; and executing the target client-side plugin according to the execution code for the target client-side plugin further comprises: listening to the execution of the target client-side plugin for the listening time period. 6.The method of claim 5, wherein listening to the execution of the target client-side plugin for the listening time period comprises: acquiring an execution result for the target client-side plugin for the listening time period; and the method further comprises: sending the execution result for the target client-side plugin to the long link service; and receiving a feedback for the target request and presenting the feedback, wherein the target model generates the feedback based on the execution result. 7.The method of claim 6, wherein the execution result includes data acquired from the target client-side plugin, the client-side plugin registration information includes registration information for a second client-side plugin, and the method further comprises: receiving a second invoking command, wherein the second invoking command is determined by the target model for the target request according to the execution result and the client-side plugin registration information; executing the second client-side plugin according to the second invoking command; acquiring a second execution result for the second client-side plugin; and ​ ​ ​ ​ ​ sending, to the long link service, a second execution result for the second end plug-in; and wherein the target model generates the feedback based on the execution result and the second execution result. 8.The method of claim 4, wherein executing the target end plug-in according to the execution code for the target end plug-in comprises: in response to the execution code being located at a native end and the target end plug-in being at a web end, invoking the target end plug-in according to the execution code through a first channel; and in response to the execution code being located at a web end and the target end plug-in being at a native end, invoking the target end plug-in according to the execution code through the first channel. 9.The method of claim 1, wherein the target end plug-in has a first graphical user interface, the method further comprising: presenting a language user interface, wherein the language user interface is configured to receive data of a plurality of modalities; in response to receiving input data from the user from the language user interface, taking the input data as the target request; and in response to the target end plug-in being invoked, presenting the first graphical user interface. 10.The method of claim 1, further comprising: presenting a second graphical user interface, wherein a plurality of function controls are disposed in the second graphical user interface, wherein the plurality of function controls comprise a target function control for the target end plug-in; and in response to the target function control in the second graphical user interface being triggered, determining the target request. 11.The method of claim 10, wherein a floating component is disposed on the second graphical user interface, the method further comprising: in response to the floating component being triggered, presenting a second language user interface, wherein the second language user interface is configured to receive data of a plurality of modalities; in response to receiving input data from the second language user interface, taking the input data as the target request. 12.A method for invoking an end plug-in, comprising: receiving a target request, the target request involving invoking a target end plug-in; generating, using a target model, an invocation request according to the target request, the invocation request indicating invoking the target end plug-in; and sending, to a long link service, the invocation request to generate an invocation command according to the invocation request using a preset communication protocol. 13.An electronic device, comprising: at least one processing unit; at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, cause the electronic device to perform the method according to any one of claims 1 to 12. 14.A computer readable storage medium having stored thereon a computer program, the computer program, when executed by a processor, implement the method according to any one of claims 1 to 12. 15.A computer program product having stored thereon a computer program, the computer program, when executed by a processor, implement the method according to any one of claims 1 to 12. ​

Images