Method for running application, computing device, and storage medium

Abstract

A method for running an application, comprising: if demand information of a user is received, determining at least one task about the demand information by means of a pre-trained artificial intelligence model; at least on the basis of the content of the task, by means of the pre-trained artificial intelligence model, determining at least one application for executing the task, execution parameters and modal information about interaction, so as to generate execution information about the task; on the basis of the generated execution information, determining whether running the at least one application requires a user to participate in an interaction event; if it is determined that the user is required to participate in the interaction event, on the basis of the determined modal information, sending interaction request information to the user by means of at least one modal type; and, in response to receiving interaction feedback information of the user, executing the task related to the received interaction feedback information. Also provided are a computing device for running an application and a storage medium.

Description

Method, computing device, and storage medium for running an application

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese Patent Application No. 202311491162.5 filed on November 9, 2023, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] Embodiments of the present invention generally relate to the field of artificial intelligence, and more particularly to a method, a computing device, and a storage medium for running an application. Background Art

[0004] With the development of computer technology and the widespread use of the internet, massive amounts of data and increased computing power have driven the development of artificial intelligence (AI), giving rise to big AI models. By learning and training on vast amounts of data, big AI models can understand human language and semantics and extract useful information from them. As a result, big AI models have gradually become popular and applied across various fields.

[0005] However, while large AI models possess certain comprehension, reasoning, and planning capabilities, and can understand human intent to a certain extent, they still lack ultimate execution capabilities. Existing automation execution tools, such as conventional automation scripts, cannot adjust parameters during execution and can only handle non-interactive automation. Furthermore, existing application execution tools are unable to promptly select the appropriate modality for interactive events within an application, enabling users to be informed and participate in the interaction, making it difficult to meet user needs in a timely manner.

[0006] In summary, the traditional method for running applications has the following shortcomings: when automatically running interactive application software, it is impossible for users to participate in interactive events in a timely manner, resulting in that user needs cannot be met.

[0007] Summary of the Invention

[0008] In response to the above problems, the present invention provides a method, a computing device, and a storage medium for running an application.

[0009] According to a first aspect of the present invention, a method for running an application is provided, comprising: in response to receiving user demand information, determining at least one task related to the demand information via a pre-trained artificial intelligence model; based at least on the content of the task, determining at least one application, execution parameters and modal information about the interaction for performing the task via a pre-trained artificial intelligence model to generate execution information about the task; based on the generated execution information, determining whether running at least one application requires user participation in an interaction event; in response to determining that user participation in an interaction event is required, sending interaction request information to the user through at least one modal type based on the determined modal information; and in response to receiving user interaction feedback information, executing tasks related to the received interaction feedback information.

[0010] According to a second aspect of the present invention, a computing device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor so as to enable the at least one processor to perform the method of the first aspect of the present invention.

[0011] In a third aspect of the present invention, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method of the first aspect of the present invention.

[0012] In some embodiments, determining at least one application for performing a task, execution parameters, and modal information about the interaction includes: for each interaction event, determining at least one modal type for sending interaction request information to the user and at least one modal type for receiving interaction feedback information from the user; in the same interaction event, the modal type for sending interaction request information to the user and the modal type for receiving interaction feedback information from the user are the same, different, or partially the same.

[0013] In some embodiments, based on the determined modal information, sending interaction request information to the user through at least one modality includes: through an interactive device used to perform interactive events, or through other devices communicatively connected to the interactive device, sensing the user's status, the working status of the interactive device and / or the spatial position relationship between the user and the interactive device, so as to determine the modal type used to send interaction request information to the user.

[0014] In some embodiments, determining at least one application for performing a task, execution parameters, and modal information about the interaction includes: determining the modal information for performing the task based on the modal types supported by the interactive device and / or other devices, the modal types including at least one of voice, text, image, video, vibration, touch, click, shake, and flash.

[0015] In some embodiments, executing a task related to the received interaction feedback information includes: determining one or more of the following based on the received user interaction feedback information in order to execute the task related to the received interaction feedback information: input information required by the application used to execute the task; modifying the parameters used to execute the task; adjusting the steps of executing the task; adjusting the priority between tasks; adjusting the time of executing the task; changing the application used to execute the task; and merging or canceling the execution of tasks.

[0016] In some embodiments, determining at least one application, execution parameters, and modal information about the interaction for performing a task includes: in response to determining that there are multiple tasks regarding requirement information, determining the priority between the multiple tasks and / or the event type of each task in the multiple tasks via a pre-trained artificial intelligence model; and based on the determined priority between the multiple tasks and / or the event type of each task, determining an execution channel for each task, the execution channel being related to the execution order of the tasks.

[0017] In some embodiments, determining at least one application, execution parameters, and modal information about the interaction for performing a task also includes: in response to not receiving interaction feedback information from the user within a predetermined response time after sending interaction request information to the user, determining an interaction request to be fed back; based on the interaction request to be fed back, determining a new execution channel and modality type for the interaction request to be fed back via a pre-trained artificial intelligence model; and in response to determining that there are multiple interaction requests to be fed back, determining the priority of each of the multiple interaction requests to be fed back, so as to merge the interaction requests to be fed back with the same priority among the multiple interaction requests to be fed back.

[0018] In some embodiments, the execution parameters include at least one of the following: the content of the task, the steps of the task, the execution order of the steps of the task, the dependency relationship between the steps of multiple tasks, the number of tasks, the execution priority between multiple tasks, the application required to run to execute the task, the interactive events in the task, the triggering rules of the interactive events in the task, the input information required for the interactive events, and the modal types supported by the interactive events.

[0019] In some embodiments, it also includes storing at least one of the following based on the execution process and / or execution results of the task: operation records of each step of task execution, execution result information, adopted modality information, execution channel information, execution priority information, interaction process information about interaction events, user status information, environmental information, time information, user input information of interaction feedback information, and user emotion information.

[0020] In some embodiments, some embodiments also include: determining whether a user's backtracking request is received based on the stored information; in response to determining that the user's backtracking request is received, providing the user with process and / or result information about the task execution, so as to obtain the user's evaluation information about the task execution; based on the stored information and the user's evaluation information about the task execution, generating training data for use in optimized training of the pre-trained artificial intelligence model.

[0021] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and other features, advantages and aspects of the embodiments of the present invention will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements.

[0023] FIG1 shows a schematic diagram of a system for implementing a method for running an application according to an embodiment of the present invention.

[0024] FIG2 shows a flowchart of a method for running an application according to an embodiment of the present invention.

[0025] FIG3 shows a flowchart of a method for determining at least one application for executing a task, execution parameters, and modality information about interaction according to an embodiment of the present invention.

[0026] FIG4 shows a flowchart of another method for determining at least one application for executing a task, execution parameters, and modality information about interaction according to an embodiment of the present invention.

[0027] FIG5 shows a flowchart of a method for backtracking interactive events according to an embodiment of the present invention.

[0028] FIG6 shows a logical architecture diagram for implementing the method provided by an embodiment of the present invention according to an embodiment of the present invention.

[0029] FIG7 shows a block diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The following description of illustrative embodiments of the present invention is provided in conjunction with the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding. These details should be considered illustrative only. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0031] As used herein, the term "comprising" and its variations represent open inclusion, i.e., "including but not limited to." Unless otherwise stated, the terms "or" represent "and / or." The term "based on" represents "based at least in part on." The terms "an example embodiment" and "an embodiment" represent "at least one example embodiment." The term "another embodiment" represents "at least one additional embodiment." The terms "first," "second," and the like may refer to different or identical objects. Other explicit and implicit definitions may also be included below.

[0032] As described above, the traditional automation execution tools and the traditional methods for running applications have the following shortcomings: when automatically running interactive application software, users cannot be allowed to participate in interactive events in a timely manner, resulting in user needs not being met.

[0033] In addition, when existing solutions automatically run interactive software, after introducing large AI models, the uncertainty of the output of large AI models can easily lead to further execution deviations, causing the execution results to deviate from user needs.

[0034] In order to at least partially solve one or more of the above-mentioned problems and other potential problems, an exemplary embodiment of the present invention proposes a solution for running an application. In the solution of the present invention, by receiving user demand information, at least one task related to the demand information is determined via a pre-trained artificial intelligence model; based at least on the content of the task, at least one application, execution parameters and modal information about the interaction for performing the task are determined via a pre-trained artificial intelligence model to generate execution information about the task; thereby, user needs can be understood and converted into tasks.

[0035] In addition, the present invention also determines whether the operation of at least one application requires user participation in an interactive event based on the generated execution information; in response to determining that the user participation in the interactive event is required, based on the determined modal information, sends interaction request information to the user through at least one modal type; and in response to receiving the user's interactive feedback information, executes a task related to the received interactive feedback information. Thus, based on the acquired task information, when the interactive event is triggered by running the application to execute the task, the user is notified through multiple modalities to participate in the interaction and the user's interactive feedback information is received, so as to better run the application based on the user's intention to complete the task and meet the user's needs. Therefore, the present invention can facilitate the user to participate in the interactive event in a timely manner through multiple modalities when automatically running the interactive application software, so as to complete the execution of the task and meet the user's needs.

[0036] In addition, the present invention can also store at least information about execution steps, results and modalities, interaction process information of interactive events, user status information, etc. when automatically running interactive application software, so as to generate training sample data to facilitate the optimization of AI models, thereby obtaining execution results that better meet user needs; and the above-mentioned various types of stored data can also be used for backtracking supervision to improve the security of the system, and can also facilitate users to understand the automatic execution process to confirm the task execution status and results.

[0037] FIG1 shows a schematic diagram of a system 100 for implementing a method for running an application according to an embodiment of the present invention. As shown in FIG1 , system 100 includes a computing device 110, an interactive device 130, a network 140, and other devices 150. Computing device 110, interactive device 130, other devices 150, a server 160, and a storage device 170 can exchange data via network 140 (e.g., the Internet). The interactive device 130 can also be communicatively connected to the computing device 110 and the storage device 170; and the other devices 150 can also be communicatively connected to the computing device 110, the storage device 170, and the interactive device 130.

[0038] Regarding the computing device 110, it is used, for example, to obtain user demand information from the interactive device 130; and to determine at least one task regarding the demand information via a pre-trained artificial intelligence model based on the obtained user demand information; at least based on the content of the task, determine via a pre-trained artificial intelligence model at least one application, execution parameters and modal information about the interaction for performing the task to generate execution information about the task; based on the generated execution information, determine whether running at least one application requires user participation in an interactive event; and in response to determining that user participation in an interactive event is required, the computing device 110 sends interaction request information to the user through at least one modal type supported by the interactive device 130 or other device 150 based on the determined modal information; and in response to receiving interaction feedback information from the user, execute tasks related to the received interaction feedback information.

[0039] The interactive device 130 can, for example, collect various input information about the user and, based on the collected input information, extract the user's natural language input information. The interactive device 130 can also provide the user's natural language input information to the computing device 110 to obtain user demand information indicated by the natural language input information. The interactive device 130 can also, based on instructions from the computing device 110, send interaction request information to the user via at least one modal type, or receive user-performed operations to obtain interaction feedback information and provide the interaction feedback information to the computing device 110. The interactive device 130 can also obtain, notify, and display the results of task execution to the user.

[0040] The interactive device 130 may include but is not limited to at least one of the following: a mobile terminal, a mobile phone, a laptop computer, a tablet computer, a PDA, a desktop computer, an intelligent voice interactive device, a smart home appliance, a smart bracelet, a smart watch, a touch screen, an AR device, etc.

[0041] Other devices 150. The system may include multiple other devices 150. These devices at least have data acquisition capabilities, such as image acquisition, sound acquisition, location acquisition, and gravity sensing. Furthermore, these other devices 150 can communicate with the interactive device 130 and the computing device 110. Interactive devices 130 may include, but are not limited to, at least one of the following: a camera, a webcam, a microphone, an infrared sensor, a locator, a smart bracelet, a smart watch, a smart appliance, a desktop computer, a laptop computer, a mobile terminal, an AR device, and the like.

[0042] Storage device 170, for example, a cloud storage device, may also be integrated with interactive device 130, or may be partially integrated with interactive device 130 and partially deployed as a cloud storage device. Storage device 170 may be communicatively connected to interactive device 130, other devices 150, computing device 110, and server 160, or may interact with device data in the scene via network 140.

[0043] The server 160 may be deployed with a pre-trained artificial intelligence model for providing services to the user device 150 , or the pre-trained artificial intelligence model may be partially or fully deployed on the user device 150 .

[0044] The computing device 110 may have one or more processing units, including specialized processing units such as GPUs, FPGAs, and ASICs, as well as general-purpose processing units such as CPUs. Furthermore, one or more virtual machines may also be running on each computing device 110. In some embodiments, the interactive device 130 is integrated with the computing device 110, separate from the interactive device 130, or partially integrated. For example, the computing device 110 is a processing unit integrated within the interactive device 130; for example, a portion of the computing device 110 is integrated with the interactive device 130, while a portion is deployed on a cloud server, with the two working together to implement the solutions provided in the embodiments of the present invention.

[0045] In some embodiments, the computing device 110 includes, for example, a task generation module 112 , an execution information determination module 114 , an interaction event determination module 116 , an interaction request module 118 , and a task execution module 120 .

[0046] The task generating module 112 is configured to determine at least one task related to the demand information via a pre-trained artificial intelligence model in response to receiving the demand information of the user.

[0047] Regarding the execution information determination module 114, it is used to determine at least one application, execution parameters and modal information about the interaction for executing the task based on at least the content of the task via a pre-trained artificial intelligence model to generate execution information about the task.

[0048] Regarding the interactive event determination module 116, it is used to determine whether running at least one application requires user participation in an interactive event based on the generated execution information.

[0049] Regarding the interaction request module 118, it is used to send interaction request information to the user through at least one modality type based on the determined modality information in response to determining that the user needs to participate in the interaction event.

[0050] The task execution module 120 is configured to execute a task related to the received interaction feedback information in response to receiving the interaction feedback information from the user.

[0051] [Corrected 10.01.2025 in accordance with Rule 91] Figure 2 illustrates a flowchart of a method 200 for running an application according to an embodiment of the present invention. Method 200 may be executed by computing device 110, as shown in Figure 1 , or by electronic device 700, as shown in Figure 7 , or by interactive device 130, as shown in Figure 1 . It should be understood that method 200 may include additional steps not shown and / or may omit steps shown, and the scope of the present invention is not limited in this respect.

[0052] In step 202 , upon receiving user demand information, the computing device 110 determines at least one task related to the demand information via a pre-trained artificial intelligence model.

[0053] Regarding user demand information, for example, various input information about the user is collected through the interactive device 130 and other devices 150, and the various input information includes at least voice information, text information, image information and video information; based on the collected various input information, the user's natural language input information is extracted through voice, text and image recognition technologies, so as to understand the user's intention and obtain the user's demand information.

[0054] For example, the user inputs a voice message A to the interactive device 130, "Send an email to Li Si, saying that I will meet him at the coffee shop we often go to this Sunday afternoon at 3 pm, and please help me check where the SF Express delivery is." The interactive device 130 sends the demand information in the voice message to the computing device 110, or directly sends the voice message to the computing device 110. The interactive device 130 or the computing device 110 can identify the intention expressed by the user input information by calling the AI ​​model to obtain the user's demand information.

[0055] Regarding determining at least one task regarding demand information, continuing with the above example, based on voice information A, task information A is generated {Task A1: Open the email client, the recipient is Li Si, and the email address is lisi@example.com; Task A2: Open the express client and query SF Express's express delivery; Task A3: Fill in the email content: "Hi, Li Si, how about we meet at the "Little Cafe" on Renmin Road where we often go at 3 pm this Sunday, that is, 3 pm on August 16, I look forward to your reply"; Task A4: Send an email}.

[0056] In this way, tasks related to the needs can be generated according to the user's demand information, and the number and type of tasks depend on the user's actual needs.

[0057] In step 204 , the computing device 110 determines at least one application, execution parameters, and modality information about the interaction for performing the task based on at least the content of the task via a pre-trained artificial intelligence model to generate execution information about the task.

[0058] Continuing with the above example, based on tasks A1-A4 included in task information A, it can be determined that at least two applications are required to execute them. A1, A3, and A4 are executed through email applications, and A2 is executed through express query applications, search engines, etc. For example, the recipient information, receiving address, and email content required to be filled in the emails in A1 and A3 are all execution parameters.

[0059] In some embodiments, determining at least one application for performing a task, execution parameters, and modal information about the interaction includes: determining the modal information for performing the task based on the modal types supported by the interactive device and / or other devices, the modal types including at least one of voice, text, image, video, vibration, touch, click, shake, and flash.

[0060] For example, in Task A1 in the above example, if the other device 150 is a smartwatch, the request information can be sent to the user through the smartwatch's vibration, prompt, display, and other functions. When the user provides feedback, they can use the other device 150 or the interactive device 130. This solution supports multiple modal types across multiple devices, allowing them to participate in interactive events and fully utilize device resources.

[0061] At step 206 , the computing device 110 determines whether running the at least one application requires user participation in an interaction event based on the generated execution information.

[0062] For example, in the above example, of the four tasks contained in task information A, A1, A2, and A3 require user participation in interactive events, while A4 does not require user participation in interactions. After the user confirms the email recipient, recipient address, and email content based on A1 and A3, the tasks can be automatically executed without the user's participation in the interaction.

[0063] For example, the computing device 110 can also generate historical feedback information about the user based on feedback from each interaction event the user participates in, so that when the same or similar interaction event is triggered, the interaction event can be automatically executed based on the historical feedback information. The computing device 110 can also call a pre-trained artificial intelligence model based on the historical feedback information to generate execution parameters for the current interaction event, thereby simulating the user's participation in the interaction event. For example, after completing Task A1 in the above example, the user confirms Li Si and Li Si's email address through the interaction event. Based on the user's subsequent request information, Task B1 {Task B1: Send an email to Li Si} is generated. At this time, when executing Task B1, based on the historical feedback information, the user does not need to participate in the interaction event again. It should be understood that if there are two Li Si in the user's address book when executing Task B1, the interaction event will still need to be triggered, and the user will determine which "Li Si" to send the email to and determine the email address.

[0064] Therefore, the solution provided by the embodiment of the present invention can gradually understand the user's preferences, intentions and usage habits as the user uses it, thereby simulating the user's participation in interactive events and further simplifying the user's operations.

[0065] In step 208 , if the computing device 110 determines that the user needs to participate in the interaction event, based on the determined modality information, interaction request information is sent to the user through at least one modality type.

[0066] In some embodiments, determining at least one application for performing a task, execution parameters, and modal information about the interaction includes: for each interaction event, determining at least one modal type for sending interaction request information to the user and at least one modal type for receiving interaction feedback information from the user; in the same interaction event, the modal type for sending interaction request information to the user and the modal type for receiving interaction feedback information from the user are the same, different, or partially the same.

[0067] Continuing with the above example, based on tasks A1-A4 included in task information A, for example, tasks A1, A2, and A3 all require user interaction; regarding the sending interaction request information for task A1, it is necessary to determine a modal type for notifying the user of the email address and recipient entered, such as by calling the display screen of interactive device 130 to display these contents to the user; regarding the interactive feedback information for task A1, the user's interactive feedback information can be received through modalities such as voice, touch screen clicks, etc., such as the user clicking to confirm based on the information on the display screen, or voice inputting "confirm". Regarding the sending interaction request information for task A2, for example, a voice prompt message "You have multiple SF Express deliveries recently. Which one do you want to check?" is sent to the user through voice modality.

[0068] It is worth noting that in sending interaction requests to users and receiving interaction feedback information from users, this solution can support the use of different modal types, making full use of the information collection or information sending capabilities of the interactive device 130 and other devices 150, thereby supporting multi-modal interaction requests and feedback, and facilitating efficient acquisition of user feedback.

[0069] In step 210 , if the computing device 110 receives interaction feedback information from the user, a task related to the received interaction feedback information is executed.

[0070] Continuing with the above example, when executing task A2, based on the interaction request information sent, the user's interaction feedback information A21 "check the express sent to Teacher Li" is received, and based on the received interaction feedback information A21, the task is continued to be executed.

[0071] For example, based on the determined modal information, computing device 110 sends the task execution result to the user via at least one modal type. Continuing with the above example, if a query finds that there is only one recipient with the surname "Li" in recent SF Express delivery records, the logistics information of this delivery is used as the task execution result. The task execution result can then be fed back to the user by selecting a modal type, such as a voice prompt that reads, "The SF Express delivery you sent to Mr. Li has arrived in Nanjing and is expected to be delivered tomorrow."

[0072] In the above-mentioned scheme, the scheme provided by the embodiment of the present invention can generate tasks related to the needs based on the user's needs, and match the corresponding applications to perform the tasks. When the interaction event is triggered when the application is automatically run, the various modal types supported by the interactive device and other devices can be used to facilitate user participation in the interaction, so as to collect interaction feedback information and understand the user's intentions, thereby completing the execution of the task efficiently and with high quality to meet the user's needs.

[0073] In some embodiments, the execution parameters include at least one of the following: the content of the task, the steps of the task, the execution order of the steps of the task, the dependency relationship between the steps of multiple tasks, the number of tasks, the execution priority between multiple tasks, the application required to run to execute the task, the interactive events in the task, the triggering rules of the interactive events in the task, the input information required for the interactive events, and the modal types supported by the interactive events.

[0074] In some embodiments, based on the determined modal information, sending interaction request information to the user through at least one modality includes: through an interactive device used to perform interactive events, or through other devices communicatively connected to the interactive device, sensing the user's status, the working status of the interactive device and / or the spatial position relationship between the user and the interactive device, so as to determine the modal type used to send interaction request information to the user.

[0075] Regarding perception, for example, the user's exercise state (such as running, playing ball), the usage state of the user and the interactive device (such as the user is looking at the screen of the interactive device, such as the user is on a call), the working state of the interactive device (such as playing music, playing videos, recording videos, etc.), the spatial position relationship between the user and the interactive device (such as the interactive device is not next to the user, the interactive device is in the user's pocket), it should be understood that in addition to the above examples of "perception", there can be more types of user states, working states of interactive devices and / or spatial position relationships between users and interactive devices, which are determined by the specific circumstances and the interactive device, sensor devices, modal devices, etc. equipped with other devices.

[0076] Regarding the above-mentioned perception, it can be through the interactive device itself, such as a camera, gravity sensor, screen, fingerprint recognition, face recognition, gravity sensor, etc.; it can also be through devices connected to the interactive device, such as Bluetooth headsets, smart bracelets and watches, cameras, microphones, displacement sensors and other devices to perceive the above-mentioned various states.

[0077] For example, if the user is aware that they are playing a game, it is obviously not appropriate to send an interaction request via a display prompt. Vibration prompts, sound prompts, or even a sports watch worn by the user would be more appropriate. For example, if the user is aware that they are in a call, voice mode is not appropriate. Similarly, if the user is in a voice call on a handheld device, image or video mode is not appropriate either. For example, vibration mode may be used to send an interaction request to the user.

[0078] For example, when the computing device 110 is executing task A1 {Task A1: open the email client, the recipient is Li Si, and the email address is lisi@example.com}, it is assumed that the user's state is perceived to be looking at the screen; a prompt is generated based on the user state, the content to be interacted, the system state and other information, and the generated prompt is input into a predetermined artificial intelligence model; for example, the prompt is "Suppose you are an interactive system, your task is to combine the user state and use the most appropriate method to interact with the user and obtain user input; the content that needs to be interacted is [the user needs to confirm whether the email address is correct], the user state is [looking at the screen, watching a video], and the interaction methods you can use are as follows [pop up a dialog box, play a voice prompt, vibrate prompt], please select The most appropriate method and the content of the interaction should be output. "The output of the predetermined artificial intelligence model is as follows" At this time, a [pop-up dialog box] should be selected, with the content [Please confirm whether the following email address is correct?], and the display buttons are "Correct", "Incorrect", and "Talk later". The user input can be clicking a virtual button or using voice. If voice is obtained at this time, voice recognition is completed first, and then the recognition content is fed back to the predetermined artificial intelligence model. The artificial intelligence determines whether the expected user feedback has been obtained, and based on the user's interactive feedback information, continues the next process, such as continuing the task, or initiating the interaction request again if the user feedback information is insufficient to complete the task, or the user feedback information requires postponing or canceling the task, etc.

[0079] For example, when the computing device 110 is executing task A3, the user state it perceives is "the device is around the user, and the user's screen is not annotated." The user state, content to be interacted with, system state, etc. are used to generate prompts and input into a predetermined artificial intelligence model. For example, the prompt is "Suppose you are an interactive system. Your task is to combine the user state and use the most appropriate method to interact with the user and obtain user input. The content that needs to be interacted with is [the user needs to confirm whether the email content is appropriate], and the user state is [the device is around the user, and the screen is not annotated]. The interaction methods you can use are as follows [pop up a dialog box, play a voice prompt, and vibrate a prompt]. Please select the most appropriate method and output the content of the interaction." The artificial intelligence model output is such as "You should choose [Play Voice] at this time, and the content is [Please confirm whether the content of the following email is correct. Do you need it to be read aloud for you?]." For example, the user uses the voice mode and sends the feedback information "Please read aloud". After the voice recognizes the user's intention, it drives the player in the interactive device 130 or other devices to read the email content aloud; the user uses voice feedback "Content confirmed, please change the address from Li Si to Lao Li", and updates the email content based on the feedback information obtained.

[0080] Therefore, the solution provided by the embodiment of the present invention can make full use of the perception capabilities of the interactive device and other devices to perceive the user status, the working status of the interactive device and / or the spatial position relationship between the user and the interactive device, so as to determine the modality that is more suitable for the current user from multiple modal types based on the above-mentioned multiple perception status information, so that the user can receive interaction request information in a timely manner, which facilitates the user to participate in interaction events.

[0081] In some embodiments, executing a task related to the received interaction feedback information includes: determining one or more of the following based on the received user interaction feedback information in order to execute the task related to the received interaction feedback information: input information required by the application used to execute the task; modifying the parameters used to execute the task; adjusting the steps of executing the task; adjusting the priority between tasks; adjusting the time of executing the task; changing the application used to execute the task; and merging or canceling the execution of tasks.

[0082] For example, taking the task information A in the above example, task A1 {Task A1: open the email client, the recipient is Li Si, and the email address is lisi@example.com}, when the user interacts and provides feedback, the feedback information sent is "Send Li Si a date for tomorrow directly via WeChat." Based on the above feedback information received, the original execution application of task A1 is replaced with the application "WeChat" to execute A1. At the same time, the corresponding content and information in tasks A3 and A4 are also adjusted.

[0083] For example, taking the task information A in the above example, the interactive feedback information received by task A1 is "Change the time to Saturday", then the content of the email required to be entered when executing task A3 needs to be adjusted; for example, the query result fed back to the user by task A2 {Task A2: Open the express client and query SF Express's express} is "Your SF Express is not available at the moment", and the interactive feedback information received from the user is "Please help me check the express information later", then the execution time of task A2 will be adjusted to "later", and task A2 can be executed again in a few hours depending on the specific situation.

[0084] Therefore, the solution provided by the embodiment of the present invention can adjust the execution time, execution method, execution object, execution application used, etc. of the task based on the user's interactive feedback information and the various information received, and make multi-faceted adjustments to respond to the user's new needs in a timely manner.

[0085] [Corrected 10.01.2025 in accordance with Rule 91] Figure 3 illustrates a flow chart of a method 300 for determining at least one application for performing a task, execution parameters, and modality information regarding interaction, according to an embodiment of the present invention. Method 300 may be executed by computing device 110, as shown in Figure 1 , or by electronic device 700, as shown in Figure 7 , or by interaction device 130, as shown in Figure 1 . It should be understood that method 300 may include additional steps not shown and / or may omit steps shown, and the scope of the present invention is not limited in this respect.

[0086] In step 302 , if it is determined that there are multiple tasks regarding the requirement information, the computing device 110 determines priorities among the multiple tasks and / or an event type of each of the multiple tasks via a pre-trained artificial intelligence model.

[0087] In step 304 , the computing device 110 determines an execution channel for each task based on the determined priorities among the multiple tasks and / or the event type of each task, where the execution channel is related to the execution order of the tasks.

[0088] Regarding the event type of the task, such as the time urgency of the task, the importance of the task, the dependency relationship between tasks, etc., for example, in the above example, task A3 depends on the execution and completion of task A1, and task A4 depends on the completion of tasks A1 and A3.

[0089] For example, continuing with the above task information A, it is determined that tasks A1, A2, and A3 require user participation in interactive events, and the priority is determined to be A1>A3>A2 (i.e., confirming the email recipient information>confirming the email content>checking the express delivery information); based on the priority of the three tasks, A1 is assigned to channel 1, A2 is assigned to channel 3, and A3 is assigned to channel 4, and the execution of the tasks is completed with the channel priority of channel 1>channel 2>channel 3.

[0090] In this way, the priority of tasks can be adjusted based on the importance of each task with respect to user needs, the urgency of events, and the relevance of tasks.

[0091] [Corrected 10.01.2025 in accordance with Rule 91] Figure 4 illustrates a flowchart of another method 400 for determining at least one application for performing a task, execution parameters, and modality information regarding interaction, according to an embodiment of the present invention. Method 400 may be executed by computing device 110, as shown in Figure 1 , or by electronic device 700, as shown in Figure 7 , or by interaction device 130, as shown in Figure 1 . It should be understood that method 400 may include additional steps not shown and / or may omit steps shown, and the scope of the present invention is not limited in this respect.

[0092] In step 402 , if the computing device 110 does not receive interaction feedback information from the user within a predetermined response time after sending the interaction request information to the user, an interaction request to be fed back is determined.

[0093] In step 404 , the computing device 110 determines a new execution channel and modality type for the interaction request to be fed back based on the interaction request to be fed back via a pre-trained artificial intelligence model.

[0094] For example, when executing task A2, assuming that it is perceived that "the interactive device is in the user's pocket", the computing device 110 generates a prompt based on the user status, the content to be interacted with, the system status, etc., and inputs a predetermined artificial intelligence model, such as the prompt "Suppose you are an interactive system. Your task is to combine the user status and use the most appropriate method to interact with the user and obtain user input. The content that needs to be interacted with is [The user needs to confirm whether the target express information is correct], the user status is [The device is in the user's pocket], and the interaction methods you can use are as follows [Pop up a dialog box, play a voice prompt, and vibrate a prompt]. Please select the most appropriate method and output the content of the interaction." For example, the predetermined artificial intelligence model outputs "At this time, you should use [Vibration prompt], and the content is [Please confirm whether the following target express information is correct]" The predetermined feedback time is to wait for 1 minute. If no user feedback is received after 1 minute of sending the interaction request, the interaction request sent in the above task A2 is determined as the interaction request A2f awaiting feedback.

[0095] Continuing with the above example, for example, a new execution channel and modal type are determined for the interaction request A2f to be fed back. For example, the new execution channel is determined to be channel 2, the modal type is "vibration prompt and voice prompt", and the execution parameters include "vibrate for 2 seconds every 5 seconds, last for 1 minute, with an interval of minutes, and repeat for half an hour; the voice is played every 30 seconds and paused after 30 minutes".

[0096] In step 406 , in response to determining that there are multiple interaction requests to be fed back, the computing device 110 determines the priority of each of the multiple interaction requests to be fed back, so as to merge the interaction requests to be fed back with the same priority among the multiple interaction requests to be fed back.

[0097] For example, if there is no interactive feedback from the user for a long time and several interactive requests are queued, the interactive requests with the same priority or specified priority in the queue can be merged to generate a new prompt and input into the predetermined artificial intelligence model. For example, the prompt word is "Suppose you are an interactive system. Your task is to combine the user status and adopt the most appropriate method to interact with the user and obtain user input. There are currently multiple contents that require interaction. Please merge them and select the most appropriate interaction method; the user status is [looking at the screen, watching a video], and the interaction methods you can use are as follows [pop-up dialog box, play voice prompt, vibration prompt], and the tasks to be processed are [confirm WeChat content; confirm payment information; confirm whether to accept the meeting invitation]"; for example, the artificial intelligence model output is "At this time, you should use [pop-up dialog box], the content of which is [Please confirm whether to perform the following operations, WeChat content is: xxx, pay Li Si 100 yuan, accept the meeting invitation], and the options provided on the interface are "All correct", "Confirm separately", and "Talk later".

[0098] Therefore, through the above solution, multiple accumulated interaction events can be merged and processed, and the parallel processing of multiple interaction events can be achieved, thereby improving processing efficiency.

[0099] [Corrected 10.01.2025 in accordance with Rule 91] Figure 5 shows a flow chart of a method 500 for backtracking an interaction event according to an embodiment of the present invention. Method 500 may be executed by computing device 110 as shown in Figure 1 or by electronic device 700 as shown in Figure 7. It should be understood that method 500 may include additional steps not shown and / or may omit steps shown, and the scope of the present invention is not limited in this respect.

[0100] At step 502 , the computing device 110 determines whether a traceback request from a user is received based on the stored information.

[0101] In some embodiments, the computing device 110 and / or the interactive device 130 is also used to store at least one of the following based on the execution process and / or execution results of the task: operation records of each step of task execution, execution result information, adopted modality information, execution channel information, execution priority information, interaction process information about interaction events, user status information, environmental information, time information, user input information of interaction feedback information, and user emotion information.

[0102] For example, when the task is completed, the entire process of running the application will be stored, such as by recording the screen of the running process, saving the running log, etc. In addition, the user's communication process during the interaction time will also be stored. For example, in the above A2, when the user is watching a video, the user selects "Talk later" for the dialog box that pops up for the interaction, and the triggering timing of the interaction time or the interaction mode selection is unreasonable; or the emotional expressions such as "Talk later" and "Exit" used for voice feedback need to be stored. At the same time, after detecting a similar user dissatisfaction situation, it is necessary to store the environmental information, time information, etc. when the interaction event occurs to provide data for subsequent improvements.

[0103] In some embodiments, backtracking the execution process can be used for security supervision to improve the security of automatically executed tasks; for example, in a TOB (To Business, for enterprises or specific users) scenario, it is used for the enterprise to conduct security supervision of the execution process, for example, by backtracking operations, it is possible to determine whether the employee's operations have security violations; for example, in a TOC (To Customer, for end users) scenario, it can be used for system optimization. For example, if a user forgets that he has sent an email before and repeats the same instruction, in addition to feedback on the execution of the same task, the specific information of the same task can also be backtracked to provide the user with the specific execution status and results; it can also facilitate user recollection afterwards, for example, if a user remembers sending an email but wants to confirm whether it was sent to the correct object.

[0104] In step 504 , if the computing device 110 determines that a backtracking request from the user is received, the computing device 110 provides the user with information about the process and / or result of the task execution, so as to obtain the user's evaluation information about the task execution.

[0105] For example, in the above example task A1, if the user does not receive a reply from Li Si after sending the email, the user can verify whether the sending information is correct by backtracking. For example, the user's backtracking request can be sent through subscription, query, etc.

[0106] In step 506 , the computing device 110 generates training data based on the stored information and the user's evaluation information on task execution for use in optimized training of the pre-trained artificial intelligence model.

[0107] Therefore, by storing various data related to application operation, task execution and interaction events, it is possible to provide training data for the predetermined artificial intelligence model, so as to optimize the artificial intelligence model's modality selection, interaction time selection, task channel selection, etc., and the ability to generate task execution parameters, so as to meet user needs more efficiently.

[0108] In some embodiments, the interactive device 130 is a mobile terminal device such as a mobile phone, a tablet computer, etc., and the computing device 110 is at least partially a computing unit integrated in the interactive device 130, such as a CPU, a GPU, etc. The method for running an application provided in an embodiment of the present invention can be partially or completely deployed in the interactive device 130 in the form of software, programs, etc., such as being installed in its internal storage.

[0109] Please refer to Figure 6, which illustrates a logical architecture diagram for implementing methods 200-500 provided in an embodiment of the present invention. The interactive device 130 is configured with a control module 610, an application set 640, and multiple modal types. The application set 640 can provide multiple applications, and the multiple modal types are determined by the hardware and software of the interactive device 130; the storage module 660 can be configured in the memory of the interactive device, or in a cloud server or cloud storage; the artificial intelligence model 650 is a large AI model provided by a third party, or can be trained based on an AI model provided by a third party and then configured on a cloud server or locally on the interactive device; the multimodal feedback system 620 includes the interaction types that can be supported by the interactive device 130 and other devices 150.

[0110] Among them, the control module 610 can configure the priority of task execution through multiple channels 630, and control the corresponding modality to send interaction requests to the user or receive interaction feedback information from the user; the control module 610 can also generate prompts for the acquired user demand information, user input information, user feedback information, etc., to input into the artificial intelligence model 660, and based on the content output by the model 660, call the application in the application set 640 to execute the task, and call the modality type contained in the multimodal feedback system 620 through the channel 630 to complete the interaction event with the user.

[0111] The storage module 660 can be connected to the control module 660, the artificial intelligence model 650 and the application set 640 respectively to store various information related to application operation, task execution and interactive events.

[0112] In summary, the embodiments of the present invention have the following technical effects:

[0113] First, it can make the process of AI model calling applications transparent, reducing the distrust of users caused by the uncertainty of AI models.

[0114] Secondly, by adding interactive channels and multiple modes, the hit rate of AI model calling applications can be improved, which can be more in line with the user's intentions; it can also improve the customizability of AI model calling applications. Users can customize the parameters in the execution process according to their own needs, rather than mechanically executing the parameters after the AI ​​model generates them; it improves the user participation in the interactive link so that the execution results are more in line with user needs.

[0115] Then, full supervision and backtracking of the application operation and task execution process are added, which not only enhances user trust but also can be used as training samples for optimized training of AI models.

[0116] Fig. 7 shows a schematic step diagram of an example electronic device 700 that can be used to implement an embodiment of the present specification. For example, the computing device 110 as shown in Figure 1 can be implemented by the electronic device 700. As shown in the figure, the electronic device 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to the computer program instructions stored in a read-only memory (ROM) 702 or the computer program instructions loaded into a random access memory (RAM) 703 from a storage unit 708. In the random access memory 703, various programs and data required for the operation of the electronic device 700 can also be stored. The central processing unit 701, the read-only memory 702 and the random access memory 703 are connected to each other by a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0117] Multiple components in electronic device 700 are connected to input / output interface 705, including: input unit 706, such as a keyboard, mouse, microphone, etc.; output unit 707, such as various types of displays, speakers, etc.; storage unit 708, such as a magnetic disk, optical disk, etc.; and communication unit 709, such as a network card, modem, wireless communication transceiver, etc. The communication unit 709 allows device 700 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0118] The various processes and procedures described above, such as methods 200 to 500, may be performed by the central processing unit 701. For example, in some embodiments, the methods 200 to 500 may be implemented as a computer software program that is tangibly embodied in a machine-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed onto the device 700 via the read-only memory 702 and / or the communication unit 709. When the computer program is loaded into the random access memory 703 and executed by the central processing unit 701, one or more actions of the methods 200 to 500 described above may be performed.

[0119] The present invention relates to methods, apparatuses, systems, electronic devices, computer-readable storage media and / or computer program products. The computer program products may include computer-readable program instructions for executing various aspects of the present invention.

[0120] Computer-readable storage media can be a tangible device that can hold and store instructions used by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination thereof. The computer-readable storage media used herein is not to be interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by a fiber optic cable), or an electrical signal transmitted by a wire.

[0121] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge computing devices. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0122] The computer program instructions for performing the operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, the state information of the computer-readable program instructions is used to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), which may execute the computer-readable program instructions to implement various aspects of the present invention.

[0123] Various aspects of the present invention are described herein with reference to flowcharts and / or step diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each step of the flowcharts and / or step diagrams, and any combination of the steps in the flowcharts and / or step diagrams, can be implemented by computer-readable program instructions.

[0124] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine such that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more steps in the flowchart and / or step diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more steps in the flowchart and / or step diagram.

[0125] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more steps in the flowchart and / or step diagram.

[0126] The flowcharts and step diagrams in the accompanying drawings show the possible architectures, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each step in the flowchart or step diagram can represent a module, program segment or part of an instruction, and a module, program segment or part of an instruction contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the steps can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive steps can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each step in the step diagram and / or flowchart, and the combination of the steps in the step diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions.

Claims

1. A method for running an application, comprising: In response to receiving user requirement information, determining, via a pre-trained artificial intelligence model, at least one task regarding the requirement information; Determining, at least based on the content of the task, via the pre-trained artificial intelligence model, at least one application for executing the task, execution parameters, and modality information regarding interaction to generate execution information regarding the task; Based on the generated execution information, determining whether user participation in an interaction event is required to run the at least one application; In response to determining that user participation in an interaction event is required, based on the determined modality information, sending interaction request information to the user via at least one modality type; And In response to receiving user interaction feedback information, performing a task related to the received interaction feedback information.

2. The method according to claim 1, wherein Determining at least one application for executing the task, execution parameters, and modality information regarding interaction includes: Regarding each interaction event, determining at least one modality type for sending interaction request information to the user and at least one modality type for receiving user interaction feedback information. In the same interaction event, the modality type for sending interaction request information to the user and the modality type for receiving user interaction feedback information may be the same, different, or partially the same.

3. The method according to claim 1, wherein Sending interaction request information to the user via at least one modality based on the determined modality information includes: Perceiving the state of the user, the working state of the interaction device, and / or the spatial position relationship between the user and the interaction device through the interaction device for executing the interaction event or through other devices communicatively connected to the interaction device, so as to determine the modality type for sending interaction request information to the user.

4. The method according to any one of claims 3, wherein Determining at least one application for executing the task, execution parameters, and modality information regarding interaction includes: Based on the modality types supported by the interaction device and / or the other devices, determining the modality information for executing the task, where the modality types include at least one of voice, text, image, video, vibration, touch, click, shake, and flash.

5. The method according to claim 1, wherein, Performing a task related to the received interaction feedback information includes: Based on the received user interaction feedback information, determining one or more of the following to perform a task related to the received interaction feedback information: Input information required by the application for performing the task; Modifying the parameters for performing the task; Adjusting the steps for performing the task; Adjusting the priority between tasks; Adjusting the time for performing the task; Replacing the application for performing the task; and Merging or canceling the execution of the task.

6. The method according to claim 1, wherein, Determining at least one application for executing the task, execution parameters, and modality information regarding interaction includes: In response to determining that there are multiple tasks regarding the requirement information, via the pre-trained artificial intelligence model, determining the priority between the multiple tasks and / or the event type of each task in the multiple tasks; and Based on the determined priority between the multiple tasks and / or the event type of each task, determining the execution channel for each task, where the execution channel is related to the execution sequence of the task.

7. The method according to claim 6, wherein Determining at least one application, execution parameters, and modal information regarding the interaction for performing the task further includes: Determining an interaction request to be fed back in response to not receiving the user's interaction feedback information within a predetermined response time after sending the interaction request information to the user; Based on the interaction request to be fed back, determining a new execution channel and modal type for the interaction request to be fed back via the pre-trained artificial intelligence model; and In response to determining that there are multiple interaction requests to be fed back, determining the priority of each interaction request to be fed back among the multiple interaction requests to be fed back, so as to merge the interaction requests to be fed back with the same priority among the multiple interaction requests to be fed back.

8. The method according to any one of claims 1-7, wherein The execution parameters include at least one of the following: The content of the task, the steps of the task, the execution sequence of the steps of the task, the dependency relationship between the steps of multiple tasks, the number of tasks, the execution priority between multiple tasks, the applications required to run for performing the task, the interaction events in the task, the triggering rules of the interaction events in the task, the input information required by the interaction events, and the modal types supported by the interaction events.

9. The method according to any one of claims 1-7, further includes storing at least one of the following based on the execution process and / or execution result of the task: Operation records of each step of the task execution, execution result information, the modal information adopted, execution channel information, execution priority information, interaction process information regarding interaction events, user status information, environmental information, time information, user input information of the interaction feedback information, and user emotion information.

10. The method according to claim 9, further includes: Based on the stored information, determining whether a user's backtracking request is received; In response to determining that a user's backtracking request is received, providing the user with information regarding the execution process and / or result of the task, so as to obtain the user's evaluation information regarding the task execution; Based on the stored information and the user's evaluation information regarding the task execution, generating training data for use in the optimization training of the pre-trained artificial intelligence model.

11. A computing device, including: At least one processor; And a memory communicatively connected to the at least one processor; The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method according to any one of claims 1-10.

12. A non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the method according to any one of claims 1-10.

Images