Multimedia playing method and device, terminal and storage medium

Abstract

The embodiment of the invention discloses a multimedia playing method and device, a terminal and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: receiving an automatic multimedia playing instruction, wherein the automatic multimedia playing instruction is used for indicating to execute an automatic multimedia playing task; based on an automatic execution script corresponding to the multimedia automatic playing task, executing the multimedia automatic playing task in a task execution window running in the background; and in the execution process of the multimedia automatic playing task, displaying the real-time playing condition of the multimedia through a real-time activity reminding control.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a multimedia playback method, device, terminal, and storage medium. Background Art

[0002] When using a multimedia application for multimedia playback, the multimedia application must remain in the foreground. During multimedia playback, users need to manually operate the multimedia application based on the actual playback situation, such as manually pausing, manually resuming playback, and manually switching to the next segment. Summary of the Invention

[0003] The embodiments of the present application provide a multimedia playback method, device, terminal, and storage medium. The technical solution is as follows:

[0004] In one aspect, an embodiment of the present application provides a multimedia playback method, the method comprising:

[0005] receiving a multimedia automatic play instruction, wherein the multimedia automatic play instruction is used to instruct execution of a multimedia automatic play task;

[0006] Based on the automation execution script corresponding to the multimedia automatic playback task, executing the multimedia automatic playback task in the task execution window running in the background;

[0007] During the execution of the multimedia automatic playback task, the real-time playback status of the multimedia is displayed through the real-time activity reminder control.

[0008] On the other hand, an embodiment of the present application provides a multimedia playback device, the device comprising:

[0009] An instruction receiving module is used to receive a multimedia automatic play instruction, wherein the multimedia automatic play instruction is used to instruct the execution of a multimedia automatic play task;

[0010] An execution module, configured to execute the multimedia automatic playback task in a task execution window running in the background based on an automated execution script corresponding to the multimedia automatic playback task;

[0011] The display module is used to display the real-time playback status of the multimedia through a real-time activity reminder control during the execution of the multimedia automatic playback task.

[0012] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores at least one computer instruction, and the at least one computer instruction is used to be executed by a processor to implement the multimedia playback method as described in the above aspects.

[0013] On the other hand, an embodiment of the present application provides a computer program product, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium; a processor reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to implement the multimedia playback method described in the above aspects.

[0014] In an embodiment of the present application, when there is a need to automatically play multimedia, the terminal can run a task execution window in the background and execute the multimedia automatic playback task in the task execution window based on the automated execution script corresponding to the multimedia automatic playback task. Since the multimedia automatic playback task is automatically executed in the task execution window running in the background, the user's manual operation process is eliminated and other applications running in the foreground are not affected. In addition, during the execution of the multimedia automatic playback task, the terminal displays the real-time playback status of the multimedia through a real-time activity reminder control, which reduces interference with the foreground application while improving the user's perception of the real-time execution status of the task. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A schematic diagram of the system architecture of an exemplary embodiment of the present application is shown;

[0016] Figure 2 A flowchart of a multimedia playback method provided by an exemplary embodiment of the present application is shown;

[0017] Figure 3 This is a schematic diagram of an interface for triggering and executing a multimedia automatic playback task according to an exemplary embodiment of the present application;

[0018] Figure 4 This is a schematic diagram of an interface for triggering and executing a multimedia automatic playback task according to another exemplary embodiment of the present application;

[0019] Figure 5 is a flowchart of a multimedia pause state processing process shown in an exemplary embodiment of the present application;

[0020] Figure 6 This is a schematic diagram illustrating an implementation of a pause and resume process according to an exemplary embodiment of the present application;

[0021] Figure 7 is a schematic diagram illustrating an implementation of a manual intervention process according to an exemplary embodiment of the present application;

[0022] Figure 8 This is a schematic diagram illustrating an implementation of an abnormal recovery process according to an exemplary embodiment of the present application;

[0023] Figure 9is a flow chart illustrating a simulation operation execution process according to an exemplary embodiment of the present application;

[0024] Figure 10 is a flowchart illustrating an exception handling process according to an exemplary embodiment of the present application;

[0025] Figure 11 This is a flowchart of a process in which a simulation operation execution engine executes an automated execution script, as shown in an exemplary embodiment of the present application;

[0026] Figure 12 This is a flowchart of a process from creation to cancellation of a task execution window shown in an exemplary embodiment of the present application;

[0027] Figure 13 is a schematic diagram illustrating a form transformation process of a real-time activity reminder control according to an exemplary embodiment of the present application;

[0028] Figure 14 This is a schematic diagram of a strong reminder process of a real-time activity reminder control according to an exemplary embodiment of the present application;

[0029] Figure 15 is a schematic diagram illustrating a form transformation process of a real-time activity reminder control according to another exemplary embodiment of the present application;

[0030] Figure 16 This is a schematic diagram illustrating a process of bringing a task execution window back to the foreground for display according to an exemplary embodiment of the present application;

[0031] Figure 17 This is an interaction sequence diagram of an adapter, a simulation operation execution engine, a real-time activity reminder control controller, and a window controller according to an exemplary embodiment of the present application;

[0032] Figure 18 A structural block diagram of a multimedia playback device provided by an exemplary embodiment of the present application is shown;

[0033] Figure 19 The figure shows a structural block diagram of a terminal provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0035] In this document, "plurality" refers to two or more. "And / or" describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates an "or" relationship between the associated objects.

[0036] Please refer to Figure 1 , which shows a schematic diagram of a system architecture shown in an exemplary embodiment of the present application. The system architecture includes a cloud side 110 and a device side 120.

[0037] The cloud side 110 may be a backend server that provides automated execution services, and the backend server may be a server, a server cluster, or a cloud computing center. In some embodiments, the cloud side 110 is provided with a management platform, and the management platform may be further divided into an atomic capability / script management platform 111, an atomic capability / script configuration entry platform 112, and an instruction management platform 113.

[0038] The Atomic Capability / Script Configuration Entry Platform serves as a service library, supporting the configuration and entry of automated execution scripts and the atomic capabilities supported by terminals. The Atomic Capability / Script Management Platform, a ubiquitous cloud platform, manages the atomic capabilities and automated execution scripts supported by terminals and provides automated execution scripts to terminals based on their needs. The Instruction Management Platform manages the configuration of automated execution instructions, such as pre-configured natural language automated execution instructions for implementing specific automated tasks, or the entry of instructions for implementing atomic capabilities.

[0039] The terminal side 120 may be a terminal with an automated task execution function, such as a smartphone, tablet computer, wearable device, or personal computer, etc. In some embodiments, the terminal side 120 is provided with a unified call SDK 121 , an intent execution framework independent process 122 , and an intent distribution framework 123 .

[0040] Among them, the unified call SDK belongs to the capability caller, which is used to provide interactive entrances including voice assistant, suggestion entrance, global search, voice command, etc., so that users can use the automated task execution function through multiple interactive entrances. The independent process of the intention execution framework serves as the execution subject of the automated task, supports the management, generation, orchestration, scheduling and execution of automated execution scripts, and supports real-time interaction with interactive components (such as real-time activity reminder controls) to display the real-time task execution status through interactive components, and receive user operations when manual intervention is required. The intent distribution framework can further include intent perception, intent decision-making and intent distribution (distributing the identified and decided intent to various entrances for the entrance to display the intent) modules, and supports interaction with the unified call SDK to deliver the intention confirmed by the user (the subsequent independent process of the intent execution framework executes the automated task to realize the intention).

[0041] In one possible design, the cloud-side 110 management platform, the client-side 120 unified call SDK, and the intent execution framework standalone process are implemented as the intent execution framework, which can independently accept and implement the caller's request. In another possible design, the intent execution framework can interact with the intent distribution framework to obtain the request automatically identified by the intent distribution framework and confirmed by the user.

[0042] For the convenience of description, the following embodiments are described by taking the automatic execution method executed by a terminal as an example.

[0043] Please refer to Figure 2 , which shows a flowchart of a multimedia playback method provided by an exemplary embodiment of the present application. This embodiment is described using the method applied to a terminal as an example. The method may include the following steps:

[0044] Step 201: Receive a multimedia automatic play instruction, where the multimedia automatic play instruction is used to instruct execution of a multimedia automatic play task.

[0045] Optionally, the multimedia automatic play task may be passively identified by the terminal, or actively identified by the terminal.

[0046] Optionally, the multimedia automatic playback instruction is an automated execution instruction issued by the user (such as through voice, clicking on an automated task desktop component, etc.), and the multimedia automatic playback task is identified from the automated execution instruction, and / or the terminal receives a confirmation execution instruction after identifying the existence of an executable multimedia automatic playback task and prompting.

[0047] The terminal may identify whether there is a multimedia automatic playback task based on at least one of a current foreground application, a current display interface, a current scene, and a user operation.

[0048] The multimedia automatic play task in the embodiment of the present application refers to a task of automatically controlling multimedia play within a multimedia application. Moreover, each task step of the multimedia automatic play task is completely executed independently by the terminal, or some task steps require slight manual intervention.

[0049] Step 202 : Based on the automation execution script corresponding to the multimedia automatic play task, the multimedia automatic play task is executed in the task execution window running in the background.

[0050] In some embodiments, the automated execution script includes operation information and interface information for each step of the operation required to implement the multimedia automatic playback task.

[0051] Optionally, the automation execution script may be a preset script, a script obtained by arranging and combining preset scripts, or a script automatically generated based on automation task requirements.

[0052] Optionally, the process of determining the automated execution of the script may be executed locally by the terminal, or by the terminal with the aid of a server, or by the terminal and the server in collaboration.

[0053] The multimedia automatic playback task in the embodiment of the present application is executed in a task execution window. This task execution window is different from a conventional application window. Instead, it has undergone a series of capability modifications to ensure that the multimedia automatic playback task in the task execution window can be executed normally in both the foreground and background states. The following embodiment will explain the specific capabilities of the task execution window in detail.

[0054] In some embodiments, during the execution of a multimedia automatic play task, a multimedia application to which the multimedia automatic play task belongs is run in the task execution window, wherein the multimedia application to which the multimedia automatic play task belongs refers to the multimedia application in which multimedia automatic play needs to be performed.

[0055] The multimedia application can be a video conferencing application, a short video application, a video playback application, an audio playback application, a live broadcast application or an online course application, etc., and the embodiments of the present application are not limited to this.

[0056] The task execution window in the embodiment of the present application supports background operation. Here, background operation means that the task execution window is located outside the screen display area, that is, the user cannot view the real-time execution status of the automated task in the task execution window through the screen (the desktop or the original foreground application window remains displayed).

[0057] Furthermore, when the task execution window is running in the background, the desktop or other application windows can be displayed in the foreground of the terminal, and the user can perform operations in the desktop or the foreground running window.

[0058] Of course, the task execution window, like ordinary application windows, supports foreground operation. Foreground operation means that the task execution window is located in the screen display area, that is, the user can view the real-time execution status of the automated task in the task execution window through the screen (the task execution window running in the foreground will block the desktop or the original foreground application window).

[0059] In some embodiments, the task execution window supports switching between foreground running and background running states, wherein the switching can be manually triggered by the user or automatically triggered by the terminal.

[0060] Regarding the execution method of multimedia automatic playback tasks, in one possible implementation, the terminal performs multimedia automatic playback tasks based on an automated execution script through simulated operations and / or with the help of accessibility services. The simulated operations are used to simulate interface operations such as clicks, drags, and slides within the interface based on coordinates. When executing automated tasks through accessibility services, relevant permissions need to be enabled (for example, enabling the Accessibility Service in the system settings).

[0061] Since there are large differences in the in-application functions, interface layouts, etc. of different multimedia applications, and there may be differences in functions or interface layouts between different application versions of the same multimedia application, in one possible implementation, the terminal determines an automated execution script that matches the multimedia application and application version to which the multimedia automatic playback task belongs, so as to ensure the accuracy of the operations performed based on the automated execution script.

[0062] In some embodiments, in order to ensure that multimedia can be played normally in the background state, in an embodiment of the present application, the multimedia application running in the task execution window is set to stop playing only when the audio focus is preempted, that is, when the audio focus is not preempted by other multimedia applications, even if the task execution window is in the background running state, the multimedia application still maintains multimedia playback.

[0063] In addition, to prevent the audio during multimedia background playback from interfering with the user's use of other foreground applications, in one possible implementation, the terminal mutes the multimedia application running in the task execution window. The muting process can be achieved by adjusting the volume of the multimedia application to the lowest level using the application volume control function provided by the operating system (which can control the volume of different applications separately, rather than using a unified volume for different applications).

[0064] Step 203: During the execution of the multimedia automatic playback task, the real-time playback status of the multimedia is displayed through a real-time activity reminder control.

[0065] Generally, the execution process of the multimedia automatic playback task needs to continue or be maintained for a long time. Therefore, in order to enable users to understand the real-time situation in a timely manner (especially when the task execution window is running in the background), the terminal displays the real-time playback status of the multimedia through a real-time activity reminder control (used to display real-time activity information of real-time activities).

[0066] The real-time multimedia playback status includes but is not limited to at least one of the following: multimedia information of the currently playing multimedia (such as multimedia name), multimedia status (pause or play), and multimedia playback progress (such as the playback time point or playback percentage).

[0067] In some embodiments, when the task execution window is in the foreground and background running states, the terminal displays the real-time playback status of the multimedia through the real-time activity reminder control.

[0068] Optionally, the real-time activity reminder control has a smaller size (compared to the task execution window) and is used to display key real-time playback conditions, which may include multimedia playback status.

[0069] In some embodiments, the real-time activity reminder control has multiple forms. The size of the real-time activity reminder control in different forms varies, and accordingly, the amount and type of information that can be displayed vary. The terminal can switch the form of the real-time activity reminder control based on the real-time playback status of multimedia.

[0070] In some embodiments, the real-time activity reminder control supports interaction, that is, by triggering the real-time activity reminder control or triggering the sub-control in the real-time activity reminder control, specific functions can be achieved, such as triggering the form switching of the real-time activity reminder control, switching the running status of the task execution window, performing manual intervention, etc. The interactive process of the real-time activity reminder control will be described in detail in the following embodiments.

[0071] Indicative, such as Figure 3 As shown, when the desktop 301 is displayed, when a user voice command is received, the terminal displays a text-based natural language instruction 302. When the natural language instruction 302 is recognized as a multimedia automatic playback request, the terminal determines the automation execution script corresponding to the multimedia automatic playback task and executes the multimedia automatic playback task in the task execution window running in the background based on the automation execution script. During the execution of the multimedia automatic playback task, the terminal displays the real-time playback status of the multimedia through the real-time activity reminder control 303.

[0072] In summary, in an embodiment of the present application, when there is a need to automatically play multimedia, the terminal can run a task execution window in the background and execute the multimedia automatic playback task in the task execution window based on the automated execution script corresponding to the multimedia automatic playback task. Since the multimedia automatic playback task is automatically executed in the task execution window running in the background, the user's manual operation process is eliminated and other applications running in the foreground are not affected. In addition, during the execution of the multimedia automatic playback task, the terminal displays the real-time playback status of the multimedia through a real-time activity reminder control, thereby reducing interference with the foreground application and improving the user's perception of the real-time execution of the task.

[0073] In addition to the user actively triggering the execution of the multimedia automatic playback task, in another possible implementation, the terminal identifies the current interface. If the current interface is identified as a target interface, the terminal displays a task execution prompt message, where the target interface is an interface with a multimedia automatic playback requirement. The task execution prompt message is used to indicate the existence of an executable multimedia automatic playback task. Furthermore, the terminal receives a task execution operation in response to the task execution prompt message, i.e., determines that a multimedia automatic playback instruction has been received.

[0074] Optionally, the terminal can determine whether it is currently in the target interface by monitoring the activity.

[0075] Optionally, the terminal may provide an automated task execution prompt in the form of a real-time activity reminder control, a banner notification, a desktop component, etc. Upon receiving a confirmation operation for the automated task execution prompt, the terminal executes the automated task.

[0076] In a possible embodiment, an automatic execution script for realizing the multimedia automatic play task is pre-set. When the terminal recognizes that it is in the target interface and receives a task execution operation for the task execution prompt information, the terminal executes the multimedia automatic play task based on the pre-set automatic execution script.

[0077] Indicative, such as Figure 4 As shown, an automated execution script for realizing the automated task of "multimedia automatic playback" is pre-set, and the video playback interface of the online course application A is set as the target interface corresponding to the task. When the online course application A is turned on and the video playback interface 401 is opened, the terminal displays an automated task execution prompt 402, prompting the user to automatically play the online course video in the background. When a click operation is received on the execution button in the automated task execution prompt 402, the terminal executes the multimedia automatic playback task in the task execution window running in the background based on the preset automated script. During the execution of the multimedia automatic playback task, the terminal displays the real-time playback status of the online course through the real-time activity reminder control 403.

[0078] Of course, in other possible implementations, the terminal can also identify whether there is a current demand for automatic multimedia playback based on the current scenario, user historical habits and other information. For example, based on the user's historical habits, it is determined to start listening to online classes at 6 o'clock every night, so that the task execution prompt information is displayed at 6 o'clock. The embodiment of the present application does not limit the specific conditions for triggering the display of the task execution prompt information.

[0079] In some embodiments, when the multimedia automatic play instruction has not only an automatic play requirement but also a summary requirement, during the execution of the multimedia automatic play task, the terminal summarizes the multimedia content and inputs the multimedia content summary information when the automatic task is completed.

[0080] In a possible implementation, the terminal may convert the audio into text and perform summary processing on the text during multimedia playback, and / or perform image semantic recognition on the video content and perform summary processing on the image semantic recognition results to obtain multimedia content summary information.

[0081] Of course, in some other embodiments, when the multimedia automatic playback instruction also has requirements such as screenshots and recording, the terminal can execute them synchronously during the execution of the multimedia automatic playback task, and the embodiments of the present application do not limit this.

[0082] The difference between the task execution window and the normal application window

[0083] In order to ensure the normal execution of automated tasks in the task execution window in various states (including foreground and background states), the task execution window needs to undergo a series of functional improvements based on the ordinary application window.

[0084] In a possible design, the task execution window differs from the normal application window in at least the following ways:

[0085] 1. Whether to block the input method display function.

[0086] To ensure that users can enter text normally within the application, the normal application window does not block the input method display function. That is, when there is a need for text input, the normal application window will display the text input area (virtual keyboard), and the user can complete the text input by clicking in the text input area. For example, when receiving a click operation on the text input box, the terminal automatically displays the text input area in the lower half of the screen. When the text input is completed, the terminal stops displaying the text input area.

[0087] However, when the automated task requires text input, if text input is achieved by simulating text input click operations in the text input area, the input efficiency is low (for example, it is necessary to simulate the click of each pinyin) and the accuracy of the text input cannot be guaranteed (for example, when selecting text based on pinyin, text selection errors may occur).

[0088] Therefore, the task execution window in the embodiment of the present application needs to shield the input method display function. That is, when there is a text input requirement, the text input area will not appear in the task execution window.

[0089] Optionally, when the multimedia automatic playback task requires text input, the terminal can fill in the text in the interface by importing text or copying and pasting text (for example, using a clipboard).

[0090] 2. Whether a mask is set.

[0091] In order to ensure normal user interface operation within the application interface, ordinary application windows usually do not have a mask set.

[0092] In order to avoid the failure of automated task execution due to user manual interface operations (the task execution window is in the foreground running state) during the execution of the automated task, the task execution window in the embodiment of the present application needs to be set with a mask to shield the user's manual interface operations in the task execution window through the mask.

[0093] In some embodiments, the size of the mask matches the size of the task execution window and is located on top of the task execution window.

[0094] It should be noted that, in addition to the above-mentioned differences, in some embodiments, the accessibility of the task execution window also needs to be modified so that the modified task execution window can support normal use of accessibility functions even in the background and non-focus state.

[0095] The execution process of multimedia automatic playback tasks

[0096] During multimedia playback, multimedia playback may be paused due to various reasons. In order to be able to promptly detect playback pauses (especially to mute multimedia played in the background) and resume playback, in a possible implementation, as Figure 5 As shown, the above step 202 may include the following sub-steps:

[0097] Step 202A: Based on the automated execution script, perform multimedia status detection on the task execution window running in the background.

[0098] In a possible implementation, the terminal may obtain the multimedia state of the multimedia application running in the task execution window through the system interface, wherein the multimedia state includes a play state and a pause state.

[0099] Optionally, in addition to being able to obtain the multimedia status, the terminal can also obtain information such as the multimedia playback duration, the total multimedia duration, and the multimedia name, which is not limited in this embodiment of the present application.

[0100] Because multimedia playback typically requires a long duration, to avoid the impact of frequent multimedia status checks on power consumption, in one possible implementation, the automated execution script includes a time interval for multimedia status checks. Accordingly, the terminal periodically performs multimedia status checks on the task execution window running in the background based on the time interval indicated by the automated execution script.

[0101] Indicatively, during the execution of the multimedia automatic play task, the terminal performs periodic multimedia status detection on the task execution window every 60 seconds.

[0102] Optionally, if the multimedia status detection result is the playing state, the terminal continues to perform multimedia status detection. If the multimedia status detection result is the pause state, the terminal performs the following step 202B.

[0103] Step 202B: When the multimedia status detection result is a paused state, a simulation operation is performed in the task execution window running in the background based on the automated execution script. The simulation operation is an interface operation required to implement the multimedia automatic playback task.

[0104] When the multimedia is in a paused state, in order to enable the multimedia automatic play task to continue normally, the terminal performs a simulation operation in the task execution window running in the background based on the automatic execution script to resume the multimedia play state.

[0105] Optionally, the simulated operation may include various manual interface operations such as simulated click operation, simulated long press operation, simulated drag operation, simulated sliding operation, etc. The embodiment of the present application does not limit the specific operation type of the simulated operation.

[0106] It's important to note that compared to using accessibility services to automate tasks, automating tasks through simulated operations is more applicable to complex interfaces and is more adaptable to dynamic content (such as those where the position of interface elements or content frequently changes). Furthermore, unlike using accessibility services, which require the application to remain in the foreground or enable picture-in-picture mode, simulated operations can also be performed normally when the task execution window is in the background.

[0107] Since the multimedia may enter a pause state before or after it is finished playing, and there are significant differences in the interface operations required to resume multimedia playback between the unfinished and completed states, in order to improve the accuracy of the simulated operations performed, the terminal can determine which simulated operation to perform based on whether the current multimedia has been played.

[0108] In one possible implementation, when the multimedia status detection result is a paused state and the current multimedia has not been played completely, based on the automated execution script, the terminal performs a playback resumption operation in the task execution window running in the background. The playback resumption operation is a simulated operation to resume playback of the current multimedia.

[0109] When the multimedia status detection result is paused and the current multimedia is finished playing, a play switching operation is performed in the task execution window running in the background based on the automated execution script. The play switching operation is a simulation operation of switching to play the next multimedia.

[0110] Optionally, a first sub-script and a second sub-script in the script are automatically executed, wherein the first sub-script is a sub-script for performing a playback resumption operation, and the second sub-script is a sub-script for performing a playback switching operation. When the current multimedia is in a paused state, the terminal determines whether to execute the first sub-script or the second sub-script based on whether the current multimedia has been played.

[0111] In some embodiments, the terminal can obtain the multimedia playback time and the total multimedia time of the current multimedia through the system interface. When the multimedia playback time is consistent with the total multimedia time, the terminal determines that the current multimedia playback is completed; when the multimedia playback time is less than the total multimedia time, the terminal determines that the current multimedia has not been completed.

[0112] In other possible embodiments, the terminal may also perform OCR recognition on the interface displayed in the task execution window to obtain the multimedia playback time and the total multimedia time of the current multimedia, and thereby determine whether the current multimedia has been played. This embodiment of the present application does not limit this.

[0113] When multimedia playback is completed and paused, some multimedia applications support automatic playback of the next multimedia, while others do not. Therefore, to avoid accidental multimedia switching, in one possible implementation, if the multimedia status detection result indicates a pause and the current multimedia playback is complete, the terminal performs another multimedia status detection after a preset time.

[0114] When the multimedia state detection result obtained by the re-detection is a pause state (indicating that it is not automatically switched to the next multimedia), the terminal performs a play switching operation in the task execution window running in the background based on the automatic execution script.

[0115] Optionally, when the multimedia status detection result obtained by the re-detection is the playing state, the terminal does not perform additional operations.

[0116] In order to avoid looping after the playback is completed, in some embodiments, when the multimedia status detection result obtained by re-detection is the playback status, the terminal determines whether the current multimedia is consistent with the historical multimedia. If not, no additional operation is performed; if consistent, the terminal performs the playback switching operation in the task execution window running in the background based on the automated execution script.

[0117] In this embodiment, the terminal identifies whether the current multimedia is paused, and when it is identified as being in a paused state, further performs a playback resumption operation or a playback switching operation in the task execution window based on the completion status of the current multimedia playback, which helps to improve the success rate of resuming playback in a playback pause scenario.

[0118] When the current multimedia is paused during playback (i.e., the current multimedia has not finished playing), there are various reasons for the playback pause. Among them, the playback pause caused by some reasons can be restored through simple simulation operations in the interface, while the playback pause caused by some reasons requires manual operations in the interface to be restored, and the playback pause caused by some reasons requires manual operations outside the multimedia application to be restored.

[0119] The following describes the processing flow for different suspension reasons.

[0120] 1. The first reason

[0121] In some embodiments, the pause caused by the first reason supports resumption through a play resume operation.

[0122] Optionally, the first reason includes the audio focus of the multimedia application running in the task execution window being preempted. For example, while the task execution window is running in the background, other multimedia applications are running in the foreground of the terminal, and the other multimedia applications preempt the audio focus, causing the multimedia in the task execution window to pause.

[0123] In some embodiments, the terminal may determine whether playback is paused due to the first reason by detecting whether the audio focus is located in an application running in the task execution window.

[0124] In a possible implementation, when the multimedia status detection result is a pause state, the current multimedia has not been played completely, and the pause reason is the first reason, the terminal performs a playback resumption operation in the task execution window running in the background based on an automated execution script.

[0125] Optionally, the playback resume operation is a simulated click operation on the playback control in the interface. After the playback resume operation is executed, the multimedia application running in the task execution window retakes the audio focus and continues to play the current multimedia.

[0126] Optionally, the terminal executes the playback resumption operation in the task execution window running in the background based on the automated execution script without user confirmation, that is, the terminal automatically executes the playback resumption operation.

[0127] In order to avoid alternating audio focus preemption due to playback resumption operations, in one possible implementation, when the multimedia status detection result is a pause state, the current multimedia has not been played completely, and the pause reason is the first reason, the terminal uses a real-time activity reminder control to provide a playback resumption prompt.

[0128] Furthermore, in response to the playback resumption confirmation operation in the real-time activity reminder control, the terminal executes the playback resumption operation in the task execution window running in the background based on the automated execution script.

[0129] In some embodiments, when a playback resumption confirmation operation is received within a real-time activity reminder control, the intent framework sends a resume playback instruction to the simulation operation execution engine, which executes the playback resumption operation in the task execution window running in the background based on the automated execution script.

[0130] Optionally, the playback resumption prompt includes a multimedia pause reason and a resume control for triggering the resumption of multimedia playback. Accordingly, in response to the triggering operation of the resume control, the terminal determines that a playback resumption confirmation operation has been first received.

[0131] It should be noted that if the playback resumption confirmation operation in the real-time activity reminder control is not received, the current multimedia remains in the paused state.

[0132] Indicative, such as Figure 6 As shown, when the multimedia application running in the task execution window pauses multimedia playback due to audio focus being preempted, the terminal issues a playback resumption prompt through the real-time activity reminder control 601. When a trigger operation is received on the resume button in the real-time activity reminder control 601, the terminal resumes the playback of the multimedia in the task execution window running in the background.

[0133] Of course, in other possible implementations, in response to a confirmation operation on the playback resumption in the real-time activity reminder control, the terminal pulls the task execution window back to the foreground, and the user manually resumes multimedia playback.

[0134] 2. The second reason

[0135] In some embodiments, the suspension caused by the second reason supports recovery through manual operation in the application interface.

[0136] Optionally, the second reason may include a multimedia application running within the task execution window requiring manual operation. For example, when the current multimedia is played to a specified node, manual verification (verification of whether a real person is watching) or manual answering is required, and the simulated operation cannot complete manual verification and manual answering.

[0137] In some embodiments, the terminal may identify the interface displayed in the task execution window through OCR to determine whether the playback is paused due to the second reason. For example, when a human intervention element (such as a human intervention prompt image or text) is identified in the interface, the terminal determines that the playback is paused due to the second reason.

[0138] In a possible implementation, when the multimedia status detection result is a pause state, the current multimedia has not been played completely, and the pause reason is the second reason, the terminal performs a manual intervention prompt through a real-time activity reminder control.

[0139] Optionally, the manual intervention prompt includes the reason for multimedia pause and a control for triggering manual intervention.

[0140] In order to facilitate the user to resume multimedia playback through manual operation when multimedia is paused due to the second reason, in response to the manual intervention confirmation operation in the real-time activity reminder control, the terminal displays the task execution window in the foreground.

[0141] Optionally, the manual intervention confirmation operation may be a triggering operation on a control in the real-time activity reminder control for triggering manual intervention.

[0142] Since the task execution window blocks the input method display function and is provided with a mask, in order to enable the user to perform manual operations in the task execution window displayed in the foreground, in a possible implementation, the terminal unblocks the input method display function of the task execution window and removes the mask.

[0143] In some embodiments, after the manual intervention is completed, the user can manually switch the task execution window to background operation, or, upon recognizing that the manual intervention is completed, the terminal switches the task execution window to background operation.

[0144] The terminal may identify the interface displayed in the task execution window by means of OCR, and determine that the manual intervention is completed when it is identified that no manual intervention elements exist in the interface.

[0145] It should be noted that when the task execution window is switched to background operation, the terminal needs to re-shield the input method display function of the task execution window and set a mask.

[0146] Indicative, such as Figure 7As shown, when multimedia playback reaches a designated point and triggers facial verification, multimedia playback is paused. The terminal then issues a manual intervention prompt via real-time activity reminder control 701. Upon receiving a trigger operation on the manual intervention button in real-time activity reminder control 701, the terminal runs a task execution window in the foreground, which currently displays facial verification interface 702. After verification is completed in facial verification interface 702, the task execution window switches back to the background and resumes multimedia playback.

[0147] 3. The third reason

[0148] In some embodiments, the pause caused by the third reason does not support resumption through the playback resumption operation and manual operation in the application interface.

[0149] Optionally, the third reason includes reasons other than the multimedia application causing the pause in the task execution window, such as low battery, network anomaly, multimedia application crash, etc.

[0150] In one possible implementation, when the multimedia status detection result is a paused state, the current multimedia has not been played completely, and the pause reason is a third reason, the terminal issues a playback abnormality prompt through a real-time activity reminder control. The pause caused by the third reason does not support recovery through playback recovery operations and manual operations within the application interface.

[0151] Optionally, the abnormal playback prompt includes the reason for the abnormal playback.

[0152] Optionally, the terminal determines a recovery method for resuming multimedia playback under the third reason, and sets an abnormality recovery control in the abnormal playback prompt. When an operation on the abnormality recovery control is received, the terminal resumes multimedia playback based on the recovery method.

[0153] Indicative, such as Figure 8 As shown, when multimedia playback is paused due to a network anomaly, the terminal issues a playback anomaly notification via real-time activity reminder control 801. Upon receiving a click operation on abnormality recovery control 801 within real-time activity reminder control 801, the terminal displays network settings interface 802. After the network is restored on network settings interface 802, the task execution window switches back to background operation and resumes multimedia playback.

[0154] In this embodiment, the terminal identifies the reason for playback pause and uses different prompts for different reasons, so that the user can further trigger processes such as automatic playback resumption and manual intervention through the prompt, thereby improving the efficiency of exception handling during the execution of multimedia automatic playback tasks.

[0155] Implementation of simulation operations

[0156] To implement the simulation operation, in some embodiments, the automation execution script includes a simulation action script corresponding to the simulation operation, and the simulation action script includes a target widget field (target widget) and a target action field (actioninfo).

[0157] The Target Control field identifies the target control in the interface that receives the simulated operation, and the Target Action field identifies the type of simulated operation. In other words, the Target Control field indicates which control in the interface to perform the interface operation on, and the Target Action field indicates which type of simulated operation to perform on the control.

[0158] In some embodiments, the automation execution script is composed of at least one execution sub-script. The execution sub-script includes a target control field and a target action field corresponding to one or more simulation steps. The terminal can execute the corresponding one or more simulation steps in the automation task based on the execution sub-script.

[0159] In one possible implementation, Figure 9 As shown, the process of performing simulation operation may include the following steps:

[0160] Step 901: Search for a target control that matches the target control field in the interface displayed in the task execution window.

[0161] To ensure the accuracy of the simulated operation, before executing the simulated operation, the terminal needs to determine the target control to be simulated in the interface displayed in the task execution window based on the target control field. The target control is a control that supports the interface operation, including but not limited to buttons, text boxes, labels, check boxes, sliders, etc., which are not limited in this embodiment of the application.

[0162] For example, in a multimedia automatic playback task scenario, the target control may include a course search text box, a play / pause button, a course selection button, and the like.

[0163] Since the position of the same control in the interface and even the displayed content may change dynamically, in order to improve the success rate and accuracy of control search, in one possible implementation, the target control field is in list form, and the list contains control matching information of at least two target controls.

[0164] Optionally, different control matching information is used to describe the target control from different dimensions. For example, at least two control matching information items are used to describe the target control from dimensions such as display content, absolute display position, relative display position, and purpose.

[0165] Optionally, the terminal matches the control information of the control in the interface with at least one item of control matching information to determine a target control that matches the target control field. The target control's control information has the highest degree of matching with the control matching information, and / or the target control's control information matches the largest number of items of control matching information.

[0166] To improve search efficiency and accuracy, in one possible design, the matching priorities of at least two items of control matching information in the list are sorted in descending order of matching accuracy. The matching priority represents the order in which the control matching information in the list is matched. The earlier the control matching information in the list is, the higher the matching priority. Matching accuracy is positively correlated with the accuracy of the target control found based on the control matching information. That is, the higher the matching accuracy of the control matching information, the higher the accuracy of the target control found based on the control matching information.

[0167] In some embodiments, the higher the matching accuracy of the control matching information, the lower the generalization degree of the control matching information (applicable to fixed control searches); conversely, the lower the matching accuracy of the control matching information, the higher the generalization degree of the control matching information (applicable to dynamically changing control searches).

[0168] After sorting at least two control matching information according to matching accuracy, the terminal matches the control information of the controls in the interface with the control matching information in descending order of matching accuracy until the target control is found, that is, the target control is searched preferentially according to the control matching information with high matching accuracy.

[0169] In one possible implementation, the terminal searches for a target control that matches the i-th control matching information in the interface displayed in the task execution window based on the i-th control matching information in the target control field, where i is a positive integer and the i-th control matching information is the control matching information located in the i-th item in the target control field.

[0170] Optionally, when searching for a target control based on the i-th control matching information in the target control field, it indicates that no matching target control is found based on the first i-1 items of control matching information in the target control field.

[0171] In the case of finding a target control that matches the i-th control matching information, the terminal stops searching for the target control based on subsequent control matching information in the list.

[0172] If no target control matching the i-th control matching information is found, the terminal searches for a target control matching the i+1-th control matching information in the interface displayed in the task execution window based on the i+1-th control matching information in the target control field.

[0173] Optionally, the matching accuracy of the i-th control matching information is higher than the matching accuracy of the (i+1)-th control matching information.

[0174] If a target control matching the (i+1)th control matching information is found, the terminal stops searching for a target control based on the subsequent control matching information in the list. If a target control matching the (i+1)th control matching information is not found, the terminal searches for a target control matching the (i+2)th control matching information in the interface displayed in the task execution window based on the (i+2)th control matching information in the target control field, until the last control matching information in the target control field is found.

[0175] In a possible design, the list includes control attribute matching information and control semantic matching information, and the matching accuracy of the control attribute matching information is higher than the matching accuracy of the control semantic matching information.

[0176] Among them, the control attribute matching information is used to describe the target control from the attribute dimension, which may include ID, text, type, position, control hierarchy (parent-child relationship), etc.; the control semantic matching information is used to describe the target control from the semantic dimension, which may include text semantics, image semantics, usage semantics, etc.

[0177] In one possible implementation, the terminal first searches for a matching target control based on the control attribute information of the control in the interface and the control attribute matching information. If the target control is found, the terminal does not further search based on the control semantic matching information. If the target control is not found, the terminal further searches for a matching target control based on the control semantic information of the control in the interface and the control semantic matching information.

[0178] In some embodiments, during the process of searching for a target control based on control attribute matching information, the terminal searches for the target control based on the control attribute matching information in the target control field and the control attribute information of each control in the interface displayed in the task execution window.

[0179] Optionally, the terminal obtains control attribute information (of the target attribute type) of the control in the interface based on the target attribute type of the control attribute matching information, thereby matching the control attribute matching information with the control attribute information.

[0180] Optionally, the found control attribute information of the target control is consistent with the control attribute matching information. It should be noted that consistency here means that the control attribute information and the control attribute matching information are completely identical, or that the information similarity between the control attribute information and the control attribute matching information is greater than a similarity threshold.

[0181] In some embodiments, during the process of searching for a target control based on control semantic matching information, the terminal determines each control in the interface displayed in the task execution window, performs semantic processing, and obtains control semantic information. The terminal searches for the target control based on the control semantic matching information and the control semantic information in the target control field, and the similarity between the control semantic information of the target control and the control semantic matching information is greater than a similarity threshold.

[0182] Optionally, the target control is pre-semantically processed to obtain control semantic matching information corresponding to the target control. The semantic processing may be embedding the target control to obtain control semantic matching information in the form of a feature vector.

[0183] Accordingly, when searching for a target control, the terminal performs semantic processing on the control in the interface to obtain control semantic information of the control. The method of performing semantic processing on the control in the interface is the same as the method of performing semantic processing on the target control.

[0184] Optionally, the control semantic information and the control semantic matching information can be image semantic information of the control appearance dimension, and / or text semantic information of the text dimension displayed by the control, and / or usage semantic information describing the purpose of the control. The embodiments of the present application are not limited to this.

[0185] Regarding the matching method of control semantic matching information and control semantic information, in one possible implementation, when the control semantic matching information and control semantic information are both feature vectors, the terminal determines the similarity between the two based on the vector distance between the feature vectors (such as cosine distance).

[0186] Optionally, the terminal determines the highest similarity between the control semantic matching information and the control semantic information, and when the highest similarity is greater than a similarity threshold, determines the control corresponding to the highest similarity as the target control. Control semantic information of different dimensions may correspond to different similarity thresholds.

[0187] Step 402: When the target control is found, the simulation operation indicated by the target action field is performed on the target control.

[0188] When the target control is found based on the control matching information, the terminal performs a simulation operation on the target control according to the target action field corresponding to the target control field.

[0189] For example, in a multimedia automatic playback task scenario, the simulated operation on the target control may include clicking on the course search text box and text input operations, clicking on the play / pause button, clicking on the course selection button, and the like.

[0190] In this embodiment, the terminal stores at least two items of control matching information of the target control in a list form in descending order of matching accuracy, and in the process of searching for the target control, matches the target control in descending order of matching accuracy. While improving the control search efficiency, it can also improve the control success rate, especially in the dynamic control search scenario.

[0191] In addition, in this embodiment, the terminal can search for the target control from at least one dimension of control properties and control semantics. Searching for the target control based on control properties can improve the control search accuracy, and searching for the target control based on control semantics can improve the control search success rate.

[0192] Exception handling during simulation operations

[0193] During application execution, the target control may not be found due to some reasons. For example, the currently displayed interface is not the target interface where the simulation operation is expected to be performed, or a pop-up window appears above the currently displayed interface.

[0194] In order to ensure the normal execution of the automation task, in a possible implementation, when the target control is not found, the terminal performs an abnormality detection on the interface displayed in the task execution window and eliminates the abnormality.

[0195] Failure to find the target control in the current interface may be caused by the following factors: the presence of page ads within the application, pop-up ads within the application, or pop-up ads caused by human intervention within the application. Accordingly, in some embodiments, the terminal may perform anomaly detection by at least one of page ad detection, human intervention pop-up detection, and ad pop-up detection.

[0196] Among them, page advertising detection is used to detect whether the currently displayed interface is a page advertisement (which may include application splash screen advertisements and in-application advertisements); manual intervention pop-up detection is used to detect whether the currently displayed interface contains pop-ups that require manual intervention to eliminate; advertising pop-up detection is used to detect whether the currently displayed interface contains advertising content in the form of pop-ups.

[0197] Optionally, when the presence of a page advertisement is detected, the terminal performs a simulated return operation in the task execution window.

[0198] In some embodiments, the terminal can perform content recognition on the interface currently displayed in the task execution window, or determine whether the currently displayed interface is a page advertisement based on the interface link of the currently displayed interface in the task execution window. The embodiments of the present application do not limit the specific method of identifying page advertisements.

[0199] The simulated return operation may be a global return operation, for example, a sliding operation from the edge of the screen to the center of the screen.

[0200] In some embodiments, after the simulated return operation is completed, the terminal performs the step of searching for the target control again.

[0201] Optionally, when a pop-up window indicating manual intervention is detected, the terminal suspends the simulation operation and executes the manual intervention process.

[0202] In some embodiments, the terminal may perform pop-up window detection on the interface currently displayed in the task execution window, and if a pop-up window is detected, further identify the content displayed in the pop-up window, or determine whether the pop-up window is a manual intervention pop-up window based on the pop-up window attribute information. The embodiments of the present application do not limit the specific method of identifying manual intervention pop-ups.

[0203] Since the manual intervention pop-up window cannot be dismissed through simulated operations, the terminal needs to suspend the simulated operation, that is, suspend the execution of the automated task, and manually dismiss the manual intervention pop-up window so that the subsequent automated task can continue. Among them, the manual intervention process includes the manual authorization process, the manual payment process, the manual verification process, etc., which is not limited to this embodiment of the application.

[0204] In some embodiments, when a pop-up window for manual intervention is detected, the terminal records the node to which the automation task is currently being executed as a pause node. When the manual intervention is identified as complete, the terminal continues to perform the simulation operation from the pause node.

[0205] Optionally, upon recognizing that the pop-up window for manual intervention in the task execution window has been removed, the terminal may continue the simulation operation from the pause point. It should be noted that if the pop-up window for manual intervention is not removed within a preset time period, the terminal may terminate the automated task to avoid wasted power due to prolonged ineffective recognition.

[0206] Optionally, the terminal starts a timer when detecting a manual intervention pop-up window. The timer duration of the timer is the preset manual intervention duration (such as 30 seconds or 1 minute). When the timer duration is reached, the terminal starts to perform the simulation operation from the pause node.

[0207] Optionally, when an advertisement pop-up window is detected, a simulated pop-up window closing operation is performed in the task execution window.

[0208] In some embodiments, the terminal may perform pop-up window detection on the interface currently displayed in the task execution window, and if a pop-up window is detected, further identify the content displayed in the pop-up window, or determine whether the pop-up window is an advertisement pop-up window based on the pop-up window attribute information. The embodiments of the present application do not limit the specific method of identifying advertisement pop-ups.

[0209] The simulated pop-up window closing operation may be a click operation on a pop-up window close control. In some embodiments, upon detecting the presence of an advertisement pop-up window, the terminal further identifies a pop-up window close control corresponding to the advertisement pop-up window, and upon identifying the pop-up window close control, performs a simulated click operation based on the control position of the pop-up window close control to close the advertisement pop-up window. The terminal may identify a pop-up window close control in the form of text or icon through text or image recognition, which is not limited in this embodiment.

[0210] In some embodiments, after the simulated pop-up window closing operation is completed, the terminal performs the step of searching for the target control again.

[0211] In some embodiments, the terminal may perform multiple anomaly detections simultaneously, or perform multiple anomaly detections sequentially.

[0212] Indicative, such as Figure 10 As shown, before executing a simulated operation on a target control, the terminal searches for the target control within the interface currently displayed in the task execution window. If the target control is found, the terminal performs the simulated operation on the target control. If the target control is not found, the terminal first detects a page advertisement. If a page advertisement is detected, the terminal performs a simulated return operation and then continues searching for the target control.

[0213] If no page advertisement is detected, the terminal further performs a manual intervention pop-up window detection. If a manual intervention pop-up window is detected, the terminal pauses the simulation operation and waits for manual intervention. After the manual intervention is completed, the terminal continues to search for the target control.

[0214] If no manual intervention pop-up window is detected, the terminal further performs ad pop-up detection. If an ad pop-up window is detected, the terminal simulates a pop-up window closing operation on the detected ad pop-up window and continues to search for the target control after the ad pop-up window is closed.

[0215] Of course, the terminal may also perform anomaly detection in other orders, which is not limited in this embodiment.

[0216] In addition, if no abnormality is detected through the above-mentioned abnormality detection method, or if an abnormality is detected but cannot be eliminated (for example, a simulated return operation or a simulated pop-up window closing operation is performed multiple times, but the target control cannot be found), the terminal can give a prompt so that the abnormality can be manually eliminated.

[0217] In this embodiment, when the target control is not identified, the terminal uses the exception detection mechanism to identify the exception type and adopts corresponding exception elimination means to eliminate the exception, thereby avoiding the inability to continue the execution of automated tasks in scenarios such as advertisements and those requiring manual intervention, which helps to improve the success rate of the execution of automated tasks.

[0218] Branch processing during simulation operation

[0219] In some interaction scenarios, before performing a simulated operation on a target control, the terminal needs to determine whether to perform a simulated operation based on the interface state, or determine which simulated operation to perform based on the interface state.

[0220] For example, during the execution of a multimedia automatic playback task, if the multimedia playback speed needs to be set to 1.0x, the terminal needs to first determine whether the current playback speed displayed on the interface meets the speed requirement, and avoid performing unnecessary simulation operations on the speed setting control when the current playback speed meets the speed requirement (excessive simulation operations may be invalid and may also lead to setting errors, such as repeatedly clicking the speed setting control to set the current playback speed from 1.0x to 2.0x).

[0221] In a possible implementation, branch processing during the simulation operation may include the following steps.

[0222] 1. Determine the interface status of the interface displayed in the task execution window.

[0223] In some embodiments, the interface state is related to the subsequent simulated operation performed on the target control. The interface state may include the current control state of the target control in the interface, the interface link of the current interface (which may directly jump to the next interface within the application without being triggered by a simulated operation), and the interface parameters of the current interface (including the interface display ratio, display area, etc.), which are not limited in this embodiment.

[0224] Optionally, the script for determining the interface status of the interface displayed in the task execution window may be included in the automation execution script, or may be automatically generated by the simulation operation execution engine based on an existing script in the automation execution script.

[0225] In one possible implementation, the simulation operation execution engine generates a script for determining the interface state before the simulation operation based on the target control in the automation execution script and the simulation operation on the target control. The script is used to determine whether the current control state of the target control in the interface is the expected control state after the simulation operation is performed, and / or to determine the control type of the target control in the interface, thereby performing a simulation operation that matches the control type.

[0226] For example, the simulation operation execution engine adds a script for determining the current playback magnification of the magnification setting control before the existing "set playback magnification to 1.0x" script.

[0227] In one possible implementation, determining the interface state as a conditional judgment can be achieved by adding a judgment node called condition_judge to the automated execution script. Accordingly, after the judgment is completed, true_action_range and false_action_range are used to guide the execution of different script branches based on different values.

[0228] In a possible implementation, the terminal may determine the interface status based on the interface tree of the interface to determine which script branch to execute.

[0229] 2. Determine the script branch in the automated execution script that matches the interface state, where different script branches correspond to different interface states.

[0230] After determining the interface state, the simulation operation execution engine further determines the script branch that matches the interface state from the multiple script branches of the automated execution script. For unselected script branches, the terminal will not execute them.

[0231] For example, if the current playback magnification of the magnification setting control is recognized as 1.0x, the simulation operation execution engine chooses to skip the script branch of "Set the playback magnification to 1.0x"; if the current playback magnification of the magnification setting control is recognized as 2.0x, the simulation operation execution engine chooses the script branch of "Set the playback magnification to 1.0x".

[0232] 3. Perform simulation operations in the task execution window based on script branches.

[0233] Furthermore, the simulation operation execution engine executes the simulation operation in the task execution window based on the script corresponding to the selected script branch. The specific implementation of this process can refer to the above-mentioned related embodiments, and this embodiment will not be described in detail here.

[0234] In this embodiment, the terminal selects and executes the script branch that matches the interface state based on the interface state, which helps to avoid the problem of failure of automated task execution due to invalid simulation operations performed in the interface or incorrect simulation operations performed, and improves the accuracy of simulation operations in the interface.

[0235] In one possible implementation, the process of performing simulated operations based on automated execution scripts is executed by a simulated operation execution engine, which can be installed independently within the intent execution framework. In one possible design, the simulated operation execution engine is connected to the application via the intent execution framework's SDK. The simulated operation execution engine's engine script is registered with the intent framework and invoked by the intent framework for execution in appropriate scenarios.

[0236] Indicative, such as Figure 11 As shown, the SDK 1101 of the intent framework sends the atomic capability key and task parameters required to implement the automated task to the execution scheduling module 1102. The execution scheduling module 1102 interacts with the instruction management module 1103 to obtain the atomic capability execution script that implements the atomic capability from the script distribution platform 1104 on the cloud side, thereby calling the simulation operation execution engine 1105 to perform the simulation operation based on the automation execution script composed of the atomic capability execution script and the task parameters. Among them, the simulation operation execution engine 1105 supports script acquisition, control search, exception handling, multi-branch execution, background execution, current page execution, parallel execution, secondary input and cache management functions.

[0237] Creation, setting, cancellation and status switching of task execution windows

[0238] Since when there is a multimedia automatic play task execution requirement, the multimedia application executing the task may be in a non-running state or in a running state, the terminal uses different methods to set or create a task execution window for multimedia applications in different states.

[0239] In a possible implementation, when the multimedia application corresponding to the multimedia automatic playback task is in a running state, the terminal sets the common application window corresponding to the multimedia application as a task execution window. The running state includes a foreground running state and a background running state.

[0240] Since a corresponding common application window already exists when a multimedia application is running in the foreground or background, when a multimedia automatic playback task needs to be executed in the multimedia application, the terminal can transform the common application window to obtain a task execution window that meets the requirements of automated task execution.

[0241] Since the task execution window and the ordinary application window differ at least in the input method display function and the mask function, when the ordinary application window corresponding to the multimedia application is set as the task execution window, in one possible implementation, the terminal shields the input method display function of the ordinary application window corresponding to the multimedia application, and sets a mask for the ordinary application window to obtain the task execution window, and the mask is used to shield user operations.

[0242] Optionally, when the multimedia application is in the foreground running state (that is, when the ordinary application window corresponding to the multimedia application is displayed in the foreground), and the multimedia application corresponding to the multimedia automatic playback task is in the running state, the terminal blocks the input method display function of the ordinary application window and sets a mask for the ordinary application window to obtain the task execution window.

[0243] Optionally, when the multimedia application is running in the background (i.e., when the ordinary application window corresponding to the multimedia application is not displayed), the terminal blocks the input method display function of the ordinary application window and does not set a mask for the ordinary application window. Subsequently, when the multimedia application is running in the foreground, the terminal sets a mask for the ordinary application window of the multimedia application to shield user operations.

[0244] Optionally, after setting the ordinary application window corresponding to the multimedia application as the task execution window, the task execution window maintains the original running state of the ordinary application window; or, the terminal sets the task execution window to the default running state, such as setting the task execution window to the background running state by default; or, the terminal determines the running state of the task execution window based on factors such as task type, multimedia application, and current foreground application.

[0245] Of course, in other possible implementations, when the multimedia application is in the running state, the terminal may also close the multimedia application, create a task execution window, and restart the multimedia application in the task execution window.

[0246] In another possible implementation, when the multimedia application corresponding to the multimedia automatic play task is not running, the terminal creates a task execution window and starts the multimedia application in the task execution window.

[0247] Since there is no corresponding common application window when the multimedia application is not running, the terminal needs to create a new task execution window for running the multimedia application, thereby starting the multimedia application in the task execution window.

[0248] In some embodiments, the terminal supports creating a task execution window by using the startactivity method and starting an application in the window.

[0249] In some embodiments, the terminal creates a task execution window in a default running state. For example, the terminal creates a task execution window running in the background by default and starts a multimedia application in the task execution window running in the background; or, the terminal determines the running state of the task execution window to be created based on factors such as task type, multimedia application, and current foreground application.

[0250] In one possible implementation, the terminal provides a background application launch function. With this function, when the target application corresponding to an automation task is not running, the terminal creates a task execution window in the background and launches the target application in the task execution window. Creating a task execution window in the background can be a default policy of the terminal, or a policy determined based on factors such as the task type of the automation task, the target application corresponding to the automation task, and the current foreground application.

[0251] When the automated task is completed or terminated (manually or automatically), in order to enable the user to continue to perform operations in the target application corresponding to the automated task, in one possible implementation, when the automated task is completed or terminated, the terminal resets the task execution window to a normal application window.

[0252] Among them, the process of resetting the task execution window to a normal application window is to unblock the input method display function and remove the mask.

[0253] Of course, in other possible implementations, when the automation task is completed or terminated, the terminal closes the task execution window, and accordingly, the target application corresponding to the automation task is also closed, thereby preventing the target application from occupying memory resources for a long time.

[0254] In the example of this application, the task execution window can be switched between two states: foreground running and background running. The switching process can be manually triggered by the user or automatically triggered by the terminal.

[0255] Regarding the method of manually switching the window running status, in one possible implementation, when the task execution window is running in the foreground, in response to a first switching operation, the terminal switches the task execution window to running in the background; when the task execution window is running in the background, in response to a second switching operation, the terminal switches the task execution window to running in the foreground.

[0256] Optionally, the first switching operation may be an operation that switches the foreground window / application to the background, such as returning to the desktop or performing an application switching gesture; the second switching operation may be an operation that launches the application or switches the application to the foreground, such as clicking the application icon corresponding to the multimedia application, triggering an operation on the multitasking interface, or launching the target application in the sidebar. The embodiments of the present application do not limit the specific methods of the first switching operation and the second switching operation.

[0257] It should be noted that when the task execution window switches from background operation to foreground operation and there is no need for manual intervention, the terminal keeps the input method display function of the task execution window blocked and masked to ensure the continued execution of the automated task and avoid accidental touch by the user.

[0258] In this embodiment, by shielding the input method display function of the task execution window and setting a mask for the task execution window, text input and user manual operations during the execution of the automated task are prevented from affecting the execution effect of the automated task, which helps to improve the task execution efficiency and execution quality of the automated task.

[0259] In addition, in this embodiment, the task execution window supports automatically switching to a normal application window when the automated task is completed or terminated, so that the user can continue to operate the multimedia application.

[0260] Schematically, during the execution of an automated task (multimedia automatic playback task), the logic of the task execution window from creation to cancellation is as follows: Figure 12 shown.

[0261] Step 1201: Identify the existence of an automation task.

[0262] Step 1202 : Detect whether the target application (multimedia application) corresponding to the automation task is started in the background. If the target application is started in the background, execute step 1203 ; if the target application is started in the foreground, execute step 1211 .

[0263] Step 1203: Create a task execution window in the background.

[0264] Step 1204: Start the target application in the task execution window.

[0265] Step 1205 , check whether the target application is locked. If the target application is not locked, execute step 1206 ; if the target application is locked, execute step 1212 .

[0266] Since the target application corresponding to the automated task may be a locked application, and a locked application needs to be verified before it can be used, when starting the target application, it is necessary to determine whether the target application is locked to determine whether the task execution window needs to be switched to the foreground to unlock the application.

[0267] Step 1206: Whether to start the simulation operation execution engine. If the simulation operation execution engine is started, step 1207 is executed; if not, step 1213 is executed.

[0268] In some embodiments, when supporting automated tasks through simulated operations and accessibility services, the terminal needs to determine whether the automated task execution process requires the use of a simulated operation execution engine. Specifically, it needs to determine whether there are any steps in the automated task that require simulated operations. If so, the simulated operation execution engine needs to be started; if not, the simulated operation execution engine is not used.

[0269] Step 1207: Add a mask to the task execution window, and execute the automation execution script of the automation task through the simulation operation execution engine.

[0270] Step 1208: During the execution of the automated task, monitor the task execution window status.

[0271] Step 1209: When it is detected that the task execution window is switched from the foreground to the background, the task execution window is moved outside the screen display area.

[0272] Step 1210: When the automated task is completed, the task execution window is set as a normal application window.

[0273] Step 1211: Start the task execution window in the foreground and execute the automated task execution logic.

[0274] Step 1212: Switch the task execution window from outside the screen display area to inside the screen display area, and unlock the application.

[0275] Step 1213: Call functions other than the simulation operation execution engine (such as accessibility functions) to execute the automation task.

[0276] Mask control settings

[0277] In some embodiments, the mask added to the task execution window is used not only to prevent the user from accidentally touching the task, but also to enable the user to control the execution process of the multimedia automatic playback task by setting a series of controls.

[0278] In a possible implementation manner, the mask is provided with at least one of a pause control, a resume control, and a termination control.

[0279] The pause control is used to control the pause execution of the multimedia automatic play task, and the resume control is used to control the resume execution of the multimedia automatic play task.

[0280] In one possible design, the pause control and the resume control are two independent controls, and during the execution of the automated task, the pause control is in a triggerable state and the resume control is in an uncatchable state; when the automated task is paused, the pause control is in an uncatchable state and the resume control is in a triggerable state.

[0281] In another possible design, the pause control and the resume control reuse the same control. During the execution of the multimedia automatic playback task, the reused control has a pause function (a pause mark can be displayed); when the multimedia automatic playback task is paused, the reused control has a resume function (a resume mark can be displayed).

[0282] Optionally, in response to a triggering operation on a pause control in the mask, the terminal pauses the multimedia automatic play task; in response to a triggering operation on a resume control in the mask, the terminal resumes the multimedia automatic play task.

[0283] To facilitate subsequent resumption of execution, when a trigger operation is received on the pause control, the terminal records the current task node of the multimedia automatic playback task; when a trigger operation is received on the resume control, the terminal resumes the multimedia automatic playback task starting from the recorded current task node.

[0284] The termination control is used to control the termination of the multimedia automatic play task. Optionally, in response to a triggering operation on the termination control in the mask, the terminal terminates the multimedia automatic play.

[0285] Optionally, after terminating the multimedia automatic playback task, the terminal sets the task execution window to a common application window, ie, cancels the mask and unblocks the input method display function.

[0286] In one possible application scenario, during the execution of a multimedia auto-play task, the user switches the task execution window from the background to the foreground through a switching operation. If the simulated operation executed in the task execution window is observed to be incorrect, the user can click the abort control on the mask to stop the multimedia auto-play task.

[0287] In this embodiment, by setting function controls on the mask, the user can control the execution process of the automation task through the function controls, thereby improving the controllability of the execution process of the multimedia automatic playback task.

[0288] Display of real-time activity reminder controls

[0289] In one possible implementation, a real-time activity reminder control for displaying the real-time playback status of multimedia has two forms: a first form and a second form. The first form of the real-time activity reminder control is smaller than the second form of the real-time activity reminder control. Therefore, displaying the first form of the real-time activity reminder control can reduce obstruction of content displayed on the desktop or application interface.

[0290] Optionally, the first form of the real-time activity reminder control can display less information than the second form of the real-time activity reminder control.

[0291] In an illustrative example, the first form is a capsule form, and the second form is a card form. The real-time activity reminder control in the capsule form is used to display a simple real-time playback situation (such as only displaying the multimedia status), while the real-time activity reminder control in the card form can display a more detailed real-time playback situation (such as displaying multimedia information, multimedia status, and multimedia playback progress).

[0292] In the embodiment of the present application, the real-time activity reminder control supports conversion between a first form and a second form.

[0293] The real-time activity reminder control is converted from the first form to the second form, which can be triggered manually by the user or automatically by the terminal.

[0294] In some embodiments, the real-time activity reminder control may be converted from the first form to the second form in at least one of the following situations:

[0295] 1. In response to a triggering operation of a first-form real-time activity providing control, the terminal displays the real-time playing status of multimedia through a second-form real-time activity reminder control.

[0296] Among them, the trigger operation can be a click operation, a long press operation, a slide operation, etc. on the control that provides the first form of real-time activity, and the embodiment of the present application does not limit this.

[0297] Indicative, such as Figure 13 As shown, during the execution of the multimedia automatic playback task, the terminal displays a first-form real-time activity reminder control 1301, which displays a brief real-time playback status (the word "playing"). When a click operation is received on the first-form real-time activity reminder control 1301, the terminal switches the first-form real-time activity reminder control 1301 to a second-form real-time activity reminder control 1302, which displays a more detailed real-time playback status (including multimedia playback progress, multimedia status, and multimedia information).

[0298] Optionally, when the display time of the second-form real-time activity reminder control reaches a first time threshold and no trigger operation on the real-time activity reminder control is received within the first time threshold, the real-time activity reminder control is converted from the second form to the first form; or, when a form conversion operation on the second-form real-time activity reminder control is received, the real-time activity reminder control is converted from the second form to the first form, wherein the form conversion operation can be a swipe-up operation on the second-form real-time activity reminder control, or a click operation on the blank display area outside the real-time activity reminder control. The embodiment of the present application does not limit the specific form of the form conversion operation.

[0299] 2. When manual intervention is required, the terminal displays the real-time multimedia playback status through the second form of real-time activity reminder control.

[0300] Since there may be some links that require manual intervention during the execution of multimedia automatic playback tasks, in order to provide strong reminders when manual intervention is required so that users can perform manual operations as soon as possible, the terminal switches the first form of real-time activity reminder control to the second form of real-time activity reminder control when manual intervention is required.

[0301] In some embodiments, when manual intervention is required, the second form of the real-time activity reminder control includes relevant information prompting manual intervention and relevant controls for performing manual intervention.

[0302] Indicative, such as Figure 14 As shown, during the execution of the multimedia automatic playback task, the terminal displays a first-form real-time activity reminder control 1401, which briefly displays the real-time playback status. When the playback reaches a specific node requiring face verification, the terminal determines that manual intervention is required, and thus switches the first-form real-time activity reminder control 1401 to a second-form real-time activity reminder control 1402. The second-form real-time activity reminder control 1402 displays detailed information about manual intervention and a control (open button) for manual intervention.

[0303] Optionally, when the display time of the real-time activity reminder control in the second form reaches a second time threshold and no trigger operation on the real-time activity reminder control is received within the second time threshold (which can be set to be greater than the first time threshold to have a strong reminder effect), the real-time activity reminder control is converted from the second form to the first form; or, when a form conversion operation on the real-time activity reminder control in the second form is received, the real-time activity reminder control is converted from the second form to the first form, wherein the form conversion operation can be a swipe up operation on the real-time activity reminder control in the second form, or a click operation on the blank display area outside the real-time activity reminder control. The embodiment of the present application does not limit the specific form of the form conversion operation.

[0304] Indicative, such as Figure 14 As shown, when the display time of the second form real-time activity reminder control 1402 reaches 10 seconds and no trigger operation is received on the second form real-time activity reminder control 1402 within 10 seconds, the terminal redisplays the first form real-time activity reminder control 1401, and the first form real-time activity reminder control 1401 displays a brief manual intervention prompt information.

[0305] 3. In the event of abnormal playback or completion of the multimedia automatic playback task, the terminal displays the real-time playback status of the multimedia through the second form of real-time activity reminder control.

[0306] In addition to providing a strong reminder when manual intervention is required, in the event of a playback anomaly (e.g., due to external factors such as the network) or when a multimedia automatic playback task is completed, the terminal switches the first-form real-time activity reminder control to a second-form real-time activity reminder control to remind the user to manually resolve the anomaly or to remind the user that the task has been completed. The second-form real-time activity reminder control displays an anomaly prompt or completion prompt.

[0307] Optionally, when the display time of the second-form real-time activity reminder control reaches a third time threshold and no trigger operation on the real-time activity reminder control is received within the third time threshold (which can be set to be greater than the first time threshold to have a strong reminder effect), the real-time activity reminder control is converted from the second form to the first form; or, when a form conversion operation on the second-form real-time activity reminder control is received, the real-time activity reminder control is converted from the second form to the first form, wherein the form conversion operation can be a swipe-up operation on the second-form real-time activity reminder control, or a click operation on the blank display area outside the real-time activity reminder control. The embodiment of the present application does not limit the specific form of the form conversion operation.

[0308] Indicative, such as Figure 15As shown, when the multimedia automatic playback task is completed, the terminal displays the second-form real-time activity reminder control 1502, which displays detailed completion prompt information. If the second-form real-time activity reminder control 1502 is displayed for 5 seconds and no trigger operation is received within 5 seconds, the terminal redisplays the first-form real-time activity reminder control 1501, which displays a brief success prompt information.

[0309] When an exception occurs during the automatic multimedia playback task, the terminal displays a second-form real-time activity reminder control 1502, which includes detailed abnormality information. If the second-form real-time activity reminder control 1502 is displayed for 5 seconds and no trigger operation is received within 5 seconds, the terminal redisplays the first-form real-time activity reminder control 1501, which includes a brief abnormality information.

[0310] In this embodiment, the real-time activity reminder control supports switching between two different sizes. When displayed in the first form with a smaller size, it can reduce the obstruction of the display interface and reduce the impact of the automated task execution process on the user's current foreground operations; when displayed in the second form with a larger size, it can display more detailed real-time playback conditions and serve as a strong reminder so that users can intervene manually or eliminate abnormalities.

[0311] Linkage between real-time activity reminder control and task execution window

[0312] In some embodiments, the second form of the real-time activity reminder control not only has the function of displaying the real-time playback status, but also has the function of linking with the task execution window. In one possible implementation, the second form of the real-time activity reminder control is provided with a sub-control, and by triggering the sub-control, the task execution window can be linked accordingly.

[0313] In some embodiments, the sub-control includes a view sub-control, wherein the view sub-control is used to trigger the display of the task execution window in the foreground.

[0314] Optionally, when the second form of the real-time activity reminder control includes a view sub-control, in response to a triggering operation on the view sub-control, the terminal displays the task execution window in the foreground. The triggering operation on the view sub-control can be a click operation, a long press operation, etc., which is not limited in this embodiment.

[0315] In a possible implementation, when there is no need for manual intervention and there is no execution abnormality in the multimedia automatic playback task, the second form of the real-time activity reminder control includes a viewing sub-control.

[0316] It should be noted that when the task execution window is triggered to be displayed in the foreground by viewing the sub-control, the task execution window keeps shielding the input method function and the window mask to avoid accidental touch by the user during the viewing process.

[0317] Indicative, such as Figure 16 As shown, during the execution of the multimedia automatic playback task, a view sub-control is displayed in the second form of the real-time activity reminder control 1601. When a click operation is received on the view sub-control, the terminal pulls the task execution window 1602, which was originally in the background running state, back to the foreground running, and the interface currently displayed in the task execution window 1602 is the multimedia playback interface.

[0318] In combination with the above embodiments, the execution process of the automation task (multimedia automatic playback task) can be roughly divided into the following stages:

[0319] 1. Atomic capability discovery and script acquisition (querying the atomic capabilities that implement automated tasks and obtaining scripts for the simulation operations required to implement the atomic capabilities);

[0320] 2. Execute simulation operations (call the simulation operation execution engine and update the real-time activity reminder control based on the script execution status of the simulation operation execution engine);

[0321] 3. Check the execution process (use the real-time activity reminder control to pull the task execution window back to the foreground for display);

[0322] 4. Manual intervention process (using real-time activity reminder controls to trigger the manual intervention process and receive the user's manual intervention operation);

[0323] 5. The task is completed (the call to the simulation operation execution engine ends and the status of the task execution window is reset).

[0324] Obviously, the above-mentioned automated task execution process requires the mutual cooperation between the simulation operation execution engine, the real-time activity reminder control controller (used to control the form and display content of the real-time activity reminder control), and the window controller (used to control the task execution window). In one possible implementation, the cooperation process of the above components is scheduled by the adapter in the intention execution framework. Schematically, the interaction process between the adapter, the operation execution engine, the real-time activity reminder control controller and the window controller is as follows: Figure 17 shown.

[0325] The adapter first interacts with the atomic capability to obtain the atomic capability configuration required to implement the automation task, and then interacts with the script agent to obtain the automation execution script to implement the automation task.

[0326] The adapter calls the simulation operation execution engine to run and instructs the window controller to move the task execution window off-screen. When the simulation operation engine begins executing the simulation operation, the adapter instructs the window controller to add a mask to the task execution window and instructs the real-time activity reminder control controller to display the real-time activity reminder control.

[0327] During the execution of the automated task, the simulation operation execution engine updates the real-time execution status to the adapter, and the adapter instructs the display content of the real-time activity reminder control to be updated.

[0328] When a view operation is received through the real-time activity reminder control, the real-time activity reminder control controller instructs the window controller to move the task execution window to the screen. The window controller notifies the adapter of the change in the task execution window state, and the adapter then instructs the real-time activity reminder control to be hidden.

[0329] When the view is completed and the return operation is executed, the window controller instructs the adapter to update the background task stack. Since the task execution window is switched back to the background running state, the adapter instructs the real-time activity reminder control to be redisplayed and instructs the window controller to move the task execution window off the screen.

[0330] When there is a need for manual intervention, the adapter instructs the real-time activity reminder control to change its form to provide a strong reminder based on the manual intervention request of the simulated operation execution engine, and instructs the window controller to remove the mask of the task execution window so that the user can perform interface operations within the task execution window.

[0331] When the automated task is completed, the simulation operation execution engine notifies the adapter that the task is completed. The adapter notifies the real-time activity reminder control to display a completion prompt and instructs the window controller to reset the window state of the task execution window, that is, to restore it to a normal application window.

[0332] Please refer to Figure 18 , which shows a structural block diagram of a multimedia playback device provided by an exemplary embodiment of the present application. The device includes:

[0333] The instruction receiving module 1801 is used to receive a multimedia automatic play instruction, wherein the multimedia automatic play instruction is used to instruct the execution of a multimedia automatic play task;

[0334] An execution module 1802 is configured to execute the multimedia automatic playback task in a task execution window running in the background based on an automation execution script corresponding to the multimedia automatic playback task;

[0335] The display module 1803 is used to display the real-time playback status of the multimedia through a real-time activity reminder control during the execution of the multimedia automatic playback task.

[0336] Optionally, the execution module 1802 is configured to:

[0337] Based on the automated execution script, performing multimedia status detection on the task execution window running in the background;

[0338] When the multimedia status detection result is a pause state, based on the automatic execution script, a simulation operation is performed in the task execution window running in the background, and the simulation operation is an interface operation required to implement the multimedia automatic playback task.

[0339] Optionally, the execution module 1802 is configured to:

[0340] When the multimedia state detection result is a pause state and the current multimedia has not been played completely, executing a playback resumption operation in the task execution window running in the background based on the automated execution script, wherein the playback resumption operation is a simulated operation of resuming playback of the current multimedia;

[0341] When the multimedia status detection result is a pause state and the current multimedia playback is completed, based on the automatic execution script, a playback switching operation is performed in the task execution window running in the background. The playback switching operation is a simulation operation of switching to play the next multimedia.

[0342] Optionally, the execution module 1802 is configured to:

[0343] When the multimedia status detection result is a paused state, the current multimedia has not been played completely, and the pause reason is the first reason, based on the automated execution script, the playback resumption operation is performed in the task execution window running in the background, and the pause caused by the first reason supports recovery through the playback resumption operation.

[0344] Optionally, the display module 1803 is further configured to:

[0345] If the multimedia status detection result is a paused state, the current multimedia has not been played to completion, and the pause reason is the second reason, a manual intervention prompt is provided through the real-time activity reminder control. The pause caused by the second reason can be resumed through manual operation in the application interface;

[0346] When the multimedia status detection result is a pause state, the current multimedia has not been played completely, and the pause reason is a third reason, a playback abnormality prompt is issued through the real-time activity reminder control, and the pause caused by the third reason does not support recovery through the playback recovery operation and manual operation in the application interface.

[0347] Optionally, the display module 1803 is further configured to:

[0348] In response to the manual intervention confirmation operation in the real-time activity reminder control, the task execution window is displayed in the foreground.

[0349] Optionally, in the background running state, the input method display function of the task execution window is shielded, and the task execution window is provided with a mask layer, the mask layer being used to shield user operations;

[0350] The execution module 1802 is further configured to:

[0351] Unshielding the input method display function of the task execution window and removing the mask.

[0352] Optionally, the execution module 1802 is further configured to:

[0353] When it is recognized that the manual intervention is completed, the task execution window is switched to background operation.

[0354] Optionally, the display module 1803 is further configured to:

[0355] When the multimedia status detection result is a paused state, the current multimedia has not been played completely, and the pause reason is the first reason, a playback resumption prompt is issued through the real-time activity reminder control;

[0356] Optionally, the execution module 1802 is configured to:

[0357] In response to the playback resumption confirmation operation in the real-time activity reminder control, the playback resumption operation is executed in the task execution window running in the background based on the automation execution script.

[0358] Optionally, the execution module 1802 is configured to:

[0359] If the multimedia status detection result is a pause state and the current multimedia playback is completed, the multimedia status detection is performed again after a preset time period;

[0360] When the multimedia status detection result obtained by re-detection is a pause state, the play switching operation is performed in the task execution window running in the background based on the automatic execution script.

[0361] Optionally, the execution module 1802 is configured to:

[0362] Based on the time interval indicated by the automatic execution script, periodic multimedia status detection is performed on the task execution window running in the background.

[0363] Optionally, the display module 1803 is used to:

[0364] The real-time playback status of multimedia is displayed through the first form of real-time activity reminder control;

[0365] The real-time playing status of the multimedia is displayed through the real-time activity reminder control of the second form, and the size of the real-time activity reminder control of the first form is smaller than the size of the real-time activity reminder control of the second form.

[0366] Optionally, the display module 1803 is used to:

[0367] In response to a triggering operation of the first-form real-time activity providing control, displaying the real-time playback status of multimedia through the second-form real-time activity reminder control;

[0368] In the event of a need for manual intervention, the real-time multimedia playback status is displayed through the second form of real-time activity reminder control;

[0369] In the case of abnormal playback or completion of the multimedia automatic playback task, the real-time playback status of the multimedia is displayed through the second form of real-time activity reminder control.

[0370] Optionally, the display module 1803 is used to:

[0371] In the case where the second form of the real-time activity reminder control includes a view sub-control, in response to a triggering operation on the view sub-control, the task execution window is displayed in the foreground.

[0372] Optionally, the task execution window is provided with a mask layer, and the mask layer is used to shield user interface operations;

[0373] The mask is provided with at least one control;

[0374] The execution module 1802 is further configured to:

[0375] In response to a triggering operation on a pause control in the mask, pausing execution of the multimedia automatic playback task;

[0376] In response to a triggering operation on a recovery control in the mask, resuming execution of the multimedia automatic playback task;

[0377] In response to a triggering operation on a termination control in the mask, the multimedia automatic playback task is terminated.

[0378] Optionally, the display module 1803 is used to:

[0379] During the execution of the multimedia automatic playback task, at least one of multimedia information, multimedia playback progress, and multimedia status of the current multimedia is displayed through the real-time activity reminder control.

[0380] Optionally, the instruction receiving module 1801 is configured to:

[0381] When it is identified that the current interface belongs to the target interface, task execution prompt information is displayed, and the target interface is an interface with multimedia automatic playback requirements. The task execution prompt information is used to prompt the existence of an executable multimedia automatic playback task; and a task execution operation for the task execution prompt information is received.

[0382] Optionally, the device further includes a summary module, configured to:

[0383] In the case where the multimedia automatic play instruction has a summary requirement, performing a content summary on the multimedia during the execution of the multimedia automatic play task;

[0384] When the automated task is completed, multimedia content summary information is input.

[0385] In summary, in an embodiment of the present application, when there is a need to automatically play multimedia, the terminal can run a task execution window in the background and execute the multimedia automatic playback task in the task execution window based on the automated execution script corresponding to the multimedia automatic playback task. Since the multimedia automatic playback task is automatically executed in the task execution window running in the background, the user's manual operation process is eliminated and other applications running in the foreground are not affected. In addition, during the execution of the multimedia automatic playback task, the terminal displays the real-time playback status of the multimedia through a real-time activity reminder control, thereby reducing interference with the foreground application and improving the user's perception of the real-time execution of the task.

[0386] It should be noted that the apparatus provided in the above embodiments is merely exemplified by the division of the above functional modules. In actual applications, the above functions can be distributed among different functional modules as needed, that is, the internal structure of the apparatus can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments are based on the same concept. The implementation process is detailed in the method embodiments and will not be repeated here.

[0387] See also Figure 19 , Figure 1919 is a block diagram of a terminal provided by an exemplary embodiment of the present application. The terminal may include one or more of the following components: a processor 1910 and a memory 1920.

[0388] Optionally, the processor 1910 utilizes various interfaces and circuits to connect various parts of the entire electronic device, and executes various functions of the electronic device and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 1920, and calling data stored in the memory 1920. Optionally, the processor 1910 can be implemented in the form of at least one hardware of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA).

[0389] The processor 1910 may integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), a neural network processing unit (NPU), and a baseband chip. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the touch screen; the NPU is used to implement artificial intelligence (AI) functions; and the baseband chip is used to process wireless communications. It is understandable that the above-mentioned baseband chip may not be integrated into the processor 1910, but may be implemented separately through a single chip.

[0390] The memory 1920 may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory 1920 includes a non-transitory computer-readable storage medium. The memory 1920 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 1920 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the various method embodiments described above, and the data storage area may store data created based on the use of the electronic device.

[0391] In addition, those skilled in the art will understand that the structure of the terminal shown in the above figures does not constitute a limitation on the terminal. The terminal may include more (such as microphones, speakers, power components, display components, sensor components) or fewer components than shown in the figure, or a combination of certain components, or different component arrangements.

[0392] An embodiment of the present application provides a computer-readable storage medium, which stores at least one computer instruction. The at least one computer instruction is used to be executed by a processor to implement the multimedia playback method described in the above embodiment.

[0393] On the other hand, an embodiment of the present application provides a computer program product, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium; a processor reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to implement the multimedia playback method described in the above embodiment.

[0394] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0395] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A multimedia playback method, characterized in that: The method comprises: receiving a multimedia automatic play instruction, wherein the multimedia automatic play instruction is used to instruct execution of a multimedia automatic play task; Based on the automation execution script corresponding to the multimedia automatic playback task, executing the multimedia automatic playback task in the task execution window running in the background; During the execution of the multimedia automatic playback task, the real-time playback status of the multimedia is displayed through the real-time activity reminder control.

2. The method according to claim 1, characterized in that The step of executing the multimedia automatic playback task in a task execution window running in the background based on the automation execution script corresponding to the multimedia automatic playback task includes: Based on the automated execution script, performing multimedia status detection on the task execution window running in the background; When the multimedia status detection result is a pause state, based on the automatic execution script, a simulation operation is performed in the task execution window running in the background, and the simulation operation is an interface operation required to implement the multimedia automatic playback task.

3. The method according to claim 2, characterized in that When the multimedia status detection result is a pause state, performing a simulation operation in the task execution window running in the background based on the automated execution script includes at least one of the following: When the multimedia state detection result is a pause state and the current multimedia has not been played completely, executing a playback resumption operation in the task execution window running in the background based on the automated execution script, wherein the playback resumption operation is a simulated operation of resuming playback of the current multimedia; When the multimedia status detection result is a pause state and the current multimedia playback is completed, based on the automatic execution script, a playback switching operation is performed in the task execution window running in the background. The playback switching operation is a simulation operation of switching to play the next multimedia.

4. The method according to claim 3, characterized in that When the multimedia state detection result is a pause state and the current multimedia has not been played completely, performing a playback resumption operation in the task execution window running in the background based on the automatic execution script includes: When the multimedia status detection result is a paused state, the current multimedia has not been played completely, and the pause reason is the first reason, based on the automated execution script, the playback resumption operation is performed in the task execution window running in the background, and the pause caused by the first reason supports recovery through the playback resumption operation.

5. The method according to claim 4, characterized in that The method further comprises: If the multimedia status detection result is a paused state, the current multimedia has not been played to completion, and the pause reason is the second reason, a manual intervention prompt is provided through the real-time activity reminder control. The pause caused by the second reason can be resumed through manual operation in the application interface; When the multimedia status detection result is a pause state, the current multimedia has not been played completely, and the pause reason is a third reason, a playback abnormality prompt is issued through the real-time activity reminder control, and the pause caused by the third reason does not support recovery through the playback recovery operation and manual operation in the application interface.

6. The method according to claim 5, characterized in that The method further comprises: In response to the manual intervention confirmation operation in the real-time activity reminder control, the task execution window is displayed in the foreground.

7. The method according to claim 6, characterized in that In the background running state, the input method display function of the task execution window is shielded, and the task execution window is provided with a mask layer, the mask layer is used to shield user operations; The method further comprises: Unshielding the input method display function of the task execution window and removing the mask.

8. The method according to claim 6, characterized in that The method further comprises: When it is recognized that the manual intervention is completed, the task execution window is switched to background operation.

9. The method according to claim 4, characterized in that The method further comprises: When the multimedia status detection result is a paused state, the current multimedia has not been played completely, and the pause reason is the first reason, a playback resumption prompt is issued through the real-time activity reminder control; The performing of the playback resumption operation in the task execution window running in the background based on the automated execution script includes: In response to the playback resumption confirmation operation in the real-time activity reminder control, the playback resumption operation is executed in the task execution window running in the background based on the automation execution script.

10. The method according to claim 3, characterized in that When the multimedia state detection result is a pause state and the current multimedia playback is completed, performing a playback switching operation in the task execution window running in the background based on the automatic execution script includes: If the multimedia status detection result is a pause state and the current multimedia playback is completed, the multimedia status detection is performed again after a preset time period; When the multimedia status detection result obtained by re-detection is a pause state, the play switching operation is performed in the task execution window running in the background based on the automatic execution script.

11. The method according to claim 2, characterized in that The performing multimedia status detection on the task execution window running in the background based on the automated execution script includes: Based on the time interval indicated by the automatic execution script, periodic multimedia status detection is performed on the task execution window running in the background.

12. The method according to claim 1, characterized in that The real-time multimedia playback status is displayed through the real-time activity reminder control, including: The real-time playback status of multimedia is displayed through the first form of real-time activity reminder control; The real-time playing status of the multimedia is displayed through the real-time activity reminder control of the second form, and the size of the real-time activity reminder control of the first form is smaller than the size of the real-time activity reminder control of the second form.

13. The method according to claim 12, characterized in that The display of the real-time multimedia playback status through the second form of the real-time activity reminder control includes at least one of the following: In response to a triggering operation of the first-form real-time activity providing control, displaying the real-time playback status of multimedia through the second-form real-time activity reminder control; In the event of a need for manual intervention, the real-time multimedia playback status is displayed through the second form of real-time activity reminder control; In the case of abnormal playback or completion of the multimedia automatic playback task, the real-time playback status of the multimedia is displayed through the second form of real-time activity reminder control.

14. The method according to claim 12, characterized in that The method further comprises: In the case where the second form of the real-time activity reminder control includes a view sub-control, in response to a triggering operation on the view sub-control, the task execution window is displayed in the foreground.

15. The method according to claim 14, characterized in that The task execution window is provided with a mask layer, and the mask layer is used to shield the user interface operation; The mask is provided with at least one control; The method further comprises: In response to a triggering operation on a pause control in the mask, pausing execution of the multimedia automatic playback task; In response to a triggering operation on a recovery control in the mask, resuming execution of the multimedia automatic playback task; In response to a triggering operation on a termination control in the mask, the multimedia automatic playback task is terminated.

16. The method according to claim 1, wherein During the execution of the multimedia automatic playback task, the real-time playback status of the multimedia is displayed through the real-time activity reminder control, including: During the execution of the multimedia automatic playback task, at least one of multimedia information, multimedia playback progress, and multimedia status of the current multimedia is displayed through the real-time activity reminder control.

17. The method according to claim 1, wherein The receiving of the multimedia automatic play instruction includes: When it is identified that the current interface belongs to the target interface, task execution prompt information is displayed, and the target interface is an interface with multimedia automatic playback requirements. The task execution prompt information is used to prompt the existence of an executable multimedia automatic playback task; and a task execution operation for the task execution prompt information is received.

18. The method according to claim 1, wherein The method further comprises: In the case where the multimedia automatic play instruction has a summary requirement, performing a content summary on the multimedia during the execution of the multimedia automatic play task; When the automated task is completed, multimedia content summary information is input.

19. A multimedia playback device, characterized in that: The device comprises: An instruction receiving module is used to receive a multimedia automatic play instruction, wherein the multimedia automatic play instruction is used to instruct the execution of a multimedia automatic play task; An execution module, configured to execute the multimedia automatic playback task in a task execution window running in the background based on an automated execution script corresponding to the multimedia automatic playback task; The display module is used to display the real-time playback status of the multimedia through a real-time activity reminder control during the execution of the multimedia automatic playback task.

20. A terminal, characterized in that: The terminal includes a processor and a memory, wherein the memory stores at least one computer instruction, and the at least one computer instruction is loaded and executed by the processor to implement the multimedia playback method according to any one of claims 1 to 18.

21. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one computer instruction, and the at least one computer instruction is used to be executed by a processor to implement the multimedia playback method according to any one of claims 1 to 18.

22. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium; a processor reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to implement the multimedia playback method according to any one of claims 1 to 18.

Images