Operation processing method and apparatus, electronic device, and storage medium

Abstract

The present disclosure relates to an operation processing method and device, electronic equipment and storage medium, and belongs to the technical field of artificial intelligence. The method comprises: in response to a user input service request instruction, obtaining target service request information; based on the target service request information, controlling at least one target application program to jump to an access page corresponding to a target service; and extracting a feedback result of the target service from the access page and sending the feedback result to the user. The present disclosure can automatically control the application program to jump to the access page corresponding to the target service according to the user demand, and feed back the feedback result of the target service to the user, thereby avoiding the multi-step interaction operation of the user, improving the user experience, and providing more intelligent services.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to an operation processing method, apparatus, electronic device and storage medium. Background Technology

[0002] With the technological advancements in smart terminals, more convenient functions can be provided to users, such as pushing weather information and reminding users to pay attention to weather changes, thereby enhancing the user experience.

[0003] However, for certain user needs, such as needing to launch the application and perform operations to obtain more specific or personalized services, like wanting to know the weather two hours in advance, users still need to perform multiple interactive operations to obtain the information, resulting in user experience bottlenecks such as operational redundancy, which affects the user experience.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] To overcome the problems existing in the related technologies, this disclosure provides an operation processing method, apparatus, electronic device and storage medium.

[0006] According to a first aspect of the present disclosure, an operation processing method is provided, including:

[0007] In response to user input of service request instructions, obtain the target service request information;

[0008] Based on the target service request information, control at least one target application to redirect to the access page corresponding to the target service;

[0009] Extract the feedback result of the target service from the access page and send the feedback result to the user.

[0010] In some exemplary embodiments of this disclosure, the process of controlling at least one target application to jump to the access page corresponding to the target service includes multiple operation interfaces;

[0011] The display status of the operation interface includes: foreground display status and background running status;

[0012] In response to the display state of the operation interface being the foreground display state, the operation interface becomes visible to the user.

[0013] Some exemplary embodiments of this disclosure also include:

[0014] In response to the background running state of the operation interface, the task execution status of the operation is displayed through a floating window.

[0015] Some exemplary embodiments of this disclosure also include:

[0016] In response to the background running state of the operation interface and upon receiving the user's command to launch the target application, the operation interface is switched to the foreground display state.

[0017] In some exemplary embodiments of this disclosure, controlling at least one target application to redirect to the access page corresponding to the target service based on the target service request information includes:

[0018] Based on the target service request information, the at least one target application and the target service are determined;

[0019] The following steps will be executed repeatedly until the user is redirected to the access page corresponding to the target service:

[0020] Get a screenshot of the window instance associated with the current processing state;

[0021] The screenshot is sent to the intelligent agent so that the intelligent agent can analyze the screenshot and generate operation instructions.

[0022] Receive the operation instructions returned by the intelligent agent;

[0023] Based on the operation instructions, control the at least one target application to perform operation processing and jump to the next processing state.

[0024] In some exemplary embodiments of this disclosure, obtaining a screenshot of the current operating interface includes:

[0025] Obtain the window hierarchy list corresponding to the operating system; wherein, the operating system contains multiple windows;

[0026] Based on the window blacklist and the window hierarchy list, the multiple windows are filtered to obtain the retained windows;

[0027] A screenshot of the current operation interface is obtained by performing a screenshot based on the reserved window.

[0028] Some exemplary embodiments of this disclosure also include:

[0029] Receive the occlusion window layer information sent by the intelligent agent;

[0030] The window blacklist is updated based on the information about the obscured window layer.

[0031] In some exemplary embodiments of this disclosure, obtaining a screenshot of the current operating interface includes:

[0032] Get the list of window levels corresponding to the operating system;

[0033] The target window is determined based on the window whitelist and the window hierarchy list;

[0034] A screenshot of the current operation interface is obtained by performing a screenshot process based on the target window.

[0035] Some exemplary embodiments of this disclosure also include:

[0036] The window whitelist is determined based on the target service request information.

[0037] In some exemplary embodiments of this disclosure, the user inputs a service request instruction, including: the user inputs a service request instruction through a smart assistant;

[0038] The target service request information includes voice and / or text formats.

[0039] According to a second aspect of the present disclosure, an operation processing apparatus is provided, comprising:

[0040] The information acquisition unit is used to acquire target service request information in response to a user's input service request command;

[0041] The control unit is used to control at least one target application to jump to the access page corresponding to the target service based on the target service request information;

[0042] The service feedback unit is used to extract feedback results of the target service from the access page and send the feedback results to the user.

[0043] In some exemplary embodiments of this disclosure, the process of controlling at least one target application to jump to the access page corresponding to the target service includes multiple operation interfaces;

[0044] The display status of the operation interface includes: foreground display status and background running status;

[0045] In response to the display state of the operation interface being the foreground display state, the operation interface becomes visible to the user.

[0046] In some exemplary embodiments of this disclosure, an interactive display unit is further included, for:

[0047] In response to the background running state of the operation interface, the task execution status of the operation is displayed through a floating window.

[0048] In some exemplary embodiments of this disclosure, an interactive display unit is further included, for:

[0049] In response to the background running state of the operation interface and upon receiving the user's command to launch the target application, the operation interface is switched to the foreground display state.

[0050] In some exemplary embodiments of this disclosure, the control unit is configured to:

[0051] Based on the target service request information, the at least one target application and the target service are determined;

[0052] The following steps will be executed repeatedly until the user is redirected to the access page corresponding to the target service:

[0053] Get a screenshot of the window instance associated with the current processing state;

[0054] The screenshot is sent to the intelligent agent so that the intelligent agent can analyze the screenshot and generate operation instructions.

[0055] Receive the operation instructions returned by the intelligent agent;

[0056] Based on the operation instructions, control the at least one target application to perform operation processing and jump to the next processing state.

[0057] In some exemplary embodiments of this disclosure, the control unit is configured to:

[0058] Obtain the window hierarchy list corresponding to the operating system; wherein, the operating system contains multiple windows;

[0059] Based on the window blacklist and the window hierarchy list, the multiple windows are filtered to obtain the retained windows;

[0060] A screenshot of the current operation interface is obtained by performing a screenshot based on the reserved window.

[0061] In some exemplary embodiments of this disclosure, a list setting unit is further included, for:

[0062] Receive the occlusion window layer information sent by the intelligent agent;

[0063] The window blacklist is updated based on the information about the obscured window layer.

[0064] In some exemplary embodiments of this disclosure, the control unit is configured to:

[0065] Get the list of window levels corresponding to the operating system;

[0066] The target window is determined based on the window whitelist and the window hierarchy list;

[0067] A screenshot of the current operation interface is obtained by performing a screenshot process based on the target window.

[0068] In some exemplary embodiments of this disclosure, the control unit is configured to:

[0069] The window whitelist is determined based on the target service request information.

[0070] In some exemplary embodiments of this disclosure, the information acquisition unit is configured such that: the user inputs a service request instruction through a smart assistant;

[0071] The target service request information includes voice and / or text formats.

[0072] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0073] processor;

[0074] Memory used to store processor-executable instructions;

[0075] The processor is configured to implement the steps of any of the operation processing methods described in the first aspect above.

[0076] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, which, when instructions in the storage medium are executed by a processor of a terminal, enables the terminal to perform any of the operation processing methods described in the first aspect above.

[0077] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the operation processing methods described in the first aspect above.

[0078] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0079] This disclosure, in response to a user's input of a service request command, obtains target service request information; based on the target service request information, controls at least one target application to jump to the access page corresponding to the target service; extracts the feedback result of the target service from the access page, and sends the feedback result to the user. This allows the application to automatically control the user's request to jump to the access page corresponding to the target service requested by the user, and to provide the user with the feedback result of the target service. This avoids multiple interactive steps for the user, improves the user experience, and provides a more intelligent service.

[0080] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0081] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0082] Figure 1 This is a flowchart illustrating an operation processing method according to an exemplary embodiment of the present disclosure.

[0083] Figure 2 This is a schematic diagram of a screen display in a specific instance according to an exemplary embodiment of the present disclosure. Figure 1 .

[0084] Figure 3 This is a schematic flowchart illustrating an implementation process of step S120 according to an exemplary embodiment of the present disclosure.

[0085] Figure 4 This is an implementation flow of step S320 according to an exemplary embodiment of the present disclosure. Figure 1 .

[0086] Figure 5 This is an implementation flow of step S320 according to an exemplary embodiment of the present disclosure. Figure 2 .

[0087] Figure 6 This is a schematic diagram of the current interface of an operating system in a specific instance according to an exemplary embodiment of the present disclosure.

[0088] Figure 7 This is a schematic diagram of a screen display in a specific instance according to an exemplary embodiment of the present disclosure. Figure 2 .

[0089] Figure 8 This is a block diagram illustrating an operation processing apparatus according to an exemplary embodiment of the present disclosure.

[0090] Figure 9 This is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0091] Exemplary embodiments of this disclosure will be described in detail herein, examples of which are illustrated in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings denote the same or similar elements unless otherwise indicated. Various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but can be changed as will become apparent upon understanding this disclosure, except for operations that must be performed in a particular order. Furthermore, for clarity and brevity, descriptions of features known in the art may be omitted.

[0092] The embodiments described below, which are examples of some of the embodiments of this disclosure, do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0093] The specific implementation methods of the embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0094] Figure 1 This is a flowchart illustrating an operation processing method according to an exemplary embodiment of the present disclosure. The operation processing method can be used in electronic devices, including but not limited to terminal devices such as smartphones, wearable devices, and smart tablets, and may also include devices such as vehicles that can provide intelligent services.

[0095] Figure 1 The operation and processing method shown includes the following steps.

[0096] In step S110, in response to the user's input of a service request instruction, the target service request information is obtained.

[0097] It should be noted that the target service request information can be in voice and / or text form. This can be the user informing the electronic device of their needs via voice, for example, saying to their smartphone, "I want to know the pickup code for the package I purchased through the XX app," or a combination of voice and text, such as waking up the in-vehicle intelligent assistant by voice and saying, "I want to go to a restaurant, please make a reservation," and then entering the restaurant name on the in-vehicle screen. The electronic device can use natural language recognition, such as deploying an NLP (Natural Language Processing) model, to identify and analyze the acquired target service request information.

[0098] In the exemplary embodiments of this disclosure, users can input service request commands through a smart assistant, such as an AI assistant, voice assistant, or chatbot. For example, users can input service request commands to the smart assistant through various methods, including voice activation, long-pressing physical buttons (including but not limited to the power button, volume buttons, and side buttons), specific gestures or actions, clicking or swiping the screen, using plugins, or operating via connected Bluetooth devices.

[0099] In step S120, based on the target service request information, at least one target application is controlled to jump to the access page corresponding to the target service.

[0100] It should be noted that the target service request information represents the user's needs, the services they expect to receive, and the relevant information required to request those services, such as applications. There can be one or multiple applications. It's understandable that some services require multiple applications to work together; for example, if a user wants to dine at a specific restaurant and make a reservation in advance, the target applications could be map software and lifestyle service software.

[0101] It should be noted that, based on the target service request information, at least one target application is controlled to perform multiple redirects until the user is redirected to the access page corresponding to the target service.

[0102] In step S130, the feedback result of the target service is extracted from the access page and sent to the user.

[0103] In specific embodiments of this disclosure, the feedback result of the target service extracted from the access page can be sent to the user in various ways. For example, it can be pushed to the user via a pop-up window, sent as a notification in the notification bar, or notified to the user via voice using a smart assistant or text-to-speech (TTS) technology. To draw the user's attention, tactile alerts can also be provided by device vibration when notifying the user, indicating that the current service request has been completed and the feedback result has been communicated to the user.

[0104] As can be seen from the above steps, the operation processing method provided in the embodiments of this disclosure obtains target service request information in response to a user's input service request instruction; controls at least one target application to jump to the access page corresponding to the target service based on the target service request information; extracts the feedback result of the target service from the access page, and sends the feedback result to the user. Thus, based on the user's needs, it can automatically control the application to jump to the access page corresponding to the target service requested by the user, and provide feedback result of the target service to the user, thereby avoiding multi-step interactive operations for the user, improving the user experience, and providing a more intelligent service.

[0105] It should be noted that when obtaining target service request information, the system can also determine the operating system's foreground page. If the foreground page is a page within the target application, the system will control the operating system to perform operations and navigate based on the current page. If the foreground page is not a page within the target application, the system will control the operating system to launch the target application and perform operations and navigate based on the target application's homepage.

[0106] In some exemplary embodiments of this disclosure, the process of controlling at least one target application to jump to the access page corresponding to the target service includes multiple operation interfaces, that is, multiple execution stages may occur, each execution stage performs an operation, corresponding to one operation interface. The display state of the operation interface includes: foreground display state and background running state. It should be noted that the foreground display state indicates that the operation interface is in a visible and active state; in response to the display state of the operation interface being in the foreground display state, the operation interface is visible to the user. The background running state indicates that the execution logic of the operation interface is running, but it is not displayed in the foreground and is not visible to the user.

[0107] In some exemplary embodiments of this disclosure, users can switch the state of the operating interface through different operations, thereby enabling the operation to run in the foreground or in the background. In response to a user's first operation command, the state of the operating interface is switched from the foreground display state to the background running state, thus hiding it from the user to avoid occupying their attention, saving screen resources or operating system foreground resources, and achieving seamless operation or background automation. It should be noted that the first operation command may include: touch gestures, clicking system navigation controls, or opening other applications. For example, a user can switch the current operating interface to the background running state by swiping up or swiping left to return, by clicking the back button or "Home" button, or by opening an application other than the target application, whose page will be displayed on top of all other pages, simultaneously switching the current operating interface to the background running state.

[0108] In specific embodiments of this disclosure, when the display state of the operation interface is in a background running state, a floating window can be provided on the screen to display the task execution status in order to provide more intelligent services and let users know the task execution status. Accordingly, the operation processing method provided in the embodiments of this disclosure further includes: in response to the display state of the operation interface being in a background running state, displaying the task execution status of the operation processing through a floating window. The floating window can be displayed above the foreground application interface currently being used by the user and is located in the non-obstructed core operation area of ​​the screen. The floating window adopts a lightweight design to avoid main task operations on the screen. It can also dynamically adjust its size according to the user's screen usage and the displayed content. For example, the floating window can include an expanded state and a summary state. The user can switch between the two states by clicking. The expanded state can be a dynamic floating panel, and the summary state can be a capsule-shaped miniature icon. The floating window can be displayed in the taskbar area or the edge area of ​​the screen, and the user can also move the display position of the floating window by dragging or other operations. Figure 2 As shown, this is a screen display of an electronic device in a specific embodiment of this disclosure. When the user puts the operation in the background, a floating window will appear in the taskbar area of ​​the electronic device to display "Task in progress...".

[0109] Understandably, to avoid obscuring other areas, floating windows can adjust their background transparency and incorporate visual hierarchy. For example, they can use transparent or semi-transparent window designs, slightly blurring the content area (frosted glass effect) or adding shadows to enhance the sense of depth without obscuring it. Alternatively, a bubble interface layout can be used. This ensures the observability of background task status while minimizing interference with the user's main task.

[0110] In a specific embodiment of this disclosure, in response to a user's second operation command, the state of the operation interface is switched from a background running state to a foreground display state, thus making it visible to the user. A smooth animation effect can be used after recognizing the user's second operation command to let the user know that they are entering the foreground, ensuring a good user experience. The second operation command may include: a touch gesture, clicking a system navigation control, clicking a full-screen button in a floating window, or a command to launch a target application. Touch gestures can be pre-set, such as a four-finger swipe from top to bottom. Clicking a system navigation control can be a double-click of the power button. A full-screen button can be provided in the floating window so that clicking it changes the operation interface to the foreground. The command to launch a target application can be that the user opens the target application from the main operation screen, or it can be done by calling up a background execution thumbnail in the operating system and clicking the target application.

[0111] It should be noted that, in response to the background running state of the operation interface, the user cannot see the operation processing. However, if the user opens the target application at this time, it may interrupt the operation processing of obtaining the target service, causing the service to fail. In this case, the user's operation on the target application is not executed, and the display state of the operation interface is switched to the foreground display state to remind the user that a task is being executed. Accordingly, the operation processing method provided in this disclosure embodiment also includes: in response to the background running state of the operation interface and receiving a user's start command for the target application, switching the display state of the operation interface to the foreground display state. It should be noted that when there are multiple target applications, in response to the background running state of the operation interface, the display state of the operation interface can be switched to the foreground display state when a start command for the target application corresponding to the current operation interface is received. For example, the process of controlling at least one target application to jump to the access page corresponding to the target service based on the target service request information includes: opening application A, performing three jumps, namely the first operation interface, the second operation interface, and the third operation interface, then opening application B, performing two jumps, namely the fourth operation interface and the fifth operation interface. When the user is in the fourth operation interface stage, they click the icon of application A on the main screen. At this point, the operation and control of application A has ended, and there is no need to switch the display state of the fourth operation interface. The home page of application A can be displayed normally in the foreground of the operating system.

[0112] It should be noted that the task execution process of controlling at least one target application to jump to the access page corresponding to the target service based on the target service request information can run on a virtual screen or on the physical main screen. For example, it can be executed silently in the background environment, run visually in the foreground environment, or run in parallel through a multi-tasking window mode.

[0113] In some exemplary embodiments of this disclosure, such as Figure 3 The diagram shown is a schematic flowchart of the implementation process of step S120, which includes the following steps.

[0114] In step S310, at least one target application and target service are determined based on the target service request information.

[0115] It should be noted that the target service request information represents the user's functional needs and may include the tasks the user wants to complete, such as "ordering takeout", "navigating to the company", "transferring money to a friend", as well as key parameters in the request, such as destination, amount, time, contact person, etc. It may also include the specified application, such as map software, browser, social software, etc.

[0116] In an exemplary embodiment of this disclosure, the target service can be obtained by analyzing the target service request information and mapping the combination of "intent + entity". For example, it may include payment services, map navigation services, food delivery ordering services, video conferencing services, smart home control services, etc. Combining the target service with the analysis of the target service request information, at least one target application is determined. In specific implementations, if the target service request information includes a user-specified application, the user-specified application shall prevail. Otherwise, by combining the device environment and user habits, an application capable of executing the target service can be automatically matched, and a specific application capable of providing the target service can be determined.

[0117] Repeat steps S320 to S360 until the user is redirected to the access page corresponding to the target service.

[0118] In step S320, a screenshot of the window instance associated with the current processing state is obtained.

[0119] In step S330, the screenshot is sent to the agent so that the agent can analyze the screenshot and generate operation instructions.

[0120] In step S340, the operation instructions returned by the intelligent agent are received.

[0121] In step S350, based on the operation instructions, at least one target application is controlled to perform operation processing and jump to the next processing state.

[0122] In step S360, it is determined whether to redirect to the access page corresponding to the target service.

[0123] In some exemplary embodiments of this disclosure, the operating system is controlled to process operations on the target application. Each operation results in a processing state, which changes to the next processing state after a new operation. It is understood that multiple tasks may be executed within the operating system, and a screenshot of the window instance associated with the current processing state can be obtained. The window instance associated with the current processing state refers to the application window (Window Instance) currently displayed or running in the background of the operating system, related to the task being executed, such as a web login page or a payment confirmation pop-up. The corresponding screenshot is obtained through system-level permissions.

[0124] It should be noted that an intelligent agent is an intelligent entity capable of perceiving the environment, making autonomous decisions, and taking actions to achieve specific goals. In the embodiments of this disclosure, the intelligent agent can perform intelligent analysis based on screenshots, analyzing what operations should be performed on the application with the desired target service in mind, thereby generating operation instructions. In specific implementations, the intelligent agent can be a UI Agent (user interface agent), which is a software program or system based on artificial intelligence, especially large language models, whose core capability is to understand and operate the graphical user interface (GUI).

[0125] To further explain, UIAgent can "see" the screen: by taking screenshots, it can "perceive" the content displayed on the screen (buttons, text boxes, icons, menus, text, etc.). It can "understand" the interface: leveraging the powerful semantic understanding and reasoning capabilities of a large language model, combined with visually perceived information, it can understand the current state of the interface, available operations, and the user's goals. It can "operate" the interface: by issuing operation commands, it can simulate the user's mouse clicks, keyboard input, scrolling, dragging, and other actions, executing the steps required to complete the task on the graphical interface. It can also "think" and "decide": based on user commands, the current interface state, and built-in knowledge / strategies, it can autonomously decide the next operation and handle some unexpected situations (such as pop-ups, interface changes). UIAgent can be deployed in the cloud, enabling centralized management, elastic scaling, cross-device collaboration, and high-performance AI capability support.

[0126] In some exemplary embodiments of this disclosure, the control operating system processes the target application based on the received operation instructions. After each operation, a processing state is presented. After a new operation, the state changes to the next processing state. After each state change, a screenshot is taken and uploaded to the UI Agent so that the UI Agent is aware of the state change and receives new operation instructions from the UI Agent to process the target application, jump to the next processing state, and repeat until the processing state corresponding to the access page is reached.

[0127] It should be noted that an operating system may have multiple windows running simultaneously. These could be visible to the user in the foreground (such as pop-ups, windows, or taskbars) or invisible in the background. Furthermore, windows may obscure or stack, so directly taking a screenshot of multiple windows within the operating system might result in unclear information, obscuring important details and affecting the agent's judgment. Therefore, when taking a screenshot, it is preferable to capture the window instance associated with the current processing state.

[0128] Accordingly, such as Figure 4 The diagram shows one implementation of step S320 provided in this embodiment of the present disclosure. Figure 1 The process includes the following steps.

[0129] In step S410, the window hierarchy list corresponding to the operating system is obtained.

[0130] It's important to note that an operating system contains multiple windows. These windows are organized into a window hierarchy list according to their display hierarchy (Z-Order), reflecting the stacking order of all visible or partially visible windows in the current system, including foreground application windows, system dialog boxes, floating windows, input method windows, etc. The window hierarchy list can be obtained through window management services provided by the operating system, such as WindowManagerService or AccessibilityService in the Android operating system.

[0131] In step S420, multiple windows are filtered based on the window blacklist and the window hierarchy list to obtain the retained windows.

[0132] It should be noted that the window blacklist is pre-set and includes at least one window type or specific window identifier that is not included in the screenshot, such as: system security pop-ups (such as password input boxes, permission request dialog boxes); privacy-sensitive windows (such as health data, biometric interfaces); full-screen advertisements or third-party floating plugins; or specific application windows that users can exclude.

[0133] In practice, by matching windows in the window hierarchy list with the blacklist, windows that do not meet the screenshot conditions are filtered out, thereby selecting the windows that can be captured.

[0134] In step S430, a screenshot is taken based on the retained window to obtain a screenshot of the current operation interface.

[0135] It should be noted that if the window to be retained is a single main window, its entire interface will be captured; if there are multiple windows to retain (such as two applications in split-screen mode), their visible areas will be captured together. Post-processing of the screenshot results is also supported, such as blurring non-retained areas to protect user privacy.

[0136] By combining window hierarchy analysis with a blacklist filtering mechanism, it is possible to accurately obtain screenshots of the operation interface related to the current task while ensuring user privacy and system security.

[0137] It should be noted that the window blacklist can be preset by the system, configured by the user, or dynamically updated according to security policies. In some exemplary embodiments of this disclosure, the window blacklist can also be dynamically updated based on feedback from the intelligent agent. It is understood that when UIAgent understands the interface, it can detect occlusions, determine the corresponding occluded window, and generate corresponding information to notify the client in order to obtain a clearer screenshot without occlusions.

[0138] In specific implementation, the operation processing method provided in this disclosure embodiment further includes: receiving occlusion window layer information sent by the intelligent agent; and updating the window blacklist based on the occlusion window layer information.

[0139] It should be noted that the occlusion window layer information represents the information of occlusion windows in the screenshot as determined by the agent during analysis. Based on this information, corresponding windows can be added to the window blacklist, thereby updating the window blacklist in real time to ensure the accuracy of the screenshot and improve the accuracy and reliability of subsequent agent analysis.

[0140] Accordingly, such as Figure 5 The diagram shows one implementation of step S320 provided in this embodiment of the present disclosure. Figure 2 The process includes the following steps.

[0141] In step S510, the window hierarchy list corresponding to the operating system is obtained.

[0142] It's important to note that an operating system contains multiple windows. These windows are organized into a window hierarchy list according to their display hierarchy (Z-Order), reflecting the stacking order of all visible or partially visible windows in the current system, including foreground application windows, system dialog boxes, floating windows, input method windows, etc. The window hierarchy list can be obtained through window management services provided by the operating system, such as WindowManagerService or AccessibilityService in the Android operating system.

[0143] In step S520, the target window is determined based on the window whitelist and the window hierarchy list.

[0144] Understandably, when taking a screenshot, if the target window is clearly known, the screenshot can be taken directly. However, if the target window is uncertain or unknown, a different method can be used. Figure 4 The process shown is used to filter, or exclude, screenshots.

[0145] It should be noted that the window whitelist represents a set of windows allowed to participate in screenshots, containing one or more of the following identifiers: application package name (e.g., com.example.browser), window class name (e.g., android.app.Dialog), window type (e.g., application window TYPE_APPLICATION), and specific title or label (e.g., a page containing the words "Order Confirmation"). The filtering logic can be to iterate through the window hierarchy list, checking each window individually to see if it matches any condition in the whitelist; only windows that meet the conditions can be selected as target windows; if multiple matching windows exist, the topmost visible window can be selected based on the hierarchy order, or multiple window regions can be merged.

[0146] In some embodiments of this disclosure, the window whitelist can be known through user instructions. For example, if a user tells a smart assistant, "I want to know the pickup code for product A purchased in XX," then the window whitelist may include the XX application. Accordingly, the operation processing method provided in the embodiments of this disclosure further includes: determining the window whitelist based on target service request information.

[0147] In step S530, a screenshot is taken based on the target window to obtain a screenshot of the current operation interface.

[0148] In some exemplary embodiments of this disclosure, after determining the target window, the operating system performs a screenshot operation on the interface area corresponding to the window to generate image data. This can be done by calling the system screenshot interface (such as PixelCopy or MediaProjection) or by directly reading the buffer of the target window through a graphics compositor (such as SurfaceFlinger).

[0149] To better illustrate the operation processing method provided in the embodiments of this disclosure, a specific example is provided for further explanation.

[0150] This specific example applies to a smartphone, providing users with a "one-step access" function. Users can activate this function by calling the smart assistant. Users can wake up the smart assistant by long-pressing the power button, using voice commands to say a specific word, or using the navigation keys. Then, they can tell the smart assistant via voice or keyboard input that they want to know the logistics pickup code for their most recent order purchased through the XX e-commerce platform.

[0151] Responding to user input, the system identifies the user's target service as "Get Pickup Code" and the target application as XX application. It can be determined that the window whitelist includes XX application. The system also identifies the current operating system interface, such as... Figure 6 As shown, this is a page in the XX application. Screenshotting is performed using a window whitelist to capture the operation interface corresponding to the current processing state and upload it to the UI Agent in the cloud for analysis. After analysis, the UI Agent sends operation instructions to the smartphone. The smartphone performs simulated operations based on the operation instructions. For example, by clicking on order management and then clicking on the logistics details in the first order details, the user is redirected to an access page containing the pickup code. The pickup code "A1056" is extracted from this access page and the user is informed by voice through the smart assistant that "the pickup code is A1056".

[0152] During smartphone processing, users can use an up-swipe gesture to collapse the interface, allowing it to run in the background, and then open other applications to browse other information or perform clicks, such as... Figure 2 and Figure 7As shown, when a user clicks on another app to browse videos, an oval floating window appears in the status bar area at the top of the screen, informing the user that the service is in progress. If the user wants to check the progress, they can click on the floating window to bring the corresponding interface to the foreground. The floating window can incorporate animations (such as a rotating icon) to indicate "running" and a green checkmark to indicate "completed," enhancing readability. Furthermore, the floating window uses a lightweight oval design and is located in the status bar area at the top of the screen, without obstructing the main content area, thus avoiding affecting the user's use of the smartphone and improving the user experience. Or, as shown... Figure 7 As shown, this function is represented by a lighter, circular floating button.

[0153] As can be seen from the above process, improving user operation efficiency and realizing a "one-step access" service loop is achieved. Users only need to use a single command (such as "Check the pickup code of the package I just ordered") to trigger the complete process from application activation, page jump, data extraction to result feedback; users do not need to manually open the app or click through the search, greatly reducing the operation steps and truly realizing a "hands-free" intelligent interactive experience.

[0154] Precise screenshotting based on a whitelist mechanism ensures privacy and security, capturing only the interface of the target application (XX App). The system automatically filters out other irrelevant or sensitive windows (such as banking apps and SMS interfaces), thus preventing the leakage of user privacy information during background automation and complying with data security standards such as GDPR and CCPA. By supporting seamless switching between foreground and background, it balances task continuity with user freedom. Smartphones can silently perform UI operations in the background (such as clicking on an order or entering a logistics details page) while allowing users to freely use other applications. A floating window in the status bar provides real-time feedback on task progress, keeping users aware of the task. Users can restore the foreground interface at any time by clicking the floating window to check the execution progress, achieving a balance between "seamless execution and controllable visibility." It breaks through the limitation of traditional automation requiring exclusive screen time, improving the multi-tasking concurrent experience.

[0155] By collaborating with the terminal through a cloud-based UI Agent, intelligent decision-making and reliable execution are achieved. After a screenshot is uploaded to the cloud-based UI Agent, the interface structure is analyzed using large models such as visual recognition, OCR, and semantic understanding to intelligently generate operation paths. Instructions are then sent to the terminal device to execute simulated clicks, swipes, and other operations. It has cross-version adaptation capabilities, and even if the App interface is updated, it can still continue to run through visual understanding, forming a closed-loop intelligent agent architecture of "terminal perception → cloud decision-making → terminal execution", which improves the robustness and generalization ability of the automation system.

[0156] It should be noted that the acquisition, storage, use, and processing of information or data in this disclosed technical solution comply with the relevant provisions of national laws and regulations.

[0157] Figure 8 This is a block diagram of an operation processing apparatus illustrated according to some exemplary embodiments of the present disclosure.

[0158] Reference Figure 8 The device 800 includes: an information acquisition unit 801, a control unit 802, and a service feedback unit 803.

[0159] The information acquisition unit 801 is used to acquire target service request information in response to a user input service request command;

[0160] The control unit 802 is used to control at least one target application to jump to the access page corresponding to the target service based on the target service request information;

[0161] The service feedback unit 803 is used to extract the feedback result of the target service from the access page and send the feedback result to the user.

[0162] In some exemplary embodiments of this disclosure, the information acquisition unit 801 is configured such that a user inputs a service request instruction through a smart assistant. It should be noted that the target service request information includes voice and / or text formats.

[0163] In some exemplary embodiments of this disclosure, the process of controlling at least one target application to jump to the access page corresponding to the target service includes multiple operation interfaces;

[0164] The display status of the user interface includes: foreground display status and background running status;

[0165] The interface is displayed in the foreground and is visible to the user.

[0166] In some exemplary embodiments of this disclosure, it further includes: an interactive display unit, configured to display the task execution status of the operation processing through a floating window in response to the display state of the operation interface being a background running state.

[0167] In some exemplary embodiments of this disclosure, the interactive display unit is configured to: in response to the display state of the operation interface being in a background running state and receiving a user's launch command for the target application, switch the display state of the operation interface to a foreground display state.

[0168] In some exemplary embodiments of this disclosure, the control unit 802 is configured to:

[0169] Based on the target service request information, identify at least one target application and one target service;

[0170] Repeat the following steps until you are redirected to the access page corresponding to the target service:

[0171] Get a screenshot of the window instance associated with the current processing state;

[0172] Send the screenshot to the agent so that the agent can analyze the screenshot and generate operation instructions.

[0173] Receive operation instructions returned by the intelligent agent;

[0174] Based on the operation instructions, control at least one target application to perform operation processing and jump to the next processing state.

[0175] In some exemplary embodiments of this disclosure, the control unit 802 is configured to:

[0176] Retrieve the list of window levels corresponding to the operating system; the operating system contains multiple windows.

[0177] Based on the window blacklist and the window hierarchy list, multiple windows are filtered to obtain the retained windows;

[0178] The screenshot is taken based on the preserved window to obtain a screenshot of the current operation interface.

[0179] In some exemplary embodiments of this disclosure, a list setting unit is further included, for:

[0180] Receive occlusion window layer information sent by the intelligent agent;

[0181] Update the window blacklist based on the information of the occluded window layer.

[0182] In some exemplary embodiments of this disclosure, the control unit 802 is configured to:

[0183] Get the list of window levels corresponding to the operating system;

[0184] The target window is determined based on the window whitelist and the window hierarchy list;

[0185] The screenshot is taken based on the target window to obtain a screenshot of the current operating interface.

[0186] In some exemplary embodiments of this disclosure, the control unit 802 is configured to:

[0187] Determine the window whitelist based on the target service request information.

[0188] Regarding the apparatus in the above embodiments, the specific manner in which each unit performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0189] Figure 9 This is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. For example, device 900 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness device, personal digital assistant, etc.

[0190] Reference Figure 9 The device 900 may include one or more of the following components: a processing component 902, a memory 904, a power supply component 906, a multimedia component 908, an audio component 910, an input / output (I / O) interface 912, a sensor component 914, and a communication component 916.

[0191] Processing component 902 typically controls the overall operation of device 900, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 902 may include one or more processors 920 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 902 may include one or more modules to facilitate interaction between processing component 902 and other components. For example, processing component 902 may include a multimedia module to facilitate interaction between multimedia component 908 and processing component 902.

[0192] Memory 904 is configured to store various types of data to support the operation of device 900. Examples of this data include instructions for any application or method operating on device 900, contact data, phonebook data, messages, pictures, videos, etc. Memory 904 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0193] Power supply component 906 provides power to various components of device 900. Power supply component 906 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 900.

[0194] Multimedia component 908 includes a screen that provides an output interface between the device 900 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 908 includes a front-facing camera and / or a rear-facing camera. When the device 900 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0195] Audio component 910 is configured to output and / or input audio signals. For example, audio component 910 includes a microphone (MIC) configured to receive external audio signals when device 900 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 904 or transmitted via communication component 916. In some embodiments, audio component 910 also includes a speaker for outputting audio signals.

[0196] I / O interface 912 provides an interface between processing component 902 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0197] Sensor assembly 914 includes one or more sensors for providing status assessments of various aspects of device 900. For example, sensor assembly 914 may detect the on / off state of device 900, the relative positioning of components such as the display and keypad of device 900, changes in position of device 900 or a component of device 900, the presence or absence of user contact with device 900, orientation or acceleration / deceleration of device 900, and temperature changes of device 900. Sensor assembly 914 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 914 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 914 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0198] Communication component 916 is configured to facilitate wired or wireless communication between device 900 and other devices. Device 900 can access wireless networks based on communication standards, such as WiFi, 3G, 4G, 5G, other communication standards, or combinations thereof. In some embodiments of this disclosure, communication component 916 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In some embodiments of this disclosure, communication component 916 further includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0199] In some embodiments of this disclosure, the apparatus 900 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0200] In some embodiments of this disclosure, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 904 including instructions that can be executed by a processor 920 of device 900 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0201] In some embodiments of this disclosure, a non-transitory computer-readable storage medium is provided, which, when the instructions in the storage medium are executed by the processor of a mobile terminal, enables the mobile terminal to perform the above-described operation processing method.

[0202] In some embodiments of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described operation processing method.

[0203] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0204] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An operation processing method, characterized in that, include: In response to user input of service request instructions, obtain the target service request information; Based on the target service request information, control at least one target application to redirect to the access page corresponding to the target service; Extract the feedback result of the target service from the access page and send the feedback result to the user.

2. The operation processing method according to claim 1, characterized in that, The process of controlling at least one target application to redirect to the access page corresponding to the target service includes multiple operation interfaces; The display status of the operation interface includes: foreground display status and background running status; In response to the display state of the operation interface being the foreground display state, the operation interface becomes visible to the user.

3. The operation processing method according to claim 2, characterized in that, Also includes: In response to the background running state of the operation interface, the task execution status of the operation is displayed through a floating window.

4. The operation processing method according to claim 2, characterized in that, Also includes: In response to the background running state of the operation interface and upon receiving the user's command to launch the target application, the operation interface is switched to the foreground display state.

5. The operation processing method according to claim 1, characterized in that, Based on the target service request information, control at least one target application to redirect to the access page corresponding to the target service, including: Based on the target service request information, the at least one target application and the target service are determined; The following steps will be executed repeatedly until the user is redirected to the access page corresponding to the target service: Get a screenshot of the window instance associated with the current processing state; The screenshot is sent to the intelligent agent so that the intelligent agent can analyze the screenshot and generate operation instructions. Receive the operation instructions returned by the intelligent agent; Based on the operation instructions, control the at least one target application to perform operation processing and jump to the next processing state.

6. The operation processing method according to claim 5, characterized in that, Capture a screenshot of the current user interface, including: Obtain the window hierarchy list corresponding to the operating system; wherein, the operating system contains multiple windows; Based on the window blacklist and the window hierarchy list, the multiple windows are filtered to obtain the retained windows; A screenshot of the current operation interface is obtained by performing a screenshot based on the reserved window.

7. The operation processing method according to claim 6, characterized in that, Also includes: Receive the occlusion window layer information sent by the intelligent agent; The window blacklist is updated based on the information about the obscured window layer.

8. The operation processing method according to claim 5, characterized in that, Capture a screenshot of the current user interface, including: Get the list of window levels corresponding to the operating system; The target window is determined based on the window whitelist and the window hierarchy list; A screenshot of the current operation interface is obtained by performing a screenshot process based on the target window.

9. The operation processing method according to claim 8, characterized in that, Also includes: The window whitelist is determined based on the target service request information.

10. The operation processing method according to any one of claims 1 to 9, characterized in that, User input service request commands, including: user input service request commands through a smart assistant; The target service request information includes voice and / or text formats.

11. An operation processing apparatus, characterized in that, include: The information acquisition unit is used to acquire target service request information in response to a user's input service request command; The control unit is used to control at least one target application to jump to the access page corresponding to the target service based on the target service request information; The service feedback unit is used to extract feedback results of the target service from the access page and send the feedback results to the user.

12. The operation processing apparatus according to claim 11, characterized in that, The process of controlling at least one target application to redirect to the access page corresponding to the target service includes multiple operation interfaces; The display status of the operation interface includes: foreground display status and background running status; In response to the display state of the operation interface being the foreground display state, the operation interface becomes visible to the user.

13. The operation processing apparatus according to claim 12, characterized in that, Also includes: Interactive display unit, used for: In response to the background running state of the operation interface, the task execution status of the operation is displayed through a floating window.

14. The operation processing apparatus according to claim 12, characterized in that, Also includes: Interactive display unit, used for: In response to the background running state of the operation interface and upon receiving the user's command to launch the target application, the operation interface is switched to the foreground display state.

15. The operation processing apparatus according to claim 11, characterized in that, The control unit is configured to: Based on the target service request information, the at least one target application and the target service are determined; The following steps will be executed repeatedly until the user is redirected to the access page corresponding to the target service: Get a screenshot of the window instance associated with the current processing state; The screenshot is sent to the intelligent agent so that the intelligent agent can analyze the screenshot and generate operation instructions. Receive the operation instructions returned by the intelligent agent; Based on the operation instructions, control the at least one target application to perform operation processing and jump to the next processing state.

16. The operation processing apparatus according to claim 15, characterized in that, The control unit is configured to: Obtain the window hierarchy list corresponding to the operating system; wherein, the operating system contains multiple windows; Based on the window blacklist and the window hierarchy list, the multiple windows are filtered to obtain the retained windows; A screenshot of the current operation interface is obtained by performing a screenshot based on the reserved window.

17. The operation processing apparatus according to claim 16, characterized in that, Also includes: List setting unit, used for: Receive the occlusion window layer information sent by the intelligent agent; The window blacklist is updated based on the information about the obscured window layer.

18. The operation processing apparatus according to claim 15, characterized in that, The control unit is configured to: Get the list of window levels corresponding to the operating system; The target window is determined based on the window whitelist and the window hierarchy list; A screenshot of the current operation interface is obtained by performing a screenshot process based on the target window.

19. The operation processing apparatus according to claim 18, characterized in that, The control unit is configured to: The window whitelist is determined based on the target service request information.

20. The operation processing apparatus according to any one of claims 11 to 19, characterized in that, The information acquisition unit is configured to allow the user to input a service request command through a smart assistant; The target service request information includes voice and / or text formats.

21. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the operation processing method according to any one of claims 1 to 10.

22. A non-transitory computer-readable storage medium, wherein when instructions in the storage medium are executed by a processor of a terminal, the terminal is enabled to perform the steps of an operation processing method according to any one of claims 1 to 10.

23. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the operation processing method as described in any one of claims 1 to 10.

Images