Interface display method and device, equipment and storage medium

Abstract

The invention discloses an interface display method and device, equipment and a storage medium. The method comprises the steps that an image of a current display interface of electronic equipment and a target task currently executed by the electronic equipment are acquired; the image and task information of the target task are input into a target model, a control instruction output by the target model is obtained, and the control instruction is used for operating a current display interface to complete the target task; determining a first area corresponding to the control instruction in the current display interface; displaying the first area and the second area with different display effects; the second area is the other area except the first area in the current display interface.

Description

Technical Field

[0001] This application relates to the field of display technology, and in particular to an interface display method, apparatus, device, and storage medium. Background Technology

[0002] Currently, if it is necessary to automatically control an application in an electronic device, it is generally necessary to obtain the interface and permissions of the application to be controlled, and then control the program based on the underlying logic. This method varies for different applications, is complex and cumbersome, and has low efficiency. Furthermore, when the electronic device is being controlled, the screen changes, which can easily make the user unclear about the current status and unable to clearly display the operation being performed, resulting in low interface readability. Summary of the Invention

[0003] In view of this, this application provides an interface display method, apparatus, device, and storage medium.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] This application provides a method for displaying an interface, the method including:

[0006] Acquire an image of the current display interface of the electronic device, as well as the target task that the electronic device is performing;

[0007] Input the image and the task information of the target task into the target model, obtain the control instructions output by the target model, and use the control instructions to operate the current display interface to complete the target task;

[0008] Determine the first region corresponding to the control command within the current display interface;

[0009] The first and second areas are displayed with different display effects; the second area is the other areas in the current display interface besides the first area.

[0010] In the above interface display method, determining the first area corresponding to the control instruction within the current display interface includes at least one of the following: determining the area where the interactive control controlled by the control instruction is located within the current display interface as the first area; determining the window area corresponding to the interactive control controlled by the control instruction within the current display interface as the first area; or determining the area within the current display interface within a preset range centered on the operation position corresponding to the control instruction as the first area.

[0011] In the above interface display method, displaying the first area and the second area with different display effects includes: displaying the first area with a first display effect and displaying the second area with a second display effect; wherein the transparency corresponding to the first display effect is greater than the transparency corresponding to the second display effect; or, the brightness corresponding to the first display effect is greater than the brightness corresponding to the second display effect.

[0012] In the above interface display method, after obtaining the control command output by the target model, the method further includes: determining the control action indicated by the control command; and outputting prompt information corresponding to the control action.

[0013] In the above interface display method, the output of prompt information corresponding to the control action includes at least one of the following: outputting an animation corresponding to the control action within the current display interface; controlling the physical device corresponding to the control action to prompt the control operation in the form of light; outputting sound information corresponding to the control operation; and inputting text description information corresponding to the control operation.

[0014] In the above interface display method, outputting the animation corresponding to the control action within the current display interface includes: determining the physical input device corresponding to the control action; displaying the device image information corresponding to the physical input device on the current display interface; and controlling the device image information to present dynamic effects according to the control action.

[0015] In the above interface display method, the output sound information corresponding to the control operation includes at least one of the following: determining analog sound information based on the physical input device corresponding to the control action, the analog sound information representing the sound generated by the physical input device when it is operated, and outputting the analog sound information; obtaining the voice broadcast information corresponding to the control action and outputting the voice broadcast information.

[0016] This application provides an interface display device, including:

[0017] The acquisition module is used to acquire the image of the current display interface of the electronic device, as well as the target task that the electronic device is performing;

[0018] The decision module is used to input the image and the task information of the target task into the target model, obtain the control instructions output by the target model, and use the control instructions to operate the current display interface to complete the target task.

[0019] The determination module is used to determine the first area corresponding to the control command within the current display interface;

[0020] The display module is used to display the first area and the second area with different display effects; the second area is the area other than the first area within the current display interface.

[0021] An embodiment of the application provides an electronic device, including: a processor, a memory, and a communication bus;

[0022] A communication bus is used to establish communication connections between the processor and memory.

[0023] The processor is used to execute computer programs stored in memory to implement the above-described interface display method.

[0024] The application provides a computer-readable storage medium that stores one or more computer programs, which can be executed by one or more processors to implement the above-described interface display method.

[0025] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the technical solutions provided in the embodiments of this application. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort, wherein:

[0027] Figure 1 A flowchart illustrating an interface display method provided in an embodiment of this application;

[0028] Figure 2 A schematic diagram illustrating an exemplary process for determining a first region, provided for an embodiment of this application;

[0029] Figure 3 A schematic diagram illustrating an exemplary process for outputting prompt information, provided as an embodiment of this application;

[0030] Figure 4 A schematic diagram illustrating an exemplary method of outputting information, provided for an embodiment of this application;

[0031] Figure 5 A flowchart illustrating an exemplary animation display method provided in this application embodiment;

[0032] Figure 6 A schematic diagram of an exemplary mouse icon provided for an embodiment of this application;

[0033] Figure 7 A simplified schematic diagram of an exemplary keyboard provided for embodiments of this application;

[0034] Figure 8A schematic diagram illustrating an exemplary method of outputting sound information, provided as an embodiment of this application;

[0035] Figure 9 This is a schematic diagram of the structure of an interface display device provided in an embodiment of this application;

[0036] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0037] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It is understood that the specific embodiments described herein are merely for explaining the relevant application and not for limiting the application. Furthermore, it should be noted that, for ease of description, only the parts relevant to the application are shown in the accompanying drawings.

[0038] In the following description, references to "some embodiments" refer to a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same or different subsets of all possible embodiments and may be combined with each other without conflict. Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiments of this application pertain. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit the application. It should also be noted that, for ease of description, only the parts relevant to the application are shown in the accompanying drawings.

[0039] Large Action Models (LAMs) are an important research and application area in the fields of large models, artificial intelligence, and AI PCs. How to ensure the efficient and reasonable operation of LAMs is a crucial aspect of their application. Currently, in applications where LAMs control keyboard or mouse input, the screen automatically changes, which can easily lead to users being unclear about the current state and the ongoing operations of the LAM.

[0040] Existing solutions include: LAM directly controlling keyboard and mouse operations without prompts, or highlighting the currently interacting AI. Neither of these solutions allows users to clearly understand the operations that LAM is performing.

[0041] To address the aforementioned problems, this application provides an interface display method implemented by an electronic device, such as... Figure 1 As shown, the process includes the following steps S101 to S104:

[0042] Step S101: Obtain the image of the current display interface of the electronic device and the target task being performed by the electronic device.

[0043] In the embodiments of this application, the electronic device is a device with an interface display function, which may be a tablet computer, a laptop computer, a handheld computer, a personal digital assistant (PDA), a desktop computer, etc. No specific electronic device is limited here.

[0044] In the embodiments of this application, the electronic device can acquire an image of the current display interface of the electronic device through a camera, or take a screenshot of the current display interface using a screenshot tool, or capture the image of the current display interface in real time. Of course, the image of the current display interface can also be acquired in other forms.

[0045] In the embodiments of this application, the target task can be a task being performed by the electronic device. For example, the target task can be "sending an email", "buying a high-speed rail ticket", "submitting a leave application", etc.

[0046] In embodiments of this application, the acquisition of the target task can rely on user input, speech recognition, task planning module output, or recommendations based on information obtained from image recognition. For example, if a ticket-buying website is opened, the target task can be directly determined to be buying tickets.

[0047] Step S102: Input the image and the task information of the target task into the target model, and obtain the control instructions output by the target model. The control instructions are used to operate the current display interface to complete the target task.

[0048] In the embodiments of this application, the image of the currently displayed interface can help the target model understand the current interface context more accurately, and the target task can help the target model understand the user intent more accurately. By inputting the image and the task information of the target task into the target model, the relationship between the current interface context and the user intent can be known.

[0049] In the embodiments of this application, the target model can be a decision model built based on artificial intelligence technology, such as a large language model or a multimodal large model. The target model can integrate multiple technologies such as image recognition, natural language processing, and task planning. Based on the content of the current display interface of the electronic device and the user's task requirements, it outputs operation instructions, such as clicking the 'OK' button, entering 'hello world', selecting the third option, and other corresponding keyboard and mouse operation instructions, including action information representing clicking the left mouse button and the coordinate information of the mouse click position in the interface.

[0050] For example, the target model is used to map the image of the current display interface of the electronic device to the information of the target task that the electronic device is performing as a specific interactive behavior.

[0051] The task information for the target task could be: "Send an email to A", "Buy a high-speed rail ticket for U from S to M with time L", "Submit a leave application for n for time period o", etc.

[0052] In the embodiments of this application, if the image displays a ticket purchase website, based on the image corresponding to the ticket purchase website and the task information "buy a high-speed rail ticket for U from S to M with time L", the control instructions output by the target model are: click the target position on the current display interface, which corresponds to the icon of the ticket purchase entrance, or move the scroll bar from the first position to the second position, which corresponds to the different train schedules displayed on the browsing page on the current display interface.

[0053] Step S103: Determine the first area corresponding to the control command within the current display interface.

[0054] In the embodiments of this application, a region corresponding to the control command within the current display interface is determined, and this region is designated as the first region. The first region is the region within the current display interface associated with the control command output by the target model.

[0055] For example, if the control command output by the target model is: click the target location on the current display interface, then the area where the ticket purchase entrance icon is located at the target location is the first area.

[0056] For example, if the control command output by the target model is to move the scroll bar from the first position to the second position, then the area corresponding to the movement of the scroll bar in the current display interface is the first area.

[0057] Step S104: Display the first area and the second area with different display effects; the second area is the area other than the first area in the current display interface.

[0058] In the embodiments of this application, a first area and a second area are displayed with different display effects. The second area is the area within the current display interface other than the first area.

[0059] For example, the second region can be masked, blurred, darkened, etc., while the first region remains unchanged, or the first and second regions can be changed in different directions. For example, the first region can be darkened while the second region can be highlighted, the first region can be made clear while the second region is blurred, and so on.

[0060] In this way, control commands can be directly determined based on the target model through the images and task information of the current interface. This allows for control of the target program executing the target task based on the control commands, without the need for different processing for different applications, thus enhancing applicability. Furthermore, different display effects are used for the controlled and uncontrolled areas, making it clear that the current operation is in progress and improving the readability of the current display interface.

[0061] In some embodiments, when the electronic device performs the above step S103, such as Figure 2 Alternatively, the following steps S201 can be performed:

[0062] Step S201: Determine the area where the interactive control controlled by the control instruction is located within the currently displayed interface as the first area.

[0063] In the embodiments of this application, interactive controls refer to graphical components that allow users to interact with electronic devices, such as buttons, input boxes, drop-down menus, etc. Interactive controls typically have a clear operational intent and functional direction. The area where an interactive control is located refers to the position of the interactive control on the screen and the spatial range occupied by the interactive control.

[0064] In embodiments of this application, the electronic device can define the area where the interactive control is located as a first area, so that the user can accurately locate the target object currently being operated on.

[0065] In this way, by setting specific interaction logic, the target object of the current automatic control operation can be accurately identified, thereby focusing attention on providing status prompts to the first area, which can improve the user experience and trust.

[0066] In other embodiments, when the electronic device performs step S103 as described above, such as Figure 2 Alternatively, the following steps S202 can be performed:

[0067] Step S202: Determine the window area corresponding to the interactive control controlled by the control instruction within the current display interface as the first area.

[0068] In the embodiments of this application, the window area refers to the overall space occupied by the application window to which the interactive control belongs on the screen. Each application displays content in the form of a window when it runs, and the window area includes all controls, title bars, borders and other content related to the window area.

[0069] In the embodiments of this application, by defining the entire area of ​​the window where the interactive control is located as the first area, the electronic device can highlight the currently automatically executed operation in a wider range, prevent the user from mistakenly thinking that the operation occurred in another location, and also improve the readability of the display interface corresponding to the current operation.

[0070] In the embodiments of this application, considering that the window corresponding to the interactive control occupies the entire screen, for example, if the current target task is to buy tickets, and the ticket buying website displayed on the current display interface occupies the entire screen, then the electronic device will not display the entire website window. For example, if the current action is to slide the slider, then only the area displaying the schedule related to the slider will be displayed, and other content such as the title bar and borders will not belong to the first area.

[0071] In the embodiments of this application, considering that displaying the area where the interactive control is located may affect the readability of other data in the window, the window area corresponding to the interactive control is defined as the first area for explicit display. This allows users to understand the current operation context more intuitively, thereby effectively reducing the risk of misoperation and enhancing users' trust in the automated process.

[0072] In some other embodiments, when the electronic device performs step S103 as described above, such as Figure 2 Alternatively, the following steps S203 can be performed:

[0073] Step S203: Determine the area within a preset range centered on the operation position corresponding to the control command within the currently displayed interface as the first area.

[0074] In the embodiments of this application, the operation position refers to the specific coordinate point of the operation (such as clicking, swiping, dragging, etc.). The electronic device calculates and determines a center point as a reference point based on the operation position, and expands a rectangular or circular area around the center point as a preset range. This preset range is used to define the boundary of the first area. The size of the preset range can be set according to actual needs, such as 50×50 pixels, 100×100 pixels, etc.

[0075] In the embodiments of this application, the electronic device can flexibly adapt to different types of interaction scenarios without relying on specific controls or windows by defining an area centered on the operation position, thereby improving the versatility and applicability of the method of the present invention.

[0076] In this way, by introducing configurable operation modules and dynamic interface layout mechanisms, it can more flexibly adapt to different operation types and interface layouts, thereby supporting a wider range of application scenarios and significantly improving the versatility and applicability of the solution.

[0077] In this embodiment of the application, by subdividing the first area corresponding to the control command into three types—the area where the interactive control is located, the window area to which the control belongs, and the preset range area centered on the operation position—the interface part that LAM is operating can be described more comprehensively and accurately. This method of subdividing the first area into three types can effectively distinguish the current operation from other unrelated interface elements. This method can improve the user's perception of the LAM operation process and enhance the user's trust and satisfaction with automation.

[0078] In some embodiments, when the electronic device performs step S104, it may also perform the following steps: displaying a first area with a first display effect and displaying a second area with a second display effect; wherein the transparency corresponding to the first display effect is greater than the transparency corresponding to the second display effect; or, the brightness corresponding to the first display effect is greater than the brightness corresponding to the second display effect.

[0079] In the embodiments of this application, the first display effect displays a first area, and the second display effect displays a second area in order to distinguish between the first area and the second area, namely the area where LAM is operating and the non-interactive area. By using different display attributes, users can more intuitively identify the area currently in the automated control state. Setting different display attributes also improves users' awareness of the electronic device's status and enhances users' trust in the operation of LAM.

[0080] In the embodiments of this application, transparency refers to the degree of opacity of screen display elements; a higher value indicates clearer visibility. Brightness, on the other hand, refers to the brightness of screen pixels. Increasing the transparency or brightness of the first area to make it more visually prominent allows users to easily determine which parts are being automatically operated by LAM, effectively reducing the risk of accidental operation.

[0081] For example, when the LAM performs a mouse click or keyboard input, the area containing the interactive control can be set to high transparency or high brightness, while the surrounding non-interactive areas are masked, making them darker or blurry. This ensures that only the area of ​​the LAM being used remains clearly visible on the screen. This display method guides the user's visual attention, allowing them to focus more on the current operation.

[0082] Furthermore, distinguishing the first and second areas by different display effects is more natural and intuitive than traditional methods such as highlighting frames, and is more suitable for automated task scenarios.

[0083] In some embodiments, after the electronic device executes the "obtain control command output by the target model" step S102 described above, as follows: Figure 3As shown, the following steps S301 and S302 can also be performed:

[0084] Step S301: Determine the control action indicated by the control command.

[0085] In the embodiments of this application, control commands output by the LAM are parsed and identified to clarify the user interface operation corresponding to the intent of the control commands. For example, control actions may include clicking the left, right, or middle mouse button, or inputting a specific keyboard key combination. By identifying specific control actions, a basis can be provided for generating subsequent prompts, thereby allowing the user to clearly understand the type of operation currently being performed by the LAM.

[0086] In the embodiments of this application, the electronic device determines the corresponding control action based on the control command. For example, if the control command output by the target model is: click the target position on the current display interface, then the control action is: click the left mouse button at the target position on the current display interface. The target position corresponds to an icon that can be used as a ticket purchase entry, indicating that the user is entering the ticket purchase page.

[0087] For example, if the control command output by the target model is: move the slider from the first position to the second position, then the control action is: click the left mouse button at the "sliding bar" and slide the middle scroll wheel to move the slider from the first position to the second position.

[0088] Step S302: Output the prompt information corresponding to the control action.

[0089] In embodiments of this application, the electronic device can output prompt information corresponding to the control action.

[0090] In this step, the electronic device generates and displays corresponding prompts based on the control action determined in step S301. The prompts can take the form of graphical icons (such as simplified icons of the left / right / middle mouse button / keyboard), text descriptions, animations, or sound alerts. These prompts will be displayed on the user interface in real time so that the user can intuitively understand the operation being performed by the LAM.

[0091] For example, when the LAM is about to simulate clicking the left mouse button, the electronic device can display a left mouse button icon near the operating area; when the LAM is about to enter a password, the electronic device can display a keyboard icon near the keyboard area, and the electronic device may also provide text prompts to indicate that a password is being entered.

[0092] In this embodiment of the application, by outputting prompts corresponding to the control actions, the current operation content of LAM can be displayed intuitively. By adopting the method of outputting prompts corresponding to the control actions, users' confusion about the automatic control status of the system can be reduced, thereby significantly reducing the risk of misoperation, and thus improving the availability of the system and user satisfaction.

[0093] In this way, by determining the control actions indicated by the control commands and outputting corresponding prompts, a clear and visual presentation of the LAM's operating status can be achieved. This method effectively distinguishes LAM operations from other user operations, allowing the control system to prevent users from unknowingly interfering with the automated process, thereby improving the overall system stability and user experience.

[0094] In some embodiments, when the electronic device performs step S302 as described above, such as Figure 4 Alternatively, the following steps S401 can be performed:

[0095] Step S401: Output the animation corresponding to the control action within the current display interface.

[0096] In the embodiments of this application, the electronic device outputs an animation corresponding to the control action within the current display interface.

[0097] In the embodiments of this application, animation refers to displaying the control action of the automated operation as a dynamic image on the display interface while the user is performing an automated operation. For example, when the automated operation is clicking the left mouse button, a blinking or scaling button animation can be displayed near the target control; when the automated operation is a drag control action, a drag trajectory line with an arrow can be displayed. These animations corresponding to the control actions not only intuitively reflect the current control action type, but also enhance the user's attention and increase the user's trust in the system.

[0098] In the embodiments of this application, the animation corresponding to the output control action in the current display interface can be displayed in a first area or a second area. If it is displayed in the second area, the display effect of the animation is different from the second display effect.

[0099] In the embodiments of this application, the presentation of animations corresponding to control actions can be diversified, such as scaling effects, highlighting and flashing, movement trajectories, etc., depending on the control action being performed. The design of the animations corresponding to the control actions should ensure that they do not interfere with the user's observation of the main interface elements, while clearly conveying the intent of the control action. Dynamic feedback mechanisms are more attractive to users than static prompts, and they help users understand that the system is currently in an automatic operation state.

[0100] In this way, through a visual interface, the system can display the interactive control actions being performed by the automated operating system to the user in real time. The way the system presents this information helps improve the user experience and further enhances the user's trust in the automated system.

[0101] In another embodiment, when the electronic device performs step S302 as described above, such as Figure 4 Alternatively, the following steps S402 can be performed:

[0102] Step S402: Control the physical device corresponding to the control action and prompt the control operation with light.

[0103] In the embodiments of this application, the electronic device can control the physical device corresponding to the action and prompt the control operation in the form of light.

[0104] For example, if the control action of an electronic device is clicking the left mouse button and scrolling the wheel, then the left mouse button or scrolling wheel will light up to indicate that the left mouse button is being clicked and the wheel is being scrolled. If the electronic device is a laptop, then the touchpad and the left touchpad button will light up accordingly. If the control action of the electronic device is clicking specific keys on the keyboard, then the corresponding keys will also light up. It should be noted that other keys that are not being used will remain unlit.

[0105] In the embodiments of this application, the control operation is indicated by operating a physical device corresponding to the control action and using light to indicate the control operation. This can be achieved by using changes in LED lights on external hardware devices (such as keyboards, mice, smart light strips, etc.) to indicate that an automated operation is performing a certain control action.

[0106] For example, when the left mouse button is pressed during an automated operation, the indicator light corresponding to the left mouse button will briefly illuminate; when the automated operation performs keyboard input, the backlight color on the keyboard may change or flash. This method of using LED lighting changes on external hardware devices further enhances the user's awareness of the current control action status through physical feedback.

[0107] In the embodiments of this application, the control operation is indicated by lights, which can distinguish different types of control actions through different colors, frequencies, and positions. For example, a red light may represent a click control action, a blue light may represent a swipe control action, and a green light may represent the completion of a control action. Using lights to indicate control operations is particularly suitable for multitasking scenarios, allowing users to perceive the control action being performed even without looking at the screen.

[0108] In this way, by using physical devices to provide additional sensory feedback, users' perception of automated operations can be enhanced, further improving the efficiency and safety of human-machine collaboration.

[0109] In another embodiment, when the electronic device performs step S302 as described above, such as Figure 4 Alternatively, the following steps S403 can be performed:

[0110] Step S403: Output the sound information corresponding to the control operation.

[0111] In the embodiments of this application, the sound information corresponding to the control operation output by the electronic device refers to playing a short sound prompt through a speaker or headphones when each control action is performed, so as to inform the user of the type of control action currently being performed.

[0112] For example, a beep sound is played when the left mouse button is clicked; a swish sound is played when the automation operating system performs a drag-and-drop control action; and a confirmation sound is played when the automation operating system completes a task.

[0113] In the embodiments of this application, sound information can serve as a supplement to visual cues, helping users to perceive the system status even when they cannot focus on the screen.

[0114] In the embodiments of this application, sound information can be categorized according to the type of control action and set with different tones, rhythms, and durations to facilitate quick user identification. Furthermore, sound information can be customized according to user preferences to meet the needs of different usage scenarios.

[0115] In this way, by using sound feedback, the user's perception of the control process can be enhanced. This enhanced perception leads to a more natural human-computer interaction experience, which in turn improves overall user satisfaction.

[0116] In another embodiment, when the electronic device performs step S302 as described above, such as Figure 4 Alternatively, the following steps S404 can be performed:

[0117] Step S404: Output the text description information corresponding to the control operation.

[0118] In embodiments of this application, the electronic device can output text description information corresponding to control operations. This text description information refers to a piece of text displayed on the screen that describes the specific control action currently being performed. For example, it could indicate clicking the 'OK' button, typing 'Hello World', or sliding the progress bar to 50%. The text description information can be displayed in a fixed area of ​​the interface, such as the bottom status bar or sidebar, or it can be temporarily displayed via a pop-up window or floating box.

[0119] In embodiments of this application, text description information may also be displayed in the corresponding first or second area in response to control operations, in the form of a pop-up window or a floating box.

[0120] In the embodiments of this application, if the text description information is displayed in the second area, the display effect is different from that in the second area.

[0121] In the embodiments of this application, the content of the text description information should be accurate, concise, and easy to understand. The text description information can be combined with other prompting methods (such as animations corresponding to control actions, light prompts for control operations, and sound information) to form a multi-layered information feedback system. The information feedback system formed by the combined use of text description information and other prompting methods is particularly suitable for scenarios that require detailed recording of the control action process or for user review and reference.

[0122] In this way, the user can be directly informed of the current control action through text, thereby improving the transparency and controllability of the system and enhancing the user's trust and acceptance of the automated operating system.

[0123] In this embodiment, an animation corresponding to the control action is output on the current display interface. The physical devices involved in the control action within the control system use lights to indicate the control operation, output sound information associated with the control operation, and output text description information matching the control operation. The above output information can be arbitrarily combined to provide status feedback in various ways, thereby achieving multi-dimensional status prompts. This multi-dimensional status prompt method can significantly improve the user's perception of the automated operating system's operating status, further effectively reducing user errors and enhancing user trust in the automated system.

[0124] In some embodiments, when the electronic device performs the above step S401, such as Figure 5 As shown, the following steps S501 to S502 can also be performed:

[0125] Step S501: Determine the physical input device corresponding to the control action.

[0126] In embodiments of this application, the electronic device determines the physical input device corresponding to the control action. Exemplarily, the physical input device refers to the actual hardware device used to perform the control action, such as a physical mouse, keyboard, touchpad, etc.

[0127] In the embodiments of this application, the type of operation instruction generated by the LAM is identified to determine which physical input device should be used for the operation. For example, when the LAM generates a button click instruction, the electronic device can map the button click operation in the control action generated by the LAM to the left mouse button and identify the available mouse device in the current environment as the object to perform the operation.

[0128] In the embodiments of this application, by clearly defining the correspondence between control actions and physical input devices, it can be ensured that the operations generated by LAM can be correctly mapped and executed, providing an accurate data foundation for subsequent visual prompts.

[0129] Step S502: Display the device image information corresponding to the physical input device on the current display interface.

[0130] In the embodiments of this application, device image information refers to the graphical representation of the physical input device displayed on the current display interface. For example, when the LAM is performing mouse operations, the electronic device displays an icon representing a mouse on the screen (see...). Figure 6 When keyboard operations are performed, the electronic device can display a simplified image of the keyboard (see...). Figure 7 Device image information not only helps users intuitively understand the type of operation currently in progress, but also enhances the transparency of the interaction.

[0131] In the embodiments of this application, the position and style of the device image information can be adjusted according to the actual application scenario. For example, in desktop automation tasks, the mouse icon can hover near the interactive control or the operation position of the control command, while the keyboard icon can be displayed in the corner of the screen or the taskbar area, or in a second area.

[0132] In the embodiments of this application, there is a one-to-one correspondence between device image information and physical input devices. For example, when the LAM detects a left mouse click, the electronic device can synchronously display the corresponding state change of the mouse icon on the interface, such as the effect of being pressed, so that the display state of the presented device image information is consistent with the user's operation on the physical input device.

[0133] In the embodiments of this application, for different types of input devices, the electronic device can set the corresponding image to different colors or shapes, thereby helping the user to quickly distinguish and identify various types of input devices.

[0134] Step S503: Control the image information of the control device to present dynamic effects according to the control action.

[0135] In the embodiments of this application, dynamic effects refer to the real-time changes in device image information based on different control actions performed by the LAM. For example, when the LAM performs a left mouse click operation, the mouse icon may show a shadow effect indicating a pressed state (see...). Figure 6 (61) When LAM performs a keyboard key press, the corresponding key position may be highlighted and flashed (see 61); Figure 7 (71) The dynamic changes in the presentation not only reflect the current state of the operation, but also help users to more clearly perceive the execution process of LAM.

[0136] In the embodiments of this application, the design of dynamic effects needs to take into account the user's cognitive habits and visual experience, and avoid the dynamic effects being too complex or interfering with the content of the main interface. For example, for frequently occurring simple operations (such as multiple clicks), a shorter animation time can be used; while for critical operations (such as submitting a form), the animation time can be extended to emphasize the importance of the critical operation.

[0137] In the embodiments of this application, the presentation of dynamic effects needs to be strictly synchronized with the timing of the control action. For example, when LAM performs a keyboard input, the position of the corresponding key in the keyboard image should immediately be highlighted to ensure that the system design allows the user to perceive the operation in real time.

[0138] In this way, by controlling actions to dynamically represent the device's image information, the operating status of LAM becomes more intuitive and easier to understand, helping users to grasp the operation progress and results of LAM in a timely manner, thereby improving the overall user experience and operational efficiency of the system.

[0139] In this way, by identifying the physical input device corresponding to the control action and displaying the device image information corresponding to the physical input device in the current display interface, and by controlling the device image information corresponding to the physical input device to present dynamic effects according to the control action, the operation being performed by LAM can be clearly conveyed to the user. This can enhance the user's trust in the automated process, thereby effectively reducing user errors and improving the user-friendliness of human-computer interaction.

[0140] In some embodiments, when the electronic device performs step S403 as described above, such as Figure 8 As shown, the following steps S801 can also be performed:

[0141] Step S801: Determine analog sound information based on the physical input device corresponding to the control action. The analog sound information represents the sound generated when the physical input device is operated, and output the analog sound information.

[0142] In the embodiments of this application, simulated sound information refers to audio feedback generated based on the control action currently being performed by the user, corresponding to the control action, used to enhance the user's perception of the automated operation process. For example, when LAM simulates clicking the left mouse button, a click-like mouse click sound effect will be played; when LAM simulates pressing the Enter key on the keyboard, a keystroke sound effect will be played. The design of simulated sound information aims to allow users to confirm the operation being performed by LAM through hearing, thereby improving user trust and interactive experience.

[0143] In the embodiments of this application, simulated sound information can be implemented through a preset sound effects library or generated through real-time synthesis technology, depending on system performance and application requirements. For example, in resource-constrained embedded systems, pre-recorded sound effects in compressed formats can be used; on high-performance computing platforms, more complex audio rendering algorithms can be employed to obtain more realistic sound effects.

[0144] In other embodiments, when the electronic device performs step S403 as described above, such as Figure 8 As shown, the following steps S802 can also be performed:

[0145] Step S802: Obtain the voice broadcast information corresponding to the control action and output the voice broadcast information.

[0146] In the embodiments of this application, voice broadcast information refers to the process of converting the operation content into audible language through speech synthesis when the LAM performs a specific control action, and then playing that audible language aloud so that the user can understand the current control action through voice. For example, when the LAM clicks the search button, it may announce that the search button is being clicked, helping the user understand the current operation intent. This method of outputting voice broadcast information is particularly suitable for scenarios with limited vision or requiring multitasking, such as LAMs operating in voice assistant control or accessibility mode.

[0147] In the embodiments of this application, the generation of voice broadcast information typically relies on text-to-speech (TTS) conversion technology. The content of the voice broadcast information can be automatically generated based on the operation type or defined by the developer. Furthermore, the voice broadcast information can support multiple languages ​​and tone styles to suit the needs of different user groups. For example, in educational applications, the system can use a gentler, clearer tone; while in games or entertainment scenarios, a more expressive voice style can be used.

[0148] In this embodiment, by outputting simulated sound information and / or voice broadcast information, the user's perception of the LAM's operating status can be significantly improved. Outputting simulated sound information and / or voice broadcast information enhances the user's trust in automated operations, thereby reducing the risk of misoperation and ultimately improving the overall interactive experience and system usability.

[0149] In some embodiments, if user intervention occurs during the above automation process, the process will be restarted based on the situation after user intervention. For example, if a user discovers a discrepancy in the recipient's information during email sending, the task information of the target task can be readjusted, and then the target model can be input before continuing the operation.

[0150] In the embodiments of this application, if a user intervenes to perform a control operation, for example, the user operates the current interface through an input device such as a mouse or keyboard, a pop-up window may appear asking whether it is a user error. If so, it is ignored; if it is not a mistake, the user operation is accepted, and the automated operation can be exited or the user operation can continue to be output.

[0151] In one embodiment of this application, the interface display method is implemented as follows: An image of the current display interface of the electronic device and the target task being performed by the electronic device are acquired; the image and task information of the target task are input into a target model to obtain control instructions output by the target model. These control instructions are used to operate the current display interface to complete the target task; the window area corresponding to the interactive control indicated by the control instructions within the current display interface is determined as a first area; the first area is displayed with a first display effect, and a second area is displayed with a second display effect; the second area is any area within the current display interface other than the first area. After obtaining the control instructions output by the target model, the control action indicated by the control instructions is determined; the physical device corresponding to the control action is controlled to prompt the control operation with lights.

[0152] In an embodiment of this application, another implementation of the interface display method is as follows: The image of the current display interface of the electronic device and the target task being performed by the electronic device are acquired; the image and task information of the target task are input into a target model to obtain control instructions output by the target model. These control instructions are used to operate the current display interface to complete the target task; a region within a preset range centered on the operation position corresponding to the control instructions within the current display interface is defined as a first region; the first region is displayed with a first display effect, and a second region is displayed with a second display effect; the second region is any region within the current display interface other than the first region. After obtaining the control instructions output by the target model, the control action indicated by the control instructions is determined; the physical input device corresponding to the control action is determined; the device image information corresponding to the physical input device is displayed on the current display interface; and the device image information is controlled to present a dynamic effect according to the control action.

[0153] In an embodiment of this application, an exemplary interface display method may include: acquiring a screenshot of the current display interface of an electronic device and the email sending task being performed by the electronic device; inputting the screenshot and task information of the email sending task, such as recipient, sender, and email content, into a target model to obtain control instructions output by the target model; determining the area within the current display interface where the interactive control indicated by the control instructions is located as a first area, displaying the first area with a first transparency and displaying a second area with a second transparency; the first transparency is greater than the second transparency. Furthermore, the control action indicated by the control instructions will be determined. If the left mouse button is clicked, a sound prompt for clicking the left mouse button can be output, and a mouse icon can be displayed on the display interface, with the left mouse button highlighted.

[0154] This application provides an interface display method, which includes: acquiring an image of the current display interface of an electronic device and a target task being executed by the electronic device; inputting the image and task information of the target task into a target model to obtain control instructions output by the target model, the control instructions being used to operate the current display interface to complete the target task; determining a first area corresponding to the control instructions within the current display interface; and displaying the first area and a second area with different display effects; the second area being other areas within the current display interface besides the first area. The interface display method provided by this application can directly determine control instructions based on the target model using the image and task information of the current interface. This allows for control of the target program executing the target task based on the control instructions, eliminating the need for different processing for different applications, thus increasing applicability. Furthermore, displaying the controlled and uncontrolled areas with different display effects clearly indicates the ongoing operation, improving the readability of the current display interface.

[0155] This application provides an interface display device 9, such as... Figure 9As shown, it includes:

[0156] The acquisition module 91 is used to acquire the image of the current display interface of the electronic device and the target task being performed by the electronic device;

[0157] Decision module 92 is used to input the image and target task information into the target model, obtain the control instructions output by the target model, and use the control instructions to operate the current display interface to complete the target task;

[0158] Module 93 is used to determine the first area corresponding to the control command within the current display interface;

[0159] Display module 94 is used to display the first area and the second area with different display effects; the second area is the other areas in the current display interface besides the first area.

[0160] In one embodiment of this application, the determining module 93 is further configured to: determine the area where the interactive control controlled by the control instruction is located within the current display interface as the first area; determine the window area corresponding to the interactive control controlled by the control instruction within the current display interface as the first area; and determine the area within the current display interface within a preset range centered on the operation position corresponding to the control instruction as the first area.

[0161] In one embodiment of this application, the display module 94 is further configured to display a first area with a first display effect and a second area with a second display effect; wherein the transparency corresponding to the first display effect is greater than the transparency corresponding to the second display effect; or, the brightness corresponding to the first display effect is greater than the brightness corresponding to the second display effect.

[0162] In one embodiment of this application, the display module 94 is further configured to determine the control action indicated by the control command and output prompt information corresponding to the control action.

[0163] In one embodiment of this application, the display module 94 is further configured to output an animation corresponding to the control action within the current display interface; control the physical device corresponding to the control action to prompt the control operation in the form of light; output sound information corresponding to the control operation; and output text description information corresponding to the control operation.

[0164] In one embodiment of this application, the display module 94 is further configured to determine the physical input device corresponding to the control action; display the device image information corresponding to the physical input device on the current display interface; and control the device image information to present dynamic effects according to the control action.

[0165] In one embodiment of this application, the display module 94 is further configured to determine analog sound information based on the physical input device corresponding to the control action, wherein the analog sound information represents the sound generated by the physical input device when it is operated, and output the analog sound information; obtain the voice broadcast information corresponding to the control action, and output the voice broadcast information.

[0166] This application provides an electronic device 10, such as... Figure 10 As shown, the electronic device includes: a processor 1001, a memory 1002, and a communication bus 1003;

[0167] Communication bus 1003 is used to realize the communication connection between processor 1001 and memory 1002;

[0168] The processor 1001 is used to execute the computer program stored in the memory 1002 to implement the above-described interface display method.

[0169] This application provides an electronic device that acquires an image of the current display interface of the electronic device and a target task being executed by the electronic device; inputs the image and task information of the target task into a target model to obtain control instructions output by the target model. These control instructions are used to operate the current display interface to complete the target task; a first region corresponding to the control instructions within the current display interface is determined; and a first region and a second region are displayed with different display effects. The second region is any region within the current display interface other than the first region. The electronic device provided by this application can directly determine control instructions based on the target model using the image and task information of the current interface. This allows for control of the target program executing the target task based on the control instructions, eliminating the need for different processing for different applications, thus increasing applicability. Furthermore, displaying the controlled and uncontrolled regions with different display effects clearly indicates the ongoing operation, improving the readability of the current display interface.

[0170] This application provides a computer-readable storage medium storing one or more computer programs, which can be executed by one or more processors to implement the above-described interface display method. The computer-readable storage medium can be transient or non-transient.

[0171] This application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, it implements some or all of the steps in the watermark embedding method described above. This computer program product can be implemented specifically through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied as a computer storage medium; in another optional embodiment, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0172] In some embodiments, the storage medium may be a computer-readable storage medium, which may be volatile memory, such as random-access memory (RAM); or non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), solid-state drive (SSD), ferromagnetic random access memory (FRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic surface memory, optical disk, or compact disk-read-only memory (CD-ROM); or it may be various devices including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0173] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0174] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0175] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0176] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0177] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0178] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file containing other programs or data, for example, in one or more scripts within a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files storing one or more modules, subroutines, or code sections). As an example, executable instructions may be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0179] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

[0180] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0181] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components may be combined, or integrated into another system, or some features may be ignored or not performed.

[0182] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for displaying an interface, the method comprising: Acquire an image of the current display interface of the electronic device, as well as the target task being performed by the electronic device; The image and the task information of the target task are input into the target model to obtain the control instructions output by the target model. The control instructions are used to operate the current display interface to complete the target task. Determine the first region within the currently displayed interface corresponding to the control command; The first area and the second area are displayed with different display effects; the second area is the other area within the current display interface besides the first area.

2. The interface display method according to claim 1, wherein determining the first area corresponding to the control command within the current display interface includes at least one of the following: The area within the current display interface where the interactive control indicated by the control command is located is defined as the first area; The window area corresponding to the interactive control controlled by the control instruction within the current display interface is defined as the first area; The area within a preset range centered on the operation position corresponding to the control command within the current display interface is defined as the first area.

3. The interface display method according to claim 1, wherein displaying the first area and the second area with different display effects includes: The first area is displayed with a first display effect, and the second area is displayed with a second display effect; Wherein, the transparency corresponding to the first display effect is greater than the transparency corresponding to the second display effect; or, the brightness corresponding to the first display effect is greater than the brightness corresponding to the second display effect.

4. The interface display method according to claim 1, after obtaining the control commands output by the target model, the method further includes: Determine the control action indicated by the control command; Output prompt information corresponding to the control action.

5. The interface display method according to claim 4, wherein the output of prompt information corresponding to the control action includes at least one of the following: Output an animation corresponding to the control action within the current display interface; The physical device corresponding to the control action is controlled to indicate the control operation with lights. Output the sound information corresponding to the control operation; Output the text description information corresponding to the control operation.

6. The interface display method according to claim 5, wherein outputting the animation corresponding to the control action within the current display interface includes: Determine the physical input device corresponding to the control action; The device image information corresponding to the entity input device is displayed on the current display interface; The control action controls the device to display dynamic effects on the image information.

7. The interface display method according to claim 5, wherein outputting the sound information corresponding to the control operation includes at least one of the following: Based on the physical input device corresponding to the control action, the analog sound information is determined, the analog sound information represents the sound generated when the physical input device is operated, and the analog sound information is output. Obtain the voice broadcast information corresponding to the control action, and output the voice broadcast information.

8. An interface display device, comprising: The acquisition module is used to acquire the image of the current display interface of the electronic device, as well as the target task being performed by the electronic device; The decision module is used to input the image and the task information of the target task into the target model, and obtain the control instructions output by the target model. The control instructions are used to operate the current display interface to complete the target task. The determining module is used to determine the first area corresponding to the control command within the current display interface; The display module is used to display the first area and the second area with different display effects; the second area is the other area in the current display interface besides the first area.

9. An electronic device, comprising: Processor, memory, and communication bus; The communication bus is used to realize the communication connection between the processor and the memory; The processor is configured to execute the computer program stored in the memory to implement the interface display method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing one or more computer programs, which can be executed by one or more processors to implement the interface display method of any one of claims 1 to 7.

Images