Data determination method and device

Abstract

The invention provides a data determination method and device. The method comprises the steps of obtaining operation data and application program interface data corresponding to a target application program; determining a target operation task for the target application program based on the operation data and the application program interface data; the target operation task is an operation meeting a shortcut key generation condition; and determining shortcut key recommendation data corresponding to the target operation task so as to realize a function corresponding to the target operation task based on the shortcut key recommendation data.

Description

Technical Field

[0001] This application relates to data processing technology, and more particularly to a data determination method and apparatus. Background Technology

[0002] In modern office and software environments, keyboard shortcuts have become essential tools for improving work efficiency. By quickly executing commands using keyboard shortcuts, users can reduce their reliance on the mouse and increase task processing speed. However, as software functions become increasingly complex, the number of keyboard shortcuts and their usage scenarios have also increased significantly, posing a considerable challenge to users' memorization and learning. Summary of the Invention

[0003] This application provides a data determination method and apparatus that can reduce the burden on users to manually set shortcut keys, and dynamically optimize shortcut key configurations according to users' actual usage habits, making it more personalized and intelligent.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides a data determination method, the data determination method including: Obtain the operation data and application interface data corresponding to the target application; Based on the operation data and the application interface data, a target operation task is determined for the target application; the target operation task is an operation that satisfies the conditions for generating shortcut keys. Determine the shortcut key recommendation data corresponding to the target operation task, so as to implement the function corresponding to the target operation task based on the shortcut key recommendation data.

[0005] In some embodiments, determining the target operation task for the application based on the operation data and the application interface data includes: determining the association between the operation data and the application interface data; and determining the target operation task based on the association.

[0006] In some embodiments, the operation data includes multiple operation events, and the application interface data includes application interface elements; determining the association between the operation data and the application interface data includes: determining the timestamp corresponding to each operation event; sorting the operation events based on the timestamps to obtain a first sorting result; and establishing an association between the operation events and the application interface elements based on the first sorting result and the application interface elements corresponding to the occurrence of each operation event.

[0007] In some embodiments, the application interface elements include first application interface elements in functional areas and second application interface elements in non-functional areas; determining the target operation task based on the association includes: within a first preset time period, if the number of occurrences of an operation event corresponding to the same first application interface element meets a first threshold condition, determining the operation task corresponding to the operation event as the target operation task; within a second preset time period, if the number of occurrences of an operation event corresponding to a specific second application interface element meets a second threshold condition, determining the target first application interface element within the functional area of ​​the interface where the specific second application interface element is located; and determining the operation task of the operation event corresponding to the target first application interface element as the target operation task.

[0008] In some embodiments, determining the target operation task based on the association relationship includes: performing semantic understanding on the application interface elements to obtain semantic understanding results; and determining the target operation task based on the semantic understanding results and the association relationship.

[0009] In some embodiments, determining the target operation task based on the semantic understanding result and the association relationship includes: if the function implemented by a series of consecutive operation events is the same as the semantic understanding result corresponding to a third application interface element, determining the operation task corresponding to the series of consecutive operation events as the target operation task; wherein the third application interface element corresponding to the series of consecutive operation events is located on the interface of the target application; the third application interface element includes at least text, icons, or prompts on the application interface; or, determining the consecutive operation tasks corresponding to the series of application interface elements based on the semantic understanding result of the series of application interface elements; if all operation events corresponding to the series of consecutive application interface elements include other operation events besides implementing the consecutive operation task, determining the consecutive operation task as the target operation task; wherein the semantic understanding result includes at least the context association information between the series of consecutive application interface elements.

[0010] In some embodiments, the operation data includes multiple operation events, and the application interface data includes interface transition information of the application interface; determining the target operation task for the application based on the operation data and the application interface data includes: determining multiple stage tasks for the target application based on the multiple operation events and the interface transition information of the application interface; identifying candidate operation events between two adjacent stage tasks; if the candidate operation event is different from a preset cross-stage operation event, determining the task corresponding to the candidate operation event as the target operation task; wherein, the preset cross-stage operation event represents the operation event with the least time consumption when executing the two adjacent stage tasks.

[0011] In some embodiments, the operation data includes multiple operation events, and the application interface data includes application interface elements; determining the shortcut key recommendation data corresponding to the target operation task includes: creating descriptive information corresponding to the target operation task based on the operation events and application interface elements corresponding to the target operation task; inputting the descriptive information and the historical shortcut key information of the target application into the target model to obtain the shortcut key recommendation data; the descriptive information can guide the target model to perform reasoning, and the shortcut key recommendation data is different from the existing shortcut keys in the target application, other applications different from the target application, and the operating system.

[0012] In some embodiments, the shortcut key recommendation data includes shortcut key identification information; the data determination method further includes: after detecting entry into the target application interface, outputting the identification information to a preset area of ​​the target application interface; associating the identification information with the target application interface corresponding to the target operation task; and in response to obtaining a selection operation for the identification information, executing the function corresponding to the selection operation.

[0013] This application provides a data determination device, comprising: an acquisition module for acquiring operation data and application interface data corresponding to a target application; a first determination module for determining a target operation task for the target application based on the operation data and the application interface data; wherein the target operation task is an operation that satisfies the conditions for generating shortcut keys; and a second determination module for determining recommended shortcut keys corresponding to the target operation task, so as to realize the function corresponding to the target operation task based on the recommended shortcut keys.

[0014] This application provides an electronic device, the electronic device comprising: Memory is used to store executable instructions or computer programs. The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the data determination method provided in the embodiments of this application.

[0015] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions, which, when executed by a processor, implements the data determination method provided in this application.

[0016] This application provides a computer program product, including a computer program or computer executable instructions, which, when executed by a processor, implements the data determination method provided in this application. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application; Figure 2 This is an optional flowchart illustrating the data determination method provided in the embodiments of this application. Figure 1 ; Figure 3 This is a flowchart illustrating the AI ​​Agent intelligent shortcut key recommendation method based on a multimodal large language model provided in this application embodiment; Figure 4 This is a schematic diagram of the shortcut key prompt module provided in an embodiment of this application; Figure 5 This is a schematic diagram of the architecture of the intelligent shortcut key recommendation method provided in the embodiments of this application.

[0018] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] It should be understood that the following description of the embodiments is intended to explain and illustrate the overall concept of the embodiments of this application, and should not be construed as limiting the embodiments of this application. In the specification and drawings, the same or similar reference numerals refer to the same or similar parts or components. For clarity, the drawings are not necessarily drawn to scale, and some well-known parts and structures may be omitted in the drawings.

[0021] In some embodiments, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the meaning understood by a person skilled in the art to which the embodiments of this application pertain. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. The word "a" or "an" does not exclude multiple components. The terms "comprising" or similar terms mean that the element or object preceding the word covers the elements or objects listed after the word and their equivalents, without excluding other elements or objects. The terms "connected" or similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Above," "below," "left," "right," "top," or "bottom," etc., are used only to indicate relative positional relationships, and these relative positional relationships may change accordingly when the absolute position of the described object changes. When an element such as a layer, film, region, or substrate is referred to as being "above" or "below" another element, the element may be "directly" located "above" or "below" the other element, or there may be intermediate elements present.

[0022] The following problems exist with keyboard shortcuts in related technologies: 1) Heavy memory burden: Modern software has rich functions and a large number of keyboard shortcuts, making it difficult for users to remember them all, especially for infrequently used functions. 2) High learning cost: New users or users of new software need to spend a lot of time learning and adapting to the keyboard shortcut settings, affecting user experience and efficiency. 3) Lack of personalization: Existing keyboard shortcut settings are usually predefined and cannot be dynamically adjusted and optimized according to users' personal operating habits. 4) Lack of intelligent prompts: Software usually only displays keyboard shortcut prompts in menus or toolbars, failing to provide timely and intuitive prompts based on user operating scenarios (such as the current interface or task flow).

[0023] To address these issues, software developers often pre-set a set of keyboard shortcuts for frequently used functions, which users can view in help documentation or settings menus. However, the drawback is that these pre-set shortcuts may not suit all users' habits, and their sheer number makes them difficult for users to memorize. Alternatively, software may allow users to customize or modify shortcuts to suit their personal preferences. The downside is that the customization process is cumbersome, requiring users to spend time configuring and possess sufficient technical knowledge. Some software displays shortcut hints in menus or toolbars to help users remember them. The drawback is that these hints only provide information for currently used functions and do not proactively assist users in learning and memorizing. Third-party software can manage shortcuts for the system or other software, providing features such as shortcut conflict detection. The drawback is that this requires additional software installation and still necessitates manual configuration and management by the user. Finally, some software allows users to record macros or use automated scripts to perform a series of operations. The drawback is that these settings are complex, unfriendly to ordinary users, and lack intelligent recommendations.

[0024] To address the issues of heavy memory burden, high learning cost, lack of personalization, and real-time prompts in shortcut key settings in related technologies, this application provides a data determination method. This method involves acquiring operation data and application interface data corresponding to a target application; determining a target operation task for the target application based on the operation data and application interface data; identifying the target operation task as an operation that meets the conditions for generating shortcut keys; and determining recommended shortcut key data corresponding to the target operation task, thereby implementing the function corresponding to the target operation task based on the recommended shortcut key data.

[0025] By collecting user operation data and application interface data, and combining this data with operational behavior and interface information, the system identifies target operation tasks that meet the conditions for generating shortcut keys, and then automatically recommends shortcut keys. This reduces the burden on users to manually set shortcut keys, and dynamically optimizes shortcut key configurations based on actual user habits, making it more personalized and intelligent compared to preset or custom methods.

[0026] In some embodiments, the data determination method provided in this application can be executed by an electronic device, which can be a smart terminal, a server, or an edge computing device, or an AI agent (such as a smart assistant program) running on the terminal. That is, the data determination method in each embodiment of this application can be executed by a smart terminal (such as a mobile phone or tablet), by a server, or by a smart terminal and a server working together. The server can be a physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0027] In some embodiments, an AI agent is an application of artificial intelligence technology, which can be implemented based on a target model (such as a large language model) for reasoning. The agent's behavior is determined by the target model within the agent based on the current state and external input, and can be used to specifically solve a certain type of problem. The target model provides the decision-making basis through learning and training on large amounts of data; the agent can also call upon tools, plugins, and knowledge bases to provide reasoning, decision-making, and execution capabilities. In other words, the target model provides decision support, and the agent continuously optimizes the performance of its internal target model through feedback data from practical applications (such as prompts).

[0028] Figure 1 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Figure 1 The illustrated electronic device 10 includes at least one processor 110, a memory 150, at least one network interface 120, and a user interface 130. Various components within the electronic device are coupled together via a bus system 140. It is understood that the bus system 140 is used to implement communication between these components. In addition to a data bus, the bus system 140 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 1 The general labeled all buses as Bus System 140.

[0029] The processor 110 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0030] User interface 130 includes one or more output devices 131 that enable the presentation of media content, and one or more input devices 132.

[0031] Memory 150 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. Memory 150 may optionally include one or more storage devices physically located away from processor 110. Memory 150 may include volatile memory or non-volatile memory, or both. Non-volatile memory may be read-only memory (ROM), and volatile memory may be random access memory (RAM). The memory 150 described in this application embodiment is intended to include any suitable type of memory. In some embodiments, memory 150 is capable of storing data to support various operations, examples of which include AI agents, programs, modules, and data structures, or subsets or supersets thereof, as illustrated below.

[0032] Operating system 151 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks; The network communication module 152 is used to reach other computing devices via one or more (wired or wireless) network interfaces 120, exemplary network interfaces 120 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc. The input processing module 153 is used to detect one or more inputs or interactions from one or more input devices 132.

[0033] In some embodiments, the apparatus provided in this application may be implemented in software. Figure 1 A data determination device 154 stored in memory 150 is shown. This data determination device 154 can be a data determination device in a data determination method, and can be software in the form of programs and plug-ins, including the following software modules: an acquisition module 1541, a first determination module 1542, and a second determination module 1543. These modules can be logically linked and therefore can be arbitrarily combined or further divided according to their implemented functions. The functions of each module will be described below.

[0034] In other embodiments, the apparatus provided in this application can also be implemented in hardware. As an example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the data determination method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0035] The technical solution of this application will now be described in detail with reference to the accompanying drawings.

[0036] Figure 2 This is an optional flowchart illustrating the data determination method provided in the embodiments of this application. Figure 1 ,like Figure 2 As shown, the data determination method provided in this application embodiment can be implemented through steps S201 to S203: Step S201: Obtain the operation data and application interface data corresponding to the target application.

[0037] Here, the target application can refer to an application that requires shortcut key settings, and the operation data can refer to various operation records generated by the user during the use of the target application, such as clicks, drags, and keyboard input events. Each operation event can include information such as a timestamp, operation type, and corresponding operation object in the operation data to reflect the user's interaction behavior pattern in a specific interface. For example, in Photoshop, the user's action of clicking the layer merge button three times consecutively through the menu will be recorded as operation data, accompanied by a timestamp, for subsequent analysis of the user's operating habits and to analyze whether there are inefficient behaviors.

[0038] Application interface data refers to the graphical interface information of an application currently displayed on a user's device, acquired through image capture or video streaming. It can include at least the visual elements and structural information of the application interface, such as the position, content, and state of interface elements like buttons, menu items, text boxes, and icons. Application interface data can be used to identify buttons or text on the interface through Optical Character Recognition (OCR), combined with Computer Vision (CV) models to detect the types of controls within the application interface data, thereby understanding the current operating environment. For example, the content of the current Photoshop window can be captured in real-time, and OCR and CV technologies can be used to identify layer names, tool options, menu items, and other information in the application interface data to support subsequent target operation task recognition and shortcut key recommendations.

[0039] In this embodiment, the AI ​​Agent can record the user's operations on the target application. For example, user operation events can be captured through underlying application interface hooks (API hooks, such as Windows UI Automation), and application interface data can be obtained through screenshots or real-time video streams. To ensure data security and privacy protection, this embodiment only collects data related to user operations and does not involve the storage or uploading of sensitive information in the data.

[0040] Here, the AI ​​Agent can run in the background and continuously collect user operation data and application interface data without interfering with the user's normal use.

[0041] Step S202: Based on the operation data and application interface data, determine the target operation task for the target application; the target operation task is the operation that meets the conditions for generating shortcut keys.

[0042] In this embodiment, meeting the conditions for generating shortcut keys can refer to certain operations having characteristics such as high frequency, low efficiency, and complex paths. These operations can become suitable targets for shortcut key optimization. That is, the target operation task refers to repetitive but inefficient operation behaviors. For example, if a user frequently clicks the layer merge button in Photoshop through multi-level menus and has never used a shortcut key, then manually expanding multiple menus to find the layer merge function can be considered a target operation task.

[0043] Here, the target operation task can refer to an operation with high repetitiveness or inefficiency, identified by analyzing operation data and application interface data. In determining the target operation task, multiple factors can be considered. For example, sorting by timestamps can identify the same operations repeatedly performed by the user within a short period; classifying application interface data can determine whether the user is using a complex menu path to complete a function.

[0044] Step S203: Determine the recommended shortcut keys for the target operation task, and implement the function corresponding to the target operation task based on the recommended shortcut keys.

[0045] In some embodiments, recommended keyboard shortcuts may refer to shortcut combinations automatically generated and suggested to the user based on their historical operating habits and the current context. Recommended shortcuts not only need to consider user preferences but also need to avoid conflicts with existing shortcuts on the device and adapt to different application scenarios. For example, in Photoshop, if a user frequently clicks the layer merge function through the menu, Ctrl+E can be recommended as the shortcut for merging layers.

[0046] In some embodiments, a multimodal large language model can refer to an artificial intelligence model capable of processing multiple types of data (such as text, images, audio, etc.). This model enables information fusion and semantic understanding across different modalities. When generating shortcut key recommendation data, a multimodal large language model can be used to combine operational data and application interface data to analyze user operating habits and intentions. The model can predict potential shortcut key functions required for the current task based on factors such as operation sequences, frequency, paths, and times in the operational data, and generate corresponding shortcut key recommendations. For example, if a user has clicked the layer merge button multiple times in the past 5 minutes, and there are multiple unnamed layers in the current application interface data, Ctrl+E can be recommended as a shortcut key.

[0047] This application embodiment can generate shortcut key recommendation data based on the model's output, and ensure that the recommended shortcut keys do not conflict with existing shortcut keys or global shortcut keys. It can also prompt the user to confirm whether to accept the recommended shortcut key settings, and allow the user to modify or reject them, thereby improving user efficiency and user experience.

[0048] This application embodiment constructs user operation behavior in a specific application by collecting operation data and application interface data, identifies target operation tasks that meet the conditions for generating shortcut keys based on the operation data and application interface data, and finally generates shortcut key recommendation data. This not only reduces the burden of users manually setting shortcut keys, but also dynamically optimizes shortcut key configuration according to the user's actual usage habits, enabling users to complete target operation tasks more efficiently.

[0049] In some embodiments, step S202 can be implemented by steps S2021 and S2022: Step S2021: Determine the relationship between operation data and application interface data.

[0050] Here, the association between user action data and application interface data refers to establishing a mapping relationship between user actions (such as clicking, dragging, keyboard input, etc.) and elements in the target application interface (such as buttons, menu items, text boxes, etc.). This association can be achieved through methods such as timestamp alignment, spatial location matching, and contextual semantic analysis. For example, in Photoshop, if a user repeatedly clicks the "Merge Layers" option under the Layers menu, it can be identified that this series of actions has a strong correlation with the Layers menu and its sub-items on the interface.

[0051] By establishing a correlation between operational data and target application interface data, we can more accurately understand user behavior, improve the accuracy and relevance of shortcut recommendations, avoid incorrect or irrelevant shortcut recommendations, and thus enhance user experience and work efficiency.

[0052] Here, a multimodal large language model can be used to understand which button the user clicked on the target application interface, and combined with the current interface content and historical operation records, the user's true intention can be determined.

[0053] Step S2022: Determine the target operation task based on the association relationship.

[0054] This application's embodiments can, based on established relationships and combined with a multimodal large language model's understanding of user operation scenarios, determine the type of the currently ongoing operation task and infer the target operation task the user may want to perform. For example, when it is detected that the user accesses the save function multiple times through the menu path, it can be determined that the target operation task is to save a digital document, and Ctrl+S can be recommended as a personalized shortcut key.

[0055] In some embodiments, factors such as user operation frequency and the coherence of the target operation task flow may also be considered to optimize the accuracy of target operation task identification.

[0056] This application embodiment applies the correlation between operation data and target application interface data to the identification of target operation tasks, which can accurately determine the user's current intent and generate personalized shortcut key recommendations based on the determination result, thereby improving the user's operation efficiency and user experience.

[0057] In some embodiments, the operation data includes multiple operation events, and the application interface data includes application interface elements.

[0058] An operation event refers to a series of specific actions performed by a user while using software, such as clicking a button, dragging a control, or entering text. Each operation event has a corresponding time point. To accurately describe the chronological order of different operation events, a unique timestamp can be assigned to each operation event. Timestamps are recorded with millisecond precision, ensuring that the time intervals between different operation events can be accurately captured.

[0059] Application interface elements refer to components that users interact with on the software interface, such as buttons, menu items, input boxes, and toolbar icons. Each action event occurs on a specific interface element, such as clicking the save button, dragging a scroll bar, or selecting an option from a drop-down menu. By establishing relationships between action events and their corresponding interface elements, the user's intentions within the interface can be understood more accurately.

[0060] Correspondingly, step S2021 can be achieved through steps S1 to S3: Step S1: Determine the timestamp corresponding to each operation event.

[0061] Timestamps can be used to determine which operations occurred first and which occurred later, thus constructing a complete sequence of operations. For example, in Photoshop, when a user clicks the layer merge button multiple times in succession, timestamps can be used to analyze the events generated by these clicks to determine if they occurred in quick succession and whether there is a possibility of duplicate operations.

[0062] Step S2: Sort each operation event based on the timestamp to obtain the first sorting result.

[0063] Based on the acquired timestamp information, all operation events can be sorted chronologically to generate a first sorting result. This first sorting result ensures the consistency of the operation events' temporal sequence, facilitating further analysis of user behavior patterns. For example, by observing the chronological arrangement of operation events, it can be determined whether a user used a non-optimal path to complete a task.

[0064] In some embodiments, the first sorting result can serve as the basis for judging inefficient operations. For example, if an operation event appears multiple times in the sorting with short intervals, it may indicate that the user is unfamiliar with the function corresponding to that operation event or has made a mistake. By combining interface semantic analysis, it can be confirmed whether an operation event is an inefficient operation, and if it is confirmed to be an inefficient operation, a corresponding shortcut key prompt can be generated.

[0065] Step S3: Based on the first sorting result and the application interface elements corresponding to each operation event, establish the association between the operation events and the application interface elements.

[0066] By combining the initial ranking results with the corresponding application interface elements at the time of each operation event, a mapping table can be constructed between operation events and application interface elements, i.e., the association between operation events and application interface elements, for subsequent shortcut key recommendations and behavior analysis. For example, when a user clicks the export button multiple times, the interface element corresponding to that operation event can be identified, and combined with the user's historical operation path, it can be determined whether a faster method (such as setting a shortcut key Ctrl+S) should be recommended to replace manually clicking the export button.

[0067] This application utilizes timestamps to sort operation events, which helps to construct a clear operation sequence and further establish a precise mapping relationship with interface elements. This allows for a deeper understanding of the user's actual operation scenarios, providing accurate data support for personalized shortcut recommendations, thereby significantly improving the user's operation efficiency and user experience.

[0068] In some embodiments, application interface elements include a first application interface element with functional areas and a second application interface element with non-functional areas. Here, functional areas may refer to areas that carry business functions, such as menu items, buttons, and toolbars; non-functional areas may refer to areas that do not participate in business function logic, such as margins at the edges of the interface, blank areas, or the interface background.

[0069] Correspondingly, step S2022 can be achieved through steps S11 and S12: Step S11: If the number of occurrences of an operation event corresponding to the same first application interface element within a first preset time period meets the first threshold condition, the operation task corresponding to the operation event is determined as the target operation task.

[0070] The first preset duration can refer to the time window range within which a user repeatedly performs operations on the same first application interface element, such as 5 minutes. The specific time can be adjusted according to the software type and user behavior habits.

[0071] The first application interface element can refer to interactive elements in the functional area, such as menu items, buttons, and toolbar icons. An action event can refer to an action performed by the user on the first application interface element, such as clicking, dragging, or typing.

[0072] The first threshold condition is the setting of the frequency of operation events. For example, if the number of times a certain operation event occurs exceeds the set threshold (such as 3 times) within the first preset time period, that is, the operation event is clicked repeatedly at a high frequency, then the operation event is considered an inefficient event and a shortcut key can be recommended.

[0073] This method of judgment can identify frequently used functions and generate shortcut recommendations accordingly, thereby improving user efficiency.

[0074] Step S12: Within the second preset time period, if the number of occurrences of the operation event corresponding to the specific second application interface element meets the second threshold condition, determine the target first application interface element in the functional area of ​​the interface where the specific second application interface element is located; determine the operation task of the operation event corresponding to the target first application interface element as the target operation task.

[0075] The second preset duration can refer to the time window range used to detect inefficient operations. The duration can be the same as or different from the first preset duration, for example, 8 minutes.

[0076] The second application interface element is not a functional control, but it may be accidentally touched or used as part of the navigation path in actual use. The second threshold condition is a criterion for judging the frequency of operation events on non-functional area elements. For example, if a certain non-functional area is clicked more than a certain threshold (e.g., 2 times) within a second preset time period, it indicates that the user may have an unclear operation path, or the path to perform the operation in the non-functional area is too long (e.g., the user needs to expand / click 3 or more layers of menus to reach the target function).

[0077] At this point, the first application interface element corresponding to the operation can be identified (for example, the user needs to click three times in a non-functional area to paste a formula, while the functional area has a corresponding formula pasting element), and the operation task of this first application interface element can be marked as the target operation task. By identifying and marking the target operation task, redundant behaviors in the user's operation path can be discovered, thereby improving overall operation efficiency.

[0078] By combining the second preset duration and the second threshold condition, abnormal user behavior in non-functional areas can be identified, the user's true intended target operation task can be located, and shortcut key optimization recommendations can be made for the target operation task, thereby improving user experience and reducing learning costs.

[0079] This application embodiment can identify inefficient user operations by setting operation frequency thresholds for different areas, such as functional areas and non-functional areas, to discover tasks that truly require shortcut key support, reduce unnecessary mouse clicks, improve user work efficiency, and make the system more flexible and adaptable.

[0080] In some embodiments, step S2022 can also be implemented through steps S21 and S22: Step S21: Perform semantic understanding on the application interface elements to obtain semantic understanding results.

[0081] Here, semantic understanding refers to the process of content recognition and semantic parsing of elements in an application interface using a multimodal large language model. The results of semantic understanding can include the functional description of buttons, the operational intent of menu items, and the content type of input boxes. For example, in the Photoshop interface, the layer merge button is not only recognized as a graphical control but also parsed as having the semantic function of merging multiple layers into one.

[0082] There is a correlation between the semantic understanding process and the user's historical operation data. For example, when a user clicks the export as PDF option multiple times, the semantic understanding process knows that the operation is to save the current image as a PDF file, rather than to modify the image content, thus avoiding incorrect recommendations of editing-related shortcuts.

[0083] Step S22: Based on the semantic understanding results and the relationships, determine the target operation task.

[0084] The target operation task can refer to the specific functional requirement of the software operation that the user is currently performing. Determining this target operation task can be jointly determined by semantic understanding results and the user's historical operation behavior data. For example, the core operation intent of the current application interface can be extracted based on semantic understanding results, and then combined with the user's historical operation behavior data (such as frequently used functions, repetitive operation paths, etc.) to comprehensively determine the task that the user is most likely to perform. For example, if a user clicks the layer merge button multiple times in Photoshop, and repeatedly clicks the layer merge button after editing, then it can be inferred that the user's target operation task is to quickly complete the layer organization.

[0085] This application embodiment determines the target operation task by performing semantic understanding on application interface elements and determining the logical relationship between the semantic understanding results and the user's historical operation behavior data and application interface elements. It can extract meaning information from the text, icons, etc. of interface elements, assisting in determining whether the operation task can generate a shortcut key, enhancing the ability to understand complex interfaces, and improving the quality of shortcut key recommendations.

[0086] In some embodiments, step S22 can be implemented by steps S221 and S222, wherein steps S221 and S222 are parallel solutions: Step S221: If the function implemented by multiple consecutive operation events is the same as the semantic understanding result corresponding to the third application interface element, the operation task corresponding to the multiple consecutive operation events is determined as the target operation task; wherein, the third application interface element corresponding to the multiple consecutive operation events is located on the interface of the target application; the third application interface element includes at least text, icons or prompts on the application interface.

[0087] Here, multiple consecutive operation events can refer to a series of related operations performed by a user within a certain time window, exhibiting continuity in time and similarity in function. Examples include repeatedly clicking the same function button or entering the same menu path multiple times.

[0088] Third-party application interface elements refer to visual elements on the target application interface that can express the user's intent, such as text, icons, and prompts. These elements can be identified using OCR and computer vision technologies, and semantically understood in conjunction with the context, thereby determining their meaning and function.

[0089] When a user repeatedly performs a series of operations on a specific interface, and the functions corresponding to these operations are consistent with the semantic understanding result of a third application interface element (such as a save button) on that specific interface, it can be inferred that the user's intention is to save the file. The series of operations performed by the user can be defined as the target operation task corresponding to that third application interface element (such as a save button).

[0090] Here, semantic parsing is performed on the third application interface elements, and combined with user interaction actions, it is used to detect whether multiple consecutive operations are consistent with the interface design intent. Figure 1 If they match, then the user's operation is determined to be inefficient.

[0091] Step S222: Based on the semantic understanding results of multiple consecutive application interface elements, determine the consecutive operation tasks corresponding to the multiple application interface elements; if all operation events corresponding to the multiple consecutive application interface elements include other operation events besides implementing the consecutive operation tasks, determine the consecutive operation tasks as the target operation tasks; wherein, the semantic understanding results include at least the contextual association information between the multiple consecutive application interface elements.

[0092] In some embodiments, contextual information can refer to the logical relationships extracted from third-party application interface elements and combinations thereof in a multimodal large language model through semantic understanding. For example, the logical relationship of input content -> confirmation -> export can identify the task flow that the user is currently executing.

[0093] A series of consecutive application interface elements can refer to semantically related interface components that appear sequentially in the same interface or across interfaces. For example, an edit box, a confirmation button, and an export option together constitute a complete task flow.

[0094] If a user triggers multiple third-party application interface elements in a certain order on the interface, and the semantic understanding results of the third-party application interface elements indicate that the third-party application interface elements together constitute a coherent operation task, wherein the semantic understanding results of the third-party application interface elements also include some unrelated operation events, this coherent operation task can be taken as the target operation task.

[0095] This application's embodiments combine the context information of the current interface to determine whether the user is executing a continuous task flow. If there is a break in the operation (e.g., an operation unrelated to the current context suddenly appears in the task flow), it may indicate that the user has a misunderstanding of the current interface or tool, resulting in inefficient continuous tasks.

[0096] This application embodiment identifies highly repetitive task flows by analyzing the consistency between continuous operation events and the semantics of interface elements. This serves as the basis for recommending shortcut keys and can effectively capture complex operation tasks that users frequently use but are not covered by shortcut keys.

[0097] In some embodiments, the operation data includes multiple operation events, and the application interface data includes interface navigation information of the application interface.

[0098] Operational data can refer to all interactive behavior records generated by the user during the use of the target application, including but not limited to specific operation events such as clicking, dragging, keyboard input, and menu selection.

[0099] Interface transition information refers to the process information a user uses to switch between different interfaces of a target application, including page paths, changes in interface elements, and navigation levels. Interface transition information reflects the specific steps a user takes when performing a function. For example, in Word, when a user completes the export operation via File -> Save As -> PDF, the recorded interface transition information can include the complete interface switching path from the initial interface to the final export options interface.

[0100] Correspondingly, step S2022 can also be achieved through steps S31 to S33: Step S31: Based on multiple operation events and interface jump information of the application interface, determine multiple stage tasks for the target application.

[0101] Multiple-stage tasks can refer to a series of logically related subtasks divided according to user actions, with each subtask corresponding to a specific functional module or operation process. For example, in image processing software, a user may sequentially complete four stages: opening an image, adjusting brightness, adding filters, and saving the image.

[0102] By identifying multiple phases of tasks, a model can be used to perform cluster analysis on operation events and interface transition information of the target application interface to find continuous behavioral sequences with similar contexts and operational purposes.

[0103] This application embodiment, by breaking down user behavior into stages, can better determine the user's operational needs at different stages of the task.

[0104] Step S32: Identify candidate operation events between two adjacent stage tasks.

[0105] Candidate operation events can refer to operation events that occur between two adjacent task phases. For example, between the tasks of editing and saving a document, if the user frequently switches to other windows and then returns, such window switching operations are considered candidate operation events.

[0106] Step S33: If the candidate operation event and the preset cross-stage operation event are different, the task corresponding to the candidate operation event is determined as the target operation task; wherein, the preset cross-stage operation event represents the operation event with the least time consumption when executing two adjacent stage tasks.

[0107] Preset cross-stage operation events refer to efficient operation events pre-defined in the knowledge base based on historical data and best practices. Preset cross-stage operation events represent the most reasonable and fastest transition between two adjacent task stages. For example, for the task stages of editing and saving a document, the preset cross-stage operation event might be to directly press Ctrl+S to save, rather than clicking through menus step by step.

[0108] In some embodiments, the knowledge base includes pre-determined best practices for performing cross-stage tasks in an application, which may be an empirical database containing a large number of historical user operation paths for efficiently completing cross-stage tasks.

[0109] A target operation task refers to an operation task that differs from the preset cross-stage operation event between two adjacent stages, indicating potential for optimization. A target operation task signifies that the current operation method is inefficient or redundant, thus requiring identification and improvement measures. For example, if a user makes unnecessary interface switches between the editing and saving stages, shortcut key prompts can be provided.

[0110] In this embodiment, by comparing the differences between candidate operation events and preset cross-stage operation events, the non-optimal operation method currently used by the user can be identified, optimization suggestions can be made in a timely manner, the user's operation time can be reduced, and the smoothness and efficiency of the entire workflow can be improved.

[0111] In some embodiments, the operation data includes multiple operation events, and the application interface data includes application interface elements; correspondingly, step S203 can be implemented through steps S2031 and S2032: Step S2031: Based on the operation events and application interface elements corresponding to the target operation task, create description information corresponding to the target operation task.

[0112] Description information refers to structured data that combines operation events and application interface elements to characterize user operation patterns and intentions. Description information can include multi-dimensional features such as prompts, operation paths, operation frequency, and interface states, serving as the basis for subsequent shortcut recommendation generation. For example, description information might state: "The user clicked the 'PNG format' option under the 'Export' menu multiple times in the past 5 minutes."

[0113] In some embodiments, prompts can be generated by a lightweight model, different from the target model, based on operation events and application interface elements corresponding to the target operation task. For example, a prompt might be generated when a user repeatedly right-clicks a layer in Photoshop, with the intent to copy, group, or merge. The target model then generates shortcut key recommendations for copying, grouping, or merging based on the prompts.

[0114] This approach of combining operation events with application interface elements to generate descriptive information can more comprehensively capture the user's actual operation scenarios and needs.

[0115] Step S2032: Input the description information and the historical shortcut key information of the target application into the target model to obtain shortcut key recommendation data; the description information can guide the target model to perform reasoning, and the shortcut key recommendation data is different from the existing shortcut keys in the target application, other applications different from the target application, and the operating system.

[0116] Here, the target model can refer to a deep learning model built on a multimodal large language model (such as Gemini 2.0), capable of processing and understanding various data types such as text, images, and operation logs. Through pre-training, the target model can identify user operation patterns and predict user needs based on descriptive information, and generate personalized shortcut key recommendation data accordingly. The target model has high generalization ability and adaptability, and can dynamically adjust recommendation strategies for different users and application scenarios.

[0117] Keyboard shortcut history information refers to a record of shortcuts previously set or used by the target application. This information can originate from user-defined settings, software default configurations, or a global shortcut database. Analyzing keyboard shortcut history information helps avoid repeatedly recommending existing shortcuts and ensures that newly recommended shortcuts align with the user's previous usage habits.

[0118] Shortcut key recommendation data refers to a set of candidate shortcut keys generated by an artificial intelligence model based on descriptive information and historical shortcut key information. Shortcut key recommendation data can include not only specific key combinations (such as Ctrl+E), but also additional information such as recommendation reasons and confidence scores.

[0119] Here, the shortcut key recommendation data undergoes conflict detection and compatibility analysis to ensure that the recommended shortcut key data does not conflict with the existing shortcut keys in the device and is suitable for the current operating scenario.

[0120] In some embodiments, by inputting descriptive information and historical shortcut key information into an artificial intelligence model, the artificial intelligence model can comprehensively consider the user's operating habits, current interface state, and historical behavior to generate optimal shortcut key suggestions. This enables intelligent and personalized shortcut key recommendation data, avoiding shortcut key conflicts and improving the user's learning efficiency and acceptance.

[0121] This application's embodiments utilize an artificial intelligence model to generate shortcut key recommendations, taking into account not only the description information of the current operation task but also historical shortcut key data, ensuring that the recommended shortcut keys are both reasonable and conflict-free.

[0122] In some embodiments, the shortcut key recommendation data includes shortcut key identification information. Identification information can refer to unique identifiers used to represent the shortcut key function or operation, such as combinations like Ctrl+S -> Save, Ctrl+C -> Copy, etc.

[0123] The data determination method provided in this application embodiment may further include steps S41 and S42: Step S41: After detecting entry into the target application interface, output the identification information to the preset area of ​​the target application interface; associate the identification information with the target application interface corresponding to the target operation task.

[0124] In this embodiment, the preset area can be the lower right corner of the screen or other display positions that do not interfere with the main operation flow. When the user switches to the target application interface, the state of the application interface can be automatically detected, and identification information related to the application interface can be displayed in the preset area. This ensures that the user can obtain the required identification information at any time during operation, thereby reducing reliance on menu navigation and avoiding the need for the user to actively search for a shortcut list, thus improving the user's operating efficiency.

[0125] Here, association can be achieved through a multimodal large language model. Association refers to binding identification information to the specific operation tasks supported by the target application interface, ensuring that when a user performs a certain type of operation task, only identification information related to that type of operation task is displayed. For example, in Photoshop's image editing interface, if the user is recognizing that they are performing an image editing operation, only identification information related to the current image editing operation, such as Ctrl+E -> Merge Layers, Ctrl+T -> Free Transform, will be displayed; when the user enters the export stage, the operation command Ctrl+Shift+E -> Export Layer may be prompted.

[0126] Step S42: In response to obtaining the selection operation for the identification information, execute the function corresponding to the selection operation.

[0127] Upon detecting entry into the target application interface, the identification information is output to a preset area to ensure that the user can obtain the required identification information in a timely manner during actual operation.

[0128] In response to a selection action on the identifier (such as clicking the identifier or typing the identifier), the function corresponding to that selection action is executed. This can reduce repetitive clicks and inefficient operations by users, thereby significantly improving user efficiency and user experience.

[0129] This application's embodiments can display shortcut key prompts to users, helping them quickly learn and apply new shortcut keys. Simultaneously, users can directly trigger target functions by selecting an operation, further improving operational efficiency.

[0130] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0131] To address the problems existing in related technologies, this application proposes an AI Agent-based method that utilizes a multimodal large language model to monitor user behavior when using software. After a period of learning and analysis, it can automatically recommend suitable shortcut key settings (using physical keyboard keys) and provide shortcut key function prompts in the lower right corner of the screen. Key features include: 1) Application of a multimodal large language model: The multimodal large language model can process and understand various user operation modes, including text, images, mouse operations, and keyboard input. 2) User behavior monitoring and learning: The AI ​​Agent monitors user operation behavior and collects operation data without interfering with normal user operation. The AI ​​Agent analyzes user operating habits and frequently used functions through the multimodal large language model. 3) Automated shortcut key recommendation: Based on the analysis results, the most suitable shortcut key settings are automatically recommended to the user. The generated shortcut key settings comprehensively consider user operating habits and proactively identify and avoid potential shortcut key conflicts. 4) Shortcut key function prompts: Real-time shortcut key prompts are provided in the lower right corner of the screen to help users quickly understand the corresponding shortcut keys when needed. The prompt content is dynamically adjusted according to the user's current operating context. 5) General technical framework design: A general technical framework that can be applied to a variety of software was designed, eliminating the need for separate development for each software.

[0132] Figure 3 This is a flowchart illustrating the AI ​​Agent intelligent shortcut key recommendation method based on a multimodal large language model provided in this application embodiment, as follows: Figure 3 As shown, the intelligent shortcut key recommendation method may include steps S301 to S308: Step S301: The user opens the target software.

[0133] Install and launch the AI ​​Agent on the electronic device, and simultaneously select and open the target software (i.e., the target application). The AI ​​Agent begins running and initializes the multimodal large language model. The target software needs to be selected in advance because the AI ​​Agent needs to monitor and analyze the target software to collect accurate operational data.

[0134] Step S302: Conduct user behavior monitoring and data collection.

[0135] The AI ​​Agent monitors user actions in the background, collecting multimodal data (i.e., action data). It captures action events (clicks / shortcut key triggers) through low-level API hooks (such as Windows UI Automation). It uses Optical Character Recognition (OCR) technology to recognize interface button text and combines this with a YOLO CV model to detect interface control types (buttons / menus / canvas areas) (i.e., application interface data). Data collection ensures it complies with privacy policies and does not infringe on user privacy.

[0136] The AI ​​Agent correlates data from various modalities in several ways: It adds timestamps to each operation event to ensure temporal alignment of data from different modalities. It uses OCR and CV technologies to identify the UI elements where the current operation event occurs and associates the operation event with the UI context. Technicians arrange the operation events chronologically to construct an operation sequence and analyze the correlations between the operations. By analyzing the operation sequence and UI context, it identifies the task the user is performing and associates the operation event with the task.

[0137] By incorporating the multimodal understanding capabilities of large language models, researchers input data from different modalities into the large language model, enabling it to automatically learn the relationships between these data.

[0138] Step S303: Perform multimodal data processing and preprocessing.

[0139] The collected data is cleaned and preprocessed to prepare it as input for a multimodal large language model.

[0140] Data cleaning can include: removing duplicate operation records, such as clicking the same button multiple times consecutively; filtering out irrelevant operation records, such as filtering out mouse movements on the screen; and correcting erroneous operation records, such as identifying and correcting erroneous clicks triggered by user misoperation.

[0141] Data preprocessing may include: converting operation records into an input format that machine learning models can understand, for example, converting user click events on the interface into screen coordinate information and corresponding button names; extracting key information from the operation records, such as operation type, operation goal, and operation time; and dividing the operation records into time windows, for example, dividing operation records into 5-minute time windows.

[0142] Define inefficient operation characteristics (i.e., meet the conditions for generating shortcut keys): such as clicking the same function button 3 times in a row, long path menu operation (level > 2).

[0143] Time window analysis: statistical analysis of user behavior patterns at different task stages (e.g., Photoshop retouching stage vs. export stage).

[0144] The logic for data processing and preprocessing is as follows: First, identify the user's task stage. For example, by analyzing the user's operation sequence and interface content, determine whether the user is performing image retouching or image export. Based on the task stage, adjust the data preprocessing strategy. For instance, in the image retouching stage, focus more on the user's drawing operations; in the image export stage, focus more on the user's saving operations.

[0145] Step S304: Input the preprocessed data into the multimodal large language model.

[0146] A multimodal large language model (i.e., the target model) analyzes the data to understand the user's operating habits and intentions. For example, the input might be that the user clicked the layer merge button 3 times via the second-level menu in the past 5 minutes. The current interface is a Photoshop canvas with 5 unnamed layers, and the history shows that Ctrl+E (the layer merge shortcut) has never been used. The model output could be in the following form: json { "recommendation":"Ctrl+E", "reason": "The system detected that the user merged layers multiple times via the menu, and there are currently multiple unnamed layers, indicating a high probability of a merge operation." "confidence": 0.87, The recommended shortcut key is Ctrl+E. The reason is that the user has been detected merging layers multiple times through the menu, and there are currently multiple unnamed layers, so the probability of merging operations is high. Confidence level: 0.87.

[0147] Here, the LLM model outputs the keyboard shortcuts to be displayed for different applications and different interfaces within those applications, based on user behavior and interface scenarios. The model considers the following factors: application functionality: different applications have different functions, thus requiring different keyboard shortcuts; application interface: different application interfaces have different layouts and elements, thus requiring different keyboard shortcut suggestions; user operating habits: different users have different operating habits, thus requiring different keyboard shortcut recommendations; user's current operating context: different operating contexts require different keyboard shortcuts.

[0148] Step S305: Based on the analysis results of the model, generate a shortcut key recommendation scheme.

[0149] Perform a shortcut conflict check to ensure that the new shortcut will not conflict with existing shortcuts or system shortcuts.

[0150] Keyboard shortcut conflict detection needs to consider the following aspects: Intra-application shortcut conflicts: Ensure that recommended shortcuts do not conflict with existing shortcuts in the application; Shortcut conflicts between different applications: Try to avoid conflicts between recommended shortcuts and commonly used shortcuts in other applications; System-wide shortcut conflicts: Ensure that recommended shortcuts do not conflict with global shortcuts in the operating system or other software.

[0151] Step S306: Apply the recommended shortcut key settings to the software and prompt the user for confirmation.

[0152] Apply the recommended keyboard shortcuts to the software, or prompt the user for confirmation. Here, prompting the user for confirmation can support voice feedback, such as: "Do you want to enable auto-save keyboard shortcuts for the current user?".

[0153] Users can choose to accept, modify, or reject the recommended keyboard shortcut settings.

[0154] Step S307: When the user uses the software, the shortcut key prompt module starts working.

[0155] When the user uses the software, the shortcut key suggestion module starts working. This module displays shortcut key suggestions related to the current operation in the lower right corner of the screen. Figure 4 This is a schematic diagram of the shortcut key prompt module provided in the embodiments of this application, such as... Figure 4 As shown, the shortcut key prompt module 401 is displayed in the lower right corner of the screen, and 402 is a physical button, such as a keyboard.

[0156] Step S308: Collect user feedback on shortcut key settings and prompts.

[0157] Based on user feedback, we adjust model parameters and recommendation strategies, and continue to optimize them.

[0158] Figure 5 This is a schematic diagram of the architecture of the intelligent shortcut key recommendation method provided in the embodiments of this application, such as... Figure 5As shown, the architecture includes a user 501, a software interface 502, a data acquisition module 503, a behavior analysis engine 504, a multimodal large language model 505, and a recommendation strategy controller 506. The user 501 performs operations in the software (such as clicking "Save" multiple times through the menu). The software interface 502 records the operation log (click path, time consumption) and captures the current screen content. The data acquisition module 503 extracts interface text (such as button names) through OCR and identifies control types (such as menu levels) through CV. The behavior analysis engine 504 identifies high-frequency and low-efficiency operations (such as clicking the same function 3 times) and divides tasks into stages (such as edit / export). The high-frequency and low-efficiency operations and task stage divisions are input into the multimodal large language model 505 to obtain a shortcut key recommendation scheme. The recommendation strategy controller 506 confirms with the user whether to generate shortcut keys. If so, the software interface 502 displays a floating card (at least including shortcut key combinations) in the lower right corner of the screen, and the user 501 provides feedback on the shortcut key selection.

[0159] To better implement the above-described embodiments, this application also provides specific implementation methods for determining inefficient operations based on multidimensional data.

[0160] Inefficient operations (i.e., target operation tasks) can be identified through user behavior feature analysis and interface semantic understanding. They can be identified through user behavior feature analysis, such as a Photoshop user clicking the menu five times consecutively to merge layers. They can also be identified through temporal behavior patterns, using a hybrid network of Temporal Convolutional Network - Long Short-Term Memory Network (TCN-LSTM) to analyze operation sequences. Inefficient operations are characterized by high-frequency undo / redo operations, periodic repetitive actions, and abnormal pauses. Furthermore, they can be identified by constructing an interface interaction heatmap matrix. The inefficiency criterion is that the high-density click area deviates from the core functional area by more than a threshold (e.g., dense operations appearing outside the Photoshop toolbar area).

[0161] Inefficient operations can also be identified through interface semantic understanding: for example, semantic consistency detection uses a multimodal large language model to perform semantic parsing of the current interface elements (text, icons, prompts, etc.), and then combines this with the user's interaction actions to detect whether the user's actions are consistent with the interface design. Figure 1For example, if a user still performs multiple redundant steps when there are clear shortcut prompts on the interface, this behavior can be judged as inefficient. Another approach is to analyze the continuity of content and task flow: combining the context information of the current interface, determine whether the user is executing a continuous task flow (i.e., multiple consecutive operation events). If there are breaks in the operation (e.g., an operation suddenly appears in the task flow that is unrelated to the current context), it may indicate a misunderstanding of the user's understanding of the current interface or tool, leading to inefficient operation.

[0162] This application also provides an embodiment for identifying and illustrating inefficient operations in cross-stage tasks. To identify inefficient operations in cross-stage tasks (such as image editing -> export), the following steps can be taken: First, track the complete workflow of the user during use and record the continuous operation sequence and interface visual changes generated when the user crosses different interfaces, different applications or different operation modes (such as editing, setting, exporting, etc.).

[0163] Secondly, understanding the stages and goals, the Multimodal Large Language Model (MLLM) analyzes the operations and interface screenshots to infer the user's overall task goals (such as batch processing and exporting photos) and the specific stage they are currently in (photo editing stage, export settings stage).

[0164] Next, identify cross-stage patterns, such as repetitive work detection: detect whether the user repeatedly performs operations on multiple items (images, files) in a later stage (such as export) that could have been completed in an earlier stage (such as image retouching) through actions and batch scripts (e.g., setting the same export parameters one by one). Inefficient data transfer detection: analyze the data transfer methods during stage transitions, including a comparison between pasting screenshots and copying and pasting data, as well as cases where inter-application integration features are not used.

[0165] Next, compare the identified patterns with the pre-stored knowledge base (which may include the application's best practices, batch processing functions, and integration features).

[0166] Finally, in the judgment and recommendation process, if the user's cross-stage operation mode differs significantly from the efficient methods in the knowledge base (i.e., preset cross-stage operation events) (e.g., not using the batch export function, but choosing to export one by one), then the operation mode will be judged as an inefficient mode, and the user will be recommended to use the corresponding batch processing or integrated processing function.

[0167] This application's embodiments can improve work efficiency: by automatically recommending and setting shortcut keys, users can perform frequently used operations more quickly. It reduces learning costs: users don't need to spend a lot of time memorizing shortcut keys; real-time shortcut key prompts help users gradually become familiar with them. It achieves a personalized experience: shortcut key settings are based on the user's personal usage habits, providing a more personalized operating experience. It has a high degree of intelligence: utilizing a multimodal large language model, it achieves a deep understanding of user behavior and provides more accurate shortcut key recommendations. It is highly versatile: the framework design is versatile and applicable to various software and application scenarios.

[0168] It should be noted that the description of the device embodiments in this application is similar to the description of the method embodiments described above, and has similar beneficial effects as the method embodiments; therefore, it will not be repeated. For technical details not disclosed in the device embodiments, please refer to the description of the method embodiments in this application for understanding.

[0169] In other embodiments, the apparatus provided in this application can also be implemented in hardware. As an example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the audio processing method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0170] It should be noted that, in the embodiments of this application, if the above-described audio processing method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a terminal to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0171] This application provides a storage medium storing executable instructions. When the executable instructions are executed by a processor, the processor will execute the audio processing method provided in this application.

[0172] In some embodiments, the storage medium may be a computer-readable storage medium, such as a ferromagnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic surface memory, optical disc, or a compact disk-read-only memory (CD-ROM); or it may be a device that includes one or any combination of the above-mentioned memories.

[0173] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0174] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts within a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files storing one or more modules, subroutines, or code sections). As an example, executable instructions may be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0175] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0176] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, or apparatus that includes a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components may be combined, or integrated into another system, or some features may be ignored or not performed.

[0177] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A data determination method, comprising: obtaining operation data and application interface data corresponding to a target application program; determining a target operation task for the target application program based on the operation data and the application interface data; the target operation task being an operation that meets a shortcut key generation condition; determining shortcut key recommendation data corresponding to the target operation task, so as to realize a function corresponding to the target operation task based on the shortcut key recommendation data.

2. The data determination method of claim 1, wherein determining a target operation task for the application program based on the operation data and the application interface data comprises: determining an association between the operation data and the application interface data; determining the target operation task based on the association.

3. The data determination method of claim 2, wherein the operation data comprises a plurality of operation events, and the application interface data comprises application interface elements; determining an association between the operation data and the application interface data comprises: determining a timestamp corresponding to each operation event; sorting the operation events based on the timestamps to obtain a first sorting result; based on the first sorting result and the application interface elements corresponding to the operation events, establishing an association between the operation events and the application interface elements.

4. The data determination method of claim 3, wherein the application interface elements comprise first application interface elements of a functional area and second application interface elements of a non-functional area; determining the target operation task based on the association comprises: if, within a first preset time period, the number of operation events corresponding to a same first application interface element meets a first threshold condition, determining an operation task corresponding to the operation events as a target operation task; if, within a second preset time period, the number of operation events corresponding to a specific second application interface element meets a second threshold condition, determining a target first application interface element in a functional area in an interface in which the specific second application interface element is located; and determining an operation task of operation events corresponding to the target first application interface element as a target operation task.

5. The data determination method of claim 3, wherein determining the target operation task based on the association comprises: performing semantic understanding on the application interface elements to obtain a semantic understanding result; determining the target operation task based on the semantic understanding result and the association.

6. The data determination method of claim 5, wherein determining the target operation task based on the semantic understanding result and the association comprises: If the function performed by multiple consecutive operation events is the same as the semantic understanding result corresponding to the third application interface element, the operation task corresponding to the multiple consecutive operation events is determined as the target operation task; wherein, the third application interface element corresponding to the multiple consecutive operation events is located on the interface of the target application; the third application interface element includes at least text, icons or prompts on the application interface. or, Based on the semantic understanding results of multiple consecutive application interface elements, a continuous operation task corresponding to the multiple application interface elements is determined; if all operation events corresponding to the multiple consecutive application interface elements include other operation events besides implementing the continuous operation task, the continuous operation task is determined as the target operation task; wherein, the semantic understanding results include at least the contextual association information between the multiple consecutive application interface elements.

7. The data determination method according to any one of claims 1 to 6, wherein the operation data includes multiple operation events, and the application interface data includes interface jump information of the application interface; The step of determining the target operation task for the application based on the operation data and the application interface data includes: Based on the multiple operation events and the interface jump information of the application interface, multiple stage tasks for the target application are determined. Identify candidate operation events between two adjacent phase tasks; If the candidate operation event is different from the preset cross-stage operation event, the task corresponding to the candidate operation event is determined as the target operation task; wherein, the preset cross-stage operation event represents the operation event with the least time consumption when executing the two adjacent stage tasks.

8. The data determination method according to any one of claims 1 to 6, wherein the operation data includes a plurality of operation events, and the data of the application interface includes application interface elements; The process of determining the shortcut key recommendation data corresponding to the target operation task includes: Based on the operation events and application interface elements corresponding to the target operation task, create description information corresponding to the target operation task; The description information and the historical shortcut key information of the target application are input into the target model to obtain the shortcut key recommendation data; the description information can guide the target model to perform reasoning, and the shortcut key recommendation data is different from the existing shortcut keys in the target application, other applications different from the target application, and operating systems.

9. The data determination method according to any one of claims 1 to 6, wherein the shortcut recommendation data includes identification information of a shortcut key. The data determination method further includes: Upon detecting entry into the target application interface, the identification information is output to a preset area of ​​the target application interface; the identification information is then associated with the target application interface corresponding to the target operation task. In response to obtaining a selection operation for the identification information, the function corresponding to the selection operation is executed.

10. A data determining apparatus, the data determining apparatus comprising: The acquisition module is used to acquire operation data and application interface data corresponding to the target application. The first determining module is used to determine the target operation task for the target application based on the operation data and the application interface data. The target operation task is an operation that meets the conditions for generating shortcut keys; The second determining module is used to determine the shortcut key recommendation data corresponding to the target operation task, so as to realize the function corresponding to the target operation task based on the shortcut key recommendation data.

Images