Operation decision-making method and device

Abstract

The invention provides an operation decision-making method and device, and relates to the technical field of artificial intelligence, in particular to the technical field of intelligent agent decision-making cache, large models and cloud mobile phones. A specific embodiment of the method comprises the following steps: performing perceptual hash calculation on a user interface screenshot to generate an image perceptual hash value; performing hash calculation on the user intention text to generate a user intention hash value; performing hash calculation on the operation context to generate an operation context hash value; generating a composite cache key based on the image perception hash value, the user intention hash value and the operation context hash value; the cache is queried based on the composite cache key, in response to query success, an operation decision result corresponding to the composite cache key is obtained from the cache, and the cache correspondingly stores the reference composite cache key and the reference operation decision result.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to the fields of intelligent agent decision caching, large models, and cloud phone technology. Background Technology

[0002] With the development of artificial intelligence technology, visual understanding capabilities based on Large Language Models (LLMs) (Vision-Language Models, VLAs) have become an important direction in the field of mobile device automation. Traditional mobile automation solutions mainly fall into three categories: coordinate-based script automation, element recognition-based automation, and machine learning-based automation. In recent years, multimodal large models, with their strong visual understanding and reasoning capabilities, have driven the implementation of general mobile automation based on AI (Artificial Intelligence) agents. These models are adaptable to scenarios such as batch automation and regression testing, significantly improving the versatility and adaptability of automation.

[0003] There are currently several technical approaches to automation in mobile devices: one is response caching based on text hashing; the second is caching based on pixel-level image similarity; the third is structured caching based on UI (User Interface) element trees; and the fourth is model-native Prompt Cache. These approaches provide technical paths for optimizing automation efficiency from different dimensions. Summary of the Invention

[0004] This disclosure provides an operational decision-making method, apparatus, device, storage medium, and program product.

[0005] In a first aspect, embodiments of this disclosure propose an operation decision method, comprising: performing a perceptual hash calculation on a screenshot of a user interface to generate an image perceptual hash value; performing a hash calculation on user intent text to generate a user intent hash value; performing a hash calculation on an operation context to generate an operation context hash value; generating a composite cache key based on the image perceptual hash value, the user intent hash value, and the operation context hash value; querying the cache based on the composite cache key; and, in response to a successful query, retrieving the operation decision result corresponding to the composite cache key from the cache, wherein the cache correspondingly stores a reference composite cache key and a reference operation decision result.

[0006] Secondly, embodiments of this disclosure propose an operation decision-making device, comprising: a first calculation module configured to perform perceptual hash calculation on a screenshot of a user interface to generate an image perceptual hash value; a second calculation module configured to perform hash calculation on user intent text to generate a user intent hash value; a third calculation module configured to perform hash calculation on an operation context to generate an operation context hash value; a generation module configured to generate a composite cache key based on the image perceptual hash value, the user intent hash value, and the operation context hash value; and a query module configured to query the cache based on the composite cache key, and in response to a successful query, retrieve the operation decision result corresponding to the composite cache key from the cache, and the cache correspondingly stores a reference composite cache key and a reference operation decision result.

[0007] Thirdly, embodiments of this disclosure provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method as described in the first aspect.

[0008] Fourthly, embodiments of this disclosure provide a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the methods described in the first aspect.

[0009] Fifthly, embodiments of this disclosure provide a computer program product including a computer program that, when executed by a processor, implements the method described in the first aspect.

[0010] The key or essential features of the embodiments disclosed herein are not intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings. The drawings are provided for a better understanding of the invention and are not intended to limit the scope of this disclosure. Wherein: Figure 1 This is a flowchart of an embodiment of the operational decision-making method according to this disclosure; Figure 2 This is a flowchart of yet another embodiment of the operational decision-making method according to this disclosure; Figure 3 This is a layered architecture diagram of an intelligent agent decision caching system; Figure 4 This is a sequence diagram of the entire process of intelligent agent decision caching; Figure 5 This is a flowchart of the closed-loop management process for the success rate of intelligent agent caching decisions; Figure 6 This is a sequence diagram of intelligent agent loop detection and caching decision protection; Figure 7 This is a sequence diagram of temporary disabling and recovery after a smart agent cache decision fails; Figure 8 This is a flowchart of cache decision-making and loop protection driven by the execution results of intelligent agent tools; Figure 9 This is a full-link flowchart of the intelligent agent decision caching process; Figure 10 This is a schematic diagram of the architecture of a globally shared cache in a multi-user cloud phone scenario; Figure 11 This is a flowchart of the screenshot hash value generation process based on perceptual hashing; Figure 12 This is a flowchart of Hamming distance calculation and cache matching for image-aware hashing; Figure 13 This is a multi-module collaborative timing flowchart of the intelligent agent decision caching; Figure 14 This is a flowchart of the LRU priority eviction and new decision insertion process for the decision cache; Figure 15 It is a logical diagram relating the parameter configuration and matching accuracy of the image-aware hash cache; Figure 16 This is a schematic diagram of the structure of an embodiment of the operation decision device according to the present disclosure; Figure 17 This is a block diagram of an electronic device used to implement the operation decision method of the embodiments of this disclosure. Detailed Implementation

[0012] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0013] It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0014] Figure 1 A flow 100 of an embodiment of an operational decision-making method according to the present disclosure is shown. The operational decision-making method includes the following steps: Step 101: Perform perceptual hash calculation on the user interface screenshot to generate an image perceptual hash value.

[0015] In this embodiment, the execution entity of the operation decision method can perform perceptual hash calculation on the user interface screenshot to generate an image perceptual hash value.

[0016] Compared to traditional hashing, the pHash (Perceptual Hash) algorithm preserves the semantic features of images, is insensitive to slight color changes, minor displacements, compression, and scaling, while being sensitive to semantic changes such as the addition or deletion of UI elements and text alterations. It accurately balances the accuracy and robustness of UI state recognition. The specific steps are as follows: The first step is to shrink the user interface screenshot to a preset size (usually 32×32 pixels) to generate a reduced image, thereby reducing computational complexity and preserving core features.

[0017] The second step is to convert the reduced image to grayscale to generate a grayscale image, thus eliminating the interference of color factors on hash calculation.

[0018] The third step is to perform a Discrete Cosine Transform (DCT) on the grayscale image to generate frequency domain features and extract the frequency domain information of the image.

[0019] The fourth step is to extract the 8×8 low-frequency coefficients in the upper left corner from the frequency domain features. The low-frequency coefficients can reflect the overall structural features of the image and ignore the interference of details.

[0020] The fifth step is to calculate the average value of the low-frequency coefficients, which will serve as the criterion for subsequent hash value generation.

[0021] The sixth step involves comparing each low-frequency coefficient with the average value. If the coefficient is greater than the average value, it is recorded as 1; otherwise, it is recorded as 0. Finally, a fixed-length binary image perception hash value is generated.

[0022] Step 102: Perform a hash calculation on the user intent text to generate a user intent hash value.

[0023] In this embodiment, the aforementioned execution entity can perform hash calculation on the user intent text to generate a user intent hash value.

[0024] User intent text refers to natural language text that represents the user's core task requirements and, after standardization, can be transformed into structured information. It is a crucial input for AI agents to understand user commands and, together with user interface screenshots and operational context, construct a composite cache key to generate appropriate operational decisions. The specific steps for hashing user intent text are as follows: The first step is to standardize the user intent text, including removing leading and trailing spaces and converting it to lowercase, to generate standardized user intent text and ensure that intent text with the same semantics is processed in a consistent manner.

[0025] The second step is to construct a user intent string based on the standardized user intent text and the iteration round, in the format "{normalized}:iter {iteration}", to distinguish the same intent in different iteration rounds.

[0026] The third step is to perform MD5 (Message-Digest Algorithm 5) hash calculation on the user intent string, and extract a preset number of characters (usually 8 characters) from the beginning of the calculation result to generate a compact user intent hash value, which is convenient for storage and matching.

[0027] Step 103: Perform a hash calculation on the operation context to generate an operation context hash value.

[0028] In this embodiment, the aforementioned execution entity can perform hash calculation on the operation context to generate an operation context hash value.

[0029] The operation context can be the tool call history of the most recent N steps (N defaults to 3), and its hash calculation can solve the decision-making problem of the same UI under different operation sequences. The specific steps are as follows: The first step is to connect the operation contexts of each step in sequence using preset symbols (such as "->") to generate an operation context string, for example, click(x=100,y=200)->type(text='hello')->press_back().

[0030] The second step is to perform an MD5 hash calculation on the operation context string and extract a preset number of characters (usually 8 characters) from the beginning of the calculation result to generate the operation context hash value.

[0031] It should be noted that if the tool history is empty, it can return the fixed value "nocontext".

[0032] Step 104: Generate a composite cache key based on the image perception hash value, the user intent hash value, and the operation context hash value.

[0033] In this embodiment, the aforementioned execution entity can generate a composite cache key based on the image perception hash value, the user intent hash value, and the operation context hash value.

[0034] Composite cache keys can adopt a three-dimensional structure of "ImageHash:PromptHash:ContextHash", such as "a1b2c3d4e5f6:3d7e9fa2:a3b4c5d6". This three-dimensional structure can accurately match combined scenarios of "UI state + user intent + operation history", avoiding mismatch problems caused by single-dimensional cache keys, and ensuring that scenarios with the same UI and the same intent but different operation histories can be correctly distinguished. For example, in the settings page, the back operation may refer to returning to the home page or returning to the previous level, depending on the entry path. The three-dimensional cache key can accurately distinguish between these two scenarios through context hashing, avoiding false hits.

[0035] Step 105: Query the cache based on the composite cache key. In response to a successful query, retrieve the operation decision result corresponding to the composite cache key from the cache. The cache also stores the reference composite cache key and the reference operation decision result.

[0036] In this embodiment, the execution entity can query the cache based on the composite cache key. If the query is successful, the operation decision result corresponding to the composite cache key can be obtained from the cache.

[0037] The cache can store a reference composite cache key and a reference operation decision result. The cache query process can employ a strategy of prioritizing exact matching and using fuzzy matching as a fallback. The specific steps are as follows: The first step is to perform an exact match between the composite cache key and a reference composite cache key in the cache. An exact match can be a full consistency comparison. The composite cache key is compared to a reference composite cache key in the cache for complete consistency; if a reference composite cache key exists in the cache that is exactly the same as the composite cache key, the exact match is successful; otherwise, the exact match fails.

[0038] The second step is to retrieve the candidate operation decision result corresponding to the candidate composite cache key that is exactly matched with the composite cache key from the cache, and use it as the operation decision result to ensure fast matching in the scenario of complete consistency.

[0039] The third step, in response to a failure of an exact match, involves a fuzzy match between the composite cache key and a reference composite cache key in the cache. The core of fuzzy matching can be calculating the Hamming distance (i.e., the number of different bits in the two binary hash values) between the current image perceptual hash value and the reference image perceptual hash value in the reference composite cache key. The Hamming distance is calculated between the image perceptual hash value and the reference image perceptual hash value in the reference composite cache key. If a reference composite cache key exists in the cache with a Hamming distance not greater than a preset distance threshold (default value is 5, adjustable within the range of 0-10 depending on the scenario), the fuzzy match succeeds; otherwise, the fuzzy match fails.

[0040] The fourth step is to respond to a successful fuzzy match by retrieving the candidate operation decision result corresponding to the candidate composite cache key that is successfully matched with the composite cache key (with the smallest Hamming distance) from the cache. This result is used as the operation decision result to adapt to scenarios where the UI has minor changes but the semantics remain consistent.

[0041] This disclosure provides an operation decision-making method that achieves efficient identification and decision reuse of recurring UI states through a three-dimensional composite cache key and a "precise + fuzzy" matching strategy. This significantly reduces the number of large model calls, lowers costs, and improves response speed. Simultaneously, the introduction of a perceptual hash algorithm ensures the robustness of UI state recognition, adapting to various minor UI changes in mobile device automation scenarios.

[0042] Figure 2 A flow 200 of another embodiment of the operational decision-making method according to this disclosure is shown. The operational decision-making method includes the following steps: Step 201: Perform perceptual hash calculation on the user interface screenshot to generate an image perceptual hash value.

[0043] Step 202: Perform a hash calculation on the user intent text to generate a user intent hash value.

[0044] Step 203: Perform a hash calculation on the operation context to generate an operation context hash value.

[0045] Step 204: Generate a composite cache key based on the image perception hash value, the user intent hash value, and the operation context hash value.

[0046] Step 205: Query the cache based on the composite cache key to determine whether the query was successful.

[0047] Step 206: In response to a successful query, retrieve the operation decision result corresponding to the composite cache key from the cache, and store the reference composite cache key and the reference operation decision result in the cache.

[0048] In this embodiment, the specific operations of steps 201-206 have been described. Figure 1 Steps 101-105 in the illustrated embodiments are described in detail and will not be repeated here.

[0049] Step 207: In response to query failure, input the user interface screenshot, user intent text, and operation context into the large language model, and output the operation decision result.

[0050] In this embodiment, if the query fails, the aforementioned execution entity can input the user interface screenshot, user intent text, and operation context into the large language model and output the operation decision result.

[0051] Query failures can include cases where neither exact match nor fuzzy match is found. In such cases, reasoning and decision-making are required through a large language model. The large language model can receive visual information from user interface screenshots, standardized user intent text, and operation context. Based on the ReAct (Reasoning + Acting) model, it performs multimodal understanding and reasoning to generate operation decision results that include information such as tool call type and parameter configuration (e.g., {"action":"tap","x":360,"y":384}).

[0052] Step 208: In response to query failure, store the composite cache key and operation decision result in the cache.

[0053] In this embodiment, if the query fails, the aforementioned execution entity can store the composite cache key and the operation decision result in the cache.

[0054] The cache storage procedure can combine the LRU (Least Recently Used) eviction policy and TTL (Time To Live) management to ensure efficient utilization of cache resources. The specific steps are as follows: The first step is to count the number of cached entries currently stored in the cache and determine whether it exceeds the preset entry threshold (the default value is 1000 entries, which can be configured).

[0055] The second step is to directly store the composite cache key and the operation decision result in the cache in response to the determination that the number of cache entries is not greater than the preset entry number threshold. The cache entries can contain information such as image hash, hint hash, context hash, user hint, tool call string, inference digest, screenshot URL (Uniform Resource Locator), creation time, iteration round, number of hits, execution success rate, tool history, etc.

[0056] The third step involves deleting a preset percentage (usually 20%) of cached entries from the cache in ascending order of priority, in response to the determination that the number of cached entries exceeds a preset threshold. The priority calculation rule can be "Priority = (hit_count × 10) + age_hours". Here, hit_count is the number of hits, and age_hours is the duration (in hours) the cached entry has existed. Entries with lower priority are evicted first. Subsequently, a new composite cache key and the operation decision result are stored in the cache.

[0057] Cache entries can also undergo lifecycle management and optimization. The specific steps are as follows: The first step is to execute the operation decision and generate the operation execution result (success or failure).

[0058] The second step involves updating the success rate of the operation decision based on the execution results using a moving average algorithm. The calculation formula is "new_rate = (old_rate × hit_count + current_result) / (hit_count + 1)". Here, old_rate represents the historical success rate (0.0-1.0), and current_result represents the current execution result (1.0 for success, 0.0 for failure). This algorithm adaptively balances the weights of historical and new data; the more hits, the smaller the impact of a single result on the overall success rate.

[0059] The third step is to delete the corresponding composite cache key and operation decision result from the cache when the success rate is determined to be lower than the preset success rate threshold (the default value is 0.6, which can be configured in the range of 0.0-1.0), automatically eliminating low-quality cache and avoiding the reuse of wrong decisions.

[0060] The fourth step is to update the remaining disabled round counter based on the operation execution result. If the operation decision result comes from the large language model or the execution is successful, the remaining disabled round counter remains unchanged; if the operation decision result comes from the cache and the execution fails, the remaining disabled round counter is set to a preset value (default 2 rounds, configurable within the range of 1-5).

[0061] The fifth step is to respond to the determination that the remaining disabled round counter is 0 and retrieve the operation decision result from the cache normally. If the remaining disabled round counter reaches the first preset number of times threshold, the cache query is skipped in this round, and the user interface screenshot, user intent text and operation context are input into the large language model to output the decision. The remaining disabled round counter is decremented by 1 to provide a cooling-off period for the system to re-evaluate after the cache fails.

[0062] Step 6: Compare the current image perception hash value with the previous image perception hash value, and update the consecutive identical hash counter. If the hash values ​​are the same, increment the counter by 1; otherwise, set the counter to 0.

[0063] Step 7: In response to the determination that the consecutive identical counters have reached the second preset number threshold (default 2 times, configurable within the range of 1-5), it is marked as a possible infinite loop, the cache query is skipped, and the large language model is forcibly called to re-infer, breaking the infinite loop pattern of "error cache → execution failure → UI unchanged → hit error cache again".

[0064] This disclosure provides an operational decision-making method that, based on a basic process, supplements cache storage optimization, success rate feedback loop, failure protection, and loop detection mechanisms. This further improves the accuracy of cache decisions, system robustness, and resource utilization efficiency. Simultaneously, through a non-intrusive design, it can be seamlessly integrated into existing AI agent workflows. This method can significantly reduce AI model invocation costs, greatly improve response speed, and significantly increase cache hit rate while ensuring decision accuracy.

[0065] Figure 3 The diagram shows the layered architecture of the intelligent agent decision caching system. The architecture is divided into an application layer, an interface layer, a cache core layer, and a data storage layer from top to bottom. Each layer achieves efficient management and full-process collaboration of the decision cache through modular division of labor.

[0066] The application layer includes an AI agent workflow engine (LangGraph), a task scheduler, and a cost statistics module. The AI ​​agent workflow engine is the core scheduling component of the system, responsible for receiving user task instructions and driving the entire process of subsequent screenshot collection, hash calculation, cache query, etc. The task scheduler supports parallel / serial scheduling of multiple tasks, adapting to scenarios such as batch device automation and multi-user concurrency. The cost statistics module collects data such as cache hit count, token consumption, and API (Application Programming Interface) call cost in real time, providing a quantitative basis for system cost optimization.

[0067] The interface layer provides three types of standardized interfaces to achieve decoupled communication between the application layer and the cache layer: Get_cached_decision interface: receives cache query requests from the application layer and triggers the matching logic of the cache layer; cache_decision interface: responsible for writing the newly generated operation decision and the corresponding composite cache key into the cache; clear / stats / close management interface: supports operation and maintenance operations such as cache cleanup, statistical data query, and cache instance shutdown, improving the maintainability of the system.

[0068] The core of the cache layer is the main carrier of system functionality, comprising four functional modules: Composite Key Generator: Generates a three-dimensional composite cache key based on "ImageHash + PromptHash + ContextHash" to ensure accurate matching of "UI state + user intent + operation history"; Fuzzy Matching Engine: Quantizes the similarity of image-aware hashes through Hamming distance calculation, enabling fuzzy matching in scenarios with minor UI changes; Cache Storage Manager: Integrates LRU eviction policy and TTL expiration management (TTL defaults to 86400 seconds, configurable from 600 to 86400 seconds), dynamically cleaning up low-value, expired cache entries and optimizing cache resource usage; Image-Aware Hash Calculator: Processes UI screenshots using the pHash algorithm to generate image-aware hash values ​​that balance robustness and efficiency, being insensitive to compression, scaling, and small displacements, but sensitive to semantic changes.

[0069] The data storage layer contains three types of data carriers: Decision cache table (in-memory dictionary): stores composite cache keys and corresponding operation decision results, supporting high-speed read and write; Statistical data: records indicators such as cache hit count, cost consumption, and execution success rate, providing data support for system optimization; Configuration parameters: manages system parameters such as Hamming distance threshold, TTL duration, and LRU eviction ratio, supporting scenario-based adaptation.

[0070] The core innovation of this architecture lies in its modular layered design and global shared caching mechanism: based on the deterministic logic of "same UI + same prompt + same context → same decision", when 100 devices perform the same task, only the first device needs to call the large language model to generate the decision, and subsequent devices can directly reuse the cache, which can reduce API call costs and form a significant scale effect; the introduction of double-checked locking, three-layer locking mechanism (write_lock, cache_lock, pending_writes) and TOCTOU (Time-of-check to Time-of-use, the check time and the use time are inconsistent) race condition protection solves the cache access conflict problem in high-concurrency scenarios and ensures data consistency and system stability.

[0071] This architecture features a non-intrusive design, allowing for on / off control via configuration files without requiring modification of business code. When disabled, the system incurs zero performance overhead and can be seamlessly integrated with existing cost statistics and logging systems without affecting the stability of existing workflows.

[0072] Figure 4 The diagram shows the complete process sequence of the intelligent agent decision caching, clearly presenting the interaction logic of core components such as the workflow engine, cache manager, hash calculator, fuzzy matching engine, cache storage, and LLM model, fully covering the entire chain of "cache query - matching judgment - decision execution - cache update".

[0073] The timing sequence can be as follows: Task initiation and cache query triggering: After receiving a user's task request, the workflow engine first obtains the current UI screenshot and then simultaneously sends a cache query request to the cache manager.

[0074] Image hash generation: The cache manager triggers the hash calculator to execute the pHash algorithm, converting the UI screenshot into a 256-bit image-aware hash value. Simultaneously, it combines the user intent hash and the operation context hash to generate a three-dimensional composite cache key. The hash calculation is executed asynchronously and does not block the main process.

[0075] Exact match determination: The cache manager performs a full match comparison between the composite cache key and the reference key in the cache storage. If an exact match is successful, it further checks whether the TTL of the cache entry has expired. If it has not expired, it updates the hit count (hit_count += 1), returns the cache decision to the workflow engine, and skips the LLM model call.

[0076] Fuzzy matching fallback: If exact matching fails, the fuzzy matching engine starts, traverses all candidate entries in the cache, and calculates the Hamming distance between the current image perceptual hash and the reference hash one by one. If there is a candidate entry with a Hamming distance ≤ a preset threshold, it is determined that the fuzzy matching is successful, and the cache decision of the best match is returned to the workflow engine.

[0077] LLM inference when a match is missed: If both exact and fuzzy matching fail, the cache manager returns a miss signal, and the workflow engine calls the LLM model inference to generate a new operation decision result.

[0078] Asynchronous caching of new decisions: As the workflow engine sends new decisions to the tool execution module, it triggers an asynchronous process: the hash calculator regenerates the image hash corresponding to the new decision, and the cache manager writes the composite cache key and the new decision into the cache storage; if the number of cache entries exceeds the threshold, the LRU cleanup mechanism is triggered. Asynchronous caching does not block the main process, thus avoiding impacting response speed.

[0079] Execution result feedback and cache update: After the tool execution module completes the decision execution, it feeds back the result (success / failure) to the workflow engine. The workflow engine then synchronizes with the cache manager to update the execution success rate of the corresponding cache entry.

[0080] In this process, the total time spent in cache hit scenarios is significantly reduced compared to the traditional full LLM call solution, and the response speed is significantly improved. At the same time, TTL verification and LRU cleanup ensure the effectiveness of the cache and resource utilization.

[0081] Figure 5The flowchart of the intelligent agent cache decision success rate closed-loop management is shown. Through the closed-loop logic of "cache decision execution → result feedback → success rate calculation → cache retention / cleanup", the cache quality is dynamically optimized.

[0082] The process can be as follows: Cache decision execution: The cache manager sends the cache decisions obtained from the query to the execution tool. After the execution tool completes the operation, it records the result of this execution (success / failure).

[0083] Success rate update: Based on the execution result, the cache manager calls the moving average algorithm to update the execution success rate corresponding to the cache decision (formula: new_rate = (old_rate × hit_count + current_result) / (hit_count + 1), where current_result is recorded as 1 for success and 0 for failure).

[0084] Success rate verification and cache optimization: The cache manager checks whether the success rate after the update reaches the preset threshold (default 0.6): If the success rate is ≥0.6, the cache entry is retained and used for subsequent query matching; if the success rate is <0.6, the composite cache key and result corresponding to the cache decision are automatically deleted to avoid low-quality cache polluting the system.

[0085] This closed-loop mechanism can adapt to changes in decision effectiveness: for example, if a cache entry hits once and has a historical success rate of 0.9, and the current execution fails, the new success rate is (0.9×1 + 0) / 2 = 0.45 (below the threshold and deleted); if it hits 10 times and has a historical success rate of 0.9, the new success rate after this failure is (0.9×10 + 0) / 11 ≈ 0.82 (above the threshold and retained), thus achieving precise and intelligent control of cache quality.

[0086] This closed-loop management significantly reduces the system's false matching rate, greatly improves the accuracy of decision-making, and effectively prevents error caching from continuously affecting system operation.

[0087] Figure 6 The sequence diagram of intelligent agent loop detection and cache decision protection is shown. Through the logic of "UI hash continuity detection + counter threshold judgment", the automatic identification and protection of infinite loop scenarios are realized, avoiding process blockage caused by cache errors.

[0088] The timing sequence can be as follows: UI hash generation and historical comparison: The workflow engine obtains the current UI screenshot, and the cache manager calculates the corresponding UI hash; the cycle detector retrieves the historical records and compares the current UI hash with the previous UI hash.

[0089] Counter status update: If the two UI hashes are different (UI changes), the consecutively identical counter (consecutive_count) is set to 0, and the workflow engine executes the normal cache query process; if the two UI hashes are the same (UI remains unchanged), the consecutively identical counter is incremented by 1.

[0090] Loop Threshold Judgment and Protection Trigger: If the consequent_count does not reach the preset threshold (default 2 times), the workflow engine continues to query the cache normally; if the consequent_count reaches the threshold, it is judged as a suspected infinite loop, the workflow engine skips the cache query, forces the LLM to re-infer and generate a new decision, thereby breaking the loop of "UI unchanged → cache hit → execution failure → UI still unchanged".

[0091] Baseline Update: Regardless of whether protection is triggered, the loop detector updates the previous UI hash to the current UI hash, providing a comparison baseline for the next iteration.

[0092] This mechanism achieves accurate identification and rapid loop breaking in infinite loop scenarios through lightweight hash comparison and counter logic, effectively improving the robustness of the system in complex scenarios.

[0093] It should be noted that in the configuration parameters, enable_loop_detection controls whether loop detection is enabled (default is true), and loop_detection_threshold can be adjusted in the range of 1-5 according to the actual scenario.

[0094] Figure 7 The diagram illustrates the temporary disabling and recovery sequence after a smart agent cache decision fails. Through the logic of "failure marking → disabling cache → decreasing rounds → recovering queries", a cooling-off period is provided for the erroneous cache to avoid repeated hits on invalid decisions.

[0095] The timing sequence can be as follows: Cache decision execution and failure recording: model_node (large language model node) queries the cache and obtains the decision, and sends it to tool_node (tool execution node) for execution; if tool_node returns an execution failure result, the system calls record_cache_failure() to record the cache decision failure event.

[0096] Disable round initialization: After receiving a failure signal, the cache manager sets the cache disable round counter (cache_disable_rounds) to the default value (2 rounds by default).

[0097] Next iteration: Cache disabled: Enter the next task iteration. The system first checks the cache_disable_rounds status: Since cache_disable_rounds>0, the cache lookup process is skipped, and LLM is directly called to re-infer and generate a new decision; after execution, cache_disable_rounds is decremented by 1 (becomes 1).

[0098] The next iteration: Disabling rounds decrement / restore: If cache_disable_rounds is still greater than 0, continue to skip cache lookups and call LLM inference, while decrementing the counter by 1; when cache_disable_rounds is reduced to 0, the cache disabling period ends, and the system resumes the normal cache lookup process.

[0099] This process, together with the loop detection mechanism, forms a dual protection: even if the risk of looping still exists after the disabling period ends, loop detection can further trigger the protection logic, effectively ensuring the stability of the system in the scenario of cache decision failure.

[0100] It should be noted that in the configuration parameters, enable_failure_protection controls whether failure protection is enabled (default is true), and cache_disabled_rounds_after_failure can be adjusted in the range of 1-5 according to the complexity of the task.

[0101] Figure 8 The flowchart illustrates the cache decision-making and loop protection process driven by the execution result of the intelligent agent tool. It integrates the two-way protection logic of "caching strategy adjustment + loop risk detection" with the tool execution result as the core trigger condition.

[0102] The process can be as follows: Tool Execution and Result Branching: After the tool execution module completes the decision execution, it enters different branches based on the result (success / failure): If the execution is successful: the system maintains the current caching strategy and continues to use the cache for subsequent decisions; if the execution fails: the cache decision failure event is recorded and the cache disabling process is triggered (disabling for 2 rounds by default).

[0103] Cache disabling and LLM inference: When entering the next iteration, first check the cache disabling status, skip the cache query process, and directly call LLM to re-infer and generate a new decision.

[0104] UI state detection after new decision: After the new decision is executed based on LLM, check whether the UI has changed: If the UI has changed: it is determined that the new decision may have corrected the error, and the normal caching logic can be restored subsequently; If the UI has not changed: trigger the loop detection mechanism, force the LLM to be called again to re-infer, and avoid getting stuck in an infinite loop.

[0105] This process uses the tool's execution results as a guide to achieve dynamic and adaptive adjustment of caching strategies and protection mechanisms. It avoids repeated hits on incorrect decisions by disabling caching and blocks loop risks by detecting UI status, thus ensuring the effectiveness and stability of system decisions.

[0106] It should be noted that the system also maintains consecutive_failures (the current consecutive failure count), which is cleared after a success, providing data support for operation and maintenance analysis.

[0107] Figure 9 The diagram illustrates the entire process flow of intelligent agent decision caching, with "user request → workflow scheduling → cache matching → decision execution → cost statistics" as the core link. It integrates modules such as hash calculation, cache management, and large model inference to achieve end-to-end closed-loop management of decision caching.

[0108] The process can be as follows: User requests and workflow initiation: Users initiate commands at the user request layer (such as "open social media apps"), and the AI ​​agent workflow (LangGraph) receives the request and initiates the entire process.

[0109] UI screenshot and hash calculation: The workflow first calls the screenshot acquisition module to collect the current UI screenshot, and then generates the image-aware hash value through the image hash calculation module.

[0110] Cache query and decision routing: Cache queries are performed based on the generated hash value, triggering two branch logics: Cache hit: The cache decision is obtained directly from the image hash cache of the cache management layer, skipping the large model call; Cache miss: The large model is called to generate a new decision, which is converted into an executable instruction by the parsing tool and then asynchronously stored in the cache (if the cache capacity exceeds the limit, the LRU eviction policy is triggered).

[0111] Execution and Cost Statistics: After the tool execution module completes the decision execution, the cost statistics layer synchronously updates the cache hit count and token saving statistics, and generates a cost report to provide data support for resource optimization. The cost report includes indicators such as actual cost, cache savings, no-cache cost, and cost saving rate.

[0112] This end-to-end process enables end-to-end control of decision caching, significantly reducing API calls and execution costs in scenarios such as batch automation and regression testing. For example, to perform 5 steps on 100 cloud phones in batches, the traditional solution requires 500 API calls, with a total cost of ¥25 per batch and a total time of 33 minutes; this solution requires only 5 API calls, with a total cost of ¥0.25 per batch and a total time of 1.5 minutes, resulting in a 99% cost reduction and a 94% time saving.

[0113] Figure 10 This diagram illustrates the architecture of a globally shared cache in a multi-user cloud phone scenario. The core logic of this architecture leverages the principle of "single cache instance + multi-user reuse" to achieve economies of scale in decision caching. Global shared architecture: This solution covers multiple users / devices, including user A / device 1, user B / device 2, user C / device 3, and devices 4-100. All entities share the same global cache instance, breaking the cache isolation of a single user / device. Traditional caching systems use a user / session isolation design, while this solution's global singleton pattern enables cross-user reuse.

[0114] Cache sharing strategy: Based on the deterministic mapping logic of "same UI + same prompt → same decision", cross-user cache reuse is realized: When user A / device 1 executes the task for the first time, it calls LLM to generate a decision and writes it to the global cache, thus completing the cache establishment; when user B / device 2, user C / device 3 and device 4-100 subsequently execute the same task, they directly hit the global cache and reuse the existing decision results without having to call LLM again.

[0115] Aggregation effect reinforcement: As the number of users / devices increases, the number of effective decision entries in the global cache continues to accumulate, the cache hit rate gradually increases, forming a positive cycle of "more use → richer cache → higher hit rate → lower cost".

[0116] Taking a scenario where 100 cloud phones perform 5 steps in batches as an example: only 5 LLM calls from user A / device 1 are needed to establish the cache. The subsequent 495 operations from the remaining 99 devices all hit the cache, significantly reducing the number of API calls, lowering the total cost and time, and significantly improving the execution efficiency in multi-device scenarios. The theoretical basis of this architecture is that AI agent decision-making is deterministic (temperature = 0), the same input corresponds to the same decision, and it is independent of user identity; therefore, the cache can be securely shared across users.

[0117] To ensure concurrency safety, the architecture employs a double-checked locking mode, a three-layer locking mechanism (write_lock, cache_lock, pending_writes), and TOCTOU race condition protection to ensure cache access security and data consistency in high-concurrency scenarios.

[0118] Figure 11 The flowchart of screenshot hash value generation based on perceptual hashing is shown. Through the logic of "asynchronous processing + multi-specification hash adaptation", the efficient and flexible generation of image feature identifiers is achieved.

[0119] The process can be as follows: Asynchronous input processing: Taking the screenshot URL as input triggers an asynchronous image download, avoiding blocking the main process and improving the system's concurrency capabilities. The download timeout is set to 10 seconds to ensure process stability.

[0120] Image format standardization: The downloaded images are converted into PIL image objects, providing a unified standard data format for subsequent pHash algorithm processing.

[0121] pHash Feature Extraction: The pHash algorithm is applied to PIL image objects to extract visual feature information, ensuring that the hash value is robust to compression, scaling, and small displacements. The pHash algorithm principle includes: reducing the size to 32×32 pixels, converting to grayscale image, calculating DCT, retaining the 8×8 low-frequency coefficients in the upper left corner, calculating the average value, and generating a binary hash value.

[0122] Multiple hash size options: Based on the scenario's requirements for "efficiency-precision", the corresponding branch is selected through the hash size parameter to generate three hash value sizes: 8×8 size: outputs a 64-bit hash value, suitable for scenarios where computational efficiency is prioritized and precision requirements are moderate; 16×16 size: outputs a 256-bit hash value, balancing precision and efficiency, and is the default recommended size; 32×32 size: outputs a 1024-bit hash value, suitable for scenarios with extremely high matching precision requirements.

[0123] Output: The generated hash value of the corresponding length is converted into a string format and returned, which can be used for subsequent composite cache key construction or cache matching operations.

[0124] This process ensures system efficiency through asynchronous processing and adapts to the accuracy requirements of different scenarios with multi-specification hashing, providing basic image feature support for the accuracy and robustness of cache matching.

[0125] Figure 12 The flowchart of Hamming distance calculation and cache matching for image-aware hashing is shown. Through a two-layer logic of "precise matching + Hamming distance quantized fuzzy matching", the accuracy and adaptability of cache query are achieved.

[0126] The process can be divided into two parts: Hamming distance calculation logic (top left module): Taking the perceptual hash values ​​(H1, H2) of two images as input, it compares the hash bits through a bitwise XOR operation, and counts the number of bits where the XOR result is 1. This number of bits is the Hamming distance. For example, image 1 has a hash of 1010110100110101, and image 2 has a hash of 1011110100110101. They differ by only one bit, so the Hamming distance is 1. When the threshold is set to 5, they are considered similar.

[0127] The main process of cache matching is as follows: using the current image hash H1 as the query basis, it is executed with precise priority and fuzzy fallback: Exact match determination: Check if there is a hash value in the cache that is exactly the same as H1: If it exists, it is determined as an exact match and the corresponding cache decision is returned directly; if it does not exist, iterate through all cache reference hash values ​​and calculate the Hamming distance between H1 and the reference hash one by one.

[0128] Fuzzy matching determination: Based on the calculated Hamming distance, determine whether the distance is less than or equal to the threshold. If it is, it is determined to be a fuzzy hit and the corresponding cached decision is returned. If it is not, it is determined to be a miss and a new decision generation process is triggered.

[0129] This process quantifies hash similarity using Hamming distance, ensuring efficient and accurate matching in scenarios with perfect consistency, while also adapting to scenarios with minor UI changes. Through fuzzy matching and reuse of effective decisions, it improves the adaptability of the cache and the overall hit rate.

[0130] It should be noted that the Hamming distance threshold can be configured in the range of 0-10 via the image_hamming_threshold parameter, with a default value of 5.

[0131] Figure 13 The diagram illustrates the multi-module collaborative timeline of the intelligent agent decision caching system, clearly presenting the interaction logic between users, workflow engine, cache manager, image hasher, large model, and tool executor, achieving a fully closed-loop collaboration of "cache query - decision execution - result feedback - cache optimization".

[0132] The timing sequence can be as follows: Task initiation and UI capture: When a user initiates the command "Execute task: Open social software", the workflow engine receives the command, triggers the cache manager to capture the current UI, and obtains the UI screenshot of the current interface.

[0133] Hash calculation and cache query: The workflow engine transmits the UI screenshot to the image hasher to generate an image hash value (such as hash_val=1a2b3c4d); the image hasher synchronizes the hash value to the cache manager, and the cache manager queries the cache table.

[0134] Cache hit execution and feedback: If a cache hit occurs, the cache manager returns the cache key (cache_key) and updates the hit count (hit_count=1); the workflow engine sends the cache decision to the tool executor, and the tool executor returns the "execution result (success / failure)" after completing the call.

[0135] Success rate update and cache optimization: The workflow engine synchronizes the execution results to the cache manager, which calculates the success rate of the cache entry. If the success rate is less than the threshold, the low-quality cache eviction process is automatically triggered to remove the entry.

[0136] Fallback handling for cache misses: If a cache miss occurs, the workflow engine calls the large model to generate a new decision; the new decision is synchronized to the tool executor for execution, and at the same time, an asynchronous cache storage process is triggered to store the new decision and its corresponding hash value into the cache.

[0137] This process, through the sequential linkage of multiple modules, achieves efficient decision-making and reuse in cache hit scenarios, optimizes cache quality through success rate feedback, and ensures the reliability and stability of task execution by relying on large-scale model inference as a fallback. This multi-module collaboration enables the system to demonstrate significant advantages in scenarios such as batch mobile device automation, regression testing, and automation of repetitive operations.

[0138] Figure 14 The flowchart of LRU priority eviction and new decision insertion for decision cache is shown. Through the logic of "capacity determination → priority calculation → low priority eviction", the dynamic optimization of cache space and the orderly storage of new decisions are realized.

[0139] The process can be as follows: New decision insertion trigger: After the workflow engine generates a new decision, it triggers a request to add the new decision to the cache.

[0140] Cache capacity determination: The cache module determines whether the current number of entries exceeds the preset maximum threshold (max_entries). If it does not exceed the threshold, the new decision is directly inserted into the cache; if it exceeds the threshold, the LRU priority eviction process is initiated. The default value of max_entries is 1000 entries, which can be configured according to actual needs.

[0141] Priority Calculation and Sorting: The priority of each cached entry is calculated according to the formula (core dimension: hit_count (number of hits) + creat_at (creation time)). The specific formula is Priority = (hit_count × 10) + age_hours, where age_hours is the length of time the cached entry has existed (in hours). All entries are sorted from low to high priority.

[0142] Low-priority eviction and new decision insertion: Delete the 20% of cache entries with the lowest priority, free up space, and then insert the new decision into the cache.

[0143] This process prioritizes high-value cache entries based on the "number of hits + creation time" dimension, while ensuring efficient use of cache space through a fixed-ratio eviction mechanism; for example, entries with low hit counts and early creation times will be evicted first, ensuring that new decisions can be successfully stored in the cache.

[0144] Figure 15 The diagram illustrates the relationship between parameter configuration and matching accuracy of the image-aware hash cache. By combining different values ​​of the core parameters, scenario-based adaptation of "efficiency-accuracy-hit rate" is achieved.

[0145] The core configuration parameters and their corresponding effects are as follows: enable_image_hash (image hash switch): false: completely disable caching, suitable for scenarios where reuse decisions are not required; true: enable caching, which is the default configuration.

[0146] `image_hash_size` (hash size): 8: Generates a 64-bit hash, corresponding to fast / low-precision mode; 16: Generates a 256-bit hash, corresponding to recommended / balanced mode (default); 32: Generates a 1024-bit hash, corresponding to slow / high-precision mode. Larger values ​​result in more accurate hashes and higher matching precision, but with slightly higher computational overhead.

[0147] image_hamming_threshold (Hamming distance threshold): 0: Exact match only, corresponding to perfect match mode; 5: Balances accuracy and hit rate, corresponding to recommended / moderate mode (default); 10: Loose match, corresponding to loose match mode. Smaller values ​​result in stricter matching and a lower false match rate, but may decrease the hit rate; larger values ​​result in looser matching, increasing the hit rate but also increasing the risk of false matches.

[0148] image_hash_ttl (cache TTL): Controls the validity period of cache entries. The default is 86400 seconds (1 day), and the configurable range is 600-86400 seconds. The longer the duration, the higher the hit rate, but it may store expiration decisions.

[0149] image_hash_max_entries (maximum number of cache entries): controls the upper limit of cache capacity, with a default of 1000 entries to avoid excessive resource consumption. Exceeding this number will trigger LRU eviction.

[0150] image_success_rate_threshold (success rate threshold): Default is 0.6 (60%), configurable within the range of 0.0-1.0. Caches below this value will be automatically deleted. Setting it to 0.8 is stricter, ensuring high-quality caching; setting it to 0.4 is more lenient, tolerating more failures.

[0151] Context-aware configuration: enable_context_aware (default true): When enabled, it uses a three-dimensional cache key, resulting in higher matching accuracy and the ability to distinguish different operation sequences; when disabled, it uses a two-dimensional cache key, which is slightly faster in matching but less accurate. context_history_depth (default 3): Retains the most recent N tool calls for building the context hash.

[0152] Loop detection configuration: enable_loop_detection (default true): Identifies dead loop patterns through UI hash continuity detection; loop_detection_threshold (default 2): Skips cache lookup and forces LLM to re-infer when UI hash is the same N times consecutively.

[0153] Failure protection configuration: enable_failure_protection (default true): temporarily disables caching after tool execution failure, giving the system a cool-down period; cache_disabled_rounds_after_failure (default 2): the cache is not used in the next N iterations after the tool fails.

[0154] Scenario-based configuration examples: Batch mobile device automation: Configure TTL=1800 seconds, Hamming distance threshold=5, maximum number of entries=2000; Regression testing: Configure TTL=86400 seconds, Hamming distance threshold=3, maximum number of entries=1000; Low-frequency mutation scenario: Configure TTL=3600 seconds, Hamming distance threshold=7, maximum number of entries=500.

[0155] This configuration logic, through the differentiated values ​​of parameters, can adapt to different scenario requirements such as low precision and high speed, high precision and low speed, and balanced type, providing a clear reference for scenario-based optimization of caching strategies.

[0156] Further reference Figure 16 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of an operation decision device, which is similar to... Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0157] like Figure 16As shown, the operation decision device 1600 of this embodiment may include: a first calculation module 1601, a second calculation module 1602, a third calculation module 1603, a generation module 1604, and a query module 1605. The first calculation module 1601 is configured to perform perceptual hash calculation on a screenshot of the user interface to generate an image perceptual hash value; the second calculation module 1602 is configured to perform hash calculation on user intent text to generate a user intent hash value; the third calculation module 1603 is configured to perform hash calculation on the operation context to generate an operation context hash value; the generation module 1604 is configured to generate a composite cache key based on the image perceptual hash value, the user intent hash value, and the operation context hash value; the query module 1605 is configured to query the cache based on the composite cache key, and in response to a successful query, retrieve the operation decision result corresponding to the composite cache key from the cache, and the cache correspondingly stores a reference composite cache key and a reference operation decision result.

[0158] In this embodiment, the specific processing and technical effects of the first calculation module 1601, the second calculation module 1602, the third calculation module 1603, the generation module 1604, and the query module 1605 in the operation decision device 1600 can be referred to respectively. Figure 1 The relevant descriptions of steps 101-105 in the corresponding embodiments will not be repeated here.

[0159] In some optional implementations of this embodiment, the first calculation module 1601 is further configured to: reduce the user interface screenshot to a preset size to generate a reduced image; perform grayscale conversion on the reduced image to generate a grayscale image; perform discrete cosine transform on the grayscale image to generate frequency domain features; extract low-frequency coefficients from the frequency domain features; calculate the average value of the low-frequency coefficients; compare the low-frequency coefficients with the average value to generate an image perceptual hash value.

[0160] In some optional implementations of this embodiment, the second calculation module 1602 is further configured to: standardize the user intent text to generate standardized user intent text; construct a user intent string based on the standardized user intent text; perform hash calculation on the user intent string and extract a preset number of characters from the beginning of the calculation result to generate a user intent hash value.

[0161] In some optional implementations of this embodiment, the third calculation module 1603 is further configured to: connect the operation contexts sequentially using preset symbols to generate an operation context string; perform a hash calculation on the operation context string, and extract a preset number of characters before the calculation result to generate an operation context hash value.

[0162] In some optional implementations of this embodiment, the query module 1605 is further configured to: perform an exact match between the composite cache key and a reference composite cache key in the cache; in response to a successful exact match, retrieve the candidate operation decision result corresponding to the candidate composite cache key that is an exact match with the composite cache key from the cache, and use it as the operation decision result.

[0163] In some optional implementations of this embodiment, the query module 1605 is further configured to: perform a complete consistency comparison between the composite cache key and the reference composite cache key in the cache; if there is a reference composite cache key in the cache that is completely consistent with the composite cache key, then the exact match is successful; if there is no reference composite cache key in the cache that is completely consistent with the composite cache key, then the exact match fails.

[0164] In some optional implementations of this embodiment, the query module 1605 is further configured to: in response to a failure of exact matching, perform a fuzzy match between the composite cache key and a reference composite cache key in the cache; in response to a successful fuzzy match, retrieve the candidate operation decision result corresponding to the candidate composite cache key that is successfully fuzzy matched with the composite cache key from the cache, and use it as the operation decision result.

[0165] In some optional implementations of this embodiment, the query module 1605 is further configured to: calculate the Hamming distance between the image perceptual hash value and the reference image perceptual hash value in the reference composite cache key in the cache; if there is a reference composite cache key in the cache with a Hamming distance not greater than a preset distance threshold, then the fuzzy matching is successful; if there is no reference composite cache key in the cache with a Hamming distance not greater than the preset distance threshold, then the fuzzy matching fails.

[0166] In some optional implementations of this embodiment, the operation decision device 1600 further includes: an inference module, configured to input a user interface screenshot, user intent text and operation context into a large language model in response to query failure, and output the operation decision result.

[0167] In some optional implementations of this embodiment, the operation decision device 1600 further includes a storage module configured to store the composite cache key and the operation decision result in a cache in response to a query failure.

[0168] In some optional implementations of this embodiment, the storage module is further configured to: count the number of cache entries currently stored in the cache; in response to determining that the number of cache entries is not greater than a preset entry number threshold, store the composite cache key and the operation decision result in the cache accordingly; in response to determining that the number of cache entries is greater than the preset entry number threshold, delete a preset proportion of cache entries from the cache in order of priority from low to high, and store the composite cache key and the operation decision result in the cache accordingly.

[0169] In some optional implementations of this embodiment, the operation decision device 1600 further includes: an execution module configured to execute the operation decision result and generate an operation execution result; a first update module configured to update the execution success rate of the operation decision result based on the operation execution result; and a deletion module configured to delete the composite cache key and the operation decision result from the cache in response to determining that the success rate is lower than a preset success rate threshold.

[0170] In some optional implementations of this embodiment, the operation decision device 1600 further includes: a second update module, configured to update the remaining disabled round counter based on the operation execution result; in response to determining that the remaining disabled round counter is 0, obtain the operation decision result from the cache; in response to determining that the remaining disabled round counter has reached a first preset number threshold, input the user interface screenshot, user intent text and operation context into the large language model, output the operation decision result, and decrement the remaining disabled round counter by 1.

[0171] In some optional implementations of this embodiment, the second update module is further configured to: keep the remaining disabled round counter unchanged in response to the operation decision result coming from the large language model or the execution being successful; and set the remaining disabled round counter to a preset value in response to the operation decision result coming from the cache and the execution failing.

[0172] In some optional implementations of this embodiment, the operation decision device 1600 further includes: a third update module, configured to compare the image perception hash value with the previous image perception hash value and update the consecutive identical counter; in response to determining that the consecutive identical counter has reached a second preset number threshold, inputting the user interface screenshot, user intent text and operation context into the large language model, and outputting the operation decision result.

[0173] In some optional implementations of this embodiment, the third update module is further configured to: increment the consecutive identical counter by 1 in response to determining that the hash values ​​are the same; and set the consecutive identical counter to 0 in response to determining that the hash values ​​are different.

[0174] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0175] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0176] Figure 17A schematic block diagram of an example electronic device 1700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0177] like Figure 17 As shown, device 1700 includes a computing unit 1701, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1702 or a computer program loaded from storage unit 1708 into random access memory (RAM) 1703. The RAM 1703 may also store various programs and data required for the operation of device 1700. The computing unit 1701, ROM 1702, and RAM 1703 are interconnected via bus 1704. Input / output (I / O) interface 1705 is also connected to bus 1704.

[0178] Multiple components in device 1700 are connected to I / O interface 1705, including: input unit 1706, such as a keyboard, mouse, etc.; output unit 1707, such as various types of displays, speakers, etc.; storage unit 1708, such as a disk, optical disk, etc.; and communication unit 1709, such as a network card, modem, wireless transceiver, etc. Communication unit 1709 allows device 1700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0179] The computing unit 1701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1701 performs the various methods and processes described above, such as the operation decision method. For example, in some embodiments, the operation decision method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1708. In some embodiments, part or all of the computer program may be loaded and / or installed on device 1700 via ROM 1702 and / or communication unit 1709. When the computer program is loaded into RAM 1703 and executed by the computing unit 1701, one or more steps of the operation decision method described above may be performed. Alternatively, in other embodiments, the computing unit 1701 may be configured to perform the operation decision method by any other suitable means (e.g., by means of firmware).

[0180] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0181] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0182] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0183] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0184] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0185] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, distributed system servers, or servers incorporating blockchain technology.

[0186] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution provided in this disclosure can be achieved, and this is not limited herein.

[0187] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An operational decision-making method, comprising: Perform perceptual hash calculation on user interface screenshots to generate image perceptual hash values; Perform hash calculation on the user intent text to generate a user intent hash value; Perform a hash calculation on the operation context to generate an operation context hash value; A composite cache key is generated based on the image perception hash value, the user intent hash value, and the operation context hash value. Based on the composite cache key, the cache is queried. In response to a successful query, the operation decision result corresponding to the composite cache key is obtained from the cache. The cache also stores a reference composite cache key and a reference operation decision result.

2. The method according to claim 1, wherein, The step of performing perceptual hash calculation on the user interface screenshot to generate an image perceptual hash value includes: The user interface screenshot is reduced to a preset size to generate a scaled-down image; The reduced image is converted to grayscale to generate a grayscale image; Perform a discrete cosine transform on the grayscale image to generate frequency domain features; Extract low-frequency coefficients from the frequency domain features; Calculate the average value of the low-frequency coefficients; The low-frequency coefficients are compared with the average value to generate the image perception hash value.

3. The method according to claim 1, wherein, The step of hashing the user intent text to generate a user intent hash value includes: The user intent text is standardized to generate standardized user intent text; Based on the standardized user intent text, a user intent string is constructed; The user intent string is hashed, and a preset number of characters are extracted from the beginning of the hash result to generate the user intent hash value.

4. The method according to claim 1, wherein, The step of performing a hash calculation on the operation context to generate an operation context hash value includes: The operation context is sequentially connected using preset symbols to generate an operation context string; The operation context string is hashed, and a preset number of characters are extracted from the beginning of the calculation result to generate the operation context hash value.

5. The method according to claim 1, wherein, The step of querying the cache based on the composite cache key, in response to a successful query, retrieving the operation decision result corresponding to the composite cache key from the cache, includes: The composite cache key is precisely matched with the reference composite cache key in the cache; In response to a successful exact match, the candidate operation decision result corresponding to the candidate composite cache key that is a successful exact match with the composite cache key is obtained from the cache and used as the operation decision result.

6. The method according to claim 5, wherein, The step of precisely matching the composite cache key with the reference composite cache key in the cache includes: Perform a complete consistency comparison between the composite cache key and the reference composite cache key in the cache; If a reference composite cache key exists in the cache that is exactly the same as the composite cache key, then an exact match is successful; If there is no reference composite cache key in the cache that is exactly the same as the composite cache key, then an exact match fails.

7. The method according to claim 5, wherein, The step of querying the cache based on the composite cache key, in response to a successful query, retrieving the operation decision result corresponding to the composite cache key from the cache, includes: In response to a failure of an exact match, the composite cache key is fuzzily matched with a reference composite cache key in the cache. In response to a successful fuzzy match, the candidate operation decision result corresponding to the candidate composite cache key that successfully matches the composite cache key is obtained from the cache and used as the operation decision result.

8. The method according to claim 7, wherein, The step of performing a fuzzy match between the composite cache key and a reference composite cache key in the cache includes: Calculate the Hamming distance between the image perceptual hash value and the reference image perceptual hash value in the reference composite cache key in the cache; If there is a reference composite cache key in the cache with a Hamming distance not greater than a preset distance threshold, then the fuzzy match is successful; If there is no reference composite cache key in the cache whose Hamming distance is not greater than a preset distance threshold, then the fuzzy matching fails.

9. The method according to claim 1, wherein, The method further includes: In response to a query failure, the user interface screenshot, the user intent text, and the operation context are input into the large language model, and the operation decision result is output.

10. The method according to claim 9, wherein, The method further includes: In response to a query failure, the composite cache key and the operation decision result are stored in the cache.

11. The method according to claim 10, wherein, The step of storing the composite cache key and the operation decision result in the cache includes: Count the number of cache entries currently stored in the cache; In response to determining that the number of cache entries is not greater than a preset entry number threshold, the composite cache key and the operation decision result are stored in the cache accordingly; In response to determining that the number of cache entries is greater than a preset entry number threshold, a preset proportion of cache entries are deleted from the cache in order of priority from low to high, and the composite cache key and the operation decision result are stored in the cache accordingly.

12. The method according to claim 10, wherein, The method further includes: Execute the operation decision result and generate the operation execution result; Based on the operation execution result, update the execution success rate of the operation decision result; In response to determining that the success rate is lower than a preset success rate threshold, the composite cache key and the operation decision result are deleted from the cache.

13. The method according to claim 12, wherein, The method further includes: Based on the results of the operation, update the counter for the remaining disabled rounds; In response to determining that the remaining disabled round counter is 0, the operation decision result is obtained from the cache; In response to determining that the remaining disabled round counter has reached a first preset number of times threshold, the user interface screenshot, the user intent text, and the operation context are input into the large language model, the operation decision result is output, and the remaining disabled round counter is decremented by 1.

14. The method according to claim 13, wherein, The step of updating the remaining disabled round counter based on the operation execution result includes: In response to the operation decision result being derived from the large language model, or if the operation is successful, the remaining disabled round counter remains unchanged; In response to the operation decision result coming from the cache and the execution failing, the remaining disabled round counter is set to a preset value.

15. The method according to claim 1, wherein, The method further includes: Compare the image perception hash value with the previous image perception hash value, and update the consecutive identical counter; In response to determining that the consecutive identical counters have reached a second preset number threshold, the user interface screenshot, the user intent text, and the operation context are input into the large language model, and the operation decision result is output.

16. The method according to claim 15, wherein, The step of comparing the image-aware hash value with the previous image-aware hash value and updating the consecutively identical counter includes: In response to determining that the hash values ​​are the same, the consecutive identical counter is incremented by 1; In response to determining that the hash values ​​are different, the consecutive identical counters are set to 0.

17. An operational decision-making device, comprising: The first calculation module is configured to perform perceptual hash calculation on the user interface screenshot and generate an image perceptual hash value. The second calculation module is configured to perform hash calculation on the user intent text to generate a user intent hash value; The third calculation module is configured to perform hash calculations on the operation context and generate an operation context hash value. The generation module is configured to generate a composite cache key based on the image perception hash value, the user intent hash value, and the operation context hash value; The query module is configured to query the cache based on the composite cache key. In response to a successful query, it retrieves the operation decision result corresponding to the composite cache key from the cache. The cache stores a reference composite cache key and a reference operation decision result.

18. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-16.

19. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1-16.

20. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-16.

Images