LLM-based dynamic automatic detection method, equipment, and media for iOS vest packages

Abstract

The present invention discloses a dynamic automatic detection method, device and medium for iOS vest packages based on LLM, comprising: obtaining application store metadata of a target iOS application, generating a trigger keyword dictionary through a large language model in combination with an example of a vest package triggering method, and building an automated task list based on the dictionary; selecting a task as a current automated task and testing the iOS application; in response to the current page status of the iOS application, the model outputs a page summary and a list of interactive elements; in response to the page summary, the list of interactive elements, the current automated task and the list of historical operations, the model outputs a target interactive element and a target operation; performing a target operation on the iOS application according to the target interactive element to obtain an updated page status of the iOS application; inputting the updated page status of the iOS application and the application store metadata into the model, judging whether the page status of the iOS application is consistent with the application store metadata, and if not, judging that the iOS application is a vest package.

Description

Technical Field

[0001] The present invention belongs to the technical field of iOS application detection, and in particular to an LLM-based dynamic automatic detection method, device, and medium for iOS vest packages. Background Art

[0002] iOS is an operating system widely used on Apple mobile devices, such as the iPhone and iPad. With the rapid development of the internet and the widespread adoption of iOS applications, more and more criminals are using iOS apps to disseminate illegal content and profit from it.

[0003] As the primary distribution channel for iOS apps and the primary platform for iOS users to access apps, the App Store has inevitably become a prime target for criminals distributing illegal apps. To circumvent App Store review, criminals have increasingly adopted a "chameleon app" strategy. These apps disguise themselves as legitimate apps during the review process, revealing hidden illegal functionality (such as gambling or pirated apps) only after completing specific triggering actions.

[0004] Existing malware detection solutions include static analysis to uncover hidden UI semantic differences, and analysis based on user reviews and metadata. However, static analysis is limited in effectiveness against malware with mixed development models or dynamically loaded content. Detection methods based on user reviews and metadata rely on the quality and quantity of reviews and the timeliness of analysis, resulting in time lags and difficulty in quickly responding to emerging threats. They also have a relatively high probability of missed and false positives.

[0005] Therefore, it is urgent to propose a detection method for illegal applications on the iOS platform that can effectively identify and analyze various types of vest packages and improve the accuracy and coverage of illegal behavior identification. Summary of the Invention

[0006] In view of the shortcomings of the existing technology, the present invention provides a dynamic and automatic detection method, device and medium for iOS vest packages based on LLM.

[0007] In a first aspect, an embodiment of the present invention provides a method for dynamic and automated detection of iOS vest packs based on LLM, the method comprising the following steps:

[0008] Get the App Store metadata of the iOS app to be tested;

[0009] In response to the App Store metadata of the iOS app to be detected and the vest package triggering method example, the large language model outputs a trigger keyword dictionary list; based on the trigger keyword dictionary list, the vest package triggering method is constructed and sorted to obtain an automated task list;

[0010] Select a task from the automation task list as the current automation task, and initialize an empty set as the history operation list;

[0011] Dynamically test the iOS app to be tested according to the current automated task, obtain and process the current page state of the iOS app; in response to the processed current page state, the large language model outputs a page summary and a list of interactive elements;

[0012] In response to the page summary, the list of interactive elements, the current automated task, and the list of historical operations, the large language model outputs a target interactive element and a target operation;

[0013] Execute the target operation on the iOS application according to the target interactive element, obtain the updated page status of the iOS application, and update the historical operation list;

[0014] The updated page status and app store metadata of the iOS app are input into the large language model to determine whether the page status of the iOS app is consistent with the app store metadata. If they are inconsistent, the iOS app to be detected is determined to be a vest package.

[0015] In a second aspect, an embodiment of the present invention provides an electronic device comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned LLM-based dynamic automatic detection method for iOS vest packages.

[0016] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned LLM-based dynamic automatic detection method for iOS vest packages.

[0017] In a fourth aspect, an embodiment of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the above-mentioned LLM-based dynamic automatic detection method for iOS vest packages.

[0018] Compared with the prior art, the present invention has the following beneficial effects:

[0019] The present invention provides an LLM-based dynamic automated detection method for iOS vest packages. By dynamically testing the iOS application to be detected, the application store metadata and vest package triggering method examples of the iOS application to be detected are merged as known iOS vest package knowledge, and the LLM is input for semantic analysis to obtain potential vest package triggering methods, and an automated task list is obtained. To a certain extent, the problem of lag in the application store metadata of the iOS application is solved, and the robustness of vest package detection is improved. The present invention effectively solves the problem that static solutions cannot cope with hybrid development modes or dynamically loaded content, and can more accurately identify the application as a vest package. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 A flowchart of a dynamic and automated detection method for iOS vest packages based on LLM provided by an embodiment of the present invention;

[0022] Figure 2 A schematic diagram of a trigger method list provided in an embodiment of the present invention;

[0023] Figure 3 A schematic diagram of pruning a view tree and adding semantic information provided by an embodiment of the present invention;

[0024] Figure 4 A schematic diagram of extracting a page summary and a list of interactive elements provided by an embodiment of the present invention;

[0025] Figure 5 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0027] It should be noted that, unless there is any conflict, the features in the following embodiments and implementations may be combined with each other.

[0028] In order to clearly explain the technical solutions provided by the embodiments of the present application, some concepts that may appear in subsequent embodiments will be introduced first.

[0029] Large Language Models (LLMs) are natural language processing models trained using deep learning techniques, particularly large-scale text data. They can understand, generate, translate, and reason about complex relationships in language. They are widely used in fields such as natural language understanding, automated generation, and conversational systems. In the field of mobile task automation, they are also often used to understand the current page and make decisions.

[0030] Prompts are text or instructions given to a large language model to guide it in generating specific outputs. Prompts are typically concise and can be questions, commands, or context, helping the model focus on specific tasks or content. Through carefully designed prompts, users can implement functional invocations of the LLM to complete various natural language processing tasks.

[0031] Structured output: This refers to the clearly structured output generated by the LLM based on the input prompts. It can be designed as nested fields, lists, dictionaries, JSON, and other data formats. By designing a specific output format, we can guide the understanding focus of the large language model and ensure that the output is highly readable and actionable.

[0032] The view hierarchy is an abstract hierarchical representation of UI (user interface) elements within an application. It reflects the overall page layout, the properties of each element, and the parent-child relationships. It is typically displayed as a tree structure, with a root node representing the entire interface and child nodes representing specific UI elements (such as buttons, text boxes, and images). Each node contains the element's various properties (such as type, name, value, location, and visibility). The view tree helps developers understand and control the hierarchy of interface layouts and plays a crucial role in UI automated testing.

[0033] like Figure 1 As shown, an embodiment of the present invention provides a dynamic and automatic detection method for iOS vest packages based on LLM, the method comprising the following steps:

[0034] Step S1: Obtain the App Store metadata of the iOS application to be detected.

[0035] Specifically, step S1 includes the following sub-steps:

[0036] Step S101 : collecting the application name, developer name, application category, application introduction, and user comments of the iOS application to be detected displayed in the App Store as the App Store metadata of the iOS application.

[0037] For example, the App Store metadata of the iOS application is obtained by using a Selenium script to access the link of the iOS application to be tested on the App Store, and obtaining the App Store metadata such as the application name, developer name, application category, application description and user comments from the application display page.

[0038] Step S102 : For the iOS application to be tested, the IPA file of the application is downloaded from the App Store, thereby installing the iOS application to be tested on the test device.

[0039] For example, download the app's IPA file from the App Store. First, use the ipatool-py tool to download the latest version of the app to be tested from the App Store. Then, use the tidevice tool to install it in batches onto test phones. The IPA file (iPhone Application Archive, the installation package file for iOS apps) contains everything needed to install and run the app on iOS devices (such as iPhones and iPads).

[0040] Step S2: In response to the App Store metadata of the iOS application to be detected and the vest package triggering method example, the large language model outputs a trigger keyword dictionary list; based on the trigger keyword dictionary list, the vest package triggering method is constructed and sorted to obtain an automated task list.

[0041] Specifically, step S2 includes the following sub-steps:

[0042] Step S201 : obtaining user review text according to the application store metadata of the iOS application to be detected, and preprocessing the user review text to obtain preprocessed user review text.

[0043] Specifically, in this example, the upper limit of the number of user comment texts obtained is set to 200, all comments retain the title and content, and are stored in the format of "title: content", the upper limit of the number of words for each comment text is 40, and all comment texts are spliced ​​with separators; as user comment text, the upper limit of the number of words in the user comment text is 800, and the part exceeding the upper limit will be deleted.

[0044] Step S202: Based on a known example of a vest package triggering method, a first prompt word and a corresponding first structured output are designed; the first prompt word includes an example of a vest package triggering method, a user comment text, and a demand instruction; the first structured output is a trigger keyword dictionary list, and each trigger keyword dictionary in the trigger keyword dictionary list includes a confidence score and a keyword area; the keyword area includes: trigger time, trigger geographical conditions, trigger waiting time, interaction location, number of interactions, text input location, and text input content; the confidence score represents the LLM's evaluation of the likelihood of the trigger keyword dictionary.

[0045] For example, in this example, the first prompt word sample designed is as follows (the part to be filled is indicated by []):

[0046] Instructions: Disguised apps are apps whose actual functionality differs from that displayed on the App Store. They require specific triggering conditions to "transform" the app into what appears to be a disguised app. Basic information and reviews should be used to determine whether the app is a disguised app and how to trigger it.

[0047] 1. Trigger method

[0048] First, we provide a series of triggering methods for disguised applications:

[0049] [1] Open / Open multiple times / Wait: such as open directly, automatically change after restart, and wait 10 seconds after opening.

[0050] [2] Specific time and space conditions: such as being located in China and being enabled during a specific time period.

[0051] [3] Specific behavior: specific click operations, such as clicking a blank space 5 times, clicking the homepage icon 5 times in succession, or watching an ad.

[0052] [4] Specific input content or options: Enter specific content in the text box or select specific content in the selection box (such as problem feedback, my suggestions, parameter bar, search box) or copy specific content to the clipboard.

[0053] These methods can be used in combination: for example, input 999999 on the calculator and click calculate three times (a combination of method [3] and method [4]); click the loading bar on the screen 10 times and then re-enter (a combination of method [1] and method [3]).

[0054] 2. Demand

[0055] Based on the review information, please identify keywords that may be related to the four trigger methods, including: waiting time, geographical / time conditions, click location, number of clicks, specific input location, and specific input text.

[0056] 3. Additional Notes

[0057] Pay special attention to phrases in reviews that have the same title and content (typically 3-5 Chinese phrases that are not interrogative). These may be specific input. Only phrases unrelated to the app, appearing multiple times in the review, or clearly associated with words like "trigger" or "transform" are considered specific input.

[0058] Examples of specific input content: 666, ys777, big-eared Tutu, big monkey, hello 123, be confident to be beautiful, I love China, 999999, 5201080, small tomato, and montage.

[0059] Examples of specific input locations: feedback, text input box, search box, clipboard.

[0060] 4. Input

[0061]

User comment text

[0062] Step S203: Fill the pre-processed user comment text into the prompt word, input it into the large language model for semantic analysis, and obtain the trigger keyword dictionary list predicted by the LLM.

[0063] Step S204: construct and piece together a trigger method short sentence according to the keyword area corresponding to each trigger keyword dictionary in the trigger keyword dictionary list, thereby generating a vest package trigger method corresponding to each trigger keyword dictionary.

[0064] For example, in this example, all keyword regions and corresponding trigger method phrases are as follows:

[0065] Trigger time: open the app at {} time;

[0066] Trigger geographic conditions: open the app at {} geographic location;

[0067] Trigger wait time: wait {} seconds;

[0068] Interaction position and number of interactions: Click {} consecutively at {} position;

[0069] Text input position and text input content: Enter {} content in {} position.

[0070] All areas are initially set to None. For the two pairs of keyword areas (interaction position and interaction count, and text input position and text input content), if any keyword area in each pair has a predicted element (not None), a corresponding trigger method phrase will be constructed, and the other unpredicted keyword area will be filled with default words.

[0071] Furthermore, the default words in the keyword area are as follows: interaction position: anywhere on the page; number of interactions: 1; text input position: text input box; text input content: potential trigger words.

[0072] Furthermore, after splicing all the trigger method sentences, add the prompt words at the end: "Restart the application, complete the trigger, and explore from different angles and page semantics (pay special attention to function icons, about, mine, feedback, search, settings, properties, personal and other functions)".

[0073] For example, Figure 2 As shown, there are three trigger keyword dictionary items in the trigger keyword dictionary list of an iOS application predicted by LLM. In the first trigger keyword dictionary, only the keyword area "trigger waiting time" is successfully filled, so only the trigger method short sentence "wait 10 seconds" is generated. After splicing the prompt words, "wait 10 seconds; restart the application to complete the trigger, and explore from different angles and page semantics (pay special attention to function icons, about, mine, feedback, search, settings, properties, personal and other functions)" is obtained as the first vest package trigger method; the second trigger keyword dictionary generates "enter hello 123 content in the problem feedback position", and the second vest package trigger method is obtained in the same way; the third trigger keyword dictionary first generates " For the short sentence "Open the APP at 12 o'clock in the morning", since the interaction position is predicted to be "home icon" and the number of interactions is None, the short sentence "Click {} continuously at the home icon position" is generated and filled with the default word 1 for the number of interactions, forming "Click 1 continuously at the home icon position", which is spliced ​​into "Open the APP at 12 o'clock in the morning, click 1 continuously at the home icon position", and then the prompt words are spliced ​​to get "Open the APP at 12 o'clock in the morning, click 1 continuously at the home icon position; restart the application to complete the triggering, and explore from different angles and page semantics (pay special attention to function icons, about, mine, feedback, search, settings, properties, personal and other functions)", and finally the third vest package triggering method is obtained.

[0074] Step S205, determine whether the vest package triggering method depends on the context of the iOS application, and sort the vest package triggering method that does not depend on the context of the iOS application before the vest package triggering method that depends on the context of the iOS application. The vest package triggering methods that do not depend on the iOS application or depend on the context of the iOS application are sorted from high to low according to the confidence score, thereby obtaining an automated task list.

[0075] Specifically, in this example, the vest package triggering method that does not rely on the iOS application context represents a vest package triggering method composed of a trigger keyword dictionary in which the three keyword areas of the interaction position, text input position and text input content are all None.

[0076] Step S3: Select a task from the automation task list as the current automation task, and initialize an empty set as the history operation list.

[0077] Step S4: Dynamically test the iOS application to be tested according to the current automation task, obtain and process the current page status of the iOS application; in response to the processed current page status, the large language model outputs a page summary and a list of interactive elements.

[0078] Specifically, step S4 includes the following sub-steps:

[0079] Step S401: Run the iOS application to be tested in a dynamic test environment built based on Appium; if the current automation task involves time and geographic location or clipboard content, the current device environment needs to be adjusted.

[0080] Exemplarily, if the current automation task involves time and geographic location or clipboard content, the process of adjusting the current device environment includes: using a Python script and Appium tools to connect to the device's communication driver; if the current automation task involves time and geographic location, using a predefined code sequence (modification time is through sshroot@localhost -p 2222 followed by the command date MMDDhhmmYYYY, and geographic location uses the driver.set_location function) to adjust the current device environment; if the current automation task involves clipboard content, writing the target content to the clipboard through the driver.set_clipboard_text method.

[0081] Step S402: Acquire and process the current page state of the iOS application, where the current page state includes a page screenshot and a view tree of the current page.

[0082] Exemplarily, the page status of the current page is obtained, and the PNG file of the page screenshot and the XML file of the view tree are obtained through the driver.save_screenshot and driver.page_source methods.

[0083] Step S403: prune the view tree of the current page to delete invalid information in the view tree of the current page; perform optical character recognition on the page screenshot and add the recognized semantic information to the view tree; perform icon classification on the leaf elements in the pruned view tree that are close to square in size and add the icon categories to the view tree to obtain an enhanced view tree.

[0084] Specifically, if Figure 3As shown, in this example, the invalid information in the view tree of the current page is deleted: the view tree is pruned, irrelevant elements and non-essential attribute information are deleted, and parent-child element pairs with only one child element and elements with the same position information (including coordinates and sizes) are recursively merged. Then, the nodes are rearranged from left to right and from top to bottom according to the position information to ensure that each visual area corresponds to an element in the view tree.

[0085] Perform optical character recognition on the page screenshot and add the recognized semantic information to the view tree: Use the PaddleOCR model on the page screenshot to extract all text and corresponding positions, traverse all elements in the view tree, and if the text is not included in any attributes of the minimum area matching element containing the position, then add the text as an additional attribute orc_text to the element;

[0086] For all leaf elements with a size close to a square in the position information of the pruned view tree (in this example, the leaf element is an element without other child elements, and the element close to a square is an element with an aspect ratio between 0.95 and 1.05), it is assumed that the element is an icon. The EfficientNet-B0 model trained with RICO Semantics and some icons from the vest package as the data set is used to perform the icon classification task, and the functional category of the icon is obtained (such as ICON_X for the × icon, ICON_SETTINGS for the settings icon, and ICON_MAGNIFYING_GLASS for the magnifying glass icon). The category is added to the element as an additional attribute icon_label.

[0087] It should be noted that in this example, by pruning the view tree and adding semantic information, the problem of LLM being unable to accurately infer the clickable elements and corresponding functions in the page status is effectively solved.

[0088] Step S404, design a second prompt word and its corresponding second structured output; the second prompt word is a text paragraph containing the goal, requirements, and view tree, and a picture part containing a page screenshot. The second structured output is a list of interactive elements, each interactive element contains the corresponding ID, location information, semantic information and other attributes of the target element in the view tree.

[0089] For example, in this example, the second prompt word sample is designed as follows (the part to be filled is indicated by []):

[0090] Perform a comprehensive analysis of the screenshot and its runtime UI elements:

[0091] 1. Tasks and Key Points

[0092] Accurately match the ID and bounding box of each element to the screenshot. Discard the element if the bounding box doesn't match exactly. If the UI differs significantly from the screenshot, it's recommended to wait for synchronization. Prioritize key elements such as close, skip, navigation, action buttons, and function icons (including search, submit, settings, tools, favorites, likes, location, back, etc.). Subviews under the same parent view are sorted from top to bottom and from left to right. If there is a pop-up ad, its close button must be located (usually in the four corners, marked as 'ICON_X').

[0093] 2. Requirements and boundaries:

[0094] Ignore the native iOS status bar / navigation bar and keyboard. Must include text boxes, buttons, function icons, and valid static text and images (especially those nested in clickable cells). Focus on the ocr_text attribute element obtained through independent OCR processing and the icon_label attribute element obtained through independent CNN image classification. Actively expand included elements to cover any adjacent elements.

[0095] 3. Restrictions and Conditions:

[0096] Use a general description to present the UI hierarchy (top-down, left-to-right layout), including components such as the title bar, sidebar, top bar, tabs, footer, and menu (if any). Only fully visible alert boxes that occupy the entire screen are identified as emergency top-level pop-ups. Pop-ups containing promotional copy (such as "Download" and "Details") must be marked as ads. Ads located in corners that do not obscure other UI elements are not considered emergency top-level pop-ups. Discard all promotional copy and buttons within the ad pop-up, retaining only the close button. All visible feedback / status / error prompts (including reminders / toast message prompt boxes / pop-up text / bottom status bar) must be included, and indicate whether the UI is in a loading state.

[0097] 4. Output format:

[0098] Strictly following the output structure of two core modules, "hint_elements" means storing non-interactive / semi-interactive UI hints (such as status information, guidance text); "action_elements" stores clear interactive elements (with complete bounding boxes and function identifiers).

[0099] 5. Input

[0100]

Page Screenshot

Enhanced View Tree

[0101] Step S405 : In response to the page screenshot and the enhanced view tree of the current page, the large language model outputs a page summary and a list of interactive elements corresponding to the current page.

[0102] For example, after inputting a screenshot of an iOS application page and the second prompt word consisting of the enhanced view tree into the LLM, the page summary and interactive element list of the current page are obtained. The page summary describes the current page with a paragraph of text, and the interactive element list contains the id, category, description, text, location and other attributes of each presumed interactive element. Specifically, Figure 4 As shown, the page screenshot displays the slogan "Premium Handwritten Signature - Create your own personalized signature; different brushes give you different feelings; different colors make your signature more colorful," controls for "Start Handwriting Signature," "Share with Friends," and "Have questions? Give me feedback." The page summary output by the large language model is: On the page showcasing application features, you can choose to start signing, share, or provide feedback. Table 1 shows the list of interactive elements output by the large language model:

[0103] Table 1: Example of a list of interactive elements

[0104]

[0105] Step S5: In response to the page summary, the list of interactive elements, the current automated task, and the list of historical operations, the large language model outputs a target interactive element and a target operation.

[0106] Specifically, step S5 includes the following sub-steps:

[0107] Step S501, design a third prompt word and its corresponding third structured output; the third prompt word is a text paragraph containing goals, requirements, page summary, list of interactive elements, current task and historical operations; the third structured output is a specific operation, including a target element and an execution action, where all possible execution actions include: click (continuous click), text input, restart, task completion, and none.

[0108] Specifically, in this example, the third prompt word sample is designed as follows (the part to be filled is indicated by []):

[0109] Comprehensively analyze the current UI status and plan subsequent actions:

[0110] 1. Mission and key points:

[0111] Understand the user's ultimate task goal based on UI data and operation history. If the subtask cannot be completed on the current page, actively explore the app by clicking labels / buttons and following the on-page instructions. Learn the application's interaction patterns through historical operation records to avoid repeated and redundant operations. If the operation does not achieve the expected feedback, allow the entire process or partial steps to be retried (up to 3 times). Do not repeat the same operation sequence on the visited page unless it can trigger a new state.

[0112] 2. Basic principles:

[0113] Use the UI element list as your primary benchmark. Strictly match element IDs with action targets. Prioritize actions that advance the user's task or actions that have never been performed before; avoid irrelevant or repetitive steps. If a direct action is not possible, first try alternative navigation or follow the page's semantics before ultimately recommending "no action." If repeated actions fail to yield progress, pause the main task and instead explore native functionality to understand the page's semantics before deciding on subsequent actions.

[0114] 3. Restrictions:

[0115] Prioritize emergency top-level pop-ups and ads (close or skip with minimal effort). Emergency, unexpected pop-ups are prioritized over ads. Automatically grant the app the required system permissions. Abort dialog box operations that are irrelevant to the current task, such as selecting local files / photos. If the interface is in a loading state or no action is possible after careful evaluation, it is recommended to "wait for the next attempt." It is recommended to restart the app if the ad cannot be closed. Clicking on ad details is strictly prohibited. If no action is possible after multiple waits, or the current interface cannot be redirected, it is recommended to restart the app with caution. The task is considered completed only when it is clear that historical feedback meets the conditions. Directly input text into the text box (automatically clearing the original content) without clicking the text box first.

[0116] 4. Output specifications and logic:

[0117] Strictly follow the JSON output format description. The summary content must be concise and accurate.

[0118] 5. Input

[0119] [Page Summary] + [List of Interactive Elements] + [Current Automation Task] + [Historical Operation List]

[0120] In step S502, the page summary, the list of interactive elements, the current automated task, and the list of historical operations are filled into the third prompt word and input into the large language model for action decision. The large language model outputs the target interactive element and the target operation.

[0121] Specifically, in this example, when the target operation output by the large language model is click or continuous click, the number of clicks is also set; when the target operation output by the large language model is text input, the character string to be input is also set;

[0122] When the large language model determines that the current automation task has been completed, the target operation is set to task completion and the target interactive element is set to None;

[0123] When the large language model determines that no operation is applicable to the current automation task, the target operation is set to None and the target interactive element is set to None.

[0124] Step S6: Execute the target operation on the iOS application according to the target interactive element, obtain the updated page status of the iOS application, and update the historical operation list.

[0125] Specifically, step S6 includes the following sub-steps:

[0126] Step S601, re-obtain the view tree of the current page, determine whether the view trees corresponding to the original page and the current page obtained in step S402 have changed, if the Jaccard similarity between the view trees corresponding to the original page and the current page is lower than the first threshold (set to 0.8 in this example) and the tree edit distance is higher than the second threshold (set to 3 in this example) and the number of consecutive rollbacks does not exceed the pre-set third threshold (set to 3 in this example), then roll back to step S402 to re-obtain the current page state of the iOS application, and re-obtain the target interactive elements and target operations.

[0127] It should be noted that during the execution of the target operation, the page of the iOS application may change (such as pop-up ads), causing the target operation predicted in step S5 to be invalid. Therefore, step S601 is required to re-acquire the page status and compare whether there are obvious changes, so as to determine whether the target operation predicted in step S5 is still valid.

[0128] Step S602: Execute the target operation on the iOS application according to the target interactive element. If the target operation is click, continuous click, text input, or restart, execute the corresponding target operation on the target interactive element. If the target operation is task completion, end the current automation task. If the target operation is none, do not execute any operation.

[0129] For example, when the execution action is click (continuous click), Python script and ZXTouch tool are used to implement the click operation of specific coordinates through the zxtouch.touch method based on the position information of the element corresponding to the target element ID; when the execution action is text input, Python script and Appium tool are used to first use the driver.find_element method to find the text input box element object element through the attributes of the element corresponding to the target element ID, and then use the element.send_keys method to implement the input of the target text; when the execution action is restart, the driver.terminate_app and driver.activate_app methods are used to close and open the application.

[0130] Step S603: After executing the target operation on the iOS application according to the target interactive element, a triplet is added to the history operation list: <page summary, target interactive element, execute target operation>, thereby updating the history operation list.

[0131] It should be noted that in this example, by optimizing the recording method of the historical operation list, the problem of looping or avoiding correct operations in task automation is effectively improved.

[0132] For example, an iOS app has already gone through four cycles in the current automation task (entering "666" in the "Feedback Input" location, restarting the app, completing the trigger, and exploring from different angles and page semantics (paying special attention to the function icons, about, my, feedback, search, settings, properties, personal functions, etc.). In the first cycle, the "About App" button was clicked in the main menu. In the second cycle, 666 was entered in the feedback box on the About App page. In the third cycle, the Submit button was clicked on the About App page to provide feedback. In the fourth cycle, due to detecting that the application automatically exited (the application automatically crashed upon successful triggering), the restart execution action was selected. The historical operation list records:

[0133] 1. Start the application from SpringBoard

[0134] 2. <Main menu (contains mode selection buttons and "About App" link), "About App" button, click>

[0135] 3. On the "About App" page (including gameplay instructions and feedback area, including input boxes and submit buttons), enter "666" in the "Feedback" input box.

[0136] 4. Click the "Submit" button on the "About App" page (including gameplay instructions and feedback area, including input box and submit button).

[0137] 5. The app may have crashed and become unresponsive. Please restart the app.

[0138] Step S7: Input the updated page status of the iOS application and the App Store metadata into the large language model to determine whether the page status of the iOS application is consistent with the App Store metadata. If not, the iOS application to be detected is determined to be a vest package. Specifically, step S7 includes the following sub-steps:

[0139] Step S701, calculating the hash value of the view tree in the updated page state of the iOS application, and determining whether the updated page state is a newly appeared page state;

[0140] If the updated page status is not a new page status, the updated page status is a page that has been compared, and the LLM judgment result is assumed to be consistent, and step S702 is directly skipped.

[0141] Step S702: Design a fourth prompt word, fill the updated page status of the iOS application and the app store metadata into the fourth prompt word and input it into the large language model. The large language model determines whether the page status of the iOS application is consistent with the app store metadata.

[0142] Furthermore, the fourth prompt word is a text paragraph containing goals, requirements, application store description, view tree, and a picture part containing a page screenshot.

[0143] Furthermore, the app store description includes: app name, app category, and app introduction.

[0144] Specifically, in this example, the fourth prompt word sample is designed as follows (the part to be filled is indicated by []):

[0145] I need to determine whether an app provides pages and functionality that are completely unrelated to its description in the App Store. Please evaluate whether the runtime screenshots and view tree text are completely unrelated to the description provided in the basic information and output "Yes" or "No":

[0146] Output "Yes": The screenshots and text clearly do not match the app store description, displaying features and pages completely unrelated to the app store description. The screenshots and text display pop-up windows or prompts containing instructions for activating hidden mode or other actions (for example, "Turn on hidden mode," "Steps to unlock all features," "Please follow the instructions to unlock," "Enter 666 to enjoy full functionality").

[0147] Output "No": The screenshots and text fully conform to the app store description. The original page is obscured by marketing content (e.g., advertisements, promotions, and app stores, often containing keywords such as "ad," "download now," "shake," "go to app," "view details," and "third-party apps"), resulting in unclear functionality assessment. The displayed page is generic or neutral (e.g., "Feedback," "Personal Center," "User Agreement," and "Menu"), with no direct evidence of hidden functionality. The page is a system-level pop-up or prompt (e.g., app store, system settings, file dialog box) unrelated to the current app. The page is still loading, temporarily presenting a white or blank screen.

[0148] Input: [App Store Description] + [Page Screenshot] + [Enhanced View Tree].

[0149] Furthermore, if the updated page status is a newly appeared page status, the updated page status of the iOS application and the App Store metadata are input into the large language model to identify whether the function of the iOS application is consistent with the App Store metadata. If not, the iOS application to be detected is determined to be a sockpuppet package;

[0150] If they are consistent and the current automation task has been completed but the automation task list has not been completely traversed, then clear the history operation list, reselect a task as the current automation task, and perform dynamic testing on the iOS application to be tested according to the reselected current automation task;

[0151] If they are consistent and the current automation task is not completed and the number of iterations has not reached the upper limit, the current page status of the iOS application is retrieved again and dynamic testing of the iOS application to be tested is continued;

[0152] If they are consistent and the current automation task is not completed but the number of iterations has reached the upper limit, then it is determined that the iOS application to be tested is not a vest package;

[0153] If they are consistent and the automation task list is traversed, it is determined that the iOS application to be detected is not a vest package.

[0154] In summary, the present invention provides a dynamic and automated detection method for iOS vest packages based on LLM. By dynamically testing the iOS application to be detected, the problem that the static solution cannot cope with the mixed development mode or dynamic loading of content is effectively solved, and the application can be identified as a vest package more accurately. By merging the application store metadata of the iOS application to be detected and the known iOS vest package knowledge, inputting the LLM for semantic analysis, obtaining potential vest package triggering methods, and obtaining an automated task list, the problem of metadata lag is solved to a certain extent, and the robustness of vest package detection is improved. By pruning the view tree and adding semantic information, the problem that LLM cannot accurately infer the clickable elements and corresponding functions in the page status is effectively solved. By optimizing the recording method of the historical operation list, the problem of looping or avoiding correct operations in task automation is effectively improved.

[0155] According to an embodiment of the present invention, the present invention further provides an electronic device and a readable storage medium.

[0156] Figure 5A schematic block diagram of an electronic device that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein. The electronic device includes a computing unit 101, which can perform various appropriate actions and processes based on a computer program stored in ROM 102 or a computer program loaded from storage unit 108 into RAM 103. Various programs and data required for the operation of the electronic device can also be stored in RAM 103. The computing unit 101, ROM 102, and RAM 103 are connected to each other via a bus 104. An I / O interface 105 is also connected to bus 104.

[0157] Multiple components in the electronic device are connected to the I / O interface 105, including an input unit 106, such as a keyboard, mouse, etc.; an output unit 107, such as various types of displays, speakers, etc.; a storage unit 108, such as a magnetic disk, optical disk, etc.; and a communication unit 109, such as a network card, modem, wireless communication transceiver, etc. The communication unit 109 allows the electronic device to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0158] The computing unit 101 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the computing unit 101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 101 performs the various methods and processes described above. For example, in some embodiments, the methods in the multidimensional early warning system for pressure injuries can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 108. In some embodiments, part or all of the computer program can be loaded and / or installed onto an electronic device via the ROM 102 and / or the communication unit 109. When the computer program is loaded into the RAM 103 and executed by the computing unit 101, one or more steps of the methods in the multidimensional early warning system for pressure injuries described above can be performed. Alternatively, in other embodiments, the computing unit 101 can be configured to perform the methods in the multidimensional early warning system for pressure injuries via any other suitable means (e.g., via firmware).

[0159] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0160] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0161] In the context of the present invention, a readable storage medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A readable storage medium may be a machine-readable signal medium or a machine-readable storage medium. A readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0162] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0163] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0164] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0165] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

[0166] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A dynamic and automated detection method for iOS vest packages based on LLM, characterized in that: The method comprises the following steps: Get the App Store metadata of the iOS app to be tested; In response to the App Store metadata of the iOS app to be detected and the vest package triggering method example, the large language model outputs a trigger keyword dictionary list; based on the trigger keyword dictionary list, the vest package triggering method is constructed and sorted to obtain an automated task list; Select a task from the automation task list as the current automation task, and initialize an empty set as the history operation list; Dynamically test the iOS app to be tested according to the current automated task, obtain and process the current page state of the iOS app; in response to the processed current page state, the large language model outputs a page summary and a list of interactive elements; In response to the page summary, the list of interactive elements, the current automated task, and the list of historical operations, the large language model outputs a target interactive element and a target operation; Execute the target operation on the iOS application according to the target interactive element, obtain the updated page status of the iOS application, and update the historical operation list; The updated page status and app store metadata of the iOS app are input into the large language model to determine whether the page status of the iOS app is consistent with the app store metadata. If they are inconsistent, the iOS app to be detected is determined to be a vest package.

2. A dynamic automatic detection method for iOS vest packages based on LLM according to claim 1, characterized in that: The process of getting a list of automated tasks includes: Get user review text based on the App Store metadata of the iOS app to be tested; In response to the user comment text and the vest package triggering method example, the large language model outputs a trigger keyword dictionary list; wherein each trigger keyword dictionary in the trigger keyword dictionary list includes a confidence score and a keyword region; According to the keyword area corresponding to each trigger keyword dictionary, construct and piece together the trigger method short sentence, so as to obtain the vest package trigger method corresponding to each trigger keyword dictionary; Determine whether the armor package triggering method depends on the context of the iOS application, and sort the armor package triggering methods that do not depend on the context of the iOS application before the armor package triggering methods that depend on the context of the iOS application. The armor package triggering methods that do not depend on the iOS application or depend on the context of the iOS application are sorted from high to low according to the confidence score, thereby obtaining an automated task list.

3. A dynamic automatic detection method for iOS vest packages based on LLM according to claim 2, characterized in that: The keyword area includes: trigger time, trigger geographical conditions, trigger waiting time, interaction location, interaction times, text input location, and text input content; The vest package trigger method that does not rely on the iOS application context represents a vest package trigger method composed of a trigger keyword dictionary with no interaction position, text input position, and text input content.

4. A dynamic automatic detection method for iOS vest packages based on LLM according to claim 1, characterized in that: The process of obtaining a page summary and a list of interactive elements involves: Run the iOS application to be tested and obtain the current page status; wherein the current page status includes a page screenshot and a view tree of the current page; Prune the view tree of the current page to delete invalid information in the view tree of the current page; perform optical character recognition on the page screenshot and add the recognized semantic information to the view tree; classify the leaf elements with a size close to square in the pruned view tree into icons and add the icon categories to the view tree to obtain an enhanced view tree; In response to the page screenshot and the enhanced view tree of the current page, the large language model outputs a page summary and a list of interactive elements corresponding to the current page.

5. A dynamic automatic detection method for iOS vest packages based on LLM according to claim 1, characterized in that: In response to the page summary, the list of interactive elements, the current automated task, and the list of historical operations, the process of the large language model outputting target interactive elements and target operations includes: When the target operation output by the large language model is click or continuous click, the number of clicks is also set; when the target operation output by the large language model is text input, the string to be input is also set; When the large language model determines that the current automated task has been completed, the target operation is set to task completed and the target interactive element is set to none; When the large language model determines that no operation is applicable to the current automation task, the target operation is set to none and the target interactive element is set to none.

6. The LLM-based iOS vest package dynamic automatic detection method according to claim 1 is characterized in that: Execute the target operation on the iOS application based on the target interactive element, obtain the updated page status of the iOS application, and update the history operation list. The process includes: Re-acquire the view tree of the current page and determine whether the view trees corresponding to the original page and the current page have changed. If the similarity between the view trees corresponding to the original page and the current page is lower than a first threshold and the tree edit distance is higher than a second threshold, and the number of times the current page state of the iOS application has been re-acquired does not exceed a third threshold, then re-acquire the current page state of the iOS application and re-acquire the target interactive element and target operation. Execute the target action on the iOS app based on the target interactive element. If the target action is click, continuous click, text input, or restart, execute the corresponding target action on the target interactive element. If the target action is task completion, end the current automation task. If the target action is none, do nothing. After executing the target operation on the iOS application according to the target interactive element, a triplet is added to the history operation list: <page summary, target interactive element, target operation performed>, thereby updating the history operation list.

7. The LLM-based iOS vest package dynamic automatic detection method according to claim 1 is characterized in that: The updated page status of the iOS app and the App Store metadata are input into the large language model. The process of determining whether the page status of the iOS app is consistent with the App Store metadata includes the following: Calculate the hash value of the view tree in the updated page state of the iOS application to determine whether the updated page state is a new page state; If the updated page status is a newly appeared page status, the updated page status of the iOS application and the App Store metadata are input into the large language model to identify whether the functions of the iOS application are consistent with the App Store metadata. If they are inconsistent, the iOS application to be tested is determined to be a vest package; if they are consistent and the current automation task has been completed and the automation task list has not been traversed, the historical operation list is cleared, and a task is reselected as the current automation task, and the iOS application to be tested is dynamically tested according to the reselected current automation task; if they are consistent and the current automation task is not completed and the number of iterations has not reached the upper limit, the current page status of the iOS application is re-obtained, and the dynamic testing of the iOS application to be tested continues; if they are consistent and the current automation task is not completed but the number of iterations has reached the upper limit, the iOS application to be tested is determined not to be a vest package; if they are consistent and the automation task list has been traversed, the iOS application to be tested is determined not to be a vest package.

8. An electronic device comprising a memory and a processor, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the LLM-based iOS vest package dynamic automatic detection method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by the processor, the LLM-based dynamic automatic detection method for iOS vest packages described in any one of claims 1 to 7 is implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the LLM-based dynamic automatic detection method for iOS vest packages described in any one of claims 1 to 7 is implemented.

Images