LLM-based iOS waistcoat dynamic automatic detection method, device and medium

Abstract

The invention discloses an LLM (Logical Language Model)-based iOS (iphone Operating System) waistcoat dynamic automatic detection method and device and a medium. The method comprises the following steps: acquiring application store metadata of a target iOS application, generating a trigger keyword dictionary through a large language model in combination with a waistcoat triggering method example, and constructing an automatic task list based on the dictionary; selecting a task as a current automatic task, and testing the iOS application; in response to the current page state of the iOS application, the model outputs a page abstract and an interactive element list; in response to the page abstract, the interactive element list, the current automation task and the historical operation list, the model outputs target interactive elements and target operations; executing a target operation on the iOS application according to the target interactive element to obtain an updated page state of the iOS application; the updated page state of the iOS application and the application store metadata are input into the model, whether the page state of the iOS application is consistent with the application store metadata or not is judged, and if not, the iOS application is a waistcoat bag.

Description

Technical Field

[0001] The present invention belongs to the technical field of iOS application detection, and in particular to a dynamic automated detection method, device, and medium for iOS chameleon apps based on LLM. Background Art

[0002] iOS is an operating system widely used in Apple's mobile devices (such as iPhones and iPads). With the rapid development of the Internet and the wide popularity of iOS applications, more and more lawbreakers have started to spread illegal content and make profits through iOS applications.

[0003] Since the App Store is the main distribution channel for iOS applications and the main platform for iOS users to obtain applications, it has inevitably become the main target for lawbreakers to distribute illegal applications. In order to bypass the review of the App Store platform, lawbreakers have gradually adopted a "chameleon app" strategy. These applications disguise themselves as legitimate application programs during the review process and only show hidden illegal functions (such as gambling or pirated application programs) after completing specific trigger operations.

[0004] Existing chameleon app detection solutions include analysis methods based on static analysis to discover hidden UI semantic differences and analysis methods based on user reviews and metadata. However, static analysis has limited effectiveness in dealing with chameleon apps in hybrid development modes or with dynamically loaded content; detection methods based on user reviews and metadata rely on the quality, quantity accumulation, and timeliness of reviews, have a time lag, are difficult to quickly respond to newly emerging threats, and also have a relatively high probability of false negatives and false positives.

[0005] Therefore, there is an urgent need to propose a detection method for illegal applications on the iOS platform that can effectively identify and analyze various types of chameleon apps and improve the accuracy and coverage of illegal behavior identification. Summary of the Invention

[0006] Aiming at the deficiencies of the prior art, the present invention provides a dynamic automated detection method, device, and medium for iOS chameleon apps based on LLM.

[0007] In a first aspect, an embodiment of the present invention provides a dynamic automated detection method for iOS chameleon apps based on LLM, the method comprising the following steps:

[0008] Obtain the application store metadata of the iOS application to be detected;

[0009] In response to the app store metadata of the iOS app to be detected and an example of the trigger method for shadow apps, the large language model outputs a list of trigger keyword dictionaries; based on the list of trigger keyword dictionaries, construct and sort the trigger methods for shadow apps to obtain an automated task list;

[0010] Select a task from the automated task list as the current automated task, and initialize an empty set as the historical operation list;

[0011] Perform dynamic testing on the iOS app to be detected according to the current automated task, and obtain and process the current page state of the iOS app; in response to the processed current page state, the large language model outputs a page summary and a list of interactive elements;

[0012] In response to the page summary, the list of interactive elements, the current automated task, and the historical operation list, the large language model outputs the target interactive element and the target operation;

[0013] Execute the target operation on the iOS app according to the target interactive element to obtain the updated page state of the iOS app, and update the historical operation list;

[0014] Input the updated page state of the iOS app and the app store metadata into the large language model, and determine whether the page state of the iOS app is consistent with the app store metadata. If not, it is determined that the iOS app to be detected is a shadow app.

[0015] In a second aspect, an embodiment of the present invention provides an electronic device, including a memory and a processor, the memory is coupled to the processor; wherein, the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned dynamic automated detection method for iOS shadow apps based on LLM.

[0016] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the above-mentioned dynamic automated detection method for iOS shadow apps based on LLM.

[0017] In a fourth aspect, an embodiment of the present invention provides a computer program product, including a computer program / instructions, and when the computer program / instructions are executed by a processor, it implements the above-mentioned dynamic automated detection method for iOS shadow apps based on LLM.

[0018] Compared with the prior art, the beneficial effects of the present invention are:

[0019] The present invention provides a dynamic automated detection method for iOS clone apps based on LLM. By dynamically testing the iOS app to be detected, merging the app store metadata of the iOS app to be detected and the example of the triggering method for clone apps as the known knowledge of iOS clone apps, inputting it into the LLM for semantic analysis, obtaining the potential triggering methods for clone apps, and getting the automated task list, it solves to a certain extent the problem of the lag in the app store metadata of iOS apps and improves the robustness of clone app detection; the present invention effectively solves the problem that the static solution cannot cope with the hybrid development mode or dynamically loaded content, and can more accurately identify the app as a clone app. Brief Description of the Drawings

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0021] Figure 1 It is a flowchart of a dynamic automated detection method for iOS clone apps based on LLM provided by an embodiment of the present invention;

[0022] Figure 2 It is a schematic diagram of the list of triggering methods provided by an embodiment of the present invention;

[0023] Figure 3 It is a schematic diagram of pruning the view tree and adding semantic information provided by an embodiment of the present invention;

[0024] Figure 4 It is a schematic diagram of extracting the page summary and the list of interactive elements provided by an embodiment of the present invention;

[0025] Figure 5 It is a schematic diagram of an electronic device provided by an embodiment of the present invention. Detailed Embodiments

[0026] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0027] It should be noted that, without conflict, the features in the following embodiments and implementation manners can be combined with each other.

[0028] To clearly illustrate the technical solutions provided by the embodiments of the present application, some concepts that may appear in subsequent embodiments will be introduced first.

[0029] Large Language Model (LLM): It is a natural language processing model trained through deep learning technology, especially based on large-scale text data. It can understand, generate, translate, and reason about complex relationships in language, and is widely used in fields such as natural language understanding, automated generation, and dialogue systems. In the field of mobile task automation, it is also often used to understand the current page and make decisions.

[0030] Prompt: It is the text or instruction input to the large language model to guide the model to generate specific outputs. Prompts are usually concise and clear, and can be a question, command, or context, helping the model focus on specific tasks or content. Through carefully designed prompts, users can achieve functional calls to the LLM to complete various natural language processing tasks.

[0031] Structured Output: It refers to the output result with a clear organizational structure generated by the LLM according to the input prompt. It can be designed in data formats such as nested fields, lists, dictionaries, JSON, etc. By designing specific output formats, the understanding focus of the large language model can be guided to ensure that the output has high readability and high operability.

[0032] View Hierarchy: It refers to an abstract hierarchical structure representation among UI (User Interface) elements in an application, reflecting the layout of the overall page and the attributes and parent-child relationships of each element component. It is usually displayed in a tree structure, where the root node represents the entire interface, and each child node represents a specific UI element (such as buttons, text boxes, images, etc.). Each node contains various attributes of the element (such as type, name, value, location information, visibility, etc.). The view hierarchy helps developers understand and control the hierarchical structure of the interface layout and plays a crucial role in UI automation testing.

[0033] As Figure 1 shown, the embodiments of the present invention provide an iOS alternative package dynamic automation detection method based on LLM. The method includes the following steps:

[0034] Step S1, obtain the application store metadata of the iOS application to be detected.

[0035] Specifically, the step S1 includes the following sub-steps:

[0036] Step S101: Collect the application name, developer name, application category, application description, and user reviews displayed in the app store for the iOS application to be detected as the app store metadata of the iOS application.

[0037] Exemplarily, to obtain the app store metadata of the iOS application, use a Selenium script to access the link of the iOS application to be detected in the App Store, and obtain app store metadata such as the application name, developer name, application category, application description, and user reviews from the application display page.

[0038] Step S102: For the iOS application to be detected, download the IPA file of the application through the app store, thereby installing the iOS application to be detected on the test device.

[0039] Exemplarily, to download the IPA file of the application through the app store, first use the ipatool-py tool to download the latest version IPA file of the application to be detected in the app store, and then use the tidevice tool to install it in batches on the test mobile phone. Among them, the IPA file (iPhone Application Archive, the installation package file of the iOS application) contains all the content required to install and run the application on iOS devices (such as iPhone, iPad).

[0040] Step S2: In response to the app store metadata of the iOS application to be detected and the example of the trigger method for the shadow package, the large language model outputs a list of trigger keyword dictionaries; according to the list of trigger keyword dictionaries, construct and sort the trigger methods for the shadow package to obtain an automated task list.

[0041] Specifically, step S2 includes the following sub-steps:

[0042] Step S201: Obtain the user review text according to the app store metadata of the iOS application to be detected, and preprocess the user review text to obtain the preprocessed user review text.

[0043] Specifically, in this example, set the upper limit of the number of user review texts obtained to 200. All reviews retain the title and content, and are stored in the format of "Title: Content". The upper limit of the number of words for each review text is 40, and all review texts are concatenated through a delimiter; as the user review text, the upper limit of the number of words for the user review text is 800, and the part exceeding the limit will be deleted.

[0044] Step S202: Based on the known example of the trigger method for the shadow package, design the first prompt word and the corresponding first structured output. The first prompt word includes the example of the shadow package trigger method, user review text, and requirement instructions. The first structured output is a list of trigger keyword dictionaries. Each trigger keyword dictionary in the list of trigger keyword dictionaries includes a confidence score and a keyword area. The keyword area includes: trigger time, trigger geographical condition, trigger waiting time, interaction location, number of interactions, text input location, and text input content. The confidence score represents the evaluation of the likelihood of this trigger keyword dictionary by the LLM.

[0045] Exemplarily, in this example, the sample of the first prompt word designed is as follows (the parts to be filled are indicated by [ ]):

[0046] Instructions for use: A disguised application refers to an application whose actual function does not match the function shown in the app store and requires specific conditions to trigger the "transformation" of the APP that conforms to the display page. It is necessary to infer whether it is a disguised application and the trigger method based on the basic information and comments.

[0047] 1. Trigger method

[0048] First, provide a series of trigger methods for the disguised application:

[0049] [1] Open / multiple open / wait: Such as directly open, restart and automatically transform, wait for 10 seconds after opening.

[0050] [2] Specific time and space conditions: Such as locate in China, turn on during a specific time period.

[0051] [3] Specific actions: Specific click operations, such as click on the blank space 5 times, continuously click on the home page icon 5 times, watch an advertisement.

[0052] [4] Specific input content or options: Enter specific content in the text box or select specific content in the selection box (such as problem feedback, my suggestions, parameter column, search box) or copy specific content to the clipboard.

[0053] These methods can be used in combination: For example, enter 999999 in the calculator and then click the calculate button three times (a combination of method [3] and method [4]); click on the screen loading bar 10 times and then re-enter (a combination of method [1] and method [3]).

[0054] 2. Requirements

[0055] Please identify the keywords that may be related to the four trigger methods based on the comment information, including: waiting duration, geographical / time conditions, click location, number of clicks, specific input location, and specific input text.

[0056] 3. Supplementary instructions

[0057] Special attention should be paid to phrases in the comments where the title is the same as the content (usually Chinese phrases of 3 to 5 characters and not interrogative sentences), which may be specific input content. Only when the phrase has nothing to do with the application and appears multiple times in the comments, or is clearly associated with words such as "trigger / transformation", will it be regarded as specific input content.

[0058] Examples of specific input content: 666, ys777, Big Ear Tutu, Big Macaque, Hello123, To become beautiful, one must be confident, I love China, 999999, 5201080, Little Persimmon, Montage.

[0059] Examples of specific input locations: Feedback, Text Input Box, Search Box, Clipboard.

[0060] 4. Input

[0061]

User Comment Text

[0062] Step S203, fill the preprocessed user comment text into the prompt words and input it into the large language model for semantic analysis to obtain the list of trigger keyword dictionaries predicted by the LLM.

[0063] Step S204, construct and splice the trigger method short sentences according to the keyword regions corresponding to each trigger keyword dictionary in the list of trigger keyword dictionaries, so as to obtain the trigger methods for the split APKs corresponding to each trigger keyword dictionary.

[0064] Exemplarily, in this example, all keyword regions and the corresponding trigger method short sentences are as follows:

[0065] Trigger time: Open the APP at {} time;

[0066] Trigger geographical condition: Open the APP at {} geographical location;

[0067] Trigger waiting time: Wait for {} seconds;

[0068] Interaction location and interaction times: Click continuously {} times at the {} location;

[0069] Text input location and text input content: Input {} content at the {} location.

[0070] All regions are initially set to None. For the two pairs of keyword regions of interaction location and interaction times, and text input location and text input content, when there is a predicted element (not None) in any one of the keyword regions in each pair, the corresponding trigger method short sentence will be constructed, and the other unpredicted keyword region will be filled with default words.

[0071] Further, the default words in the keyword area are as follows: Interaction position: Any position on the page; Interaction times: 1; Text input position: Text input box; Text input content: Potential trigger word.

[0072] Further, after concatenating all the trigger method short sentences, add the prompt "Restart the application, complete the trigger, and explore from different angles and page semantics (pay special attention to function icons, About, Mine, Feedback, Search, Settings, Properties, Personal, etc. functions first)" at the end.

[0073] Exemplarily, as Figure 2 shown, there are three trigger keyword dictionary items in the trigger keyword dictionary list predicted by the LLM for a certain iOS application. In the first trigger keyword dictionary, only one keyword area "Trigger waiting time" is successfully filled, so only the trigger method short sentence "Wait for 10 seconds" is generated. After concatenating the prompt, we get "Wait for 10 seconds; Restart the application, complete the trigger, and explore from different angles and page semantics (pay special attention to function icons, About, Mine, Feedback, Search, Settings, Properties, Personal, etc. functions first)", which is used as the first alternative package trigger method; the second trigger keyword dictionary generates "Enter the content 'Hello 123' at the problem feedback position", and similarly, we get the second alternative package trigger method; for the third trigger keyword dictionary, first generate the short sentence "Open the APP at 12:00 am". Since the interaction position is predicted as "Home icon" and the interaction times is None, generate the short sentence "Click {} times continuously at the home icon position" and fill it with the default word 1 for interaction times, forming "Click 1 time continuously at the home icon position". After concatenating, we get "Open the APP at 12:00 am, click 1 time continuously at the home icon position", and then concatenate the prompt to get "Open the APP at 12:00 am, click 1 time continuously at the home icon position; Restart the application, complete the trigger, and explore from different angles and page semantics (pay special attention to function icons, About, Mine, Feedback, Search, Settings, Properties, Personal, etc. functions first)", and finally get the third alternative package trigger method.

[0074] Step S205, determine whether the alternative package trigger method depends on the context of the iOS application. Arrange the alternative package trigger methods that do not depend on the context of the iOS application before those that depend on the context of the iOS application. The alternative package trigger methods that do not depend on the iOS application or depend on the context of the iOS application are sorted from high to low according to the confidence score, so as to obtain an automated task list.

[0075] Specifically, in this example, the alternative package trigger method that does not depend on the iOS application context represents the alternative package trigger method composed of trigger keyword dictionaries where the three keyword areas of interaction position, text input position, and text input content are all None.

[0076] Step S3, select a task from the automated task list as the current automated task, and initialize an empty set as the historical operation list.

[0077] Step S4, perform dynamic testing on the iOS application to be detected according to the current automated task, obtain and process the current page state of the iOS application; in response to the processed current page state, the large language model outputs a page summary and a list of interactive elements.

[0078] Specifically, step S4 includes the following sub-steps:

[0079] Step S401, run the iOS application to be detected in a dynamic testing environment built based on Appium; if the current automated task involves time, geographical location or clipboard content, the current device environment needs to be adjusted.

[0080] Exemplarily, if the current automated task involves time, geographical location or clipboard content, the process of adjusting the current device environment includes: using a Python script and the Appium tool to connect to the communication driver (driver) of the device; if the current automated task involves time and geographical location, use a predefined code sequence (the modified time is through ssh root@localhost -p 2222 and then use the command date MMDDhhmmYYYY, and the geographical location uses the driver.set_location function) to adjust the current device environment; if the current automated task involves clipboard content, write the target content to the clipboard through the driver.set_clipboard_text method.

[0081] Step S402, obtain and process the current page state of the iOS application, where the current page state includes a page screenshot and a view tree of the current page.

[0082] Exemplarily, to obtain the page state of the current page, obtain the png file of the page screenshot and the XML file of the view tree through the driver.save_screenshot and driver.page_source methods.

[0083] Step S403, perform pruning operations on the view tree of the current page to delete invalid information in the view tree of the current page; perform optical character recognition on the page screenshot, and supplement the recognized semantic information to the view tree; classify the leaf elements with dimensions close to a square in the pruned view tree, and supplement the icon categories to the view tree to obtain an enhanced view tree.

[0084] Specifically, as Figure 3As shown in the figure, in this example, invalid information in the view tree of the current page is deleted: pruning operations are performed on the view tree to delete irrelevant elements and unnecessary attribute information, recursively merging parent-child element pairs with only one child element and elements with the same position information (including coordinates and dimensions), and then rearranging the nodes in the order of position information from left to right and top to bottom to ensure that each visual area corresponds one-to-one with the elements in the view tree.

[0085] Optical character recognition is performed on the page screenshot, and the recognized semantic information is supplemented into the view tree: the PaddleOCR model is used for the page screenshot to extract all texts and their corresponding positions. Traverse all elements in the view tree. If the text is not included in any of the attributes of the smallest area matching element that contains this position, then this text is added as an additional attribute orc_text to this element.

[0086] For leaf elements in the pruned view tree whose size in all position information is close to a square (in this example, the leaf element is an element without other child elements, and the element close to a square has an aspect ratio between 0.95 and 1.05), assuming this element is an icon, use the EfficientNet-B0 model trained with RICO Semantics and some icons from variant packages as the dataset to perform the icon classification task, and obtain the functional category of this icon (such as ICON_X represents the × icon, ICON_SETTINGS represents the settings icon, ICON_MAGNIFYING_GLASS represents the magnifying glass icon), and add this category as an additional attribute icon_label to this element.

[0087] It should be noted that in this example, by pruning the view tree and adding semantic information, the problem that the LLM cannot accurately infer the clickable elements and corresponding functions in the page state is effectively solved.

[0088] Step S404, design the second prompt word and its corresponding second structured output; the second prompt word is a text paragraph containing the goal, requirements, and view tree and a picture part containing the page screenshot, and the second structured output is a list of interactive elements, and each interactive element contains attributes such as the corresponding ID of the target element in the view tree, position information, and semantic information.

[0089] Exemplarily, in this example, the sample of the designed second prompt word is as follows (the parts to be filled are represented by 【】):

[0090] Conduct a comprehensive analysis of the screenshot and its runtime UI elements:

[0091] 1. Tasks and key points

[0092] Precisely match the ID and bounding box of each element with the screenshot. Discard the element if the bounding box does not match exactly. When the UI differs significantly from the screenshot, it is recommended to wait for synchronization. Prioritize the processing of key elements such as close, skip, navigation, action buttons, and function icons (including search, submit, settings, tools, favorites, like, location, return, etc.). Sort the sub-views under the same parent view from top to bottom and from left to right. If there is an advertising pop-up window, its close button must be located (usually in the four corners, marked as 'ICON_X').

[0093] 2. Requirements and Boundaries:

[0094] Ignore the iOS native status bar / navigation bar and the keyboard. It must include text boxes, buttons, function icons, and valid static text and images (especially text and images nested in clickable cells). Focus on the ocr_text attribute elements obtained through an independent OCR processing process, and the icon_label attribute elements obtained through independent CNN image classification. Actively expand the included elements to cover adjacent elements that may exist.

[0095] 3. Restrictions and Conditions:

[0096] Present the UI hierarchical structure (layout from top to bottom and from left to right) in a general description, including components such as the title bar, sidebar, top bar, tab page, footer, menu, etc. (if any). Only identify the alert box that is fully visible and occupies the entire screen as an emergency top-level pop-up window. Pop-up windows containing promotional copy ('Download', 'Details', etc.) need to be marked as advertisements. Advertisements located in the corners and not blocking other UI elements are not considered emergency top-level pop-up windows. Discard all promotional copy and buttons within the advertising pop-up window, and only retain the close button. It must include all visible feedback / status / error prompts (including reminders / toast message prompt boxes / pop-up window text / bottom status bar), and mark whether the UI is in the loading state.

[0097] 4. Output Format:

[0098] Strictly follow the output structure containing two core modules. "hint_elements" represents storing non-interactive / semi-interactive UI hints (such as status information, guiding text); "action_elements" stores clearly interactive elements (with complete bounding boxes and function identifications).

[0099] 5. Input

[0100]

Page Screenshot

Enhanced View Tree

[0101] In step S405, in response to the page screenshot and enhanced view tree of the current page, the large language model outputs the page summary and the list of interactive elements corresponding to the current page.

[0102] Exemplarily, after inputting the second prompt composed of the page screenshot of a certain iOS application and the enhanced view tree into the LLM, a page summary and a list of interactive elements of the current page are obtained. The page summary describes the current page in a text paragraph, and the list of interactive elements contains attributes such as the id, category, description, text, location, etc. of each speculated interactive element. Specifically, as Figure 4 shown, the page screenshot shows the slogan "Advanced Handwritten Signature - Let You Write Your Own Personal Signature; Different Paintbrushes Give You Different Feelings; Different Colors Make Your Signature More Colorful", the control "Start Handwritten Signature", the control "Share with Friends", and the control "Have a Problem? Feedback to Me". The page summary output by the large language model is: On the page showing the application functions, you can choose to start signing, share, or provide feedback. The list of interactive elements output by the large language model is shown in Table 1 below:

[0103] Table 1: Example of the list of interactive elements

[0104]

[0105] Step S5, in response to the page summary, the list of interactive elements, the current automation task, and the historical operation list, the large language model outputs the target interactive element and the target operation.

[0106] Specifically, the step S5 includes the following sub-steps:

[0107] Step S501, design the third prompt and its corresponding third structured output; the third prompt is a text paragraph containing the goal, requirements, page summary, list of interactive elements, current task, and historical operations; the third structured output is a specific operation, including the target element and the execution action, where all possible execution actions include: click (continuous click), text input, restart, task completion, none.

[0108] Specifically, in this example, the sample of the designed third prompt is as follows (the part to be filled is represented by 【】):

[0109] Comprehensively analyze the current UI state and plan subsequent operations:

[0110] 1. Tasks and key points:

[0111] Understand the user's ultimate task goal based on UI data and operation history. If the current page cannot complete the subtask, actively explore the application by clicking on labels / buttons and following the page guidance. Learn the application interaction mode through historical operation records to avoid repetitive and redundant operations. If the operation fails to achieve the expected feedback, allow the entire process or some steps to be retried (up to 3 times). It is prohibited to repeat the same operation sequence on the already visited page, unless it may trigger a new state.

[0112] 2. Basic principles:

[0113] Take the UI element list as the first benchmark. Strictly match the element ID with the operation target. Prioritize the operations that can advance the user's tasks or have never been performed; avoid irrelevant / repetitive steps. If the operation cannot be performed directly, try alternative navigation or follow the page semantics guidance first, and finally recommend "no operation". When repeating the same operation without progress, suspend the main task, and instead perform exploratory operations on native functions and understand the page semantics before deciding on subsequent actions.

[0114] 3. Restrictions:

[0115] Prioritize emergency top-level pop-ups and ads (close or skip with minimal operations). Emergency unexpected pop-ups are handled with higher priority than ads. Automatically grant the app the required system permissions. Abort dialog box operations that are not related to the current task, such as selecting local files / photos. If the interface is in a loading state or no feasible operation is possible after careful evaluation, it is recommended to "wait for the next attempt". It is recommended to restart the app when the ad cannot be closed. Clicking on the ad details is strictly prohibited. If no operation is possible after multiple waits, or the current interface cannot be jumped, it is recommended to restart the app with caution. The task is only considered completed when it is clear that the historical feedback meets the conditions. Directly perform text input on the text box (automatically clear the original content) without clicking the text box first.

[0116] 4. Output specifications and logic:

[0117] Strictly follow the JSON output format description. The summary content must be concise and accurate.

[0118] 5. Input

[0119] [Page summary] + [List of interactive elements] + [Current automation task] + [Historical operation list]

[0120] Step S502, fill the page summary, the list of interactive elements, the current automated task and the list of historical operations into the third prompt word, input it into the large language model for action decision, and the large language model outputs the target interactive element and the target operation.

[0121] Specifically, in this example, when the target operation output by the large language model is click or continuous click, the number of clicks is also set; when the target operation output by the large language model is text input, the character string to be input is also set;

[0122] When the large language model determines that the current automated task has been completed, the target operation is set to task completion and the target interactive element is set to None;

[0123] When the large language model determines that the current automated task is not applicable to any operation, set the target operation to none and the target interactive element to None.

[0124] Step S6, perform the target operation on the iOS application according to the target interactive element, obtain the updated page state of the iOS application, and update the historical operation list.

[0125] Specifically, step S6 includes the following sub-steps:

[0126] Step S601, re-obtain the view tree of the current page, and determine whether the view trees corresponding to the original page and the current page obtained in step S402 have changed. If the Jaccard similarity between the view trees corresponding to the original page and the current page is lower than the first threshold (set to 0.8 in this example), the tree edit distance is higher than the second threshold (set to 3 in this example), and the number of consecutive rollbacks does not exceed the pre-set third threshold (set to 3 in this example), then roll back to step S402 to re-obtain the current page state of the iOS application, and re-obtain the target interactive element and the target operation.

[0127] It should be noted that during the execution of the target operation, the page of the iOS application may change (such as an advertisement popping up), resulting in the invalidation of the target operation predicted in step S5. Therefore, it is necessary to re-obtain the page state in step S601 and compare whether there are obvious changes, so as to determine whether the target operation predicted in step S5 is still valid.

[0128] Step S602, perform the target operation on the iOS application according to the target interactive element. If the target operation is click, continuous click, text input, or restart, perform the corresponding target operation on the target interactive element; if the target operation is task completion, end the current automated task; if the target operation is none, do not perform any operation.

[0129] Exemplarily, when the action to be performed is click (continuous click), use a Python script and the ZXTouch tool to implement a click operation at a specific coordinate based on the position information of the element corresponding to the target element ID through the zxtouch.touch method; when the action to be performed is text input, use a Python script and the Appium tool, and through the attributes of the element corresponding to the target element ID, first use the driver.find_element method to find the text input box element object element, and then use the element.send_keys method to implement the input of the target text; when the action to be performed is restart, use the driver.terminate_app and driver.activate_app methods to implement closing the application and opening the application.

[0130] Step S603: After performing the target operation on the iOS application based on the target interactive element, add a triple <page summary, target interactive element, perform target operation> to the historical operation list, thereby updating the historical operation list.

[0131] It should be noted that in this example, by optimizing the recording method of the historical operation list, the problems of possible loops or avoidance of correct operations in task automation are effectively improved.

[0132] Exemplarily, in a certain iOS application during the current automation task (inputting the content "666" at the "Feedback Input" position, restarting the application, completing the trigger, and exploring from different angles and page semantics (pay special attention to function icons, About, My, Feedback, Search, Settings, Properties, Personal, etc. functions) has gone through 4 rounds of loops. In the first round, the "About APP" button was clicked in the main menu. In the second round, "666" was input in the feedback box on the About APP page. In the third round, the submit button was clicked on the About APP page for feedback. In the fourth round, since it was detected that the application automatically exited (the application will automatically crash when the trigger is successful), the restart execution action was selected. The historical operation list records:

[0133] 1. Launch the application from SpringBoard

[0134] 2. <Main menu (including mode selection button and "About Application" link), "About Application" button, click>

[0135] 3. <"About APP" page (including game play description and feedback area, including input box and submit button), "Feedback" input box, input "666">

[0136] 4. <"About APP" page (including game play description and feedback area, including input box and submit button), "Submit" button, click>

[0137] 5. The application may have crashed and become unresponsive. Please restart the application

[0138] Step S7: Input the updated page state of the iOS application and the application store metadata into the large language model to determine whether the page state of the iOS application is consistent with the application store metadata. If not, it is determined that the iOS application to be detected is a shadow package. Specifically, step S7 includes the following sub-steps:

[0139] Step S701: Calculate the hash value of the view tree in the updated page state of the iOS application to determine whether the updated page state is a newly emerged page state;

[0140] If the updated page state is not a newly emerged page state, the updated page state is a page that has been compared, and the judgment result of the default LLM is consistent, so step S702 is directly skipped.

[0141] Step S702: Design the fourth prompt. Fill the page state after the iOS application update and the app store metadata into the fourth prompt and input it into the large language model. The large language model determines whether the page state of the iOS application is consistent with the app store metadata.

[0142] Furthermore, the fourth prompt is a text paragraph containing the target, requirements, app store description, and view tree, and a picture part containing the page screenshot.

[0143] Furthermore, the app store description includes: app name, app category, and app brief introduction.

[0144] Specifically, in this example, the sample of the designed fourth prompt is as follows (the parts to be filled are indicated by 【】):

[0145] I need to determine whether an application provides pages and functions that are completely unrelated to the description in the app store. Please evaluate whether the runtime screenshots and view tree text are completely unrelated to the description provided in the basic information and output "Yes" or "No":

[0146] Output "Yes": The screenshots and text are significantly inconsistent with the app store description, showing functions and pages that are completely unrelated to the app store description. The screenshots and text show that the pop-up window or prompt message contains instructions for activating hidden modes or other behaviors. (For example, "Enable hidden mode", "Steps to unlock all functions", "Please unlock as required", "Enter 666 to enjoy full functions").

[0147] Output "No": The screenshots and text are completely consistent with the app store description. Due to the presence of marketing-related content (such as advertisements, promotions, app stores, often containing keywords such as "advertisement", "Download now", "Shake to win", "Go to APP", "View details", "Third-party app") covering and obscuring the original page, the function evaluation is unclear; the displayed page is a general or neutral page (such as "Feedback", "Personal Center", "User Agreement", "Menu"), and no direct evidence of hidden functions is observed; the page is a system-level pop-up window or prompt (app store, system settings, file dialog box, etc.), which is irrelevant to the current application; the page is still loading and temporarily shows a white screen or blank page.

[0148] Input:

App store description

Page screenshot

Enhanced view tree

[0149] Further, if the updated page state is a newly emerged page state, input the updated page state of the iOS application and the app store metadata into the large language model to identify whether the functions of the iOS application are consistent with the app store metadata. If they are inconsistent, it is determined that the iOS application to be detected is a shadow app;

[0150] If they are consistent, the current automation task has been completed, and the automation task list has not been traversed, clear the historical operation list, reselect a task as the current automation task, and perform dynamic testing on the iOS application to be detected according to the reselected current automation task;

[0151] If they are consistent, the current automation task has not been completed, and the number of iterations has not reached the upper limit, re-obtain the current page state of the iOS application, and continue to perform dynamic testing on the iOS application to be detected;

[0152] If they are consistent, the current automation task has not been completed, but the number of iterations has reached the upper limit, it is determined that the iOS application to be detected is not a shadow app;

[0153] If they are consistent and the automation task list has been traversed, it is determined that the iOS application to be detected is not a shadow app.

[0154] In summary, the present invention provides a dynamic automated detection method for iOS shadow apps based on LLM. By performing dynamic testing on the iOS application to be detected, it effectively solves the problem that the static solution cannot handle the hybrid development mode or dynamically loaded content, and can more accurately identify the application as a shadow app; by merging the app store metadata of the iOS application to be detected and the known knowledge of iOS shadow apps, inputting them into the LLM for semantic analysis, obtaining potential shadow app trigger methods, and obtaining an automation task list, it solves the problem of the lag of metadata to a certain extent and improves the robustness of shadow app detection; by pruning the view tree and adding semantic information, it effectively solves the problem that the LLM cannot accurately infer the clickable elements and corresponding functions in the page state; by optimizing the recording method of the historical operation list, it effectively improves the problem of possible loops or avoidance of correct operations in task automation.

[0155] According to the embodiments of the present invention, the present invention also provides an electronic device and a readable storage medium.

[0156] Figure 5FIG. shows a schematic block diagram of an electronic device that can be used to implement an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as, for example, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, for example, personal digital processors, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein. The electronic device includes a computing unit 101 that can perform various appropriate actions and processes according to a computer program stored in the ROM 102 or a computer program loaded from the storage unit 108 into the RAM 103. In the RAM 103, various programs and data required for the operation of the electronic device can also be stored. The computing unit 101, the ROM 102, and the RAM 103 are connected to each other via a bus 104. The I / O interface 105 is also connected to the bus 104.

[0157] Multiple components in the electronic device are connected to the I / O interface 105, including: an input unit 106, such as a keyboard, a mouse, etc.; an output unit 107, such as various types of displays, speakers, etc.; a storage unit 108, such as a magnetic disk, an optical disk, etc.; and a communication unit 109, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 109 allows the electronic device to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0158] The computing unit 101 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 101 executes the various methods and processes described above. For example, in some embodiments, the method in the multi-dimensional early warning system for pressure injuries can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 108. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device via the ROM 102 and / or the communication unit 109. When the computer program is loaded into the RAM 103 and executed by the computing unit 101, one or more steps of the method in the multi-dimensional early warning system for pressure injuries described above can be executed. Alternatively, in other embodiments, the computing unit 101 can be configured to execute the method in the multi-dimensional early warning system for pressure injuries in any other suitable manner (e.g., by means of firmware).

[0159] The various embodiments of the systems and techniques described above in this specification can be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGA), application specific integrated circuits (ASIC), application specific standard products (ASSP), systems on chip (SOC), complex programmable logic devices (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be a special-purpose or general-purpose programmable processor that receives data and instructions from, and transmits data and instructions to, a storage system, at least one input device, and at least one output device.

[0160] The program code for implementing the methods of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions / operations specified in the flowchart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine, or entirely on the remote machine or server.

[0161] In the context of the present invention, a machine-readable storage medium can be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable storage medium can be a machine-readable signal medium or a machine-readable storage medium. The machine-readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0162] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, speech input, or tactile input).

[0163] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected to each other by digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include: local area network (LAN), wide area network (WAN), and the Internet.

[0164] A computer system can include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The client - server relationship is created by computer programs running on the respective computers and having a client - server relationship with each other. The server can be a cloud server, or a server of a distributed system, or a server incorporating blockchain.

[0165] Although embodiments of the present invention have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the appended claims and their equivalents.

[0166] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention shall be included within the scope of protection of the present invention.

Claims

1. An iOS alternative app dynamic automated detection method based on LLM, characterized in that, The method includes the following steps: Obtain the app store metadata of the iOS app to be detected; In response to the app store metadata of the iOS app to be detected and the example of the trigger method for the shadow app, the large language model outputs a list of trigger keyword dictionaries; According to the list of trigger keyword dictionaries, construct and sort the trigger methods for the shadow app to obtain an automated task list; Select a task from the automated task list as the current automated task, and initialize an empty set as the historical operation list; Perform dynamic testing on the iOS app to be detected according to the current automated task, and obtain and process the current page state of the iOS app; In response to the processed current page state, the large language model outputs a page summary and a list of interactive elements; In response to the page summary, the list of interactive elements, the current automated task, and the historical operation list, the large language model outputs the target interactive element and the target operation; Execute the target operation on the iOS app according to the target interactive element to obtain the updated page state of the iOS app, and update the historical operation list; Input the updated page state of the iOS app and the app store metadata into the large language model, and determine whether the page state of the iOS app is consistent with the app store metadata. If not, determine that the iOS app to be detected is a shadow app.

2. The dynamic automated detection method for iOS fake packages based on LLM according to claim 1, characterized in that The process of obtaining the automated task list includes: Obtain the user review text according to the app store metadata of the iOS app to be detected; In response to the user review text and the example of the trigger method for the shadow app, the large language model outputs a list of trigger keyword dictionaries; Each trigger keyword dictionary in the list of trigger keyword dictionaries includes a confidence score and a keyword region; Construct and concatenate trigger method short sentences according to the keyword region corresponding to each trigger keyword dictionary, so as to obtain the trigger method for the shadow app corresponding to each trigger keyword dictionary; Judge whether the trigger method for the shadow app depends on the context of the iOS app, and rank the trigger methods for the shadow app that do not depend on the context of the iOS app before the trigger methods for the shadow app that depend on the context of the iOS app. The trigger methods for the shadow app that do not depend on the context of the iOS app or depend on the context of the iOS app are sorted from high to low according to the confidence score respectively, so as to obtain the automated task list.

3. The method for dynamically and automatically detecting iOS fake packages based on LLM according to claim 2, wherein, The keyword region includes: trigger time, trigger geographical condition, trigger waiting time, interaction location, interaction times, text input location, text input content; The trigger method for the shadow app that does not depend on the context of the iOS app represents the trigger method for the shadow app composed of trigger keyword dictionaries with no interaction location, text input location, and text input content.

4. A dynamic automated detection method for iOS clone apps based on LLM according to claim 1, characterized in that The process of obtaining the page summary and the list of interactive elements includes: Run the iOS app to be detected and obtain the current page state; The current page state includes the page screenshot and the view tree of the current page; Perform pruning operations on the view tree of the current page to delete invalid information in the view tree of the current page; perform optical character recognition on the page screenshot and supplement the recognized semantic information to the view tree; classify the leaf elements with dimensions close to a square in the pruned view tree and supplement the icon categories to the view tree to obtain an enhanced view tree. In response to the page screenshot and the enhanced view tree of the current page, the large language model outputs the page summary and the list of interactive elements corresponding to the current page.

5. The dynamic automated detection method for iOS fake packages based on LLM according to claim 1, wherein In response to the page summary, the list of interactive elements, the current automation task, and the historical operation list, the process by which the large language model outputs the target interactive element and the target operation includes: When the target operation output by the large language model is click or continuous click, it also includes setting the number of clicks; when the target operation output by the large language model is text input, it also includes setting the string to be input. When the large language model determines that the current automation task has been completed, set the target operation to task completed and the target interactive element to none. When the large language model determines that no operation is applicable to the current automation task, set the target operation to none and the target interactive element to none.

6. The method for dynamically and automatically detecting iOS alternative packages based on LLM according to claim 1, wherein, Perform the target operation on the iOS application according to the target interactive element to obtain the updated page state of the iOS application. The process of updating the historical operation list includes: Re-obtain the view tree of the current page, and determine whether the view trees corresponding to the original page and the current page have changed. If the similarity between the view trees corresponding to the original page and the current page is lower than the first threshold, the tree edit distance is higher than the second threshold, and the number of times of re-obtaining the current page state of the iOS application does not exceed the third threshold, then re-obtain the current page state of the iOS application, and re-obtain the target interactive element and the target operation. Perform the target operation on the iOS application according to the target interactive element. If the target operation is click, continuous click, text input, or restart, perform the corresponding target operation on the target interactive element; if the target operation is task completed, end the current automation task; if the target operation is none, do not perform any operation. After performing the target operation on the iOS application according to the target interactive element, add a triple to the historical operation list: <page summary, target interactive element, perform target operation>, thereby updating the historical operation list.

7. A dynamic automated detection method for iOS alternative packages based on LLM according to claim 1, characterized in that, Input the updated page state of the iOS application and the app store metadata into the large language model. The process of determining whether the page state of the iOS application is consistent with the app store metadata includes: Calculate the hash value of the view tree in the updated page state of the iOS application and determine whether the updated page state is a newly emerged page state. If the page state after update is a newly emerged page state, input the page state after the iOS application update and the app store metadata into the large language model to identify whether the functions of the iOS application are consistent with the app store metadata. If they are inconsistent, determine that the iOS application to be detected is a shadow app; if they are consistent and the current automated task has been completed and the automated task list has not been traversed completely, clear the historical operation list, reselect a task as the current automated task, and conduct dynamic testing on the iOS application to be detected according to the reselected current automated task; if they are consistent and the current automated task has not been completed and the iteration count has not reached the upper limit, re-obtain the current page state of the iOS application and continue to conduct dynamic testing on the iOS application to be detected; if they are consistent and the current automated task has not been completed but the iteration count has reached the upper limit, determine that the iOS application to be detected is not a shadow app; if they are consistent and the automated task list has been traversed completely, determine that the iOS application to be detected is not a shadow app.

8. An electronic device, comprising a memory and a processor, characterized in that, The memory is coupled to the processor; wherein, the memory is used to store program data, and the processor is used to execute the program data to implement the method for dynamically and automatically detecting iOS shadow apps based on LLM according to any one of claims 1-7 above.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method for dynamically and automatically detecting iOS shadow apps based on LLM according to any one of claims 1-7 above.

10. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the method for dynamically and automatically detecting iOS shadow apps based on LLM according to any one of claims 1-7 above.

Images