Policy violation detection using automated mobile software application exploration

Abstract

Methods, systems, and apparatus, including computer-readable storage media for an automated policy violation determination system for software applications, including mobile software applications. A software application exploration system receives a view hierarchy containing data about the structure and organization of an application, and screenshots of a software application as input to a trained artificial intelligence (Al) model. The model ranks different sequences of inputs referred to at times as trajectories, according to how well the trajectory accomplishes a particular activity specified during model training. An example activity is to explore the software application by navigating to as many unique screens as possible within a predetermined number of actions or wall-clock time. Aspects of the disclosure provide for object hashing and comparison according to different thresholds for identifying like objects in an application at various points in time, to reduce or eliminate cycles of the same screens of the application during exploration.

Description

GOOGLE-4244POLICY VIOLATION DETECTION USING AUTOMATED MOBILE SOFTWARE APPLICATION EXPLORATION BACKGROUND

[0001] A software application explorer or crawler is a type of software built to automatically explore or “crawl” through screens or pages of a software application. A software application may be made of a number of pages or screens, with each page or screen containing elements, such as digital content in the form of text, images, video, audio, and so on, and / or user-interactable elements, such as text fields, buttons, and drop-down menus. Pages or screens can have links or digital content that cause the application to render for display different pages or screens depending on the links or objects accessed.

[0002] A web application, such as one hosted as a web page accessible over the internet, may have clearly defined pages, indicated by relative links in a sitemap indicating how each page may be accessed from one or more other pages. Other types of software applications, such as applications for mobile devices, do not have clearly-defined sitemaps or other data structures that are reliable indicators of content or functionality among various screens accessible within the application. This unreliability makes mobile application exploration more error-prone and less reliable to automate versus traditional web-based application crawling.

[0003] Software applications enable various types of content, such as advertisements, to be presented while users interact with the application. Fraudulent or ill-intentioned software developers may implement advertising monetization on their applications by tricking users into unintentionally interacting with advertisements or other content in an application. A software application store platform or other repository for downloading software applications may have written policies outlining or defining this or other prohibited behavior. Other policies may be enforced by publishers or monetization networks who provide and / or monetize digital content, such as advertisements, for display on screens of a software application while interacted with by a user. Some developers implement cloaking techniques for evading detection systems implemented by software application platforms or repositories. As a result, manual review is needed to confirm whether a software application is in violation of a policy. Automated detection methods for detecting cloaking techniques are often out-of-date and unable to keep up with rapidly changing strategies implemented by malicious application developers or publishers.GOOGLE-4244BRIEF SUMMARY

[0004] Aspects of the disclosure are directed to an automated policy violation determination system for software applications, including mobile software applications. A software application exploration system receives a view hierarchy containing data about the structure and organization of an application, and screens of a software application as input to a trained artificial intelligence (Al) model. The model generates and ranks different sequences of inputs referred to at times as trajectories, according to how well the trajectory accomplishes a particular activity specified during model training. An example activity is to explore the software application by navigating to as many unique screens as possible within a predetermined number of actions or wall-clock time. By training the Al model to explore the software application with this example activity, more of the application can be recorded for automated analysis to be provided to the system for policy violation determination. Aspects of the disclosure also provide for element hashing and comparison according to different thresholds for identifying like elements in an application at various points in time to reduce or eliminate cyclic exploration of the same screens of the application during automated exploration.

[0005] The output of the exploration can be recorded as images or videos and provided to an evaluation framework to predict whether the software application is designed to cause users to inadvertently or unintentionally interact with certain content in the application, such as advertisements, in violation of an application platform policy. Data traffic that typically does not represent genuine user interest or intent can be identified and used to determine a policy violation. The evaluation framework can implement the same or a different Al model, which can be a multimodal model trained to receive the images or video of an automated exploration of the software application, as well as one or more policy documents defining or providing examples of application behavior that is prohibited on a store or other repository providing software applications for download. Other implementations of this aspect include corresponding computer systems, apparatus, computer-readable storage media, and computer programs products recorded on one or more computer storage devices, each configured to perform the actions of the methods.BRIEF DESCRIPTION OF THE DRAWINGS

[0006] FIG. 1 is a block diagram of an example policy violation determination system using a software application exploration system, according to aspects of the disclosure.GOOGLE-4244

[0007] FIG. 2 is a block diagram of the example software application exploration system, according to aspects of the disclosure.

[0008] FIG. 3 is a block diagram of an example action-ranking model, according to aspects of the disclosure.

[0009] FIG. 4 is a block diagram of an example exploration step by an example exploration engine, according to aspects of the disclosure.

[0010] FIG. 5 is a diagram of an example graph representing exploration through a software application, according to aspects of the disclosure.

[0011] FIG. 6 is a diagram of an example evaluation framework for policy violation determination, according to aspects of the disclosure.

[0012] FIG. 7 is a flow diagram of an example process for automated software application exploration, according to aspects of the disclosure.

[0013] FIGs. 8 A and 8B are a flow diagram of an example process for exploration deduplication during automated software application exploration, according to aspects of the disclosure.

[0014] FIG. 9 is a flow diagram of an example process for software application policy determination using automated software application exploration, according to aspects of the disclosure.

[0015] FIG. 10 is a block diagram illustrating one or more models, such as for deployment in a datacenter housing one or more hardware accelerators on which the deployed models will execute for policy violation determination and / or software application exploration.

[0016] FIG. 11 is a block diagram of an example computing environment for implementing the policy violation determination system.DETAILED DESCRIPTIONOverview

[0017] Aspects of the disclosure provide a policy violation determination system for determining whether software applications violate policies for their design or use, for example when made available for download on a software application platform or used to publish and / or monetize certain types of digital content. The system is configured to perform automated exploration of software applications, such as mobile software applications, without sitemaps or static links for each software application screen or page. An artificial intelligence (Al) model, referred to as an action-ranking model, is trained on both view hierarchies andGOOGLE-4244screenshots of screens of various mobile software applications. A view hierarchy is a data structure, such as a tree or inverted tree structure, of a mobile software application representing the layout, structure, or functionality of objects of the application. Objects of the view hierarchy are aligned with objects rendered on screens (or “screen objects”) identified by the model and rendered on corresponding screens of the software application. Different classifiers are trained to classify different screens and / or activities that are performed when actions, such as interaction with various objects, are performed on the application during exploration.

[0018] Software application platforms, such as application stores; digital content publishing platforms; digital content monetization platforms, e.g., for providing advertisements in designated screens or portions of screens of a software application; and so on, implement policies prohibiting certain types of application design paradigms or behaviors. Behavioral abuse results in traffic caused by publisher advertisement implementations tricking or deceiving users. For example, applications that are designed with deceptive layouts that cause a user to inadvertently engage with an advertisement or other objects in the application is a form of behavioral abuse. Previous approaches to detecting behavioral abuse required gathering evidence manually, e.g., by an investigator analyzing the suspected application and testing for behavioral abuse. In addition to scaling poorly, manual detection approaches can often be evaded by adversaries by using techniques that cloak the policy-violating behavior. Cloaking can include not showing nefarious implementations when the application is recently published to an application store or repository, or only occasionally showing the nefarious implementations, e.g., one out of every ten impressions. Cloaking can also occur intermittently, for example based on the publisher or developer of a software application selectively enabling or disabling policy-violating behavior based on predicting when the target software application is being reviewed for violations.

[0019] The action-ranking model determines sequences of actions, e.g., user inputs that are most likely to be performed by users of the application or inputs which increase encounters of particular screen types during the trajectory. In some examples, candidate sequences of actions, sometimes referred to as candidate trajectories, are ranked based on how many unique screens are accessed when respective actions of the candidate sequences are performed. A screen can refer to objects rendered through a viewport when the software application is running. The viewport can be, for example, the display of a mobile device. Unique screens areGOOGLE-4244screens that have not been encountered during a current exploration, which can be made up of one or more sequences of actions.

[0020] Increasing how many unique screens are explored within a predetermined window or time or number of actions results in explorations that are considered to emulate a user’s interaction with the software application. By emulating user behavior to explore and use an application naturally, the system can generate output images or videos screens rendered while executing the highest-ranked trajectory of actions, which can be provided as input to an evaluation framework. The evaluation framework of the policy violation determination system can receive the video or images of an automated mobile application exploration, and classify objects shown in the output. The policy determination model classifies objects depicted in the exploration of the software application and is trained to determine whether the objects or actions taken on the objects are indicative of policy violations.

[0021] The determination output by the model can be, for example, an annotated video identifying possible violations, a text or other data output summarizing possible violations, or a combination of the preceding. The determination output can be used in downstream processes, e.g., for moderating a software application platform, banning or punishing publishers and application developers of the platform that violate the platform policy, and so on. Some policy violations may not be intentional or done with malicious intent. In some examples, courses of actions may include corrective or informative actions, such as to provide guidance to publishers or developers as to why their software applications are in violation of policy, and possible ways to correct the behavior. The policy violation determination system may be configured with predetermined explanations and potential corrective actions to suggest, based on the type of policy violation determined. Output determinations may be used as evidence to facilitate manual or automatic review of potential violations on a case-by-case basis. The evaluation framework can implement a multimodal policy determination Al model to receive the images or video as input, along with policy documents indicating or providing examples of prohibited behavior for software applications published on a store platform or other repository.

[0022] The policy violation determination model can receive, as input, policy documents as part of a prompt for identifying classified objects that have been interacted with in the output exploration in policy-violating manner. The prompt can also include a prompt prefix, system message, metadata, and so on, which may include instructions to the model forGOOGLE-4244determining the task it is being prompted to perform, or the style or nature of the response expected in response to the prompt. The policy documents can include examples or guidelines as to what violates a given policy, such as exit buttons for advertisements that change location when the screen is updated. Another example may be an advertisement that does not clearly distinguish itself from the native digital content or other objects of the application. The output determination can be used, for example, to prohibit the software application from being available for download or installation.

[0023] Policy documents can also be used as input for fine-tuning an existing model, such as a large language model. The fine-tuning process can receive labeled policy documents or subdocuments. The labels can correspond to the policy violation described or indicated in the corresponding document or subdocument. Various different techniques for fine-tuning can be applied, such as low rank adaptation (LoRA) and its variants, can be used for fine-tuning a policy violation determination model.

[0024] Because the view hierarchy may contain missing or noisy data, and view hierarchies and screenshots may not always be captured at the same time, the action-ranking model is trained to align attributes of an object indicated in a view hierarchy with objects identified in screenshot data. Alignment can refer to associating objects represented in a view hierarchy with a corresponding object rendered for display as part of a screen. Related objects, e.g., objects that are present in a view hierarchy and also rendered on a screen of the software application, can be indicated by features or attributes in the aligned data. For example, a view hierarchy may indicate a button object with attributes characterizing the function of the button when receiving user input. The button object may be rendered for display on a screen of the user interface of an application, and aligning the objects can refer to generating an association between the button object in the view hierarchy, with the button object rendered on the screen. Screens also may not always indicate the functionality of objects visible in a depicted screen, making it unclear how the application may transition to a different screen based on action taken on the current screen.

[0025] Some existing techniques may record user inputs for later replaying how a policy violation may have occurred, which is invasive and adds substantial overhead in the form of additional processing to record the actions, as well as storage requirements for storing any metadata associated with the recording. Given that software applications are examined some time after a violation may have occurred, aligning available view hierarchies with screensGOOGLE-4244that can be captured of an application’s user interface allows for the gathering of signals that can be used to determine available actions a user may have taken at the time of a violation.

[0026] By aligning the objects in both a view hierarchy and a screenshot during training, inconsistencies or errors from one can be compensated by the other and provide more input data overall for determining both the layout and functionality of each object. Aspects of the disclosure are less invasive and less computationally demanding than recording user activity for replay, e.g., because the processes described herein do not require recording user activity, and need only be performed on select applications, as opposed to recording user activity for each application of a platform for possible policy violation determination. By reducing errors between the separate modalities of screen data and view hierarchy, the overall output of the rest of the model downstream can be more accurate.

[0027] In the context of the action-ranking model’s output, more accurate output can be output explorations in the form of videos or images that more accurately emulate user behavior. Higher accuracy reduces the chance of inconclusive or erroneous determinations. As the output determinations may be used as evidence in potentially banning or punishing malicious developers, false positive results may be particularly harmful. Even if the policy determination is tuned to weight against positive determinations without a higher measure of confidence, e.g., 99 percent certainty, fewer computational resources, e.g., memory bandwidth and processing cycles, may be wasted towards generating inconclusive determinations.

[0028] Mobile software applications are harder to automatically explore than web applications, for example due to the former’s dynamicity. For example, user modes of interaction can also greatly vary, e.g., as between video games versus utility software applications versus social media applications and so on. Many different formats and native advertisement placement features result in powerful customization, resulting in diverse implementations from application to application. This makes automation more difficult than in standard web crawls.

[0029] The scores and ranking of the action-ranking model can be combined with weights based on the output of different screen-level and object-level classifiers to modify the ranking of a trajectory based on classifications of screen objects corresponding to behavior to be emulated by the system performing a given activity, such as logging in, exiting the application, and so on. Screen-level classification can refer to the classification of different objects of a software application based on screens of different objects of a software application.GOOGLE-4244Object-level classification can refer to classifying objects of a software application based on attributes for the objects in a view hierarchy for the software application. Screen-level classification can be used to identify the location of various objects of an application within a screen, e.g., the location of various user-interactable objects. Object-level classification can be used for determining which actions to perform next in an exploration. For example, in ranking trajectories based on reaching more unique screens, determinations from classifiers of log-in screens or screens that close an application or otherwise lead to screens external to the application can be used to weight the trajectory scores negatively, as reflective of behavior that does not advance the given exploration activity.

[0030] In addition, the action-ranking model can implement classifiers for identifying cloaking or adversarial design or layouts in software applications, the presence of which can be used to weight the overall score of a trajectory for exposing these behaviors in the output exploration that is provided to the policy determination model. Example screen or screen objects that can be classified include login screens, error screens, and loading screens, all three of which may include objects that, when rendered, circumvent or attempt to cloak application behavior that is prohibited by the policy of a software application platform. The additional classifiers improve the overall accuracy of generating an output exploration that emulates user behavior. This is at least because individual classifiers can identify predetermined types of policy-violating design, to expose that design through actions taken emulating how a user may be tricked or inadvertently caused to interact with the application. To that end, determining the presence of policy-violating behavior in a software application is increased, as malicious developers design software applications with user behavior in mind. The removal of policyviolating applications improves the usability of a software application hosting platform overall, at least because users may download applications with a higher degree of confidence that the applications are designed within the parameters of the policy for the platform.

[0031] The policy determination model is trained to break up policy documents into separate subdocuments as part of multiple generated multiple prompts to the model focusing on different parts of the input policy documents and whether the output exploration indicates a violation has occurred. Breaking down a policy document into multiple subdocuments can improve the performance of the prompt processing, at least because there is a smaller chance of misunderstanding by the model versus ingesting and comprehending the entire policy document. The policy determination model can output determinations as to whether or not theGOOGLE-4244output exploration depicts one or more policy violations. The determinations can indicate, which, if any, specific policy violations are detected, which can be provided to a downstream process, for example for remediation or for collecting evidence in support of banning a developer of the software application from the application platform.

[0032] By interfacing through prompts and receiving input policy documents, the policy determination model can generalize to detect multiple abuse vectors, e.g., multiple application layouts and designs that violate application platform policies. The generalizable design of the policy determination model reduces lag time between the detection of new cloaking techniques and updating policy documents, which can be processed by the model to be factored in future policy violation determinations. Separate classifiers can be added to determine whether specific activities are present in an output exploration video, the output classifications can be factored into the overall determination of a policy violation. These classifiers may be separated from classifiers of the application exploration system, for example because their focus is on specific use cases of processing output explorations, e.g., for policy violation, but are not needed in all use cases. For example, other classifiers may be added to process an output exploration of an application for other use cases, such as application testing, debugging or generating videos or images of application explorations for other purposes, such as creating training videos or demonstrations. Separating classifiers by use case can improve application exploration, for example by reducing the number of classifications performed, thereby reducing the computational costs, e.g., measured in computing cycles or memory bandwidth, for exploring a software application. Some classifications may be more computeheavy, e.g., processing over multimedia or calling various generative models that may perform more efficiently offline, after the exploration has finished.

[0033] Aspects of the disclosure provide for object hashing and comparison according to different thresholds for identifying like objects in an application at various points in time, to reduce or eliminate cyclic exploration of the same screens of the application. Classified objects can be grouped to mitigate or avoid errors during application exploration, using custom object hashing and pairwise screen and object comparison mechanisms. For example, objects of an application may shift by a few pixels or more between different screens, even though the objects may substantively be the same. This shift may be imperceptible, but enough to cause the same object to be classified as a unique object. Duplicate explorations or repeatedly reaching the same screens through the same actions results in wasted computing resources, e.g.,GOOGLE-4244more processing cycles, memory bandwidth, and network traffic utilized redundantly. By identifying and avoiding duplicate explorations, the amount of wasted computing resources is reduced, which improves the performance of the system overall, for example because explorations can finish faster and more explorations of the same or different software applications can be performed in the same amount of time. Manual review of output explorations can also be performed in less time, for example because cyclic exploration is avoided and accordingly not recorded in the output video or images.

[0034] To avoid duplicate explorations, e.g., explorations of a software application in which the same screens are visited using the same actions, aspects of the disclosure provide for an exploration-time construction of an application transition graph for tracking and eliminating duplicate screen visits, as well as storing historical exploration data that can be accessed to improve future explorations into unique screens not previously explored. The system allows for exploration of code-identical but content-unique screens of a software application, versus previous approaches relying on static code or network traffic analysis alone. A code-identical but content-unique page can be a product listings page, in which products change periodically but the overall code implementation of the page remains the same regardless of the products listed. The system can determine that two screens are the same, even if the content in the screen is different. For example, a dynamic, image-heavy homepage may look completely different even within the same exploration but should be recognized as the same screen for purposes of exploration. The system can generate an embedding or feature vector over which a distance metric can be computed. The embedding can include attributes corresponding to one or both of objects in the view hierarchy or screenshot for a current screen being explored.

[0035] The system can compare one or more attributes to determine whether a current screen matches a previous screen encountered and recorded in the application transition graph. For example, to determine whether a screen is the same as a previously encountered screen with different content or other objects, the system may compare only attributes related to structure or the layout of the screens, and not for content. By defining different thresholds for comparison, the system can determine identical matches, content-aware partial matches, content-agnostic partial matches, or structural partial matches, all of which can be used to determine whether a screen has been previously encountered during exploration. In the event of a duplicate, the system can add certain transitions from certain screens to a blocklist by aGOOGLE-4244controller executing an exploration, to prevent the controller from entering a cycle during exploration.

[0036] In addition, utility controllers can be implemented during application exploration to facilitate performing certain activities that may block or inhibit automated exploration. These utility controllers can be swapped in to operate when a core controller, e.g., a controller performing exploration according to a ranked trajectory, becomes blocked, for example due to an error message or log-in window. Utility controllers can be specifically trained for handling these blocks, before control is handed back to the core controller to continue exploration.Example Systems

[0037] FIG. 1 is a block diagram of an example policy violation determination system 100 using a software application exploration system 105, according to aspects of the disclosure. Software applications 110 can be hosted on software application platform 115, which may have a user-facing storefront or other interface for downloading applications. In some examples, software application platform 115 is a platform for publishing and / or monetizing digital content to different software applications. In those examples, the software applications 110 may be hosted by the platform 115 or by a different platform or system. Also in those examples, the software applications 110 may be provided from their source to the system 105 for exploration and to the system 100 for policy violation determination.

[0038] Software applications can be identified for exploration, for example as part of a screening process when the applications are offered for installation on the software application platform 115. For example, software applications may be selected randomly for exploration or based on a predetermined risk level. In some examples, all software applications made available for download may be selected for exploration. Any type of software application in which a view hierarchy can be generated can be explored, such as most mobile software applications. View hierarchy 192 and one or more screens 196 can be provided as input to the system 105.

[0039] Exploration navigation can be handled by the software application exploration system 105, in which sequences of actions (also referred to as trajectories) are performed for generating exploration images or video 120. Example actions can include launching intents of an application or relaunching the application itself; targeting a specific user- interactable object, e.g., by using a tap, long press, swipe (e.g., up, down, left, right, and / or at a given percentageGOOGLE-4244of the current view); editing or adding text; waiting for a specified duration; going back to an earlier view; a two- or multiple-point gesture, such as pinch to zoom; and so on. The exploration images or video 120 can be screen recordings of a viewport of a virtual or physical environment executing sequences of actions. As described in more detail with respect to FIG. 3, the software exploration system 105 generates and ranks different candidate trajectories and determines the highest-ranked trajectory for exploring unique screens within a software application.

[0040] The exploration images or video 120 are processed by the image or video processing engine 125 and the evaluation framework 130 for generating policy violation determination 135. Although not shown in FIG. 1, other possible outputs of the exploration system 105 include screen metadata or other types of metadata, screen text, and / or an application transition graph, for example as shown and described with reference to FIG. 5. The policy violation determination 135 can be input to a downstream process (not shown) for collecting evidence or taking action to suspend or moderate a software application indicated as violating a policy. The downstream process can involve manual or automated confirmation of the policy violation, for example before action is taken to remove the software application from the platform.

[0041] Policy document 140 can be one or more documents or data files describing or defining policies to be enforced by software applications hosted on the software application platform 115. The policy can be written in structured mark-up language, natural language, computer code, or a combination of the preceding. Enforcing policies improves the function of a platform in providing software applications that are consistent and unambiguous in behavior. From a user perspective, platforms that enforce these qualities of software applications are more reliable as resources for applications that may be employed in various user-driven use cases, such as data processing. Policies can enforce, for example, clear user-interface practice, in which applications do not attempt to obscure or hide unintentional interaction with certain digital content on applications, like advertisements. A policy may specify that advertisements be of a certain size or opacity, with clear and consistent user-interactable objects for exiting or removing the advertisement from display.

[0042] Policy document 140 can be updated routinely, for example in response to new application design paradigms that may violate the spirit of the types of behavior a policy is seeking to eliminate from the software application platform 115, but which may not be explicitly stated. As described with reference to FIG. 6, the evaluation framework canGOOGLE-4244implement multimodal evaluators 132, such as trained and fine-tuned Al models, for dividing the policy document 140 into a number of policy subdocuments. The policy subdocuments can break down a larger, more complex, policy into more discrete conditions, which have been found to result in more accurate determination overall of policy violations, versus processing the policy document 140 as a whole. Improved accuracy in policy violation determination can result in more efficient processing overall, e.g., through fewer wasted processing cycles in generating false positives, which are also detrimental in maintaining the reliability — and consequently the future use — of the software application platform 115 by application developers or publishers.

[0043] The output exploration images or video 120 (sometimes, “output exploration 120”) are processed by the image or video processing engine 125 to generate annotated images or video 127. The annotations correspond to where certain types of digital content items, such as advertisements, are visible in screens depicted in the output exploration 120. The image or video processing engine 125 can implement one or more Al models or other techniques for classifying certain types of digital content, which may be related to the policy document 140. For example, the policy document 140 may specify certain acceptable ways of interacting with advertisements that may be displayed in the course of using the application.

[0044] FIG. 2 is a block diagram of the example software application exploration system 105, according to aspects of the disclosure. Exploration engine 205 is configured to explore software application 110 by executing actions of different trajectories. A trained actionranking model 300 provides a scoring for each trajectory and then the exploration engine 215 can re-rank or adjust the trajectories based on other priorities or prioritized characteristics. The exploration engine 205 can load and interact with software application 110 in simulation environment 215.

[0045] Simulation environment 215 can be made of one or more virtual or physical devices, such as a virtual machine configured to launch and run a software application, or a physical device communicatively coupled with the exploration engine 205 and configured to do the same. The exploration engine can be communicatively coupled to an interactive console 230. The interactive console 230 can be configured to allow for manual or automated input to be inserted during automated exploration by the exploration engine 205 in the simulation environment 215.GOOGLE-4244

[0046] The application exploration system 105 supports multiple exploration modes, including full exploration mode and deeplink exploration mode, as well as heavily bespoke exploration modes which may be designed for particular policies. The software application exploration system 105 can be configured to support alternative modes of exploration, through custom-defined parameters. The modes described herein can be performed automatically, and in some examples, can be performed in a hybrid mode with manual intervention, for example to complete captchas present in the software application 110. The exploration can be, for example, a full exploration of all screens in the application, and / or a deep link exploration for capturing screens shown when opening various deep links in the application. Moreover, exploration may track ads encountered. When compared to log data related to the serving of the ads, policy violations, such as dead-end ads, which do not provide a way to close the ad, may be identified. Ad encounters can also indicate how well the exploration system explored an application, as ad encounters often serve as a signal for portions of the application which should be checked for ad policy compliance.

[0047] In full exploration mode, the software application exploration system 105 automatically explores the software application 110 in a manner predicted to be most like how a user would interact with the software application. The software application exploration system 105 automatically performs any needed one-time application set-up, such as personalization, selecting user preferences, completing a tutorial, logging in or bypassing a login, and collecting information about the behavior of the software application 110 during the exploration. What information is collected or prioritized can depend on the specific objective functions used to train an action-ranking model 300 corresponding to the system. The actionranking model 300 can be trained to cause the software application exploration system 105 to explore the software application with different priorities, e.g., accessing unique screens of the software application, utilizing search fields, performing a text search, browsing and selecting from a list of articles or product listings, and so on. The software application exploration system 105 can be configured to perform the exploration within a predetermined window of time, e.g., minutes or an hour at a time.

[0048] The application exploration system 105 can also crawl through deeplinked applications, such as deeplinked applications 225. As described herein, software applications such as mobile applications generally do not have URLs or a sitemap for identifying the location of a given screen relative to another screen. While deep links may be used, theirGOOGLE-4244adoption is low, with inconsistent coverage and unpredictable behavior. For example, a deep link may point to a different page depending on whether a user of the application is logged in or not, or the page itself is dynamic. The same issue can arise for elements on a view. For example, element identifiers are not unique and may change by application version, by crawl, or simply refreshing the page.

[0049] In deep link crawl mode, the software application exploration system 105 is configured to explore the software application 110 for capturing the content of the real deep link landing page. Deep Linking initially can be used to leverage web page versions of mobile application deeplinked pages. A deep link is a link in a software application to another screen in the software application, which often has a web page counterpart that can be used for comparison with web-based crawlers that do not rely on a view hierarchy. If the software application is not appropriately set up, the deeplink may fail to launch or point to the wrong destination, e.g., a setup page. The system can run in full exploration mode to get past these and any other bootstrapping walls. The system can then launch one or more deep links in succession and capture the content contained within each deeplinked page. The system can scroll to uncover content that is not initially visible in a viewport displaying screens of the software application.

[0050] The exploration engine 205 can receive adversarial configurations 235 classified by invalid traffic classifiers 240. The adversarial configurations 242 can be appended to objects in screens of an application transition graph, for example as shown and described with reference to FIG. 5. An adversarial configuration can be, for example, an arrangement to the structure or layout of an application intended to be implemented as a cloaking technique for the application to avoid policy violation determination.

[0051] Data related to explorations can be stored in storage device 235. Explorations can be incrementally stored, e.g., with changes after each action taken or response provided, and used for optionally re-running previous explorations and / or labeling for training data for the action-ranking model 300, described in more detail with reference to FIG. 3.

[0052] FIG. 3 is a block diagram of the example action-ranking model 300, according to aspects of the disclosure. The action-ranking model 300 is trained using training data 395 including a view hierarchy of a subject software application and images of different screens of the software application. The action-ranking model 300 architecture and training can be based on a two-tower model approach. An alignment model 310 of the action-ranking model 300 isGOOGLE-4244trained to align view hierarchy data with screenshot data. A classification model 320 of the action-ranking model 300 is trained to classify various responses predicted to occur based on certain actions performed and from the aligned view hierarchy and screen data. The overall output of the action-ranking model is a score of the quality of one or more actions taken in exploring a software application.

[0053] Alignment can refer to associating objects of the view hierarchy with screens that are displayed when the software application is displaying a screen or screens corresponding to the parts of the view hierarchy. Objects of the view hierarchy can specify user interactable elements, such as buttons, toggles, drop-down menus, and so on. Elements of the view hierarchy can also specify elements that are not interactable, but which may be displayed while the software application is in use, such as static text or images. An association can be a mapping between an element and the corresponding element displayed on a screen, e.g., view hierarchy code for a button can be mapped to a portion of a screenshot displaying the button.

[0054] A view hierarchy can be available for most mobile software applications, for example generated manually or automatically during the design and / or development of a mobile software application. Some mobile software applications may not have view hierarchies or may not have complete view hierarchies to cover the various objects of the application or their associated functionalities or attributes. For example, some mobile application games may not have view hierarchies because updated screens are rendered dynamically on a pixel-by-pixel basis in accordance with predetermined event logic, as opposed to rendering more relatively static pages based on receiving input to certain objects for causing the screens to transition from one to another.

[0055] The action-ranking model 300 can be based on a multi-headed attention-based transformer model architecture, with classification heads 325 trained to penalize bad actions, e.g., actions that do not advance the given activity, and ranking heads 328 that are trained using an objective function to increase scores for trajectories of actions that advance the given activity. The alignment model 310 can be trained using a supervised learning approach, with pre-labeled aligned view hierarchy and screens. An objective function can be to measure the similarity between the labeled aligned data and predicted alignments output by the alignment model 310. B ackpropagation with gradient descent can be used to update model parameter values for improving the alignment by increasing the similarity measured using the objectiveGOOGLE-4244function. The objective function can, for example, measure cross-entropy loss, LI or L2 loss, or be another suitable function.

[0056] Alignment before classification provided by the alignment model 310 and the classification model 320 of the action-ranking model 300 improves accuracy versus training a model to perform the various different classifications without first aligning the view hierarchy and the screenshot data. Higher accuracy models can perform more efficiently overall, by not wasting processing cycles or power to re-execute the model when confidence scores for the model accuracy is low. Higher accuracy reduces the chance of wasted computation downstream of the model output that may otherwise cause a cascade of errors stemming from inaccurate model output.

[0057] The classification model 320 can receive the aligned view hierarchy and screens 315 of the alignment model 310 for classifying whether providing some type of input to the software application will cause a particular result to occur. An input to a user-interactable object is an example of an action that can be taken. An output to the action can be some response from the software application based on receiving the input, of which there may be more than one and to more than one user- interactable object. An example action can be pressing a button on the software application, with an output in response being that the software application renders a new screen for display.

[0058] The view hierarchy can be organized as a collection of trees (also referred to as a “forest”) or other graph data structures. The view hierarchy can at least partially characterize a collection of objects of a user interface that may be displayed through a viewport, such as a mobile device display. The view hierarchy can be representative of a single screen, as well as various screens and transitions between the screens. For example, the view hierarchy can include text content for different elements making up a view, class names or internal identifiers for various elements, developer-provided descriptions, boundaries of various elements in a view, and element capabilities, such as being configured to receive touch tap input, swipe input, long presses, input text, and so on. A screen can capture at least a portion of a screen of a software application. The screenshot can be the output displayed by a viewport of a physical or virtual machine running the software application. Screens can be in any format suitable for image data, e.g., a PNG file.

[0059] The classification model 320 includes screen-level classifiers 322 and objectlevel classifiers 324 that are trained to perform different binary classifications and / or multi-GOOGLE-4244class to predict outputs to actions provided to the software application. Screen-level classification can refer to classification of different objects of a software application based on rendered screens of different objects of a software application. Object-level classification can refer to classifying objects of a software application based on data in a view hierarchy for the software application. Screen-level classification can be used to identify the location of various objects of an application within a screen, e.g., the location of various user-interactable objects. Object-level classification can be used for determining which actions to perform next in an exploration. For example, in ranking trajectories based on reaching more unique screens, determinations from classifiers of log-in screens or screens that close an application can be used to weight the trajectory scores negatively, as reflective of behavior that does not advance the given exploration activity.

[0060] Example binary classifications can include whether or not an action will keep the screen the same, e.g., no transition to a new screen of the software application, or whether the action will close or minimize the software application. Example multi-class classifications can predict the quality of a screen reached by performing an action. Quality can be assessed based on predetermined classes of responses that are predetermined to be of low quality, e.g., causing the application to close to minimize, reaching a login screen or a loading screen, and so on. Responses that are classified but not of the predetermined low-quality classes of responses can be assessed as high or higher quality by the classification model 320. High quality classes can refer to, for example, actions that cause responses in the form of core content of the software application, while a low quality class culd include a page which only includes a sign-up or login form.

[0061] Screens that are not classified as one of the predetermined low-quality classes and that are not duplicates of other screens are referred herein as unique screens. The intuition of valuing unique good screens over other types of screens that are in response to various actions performed is that in exploring a software application, more unique screens mean more of the software application was explored overall. Given that trajectories are finite, rewarding actions that lead to unique good screens correspond to using the trajectories more efficiently to explore the software application in a manner that mirrors how a user may interact with the software application.

[0062] Various other classifiers, e.g., binary classifiers for determining whether an action leads to closing or minimizing the software application, can be used as a filter inGOOGLE-4244predicting higher versus lower quality responses. Quality classification can be numerical, e.g., ascribing a numerical output to the likelihood an action leads to an undesired response, e.g., closing the application, or not. For example, one binary classifier can be to classify a screen as being a login screen, e.g., containing a username and password field for logging into the software application. A login screen classifier may be used to weight the overall score of the trajectory that leads to a login screen. The value of the weight can depend on the activity sought to be emulated. For example, if the activity of the system is to explore new screens, then a trajectory leading to a login screen may be weighted to a lower value, to reflect that the exploration cannot continue to find new screens from this trajectory unless a login is made (and which may not always be available as an option to the exploration engine 205).

[0063] Ranking heads 328 can receive output classifications 317 and the aligned view hierarchy and screens 315 and be configured to predict the quality of candidate trajectories 331 of actions and responses. The actions of the candidate trajectories 331 can correspond to actions that the action-ranking model 300 predicts can be taken from different screens, e.g., different button presses or other inputs to transition between different screens. The length of the trajectories can be measured as a window, e.g., temporally, such as five minutes of exploring the software application; or, measured by the number of screens explored in a trajectory, e.g., seven screens in a window. Training data 395 can be partitioned in windows of the same size as what the action-ranking model 300 is trained to predict. The ranking heads 328 can output a score of a selected trajectory of the candidate trajectories 331 as output data 370, corresponding to the overall quality of the trajectory of actions and responses. The model can decay or weight different scores depending on the number of actions leading to, for example, screens that are not predetermined as being of low quality. One action to a high-quality screen may be weighed more heavily than two or three actions to reach the same screen.

[0064] The score in the output data 370 can be a numerical value, e.g., from zero to one. A score of zero can indicate that none of the actions in a given trajectory led to a unique good screen. On the other hand, a score of one can indicate that every action in a given trajectory led to a unique good screen. The output score can also be accompanied with the binary classifications made as part of processing the input data, as well as data corresponding to the actions of the highest-ranked trajectory for the exploration engine 205 to execute. The objective function for training the classification model 320 can be a measure of similarity between the labeled response and the predicted response, which can be used to improve theGOOGLE-4244similarity in subsequent training iterations through model weight updates. The classifiers 322, 324 can receive aligned view hierarchy data and screens 315, or a combination of the two as input for generating a classification.

[0065] The ranking heads 328 can be trained to generate and rank trajectories emulating different behaviors through received training data 395 including sequences of user actions on similar software applications, which represent a desired behavior to emulate. The objective function can be selected to measure the degree of similarity between a ground-truth sequence of actions, and an output set of actions for exploring a software application according to the behavior of interest.

[0066] The classification heads 325 and the ranking heads 328 can be implemented according to various different ensembles of model architectures, as well as symbolic or rules-based systems for generating a score for one or more actions. The exploration engine 205 can determine which type of classification or ranking head to use based on machine-learned or manually set associations between different classifications and corresponding model architecture. For example, some types of classifications in some contexts, such as determining whether an action leads to a login, a text search bar, a full- screen advertisement, and so on, may lead to a rules-based head for classifying the response to the action.

[0067] The classification model 320 can be trained according to a supervised learning approach, with training data 395 including a current state of a software application, e.g., indicated by aligned view hierarchy and screenshot data, and a current action. The training data 395 can be labeled with the response taken by the software application when the current action is taken. The training data can include trajectories of actions and responses. The exploration engine 205 can record trajectories taken for various different software applications for use as training data 395 for the classification model 320. At inference, e.g., when the action-ranking model 300 is accessed during an exploration step as shown in FIG. 4, inference data 390 can include a view hierarchy and image data for a current screen while the system explores the application, e.g., as shown and described with reference to FIGs. 4 and 5. In some examples, fewer types of data, e.g., only a view hierarchy, may be used. The view hierarchy can be annotated with additional signals generated during preprocessing, e.g., as performed by preprocessing engine 415 of the exploration engine 205.

[0068] The combination of the view hierarchy and the screenshot data supplements model training versus training on screenshot data alone. One way the view hierarchyGOOGLE-4244supplements the screenshot data is by allowing the action-ranking model 300 to contextualize screens by providing indications of what objects are expected to be on a given screen. For example, objects of the view hierarchy can indicate what user-interactable objects may be present in a corresponding screen. This indication reduces or eliminates the need for more computational-intensive image processing, such as annotating screens with bounding boxes for the various displayed elements. Reducing the need for this type of processing allows for faster application exploration overall, at least because fewer processing cycles are needed for processing the inference data 390.

[0069] Combining the view hierarchy and the screen data can also allow for the model 300 to cross-reference between the two types of data, for example to correct and account for errors in the view hierarchy. For example, an object in the view hierarchy may indicate it is interactable in the software application, but the corresponding object when rendered on a screen of the application may end up not be interactable. One reason for this disparity is that the view hierarchy may be manually generated and not up-to-date with application functionality, but regardless provides additional information for mapping the application more accurately overall, when processed in conjunction with screen data.

[0070] Combining classification and action-ranking using both view hierarchies and screens reduces errors or noise that may otherwise be propagated from view hierarchies or screenshots alone, thereby increasing the accuracy of the system in exploring the software application according to different predefined activities. Aligning the view hierarchy and screenshot data allows for covering inconsistencies or ambiguity in either data modality alone. As a result, mobile application exploration automation is enabled, further enabling use cases such as policy determination, but also for other use cases, such as application testing or quality control.

[0071] The action-ranking model 300 can implement rule-based and model-trained engines, for example as screen-level classifiers 322 and / or object-level classifiers 324. Rulebased engines can target specific application development frameworks, e.g., defining various objects of an application and the predetermined behavior of the application when the object receives user input. Model-trained systems can be trained on sequences of application screenshots to predict application behavior when different elements receive user input.

[0072] FIG. 4 is a block diagram of an example exploration step 400 performed by the exploration engine 205, according to aspects of the disclosure. To perform an exploration ofGOOGLE-4244the software application 110, the exploration engine 205 executes streams of send-acknowledge request-responses. The environment 215, e.g., a client device or virtual machine, sends action results 405 and a screen state 410 to the exploration engine 205. The screen state 410 can include a view hierarchy for the software application 110, screen data, e.g., an image of a current screen, as well as any metadata associated with the image. The current screen can be one of screen(s) 196 initially provided to the system 105. The view hierarchy can be view hierarchy 192, also shown with reference to FIG. 1. The exploration engine 205 processes the action results 405 and the screen state 410 to generate next action 490. The environment 215 performs the action output by the exploration engine 205 and sends another action result 405 and a new screen state 410, beginning another cycle. The exploration engine 205 can continue executing the stream as one or more cycles of the above, until reaching a termination condition. A termination condition can be, for example, reaching the end of the predetermined window of time, determining that all deeplinks were explored, or all screens in the software application were reached.

[0073] The exploration engine 205 is configured to store information related to the crawl, e.g., received screen state 410 and the action results 405 of the previous iteration, and performs preprocessing and crawl logic tasks described presently. The preprocessing engine 415 receives the screen state 410 and can perform one or more preprocessing operations to clean up invalid, noisy, or irrelevant signals. For example, preprocessing operations can include removing predetermined unwanted elements captured as part of an image of the current screen, such as the top notification bar or bottom navigation bar of an operating system running the software application 110, invisible or non- visible segments indicated by view hierarchy, advertisements, and so on. Aligned objects in the view hierarchy can also be removed as part of preprocessing. The preprocessing engine 415 also sends the screen state 410 to the actionranking model 300 and appends scores and classifications generated by the action-ranking model 300 to the screen state 410. The preprocessing engine 415 can be configured to append signals to the view hierarchy and / or screens of a software application being explored. Additional preprocessing and corresponding signals that can be added include optical-character recognition (OCR) with recognized text appended to corresponding screens; icon detection, e.g., icons indicative of common functions, such as advancing or retreating from a current screen, closing a window, etc.; labels corresponding to policy / policy versions to apply; andGOOGLE-4244other classification operations and corresponding labels annotated or appended on the view hierarchy and / or screens.

[0074] In addition, the preprocessing engine 415 can mark advertisements in an application as invisible, to prevent interaction during exploration. The text appearing in the software application can also be translated to a target language the action-ranking model 300 is trained to process data in. The preprocessing engine 415 can also mark elements that are too small or too close to the edge of the screen, e.g., based on a predetermined size or distance from the edge, respectively. These marked elements may be excluded from interaction during crawling, for example because they are considered unlikely to be interacted with by a user.

[0075] As described in more detail with respect to FIG. 5, graph engine 420 is configured to incrementally update an application transition graph representing the software application and the current exploration. For example, vertices or nodes of the application transition graph can include screen data encountered so far, while edges or links between nodes can represent the action taken to reach the screen from another screen. The graph engine 420 can perform updates before downstream elements, e.g., agent 425, activity selector 430, and / or activity controller 435 make any decision as to the next action to take. Classifications 317 from the classification model 320 can also be appended to the graph. The classifications 317 can be used to identify which controllers are applicable to which screens, for example because difference controllers 425 as described herein may be used to perform various tasks during application exploration.

[0076] Agent 425 runs at the start of each step in full exploration mode and is configured to determine an activity or action to take based on scores provided by the actionranking model 300. An activity can be represented as a set of tasks. A task is a discrete sequence of actions that a controller can take, which may be repeated over the course of an activity. Examples of tasks that a controller 435 can perform include capturing content at a deep link, progressing through a sequence of loading screens, exploring one product page or one category page from a parent page of multiple products or categories, or one login attempt at a login screen. Breaking the activities into tasks can allow for more granular control in adjusting the exploration, e.g., by setting a minimum or maximum time or number of attempts in performing each task.

[0077] Although the example activity described herein has been to explore unique screens, the exploration engine 205 is configured to perform any of a number of activities on aGOOGLE-4244software application, which may be used for purposes other than policy violation determination, such as application testing or debugging. Other activities that can be performed by activity controllers 435 can emulate various types of user behaviors, such as waiting for a screen to finish loading, relaunching the software application, closing a pop-up notification, and so on.

[0078] To determine an action to take, the agent 425 can weight scores from trajectories output by the action-ranking model 300 based on corresponding classifications for the type of behavior being emulated. For example, classifying an action on a particular screen as causing a pop-up to occur can weigh the action more heavily if the behavior to emulate by the exploration engine 205 is causing a pop-up to occur in response to an action. The value of the weights for each classification can be empirically determined, for example, after ablating different combinations of classifications and comparing the impact on the output exploration, e.g., based on how many unique screens are explored or against other measurements depending on the activity being emulated. In some examples, the weights are set uniformly for various classifications used to weight the scores.

[0079] The agent 435 can weight trajectory scores according to other factors that may be specific to a particular implementation of the exploration system 100 and / or specific to the exploration of a particular software application. For example, certain actions might be associated with certain latencies to perform, affecting the speed and performance in executing a trajectory to explore an application. The agent 435 may maintain data corresponding to these latencies or other performance characteristics, and use that data to further weight the trajectory scores. To that end, the agent 435 can convert the output of the action-ranking model 300 into a set of scores that factor in time costs or other costs in performing certain trajectories, which the model 300 may not account for as it is agnostic to different computing or simulation environments.

[0080] Activity selector 430 can be configured to determine which controllers to invoke as part of performing an activity. The activity selector 430 can select an appropriate controller of the controllers 435 based on the activity predetermined to perform, e.g., exploring the software application 110 for unique screens. Activities can be divided into two categories, utility activities and core activities. Corresponding controllers of the activity controllers 435 can be assigned to one, both, or other categories of activities. Utility activities can be related to activities for improving the efficiency of core activities. Example utility activities includeGOOGLE-4244unlocking and reaching screens, e.g., behind login screens or other barriers or screening certain types of content, including login, loading, popup dismissal, or out-of-application screens. Core activities can be related to exploring for capture unique screens or other types of content, such as through deep link capture, parent-detail, and / or text-search.

[0081] The activity selector 430 can determine whether the controllers will perform actions based on a (potentially agent re-weighted) score of a trajectory, or if actions will be taken as determined by a corresponding controller. Multiple controllers may be configured to explore the software application 110 according to the same or different activity. Controllers for core activities may be preempted by controllers for utility activities, for example, when the core controllers are stuck behind a login screen. The exploration engine 205 can temporarily run a utility controller for navigating past a login screen, e.g., by providing an appropriate username and password, before passing control back to the core controller to continue application exploration.

[0082] Controllers can be configured with their classifiers and ranking mechanisms, trained more narrowly on specific types of activities, e.g., to output trajectories that are ranked based on how efficiently a utility activity, such as navigating past a login screen, is performed. Efficiency can be measured, for example based on the number of actions in a trajectory to perform a certain activity, and / or based on temporal metrics, e.g., the wall-clock time needed to perform the activity. The controllers 435 can be trained using a supervised learning technique, based on updating model parameter values for classifiers and other Al models implemented by the controllers 435, after backpropagating the error between output actions from a controller relative to ground-truth trajectories of actions for performing a given activity.

[0083] The activity selector 430 can compare scores from trajectories output from the action-ranking model 300, with scores output from the various controllers 435 and select which trajectory to perform based on the scores. In some cases, the activity selector 430 may preempt actions suggested by the output from the model 300 for actions by a controller. The determination allows for generalizability in exploration by the model 300, e.g., for exploring new and varied applications, with performing certain common sequences of actions, e.g., passing a login page, more efficiently.

[0084] The activity controllers 435 are configured to cause the selected action to be taken on the software application 110 in the environment 215. Interactive console 230 can be configured to allow for manual input, for example in case an override to an action takenGOOGLE-4244automatically is needed for testing or debugging purposes. There may be multiple activity controllers for the same activity, or a single controller per activity.

[0085] The activity selector 435 can determine which type of classification or ranking head to use based on machine-learned or manually set associations between different classifications and corresponding model architecture. Based on the activity to emulate, the activity selector can select which classification heads and / or ranking heads to use for classification and ranking, respectively. As described with reference to FIG. 3, different classification heads can be trained for classifying different objects and screens, and different ranking heads can rank candidate trajectories differently based on the activity targeted for emulation. The various heads can be associated with different core or utility activities, and the activity selector 430 can determine the appropriate heads based on a computed priority score for each head.

[0086] To compute a priority score, the activity selector 430 can receive an importance score for each utility activity towards completing a core activity, e.g., performing a login for an application that requires a login, as part of exploring the application for finding unique screens. The activity selector 430 can also receive a confidence value corresponding to confidence in a particular head for performing an accurate classification for performing a given activity. The activity selector 430 can also receive an expected value of an action performed towards achieving a given core activity when an action selected based on a particular score for a trajectory ranked by a ranking head is performed. For example, one ranking head may generate a score for an output trajectory that has a different value than another output trajectory if another ranking head were used. The various inputs, e.g., the expected values, the confidence values, and the importance scores, can be weighted as part of combining to generate an overall priority score. The weights can be predetermined, based on empirical or statistical analysis of previous explorations for a given activity, or a combination of the two. In some examples, weights are adjusted manually or automatically on a per-exploration basis.

[0087] The exploration engine 205 may explore a software application multiple times, e.g., over a period of several days, weeks, or longer. An output recording, e.g., a video or images, can be taken during exploration and provided to the evaluation framework. Provided that the structure or layout of the software application has not changed too drastically, e.g., from an update, or if the content is highly dynamic, such as in a news application, the information contained in previous explorations can be used to facilitate future explorations. ForGOOGLE-4244example, action memory is used and stored by the system 205 as a global record of actions and results of those actions, from previous explorations. The trajectory scores of each action may be stored, and the system can compare the trajectory scores to determine which actions ranked lower than others. This scorekeeping and comparison can cause future explorations to avoid lower-valued trajectories, e.g., ones that do not lead to new screens, or lead out of the software application.

[0088] FIG. 5 is a diagram of an example application transition graph 500 according to aspects of the disclosure. Arrows between screens indicate actions 508A-508E taken and can have a corresponding trajectory score associated with the scores of actions taken to arrive at that action. Each screen 502A-502D has a corresponding screen label 504A-504D identifying the screen. Each screen also includes a respective screen score 506A-D, indicating the score of the trajectory of actions leading up to the given screen.

[0089] The graph engine 420 of the exploration engine 205 can generate a graph such as the graph 500 for tracking the current state of an exploration, past decision-making, and all available paths. Tracking current and past actions through a graph also enables the exploration engine to navigate to other screens if there are no further actions on the current screen or determine shorter trajectories for reaching the same screen. The graph 500 can be a data structure representing the application state as described above and may be updated once for each iteration of exploration. As described in more detail with reference to FIG. 4, the exploration occurs over a series of steps. By updating the graph 500 once per iteration, the graph is kept immutable while running the exploration, reducing potential errors from mismatched versions of the graph 500.

[0090] Actions 508A-508E represent different actions taken during exploration. Some actions lead to new screens, such as action 508A to screen 502B, or action 508D to screen 502C. Other actions may result in exiting or leaving the application, such as action 508C, which does not point to a screen. Some actions can result in looping between multiple screens, such as actions 508F and action 508G going back and forth between screens 502C and 502D.

[0091] Copies of the same screen, e.g., screen copy 510A representing a copy of screen 502A, are stored as part of the same vertex. When the exploration engine 205 encounters a screen, the engine compares that screen to all existing screens encountered to determine a match, a process referred to herein as deduplication. The exploration engine 205 is configured to use a custom object hashing comparison function to identify duplicate objects classified byGOOGLE-4244the action-ranking model 300. The classified objects are gathered, e.g., according to a preorder traversal, and then compared on an object-by-object basis, for example using structural matching as described in more detail, below.

[0092] Because the same object may be rendered at different times with a difference of a few pixels in placement, the exploration engine 205 can compute the difference between the minimum bounding box of both objects and their intersection, e.g., the amount of screen area that has changed, normalizing by the size of the screen. Traditional metrics such as intersection over union (IOU) do not work as well, because this approach does not account for relative rectangle sizes. To account for dynamic screens, e.g., in which the same screen may show different content at different times, the exploration engine 205 uses the view hierarchy in addition to structural matching to determine whether objects are the same or not. For example, the view hierarchy can include developer-provided annotations, such as class names or code descriptions, as well as action capabilities, hint text, or hidden / occluded objects. For example, because objects can contain child objects, comparing object similarity can include a measure of comparing structure similarity.

[0093] By using the view hierarchy data instead of image comparison alone, comparison is less prone to failure, for example if objects are ordered in a slightly different way, or if the hierarchy is subtly different, or the object attributes themselves are slightly changed. Comparisons can be made over objects in perceptually-different screens with structurally-different hierarchies, e.g., scrolled pages. Different use cases can be supported requiring different levels of approximate matching. For instance, object tracking may require a stricter match than object comparisons as part of deduplicating screens during navigations, as items may look different on a dynamic home screen. Techniques for comparing images alone do not account for semantically or structurally similar pages, even when the pages may be perceptually similar. A dynamic, image-heavy homepage may look completely different even within the same application crawl.

[0094] Different hashes are created from feature vectors or embeddings of attributes for each object. Each hash can be of a class corresponding to a comparison threshold, wherein hashes of the same class may be compared with one another. The exploration engine 205 is configured to condense attribute data into a vector of features, with which a distance metric such as cosine distance can be applied. A co-occurrence model can be trained to determine which objects are similar to one another, although any machine learning technique forGOOGLE-4244generating embeddings can be used. The models can be trained using a supervised learning approach, to reduce the error between a classification of two objects as similar, with a groundtruth label indicating object similarity. Example models for generating embeddings can be leveraged and can be trained for lexical and / or semantic similarity.

[0095] Object hashes can be compared through a combination of object attribute similarity (OAS) and object structural similarity (OSS). Object attribute similarity answers the question of how similar two objects are, without considering connections to other objects, e.g., child or parent objects. Attributes for an object in a view hierarchy can include class and resource names, content items within an object, the object layout, styling, functions available to the object, and child or parent objects. To perform OAS, the exploration engine 205 can generate embedding or feature vectors over which a distance metric can be used to compare objects, avoiding the cost of having to perform pairwise comparisons of all attributes of all objects. The distance metric can be continuous and comparable.

[0096] Any object can be queried for similarity across an index of all objects generated from objects encountered during exploration. The exploration engine 205 can compute the object structural similarity metric, which is tuned based on respective OAS values for individual object comparisons. OSS similarity compares individual objects, e.g., a previously explored object with a current object, while factoring in the parent-child relationships of the compared objects. OSS can be used for comparing screens rendering multiple objects, while OAS may compare individual objects. A screen of objects may be represented as a tree with a root node representing the overall screen, and individual children nodes representing individual objects that would be rendered on the screen.

[0097] Additionally, different object comparison requirements can take such a metric, and apply different thresholds for other purposes. For example, other comparisons can be between object tracking across screens, e.g., a scrolled page, requiring a high threshold of similarity in which content should be an exact or near-exact match. Object comparison for the purposes of navigation, e.g., to determine if one screen is unique versus previously encountered screen, may have a lower threshold and potentially be content- agnostic. In this way, threshold for similarity can be set to make hash comparisons more robust in measuring similarity instead of only exact-matching.

[0098] Object structural similarity (OSS) is a measure of similarity between two objects in the context within the view hierarchy corresponding to objects rendered on a screen. OSSGOOGLE-4244takes into account both objects’ children, if any, as well as their OAS. Relationships among objects can be modeled by the exploration engine 205 by an n-ary labeled tree or other tree structure. Because objects encode positioning within their attributes, the list-view of a corresponding tree need not be ordered in a particular way. As with OAS, OSS can be determined by parsing the object relationship structure to allow for computation of a distance metric. The distance metric can be used to compare the OSS of objects according to different thresholds of comparison.

[0099] An example threshold of comparison is identical matching, in which all attributes are hashed and XORed together to determine a match. Example objects may be considered to identically match, for example, when the objects match under both content-aware partial matching and content- agnostic partial matching, described below.

[0100] Another example is content-aware partial matching, in which all attributes for the compared objects are hashed except layout and XORed. Example attributes that can be compared include digital content, object identifier, developer comments or description in the view hierarchy for the object, and so on. This threshold can be used for comparing objects that may have shifted positions in the screen, but otherwise should be identical. For example, objects that are similar under this threshold are similar irrespective of location on the screen.

[0101] Another example threshold is content- agnostic partial matching, in which all attributes for the compared objects are hashed and compared except for content. Another example threshold is structural partial match, in which all attributes are hashed and compared except for content and styling. Example attributes compared under content- agnostic partial matching can include attributes representing the position of a bounding box for the object, whether the object is visible when a corresponding screen is rendered, the object type (e.g., container, image, list text, etc.), and other attributes that may relate to the structure of an object.

[0102] In one approach, a breadth-first search (BFS) or iterative deepening depth-first search (IDDFS) can be used to traverse through attributes of two objects, comparing the sorted object hashes at each level. This approach ensures that object ordering does not factor into the comparison. OSS can take into account both object-to-object comparison and tree structure, e.g., parents and children of the compare object. The exploration engine 205 can compute the OSS by comparing OAS scores for each object and summing the scores up. OAS scores can be positive values for a partial or complete match, and negative values when compared objects do not partially or identically match. A penalty can be added to the scores for missing nodes, andGOOGLE-4244an OSS threshold for similarity be determined, which when met means that the objects are the same under OSS. The exploration engine 205 can determine this threshold, for example, by determining various OSS scores for different objects of a software application during exploration, and basing the OSS threshold on the highest OSS scores that appear visually similar. The object hashes are matched using OSS for example using a threshold over the OAS distance scores as described above. The number of differences can be recorded, which can be further weighted by the level in the tree in which the differences occurred.

[0103] In some examples, the object trees can be compared using a tree edit distance. The tree edit distance is the minimum-cost sequence of node operations that transform one tree into another. The node operations permitted are inserted new nodes, deleting existing nodes, or transforming one node to another. The distance metric generated by OAS comparison can be readily used as input into transformations, and optionally, used to model the cost of insertions and deletions. Example approaches for computing the tree edit distance include Zhang-Shasha algorithm, robust tree-edit distance algorithm (RTED), pq-Gram, and the like. As described above with reference to the OSS similarity threshold, the exploration engine 205 can determine a threshold within which two trees are considered the same by tree edit distance, e.g., based on the tree edit scores computed during a current exploration.

[0104] Referring to FIG. 5, the exploration engine 205 is configured to implement cycle prevention to reduce or eliminate getting caught in cycles during exploration. An example cycle can be transitioning back and forth between screen 502C and screen 502D through action 508F and action 508G, respectively. Cycle prevention results in more efficient exploration, because fewer computing cycles and less time overall is spent exploring the same portions of a software application. More efficient exploration allows for more of the same application to be explored in the same amount of time, and / or the exploration of multiple applications. By comparing attributes of objects corresponding to screens encountered at different times, the exploration engine 205 can determine whether two screens represent the same screen, even when the content of the screens has changed between exploration steps.

[0105] Cycle prevention can be divided into two categories, preemptive blocking and post-cycle blocking. In preemptive blocking, the graph 500 can be used to keep track of actions already taken during the exploration, indicated by corresponding edges between screens. Controllers 435 are exposed to an interface for selecting the next action, which automatically omits actions that have already been taken. In this way, controllers cannot repeat actions thatGOOGEE-4244have already been taken, because those actions are not made available to the controllers 435 after being taken already.

[0106] In post-cycle blocking, the exploration engine 205 determines that exploration is proceeding in a cycle, e.g., by comparing the screens as described above. An example way to make this determination is to check actions previously taken within a window, e.g., the last five actions taken during an exploration. If the exploration engine 205 encounters a previously encountered screen using the same action as in a previous iteration, the action is added to a blocklist. The blocklist is fed to the controllers 435 to block certain actions from being selected.

[0107] Either or both of preemptive and post-cycle blocking can be used during exploration. Preemptive blocking allows for more efficient stepping through successive actions, e.g., because actions that are likely to result in a cycle are omitted. In some examples, preemption can be decayed over a predetermined amount of time or number of actions taken, for example to eventually allow for the action to be taken again to potentially reach a unique screen that was previously missed. Some actions may result in different behavior based on other actions previously taken. For example, some actions may not be available to advance to a screen until other actions are taken earlier in the trajectory, such as selecting from a list of categories on a current screen before hitting a “next” button to advance to a screen displaying the selected categories. Initially, the “next” button may be visible but disabled (e.g., “grayed out”), so the action to touch input the button is ranked lower, versus later, when earlier actions in the trajectory select categories and enable the “next” button. As another example, some objects may not be fully loaded in, and therefore not interactable, the first time a screen is reached. Post-cycle blocking can be used as a stronger signal, e.g., one that does not decay as with preemptive blocking, for example for actions that always result in cycles, such as trying to login without providing login information.

[0108] FIG. 6 is a diagram of the example evaluation framework 130 for policy violation determination, according to aspects of the disclosure. The evaluation framework 130 receives policy document 140, which may include one or more policy documents providing examples or defining prohibited behavior on a software application platform.

[0109] The evaluation framework 130 can implement a prompt generation engine 610 configured to receive the policy document 140 and generate natural language prompts for the multimodal evaluators 132. The multimodal evaluators 132 can be, for example, language models, large language models or other generative models trained to receive a prompt forGOOGLE-4244performing a processing task. The multimodal evaluators 132 can receive a prompt indicating what is part of policy for a software application platform, as well as the annotated image / video 127. The output of the multimodal evaluators 132 can be a policy violation determination 135, indicating whether the annotated image / video 127 includes instances of policy violations per the policy documents 140. The multimodal evaluators 132 can process the annotated image / video 127 frame-by-frame, and / or sample from the total number of frames available. In some instances, multimodal evaluators, such as multimodal evaluators 132, can operate as filters. For instance, multimodal evaluator(s) can identify annotated image / video 127 that satisfy or do not satisfy one or more criteria. Subsequent multimodal evaluator(s) can be programmed to act only on the annotated image / video 127 that satisfy or do not satisfy the criteria.

[0110] The image or video processing engine 125 can filter for specific content, e.g., specific advertisements shown in a frame of the video. The image or video processing engine 125 can annotate the image or video 127 to identify certain types of objects, e.g., advertisements or other types of videos, images, etc., which may be referenced or the subject of the policy document 140. The prompt generation engine 610 can be configured to generate policy subdocuments 640, which represent smaller portions of the policy document 140. For example, if a policy document has five different paragraphs indicating five different types of policy violations, the prompt generation engine 610 can summarize the policy document into five subdocuments, each representing the policy violation indicated in a respective paragraph. As part of subdocument generation, the prompt generation engine 610 can implement any of a variety of different text summarization techniques, e.g., appropriately trained large language models or other models trained for text summarization.

[0111] In some examples, the prompt generation engine 610 can revise labeled subdocuments, e.g., text, of a policy document, with the labels indicating the corresponding proposition or portion of a policy described by the subdocument. The prompt generation engine 610 can receive variations of prompts including the subdocuments, and its output can be generated and evaluated based on how similar the output is to the provided labels in determining what the subdocument describes. An Al model, such as a large language model, can be implemented by the prompt generation engine 610 for processing the subdocuments, and the model can be further fine-tuned based on the error between the output and the providedGOOGEE-4244labels. The process can be repeated until reaching a stopping criteria, e.g., a number of iterations, a minimum error threshold being met, and so on.

[0112] The subdocuments 640 can be used to generate one or more prompts for the multimodal evaluators 132. As described above, by separating the policy document into subdocuments, the multimodal evaluators 132 can perform more accurately in identifying violations, at least because each type of violation is focused on independent of one another. Higher accuracy also means that the multimodal evaluators 132 are less prone to false positive results, which can be detrimental if used in downstream processes, for example to determine whether to remove an application from a platform.

[0113] The multimodal evaluators 132 can be pre-trained large language models that are fine-tuned with examples of annotated image / videos with corresponding policy documents. The examples can be annotated with labels indicated whether the image / videos are indicative of policy violations or not.Example Methods

[0114] FIG. 7 is a flow diagram of an example process 700 for software application policy determination using automated software application exploration, according to aspects of the disclosure. A system, for example the policy violation determination system 100 of FIG.1, can perform the steps of the processes described herein, including in FIGs. 7, 8, and 9.

[0115] A system receives a view hierarchy and one or more screens of a mobile software application including data at least partially characterizing a structure or an organization of one or more objects of a plurality of screens that are rendered on a display device when the software application is executed, according to block 710.

[0116] The action-ranking model is trained to determine scores for the one or more candidate based on how many unique screens are accessed when respective actions of the one or more candidate sequences are performed. For example, the output of the action-ranking model can be a trajectory score based on a predetermined activity, such as whether the actions result in exploring more unique screens

[0117] The system generates image or video data of one or more screens of the plurality of screens rendered on the display device when a sequence of one or more actions are performed as input to the one or more objects of the software application, according to block 720. The sequence is one of one or more candidate sequences selected by a first artificial intelligence (Al) model trained to generate the one or more candidate sequences using the viewGOOGLE-4244hierarchy and the one or more screens. The first Al model can be the action-ranking model trained according to one or more objective functions corresponding to scoring the one or more candidate sequences based on how many unique screens are accessed when respective actions of the one or more candidate sequences are performed.

[0118] The system processes the image or video data of the sequence through a second Al model trained to classify digital content items present in the one or more screens and to determine whether actions performed as input to the one or more objects including the classified digital content items violate the policy of the software application platform, according to block 730. The system can process the image or video data of the sequence through a second Al model trained to classify objects present in the one or more screens. The second Al model can include multimodal evaluators, for example as shown and described with reference to FIGs. 1 and 6.

[0119] As part of performing the policy violation determination, the system can receive one or more policy documents at least partially characterizing the policy of the software application platform. The system can determine one or more policy subdocument from the one or more policy documents. The system can process by the one or more processors, the image or video data of the sequence through an Al model including a large language machine learning model to determine whether actions performed as input to the one or more objects including the classified objects violate the one or more determined subdocuments of the software application platform.

[0120] FIG. 8A is a flow diagram of an example process 800A for exploration deduplication during automated software application exploration, according to aspects of the disclosure.

[0121] The system generates a first embedding for a first screen and a second embedding for a second screen, according to block 810. An embedding for a screen includes attributes corresponding to objects in the view hierarchy corresponding to the screen and / or objects in an image corresponding to the screen.

[0122] The system compares the first embedding and the second embedding according to one or more comparison thresholds, according to block 820. The comparison thresholds can include one or more of: an identical matching threshold that is met when all attributes of both the first embedding and the second embedding match; a content-aware partial matching threshold that is met when all of the attributes of both the first and the second embedding matchGOOGLE-4244except for attributes corresponding to a respective layout of objects in either the first screen or the second screen; a content- agnostic partial matching threshold that is met when all of the attributes of both the first and the second embedding match except for attributes corresponding to respective digital content objects depicted in either the first screen or the second screen; and a structural partial matching threshold that is met when all of the attributes of both the first and the second embedding match except for attributes corresponding to a respective content or styling of objects in either the first screen or the second screen.

[0123] The system updates an application transition graph to include both the first screen and the second screen when the first embedding and the second embedding do not meet the one or more comparison thresholds, according to block 830.

[0124] The system determines, based on the application transition graph, whether a current screen has already been visited during an exploration, according to block 840. For example, if the application transition graph includes an indication of a screen copy, such as the screen copy 510A of FIG. 5, then the system can determine that the current screen has already been visited and another action can be selected and / or the action from a previous screen can be added to a blocklist.

[0125] FIG. 8B is a flow diagram of an example process 800B for aligning view hierarchies and screens during automated software application exploration, according to aspects of the disclosure.

[0126] The system generates aligned view hierarchy and screen data from a view hierarchy and one or more screens of a software application, according to block 850. The aligned view hierarchy can include one or more features or attributes relating objects in the view hierarchy with objects rendered in the one or more screens. For example, the features can associate that one object indicated in the view hierarchy is the object rendered as part of one of the one or more screens. The system can generate the aligned view hierarchy and screen data using a trained model, e.g., the alignment model 310 as shown and described with reference to FIG. 3. The model can be trained from training data including examples of view hierarchies and screens labeled with respective aligned view hierarchy and screen data, using a supervised learning approach to compute an error between aligned data and the labels. The error can be used to update model parameter values for the model, e.g., using backpropagation with gradient descent and weight update.GOOGLE-4244

[0127] The system generates an output exploration including image or video data using an Al model receiving the aligned view hierarchy and screen data as input, according to block 860. The Al model can be, for example, the action-ranking model 300 as shown and described with respect to FIG. 3.

[0128] FIG. 9 is a flow diagram of an example process 900 for automated software application exploration, according to aspects of the disclosure.

[0129] The system determines scores for one or more candidate sequences of actions based on how many unique screens are accessed when respective actions of the one or more candidate sequences are performed, according to block 910.

[0130] The system updates the scores with weights based on output from a plurality of classifiers comprising screen-level and object-level classifiers, according to block 920. Screenlevel classifiers are trained to classify objects of a software application depicted on screens of the software application and object-level classifiers are trained to classify objects of the software application indicated in the view hierarchy.

[0131] The system generates an application transition graph comprising vertices representing explored screens of the mobile software application and edges between screens indicating actions taken to transition from one connected screen to another, according to block 930.

[0132] The system updates the application transition graph when transitioning from one screen to another screen during the exploration, according to block 940. A unique screen is a screen that has not been previously represented in the application transition graph and unique screens are added as vertices to the application transition graph. The process 900 can continue for one or more steps, until a termination condition is met, and the exploration ends.

[0133] Implementations of the present disclosure can each include, but are not limited to, the following. The features may be alone or in combination with one or more other features described herein. In some examples, the following features are included in combination:(1) A method for determining violations of a policy of a software application platform, including: receiving, by one or more processors, one or more screens and a view hierarchy of a mobile software application, the view hierarchy including data at least partially characterizing a structure of one or more objects of a plurality of screens that are rendered on a display device when the mobile software application is executed; generating, by the one or more processors, image or video data of one or more screensGOOGLE-4244of the plurality of screens rendered on the display device when a sequence of one or more actions are performed as input to the one or more objects of the mobile software application, wherein the sequence is one of one or more candidate sequences selected by a first artificial intelligence (Al) model trained to generate the one or more candidate sequences using the view hierarchy and the one or more screens; and optionally processing, by the one or more processors, the image or video data of the sequence to determine whether actions performed as input to the one or more objects that violate a policy of a mobile software application platform.(2) The method of (1), wherein the first Al model is trained according to an objective function corresponding to scoring the one or more candidate sequences based on how many unique screens are accessed when respective actions of the one or more candidate sequences are performed.(3) The method of (2), further including: determining, by the one or more processors, scores for the one or more candidate sequences based on how many unique screens are accessed when respective actions of the one or more candidate sequences are performed; and updating, by the one or more processors, the scores with weights based on output from a plurality of classifiers including screen-level and object-level classifiers, wherein screen-level classifiers are trained to classify objects of the mobile software application depicted on screens of the mobile software application and object-level classifiers are trained to classify objects of the mobile software application indicated in the view hierarchy.(4) The method of either one of (2) or (3), further including: generating, by the one or more processors, an application transition graph including vertices representing explored screens of the mobile software application and edges between screens indicating actions taken to transition from one connected screen to another; and updating, by the one or more processors, the application transition graph when actions of the sequence are performed on the mobile software application transitioning from one screen to another screen or outside of the mobile software application, wherein a unique screen is a screen that has not been previously represented in the application transition graph and unique screens are added as vertices to the application transition graph.GOOGLE-4244(5) The method of (4), further including: generating, by the one or more processors, a first embedding for a first screen and a second embedding for a second screen, wherein an embedding for a screen includes attributes corresponding to objects in the view hierarchy corresponding to the screen and / or objects in an image corresponding to the screen; comparing, by the one or more processors, the first embedding and the second embedding according to one or more comparison thresholds; and updating, by the one or more processors, the application transition graph to include both the first screen and the second screen when the first embedding and the second embedding do not meet the one or more comparison thresholds.(6) The method of claim 5, wherein: the one or more comparison thresholds include one or more of: an identical matching threshold that is met when all attributes of both the first embedding and the second embedding match, a content-aware partial matching threshold that is met when all of the attributes of both the first and the second embedding match except for attributes corresponding to a respective layout of objects in either the first screen or the second screen, a content-agnostic partial matching threshold that is met when all of the attributes of both the first and the second embedding match except for attributes corresponding to respective digital content objects depicted in either the first screen or the second screen, and a structural partial matching threshold that is met when all of the attributes of both the first and the second embedding match except for attributes corresponding to a respective content or styling of objects in either the first screen or the second screen.(7) The method of any one of (1) through (6), further including: receiving, by the one or more processors, one or more policy documents at least partially characterizing the policy of the software application platform; determining, by the one or more processors, one or more policy subdocuments from the one or more policy documents; and processing, by the one or more processors, the image or video data of the sequence through a second Al model including a large language machine learning model to determine whether actions performed as input to the one or more objects including the classified objects violate the one or more determined subdocuments of the mobile software application platform.GOOGLE-4244(8) The method of any one of (1) through (7), further including processing the image or video data of the sequence through a second Al model trained to classify objects present in the one or more screens based on digital content in the objects.(9) The method of any one of (1) through (8), further including: generating, by the one or more processors, aligned view hierarchy and screen data from the view hierarchy and the one or more screens and including one or more features relating objects represented in the view hierarchy with objects rendered in the one or more screens; and generating, by the one or more processors, the image or video data using the first Al model receiving the aligned view hierarchy and screen data as input.(10) The method of (9), further including: generating, by the one or more processors, the aligned view hierarchy and screen data using the first Al model trained to generate the aligned view hierarchy and screen data from training data including examples of view hierarchies and screens labeled with respective aligned view hierarchy and screen data.(11) A system including one or more processors configured to: receive one or more screens and a view hierarchy of a mobile software application, the view hierarchy including data at least partially characterizing a structure of one or more objects of a plurality of screens that are rendered on a display device when the mobile software application is executed; generate image or video data of one or more screens of the plurality of screens rendered on the display device when a sequence of one or more actions are performed as input to the one or more objects of the mobile software application, wherein the sequence is one of one or more candidate sequences selected by a first artificial intelligence (Al) model trained to generate the one or more candidate sequences using the view hierarchy and the one or more screens; and optionally process the image or video data of the sequence to determine whether actions performed as input to the one or more objects that violate a policy of a mobile software application platform.(12) The system of (11), wherein the one or more processors are further configured to perform the method as in any one of (1) through (10).(13) One or more computer-readable storage media storing instructions that when executed by one or more processors, cause the one or more processors to perform operations including: receiving one or more screens and a view hierarchy of a mobileGOOGLE-4244software application, the view hierarchy including data at least partially characterizing a structure of one or more objects of a plurality of screens that are rendered on a display device when the mobile software application is executed; generating image or video data of one or more screens of the plurality of screens rendered on the display device when a sequence of one or more actions are performed as input to the one or more objects of the mobile software application, wherein the sequence is one of one or more candidate sequences selected by a first artificial intelligence (Al) model trained to generate the one or more candidate sequences using the view hierarchy and the one or more screens; and optionally processing the image or video data of the sequence to determine whether actions performed as input to the one or more objects that violate a policy of a mobile software application platform.(14) The one or more computer-readable storage media of (13), wherein the operations further include performing the method as in any one of (1) through (10). (15) The one or more computer-readable storage media of either one of (13) or (14), wherein the one or more computer-readable storage media are non-transitory. (16) One or more computer program products including instructions that when executed by one or more processors, cause the one or more processors to perform operations including: receiving one or more screens and a view hierarchy of a mobile software application, the view hierarchy including data at least partially characterizing a structure of one or more objects of a plurality of screens that are rendered on a display device when the mobile software application is executed; generating image or video data of one or more screens of the plurality of screens rendered on the display device when a sequence of one or more actions are performed as input to the one or more objects of the mobile software application, wherein the sequence is one of one or more candidate sequences selected by a first artificial intelligence (Al) model trained to generate the one or more candidate sequences using the view hierarchy and the one or more screens; and optionally processing the image or video data of the sequence to determine whether actions performed as input to the one or more objects that violate a policy of a mobile software application platform.(17) The one or more computer program products of (16), wherein the operations further include performing the method as in any one of (1) through (10).Example Computing EnvironmentGOOGLE-4244

[0134] FIG. 10 is a block diagram illustrating one or more models 1010, such as for deployment in a datacenter 1020 housing one or more hardware accelerators 1030 on which the deployed models will execute for policy violation determination and / or software application exploration. The hardware accelerators #30 can be any type of processor, such as a central processing unit (CPU), graphics processing unit (GPU), field-programmable gate array (FPGA), or an application- specific integrated circuit (ASIC), such as a tensor processing unit (TPU).

[0135] In some implementations, the techniques disclosed herein enable artificial intelligence to determine sequences of actions for application exploration and software application platform policy violation determination. Artificial intelligence (Al) is a segment of computer science that focuses on the creation of models that can perform tasks with little to no human intervention. Artificial intelligence systems can utilize, for example, machine learning, natural language processing, and computer vision. Machine learning, and its subsets, such as deep learning, focus on developing models that can infer outputs from data. The outputs can include, for example, predictions and / or classifications. Natural language processing focuses on analyzing and generating human language. Computer vision focuses on analyzing and interpreting images and videos. Artificial intelligence systems can include generative models that generate new content, such as images, videos, text, audio, and / or other content, in response to input prompts and / or based on other information.

[0136] An architecture of a model can refer to characteristics defining the model, such as characteristics of layers for the model, how the layers process input, or how the layers interact with one another. For example, the model can be a convolutional neural network that includes a convolution layer that receives input data, followed by a pooling layer, followed by a fully connected layer that generates a result. The architecture of the model can also define types of operations performed within each layer. For example, the architecture of a convolutional neural network may define that rectified linear unit (ReLU) activation functions are used in the fully connected layer of the network. Other example architectures can include generative models, such as language models, foundation models, and / or graphical models. One or more model architectures can be generated that can output results associated with policy determination violation and / or software application exploration, e.g., multimodal evaluators 132, image or video processing engine 125, and action-ranking model 300.GOOGEE-4244

[0137] Example machine-learned models include neural networks or other multi-layer non-linear models. Example neural networks include feed forward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some example machine-learned models can leverage an attention mechanism such as self-attention. For example, some machine-learned models can include multi-headed self-attention models (e.g., transformer models).

[0138] The model(s) can be trained using various training or learning techniques. The training can implement supervised learning, unsupervised learning, reinforcement learning, etc. The training can use techniques such as, for example, backwards propagation of errors. For example, a loss function can be backpropagated through the model(s) to update one or more parameters of the model(s) (e.g., based on a gradient of the loss function). Various loss functions can be used such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques can be used to iteratively update the parameters over a number of training iterations. A number of generalization techniques (e.g., weight decays, dropouts) can be used to improve the generalization capability of the models being trained. The training examples can be labeled with a desired output for the model when processing the labeled training examples. The label and the model output can be evaluated through a loss function to determine an error, which can be backpropagated through the model to update weights for the model. For example, a supervised learning technique can be applied to calculate an error between outputs, with a ground-truth label of a training example processed by the model.

[0139] The model(s) can be pre-trained before domain- specific alignment. For instance, a model can be pretrained over a general corpus of training data and fine-tuned on a more targeted corpus of training data. A model can be aligned using prompts that are designed to elicit domain- specific outputs. Prompts can be designed to include learned prompt values (e.g., soft prompts). The trained model(s) may be validated prior to their use using input data other than the training data and may be further updated or refined during their use based on additional feedback / inputs.

[0140] The machine learning models can be trained according to a variety of different learning techniques. Eearning techniques for training the machine learning models can include supervised learning, unsupervised learning, semi- supervised learning, and reinforcementGOOGLE-4244learning techniques. For example, training data can include multiple training examples that can be received as input by a model.

[0141] Any of a variety of loss or error functions appropriate for the type of the task the model is being trained for can be utilized, such as cross-entropy loss for classification tasks, or mean square error for regression tasks. The gradient of the error with respect to the different weights of the candidate model on candidate hardware can be calculated, for example using a backpropagation algorithm, and the weights for the model can be updated. The model can be trained until stopping criteria are met, such as a number of iterations for training, a maximum period of time, a convergence, or when a minimum accuracy threshold is met.

[0142] As another example, with respect to reinforcement learning, situations encountered by an agent, e.g., a model, a computing device, a system, a robot, etc., are mapped to actions taken by the agent in those situations to maximize the reward or value of its actions. The agent can interact with an environment through its actions. At any given time or point at which the agent is able to act, the environment can be represented as a state. The state can include any information or features about the environment that can be known by the agent. The value of a state is a measure of the total amount of reward the agent can receive from the current state and future states accessible from the current state. A value function can be defined or estimated for calculating, predicting, or estimating the value of a state. Techniques for training a machine learning model via reinforcement learning can focus on estimating or learning value functions to accurately predict value across different states of an environment.

[0143] The agent applies a policy to determine an action to take given the state of the environment. The policy can be stochastic, deterministic, or a mixture of the two. The agent can be provided a reward signal or value in response to performing the action, which can be positive, negative, or neutral. The action taken by the agent can advance the environment to a new state with an objective being to maximize the value of a state brought upon by the agent performing an action. Example reinforcement learning techniques include multi-armed bandits, Markov decision processes, Monte Carlo methods, policy gradient methods, and / or other approximate solution methods. Other approaches in reinforcement learning may not rely on estimating value functions.

[0144] The model or policy can be modified or updated until stopping criteria are met, such as a number of iterations for training, a maximum period of time, a convergence of estimated rewards or value between actions, or when a minimum value threshold is met. AGOOGLE-4244model can be a composite of multiple models or elements of a processing or training pipeline. In some examples, the models or elements are trained separately, while in other examples, the models or elements are trained end-to-end.

[0145] FIG. 11 is a block diagram of an example computing environment 1100 for implementing the policy violation determination system 100. The system 100 can be implemented on one or more devices having one or more processors in one or more locations, such as in server computing device 1115. User computing device 1112 and the server computing device 1115 can be communicatively coupled to one or more storage devices 1130 over a network 1160. The storage device(s) 1130 can be a combination of volatile and nonvolatile memory and can be at the same or different physical locations than the computing devices 1112, 1115. For example, the storage device(s) 1130 can include any type of non-transitory computer readable medium capable of storing information, such as a hard-drive, solid state drive, tape drive, optical storage, memory card, ROM, RAM, DVD, CD-ROM, write-capable, and read-only memories.

[0146] Aspects of the disclosure can be implemented in a computing system that includes a back-end element, e.g., as a data server, a middleware element, e.g., an application server, or a front-end element, e.g., user computing device 1112 having a user interface, a web browser, or an app, or any combination thereof. The elements of the system can be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet. The datacenter #20 can also be in communication with the user computing device 1112 and the server computing device 1115.

[0147] The computing system can include clients, e.g., user computing device 1112 and servers, e.g., server computing device 1115. A client and server can be remote from each other and interact through a communication network. The relationship of client and server arises by virtue of the computer programs running on the respective computers and having a client-server relationship to each other. For example, a server can transmit data, e.g., an HTML page, to a client device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the client device. Data generated at the client device, e.g., a result of the user interaction, can be received at the server from the client device.

[0148] The server computing device 1115 can include one or more processors 1113 and memory 1114. The memory 1114 can store information accessible by the processor(s) 1113,GOOGLE-4244including instructions 1121 that can be executed by the processor(s) 1113. The memory 1114 can also include data 1123 that can be retrieved, manipulated, or stored by the processor(s) 1113. The memory 1114 can be a type of non-transitory computer readable medium capable of storing information accessible by the processor(s) 1113, such as volatile and non-volatile memory. The processor(s) 1113 can include one or more central processing units (CPUs), graphic processing units (GPUs), field-programmable gate arrays (FPGAs), and / or applicationspecific integrated circuits (ASICs), such as tensor processing units (TPUs).

[0149] The instructions 1121 can include one or more instructions that when executed by the processor(s) 1113, causes the one or more processors to perform actions defined by the instructions. The instructions 1121 can be stored in object code format for direct processing by the processor(s) 1113, or in other formats including interpretable scripts or collections of independent source code modules that are interpreted on demand or compiled in advance. The instructions 1121 can include instructions for implementing the system 100 consistent with aspects of this disclosure. The system 100 can be executed using the processor(s) 1113, and / or using other processors remotely located from the server computing device 1115.

[0150] The data 1123 can be retrieved, stored, or modified by the processor(s) 1113 in accordance with the instructions 1121. The data 1123 can be stored in computer registers, in a relational or non-relational database as a table having a plurality of different fields and records, or as JSON, YAML, proto, or XML documents. The data 1123 can also be formatted in a computer-readable format such as, but not limited to, binary values, ASCII, or Unicode. Moreover, the data 1123 can include information sufficient to identify relevant information, such as numbers, descriptive text, proprietary codes, pointers, references to data stored in other memories, including other network locations, or information that is used by a function to calculate relevant data.

[0151] The user computing device 1112 can also be configured similarly to the server computing device 1115, with one or more processors 1116, memory 1117, instructions 1118, and data 1119. For example, the user computing device 1112 can be a mobile device, a laptop, a desktop computer, a game console, etc. The user computing device 1112 can also include a user output 1126, and a user input 1124. The user input 1124 can include any appropriate mechanism or technique for receiving input from a user, including acoustic input; visual input; tactile input, including touch motion or gestures, or kinetic motion or gestures or orientation motion or gestures; auditory input, speech input, etc., Example devices for user input 1124 canGOOGLE-4244include a keyboard, mouse or other point device, mechanical actuators, soft actuators, touchscreens, microphones, and sensors.

[0152] The server computing device 1115 can be configured to transmit data to the user computing device 1112, and the user computing device 1112 can be configured to display at least a portion of the received data on a display implemented as part of the user output 1126. The user output 1126 can also be used for displaying an interface between the user computing device 1112 and the server computing device 1115. The user output 1126 can alternatively or additionally include one or more speakers, transducers or other audio outputs, a haptic interface or other tactile feedback that provides non-visual and non-audible information to the platform user of the user computing device 1112.

[0153] Although FIG. 11 illustrates the processors 1113, 1116 and the memories 1114, 1117 as being within the computing devices 1115, 1112, elements described in this specification, including the processors 1113, 1116 and the memories 1114, 1117 can include multiple processors and memories that can operate in different physical locations and not within the same computing device. For example, some of the instructions 1121, 1118 and the data 1123, 1119 can be stored on a removable SD card and others within a read-only computer chip. Some or all of the instructions and data can be stored in a location physically remote from, yet still accessible by, the processors 1113, 1116. Similarly, the processors 1113, 1116 can include a collection of processors that can perform concurrent and / or sequential operation. The computing devices 1115, 1112 can each include one or more internal clocks providing timing information, which can be used for time measurement for operations and programs run by the computing devices 1115, 1112.

[0154] The server computing device 1115 can be configured to receive requests to process data from the user computing device 1112. For example, the environment 1100 can be part of a computing platform configured to provide a variety of services to users, through various user interfaces and / or APIs exposing the platform services. One or more services can be a machine learning framework or a set of tools for training or executing generative models or other machine learning models according to a specified task and training data.

[0155] The devices 1112, 1115 can be capable of direct and indirect communication over the network 1160. The devices 1115, 1112 can set up listening sockets that may accept an initiating connection for sending and receiving information. The network 1160 itself can include various configurations and protocols including the Internet, World Wide Web,GOOGLE-4244intranets, virtual private networks, wide area networks, local networks, and private networks using communication protocols proprietary to one or more companies. The network 1160 can support a variety of short- and long-range connections. The short- and long-range connections may be made over different bandwidths, such as 2.402 GHz to 2.480 GHz (commonly associated with the Bluetooth® standard), 2.4 GHz and 5 GHz (commonly associated with the Wi-Fi® communication protocol); or with a variety of communication standards, such as the LTE® standard for wireless broadband communication. The network 1160, in addition or alternatively, can also support wired connections between the devices 1112, 1115, including over various types of Ethernet connection.

[0156] Although a single server computing device 1115, user computing device 1112, and datacenter 1020 are shown in FIG. 11, it is understood that the aspects of the disclosure can be implemented according to a variety of different configurations and quantities of computing devices, including in paradigms for sequential or parallel processing, or over a distributed network of multiple devices. In some implementations, aspects of the disclosure can be performed on a single device, and any combination thereof.

[0157] Aspects of this disclosure can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, and / or in computer hardware, such as the structure disclosed herein, their structural equivalents, or combinations thereof. Aspects of this disclosure can further be implemented as one or more computer programs, such as one or more engines or modules of computer program instructions encoded on one or more tangible non-transitory computer storage media for execution by, or to control the operation of, one or more data processing apparatus.

[0158] A computer storage medium can be a machine -readable storage device, a machine-readable storage substrate, a random or serial access memory device, or combinations thereof. The computer program instructions can be encoded on an artificially generated propagated signal, such as a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. A computer program may, but need not, correspond to a file in a file system. A computer program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts, in a single file, or in multiple coordinated files, e.g., files that store one or more engines, modules, sub-programs, or portions of code.GOOGLE-4244

[0159] The term “configured” is used herein in connection with systems and computer program elements. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed software, firmware, hardware, or a combination thereof that cause the system to perform the operations or actions. For one or more computer programs to be configured to perform operations or actions means that the one or more programs include instructions that, when executed by one or more data processing apparatus, cause the apparatus to perform the operations or actions.

[0160] The term “data processing apparatus” refers to data processing hardware and encompasses various apparatus, devices, and machines for processing data, including programmable processors, a computer, or combinations thereof. The data processing apparatus can include special purpose logic circuitry, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC), such as a Tensor Processing Unit (TPU). The data processing apparatus can include code that creates an execution environment for computer programs, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or combinations thereof.

[0161] The data processing apparatus can include special-purpose hardware accelerator units for implementing machine learning models to process common and compute-intensive parts of machine learning training or production, such as inference or workloads. Machine learning models can be implemented and deployed using one or more machine learning frameworks, such as static or dynamic computational graph frameworks.

[0162] The term “computer program” refers to a program, software, a software application, an app, a module, a software module, a script, or code. The computer program can be written in any form of programming language, including compiled, interpreted, declarative, or procedural languages, or combinations thereof. The computer program can be deployed in any form, including as a standalone program or as a module, element, subroutine, or other unit suitable for use in a computing environment. The computer program can correspond to a file in a file system and can be stored in a portion of a file that holds other programs or data, such as one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, such as files that store one or more modules, sub programs, or portions of code. The computer program can be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.GOOGLE-4244

[0163] The term “database” refers to any collection of data. The data can be unstructured or structured in any manner. The data can be stored on one or more storage devices in one or more locations. For example, an index database can include multiple collections of data, each of which may be organized and accessed differently.

[0164] The term “engine” can refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. The engine can be implemented as one or more software modules or elements or can be installed on one or more computers in one or more locations. A particular engine can have one or more processors or computing devices dedicated thereto, or multiple engines can be installed and running on the same processor or computing device. In some examples, an engine can be implemented as a specially configured circuit, while in other examples, an engine can be implemented in a combination of software and hardware.

[0165] The processes and logic flows described herein can be performed by one or more computers executing one or more computer programs to perform functions by operating on input data and generating output data. The processes and logic flows can also be performed by special purpose logic circuitry, or by a combination of special purpose logic circuitry and one or more computers. While operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and elements in the embodiments described above should not be understood as requiring such separation in all examples, and it should be understood that the described program elements and systems can be integrated together in one or more software or hardware-based devices or computer-readable media.

[0166] A computer or special purpose logic circuitry executing the one or more computer programs can include a central processing unit, including general or special purpose microprocessors, for performing or executing instructions and one or more memory devices for storing the instructions and data. The central processing unit can receive instructions and data from the one or more memory devices, such as read only memory, random access memory, or combinations thereof, and can perform or execute the instructions. The computer or special purpose logic circuitry can also include, or be operatively coupled to, one or more storageGOOGLE-4244devices for storing data, such as magnetic, magneto optical disks, or optical disks, for receiving data from or transferring data to. The computer or special purpose logic circuitry can be embedded in another device, such as a mobile phone, desktop computer, a personal digital assistant (PDA), a mobile audio or video player, a game console, a tablet, a virtual-reality (VR) or augmented-reality (AR) device, a Global Positioning System (GPS), or a portable storage device, e.g., a universal serial bus (USB) flash drive, as examples. Examples of the computer or special purpose logic circuitry can include the user computing device 1112, the server computing device 1115, or the hardware accelerators 1030.

[0167] Computer readable media suitable for storing the one or more computer programs can include any form of volatile or non-volatile memory, media, or memory devices. Examples include semiconductor memory devices, e.g., EPROM, EEPROM, or flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto optical disks, CD-ROM disks, DVD-ROM disks, or combinations thereof.

[0168] Unless otherwise stated, the foregoing alternative examples are not mutually exclusive, but may be implemented in various combinations to achieve unique advantages. As these and other variations and combinations of the features discussed above can be utilized without departing from the subject matter defined by the claims, the foregoing description of the embodiments should be taken by way of illustration rather than by way of limitation of the subject matter defined by the claims. In addition, the provision of the examples described herein, as well as clauses phrased as "such as," "including" and the like, should not be interpreted as limiting the subject matter of the claims to the specific examples; rather, the examples are intended to illustrate only one of many possible examples. Further, the same reference numbers in different drawings can identify the same or similar elements.

Claims

GOOGLE-4244CLAIMS1. A method, comprising:receiving, by one or more processors, one or more screens and a view hierarchy of a mobile software application, the view hierarchy comprising data at least partially characterizing a structure of one or more objects of a plurality of screens that are rendered on a display device when the mobile software application is executed;generating, by the one or more processors, image or video data of one or more screens of the plurality of screens rendered on the display device when a sequence of one or more actions are performed as input to the one or more objects of the mobile software application, wherein the sequence is one of one or more candidate sequences selected by a first artificial intelligence (Al) model trained to generate the one or more candidate sequences using the view hierarchy and the one or more screens; andoptionally processing, by the one or more processors, the image or video data of the sequence to determine whether actions performed as input to the one or more objects that violate a policy of a mobile software application platform.

2. The method of claim 1, wherein the first Al model is trained according to an objective function corresponding to scoring the one or more candidate sequences based on how many unique screens are accessed when respective actions of the one or more candidate sequences are performed.

3. The method of claim 2, further comprising:determining, by the one or more processors, scores for the one or more candidate sequences based on how many unique screens are accessed when respective actions of the one or more candidate sequences are performed; andupdating, by the one or more processors, the scores with weights based on output from a plurality of classifiers comprising screen-level and object-level classifiers, wherein screen-level classifiers are trained to classify objects of the mobile software application depicted on screens of the mobile software application and object-level classifiers are trained to classify objects of the mobile software application indicated in the view hierarchy.GOOGLE-42444. The method of either one of claims 2 or 3, further comprising:generating, by the one or more processors, an application transition graph comprising vertices representing explored screens of the mobile software application and edges between screens indicating actions taken to transition from one connected screen to another; and updating, by the one or more processors, the application transition graph when actions of the sequence are performed on the mobile software application transitioning from one screen to another screen or outside of the mobile software application, wherein a unique screen is a screen that has not been previously represented in the application transition graph and unique screens are added as vertices to the application transition graph.

5. The method of claim 4, further comprising:generating, by the one or more processors, a first embedding for a first screen and a second embedding for a second screen, wherein an embedding for a screen comprises attributes corresponding to objects in the view hierarchy corresponding to the screen and / or objects in an image corresponding to the screen;comparing, by the one or more processors, the first embedding and the second embedding according to one or more comparison thresholds; andupdating, by the one or more processors, the application transition graph to include both the first screen and the second screen when the first embedding and the second embedding do not meet the one or more comparison thresholds.

6. The method of claim 5, wherein:the one or more comparison thresholds comprise one or more of:an identical matching threshold that is met when all attributes of both the first embedding and the second embedding match,a content-aware partial matching threshold that is met when all of the attributes of both the first and the second embedding match except for attributes corresponding to a respective layout of objects in either the first screen or the second screen,a content- agnostic partial matching threshold that is met when all of the attributes of both the first and the second embedding match except for attributes corresponding to respective digital content objects depicted in either the first screen or the second screen, and a structural partial matching threshold that is met when all of the attributes of both theGOOGLE-4244first and the second embedding match except for attributes corresponding to a respective content or styling of objects in either the first screen or the second screen.

7. The method of any one of the preceding claims, further comprising:receiving, by the one or more processors, one or more policy documents at least partially characterizing the policy of the software application platform;determining, by the one or more processors, one or more policy subdocuments from the one or more policy documents; andprocessing, by the one or more processors, the image or video data of the sequence through a second Al model comprising a large language machine learning model to determine whether actions performed as input to the one or more objects comprising the classified objects violate the one or more determined subdocuments of the mobile software application platform.

8. The method of any one of the preceding claims, further comprising processing the image or video data of the sequence through a second Al model trained to classify objects present in the one or more screens based on digital content in the objects.

9. The method of any one of the preceding claims, further comprising:generating, by the one or more processors, aligned view hierarchy and screen data from the view hierarchy and the one or more screens and comprising one or more features relating objects represented in the view hierarchy with objects rendered in the one or more screens; andgenerating, by the one or more processors, the image or video data using the first Al model receiving the aligned view hierarchy and screen data as input.

10. The method of claim 9, further comprising:generating, by the one or more processors, the aligned view hierarchy and screen data using the first Al model trained to generate the aligned view hierarchy and screen data from training data comprising examples of view hierarchies and screens labeled with respective aligned view hierarchy and screen data.GOOGLE-424411. A system comprising:one or more processors configured to:receiving, by one or more processors, one or more screens and a view hierarchy of a mobile software application, the view hierarchy comprising data at least partially characterizing a structure of one or more objects of a plurality of screens that are rendered on a display device when the mobile software application is executed;generating, by the one or more processors, image or video data of one or more screens of the plurality of screens rendered on the display device when a sequence of one or more actions are performed as input to the one or more objects of the mobile software application, wherein the sequence is one of one or more candidate sequences selected by a first artificial intelligence (Al) model trained to generate the one or more candidate sequences using the view hierarchy and the one or more screens; and optionally processing, by the one or more processors, the image or video data of the sequence to determine whether actions performed as input to the one or more objects that violate a policy of a mobile software application platform.

12. The system of claim 11, wherein the first Al model is trained according to an objective function corresponding to scoring the one or more candidate sequences based on how many unique screens are accessed when respective actions of the one or more candidate sequences are performed.

13. The system of claim 12, wherein the one or more processors are further configured to: determine scores for the one or more candidate sequences based on how many unique screens are accessed when respective actions of the one or more candidate sequences are performed; andupdate the scores with weights based on output from a plurality of classifiers comprising screen-level and object-level classifiers, wherein screen-level classifiers are trained to classify objects of the mobile software application depicted on screens of the mobile software application and object-level classifiers are trained to classify objects of the mobile software application indicated in the view hierarchy.GOOGLE-424414. The system of either one of claims 12 or 13, wherein the one or more processors are further configured to:generate an application transition graph comprising vertices representing explored screens of the mobile software application and edges between screens indicating actions taken to transition from one connected screen to another; andupdate the application transition graph when actions of the sequence are performed on the mobile software application transitioning from one screen to another screen or outside of the mobile software application, wherein a unique screen is a screen that has not been previously represented in the application transition graph and unique screens are added as vertices to the application transition graph.

15. The system of claim 14, wherein the one or more processors are further configured to: generate a first embedding for a first screen and a second embedding for a second screen, wherein an embedding for a screen comprises attributes corresponding to objects in the view hierarchy corresponding to the screen and / or objects in an image corresponding to the screen;compare the first embedding and the second embedding according to one or more comparison thresholds; andupdate the application transition graph to include both the first screen and the second screen when the first embedding and the second embedding do not meet the one or more comparison thresholds.

16. The system of claim 15, wherein:the one or more comparison thresholds comprise one or more of:an identical matching threshold that is met when all attributes of both the first embedding and the second embedding match,a content-aware partial matching threshold that is met when all of the attributes of both the first and the second embedding match except for attributes corresponding to a respective layout of objects in either the first screen or the second screen, a content- agnostic partial matching threshold that is met when all of the attributes of both the first and the second embedding match except for attributes corresponding to respective digital content objects depicted in either the first screen or theGOOGLE-4244second screen, anda structural partial matching threshold that is met when all of the attributes of both the first and the second embedding match except for attributes corresponding to a respective content or styling of objects in either the first screen or the second screen.

17. The system of any one of claims 11 through 16, wherein the one or more processors are further configured to:receive one or more policy documents at least partially characterizing the policy of the software application platform;determine one or more policy subdocuments from the one or more policy documents; andprocess the image or video data of the sequence through a second Al model comprising a large language machine learning model to determine whether actions performed as input to the one or more objects comprising the classified objects violate the one or more determined subdocuments of the mobile software application platform.

18. The system of any one of claims 11 through 17, wherein the one or more processors are further configured to process the image or video data of the sequence through a second Al model trained to classify objects present in the one or more screens based on digital content in the objects.

19. The system of any one of claims 11 through 18, wherein the one or more processors are further configured to:generate aligned view hierarchy and screen data from the view hierarchy and the one or more screens and comprising one or more features relating objects represented in the view hierarchy with objects rendered in the one or more screens; andgenerating, by the one or more processors, the image or video data using the first Al model receiving the aligned view hierarchy and screen data as input.

10. The system of claim 19, wherein the one or more processors are further configured to generate the aligned view hierarchy and screen data using the first Al model trained to generate the aligned view hierarchy and screen data from training data comprising examplesGOOGLE-4244of view hierarchies and screens labeled with respective aligned view hierarchy and screen data.

20. One or more computer-readable storage media, comprising instructions that when performed by one or more processors, causes the one or more processors to perform operations comprising:receiving one or more screens and a view hierarchy of a mobile software application, the view hierarchy comprising data at least partially characterizing a structure of one or more objects of a plurality of screens that are rendered on a display device when the mobile software application is executed;generating image or video data of one or more screens of the plurality of screens rendered on the display device when a sequence of one or more actions are performed as input to the one or more objects of the mobile software application, wherein the sequence is one of one or more candidate sequences selected by a first artificial intelligence (Al) model trained to generate the one or more candidate sequences using the view hierarchy and the one or more screens; andoptionally processing the image or video data of the sequence to determine whether actions performed as input to the one or more objects that violate a policy of a mobile software application platform.

21. The computer-readable storage media of claim 20, wherein the first Al model is trained according to an objective function corresponding to scoring the one or more candidate sequences based on how many unique screens are accessed when respective actions of the one or more candidate sequences are performed.

22. The computer-readable storage media of claim 21, wherein the operations further comprise:determining scores for the one or more candidate sequences based on how many unique screens are accessed when respective actions of the one or more candidate sequences are performed; andupdating the scores with weights based on output from a plurality of classifiers comprising screen-level and object-level classifiers, wherein screen-level classifiers areGOOGLE-4244trained to classify objects of the mobile software application depicted on screens of the mobile software application and object-level classifiers are trained to classify objects of the mobile software application indicated in the view hierarchy.

23. The computer-readable storage media of either one of claims 21 or 22, wherein the operations further comprise:generating an application transition graph comprising vertices representing explored screens of the mobile software application and edges between screens indicating actions taken to transition from one connected screen to another; andupdating the application transition graph when actions of the sequence are performed on the mobile software application transitioning from one screen to another screen or outside of the mobile software application, wherein a unique screen is a screen that has not been previously represented in the application transition graph and unique screens are added as vertices to the application transition graph.

24. The computer-readable storage media of claim 23, wherein the operations further comprise:generating a first embedding for a first screen and a second embedding for a second screen, wherein an embedding for a screen comprises attributes corresponding to objects in the view hierarchy corresponding to the screen and / or objects in an image corresponding to the screen;comparing the first embedding and the second embedding according to one or more comparison thresholds; andupdating the application transition graph to include both the first screen and the second screen when the first embedding and the second embedding do not meet the one or more comparison thresholds.

25. The computer-readable storage media of claim 24, wherein:the one or more comparison thresholds comprise one or more of:an identical matching threshold that is met when all attributes of both the first embedding and the second embedding match,a content-aware partial matching threshold that is met when all of theGOOGLE-4244attributes of both the first and the second embedding match except for attributes corresponding to a respective layout of objects in either the first screen or the second screen, a content- agnostic partial matching threshold that is met when all of the attributes of both the first and the second embedding match except for attributes corresponding to respective digital content objects depicted in either the first screen or the second screen, anda structural partial matching threshold that is met when all of the attributes of both the first and the second embedding match except for attributes corresponding to a respective content or styling of objects in either the first screen or the second screen.

26. The computer-readable media of any one of claims 20 through 25, wherein the operations further comprise:receiving one or more policy documents at least partially characterizing the policy of the software application platform;determining one or more policy subdocuments from the one or more policy documents; andprocessing the image or video data of the sequence through a second Al model comprising a large language machine learning model to determine whether actions performed as input to the one or more objects comprising the classified objects violate the one or more determined subdocuments of the mobile software application platform.

27. The computer-readable media of any one of claims 20 through 26, wherein the operations further comprise processing the image or video data of the sequence through a second Al model trained to classify objects present in the one or more screens based on digital content in the objects.

28. The computer-readable media of any one of claims 20 through 27, wherein the operations further comprise:generating aligned view hierarchy and screen data from the view hierarchy and the one or more screens and comprising one or more features relating objects represented in the view hierarchy with objects rendered in the one or more screens; andGOOGLE-4244generating, by the one or more processors, the image or video data using the first Al model receiving the aligned view hierarchy and screen data as input.

29. The computer-readable media of claim 28, wherein the operations further comprise generating the aligned view hierarchy and screen data using the first Al model trained to generate the aligned view hierarchy and screen data from training data comprising examples of view hierarchies and screens labeled with respective aligned view hierarchy and screen data.

30. The computer-readable media of any one of claims 20 through 29, wherein the computer-readable media is non-transitory.

Images