Mobile application interface data collection system and method, apparatus, storage medium

Abstract

This invention discloses a mobile application interface data acquisition system, method, device, and storage medium. It utilizes a Visual Language Model (VLM), a multi-level recognition strategy, and an adaptive rule update mechanism to achieve intelligent acquisition of mobile application interface data. By employing the technical solution of this invention, the problems of fragile rules, inefficient acquisition, and lack of self-repair in traditional solutions are solved, achieving high-precision, low-cost, and continuously available data acquisition.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and mobile application automation technology, specifically relating to a mobile application interface data acquisition system, method, device, and storage medium, which can be applied to scenarios such as mobile application monitoring, data crawling, automated testing, and user interface analysis. Background Technology

[0002] The rapid development of mobile internet applications has led to a massive demand for interface data collection, but existing technologies have the following prominent problems: 1. Poor adaptability to interface changes Frequent mobile app interface updates have become the norm in the industry. According to the latest statistics, 37% of the most popular mobile apps on the Google Play Store worldwide are updated at least once a week. Traditional rule-based matching methods using fixed XPath, ID, or CSS selectors become ineffective immediately after an interface redesign, a problem widely reported in both academia and industry.

[0003] A survey of over 50 mobile testing experts by Laishram et al. revealed that XPath-based automated test scripts often fail after each UI update, requiring manual reconfiguration of the rules. The root cause of this "brittleness" lies in the fact that XPath expressions are tightly coupled to the application's DOM structure, rather than to user intent. When the development team adjusts the layout hierarchy, modifies element IDs, or restructures components, the positioning rules immediately become ineffective, even if the functionality remains unchanged.

[0004] In real-world applications, this poor adaptability leads to high maintenance costs. A BrowserStack technical report indicates that over 70% of maintenance time in mobile automated testing projects is spent fixing locator malfunctions caused by UI changes. For teams adopting agile development models, a two-week iteration cycle means test engineers need to continuously invest resources in updating and validating locator rules, severely impacting the return on investment for automated testing.

[0005] More seriously, existing solutions lack the ability to effectively handle dynamic elements. Modern mobile applications widely employ technologies such as dynamic ID generation, conditional rendering, and lazy loading, resulting in randomness and uncertainty in element attributes. For example, component IDs generated by the React Native framework often contain hash values ​​that change after each build; and the rendering tree structure of Flutter applications differs significantly from the native view hierarchy, making it difficult for traditional UIAutomator or XCTest tools to accurately locate elements.

[0006] 2. OCR recognition has obvious limitations. While Optical Character Recognition (OCR) technology has been applied in mobile application data collection, its inherent limitations are particularly prominent in complex scenarios. Although pure OCR solutions can extract text information, they cannot understand the semantics of the interface and the logical relationships between elements, a problem that has been fully confirmed in academic research.

[0007] According to a comprehensive evaluation report by AIMultiple Research, even the most advanced OCR engines (such as Google Cloud Vision API and Tesseract 5.0) can achieve an accuracy rate of over 99% on standard printed text, but in complex scenarios such as mobile application interfaces, the recognition accuracy drops sharply to 78%-85%. This performance degradation is mainly due to the following technical bottlenecks: Complex background interference: Mobile application interfaces often use gradient backgrounds, textured patterns, or semi-transparent overlays to enhance visual effects. These design elements can severely interfere with OCR text segmentation algorithms. DocuClipper's research shows that when the contrast between text and background is less than 3:1, the OCR error rate increases by more than 40%. Richly colored background patterns can also cause OCR engines to misidentify decorative elements as text, generating a large amount of noisy data.

[0008] Multilingual processing is challenging: Global applications often use multiple languages ​​and scripting systems within the same interface. Ultralytics' technical analysis indicates that OCR engines achieve only 65%-70% accuracy when processing complex glyphs such as Arabic and Hindi, due to character ligatures and contextual dependencies. Even more problematic is the difficulty OCR can accurately segment character boundaries when the interface simultaneously contains Latin letters, Chinese characters, and emojis, frequently resulting in character overlap or segmentation errors.

[0009] Insufficient recognition of special fonts and styles: In order to differentiate their brands, many applications use custom fonts or artistic fonts. The Wikipedia entry on OCR technology

[10] clearly points out that standard OCR engines are usually only trained on common fonts such as Arial and Times New Roman, and their ability to recognize handwritten, decorative, or pixelated fonts is very limited. In addition, text rotation, slant, shadow, and outline effects will also significantly reduce recognition accuracy.

[0010] Lack of semantic understanding: This is the most fundamental limitation of OCR technology. Coursera's OCR technology course emphasizes that traditional OCR can only perform character-level or word-level recognition and cannot understand the semantic content and contextual relationships of text. For example, OCR can recognize the text of two buttons, "Add to Cart" and "Buy Now," but it cannot determine their functional differences, operational priorities, or business logic relationships. This results in OCR-extracted data lacking structure and interpretability, making it difficult to use directly for automated decision-making.

[0011] Tensorway's comprehensive research further points out that OCR performs particularly poorly when processing tables, lists, and complex layouts. Mobile application interfaces have high information density and complex layout hierarchies, and relying solely on OCR cannot restore the hierarchical relationships and interaction logic of elements, causing the extracted data to lose its original structured features.

[0012] 3. Lack of self-repair ability Existing solutions lack autonomous diagnosis and repair mechanisms when data collection fails, leaving users to passively wait for manual intervention. This results in prolonged data collection interruptions and impacts business continuity. This issue has attracted widespread attention in the fields of distributed systems and automated testing.

[0013] In their review of self-healing systems, Ghosh and Khataa point out that traditional data acquisition systems employ a passive "detect-alarm-manual repair" model, resulting in a mean time to failure (MTTR) of 2-4 hours. In critical business scenarios, this delay can cause significant losses. For example, if an e-commerce platform's price monitoring system experiences a data acquisition interruption, it may miss competitors' limited-time promotional information, directly impacting the timeliness of its pricing strategy.

[0014] Lack of Failure Mode Recognition Capabilities: Research by Cardozo and Dusparic shows that existing systems cannot automatically analyze the causes of failures, only recording error logs for manual investigation. In reality, the failure modes of mobile application data collection are limited and predictable, mainly including: element location failure, network timeout, session expiration, CAPTCHA interception, and anti-scraping mechanism triggering. However, traditional solutions lack the ability to intelligently identify and classify these modes, and cannot adopt corresponding recovery strategies for different failure types.

[0015] Lack of Adaptive Recovery Mechanisms: Saeed et al., in their research on self-healing mobile networks, proposed that systems should possess the ability to dynamically adjust their behavior based on their operational status. UiPath's Healing Agent, launched in 2025, represents the latest advancement in the industry. This system uses AI to analyze UI changes and automatically generates new element location strategies, resuming automated processes in most cases without human intervention. However, the core technologies of such commercial solutions are not fully disclosed in academia, and they primarily target web automation scenarios, with limited support for mobile applications.

[0016] Lack of Learning and Evolutionary Capabilities: HackerNoon's report on automated self-healing testing emphasizes that an ideal self-healing system should be able to learn from historical failures and continuously optimize its remediation strategies. However, most existing data collection systems are "stateless," requiring troubleshooting from scratch for each failure and failing to accumulate experience and knowledge. The design principles proposed by the Microsoft Azure Architecture Center state that self-healing capabilities should include four closed-loop components: real-time fault detection, intelligent root cause analysis, automated remediation execution, and continuous optimization feedback. Current mobile application data collection systems generally only possess the first component.

[0017] GeeksforGeeks' technical analysis further points out that the core of a self-healing system lies in "autonomy" and "adaptability." The system needs to be able to detect anomalies, diagnose problems, formulate recovery plans, and execute repair operations without human intervention, ultimately restoring the system to normal operation. This requires an organic combination of reasoning, decision-making, and execution capabilities, a level of intelligence that traditional rule-based or template-based methods struggle to achieve.

[0018] 4. Low resource utilization efficiency In mobile interface data acquisition, traditional solutions suffer from low resource utilization efficiency, primarily due to severe resource waste, redundant computation, and inefficient interface element recognition strategies. On one hand, a large amount of invalid or repetitive computation consumes valuable CPU and memory resources without generating new, useful information. A typical example is repeatedly performing useless calculations on recurring zero-value data, leading to wasted memory and computing resources. On the other hand, some interface recognition schemes are inefficient; for instance, indiscriminately scanning and recognizing the entire interface wastes computing resources on areas with low information content. These problems not only slow down data acquisition but also increase energy consumption on mobile devices. Therefore, to improve resource utilization efficiency, it is crucial to optimize algorithms to eliminate redundant computation, reduce unnecessary resource overhead, and adopt more intelligent interface recognition strategies to avoid the aforementioned waste.

[0019] 5. Lack of intelligent decision-making ability Existing automation technologies generally lack intelligent decision-making capabilities in complex task execution scenarios. The core problem lies in the system's inability to dynamically assess and generate strategies based on context. Traditional automation systems rely on predefined rules or fixed scripts to execute tasks, mechanically completing operation sequences without any awareness of business semantics or execution status. This "non-cognitive" execution method prevents the system from autonomously assessing feasibility or generating alternative decision paths when faced with environmental changes (such as interface structure updates, abnormal event interference, or task objective adjustments), often leading to task interruptions or error accumulation. In contrast, intelligent automation systems emphasize introducing contextual understanding and reasoning mechanisms during task execution, enabling adaptive decision-making and dynamic recovery based on real-time status. Therefore, improving the intelligent decision-making capabilities of automation systems is not only a key path to achieving high reliability and high generalization ability but also a fundamental shift from "static execution" to "semantic-driven decision-making." Summary of the Invention

[0020] To address the problems existing in the prior art, this invention provides a mobile application interface data acquisition system, method, device, and storage medium, which solves the problems of fragile rules, inefficient acquisition, and lack of self-repair in traditional solutions, and achieves high-precision, low-cost, and continuous availability of data acquisition.

[0021] To achieve the above objectives, the present invention provides the following solution: A mobile application interface data acquisition system, comprising: The interface information acquisition module is used to acquire the UI tree structure data of the current interface of the mobile application through the system-level interface, calculate the visibility percentage of each node in the visible area of ​​the screen, retain nodes whose visibility percentage is not lower than the preset visibility threshold, and output the set of valid nodes. The regional weight evaluation module, connected to the interface information acquisition module, is used to calculate a comprehensive weight for each node in the effective node set based on the business target parameters. This weight is composed of a weighted sum of business importance, collection frequency requirements, and semantic relevance. The module maps the nodes to discrete value levels and determines the maximum usable identification level for each node based on the value level. The rule matching module, connected to the region weight evaluation module, is used to unconditionally perform rule matching on all valid nodes based on the rule base, calculate and output the first-level identification confidence of each node; The visual recognition module, connected to the rule matching module and the confidence fusion and decision control module, is used to perform visual recognition on the current node and output the second layer of recognition confidence when activated by the confidence fusion and decision control module; The semantic understanding module, connected to the visual recognition module and the confidence fusion and decision control module, is used to call the visual language model to perform semantic analysis on the current node and output the third layer of recognition confidence when activated by the confidence fusion and decision control module. The confidence fusion and decision control module is connected to the rule matching module, visual recognition module, semantic understanding module, and self-healing and rule optimization module, respectively. It is used to fuse the confidence of each layer of recognition completed at the current recognition level. When the fused confidence meets the preset confidence threshold, the recognition result is output. When the fused confidence is insufficient and the current level has not reached the maximum usable recognition level, the next layer recognition module is activated. When the fused confidence is insufficient and the maximum usable recognition level has been reached, a trigger signal is sent to the self-healing and rule optimization module. The self-healing and rule optimization module is connected to the confidence fusion and decision control module. When a trigger signal is received, or the number of consecutive rule failures reaches a preset threshold, or the confidence of the rule is lower than a preset confidence lower limit threshold, the optimal candidate node is extracted from the current interface structure based on the weighted score of semantic similarity, positional similarity and visual feature similarity, new rules are generated, and the rule base is updated after the verification accuracy reaches a preset verification threshold. The data output module, connected to the confidence fusion and decision control module, is used to output structured collection results containing field values, fusion confidence, collection timestamps, and actual identification levels.

[0022] Preferably, the visible percentage is calculated as follows: in, The node's screen coordinate range, The coordinate range of the visible area on the device screen; the preset visibility threshold is 1%; the UI tree is extracted using a depth-first, child-to-parent recursive traversal strategy.

[0023] As a preferred option, the importance of the business Calculated in the following segmentation method: in, Explicitly specify the set of elements for business operations determined by element ID, element description attributes, XPath structure, or regular expression rule matching; Preferably, the semantic relevance Calculate as follows: in, The embedding vector is obtained by mapping the node text content through a text encoding model. For business objective vectors, Embed vectors for business objectives stored in the vector database; when the node content is an image, extract the text first through optical character recognition and then calculate the embedding vector.

[0024] Preferably, the method for classifying the value levels is as follows: The maximum available identification level The correspondence is as follows: high-value areas correspond to The corresponding mid-value area Low-value areas correspond to .

[0025] Preferably, the rule confidence level is calculated as follows: in, For rules The number of historical successful executions, The number of historical failed executions is updated after each step S6 outputs the identification result; the preset number threshold is 3 times, and the preset confidence lower limit threshold is 0.6.

[0026] Preferably, the semantic similarity, positional similarity, and visual feature similarity are calculated as follows: in, For the semantic vector of the target field, The average coordinate of historical successful location. The length of the screen diagonal. For perceptual hash functions, The hash bit length is [value]; the comprehensive score for candidate nodes is: Pick The largest candidate node is generated using new rules.

[0027] This invention also provides a method for collecting mobile application interface data, comprising: S1. Obtain the UI tree structure data of the current interface of the mobile application through the system-level interface, calculate the visibility percentage of each node in the UI tree within the visible area of ​​the screen, and retain nodes whose visibility percentage is not lower than a preset visibility threshold to form a valid node set. ; S2, For the set of valid nodes Each node in According to business target parameters Calculate the overall weight: in Due to the importance of node services, To meet the frequency requirements, The semantic relevance between node content and business objectives; S3, Based on comprehensive weighting Each node is mapped to a discrete value level, and the maximum usable identification level for each node is determined based on the value level. The higher the value level, the corresponding The larger the upper limit; S4. Perform rule matching based on the rule base unconditionally on all nodes, calculate and output the first-level identification confidence score for each node. ; S5, to Below the preset confidence threshold and For nodes with a value greater than 1 (maximum available recognition level greater than the first level), activate visual recognition and calculate the confidence score for the second layer of recognition. ;right Still below and For the highest-level node, activate semantic understanding recognition and calculate the third-layer recognition confidence score. ; S6. For each node at the current identification level The following formula has been used to complete the identification confidence of each layer: when Output the current recognition result when; and Activate the next layer of recognition and update Then return to fusion calculation; when and The time marker has insufficient confidence and triggers step S7; S7. When the number of consecutive recognition failures of a rule reaches a preset threshold, or the confidence of a rule continues to be lower than the preset confidence threshold, candidate nodes are extracted from the current interface structure. The optimal candidate node is selected based on the weighted score of semantic similarity, positional similarity and visual feature similarity between the candidate node and the target field. A new rule is generated and the rule base is updated after the accuracy is verified to reach the preset verification threshold. S8. Outputs a structured collection result containing field values, fusion confidence, collection timestamp, and actual recognition level.

[0028] The present invention also provides a mobile application interface data acquisition device, comprising: a memory and a processor, wherein the memory stores a computer program executed by the processor, and the computer program executes a mobile application interface data acquisition method when executed by the processor.

[0029] The present invention also provides a storage medium storing a computer program, which executes a mobile application interface data acquisition method when running.

[0030] Compared with the prior art, the beneficial effects of the present invention are as follows: By constructing a closed-loop control system of "structural understanding, value quantification, hierarchical identification, confidence control, and self-healing feedback", the data acquisition system is equipped with the ability to adapt to interface changes, optimize resource allocation, and autonomously recover from abnormal scenarios, thus realizing the transformation from static rule execution to dynamic intelligent decision-making. Attached Figure Description

[0031] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a structural diagram of the mobile application interface data acquisition system according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the regional value weight calculation structure; Figure 3 This is a schematic diagram of a hierarchical identification mechanism; Figure 4 Flowchart for self-healing and rule optimization Figure 5 This is a flowchart of a mobile application interface data collection method according to an embodiment of the present invention. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0035] Example 1 like Figures 1 to 4 As shown, the present invention provides a mobile application interface data acquisition system, comprising: The interface information acquisition module is used to acquire the UI tree structure data of the current interface of the mobile application through the system-level interface, calculate the visibility percentage of each node in the visible area of ​​the screen, retain nodes whose visibility percentage is not lower than the preset visibility threshold, and output the set of valid nodes. The regional weight evaluation module, connected to the interface information acquisition module, is used to calculate a comprehensive weight for each node in the effective node set based on the business target parameters. This weight is composed of a weighted sum of business importance, collection frequency requirements, and semantic relevance. The module maps the nodes to discrete value levels and determines the maximum usable identification level for each node based on the value level. The rule matching module, connected to the region weight evaluation module, is used to unconditionally perform rule matching on all valid nodes based on the rule base, calculate and output the first-level identification confidence of each node; The visual recognition module, connected to the rule matching module and the confidence fusion and decision control module, is used to perform visual recognition on the current node and output the second layer of recognition confidence when activated by the confidence fusion and decision control module; The semantic understanding module, connected to the visual recognition module and the confidence fusion and decision control module, is used to call the visual language model to perform semantic analysis on the current node and output the third layer of recognition confidence when activated by the confidence fusion and decision control module. The confidence fusion and decision control module is connected to the rule matching module, visual recognition module, semantic understanding module, and self-healing and rule optimization module, respectively. It is used to fuse the confidence of each layer of recognition completed at the current recognition level. When the fused confidence meets the preset confidence threshold, the recognition result is output. When the fused confidence is insufficient and the current level has not reached the maximum usable recognition level, the next layer recognition module is activated. When the fused confidence is insufficient and the maximum usable recognition level has been reached, a trigger signal is sent to the self-healing and rule optimization module. The self-healing and rule optimization module is connected to the confidence fusion and decision control module. When a trigger signal is received, or the number of consecutive rule failures reaches a preset threshold, or the confidence of the rule is lower than a preset confidence lower limit threshold, the optimal candidate node is extracted from the current interface structure based on the weighted score of semantic similarity, positional similarity and visual feature similarity, new rules are generated, and the rule base is updated after the verification accuracy reaches a preset verification threshold. The data output module, connected to the confidence fusion and decision control module, is used to output structured collection results containing field values, fusion confidence, collection timestamps, and actual identification levels.

[0036] The system of this invention comprises eight functional units, which are formally represented as mapping functions f1 to f8. The following "module" refers to the engineering description of the corresponding function implementation unit.

[0037] The entire system can be represented as a state mapping: in: This represents the interface structure data; This indicates the business objectives and data collection strategy parameters; This represents the historical rule base and runtime logs; This indicates the structured data collection results.

[0038] Since the activation of subsequent modules depends on the confidence output of preceding modules, the system is not a sequentially concatenated pipeline, but rather a conditional mapping combination. The three recognition sub-paths are defined as set functions: in, , Whether a node is activated is determined by both its value level and confidence threshold (see Modules 4 and 6 for details). The overall system mapping is as follows: The mapping relationships and activation conditions of the eight functional modules will be explained one by one in the following sections.

[0039] Module 1: Interface Information Acquisition Module ) 1.1 Functional Positioning The UI tree structure data of the current interface of the mobile application is obtained through the hardware system-level interface to form a structured interface tree, which serves as the basic input data source for all subsequent modules.

[0040] This module runs in the following hardware execution environments: Android devices call the AccessibilityNodeInfo interface via ADB (Android DebugBridge); iOS devices call it via the UIAccessibility framework interface. These interface calls are existing technologies.

[0041] The processing flow of this module is as follows: First, define the input and output interfaces (1.2); then, obtain all nodes through depth-first recursive traversal (1.3); finally, remove invalid nodes through visibility filtering and output the set of valid nodes (1.4). 1.3 and 1.4 constitute a two-stage processing pipeline executed sequentially. 1.2 Input and Output Input: Original interface access data Output: Structured interface tree ,in For the set of valid nodes, This is a set of parent-child relationships.

[0042] Each node is defined as a quadruple: in, For element type, To display the text, The range of screen coordinates. This extends the attribute set (including state fields such as isVisible, isClickable, resourceId, and className).

[0043] 1.3 Recursive Traversal Strategy The interface tree employs a depth-first, child-to-parent recursive traversal strategy. Let the current node be... Its child node set is traversal function Defined as: in Indicates a node Perform attribute extraction and output node JSON structured data. The child-to-parent order ensures that child node attributes are written before parent node attributes, facilitating subsequent hierarchical relationship construction.

[0044] 1.4 Visibility Filtering Function During the traversal, the visibility percentage of each node is calculated, and invalid nodes are filtered out.

[0045] Set nodes The screen coordinates are The visible area of ​​the device screen is ,but: Filtering rules: in, Visibility threshold, default. (That is, at least 1% of the node's area is within the visible area of ​​the screen). The final output is the set of valid nodes that satisfy the retention criteria. .

[0046] Innovation Description: The innovation of this module lies in the hierarchical visibility filtering mechanism based on intersection area calculation, and the child-to-parent traversal order design, which effectively reduces the interference of invalid nodes on subsequent identification. System-level interface calls are existing technologies.

[0047] Module 2: Regional Weight Evaluation Module ) 2.1 Functional Positioning For the set of valid nodes The business value of each node is quantified, and a weighted set of nodes is output. This serves as the resource scheduling and control basis for subsequent hierarchical identification strategies.

[0048] The processing flow of this module is as follows: Based on the input and output definitions (2.2), the business value of each node is calculated through the comprehensive weight function (2.3), where (1), (2), and (3) are the three weighted components of the comprehensive weight. Finally, the value range is divided according to the weight results (2.4). The three sub-functions of 2.3 are calculated in parallel and weighted summed. 2.4 uses the output of 2.3 as the basis for classification.

[0049] 2.2 Input and Output Input: Set of valid nodes Business target parameters Output: Weighted set of nodes 2.3 Comprehensive Weighting Function Constraints: ,and (1) Business Importance in The set of elements explicitly specified for business rules, determined by element ID, element description, XPath structure, or regular expression rule matching; Embed vectors for business objectives stored in the vector database; Let be the cosine similarity.

[0050] (2) Acquisition frequency requirements Based on business strategy parameters Direct configuration reflects the urgency of data collection for this field, including its value range. .

[0051] in For fields Configure the data collection interval (unit: seconds). , The longest and shortest acquisition intervals within the system are determined by logarithmic normalization to ensure the monotonicity of the mapping.

[0052] (3) Semantic relevance in The embedding vector is obtained by mapping the node text content through a text encoding model. This represents the business target vector. If the node content is an image, the text is first extracted using OCR before the embedding is calculated.

[0053] 2.4 Value Range Division The calculation results directly determine the results in module 6. The value of controls the activation range of subsequent recognition modules.

[0054] This step assesses value based on structured interface tree data, node meta-attributes, and lightweight text features, without relying on high-cost visual recognition or semantic reasoning.

[0055] Module 3: Rule Matching Module ) 3.1 Functional Positioning Based on historical rule base Perform fast localization on weighted nodes and output the first-level recognition result and confidence score. Rule matching is the recognition path with the lowest computational cost and is performed unconditionally on all valid nodes.

[0056] The processing flow of this module is as follows: Based on the input and output definitions (3.2), rule matching is performed on each node one by one, and the confidence score is calculated (3.3). 3.2 defines the weighted node set and rule base as input, and 3.3 defines the multi-attribute weighted matching structure and confidence score calculation method for each rule. In the case where the same node may match multiple rules, the rule with the highest matching confidence score is taken as the first-level recognition result output for that node.

[0057] 3.2 Input and Output Input: a set of weighted nodes rule base Output: First-level matching results 3.3 Rule Structure and Matching Confidence Each rule It consists of several attribute matching conditions: in, To match attribute names (which can be resourceId, className, text, xpath, description, etc.), For the corresponding expected value or regular expression pattern, The matching weight for this attribute, satisfying .

[0058] node With rules The matching confidence level is: in, For indicator functions, It supports both exact matching and regular expression matching modes.

[0059] node The final confidence score of the first layer is the maximum value among all rules: Innovation Description: Rule attribute types and regular expression matching are existing technologies. The innovation of this module lies in the confidence-weighted scoring mechanism and the system integration method of using rule matching results as the entry signal for hierarchical scheduling.

[0060] Module 4: Visual Recognition Module ) 4.1 Functional Positioning For nodes whose confidence level in the first layer does not reach the threshold, OCR and image feature methods are used to perform the second layer of recognition.

[0061] Activation conditions: and This means that the node belongs to a medium- or high-value area.

[0062] The processing flow of this module is as follows: Based on the input / output definition (4.2), text is extracted from the node screenshot area using the OCR recognition function, and the second-layer recognition confidence score is calculated (4.3). 4.2 defines the interface screenshot and node coordinates as inputs, and 4.3 defines the specific calculation method of the OCR recognition function, including text extraction, similarity comparison, and confidence score output. This module is only conditionally activated within the iterative control loop of module 6 and does not run independently.

[0063] 4.2 Input and Output Input: Node image region (From the screenshot) (Crop and extract) Output: Recognized text Second layer confidence 4.3 Identification Function and Confidence Calculation Text recognition calls upon an existing OCR engine: When the target field contains the desired text pattern At that time, the confidence score is calculated based on the edit distance: in Edit the distance for Levenshtein. The desired character length of the text.

[0064] When the target field has no fixed expected value, the confidence level degenerates into the OCR engine's own character-level confidence average: Innovation Note: OCR is a mature existing technology. This module is activated only when the confidence level of the first layer is insufficient and the node value level meets the conditions, and it belongs to the execution unit of the hierarchical identification and scheduling system.

[0065] Module 5: Semantic Understanding Module ) 5.1 Functional Positioning For high-value nodes whose confidence level in the first two layers has not yet reached the threshold, the Visual Language Model (VLM) is invoked to make a third-layer semantic decision.

[0066] Activation conditions: and This means that the node belongs to a high-value area.

[0067] The processing flow of this module is as follows: Based on the input / output definition (5.2), firstly, the structure of the called visual language model is explained (5.3), then the task decision function and confidence calculation method based on the model output are defined (5.4), and finally, the design of four types of standardized prompt word templates is given (5.5). The logical relationship between each section is as follows: 5.3 provides the foundation of model capabilities, 5.4 defines how to use the model output to calculate the decision results and the third-level confidence, and 5.5 provides specific input templates for the model call in 5.4. 5.4 and 5.5 together realize the complete link from model inference to structured output. This module is only conditionally activated in the iterative control loop of module 6 and does not run independently.

[0068] 5.2 Inputs and Outputs Input: Screenshot of the current interface Task target parameters Node context information Output: Action space probability distribution Third level of confidence 5.3 Model Structure Description This module calls a visual language model based on the Transformer architecture (existing technology), whose core attention calculation is as follows: in , , These are query, key, and value matrices, respectively. The dimension of the key vector.

[0069] 5.4 Task Decision Function The model receives the task context through standardized prompt word templates and outputs the action space. The probability distribution on: Action space Includes atomic operation types such as: click, swipe, scroll, text input, system operations (return to / Home / task list), etc.

[0070] 5.5 Standardized prompt word template The general structure of the prompt word template is as follows: Context injection is achieved through variable placeholder replacement. Four standard templates are defined, as shown in Table 1: Table 1 , Used for the normal execution path. , The exception recovery path is called in different decision branches of module 6.

[0071] The third level of confidence is the maximum value of the model's output probability distribution. Innovation Description: The Transformer structure and attention mechanism of VLM are existing technologies. The innovation of this module lies in its layered triggering and scheduling mechanism, as well as the engineering design of four types of standardized prompt words for mobile operation scenarios.

[0072] Module 6: Confidence Fusion and Hierarchical Decision Control Module ) 6.1 Functional Positioning This module is the core control unit for the system's hierarchical identification and scheduling, and it undertakes the following three responsibilities: 1. Determine the upper limit of available identification levels based on node value levels. 2. Perform confidence fusion at the current level and output the result. 3. Determine whether to upgrade to the next level of recognition or output the result directly. The above three items constitute an iterative control loop: Step 2 and Step 3 are executed sequentially within the same loop body, and the loop terminates when "the confidence level meets the threshold" or "the maximum available level has been reached".

[0073] The processing flow of this module is as follows: Based on the input and output definitions (6.2), the maximum usable recognition level of each node is first determined according to the node value level (6.3), then the iterative control loop is entered (6.4), and finally the complete recognition path is formally described using equivalent mathematical expressions (6.5). The logical relationship between each section is as follows: 6.3 provides the control boundary for 6.4—the upper limit of the iteration of each node is determined by the output of 6.3; within this boundary, 6.4 executes the loop logic of "fusion confidence—judging whether it meets the standard—deciding to upgrade or output"; 6.5 is the equivalent expansion form of 6.4, used to help understand the mathematical structure of the complete recognition path. The two describe the same logic and do not constitute repetition.

[0074] 6.2 Inputs and Outputs Input: node Confidence levels at each level ( ), Node Value Level Output: Final confidence level Current recognition results Hierarchical status 6.3 Maximum Available Recognition Level 6.4 Iterative Control Loop For nodes The recognition process is executed iteratively in the following order: Step A — Confidence fusion (at the current level) Down): Weight The results increase monotonically with increasing level, indicating that the identification results for higher levels have higher credibility.

[0075] Step B — Level Upgrade Determination (Based on Step A Results): in The confidence threshold is the default value. .

[0076] like Then the corresponding level recognition module will be activated. or ),renew Then return to Step A to re-merge.

[0077] like If the loop terminates, the current result will be output.

[0078] like and If the maximum available level has been reached but the threshold is still not met, the system outputs the current best result and marks it as insufficient with a confidence level, triggering... Self-healing process.

[0079] Step C — Output the recognition results: That is, the recognition output of the level at which the loop terminates is taken as the final result.

[0080] The above iterative process corresponds to the fusion and upgrade logic of step S6 in claim 6.

[0081] 6.5 Equivalent Representation of Complete Recognition Path The above iterative process, after expansion, is equivalent to the following piecewise function, used to intuitively express the activation conditions of each path: Technical Note: The iterative control loop in section 6.4 and the piecewise expansion in section 6.5 express the same logic. Section 6.5 is an equivalent expansion of section 6.4 for ease of review and understanding. They are not redundant; section 6.4 describes runtime execution logic, while section 6.5 is a formal induction of the complete path.

[0082] Module 7: Self-Healing and Rule Optimization Module ) 7.1 Functional Positioning When identification fails or rule confidence continues to decline, the system automatically diagnoses the cause of failure, generates candidate rules, and verifies and updates them, enabling dynamic evolution of the rule base without manual intervention and ensuring the robustness of the system's long-term operation.

[0083] The processing flow of this module is as follows: Based on the input and output definitions (7.2), firstly, the rule confidence level is maintained based on historical execution records (7.3), then it is determined whether the self-healing trigger condition is met (7.4). If triggered, the candidate elements are scored in multiple dimensions (7.5). After selecting the optimal candidate, a new rule is generated and updated after verification (7.6). Finally, all rules are managed in terms of health level and the inspection cycle is adjusted (7.7). The logical relationship between each section is as follows: 7.3 provides the basis for judgment in 7.4—the rule confidence level and the number of consecutive failures determine whether self-healing is triggered; after triggering in 7.4, the candidate scoring process in 7.5 is entered; the output of 7.5 provides the optimal candidate node for 7.6; after 7.6 completes the rule replacement, 7.7 is responsible for the long-term health maintenance and threshold adaptive adjustment of the rule base as a whole. 7.3 to 7.7 constitute a complete self-healing pipeline of "monitoring—triggering—scoring—replacement—maintenance".

[0084] 7.2 Inputs and Outputs Input: Historical running logs Current interface structure Target field Output: Updated rule base 7.3 Rule Confidence and Update Timing Set rules The number of successful executions in the historical execution record is The number of failures is The rule confidence level is defined as: Update timing: After each node identification result is output by module 6, the rules actually called in this identification are updated. Perform a count update—if the identification result is verified to be correct, then If identification fails or is marked by the self-healing module, then .

[0085] 7.4 Self-healing trigger conditions in For rules The number of consecutive failures, This is the threshold for the number of consecutive failures (default 3 times). This is the lower confidence threshold (default 0.6).

[0086] 7.5 Candidate Element Scoring Function when At that time, extract candidate elements from the current interface structure and score them: Constraints: , Definitions of each sub-function: semantic similarity Cosine similarity between the node text embedding vector and the target field semantic vector: Location similarity The spatial proximity of a node's location to its historically successfully located locations is measured using the inverse function of the normalized Euclidean distance. in The average of historically successful coordinates. The screen diagonal length (normalization factor).

[0087] Visual feature similarity Perceptual hash similarity between the node image region and historically successfully located regions: in For perceptual hash functions, For hash bit length, This is the Hamming distance.

[0088] 7.6 New Rule Generation and Verification Select the candidate node with the highest score The module 5 (semantic understanding module) is invoked to analyze its features and generate new rules: in Multidimensional features such as text regularity patterns, spatial relationships, and visual feature hashes are extracted from the VLM analysis results. This represents the initial confidence level.

[0089] The new rules will be validated before they are officially replaced: in This is the threshold for rule validation, with a default value of 0.8. If validation fails, the original rule is retained, and a self-healing failure log is recorded, waiting for retry in the next cycle.

[0090] 7.7 Rule-based Health Level Management The system according to The rules are maintained in a hierarchical manner, and the inspection cycle is dynamically adjusted, as shown in Table 1: Table 2: Meanwhile, the system calculates the global data collection failure rate. (Percentage of all nodes failing to identify within the statistical period) Adaptively adjust the trigger threshold: in , The default initial value, To adjust the step size, the system can proactively trigger self-healing in advance when the data acquisition environment continues to deteriorate.

[0091] Module 8: Data Output Module ) 8.1 Functional Positioning The final identification results from each node are aggregated, and the structured collected data is output in a formatted manner for consumption by the upper-layer business system.

[0092] The processing flow of this module is as follows: It receives all node identification results output by module 6, encapsulates them according to the six-tuple format defined in the output data structure (8.2), and summarizes them into a complete set of structured acquisition results. 8.2 defines the field names, identification values, fusion confidence levels, timestamps, actual identification levels, and self-healing logs contained in each output record, ensuring that downstream systems can obtain complete metadata about the acquisition process.

[0093] 8.2 Output Data Structure Each output record is defined as a six-tuple: in: Field names and element identifiers; : Identify the obtained field values; : The final confidence level after fusion; : Collection timestamp; The actual recognition level used in this data collection. ; If this data collection triggers the self-healing process, record a self-healing event summary (including failure mode, generated rule summary, and verification accuracy); otherwise, leave it empty. The data collection results for all nodes are summarized as follows: Overview of Module Logical Relationships The mapping relationships and activation conditions of the eight functional modules are summarized as follows: Closed-loop structure description: This constitutes an unconditional forward path; , by The hierarchical decision conditions are activated, forming conditional forward branches; Continuous monitoring The output is asynchronously updated and written back when the triggering condition is met. This forms an asynchronous feedback loop; The final outputs from all paths converge. The three paths together constitute the closed-loop control structure of the system.

[0094] The technological innovation of this application does not lie in the individual use of existing recognition technologies such as rule matching, optical character recognition, or visual language models, but rather in the construction of a value-driven hierarchical recognition scheduling and closed-loop adaptive control system. In existing technologies, rule matching, OCR recognition, or interface understanding based on visual language models are usually used as independent solutions or in simple series combinations, with the goal of improving recognition accuracy. However, such solutions typically fail to address the issue of how to dynamically allocate recognition resources according to business value, nor do they form a unified confidence control and feedback closed-loop mechanism.

[0095] This application constructs a system-level control structure that differs from existing technologies through the following technical features: 1. Value assessment results serve as scheduling control signals: The value levels output by the regional value assessment module are not used for sorting or simple filtering, but rather as control signals for resource scheduling, dynamically constraining the maximum available identification level for each node. Different value levels correspond to different upper limits for identification paths, allowing computing resources to be concentrated in high-value areas and avoiding the use of high-cost identification methods in low-value areas.

[0096] 2. Iterative Control Mechanism for Confidence Fusion and Layer Upgrade: The confidence fusion module fuses the recognition confidence of each layer at the current layer and decides whether to upgrade to the next layer based on a preset threshold. This process forms a terminateable iterative control loop, rather than a fixed sequential execution flow. The triggering condition for recognition upgrade is constrained by both the "current fusion confidence" and the "maximum available recognition layer," forming a dynamic scheduling boundary.

[0097] 3. Cooperative mechanism of maximum availability level constraint and self-healing trigger: When a node has reached the maximum available recognition level corresponding to its value level and the confidence level is still insufficient, the system no longer blindly increases the recognition complexity, but instead switches to a self-healing and rule optimization process. This design avoids the waste of resources caused by infinitely upgrading recognition and incorporates abnormal states into the feedback loop for structural repair.

[0098] 4. Rule Health Statistics and Threshold Adaptive Adjustment Mechanism: Rule confidence is dynamically updated based on historical success and failure rates, and trigger thresholds are adaptively adjusted in conjunction with the global collection failure rate, enabling the self-healing mechanism to continuously evolve. This mechanism establishes a linkage between runtime statistical control and structural repair.

[0099] Through the synergistic effect of the aforementioned technical features, a closed-loop control structure of "value-driven—confidence fusion—hierarchical constraints—anomaly feedback—rule evolution" is formed. This structure is not a simple splicing or parallel combination of existing identification technologies, but rather a new system-level operating mechanism built on identification resource scheduling, control boundary setting, and anomaly handling paths.

[0100] Therefore, this application achieves a dynamic balance between recognition capability, resource consumption, and system robustness, which is significantly different from traditional static rule enforcement or single recognition technology solutions.

[0101] The present invention achieves the following technical effects: Optimized resource allocation: Weighting assessment and of Constrained collaborative work ensures that low-value nodes only perform rule matching, avoiding calls to OCR or VLM for decorative elements and significantly reducing computational resource consumption.

[0102] Layered recognition control: The iterative control loop in the algorithm concentrates identification resources on high-value, low-confidence nodes, achieving a dynamic balance between identification accuracy and computational efficiency, which is different from the traditional scheme that scans all nodes uniformly.

[0103] Dynamic rule evolution: The self-healing mechanism enables the system to extract features from failure modes, generate and verify new rules autonomously without manual intervention after changes to the mobile application interface. The rule base continues to evolve, and maintenance costs are significantly reduced.

[0104] Adaptive recovery from abnormal scenarios: Standardized prompt word template ( , )and The combined effect of the self-healing process supports autonomous diagnosis and recovery from abnormal scenarios such as dynamic pop-ups, interface reconstruction, and element ID changes, realizing a fundamental shift from "passive alarm waiting for manual intervention" to "proactive diagnosis and autonomous recovery".

[0105] Example 2 like Figure 5 As shown, the present invention also provides a method for collecting mobile application interface data, comprising: S1. Obtain the UI tree structure data of the current interface of the mobile application through the system-level interface, calculate the visibility percentage of each node in the UI tree within the visible area of ​​the screen, and retain nodes whose visibility percentage is not lower than a preset visibility threshold to form a valid node set. ; S2, For the set of valid nodes Each node in According to business target parameters Calculate the overall weight: in Due to the importance of node services, To meet the frequency requirements, The semantic relevance between node content and business objectives; S3, Based on comprehensive weighting Each node is mapped to a discrete value level, and the maximum usable identification level for each node is determined based on the value level. The higher the value level, the corresponding The larger the upper limit; S4. Perform rule matching based on the rule base unconditionally on all nodes, calculate and output the first-level identification confidence score for each node. ; S5, to Below the preset confidence threshold and For nodes with a value greater than 1 (maximum available recognition level greater than the first level), activate visual recognition and calculate the confidence score for the second layer of recognition. ;right Still below and For the highest-level node, activate semantic understanding recognition and calculate the third-layer recognition confidence score. ; S6. For each node at the current identification level The following formula has been used to complete the identification confidence of each layer: when Output the current recognition result when; and Activate the next layer of recognition and update Then return to fusion calculation; when and The time marker has insufficient confidence and triggers step S7; S7. When the number of consecutive recognition failures of a rule reaches a preset threshold, or the confidence of a rule continues to be lower than the preset confidence threshold, candidate nodes are extracted from the current interface structure. The optimal candidate node is selected based on the weighted score of semantic similarity, positional similarity and visual feature similarity between the candidate node and the target field. A new rule is generated and the rule base is updated after the accuracy is verified to reach the preset verification threshold. S8. Outputs a structured collection result containing field values, fusion confidence, collection timestamp, and actual recognition level.

[0106] Example 3 The present invention also provides a mobile application interface data acquisition device, comprising: a memory and a processor, wherein the memory stores a computer program executed by the processor, and the computer program executes a mobile application interface data acquisition method when executed by the processor.

[0107] Example 4 The present invention also provides a storage medium storing a computer program, which executes a mobile application interface data acquisition method when running.

[0108] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A mobile application interface data acquisition system, characterized in that, include: The interface information acquisition module is used to acquire the UI tree structure data of the current interface of the mobile application through the system-level interface, calculate the visibility percentage of each node in the visible area of ​​the screen, retain nodes whose visibility percentage is not lower than the preset visibility threshold, and output the set of valid nodes. The regional weight evaluation module, connected to the interface information acquisition module, is used to calculate a comprehensive weight for each node in the effective node set based on the business target parameters. This weight is composed of a weighted sum of business importance, collection frequency requirements, and semantic relevance. The module maps the nodes to discrete value levels and determines the maximum usable identification level for each node based on the value level. The rule matching module, connected to the region weight evaluation module, is used to unconditionally perform rule matching on all valid nodes based on the rule base, calculate and output the first-level identification confidence of each node; The visual recognition module, connected to the rule matching module and the confidence fusion and decision control module, is used to perform visual recognition on the current node and output the second layer of recognition confidence when activated by the confidence fusion and decision control module; The semantic understanding module, connected to the visual recognition module and the confidence fusion and decision control module, is used to call the visual language model to perform semantic analysis on the current node and output the third layer of recognition confidence when activated by the confidence fusion and decision control module. The confidence fusion and decision control module is connected to the rule matching module, visual recognition module, semantic understanding module, and self-healing and rule optimization module, respectively. It is used to fuse the confidence of each layer of recognition completed at the current recognition level. When the fused confidence meets the preset confidence threshold, the recognition result is output. When the fused confidence is insufficient and the current level has not reached the maximum usable recognition level, the next layer recognition module is activated. When the fused confidence is insufficient and the maximum usable recognition level has been reached, a trigger signal is sent to the self-healing and rule optimization module. The self-healing and rule optimization module is connected to the confidence fusion and decision control module. When a trigger signal is received, or the number of consecutive rule failures reaches a preset threshold, or the confidence of the rule is lower than a preset confidence lower limit threshold, the optimal candidate node is extracted from the current interface structure based on the weighted score of semantic similarity, positional similarity and visual feature similarity, new rules are generated, and the rule base is updated after the verification accuracy reaches a preset verification threshold. The data output module, connected to the confidence fusion and decision control module, is used to output structured collection results containing field values, fusion confidence, collection timestamps, and actual identification levels.

2. The mobile application interface data acquisition system as described in claim 1, characterized in that, The visible percentage is calculated as follows: in, The node's screen coordinate range, The coordinate range of the visible area on the device screen; the preset visibility threshold is 1%; the UI tree is extracted using a depth-first, child-to-parent recursive traversal strategy.

3. The mobile application interface data acquisition system as described in claim 1, characterized in that, The importance of the business Calculated in the following segmentation method: in, Explicitly specify the set of elements for business operations determined by element ID, element description attributes, XPath structure, or regular expression rule matching.

4. The mobile application interface data acquisition system as described in claim 1, characterized in that, The semantic relevance Calculate as follows: in, The embedding vector is obtained by mapping the node text content through a text encoding model. For business objective vectors, Embed vectors for business objectives stored in the vector database; when the node content is an image, extract the text first through optical character recognition and then calculate the embedding vector.

5. The mobile application interface data acquisition system as described in claim 1, characterized in that, The method for classifying the value levels is as follows: The maximum available identification level The correspondence is as follows: high-value areas correspond to The corresponding mid-value area Low-value areas correspond to .

6. The mobile application interface data acquisition system as described in claim 1, characterized in that, The confidence level of the rule is calculated as follows: in, For rules The number of times it has been successfully executed in history The number of historical failed executions is updated after each step S6 outputs the identification result; the preset number threshold is 3 times, and the preset confidence lower limit threshold is 0.

6.

7. The mobile application interface data acquisition system as described in claim 1, characterized in that, The semantic similarity, positional similarity, and visual feature similarity are calculated as follows: in, For the semantic vector of the target field, The average coordinate of historical successful location. The length of the screen diagonal. For perceptual hash functions, The hash bit length is [value]; the comprehensive score for candidate nodes is: Pick The largest candidate node is generated using new rules.

8. A method for collecting data from a mobile application interface, characterized in that, include: S1. Obtain the UI tree structure data of the current interface of the mobile application through the system-level interface, calculate the visibility percentage of each node in the UI tree within the visible area of ​​the screen, and retain nodes whose visibility percentage is not lower than a preset visibility threshold to form a valid node set. ; S2, For the set of valid nodes Each node in According to business target parameters Calculate the overall weight: in Due to the importance of node services, To meet the frequency requirements, The semantic relevance between node content and business objectives; S3, Based on comprehensive weighting Each node is mapped to a discrete value level, and the maximum usable identification level for each node is determined based on the value level. The higher the value level, the corresponding The larger the upper limit; S4. Perform rule matching based on the rule base unconditionally on all nodes, calculate and output the first-level identification confidence score for each node. ; S5, to Below the preset confidence threshold and For nodes with a value greater than 1 (maximum available recognition level greater than the first level), activate visual recognition and calculate the confidence score for the second layer of recognition. ;right Still below and For the highest-level node, activate semantic understanding recognition and calculate the third-layer recognition confidence score. ; S6. For each node at the current identification level The following formula has been used to complete the identification confidence of each layer: when Output the current recognition result when; and Activate the next layer of recognition and update Then return to fusion calculation; when and The time marker has insufficient confidence and triggers step S7; S7. When the number of consecutive recognition failures of a rule reaches a preset threshold, or the confidence of a rule continues to be lower than the preset confidence threshold, candidate nodes are extracted from the current interface structure. The optimal candidate node is selected based on the weighted score of semantic similarity, positional similarity and visual feature similarity between the candidate node and the target field. A new rule is generated and the rule base is updated after the accuracy is verified to reach the preset verification threshold. S8. Outputs a structured collection result containing field values, fusion confidence, collection timestamp, and actual recognition level.

9. A mobile application interface data acquisition device, characterized in that, include: A memory and a processor, wherein the memory stores a computer program executed by the processor, the computer program executing the mobile application interface data acquisition method as described in claim 8 when run by the processor.

10. A storage medium, characterized in that, The storage medium stores a computer program, which executes the mobile application interface data acquisition method as described in claim 8 when it runs.

Images