Cross-platform point-free data collection and intelligent selection rule generation method and system

Abstract

The present invention provides a cross-platform data collection method and system for generating intelligent selection rules without embedding points, which relates to the field of intelligent algorithm technology, including obtaining the view hierarchy tree information and interactive event information of page elements by injecting data collection scripts, and training a page element recognition model based on a graph neural network. The model is combined with an adaptive feature fusion network to achieve cross-platform element alignment and recognition. New elements are classified using element similarity information, and selection rules are generated and corrected based on preset rule templates and context feature vectors. Finally, the rule execution engine realizes continuous monitoring and dynamic collection of target elements in cross-platform applications according to the target selection rules.

Description

Technical Field

[0001] The present invention relates to intelligent algorithm technology, and in particular to a cross-platform method and system for collecting data without embedding points and generating intelligent selection rules. Background Art

[0002] In recent years, non-point data collection technology has gradually emerged. This technology does not require developers to manually insert point-of-use code, and can automatically collect user interaction behavior data in applications. However, the existing non-point-of-use data collection technology still has some defects and shortcomings:

[0003] Poor cross-platform compatibility: Existing pointless data collection technologies are usually designed only for specific platforms or operating systems, making it difficult to achieve compatibility between applications on different platforms, resulting in low data collection efficiency.

[0004] Low accuracy in identifying page elements: Due to differences in application interface structures and element attributes on different platforms, existing point-free data collection technologies are difficult to accurately identify target page elements, resulting in inaccurate or incomplete collected data.

[0005] Low efficiency in generating selection rules: Manually writing selection rules takes a lot of time and effort, and it is difficult to ensure the accuracy and coverage of the rules. The existing automatic selection rule generation methods are inefficient, and the generated rules often fail to meet actual needs. Summary of the invention

[0006] The embodiments of the present invention provide a cross-platform method and system for data collection without embedding points and intelligent selection rule generation, which can solve the problems in the prior art.

[0007] According to a first aspect of the embodiments of the present invention,

[0008] Provides cross-platform data collection without embedding points and intelligent selection rule generation methods, including:

[0009] Inject a data collection script into the application of the terminal device to obtain the view hierarchy tree information and interaction event information of the preset page elements;

[0010] A page element feature vector is constructed based on the view hierarchy tree information, and a multi-level feature decomposition is performed on the page element feature vector to obtain a hierarchical feature set. The hierarchical feature set is associated and mapped with the interactive event information as a training sample, and weight coefficients are assigned to different levels of features in the training sample based on a graph neural network; the training sample with the weight coefficient is input into a cross-platform element alignment matrix for transfer learning, and an adaptive feature fusion network including a feature unification module, a dimension adaptation module, and a feature fusion module is constructed; the training sample is iteratively optimized through the adaptive feature fusion network to obtain a page element recognition model, and the page element recognition model is used to calculate the similarity of the newly added page elements to obtain page element similarity information;

[0011] The newly added page elements are classified according to the page element similarity information, initial selection rules are generated for the newly added page elements of different categories based on a preset selection rule template, the cross-platform elements are located by dynamically adjusting the selector weight, and the initial selection rules are corrected based on the context feature vector, and the incremental rule verification matrix is ​​used to calculate the rule parameters to generate the target selection rules; the target selection rules are deployed to the rule execution engine, and the rule execution engine includes a rule parsing module, a rule indexing module and a rule execution module, and the target page elements in the cross-platform application are continuously monitored and dynamically collected based on the rule execution engine.

[0012] In an optional embodiment,

[0013] The steps of constructing a page element feature vector based on the view hierarchy tree information, performing multi-level feature decomposition on the page element feature vector to obtain a hierarchical feature set, associating and mapping the hierarchical feature set with the interaction event information as a training sample, and assigning weight coefficients to different hierarchical features in the training sample based on a graph neural network include:

[0014] The view hierarchy tree information includes the location information, type information and attribute information of the page elements, and the interaction event information includes the user's click operation information, sliding operation information and input operation information on the page elements;

[0015] The page element feature vector is subjected to dimensionality reduction processing by principal component analysis to obtain dimensionality reduction features, and the dimensionality reduction features are subjected to multi-level decomposition by a hierarchical clustering algorithm. The multi-level decomposition includes an appearance feature layer, a structural feature layer, and a semantic feature layer. The appearance feature layer includes color space features, geometric features, and style features encoded by local sensitive hashing, the structural feature layer includes relative position features and topological features mapped by the Graph2Vec algorithm, and the semantic feature layer includes text features, functional features, and business features extracted by the BERT model, to obtain a hierarchical feature set;

[0016] The interactive event information is encoded by using a position encoding method, the click operation is encoded as a triple containing type, target and timestamp, the sliding operation is encoded as a quadruple containing type, direction, distance and duration, and the input operation is encoded as a triple containing type, content and length, so as to obtain the encoded interactive event sequence; the mutual information calculation method and dynamic time warping algorithm are used to calculate the correlation strength between the hierarchical feature set and the interactive event sequence, and a feature event correlation mapping matrix is ​​constructed;

[0017] The hierarchical feature set is constructed as a feature relationship graph, the nodes of the feature relationship graph represent features at different levels, and the edges between the nodes represent the correlation between the features. A graph convolutional network is used to perform feature aggregation on the feature relationship graph, and the aggregated features are temporally modeled through a gated recurrent unit. An adaptive gating mechanism is introduced to dynamically adjust the importance of features, and the weight coefficient of the hierarchical feature is obtained based on a weighted combination of the temporal attention score and the graph attention score, wherein the temporal attention score is calculated based on the historical evolution law of the feature, and the graph attention score is calculated based on the topological structure relationship of the feature.

[0018] In an optional embodiment,

[0019] The steps of using a mutual information calculation method and a dynamic time warping algorithm to calculate the correlation strength between the hierarchical feature set and the interactive event sequence and constructing a feature event correlation mapping matrix include:

[0020] The mutual information calculation method is used to calculate the association strength between each feature in the hierarchical feature set and each event in the interactive event sequence, wherein the association strength is calculated by the logarithm of the ratio of the joint probability of the feature and the event to the marginal probability. For the continuous value features in the hierarchical feature set, the kernel density estimation method is used to calculate the probability density based on the Gaussian kernel function;

[0021] A dynamic time warping algorithm is used to perform time alignment on the hierarchical feature set and the interaction event sequence, and a cumulative distance matrix is ​​constructed, wherein the distance value of the cumulative distance matrix is ​​calculated based on the inverse of the association strength, and the minimum cumulative distance path of the cumulative distance matrix is ​​traced back through the dynamic time warping algorithm to obtain the aligned sequence correspondence relationship;

[0022] constructing a feature event association mapping matrix based on the association strength and the sequence correspondence, wherein the element value of the feature event association mapping matrix is ​​determined by the product of the association strength and a time decay term, and the time decay term is calculated based on the time difference in the sequence correspondence;

[0023] The feature event association mapping matrix is ​​subjected to sparse processing, the element values ​​of the feature event association mapping matrix are filtered by an adaptive threshold function, and the filtered element values ​​are subjected to nonlinear transformation by an activation function to obtain an optimized feature event association mapping matrix.

[0024] In an optional embodiment,

[0025] The training samples with weight coefficients are input into the cross-platform element alignment matrix for transfer learning, and an adaptive feature fusion network including a feature unification module, a dimension adaptation module and a feature fusion module is constructed. The training samples are iteratively optimized through the adaptive feature fusion network to obtain a page element recognition model. The page element recognition model is used to calculate the similarity of the newly added page elements. The steps of obtaining the page element similarity information include:

[0026] Input the training samples with weight coefficients into the cross-platform element alignment matrix for transfer learning. The cross-platform element alignment matrix is ​​composed of the source platform feature space basis vector, the target platform feature space basis vector and the singular value matrix. The transfer learning result is obtained by minimizing the difference between the source platform feature and the target platform feature and the platform consistency constraint of the feature gradient.

[0027] Constructing an adaptive feature fusion network, wherein the feature unification module processes the structural features, visual features and semantic features respectively to obtain unified features, the dimension adaptation module performs dimension adaptive processing on the unified features through global average pooling and global maximum pooling to obtain adaptive features, and the feature fusion module uses an attention mechanism to perform dynamic weight fusion on the adaptive features to obtain fused features;

[0028] A multi-scale feature iterative optimization model is constructed based on the fusion features, wherein the multi-scale feature iterative optimization model includes a multi-scale loss function, wherein the multi-scale loss function includes a structural consistency loss function, a visual similarity loss function, and a semantic alignment loss function, and a page element recognition model is obtained by minimizing the multi-scale loss function and a cross-platform constraint item to optimize the model;

[0029] The page element recognition model is used to construct a hierarchical similarity calculation framework, which calculates the tree edit distance to obtain structural similarity, calculates the cosine similarity to obtain visual similarity, and calculates the text similarity to obtain semantic similarity, and fuses the structural similarity, visual similarity and semantic similarity through difference adaptive weights to obtain page element similarity.

[0030] In an optional embodiment,

[0031] A multi-scale feature iterative optimization model is constructed based on the fusion features, wherein the multi-scale feature iterative optimization model includes a multi-scale loss function, wherein the multi-scale loss function includes a structural consistency loss function, a visual similarity loss function, and a semantic alignment loss function. The steps of optimizing the model by minimizing the multi-scale loss function and the cross-platform constraint item to obtain a page element recognition model include:

[0032] The structural consistency loss function is used to calculate the Euclidean distance and divergence between the fused feature and the original feature, the visual similarity loss function is calculated based on the ratio of the similarity score of the positive sample pair to the similarity score of all sample pairs, and the semantic alignment loss function is calculated based on the cosine distance between the fused feature and the text description encoding;

[0033] Introducing the maximum average difference as a cross-platform constraint term to calculate the distance between the source platform feature distribution and the target platform feature distribution in the kernel function mapping space; combining the multi-scale loss function and the cross-platform constraint term to form a total loss function;

[0034] The multi-scale feature iterative optimization model is trained by adopting an alternating optimization strategy, including: in the feature extraction optimization stage, fixing the alignment module parameters and optimizing the feature extractor parameters; in the alignment module optimization stage, fixing the feature extractor parameters and optimizing the alignment module parameters;

[0035] During the training process, the weight of each loss function is adjusted in stages, including: when the training round is less than the first preset round, the weight of the visual similarity loss function is set to the first preset value; when the training round is greater than or equal to the first preset round and less than the second preset round, the weight adjustment coefficient is calculated based on the difference between the total loss function values ​​of the validation sets of two adjacent rounds, and the weight of each loss function is updated according to the weight adjustment coefficient; when the training round is greater than or equal to the second preset round, the weight of the cross-platform constraint item is adjusted to the second preset value;

[0036] The model convergence is judged based on the performance fluctuation of the validation set. When the performance fluctuation of the validation set for multiple consecutive rounds is less than the preset threshold, the training is stopped to obtain the page element recognition model.

[0037] In an optional embodiment,

[0038] The steps of classifying the newly added page elements according to the page element similarity information, generating initial selection rules for the newly added page elements of different categories based on a preset selection rule template, locating the cross-platform elements by dynamically adjusting the selector weight, and modifying the initial selection rules based on the context feature vector, and using an incremental rule verification matrix to calculate rule parameters to generate target selection rules include:

[0039] A bottom-up agglomerative hierarchical clustering algorithm is used to classify the newly added page elements to obtain a set of page element categories, wherein the agglomerative hierarchical clustering algorithm includes: initializing each newly added page element as a separate category, calculating the similarity between categories, merging the two categories with the highest similarity to form a new category, calculating a similarity threshold based on the feature distribution of the newly added page elements, and stopping the merging when the maximum similarity between categories is less than the similarity threshold, wherein the similarity threshold is determined by a linear combination of the sample similarity mean and the sample standard deviation;

[0040] For each category in the page element category set, the similarity between the feature vector of the page element of the category and the feature vectors of each template in the preset rule template library is calculated, as well as the overlap between the attribute set of the page element of the category and the attribute set of each template, and the most matching rule template is selected based on the weighted value of the feature vector similarity and the attribute set overlap, to generate an initial selection rule;

[0041] Constructing a selector set including an identifier selector, a class selector, a text selector, and a position selector, calculating the positioning accuracy of each selector for the page element in the initial selection rule, dynamically adjusting the selector weight based on the difference between the positioning accuracy of each selector and the average accuracy of all selectors, and optimizing the initial selection rule using the updated selector weight;

[0042] Extracting hierarchical relationship features of page elements in the initial selection rule, the hierarchical relationship features include parent-level features, same-level features, and child-level features, fusing the hierarchical relationship features using an attention mechanism, and revising the optimized initial selection rule through a multi-layer perceptron based on the fused hierarchical relationship features to obtain a revised selection rule;

[0043] A rule verification matrix is ​​constructed, and the matching degree between the revised selection rule and the newly added page element is calculated as the matrix element of the rule verification matrix; the rule constraint loss is calculated based on the matrix element, and the rule constraint loss is weighted iteratively optimized to obtain the rule parameter, and the rule parameter is updated to the revised selection rule to generate the target selection rule.

[0044] In an optional embodiment,

[0045] The target selection rule is deployed to a rule execution engine, which includes a rule parsing module, a rule indexing module, and a rule execution module. The steps of continuously monitoring and dynamically collecting target page elements in a cross-platform application based on the rule execution engine include:

[0046] The rule parsing module adopts a front-end-mid-end-back-end three-stage architecture to perform cross-platform rule dynamic parsing. In the front-end stage, an abstract syntax tree containing node type, selector expression, attribute set, child node list and context information is constructed. In the mid-end stage, the abstract syntax tree is optimized for redundancy elimination and dependency analysis. In the back-end stage, the optimized abstract syntax tree is converted into a rule execution instruction sequence.

[0047] The rule index module uses a hierarchical multi-level cache structure to build an adaptive rule index tree. The nodes of the adaptive rule index tree include index keys, rule sets, access frequency statistics, timestamp information, and sub-node mapping relationships. The rule cache position is dynamically adjusted through a hotspot cache manager, and rule access is predicted based on historical access sequences through a predictive cache manager.

[0048] The rule execution module adopts a distributed scheduling mechanism to perform cross-platform rule asynchronous processing, selects the optimal execution node based on the node load status, and distributes tasks to a multi-level execution queue including a real-time task queue, a regular task queue, and a batch task queue;

[0049] Based on the rule execution engine, the target page elements in the cross-platform application are continuously monitored and dynamically collected, the changes of page elements are monitored in real time through the observer mode, the rules are matched and data are collected for the changed page elements, and the collected data is transmitted according to the multi-level synchronization strategy. The multi-level synchronization strategy includes real-time data synchronization based on long connection, incremental data synchronization based on operation log and full data synchronization based on data snapshot.

[0050] According to a second aspect of the embodiments of the present invention,

[0051] Provides a cross-platform, point-free data collection and intelligent selection rule generation system, including:

[0052] The first unit is used to inject a data collection script into the application of the terminal device to obtain the view hierarchy tree information and interaction event information of the preset page element;

[0053] The second unit is used to construct a page element feature vector based on the view hierarchy tree information, perform multi-level feature decomposition on the page element feature vector to obtain a hierarchical feature set, associate the hierarchical feature set with the interactive event information as a training sample, and assign weight coefficients to different levels of features in the training sample based on a graph neural network; input the training sample with the weight coefficient into a cross-platform element alignment matrix for transfer learning, construct an adaptive feature fusion network including a feature unification module, a dimension adaptation module and a feature fusion module, iteratively optimize the training sample through the adaptive feature fusion network to obtain a page element recognition model, and use the page element recognition model to calculate the similarity of the newly added page elements to obtain page element similarity information;

[0054] The third unit is used to classify the newly added page elements according to the page element similarity information, generate initial selection rules for the newly added page elements of different categories based on a preset selection rule template, locate the cross-platform elements by dynamically adjusting the selector weight, and modify the initial selection rules based on the context feature vector, and use an incremental rule verification matrix to calculate the rule parameters to generate a target selection rule; deploy the target selection rule to a rule execution engine, the rule execution engine includes a rule parsing module, a rule indexing module and a rule execution module, and continuously monitor and dynamically collect the target page elements in the cross-platform application based on the rule execution engine.

[0055] According to a third aspect of the embodiments of the present invention,

[0056] An electronic device is provided, comprising:

[0057] processor;

[0058] a memory for storing processor-executable instructions;

[0059] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0060] A fourth aspect of the embodiments of the present invention is:

[0061] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0062] The present invention realizes automatic collection of cross-platform application data by injecting data collection scripts and building page element recognition models, without the need for manual point placement, thus significantly improving data collection efficiency. Based on graph neural networks and adaptive feature fusion networks, it can intelligently generate and optimize circle selection rules, improve the accuracy of cross-platform element positioning, and adapt to page structure changes on different platforms. Using the rule execution engine, it can continuously monitor and dynamically collect target page elements to ensure the real-time and integrity of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 A flowchart of a method for cross-platform non-embedded point data collection and intelligent selection rule generation according to an embodiment of the present invention;

[0064] Figure 2 The present invention is a schematic diagram of the structure of a cross-platform non-embedded point data collection and intelligent selection rule generation system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0066] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0067] Figure 1 Schematic diagram of the process of cross-platform non-embedded point data collection and intelligent selection rule generation method according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0068] S1. Inject a data collection script into the application of the terminal device to obtain the view hierarchy tree information and interaction event information of the preset page elements;

[0069] S2. Construct a page element feature vector based on the view hierarchy tree information, perform multi-level feature decomposition on the page element feature vector to obtain a hierarchical feature set, associate and map the hierarchical feature set with the interactive event information as a training sample, and assign weight coefficients to different levels of features in the training sample based on a graph neural network; input the training sample with the weight coefficient into a cross-platform element alignment matrix for transfer learning, construct an adaptive feature fusion network including a feature unification module, a dimension adaptation module, and a feature fusion module, iteratively optimize the training sample through the adaptive feature fusion network to obtain a page element recognition model, and use the page element recognition model to calculate the similarity of the newly added page elements to obtain page element similarity information;

[0070] S3. Classify the newly added page elements according to the page element similarity information, generate initial selection rules for different categories of newly added page elements based on a preset selection rule template, locate the cross-platform elements by dynamically adjusting the selector weight, and modify the initial selection rules based on the context feature vector, and use an incremental rule verification matrix to calculate the rule parameters to generate a target selection rule; deploy the target selection rule to a rule execution engine, which includes a rule parsing module, a rule indexing module and a rule execution module, and continuously monitor and dynamically collect the target page elements in the cross-platform application based on the rule execution engine.

[0071] In an optional embodiment,

[0072] The steps of constructing a page element feature vector based on the view hierarchy tree information, performing multi-level feature decomposition on the page element feature vector to obtain a hierarchical feature set, associating and mapping the hierarchical feature set with the interaction event information as a training sample, and assigning weight coefficients to different hierarchical features in the training sample based on a graph neural network include:

[0073] The view hierarchy tree information includes the location information, type information and attribute information of the page elements, and the interaction event information includes the user's click operation information, sliding operation information and input operation information on the page elements;

[0074] The page element feature vector is subjected to dimensionality reduction processing by principal component analysis to obtain dimensionality reduction features, and the dimensionality reduction features are subjected to multi-level decomposition by a hierarchical clustering algorithm. The multi-level decomposition includes an appearance feature layer, a structural feature layer, and a semantic feature layer. The appearance feature layer includes color space features, geometric features, and style features encoded by local sensitive hashing, the structural feature layer includes relative position features and topological features mapped by the Graph2Vec algorithm, and the semantic feature layer includes text features, functional features, and business features extracted by the BERT model, to obtain a hierarchical feature set;

[0075] The interactive event information is encoded by using a position encoding method, the click operation is encoded as a triple containing type, target and timestamp, the sliding operation is encoded as a quadruple containing type, direction, distance and duration, and the input operation is encoded as a triple containing type, content and length, so as to obtain the encoded interactive event sequence; the mutual information calculation method and dynamic time warping algorithm are used to calculate the correlation strength between the hierarchical feature set and the interactive event sequence, and a feature event correlation mapping matrix is ​​constructed;

[0076] The hierarchical feature set is constructed as a feature relationship graph, the nodes of the feature relationship graph represent features at different levels, and the edges between the nodes represent the correlation between the features. A graph convolutional network is used to perform feature aggregation on the feature relationship graph, and the aggregated features are temporally modeled through a gated recurrent unit. An adaptive gating mechanism is introduced to dynamically adjust the importance of features, and the weight coefficient of the hierarchical feature is obtained based on a weighted combination of the temporal attention score and the graph attention score, wherein the temporal attention score is calculated based on the historical evolution law of the feature, and the graph attention score is calculated based on the topological structure relationship of the feature.

[0077] For example, in order to build a more accurate user interface (UI) element interaction prediction model, a UI element hierarchical feature modeling method based on graph neural network is proposed. This method can effectively capture the complex interaction patterns between users and UI elements and improve UI design and user experience.

[0078] First, extract the view hierarchy tree information from the front-end code or UI automation testing tool. This information includes the position coordinates (e.g., upper left corner coordinates and element size), type (e.g., button, text box, picture), and attributes (e.g., ID, class name, text content, color, font) of each UI element. For example, for a page containing a login button, the button type can be extracted as "button", the position is "(100, 200, 50, 30)" (indicating that the upper left corner coordinates are (100, 200), the width is 50, and the height is 30), and the text content is "login". At the same time, collect the user's interaction event information with UI elements, including click, slide, and input operations. For example, the event of a user clicking the login button can be recorded as a triple containing the type "click", the target element ID "loginButton", and the timestamp "2024-07-27 10:00:00".

[0079] Next, based on the extracted view hierarchy tree information, a feature vector is constructed for each UI element. The feature vector contains the location information, type information, and attribute information of the element. Then, the principal component analysis method is used to reduce the dimension of the feature vector to remove redundant information and reduce the computational complexity. For example, the original feature vector of one hundred dimensions can be reduced to twenty dimensions. After that, the hierarchical clustering algorithm is used to perform multi-level decomposition on the reduced features to construct a hierarchical feature set. The decomposition process divides the features into appearance feature layer, structural feature layer, and semantic feature layer. The appearance feature layer contains color space features (e.g., RGB values ​​encoded by local sensitive hashing), geometric features (e.g., aspect ratio), and style features (e.g., font size, border style). The structural feature layer contains relative position features (e.g., relative position to the parent element) and topological features (e.g., the DOM tree structure of the element is mapped to a vector using the Graph2Vec algorithm). The semantic feature layer contains text features, such as features extracted from text content using the BERT model (Bidirectional Encoder Representation Model), functional features (e.g., the click function of a button), and business features (e.g., the login button belongs to the login module).

[0080] Then, the interaction event information is encoded. Click operations are encoded as triples containing type, target element, and timestamp; sliding operations are encoded as quadruples containing type, direction, distance, and duration; and input operations are encoded as triples containing type, content, and length. For example, an event in which a user enters "hello" in the search box can be encoded as a triple containing type "input", content "hello", and length 2. After obtaining the encoded interaction event sequence, the mutual information calculation method and dynamic time warping algorithm are used to calculate the correlation strength between the hierarchical feature set and the interaction event sequence, and construct a feature event correlation mapping matrix. This matrix reflects the correlation between features at different levels and different types of interaction events.

[0081] Finally, the hierarchical feature set is constructed as a feature relationship graph. The nodes in the graph represent features at different levels, and the edges between the nodes represent the correlation between the features. For example, there is a correlation between the color features of the appearance feature layer and the relative position features of the structural feature layer. A graph convolutional network is used to aggregate the features of the feature relationship graph and learn the representation vectors of the nodes. Then, a gated recurrent unit is used to perform temporal modeling on the aggregated features to capture the change patterns of the features over time. In order to dynamically adjust the importance of features, an adaptive gating mechanism is introduced. The temporal attention score is calculated based on the historical evolution law of the features, and the graph attention score is calculated based on the topological structure relationship of the features. Finally, the weight coefficient of the hierarchical feature is obtained by the weighted combination of the temporal attention score and the graph attention score.

[0082] Through hierarchical feature decomposition and graph neural network modeling, the present invention can more comprehensively capture the feature information of UI elements and the correlation between features, thereby improving the accuracy of interaction prediction; hierarchical feature decomposition enables the model to distinguish the impact of features at different levels on interaction, such as appearance, structure, and semantic features, thereby enhancing the interpretability of the model and facilitating UI designers to understand user behavior patterns; it can learn the common feature representation of UI elements, thereby improving the generalization ability of the model on different UI interfaces and reducing dependence on specific UIs.

[0083] In an optional embodiment,

[0084] The steps of using a mutual information calculation method and a dynamic time warping algorithm to calculate the correlation strength between the hierarchical feature set and the interactive event sequence and constructing a feature event correlation mapping matrix include:

[0085] The mutual information calculation method is used to calculate the association strength between each feature in the hierarchical feature set and each event in the interactive event sequence, wherein the association strength is calculated by the logarithm of the ratio of the joint probability of the feature and the event to the marginal probability. For the continuous value features in the hierarchical feature set, the kernel density estimation method is used to calculate the probability density based on the Gaussian kernel function;

[0086] A dynamic time warping algorithm is used to perform time alignment on the hierarchical feature set and the interaction event sequence, and a cumulative distance matrix is ​​constructed, wherein the distance value of the cumulative distance matrix is ​​calculated based on the inverse of the association strength, and the minimum cumulative distance path of the cumulative distance matrix is ​​traced back through the dynamic time warping algorithm to obtain the aligned sequence correspondence relationship;

[0087] constructing a feature event association mapping matrix based on the association strength and the sequence correspondence, wherein the element value of the feature event association mapping matrix is ​​determined by the product of the association strength and a time decay term, and the time decay term is calculated based on the time difference in the sequence correspondence;

[0088] The feature event association mapping matrix is ​​subjected to sparse processing, the element values ​​of the feature event association mapping matrix are filtered by an adaptive threshold function, and the filtered element values ​​are subjected to nonlinear transformation by an activation function to obtain an optimized feature event association mapping matrix.

[0089] Exemplarily, first, the association strength between the hierarchical feature set and the interaction event sequence is calculated. The user feature data is preprocessed, such as missing value filling and standardization. For each feature in the hierarchical feature set and each event in the interaction event sequence, the association strength between them is calculated. The association strength is calculated by the logarithm of the ratio of the joint probability of the feature and the event to the respective marginal probabilities. For continuous value features, the kernel density estimation method is used to calculate the probability density based on the Gaussian kernel function. For example, assume that the feature "user age" is a continuous value feature and the event "click on the product" is a discrete value event. It is necessary to calculate the probability density function of "user age" and the probability of "clicking on the product" respectively, as well as the probability of their joint occurrence. Then, based on these probabilities, the association strength between the "user age" feature and the "click on the product" event is calculated.

[0090] Next, the dynamic time warping algorithm is used to align the hierarchical feature set and the interaction event sequence in time. A cumulative distance matrix is ​​constructed, where the distance value of the matrix is ​​based on the inverse of the previously calculated association strength. For example, if the association strength between feature A and event B is high, then their corresponding distance values ​​in the cumulative distance matrix are small. The minimum cumulative distance path of the cumulative distance matrix is ​​traced back through the dynamic time warping algorithm to obtain the correspondence between the aligned sequences. For example, assuming that the feature sequence is [A, B, C] and the event sequence is [X, Y, Z], through the dynamic time warping algorithm, it is possible to obtain the correspondence between A and X, B and Y, and C and Z.

[0091] Then, the feature event association mapping matrix is ​​constructed based on the association strength and the sequence correspondence. The element values ​​of this matrix are determined by the product of the association strength and the time decay term. The time decay term is calculated based on the time difference in the sequence correspondence. The larger the time difference, the smaller the time decay term. For example, if the time difference between feature A and event X in the aligned sequence is small, the corresponding element values ​​in the feature event association mapping matrix are large. Assuming that the association strength between feature A and event X is 0.8, the time difference is 1, and the time decay coefficient is 0.9, the corresponding element values ​​in the feature event association mapping matrix are 0.8 × 0.9 = 0.72.

[0092] Finally, the feature event association mapping matrix is ​​sparsely processed. The element values ​​of the matrix are filtered through an adaptive threshold function, and only the elements with association strength exceeding a certain threshold are retained. The filtered element values ​​are nonlinearly transformed using an activation function, such as the Sigmoid function, to enhance the expression of the association relationship between features and events. For example, assuming the threshold is 0.5, element values ​​less than 0.5 will be set to 0. Assuming that an element value is 0.6, after the Sigmoid function transformation, its value becomes about 0.65. Finally, the optimized feature event association mapping matrix is ​​obtained.

[0093] By combining mutual information and dynamic time warping algorithms, the present invention can more accurately capture the temporal correlation between features and events, thereby improving the accuracy of the correlation relationship; the constructed feature-event correlation mapping matrix clearly shows the correlation strength between features and events, improves the interpretability of the model, and helps to understand the driving factors behind user behavior; sparse processing can reduce the dimension of the feature-event correlation mapping matrix, thereby reducing computational complexity and improving the efficiency of the model.

[0094] In an optional embodiment,

[0095] The training samples with weight coefficients are input into the cross-platform element alignment matrix for transfer learning, and an adaptive feature fusion network including a feature unification module, a dimension adaptation module and a feature fusion module is constructed. The training samples are iteratively optimized through the adaptive feature fusion network to obtain a page element recognition model. The page element recognition model is used to calculate the similarity of the newly added page elements. The steps of obtaining the page element similarity information include:

[0096] Input the training samples with weight coefficients into the cross-platform element alignment matrix for transfer learning. The cross-platform element alignment matrix is ​​composed of the source platform feature space basis vector, the target platform feature space basis vector and the singular value matrix. The transfer learning result is obtained by minimizing the difference between the source platform feature and the target platform feature and the platform consistency constraint of the feature gradient.

[0097] Constructing an adaptive feature fusion network, wherein the feature unification module processes the structural features, visual features and semantic features respectively to obtain unified features, the dimension adaptation module performs dimension adaptive processing on the unified features through global average pooling and global maximum pooling to obtain adaptive features, and the feature fusion module uses an attention mechanism to perform dynamic weight fusion on the adaptive features to obtain fused features;

[0098] A multi-scale feature iterative optimization model is constructed based on the fusion features, wherein the multi-scale feature iterative optimization model includes a multi-scale loss function, wherein the multi-scale loss function includes a structural consistency loss function, a visual similarity loss function, and a semantic alignment loss function, and a page element recognition model is obtained by minimizing the multi-scale loss function and a cross-platform constraint item to optimize the model;

[0099] The page element recognition model is used to construct a hierarchical similarity calculation framework, which calculates the tree edit distance to obtain structural similarity, calculates the cosine similarity to obtain visual similarity, and calculates the text similarity to obtain semantic similarity, and fuses the structural similarity, visual similarity and semantic similarity through difference adaptive weights to obtain page element similarity.

[0100] Exemplarily, first, prepare training sample data. Collect page element data from different platforms (e.g., Android, iOS, Web), and annotate the corresponding weight coefficient for each training sample. The weight coefficient can be set according to factors such as the importance and reliability of the sample. For example, high-quality samples are given higher weights, and low-quality samples are given lower weights. Suppose we collected 1,000 samples, 800 of which are from the Android platform, 100 from the iOS platform, and 100 from the Web platform. Each sample contains structural features (e.g., DOM tree structure), visual features (e.g., screenshot images), and semantic features (e.g., text content). The weight coefficient of the Android platform sample is set to 0.8, and the weight coefficient of the iOS and Web platform samples is set to 0.1.

[0101] Next, a cross-platform element alignment matrix is ​​constructed. This matrix consists of the source platform feature space basis vectors, the target platform feature space basis vectors, and the singular value matrix. The transfer learning results are obtained by minimizing the differences between the source platform features and the target platform features and the platform consistency constraints of the feature gradients. For example, the Android platform is used as the source platform and the iOS platform is used as the target platform. By analyzing the feature distribution of the source platform and the target platform, a matrix is ​​constructed to map the features of the source platform to the feature space of the target platform, thereby reducing the impact of platform differences.

[0102] Then, an adaptive feature fusion network is constructed. The network includes a feature unification module, a dimension adaptation module and a feature fusion module. The feature unification module processes the structural features, visual features and semantic features respectively to obtain unified features. For example, the DOM tree structure is converted into a vector representation, the screenshot image is converted into a feature vector, and the text content is converted into a word vector. The dimension adaptation module performs dimension adaptive processing on the unified features through global average pooling and global maximum pooling to obtain adaptive features. The feature fusion module uses the attention mechanism to dynamically weight the adaptive features to obtain fused features. For example, weights are dynamically assigned according to the importance of different features to fuse the structural features, visual features and semantic features into a comprehensive feature vector.

[0103] After that, a multi-scale feature iterative optimization model is constructed based on the fusion features. The model includes a multi-scale loss function, including a structural consistency loss function, a visual similarity loss function, and a semantic alignment loss function. The model is optimized by minimizing the multi-scale loss function and the cross-platform constraint term to obtain a page element recognition model. For example, through iterative training, the model parameters are continuously adjusted so that the model can accurately recognize page elements on different platforms.

[0104] Finally, the trained page element recognition model is used to construct a hierarchical similarity calculation framework. This framework calculates the tree edit distance to obtain structural similarity, calculates the cosine similarity to obtain visual similarity, and calculates the text similarity to obtain semantic similarity. It then fuses the structural similarity, visual similarity, and semantic similarity through adaptive weights of differences to obtain page element similarity. For example, for two page elements to be compared, their structural similarity, visual similarity, and semantic similarity are calculated respectively, and then the weights of the three similarities are adaptively adjusted according to the differences, and finally a comprehensive similarity score is obtained.

[0105] The present invention effectively reduces the impact of cross-platform differences on page element recognition and improves recognition accuracy through transfer learning and multi-scale feature fusion; the adaptive feature fusion network can better capture the common features of page elements on different platforms, enhances the generalization ability of the model, and enables it to adapt to page element recognition tasks on different platforms; the hierarchical similarity calculation framework can effectively fuse feature information of different dimensions and improve the efficiency of similarity calculation.

[0106] In an optional embodiment,

[0107] A multi-scale feature iterative optimization model is constructed based on the fusion features, wherein the multi-scale feature iterative optimization model includes a multi-scale loss function, wherein the multi-scale loss function includes a structural consistency loss function, a visual similarity loss function, and a semantic alignment loss function. The steps of optimizing the model by minimizing the multi-scale loss function and the cross-platform constraint item to obtain a page element recognition model include:

[0108] The structural consistency loss function is used to calculate the Euclidean distance and divergence between the fused feature and the original feature, the visual similarity loss function is calculated based on the ratio of the similarity score of the positive sample pair to the similarity score of all sample pairs, and the semantic alignment loss function is calculated based on the cosine distance between the fused feature and the text description encoding;

[0109] Introducing the maximum average difference as a cross-platform constraint term to calculate the distance between the source platform feature distribution and the target platform feature distribution in the kernel function mapping space; combining the multi-scale loss function and the cross-platform constraint term to form a total loss function;

[0110] The multi-scale feature iterative optimization model is trained by adopting an alternating optimization strategy, including: in the feature extraction optimization stage, fixing the alignment module parameters and optimizing the feature extractor parameters; in the alignment module optimization stage, fixing the feature extractor parameters and optimizing the alignment module parameters;

[0111] During the training process, the weight of each loss function is adjusted in stages, including: when the training round is less than the first preset round, the weight of the visual similarity loss function is set to the first preset value; when the training round is greater than or equal to the first preset round and less than the second preset round, the weight adjustment coefficient is calculated based on the difference between the total loss function values ​​of the validation sets of two adjacent rounds, and the weight of each loss function is updated according to the weight adjustment coefficient; when the training round is greater than or equal to the second preset round, the weight of the cross-platform constraint item is adjusted to the second preset value;

[0112] The model convergence is judged based on the performance fluctuation of the validation set. When the performance fluctuation of the validation set for multiple consecutive rounds is less than the preset threshold, the training is stopped to obtain the page element recognition model.

[0113] Exemplarily, first, a multi-scale feature iterative optimization model is constructed. The core of this model is to fuse page element features from different platforms (e.g., desktop and mobile). For example, image features, text features, and layout features can be extracted. Image features can be extracted using a pre-trained convolutional neural network, text features can be obtained using a word embedding model, and layout features can be calculated based on the position and size of the element on the page. These features are spliced ​​or fused to form an initial fused feature.

[0114] Then, a multi-scale loss function is constructed. The loss function consists of three parts: structural consistency loss, visual similarity loss, and semantic alignment loss. Structural consistency loss is used to measure the difference between the fused features and the original features. For example, the difference between the fused features and the image features can be used to measure structural consistency. The smaller the difference, the better the structural consistency. Visual similarity loss is used to measure the visual similarity of the same semantic element on different platforms. For example, the image feature similarity of the same button on different platforms can be calculated. The higher the similarity, the better the visual similarity. Semantic alignment loss is used to measure the semantic similarity between the fused features and the text description of the element. For example, the similarity between the fused features and the word embedding vector of the button text "Submit" can be used to measure the degree of semantic alignment. The higher the similarity, the better the semantic alignment. Suppose there is a button element, whose image feature on the desktop is vector A, and whose image feature on the mobile is vector B. Its text description is "Submit", and the corresponding word embedding vector is C. The structural consistency loss can calculate the sum of the differences between vectors A and B and the fused features; the visual similarity loss can calculate the ratio of the similarity scores of vectors A and B to the similarity scores of all sample pairs; the semantic alignment loss can calculate the similarity between the fused features and vector C.

[0115] Next, we introduce cross-platform constraints. Cross-platform constraints are used to constrain the feature distributions of different platforms to be as close as possible. For example, the maximum average difference can be used to measure the distance between the feature distributions of the desktop and mobile terminals in the kernel function mapping space. The smaller the distance, the closer the feature distributions of the two platforms are. Assuming that there are 100 feature vectors of buttons on the desktop and 100 feature vectors of buttons on the mobile terminal, the cross-platform constraint can calculate the distribution difference of these two sets of feature vectors in the high-dimensional space.

[0116] The multi-scale loss function and the cross-platform constraint term are combined to form the total loss function.

[0117] The model is trained using an alternating optimization strategy. In the feature extraction optimization phase, the alignment module parameters are fixed and the feature extractor parameters are optimized, for example, the weights of the convolutional neural network are updated. In the alignment module optimization phase, the feature extractor parameters are fixed and the alignment module parameters are optimized, for example, the weights of the fusion module are updated. Assuming the initial learning rate is 0.001, in the feature extraction optimization phase, the parameters of the feature extractor are updated with a learning rate of 0.001; in the alignment module optimization phase, the parameters of the alignment module are updated with a learning rate of 0.001.

[0118] During the training process, the weights of each loss function are adjusted in stages. For example, at the beginning of training, the weight of the visual similarity loss is set to a higher value, such as 0.8, to quickly learn visual similarity. When the number of training rounds reaches a certain level, such as 100 rounds, the weight adjustment coefficient is calculated based on the difference in the total loss function value of the validation set between two adjacent rounds, and the weight of each loss function is updated based on the coefficient. When the number of training rounds reaches another threshold, such as 200 rounds, the weight of the cross-platform constraint item is adjusted to a higher value, such as 0.5, to enhance cross-platform adaptability.

[0119] The model convergence is judged based on the performance fluctuation of the validation set. When the performance fluctuation of the validation set for multiple rounds (for example, 5 rounds) is less than a preset threshold (for example, 0.001), the training is stopped and the final page element recognition model is obtained.

[0120] Through the joint optimization of multi-scale loss functions and cross-platform constraints, the model can learn more robust and generalized feature representations, thereby improving the accuracy of cross-platform element recognition; the introduction of cross-platform constraints can effectively reduce the differences in feature distributions on different platforms, enhance the cross-platform adaptability of the model, and enable it to maintain good performance on different platforms; the strategy of adjusting the loss function weights in stages can enable the model to focus on different optimization objectives at different training stages, thereby improving the model training efficiency and accelerating the model convergence speed.

[0121] In an optional embodiment,

[0122] The steps of classifying the newly added page elements according to the page element similarity information, generating initial selection rules for the newly added page elements of different categories based on a preset selection rule template, locating the cross-platform elements by dynamically adjusting the selector weight, and modifying the initial selection rules based on the context feature vector, and using an incremental rule verification matrix to calculate rule parameters to generate target selection rules include:

[0123] A bottom-up agglomerative hierarchical clustering algorithm is used to classify the newly added page elements to obtain a set of page element categories, wherein the agglomerative hierarchical clustering algorithm includes: initializing each newly added page element as a separate category, calculating the similarity between categories, merging the two categories with the highest similarity to form a new category, calculating a similarity threshold based on the feature distribution of the newly added page elements, and stopping the merging when the maximum similarity between categories is less than the similarity threshold, wherein the similarity threshold is determined by a linear combination of the sample similarity mean and the sample standard deviation;

[0124] For each category in the page element category set, the similarity between the feature vector of the page element of the category and the feature vectors of each template in the preset rule template library is calculated, as well as the overlap between the attribute set of the page element of the category and the attribute set of each template, and the most matching rule template is selected based on the weighted value of the feature vector similarity and the attribute set overlap, to generate an initial selection rule;

[0125] Constructing a selector set including an identifier selector, a class selector, a text selector, and a position selector, calculating the positioning accuracy of each selector for the page element in the initial selection rule, dynamically adjusting the selector weight based on the difference between the positioning accuracy of each selector and the average accuracy of all selectors, and optimizing the initial selection rule using the updated selector weight;

[0126] Extracting hierarchical relationship features of page elements in the initial selection rule, the hierarchical relationship features include parent-level features, same-level features, and child-level features, fusing the hierarchical relationship features using an attention mechanism, and revising the optimized initial selection rule through a multi-layer perceptron based on the fused hierarchical relationship features to obtain a revised selection rule;

[0127] A rule verification matrix is ​​constructed, and the matching degree between the revised selection rule and the newly added page element is calculated as the matrix element of the rule verification matrix; the rule constraint loss is calculated based on the matrix element, and the rule constraint loss is weighted iteratively optimized to obtain the rule parameter, and the rule parameter is updated to the revised selection rule to generate the target selection rule.

[0128] Exemplarily, first, the newly added page elements are classified. Collect the newly added page elements, such as buttons, input boxes, text, etc. Then use the bottom-up agglomerative hierarchical clustering algorithm to classify these elements. Specifically, each element is first regarded as a separate category. Then calculate the similarity between categories. The calculation of similarity can be based on multiple features of the elements, such as tag name, attribute, style, text content, etc. For example, two elements with the "button" tag and similar styles have a high similarity. Merge the two categories with the highest similarity into a new category. Repeat this merging process until the maximum similarity between categories is less than the set threshold. This threshold is determined based on the sample similarity mean and sample standard deviation. For example, the threshold can be set to the mean minus 0.5 times the standard deviation. In this way, we get a set of categories for page elements. Suppose we have 10 newly added page elements, which are divided into 3 categories after clustering: button category, input box category, and text category.

[0129] Next, generate initial selection rules for each category. For each category, calculate the similarity between the feature vector of the page element of the category and the feature vector of each template in the preset rule template library. The feature vector may include information such as tag name, attribute name, attribute value, text content, etc. At the same time, calculate the overlap between the attribute set of the page element of the category and the attribute set of each template. For example, the attribute set of a category contains "class", "id", and "type", and the attribute set of a template contains "class" and "id", then their overlap is 2 / 3. Perform a weighted sum of the feature vector similarity and the attribute set overlap, select the template with the highest weighted value as the most matching rule template, and generate the initial selection rule based on this template. For example, the feature vector of the button class has the highest similarity with the "click button" template, so the "click button" template is selected to generate the initial selection rule.

[0130] Then, the initial selection rules are optimized across platforms. A selector set is constructed that includes identifier selectors (such as #id), class selectors, text selectors, and location selectors. For each category, the positioning accuracy of each selector for the page elements in that category is calculated. For example, for the button class, the accuracy of the identifier selector is 90%, the accuracy of the class selector is 80%, the accuracy of the text selector is 70%, the accuracy of the location selector is 60%, and the average accuracy of all selectors is 75%. According to the difference between the accuracy of each selector and the average accuracy, the weight of the selector is dynamically adjusted. For example, the weight of the identifier selector is increased by 15%, the weight of the class selector is increased by 5%, the weight of the text selector is reduced by 5%, and the weight of the location selector is reduced by 15%. Use the updated selector weights to optimize the initial selection rules.

[0131] Next, the selection rules are modified based on contextual features. The hierarchical relationship features of the page elements in the initial selection rules are extracted, including parent-level features, same-level features, and child-level features. For example, the parent level of a button is a div, the same level is another button and an input box, and the child level is text. These hierarchical relationship features are fused using an attention mechanism. For example, different weights are assigned according to the importance of the hierarchical relationship features, and then weighted summation is performed. Based on the fused hierarchical relationship features, the optimized initial selection rules are modified through a multi-layer perceptron.

[0132] Finally, generate the target selection rule. Construct a rule verification matrix and calculate the matching degree between the revised selection rule and the newly added page element as the matrix element. For example, if the rule can correctly select the element, the matrix element is 1, otherwise it is 0. Calculate the rule constraint loss based on the matrix elements, such as calculating the number of incorrectly selected elements. Perform weighted iterative optimization on the rule constraint loss to obtain the rule parameters. For example, adjust the weight of the selector or the weight of the hierarchical relationship feature. Update these parameters to the revised selection rule to generate the final target selection rule.

[0133] The present invention can effectively improve the accuracy of the selection rules and reduce the occurrence of erroneous selections through steps such as classification based on page element similarity, correction based on context features, and dynamic adjustment of selector weights.

[0134] In an optional embodiment,

[0135] The target selection rule is deployed to a rule execution engine, which includes a rule parsing module, a rule indexing module, and a rule execution module. The steps of continuously monitoring and dynamically collecting target page elements in a cross-platform application based on the rule execution engine include:

[0136] The rule parsing module adopts a front-end-mid-end-back-end three-stage architecture to perform cross-platform rule dynamic parsing. In the front-end stage, an abstract syntax tree containing node type, selector expression, attribute set, child node list and context information is constructed. In the mid-end stage, the abstract syntax tree is optimized for redundancy elimination and dependency analysis. In the back-end stage, the optimized abstract syntax tree is converted into a rule execution instruction sequence.

[0137] The rule index module uses a hierarchical multi-level cache structure to build an adaptive rule index tree. The nodes of the adaptive rule index tree include index keys, rule sets, access frequency statistics, timestamp information, and sub-node mapping relationships. The rule cache position is dynamically adjusted through a hotspot cache manager, and rule access is predicted based on historical access sequences through a predictive cache manager.

[0138] The rule execution module adopts a distributed scheduling mechanism to perform cross-platform rule asynchronous processing, selects the optimal execution node based on the node load status, and distributes tasks to a multi-level execution queue including a real-time task queue, a regular task queue, and a batch task queue;

[0139] Based on the rule execution engine, the target page elements in the cross-platform application are continuously monitored and dynamically collected, the changes of page elements are monitored in real time through the observer mode, the rules are matched and data are collected for the changed page elements, and the collected data is transmitted according to the multi-level synchronization strategy. The multi-level synchronization strategy includes real-time data synchronization based on long connection, incremental data synchronization based on operation log and full data synchronization based on data snapshot.

[0140] Exemplarily, first, the cross-platform rules are dynamically parsed. The rule parsing module adopts a three-stage architecture of front-end-middle-end-back-end. In the front-end stage, the rule text is parsed into an abstract syntax tree (AST). AST contains node types (such as elements, text, attributes), selector expressions (such as CSS selectors, XPath expressions), attribute sets (such as id, class, name), child node lists, and context information (such as parent nodes, sibling nodes). For example, for the rule text "div.container > a[href]", the generated AST contains a root node "div", whose node type is element, the selector expression is ".container", and the child node list contains an "a" node, and the attribute set of the "a" node contains "href". In the middle-end stage, the AST is optimized. Redundancy elimination optimization removes duplicate or useless nodes, such as duplicate selectors or empty attribute sets. Dependency analysis optimization identifies dependencies between nodes, such as parent-child relationships and sibling relationships, for subsequent rule matching optimization. In the back-end stage, the optimized AST is converted into a sequence of rule execution instructions. For example, "div.container > a[href]" is converted to "find a div element with class container, and then find a element with href attribute among its child elements".

[0141] Next, build an adaptive rule index tree. The rule index module adopts a hierarchical multi-level cache structure. The nodes of the adaptive rule index tree contain index keys (such as part of the selector expression), rule sets (pointing to the corresponding AST), access frequency statistics, timestamp information, and child node mapping relationships. For example, the rule set corresponding to the index key ".container" contains all rules that contain ".container" in the selector expression. The hotspot cache manager dynamically adjusts the rule cache location and caches rules with high access frequency into a higher level cache. For example, if the rule "div.container > a[href]" has a high access frequency, it is cached in the first-level cache. The predictive cache manager predicts rule access based on historical access sequences. For example, if the user frequently visits page A, the rules related to page A are loaded into the cache in advance.

[0142] Then, cross-platform rule asynchronous processing is performed. The rule execution module adopts a distributed scheduling mechanism. The optimal execution node is selected based on the node load status, and tasks are assigned to multi-level execution queues, including real-time task queues, regular task queues, and batch task queues. For example, tasks with high real-time requirements are assigned to the real-time task queue and executed first; regular tasks are assigned to the regular task queue; batch tasks are assigned to the batch task queue for centralized processing.

[0143] Finally, the target page elements in the cross-platform application are continuously monitored and dynamically collected. The observer mode is used to monitor page element changes in real time. For example, MutationObserver is used to monitor changes in the DOM tree. Rules are matched and data is collected for the changed page elements. For example, when a page element changes, the changed element is matched with the rule in the rule index tree. If the match is successful, the target data is collected. The collected data is transmitted according to the multi-level synchronization strategy. Real-time data synchronization is based on long connections, such as WebSocket, to transmit data to the server in real time. Incremental data synchronization is based on operation logs, which record changes in page elements, such as addition, modification, and deletion, and then synchronize the operation logs to the server. Full data synchronization is based on data snapshots, and snapshots of all collected data are generated regularly and then synchronized to the server. For example, a full data snapshot is generated every morning.

[0144] The front-end-middle-end-back-end three-stage architecture and AST optimization strategy of the present invention improve the efficiency and accuracy of rule parsing; the adaptive rule index tree and multi-level cache mechanism speed up the speed of rule matching and reduce resource consumption.

[0145] Figure 2 Schematic diagram of the structure of the cross-platform non-embedded point data collection and intelligent selection rule generation system according to an embodiment of the present invention. Figure 2 As shown, the system comprises:

[0146] The first unit is used to inject a data collection script into the application of the terminal device to obtain the view hierarchy tree information and interaction event information of the preset page element;

[0147] The second unit is used to construct a page element feature vector based on the view hierarchy tree information, perform multi-level feature decomposition on the page element feature vector to obtain a hierarchical feature set, associate the hierarchical feature set with the interactive event information as a training sample, and assign weight coefficients to different levels of features in the training sample based on a graph neural network; input the training sample with the weight coefficient into a cross-platform element alignment matrix for transfer learning, construct an adaptive feature fusion network including a feature unification module, a dimension adaptation module and a feature fusion module, iteratively optimize the training sample through the adaptive feature fusion network to obtain a page element recognition model, and use the page element recognition model to calculate the similarity of the newly added page elements to obtain page element similarity information;

[0148] The third unit is used to classify the newly added page elements according to the page element similarity information, generate initial selection rules for the newly added page elements of different categories based on a preset selection rule template, locate the cross-platform elements by dynamically adjusting the selector weight, and modify the initial selection rules based on the context feature vector, and use an incremental rule verification matrix to calculate the rule parameters to generate a target selection rule; deploy the target selection rule to a rule execution engine, the rule execution engine includes a rule parsing module, a rule indexing module and a rule execution module, and continuously monitor and dynamically collect the target page elements in the cross-platform application based on the rule execution engine.

[0149] According to a third aspect of the embodiments of the present invention,

[0150] An electronic device is provided, comprising:

[0151] processor;

[0152] a memory for storing processor-executable instructions;

[0153] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0154] A fourth aspect of the embodiments of the present invention is:

[0155] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0156] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0157] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cross-platform method for collecting data without embedding points and generating intelligent selection rules, characterized in that: include: Inject a data collection script into the application of the terminal device to obtain the view hierarchy tree information and interaction event information of the preset page elements; Constructing a page element feature vector based on the view hierarchy tree information, performing multi-level feature decomposition on the page element feature vector to obtain a hierarchical feature set, associating and mapping the hierarchical feature set with the interaction event information as a training sample, and assigning weight coefficients to different level features in the training sample based on a graph neural network; Input the training samples with weight coefficients into the cross-platform element alignment matrix for transfer learning, build an adaptive feature fusion network including a feature unification module, a dimension adaptation module and a feature fusion module, iteratively optimize the training samples through the adaptive feature fusion network to obtain a page element recognition model, use the page element recognition model to calculate the similarity of the newly added page elements, and obtain page element similarity information; The newly added page elements are classified according to the page element similarity information, initial selection rules are generated for the newly added page elements of different categories based on a preset selection rule template, the cross-platform elements are located by dynamically adjusting the selector weight, and the initial selection rules are corrected based on the context feature vector, and the incremental rule verification matrix is ​​used to calculate the rule parameters to generate the target selection rules; the target selection rules are deployed to the rule execution engine, and the rule execution engine includes a rule parsing module, a rule indexing module and a rule execution module, and the target page elements in the cross-platform application are continuously monitored and dynamically collected based on the rule execution engine.

2. The method according to claim 1, characterized in that The steps of constructing a page element feature vector based on the view hierarchy tree information, performing multi-level feature decomposition on the page element feature vector to obtain a hierarchical feature set, associating and mapping the hierarchical feature set with the interaction event information as a training sample, and assigning weight coefficients to different hierarchical features in the training sample based on a graph neural network include: The view hierarchy tree information includes the location information, type information and attribute information of the page elements, and the interaction event information includes the user's click operation information, sliding operation information and input operation information on the page elements; The page element feature vector is subjected to dimensionality reduction processing by principal component analysis to obtain dimensionality reduction features, and the dimensionality reduction features are subjected to multi-level decomposition by a hierarchical clustering algorithm. The multi-level decomposition includes an appearance feature layer, a structural feature layer, and a semantic feature layer. The appearance feature layer includes color space features, geometric features, and style features encoded by local sensitive hashing, the structural feature layer includes relative position features and topological features mapped by the Graph2Vec algorithm, and the semantic feature layer includes text features, functional features, and business features extracted by the BERT model, to obtain a hierarchical feature set; The interactive event information is encoded by using a position encoding method, the click operation is encoded as a triple containing type, target and timestamp, the sliding operation is encoded as a quadruple containing type, direction, distance and duration, and the input operation is encoded as a triple containing type, content and length, so as to obtain the encoded interactive event sequence; the mutual information calculation method and dynamic time warping algorithm are used to calculate the correlation strength between the hierarchical feature set and the interactive event sequence, and a feature event correlation mapping matrix is ​​constructed; The hierarchical feature set is constructed as a feature relationship graph, the nodes of the feature relationship graph represent features at different levels, and the edges between the nodes represent the correlation between the features. A graph convolutional network is used to perform feature aggregation on the feature relationship graph, and the aggregated features are temporally modeled through a gated recurrent unit. An adaptive gating mechanism is introduced to dynamically adjust the importance of features, and the weight coefficient of the hierarchical feature is obtained based on a weighted combination of the temporal attention score and the graph attention score, wherein the temporal attention score is calculated based on the historical evolution law of the feature, and the graph attention score is calculated based on the topological structure relationship of the feature.

3. The method according to claim 2, characterized in that The steps of using a mutual information calculation method and a dynamic time warping algorithm to calculate the correlation strength between the hierarchical feature set and the interactive event sequence and constructing a feature event correlation mapping matrix include: The mutual information calculation method is used to calculate the association strength between each feature in the hierarchical feature set and each event in the interactive event sequence, wherein the association strength is calculated by the logarithm of the ratio of the joint probability of the feature and the event to the marginal probability. For the continuous value features in the hierarchical feature set, the kernel density estimation method is used to calculate the probability density based on the Gaussian kernel function; A dynamic time warping algorithm is used to perform time alignment on the hierarchical feature set and the interaction event sequence, and a cumulative distance matrix is ​​constructed, wherein the distance value of the cumulative distance matrix is ​​calculated based on the inverse of the association strength, and the minimum cumulative distance path of the cumulative distance matrix is ​​traced back through the dynamic time warping algorithm to obtain the aligned sequence correspondence relationship; constructing a feature event association mapping matrix based on the association strength and the sequence correspondence, wherein the element value of the feature event association mapping matrix is ​​determined by the product of the association strength and a time decay term, and the time decay term is calculated based on the time difference in the sequence correspondence; The feature event association mapping matrix is ​​subjected to sparse processing, the element values ​​of the feature event association mapping matrix are filtered by an adaptive threshold function, and the filtered element values ​​are subjected to nonlinear transformation by an activation function to obtain an optimized feature event association mapping matrix.

4. The method according to claim 1, characterized in that: The training samples with weight coefficients are input into the cross-platform element alignment matrix for transfer learning, and an adaptive feature fusion network including a feature unification module, a dimension adaptation module and a feature fusion module is constructed. The training samples are iteratively optimized through the adaptive feature fusion network to obtain a page element recognition model. The page element recognition model is used to calculate the similarity of the newly added page elements. The steps of obtaining the page element similarity information include: Input the training samples with weight coefficients into the cross-platform element alignment matrix for transfer learning. The cross-platform element alignment matrix is ​​composed of the source platform feature space basis vector, the target platform feature space basis vector and the singular value matrix. The transfer learning result is obtained by minimizing the difference between the source platform feature and the target platform feature and the platform consistency constraint of the feature gradient. Constructing an adaptive feature fusion network, wherein the feature unification module processes the structural features, visual features and semantic features respectively to obtain unified features, the dimension adaptation module performs dimension adaptive processing on the unified features through global average pooling and global maximum pooling to obtain adaptive features, and the feature fusion module uses an attention mechanism to perform dynamic weight fusion on the adaptive features to obtain fused features; A multi-scale feature iterative optimization model is constructed based on the fusion features, wherein the multi-scale feature iterative optimization model includes a multi-scale loss function, wherein the multi-scale loss function includes a structural consistency loss function, a visual similarity loss function, and a semantic alignment loss function, and a page element recognition model is obtained by minimizing the multi-scale loss function and a cross-platform constraint item to optimize the model; The page element recognition model is used to construct a hierarchical similarity calculation framework, which calculates the tree edit distance to obtain structural similarity, calculates the cosine similarity to obtain visual similarity, and calculates the text similarity to obtain semantic similarity, and fuses the structural similarity, visual similarity and semantic similarity through difference adaptive weights to obtain page element similarity.

5. The method according to claim 4, characterized in that A multi-scale feature iterative optimization model is constructed based on the fusion features, wherein the multi-scale feature iterative optimization model includes a multi-scale loss function, wherein the multi-scale loss function includes a structural consistency loss function, a visual similarity loss function, and a semantic alignment loss function. The steps of optimizing the model by minimizing the multi-scale loss function and the cross-platform constraint item to obtain a page element recognition model include: The structural consistency loss function is used to calculate the Euclidean distance and divergence between the fused feature and the original feature, the visual similarity loss function is calculated based on the ratio of the similarity score of the positive sample pair to the similarity score of all sample pairs, and the semantic alignment loss function is calculated based on the cosine distance between the fused feature and the text description encoding; Introducing the maximum average difference as a cross-platform constraint term to calculate the distance between the source platform feature distribution and the target platform feature distribution in the kernel function mapping space; combining the multi-scale loss function and the cross-platform constraint term to form a total loss function; The multi-scale feature iterative optimization model is trained by adopting an alternating optimization strategy, including: in the feature extraction optimization stage, fixing the alignment module parameters and optimizing the feature extractor parameters; in the alignment module optimization stage, fixing the feature extractor parameters and optimizing the alignment module parameters; During the training process, the weight of each loss function is adjusted in stages, including: when the training round is less than the first preset round, the weight of the visual similarity loss function is set to the first preset value; when the training round is greater than or equal to the first preset round and less than the second preset round, the weight adjustment coefficient is calculated based on the difference between the total loss function values ​​of the validation sets of two adjacent rounds, and the weight of each loss function is updated according to the weight adjustment coefficient; when the training round is greater than or equal to the second preset round, the weight of the cross-platform constraint item is adjusted to the second preset value; The model convergence is judged based on the performance fluctuation of the validation set. When the performance fluctuation of the validation set for multiple consecutive rounds is less than the preset threshold, the training is stopped to obtain the page element recognition model.

6. The method according to claim 1, characterized in that The steps of classifying the newly added page elements according to the page element similarity information, generating initial selection rules for the newly added page elements of different categories based on a preset selection rule template, locating the cross-platform elements by dynamically adjusting the selector weight, and modifying the initial selection rules based on the context feature vector, and using an incremental rule verification matrix to calculate rule parameters to generate target selection rules include: A bottom-up agglomerative hierarchical clustering algorithm is used to classify the newly added page elements to obtain a set of page element categories, wherein the agglomerative hierarchical clustering algorithm includes: initializing each newly added page element as a separate category, calculating the similarity between categories, merging the two categories with the highest similarity to form a new category, calculating a similarity threshold based on the feature distribution of the newly added page elements, and stopping the merging when the maximum similarity between categories is less than the similarity threshold, wherein the similarity threshold is determined by a linear combination of the sample similarity mean and the sample standard deviation; For each category in the page element category set, the similarity between the feature vector of the page element of the category and the feature vectors of each template in the preset rule template library is calculated, as well as the overlap between the attribute set of the page element of the category and the attribute set of each template, and the most matching rule template is selected based on the weighted value of the feature vector similarity and the attribute set overlap, to generate an initial selection rule; Constructing a selector set including an identifier selector, a class selector, a text selector, and a position selector, calculating the positioning accuracy of each selector for the page element in the initial selection rule, dynamically adjusting the selector weight based on the difference between the positioning accuracy of each selector and the average accuracy of all selectors, and optimizing the initial selection rule using the updated selector weight; Extracting hierarchical relationship features of page elements in the initial selection rule, the hierarchical relationship features include parent-level features, same-level features, and child-level features, fusing the hierarchical relationship features using an attention mechanism, and revising the optimized initial selection rule through a multi-layer perceptron based on the fused hierarchical relationship features to obtain a revised selection rule; A rule verification matrix is ​​constructed, and the matching degree between the revised selection rule and the newly added page element is calculated as the matrix element of the rule verification matrix; the rule constraint loss is calculated based on the matrix element, and the rule constraint loss is weighted iteratively optimized to obtain the rule parameter, and the rule parameter is updated to the revised selection rule to generate the target selection rule.

7. The method according to claim 1, characterized in that The target selection rule is deployed to a rule execution engine, which includes a rule parsing module, a rule indexing module, and a rule execution module. The steps of continuously monitoring and dynamically collecting target page elements in a cross-platform application based on the rule execution engine include: The rule parsing module adopts a front-end-mid-end-back-end three-stage architecture to perform cross-platform rule dynamic parsing. In the front-end stage, an abstract syntax tree containing node type, selector expression, attribute set, child node list and context information is constructed. In the mid-end stage, the abstract syntax tree is optimized for redundancy elimination and dependency analysis. In the back-end stage, the optimized abstract syntax tree is converted into a rule execution instruction sequence. The rule index module uses a hierarchical multi-level cache structure to build an adaptive rule index tree. The nodes of the adaptive rule index tree include index keys, rule sets, access frequency statistics, timestamp information, and sub-node mapping relationships. The rule cache position is dynamically adjusted through a hotspot cache manager, and rule access is predicted based on historical access sequences through a predictive cache manager. The rule execution module adopts a distributed scheduling mechanism to perform cross-platform rule asynchronous processing, selects the optimal execution node based on the node load status, and distributes tasks to a multi-level execution queue including a real-time task queue, a regular task queue, and a batch task queue; Based on the rule execution engine, the target page elements in the cross-platform application are continuously monitored and dynamically collected, the changes of page elements are monitored in real time through the observer mode, the rules are matched and data are collected for the changed page elements, and the collected data is transmitted according to the multi-level synchronization strategy. The multi-level synchronization strategy includes real-time data synchronization based on long connection, incremental data synchronization based on operation log and full data synchronization based on data snapshot.

8. A cross-platform non-embedded point data collection and intelligent selection rule generation system, used to implement the method described in any one of claims 1 to 7, characterized in that: include: The first unit is used to inject a data collection script into the application of the terminal device to obtain the view hierarchy tree information and interaction event information of the preset page element; The second unit is used to construct a page element feature vector based on the view hierarchy tree information, perform multi-level feature decomposition on the page element feature vector to obtain a hierarchical feature set, associate and map the hierarchical feature set with the interaction event information as a training sample, and assign weight coefficients to different levels of features in the training sample based on a graph neural network; Input the training samples with weight coefficients into the cross-platform element alignment matrix for transfer learning, build an adaptive feature fusion network including a feature unification module, a dimension adaptation module and a feature fusion module, iteratively optimize the training samples through the adaptive feature fusion network to obtain a page element recognition model, use the page element recognition model to calculate the similarity of the newly added page elements, and obtain page element similarity information; The third unit is used to classify the newly added page elements according to the page element similarity information, generate initial selection rules for the newly added page elements of different categories based on a preset selection rule template, locate the cross-platform elements by dynamically adjusting the selector weight, and modify the initial selection rules based on the context feature vector, and use an incremental rule verification matrix to calculate the rule parameters to generate a target selection rule; deploy the target selection rule to a rule execution engine, the rule execution engine includes a rule parsing module, a rule indexing module and a rule execution module, and continuously monitor and dynamically collect the target page elements in the cross-platform application based on the rule execution engine.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images