Screening bar element recognition and interaction intent prediction method

Abstract

This application provides a method for identifying filter bar elements and predicting interactive intent. It uses a visual language model to uniformly identify filter bar controls and buttons of different styles, and combines multimodal fusion to predict interaction behavior types (pop-up / refresh / jump), solving the problems of poor versatility and lack of intent prediction in traditional tools. Automated instructions are generated based on coordinates and adapted to different device resolutions, achieving cross-platform operation with zero code maintenance. A closed-loop system of confidence verification and training set optimization improves model robustness. Ultimately, it achieves automated verification of the filtering function, replacing time-consuming manual click testing, significantly improving verification efficiency and reliability.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence recognition, and in particular to a method for identifying filter elements and predicting interactive intent. Background Technology

[0002] Currently, while some feasible solutions for page element recognition use automated tools, they lack versatility and cannot be adapted to a wide range of application scenarios and diverse recognition tasks. Furthermore, current recognition methods struggle to uncover the deeper semantics of each page element and its distribution.

[0003] Specifically, in a scenario involving filter or sort bar recognition, traditionally, manual testing is required for each filter. While some feasible automation tools can improve automation levels, they still cannot adapt to various filter bar styles and cannot predict the associated interactive behaviors of filter bar clicks, resulting in a poor user experience. All of these problems arise from the insufficient versatility and intelligence of automated tools in recognizing page elements. Therefore, a method for filter bar element recognition and interaction intent prediction is needed to solve these issues. Summary of the Invention

[0004] The purpose of this application is to at least address one of the aforementioned technical deficiencies, particularly the lack of versatility and intelligence in the existing automated tools for identifying filter elements and predicting interactive intent.

[0005] In a first aspect, this application provides a method for identifying filter bar elements and predicting interaction intent, the method comprising:

[0006] Get the target image corresponding to the target page;

[0007] The target image is input into a pre-trained visual language model to identify the filter bar control and its internal operable buttons, and output the position and size information based on the target image screenshot coordinate system.

[0008] Based on the visual semantics and page context features of the target image, predict the interaction behavior type of each operable button;

[0009] Based on the location information and the size information, an automated operation instruction is generated, and click operation, information acquisition operation and analysis operation associated with the filter bar control are executed according to the automated operation instruction and the interaction behavior type.

[0010] As an optional implementation, predicting the interaction behavior type of each operable button based on the visual semantics and page context features of the target image includes:

[0011] Extract the visual feature vectors and semantic vectors of the surrounding text for each operable button;

[0012] The association weights between visual feature vectors and semantic vectors are calculated using a multimodal fusion method.

[0013] Based on the correlation weight matrix, the interactive behavior type identifier is output.

[0014] The multimodal fusion includes:

[0015] The visual branch uses a convolutional neural network to extract the button shape and color features of each operable button;

[0016] The text branch uses a word embedding model to extract the semantics of the subject and surrounding descriptive text of each operable button;

[0017] The cosine similarity between the visual features and the text features is calculated using a cross-modal attention layer as the association weight.

[0018] As an optional implementation, the interaction behavior type identifier includes:

[0019] Pop-up type identifier, used to indicate the click behavior that triggers the floating panel or hides the pop-up;

[0020] The refresh type identifier is used to indicate the click behavior that triggers a partial update of the current area;

[0021] Jump type identifier, used to indicate the click behavior that triggers page switching.

[0022] As an optional implementation, the method further includes:

[0023] If a first page element with an interface rendering error is detected in the filter bar control and its internal operable buttons, the area ratio of each first page element with rendering error is calculated. When the abnormal area ratio exceeds a preset ratio, the operable area of ​​each first page element is optimized, and the coordinate information of a preset number of first page elements is retained according to the priority order of element semantic attributes.

[0024] If a second page element with abnormal business logic is detected in the filter bar control and its internal operable buttons, the code segment associated with the second page element is obtained from the document object model, the abnormal situation of the second page element is identified according to the code segment, and the preset fault handling process is called to process the second page element.

[0025] If a third page element for a behavior prediction task is detected in the filter bar control and its internal operable buttons, an expected behavior flow and an expected execution result are generated according to a preset behavior prediction model. In the process of determining the execution effect of the target task based on the page state change image, the execution effect of the target task is determined based on the expected behavior flow and the expected execution result.

[0026] As an optional implementation, the training process of the visual language model includes:

[0027] Obtain e-commerce training images to construct a training dataset;

[0028] For each of the e-commerce training images, the coordinate range and semantic identifier of the associated elements of the filter bar control and the internal operable buttons are marked, and the button coordinates and corresponding interaction behavior types of each operable button are marked.

[0029] A multimodal converter architecture is adopted to jointly optimize the loss function of the visual feature extraction layer, semantic understanding layer, and interaction behavior classification output layer;

[0030] Additionally, random noise and simulated display parameter fluctuations are added during training to enhance the image.

[0031] As an optional implementation, before linearly scaling the position and size information according to the current device display parameters to obtain the physical operation position information, the method further includes:

[0032] The confidence level of the output of the visual language model is verified.

[0033] When the confidence score is less than the preset threshold, the context region expansion screenshot is triggered, the target image is re-acquired, and the recognition operation is re-executed.

[0034] Additionally, the page context features of elements whose confidence scores are less than a preset threshold are recorded to optimize the training dataset.

[0035] Secondly, this application provides a device for identifying filter elements and predicting interaction intent, the device comprising:

[0036] The acquisition module is used to acquire the target image corresponding to the target page;

[0037] The processing module is used to input the target image into a pre-trained visual language model, identify the filter bar control and its internal operable buttons, and output position and size information based on the target image screenshot coordinate system;

[0038] The processing module is also used to predict the interaction behavior type of each operable button based on the visual semantics and page context features of the target image;

[0039] The processing module is further configured to generate automated operation instructions based on the location information and the size information, and execute click operations, information acquisition operations and analysis operations associated with the filter bar control according to the automated operation instructions and the interaction behavior type.

[0040] As an optional implementation, the specific method by which the processing module predicts the interaction behavior type of each operable button based on the visual semantics and page context features of the target image includes:

[0041] Extract the visual feature vectors and semantic vectors of the surrounding text for each operable button;

[0042] The association weights between visual feature vectors and semantic vectors are calculated using a multimodal fusion method.

[0043] Based on the correlation weight matrix, the interactive behavior type identifier is output.

[0044] The multimodal fusion includes:

[0045] The visual branch uses a convolutional neural network to extract the button shape and color features of each operable button;

[0046] The text branch uses a word embedding model to extract the semantics of the subject and surrounding descriptive text of each operable button;

[0047] The cosine similarity between the visual features and the text features is calculated using a cross-modal attention layer as the association weight.

[0048] Thirdly, this application provides a computer device including one or more processors and a memory storing computer-readable instructions that, when executed by the one or more processors, perform the steps of the method described in the first aspect.

[0049] Fourthly, this application provides a storage medium storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the method described in the first aspect.

[0050] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0051] Based on any of the above embodiments, a pre-trained visual language model is used to identify various styles of filter bars (e.g., horizontal tabs / vertical sidebars) and their internal operable buttons. A multimodal fusion mechanism is used to associate visual features (button shape, color) with textual semantics (surrounding descriptions) to accurately classify interaction behavior types (pop-up triggering floating panels, refreshing local areas, page switching). Position information based on the screenshot coordinate system is linearly scaled to adapt to different device resolutions, outputting directly executable physical coordinate commands. During the training phase, e-commerce scene image enhancement and multimodal joint optimization are introduced to improve the model's generalization ability to multiple samples. A runtime confidence verification mechanism expands the context for re-identification of low-confidence elements and records features to optimize the training set, forming a continuous enhancement loop. This solution overcomes the limitations of traditional tools that require manual style adaptation and cannot predict interaction intent, enabling automated tools to customize waiting strategies based on prediction types (e.g., monitoring floating layer loading for pop-ups), avoiding blind polling. Ultimately, fully automated verification of the filtering / sorting function is achieved, transforming manual click-by-click testing into efficient and reliable command execution, significantly reducing verification costs and improving test coverage. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 A flowchart illustrating a method for identifying filter elements and predicting interaction intent provided in one embodiment of this application;

[0054] Figure 2 This is an internal structural diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0055] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0056] Currently, while some feasible solutions for page element recognition use automated tools, they lack versatility and cannot be adapted to a wide range of application scenarios and diverse recognition tasks. Furthermore, current recognition methods struggle to uncover the deeper semantics of each page element and its distribution.

[0057] Specifically, in the scenario of identifying a page's operational placement, the design of these placements is often complex, containing multiple clickable sub-elements. However, some feasible automation tools cannot adapt to all types and architectures of operational placement designs, and struggle to identify the design logic behind the placements and the nesting of their internal sub-elements. Similarly, in the scenario of identifying product cards on a page, each card contains the product's attributes, sales information, and a visual representation. Some feasible automation tools cannot adapt to product lists with various layouts, nor can they achieve fine-grained instance-level segmentation to distinguish individual product cards within complex layouts. Furthermore, in the scenario of identifying filter or sorting bars, traditionally, manual testing is required. While some feasible automation tools can improve automation, they still cannot adapt to various filter bar styles and cannot predict the associated interactive behaviors of filter bar clicks, resulting in a poor user experience. All of these scenarios suffer from problems due to the insufficient versatility and intelligence of automated tools in identifying filter bar elements and predicting interaction intent. Therefore, a method for identifying filter bar elements and predicting interaction intent based on a visual language model is needed to address these issues.

[0058] Furthermore, in e-commerce scenarios, filtering and sorting functions (vertical / horizontal filter bars) are key interaction areas affecting user conversion rates. Each page update or major promotional event requires manual verification to ensure the filtering / sorting functions are triggered correctly and updated appropriately. This manual verification is time-consuming and labor-intensive. Traditional methods lack versatility and struggle to adapt uniformly to different filter bar styles (horizontal tabs, vertical sidebar buttons). More importantly, traditional tools cannot predict the expected interaction type after clicking the control from page screenshots (visual information), rendering automated click-and-verify strategies ineffective.

[0059] In summary, the technical concept of this application lies in employing a pre-trained visual language model to recognize various styles of filter bars (e.g., horizontal tabs / vertical sidebars) and their internal operable buttons. A multimodal fusion mechanism is used to associate visual features (button shape, color) with textual semantics (surrounding descriptions), accurately classifying interaction behavior types (pop-up triggering floating panels, refreshing local areas, page switching). Based on the positional information of the screenshot coordinate system, linear scaling is applied to adapt to different device resolutions, outputting directly executable physical coordinate commands. During the training phase, e-commerce scene image enhancement and multimodal joint optimization are introduced to improve the model's generalization ability to multiple samples. A runtime confidence verification mechanism expands the context for re-identifying low-confidence elements and records features to optimize the training set, forming a continuous enhancement loop. This solution overcomes the limitations of traditional tools that require manual style adaptation and cannot predict interaction intent, enabling automated tools to customize waiting strategies based on prediction types (e.g., monitoring floating layer loading for pop-ups), avoiding blind polling. Ultimately, fully automated verification of the filtering / sorting function is achieved, transforming manual click-by-click testing into efficient and reliable command execution, significantly reducing verification costs and improving test coverage.

[0060] The methods provided in this application will be described in detail below based on the corresponding implementation methods in some practical application scenarios.

[0061] The filter element recognition and interaction intent prediction architecture provided in this application is a page element recognition method based on a visual language model, which may specifically include:

[0062] Get the target image corresponding to the target page;

[0063] The target image is input into a pre-trained visual language model, which performs a recognition task based on the target page elements and outputs position and size information based on the target image screenshot coordinate system.

[0064] Based on the current device display parameters, the position information and the size information are linearly scaled to obtain the physical operation position information;

[0065] Automated operation instructions are generated based on physical operation location information, and the target task is executed according to the automated operation instructions.

[0066] Specifically, this application provides a method for identifying filter elements and predicting interactive intent based on a visual language model. The specific operation process may include: first, obtaining a complete screenshot of the target page through the system interface of the terminal device; inputting the screenshot image into a pre-trained visual language model, which employs a multimodal converter architecture and achieves semantic understanding of page elements by jointly optimizing the loss functions of the visual feature extraction layer and the semantic understanding layer; the model outputs the position and size information of the target element in the screenshot coordinate system, including the horizontal and vertical coordinates of the element's center point, as well as its width and height values; further, the screenshot coordinates can be linearly scaled according to the ratio of the current device's screen width and height parameters to a pre-stored reference resolution parameter: the horizontal coordinate is adjusted according to the ratio of the screen width to the reference width, and the vertical coordinate is adjusted according to the ratio of the screen height to the reference height; finally, based on the scaled physical operation position information, a sequence of touch commands executable by an automated testing tool is generated.

[0067] This implementation uses a pre-trained visual language model to perform semantic understanding of the target page screenshot, directly outputting the coordinates and size information of the target elements. This replaces traditional automated tools that rely on code-based positioning, solving the problems of high maintenance costs and poor versatility caused by dynamic changes in page structure. By using linear scaling, the screenshot coordinate system is converted into the device's physical coordinate system, achieving accurate operation adaptation across resolution devices. The final generated automated operation instructions form a "recognition-positioning-operation" closed loop, significantly improving the automation level and operation accuracy of filter element recognition and interactive intent prediction, providing a general infrastructure for scenarios such as nested element recognition and product card segmentation.

[0068] Based on this Figure 1 This is a flowchart illustrating a method for identifying filter elements and predicting interaction intent according to an embodiment of this application, as shown below. Figure 1 As shown, this application provides a method for identifying filter bar elements and predicting interaction intent, the method comprising:

[0069] S101. Obtain the target image corresponding to the target page;

[0070] In this application, the target object may include a filter bar control and internal operable buttons. Of course, it may also include other elements such as product card instances, main operation positions and internal nested sub-elements, and decorative objects.

[0071] S102. Input the target image into a pre-trained visual language model to identify the filter bar control and its internal operable buttons, and output the position and size information based on the target image screenshot coordinate system.

[0072] Taking a scenario where the target object includes the main body of the operation position and its internal nested sub-elements, product card instances, filter bar controls and internal operable buttons as an example, the target object recognition task in a specific application scenario can include target page element recognition, nested element recognition or filter bar element recognition.

[0073] The target page element identification is used to determine the interface rendering, business logic, and page layout of each target page element in the target page; the nested element identification is used to determine the interface rendering, business logic, and element jump association of each target page element with jump function in the target page; and the filter bar element identification is used to determine the interface rendering, business logic, and expected behavior association of each target page element with filter or sort function in the target page.

[0074] If a first page element with an interface rendering error is identified, the area percentage of each first page element with rendering error is calculated. When the percentage of the error area exceeds a preset ratio, the operable area of ​​each first page element is optimized, and the coordinate information of a preset number of first page elements is retained according to the priority order of element semantic attributes.

[0075] If a second page element with abnormal business logic is identified, the code segment associated with the second page element is obtained from the document object model, the abnormal situation of the second page element is further identified through the code language model, and the preset fault handling process is called to process the second page element.

[0076] If a third page element with a behavior prediction task to be performed is identified, the expected behavior flow and expected execution result are generated according to the preset behavior prediction model. In the process of determining the execution effect of the target task based on the page state change image, the execution effect of the target task is determined based on the expected behavior flow and the expected execution result.

[0077] Specifically, for handling UI rendering anomalies, the area percentage of the rendered element with an anomaly can be calculated. When the percentage exceeds a preset ratio, an optimization algorithm is activated to filter operable areas, prioritizing the retention of high-value element coordinates. For handling business logic anomalies, a multimodal verification mechanism can be implemented. This involves extracting the code segment associated with the target element from the document object model, inputting the code into the code analysis model to detect anomaly patterns, and invoking a preset repair procedure based on the output anomaly identifier. For handling behavior prediction anomalies, the type of interaction behavior is predicted based on a visual language model. When the actual effect does not match the prediction, the differences in page features before and after the interaction are extracted to correct the prediction model parameters.

[0078] For UI rendering anomalies, visual conflict areas are identified through area quantization, and operation targets are optimized by combining semantic priority to avoid accidental touches. For business logic anomalies, code language models are called to analyze underlying logic defects and locate the root cause of the anomaly. For behavior prediction tasks, interaction results are predicted and verification strategies are customized to form a full-link processing mechanism for abnormal scenarios, which greatly improves the operational robustness of complex pages.

[0079] As an optional implementation, taking a target object that includes a filter bar control and internal operable buttons as an example, the method further includes:

[0080] If a first page element with an interface rendering error is detected in the filter bar control and its internal operable buttons, the area ratio of each first page element with rendering error is calculated. When the abnormal area ratio exceeds a preset ratio, the operable area of ​​each first page element is optimized, and the coordinate information of a preset number of first page elements is retained according to the priority order of element semantic attributes.

[0081] If a second page element with abnormal business logic is detected in the filter bar control and its internal operable buttons, the code segment associated with the second page element is obtained from the document object model, the abnormal situation of the second page element is identified according to the code segment, and the preset fault handling process is called to process the second page element.

[0082] If a third page element for a behavior prediction task is detected in the filter bar control and its internal operable buttons, an expected behavior flow and an expected execution result are generated according to a preset behavior prediction model. In the process of determining the execution effect of the target task based on the page state change image, the execution effect of the target task is determined based on the expected behavior flow and the expected execution result.

[0083] This implementation method targets elements with abnormal interface rendering, calculates the percentage of abnormal area, optimizes operable regions, and retains key coordinate information based on the priority of element semantic attributes to ensure the accessibility of core functions. For elements with abnormal business logic, it uses document object model code segments and tree-structured coordinate sets to locate the root cause of the abnormality, triggering a preset fault handling process to avoid interruption of automated processes. For elements with predictable tasks, it generates expected processes and results through a behavior prediction model and compares the actual effects with status monitoring. This multi-dimensional abnormality handling mechanism significantly improves the system's fault tolerance and task continuity in complex page environments.

[0084] S103. Based on the visual semantics and page context features of the target image, predict the interaction behavior type of each operable button;

[0085] As an optional implementation, predicting the interaction behavior type of each operable button based on the visual semantics and page context features of the target image includes:

[0086] Extract the visual feature vectors and semantic vectors of the surrounding text for each operable button;

[0087] The association weights between visual feature vectors and semantic vectors are calculated using a multimodal fusion method.

[0088] Based on the correlation weight matrix, the interactive behavior type identifier is output.

[0089] The multimodal fusion includes:

[0090] The visual branch uses a convolutional neural network to extract the button shape and color features of each operable button;

[0091] The text branch uses a word embedding model to extract the semantics of the subject and surrounding descriptive text of each operable button;

[0092] The cosine similarity between the visual features and the text features is calculated using a cross-modal attention layer as the association weight.

[0093] This implementation employs a multimodal fusion mechanism, using a convolutional neural network to extract visual features such as button shape and color, a word embedding model to extract semantic features from the text, and a cross-modal attention layer to calculate cosine similarity as association weights. This design deeply associates visual cues with textual descriptions, overcoming the limitations of insufficient information from a single modality. Based on a weight matrix, it classifies and outputs behavior identifiers, improving the accuracy of interaction type prediction and providing more refined intent guidance for automated operations.

[0094] As an optional implementation, the interaction behavior type identifier includes:

[0095] Pop-up type identifier, used to indicate the click behavior that triggers the floating panel or hides the pop-up;

[0096] The refresh type identifier is used to indicate the click behavior that triggers a partial update of the current area;

[0097] Jump type identifier, used to indicate the click behavior that triggers page switching.

[0098] This implementation identifies three types of interaction identifiers: pop-up, refresh, and jump. The pop-up identifier corresponds to a floating panel trigger action, the refresh identifier indicates a local area update, and the jump identifier is associated with a page switching action. By accurately matching subsequent verification strategies using these categorized identifiers (e.g., pop-ups require waiting for the floating layer to load, and jumps require monitoring page transitions), automation tools can customize waiting logic for different behavior types, avoiding invalid polling and improving verification efficiency and response accuracy.

[0099] S104. Generate automated operation instructions based on the location information and the size information, and execute click operations, information acquisition operations and analysis operations associated with the filter bar control according to the automated operation instructions and the interaction behavior type.

[0100] As an optional implementation, the step of generating automated operation instructions based on the location information and the size information includes:

[0101] Based on the logical relationship between the filter bar control and its internal operable buttons, a tree-structured coordinate set containing the coordinates of the parent node and the relative coordinates of the child node is constructed;

[0102] Based on the current device display parameters, the position information and the size information are linearly scaled according to the tree-structured coordinate set to obtain the physical operation position information;

[0103] Automated operation instructions are generated based on the physical operation location information.

[0104] The generation of automated operation instructions based on physical operation location information includes:

[0105] Based on the target page elements, determine the interactive behavior instructions corresponding to the target automation tool based on the physical operation location information, and execute the information acquisition or analysis process corresponding to the target task based on the interactive behavior instructions.

[0106] Furthermore, after executing the interactive behavior instruction, the page state change image is acquired in real time, and the execution effect of the target task is determined based on the page state change image.

[0107] In the process of generating automated operation instructions, the target task type is first selected based on business needs. The target page element recognition task detects the element rendering state and layout logic; the nested element recognition task constructs hierarchical coordinates for sub-elements within the operational area; and the filter bar recognition task combines behavior prediction to generate interactive intent identifiers, enabling subsequent prediction and operation. Then, the instructions are converted, mapping the physical coordinates to an automated tool instruction set. For example, for a click operation, a touch event sequence containing coordinate position, press duration, and release duration is generated; for a swipe operation, trajectory data of the start and end points is generated. Additionally, a state verification process is included, capturing images of page state changes after instruction execution. For example, a feature comparison engine can be used to extract structured features from the image and calculate similarity with a pre-stored template. When the similarity is below a set threshold, an anomaly marking process is triggered.

[0108] This implementation transforms physical coordinates into executable instructions for automated tools, driving the tools to perform precise operations; by capturing and comparing the page status image after the operation, real-time verification of the operation effect is achieved; and by customizing the operation process according to different task types, a complete closed loop of operation-verification-feedback is formed, significantly improving the accuracy of automated testing and the reliability of functional verification.

[0109] The linear scaling process includes:

[0110] Obtain the screen width and screen height parameters of the current device, and calculate the screen aspect ratio;

[0111] Read the pre-stored reference width and reference height parameters, and calculate the reference width-to-height ratio;

[0112] Adjust the horizontal coordinates based on the ratio of the screen width parameter to the baseline width parameter;

[0113] Adjust the vertical coordinates based on the ratio of the screen height parameter to the reference height parameter;

[0114] Furthermore, when the difference between the aspect ratio of the device screen and the reference aspect ratio exceeds a preset threshold, an edge pixel filling strategy is adopted to adjust the aspect ratio of the screen.

[0115] Specifically, this application can perform the following operations during linear scaling: Obtain the actual screen width and height parameters of the terminal device, and calculate the device aspect ratio. Simultaneously, read the pre-stored reference width and height parameters, and calculate the reference aspect ratio. When the difference between the device aspect ratio and the reference aspect ratio exceeds a preset threshold, an edge pixel filling strategy is triggered: edge bands are added to both sides of the screenshot image to ensure the processed image aspect ratio matches the reference ratio. Subsequently, coordinate mapping is performed: the horizontal coordinate is scaled according to the ratio of the device width to the reference width, and the vertical coordinate is scaled according to the ratio of the device height to the reference height. This process can be implemented using a dedicated coordinate transformation engine with a built-in aspect ratio verification function, automatically calling the edge filling module to maintain the original shape of the elements.

[0116] It should be noted that the scaling ratio can be calculated based on the reference width and reference height parameters, and is a result based on the actual application scenario. The above implementation method is only a feasible example. In fact, this calculation method can also be used for local mapping, and those skilled in the art can deduce the relevant implementation details.

[0117] This implementation achieves linear coordinate mapping by dynamically calculating the aspect ratio between the device and the reference resolution, ensuring the accuracy of the operation position on devices with different resolutions. For scenarios where the device aspect ratio is distorted, an edge filling strategy is used to force the original aspect ratio of the elements to be maintained, avoiding click position offset caused by screen deformation, thereby improving the robustness of cross-device adaptation and operational reliability.

[0118] This implementation uses a pre-trained visual language model to identify filter bar controls and internal buttons, and combines visual semantics with page context features to predict interaction behavior types, solving the problem that traditional tools cannot predict interaction intent. Based on location information, it generates automated instructions to execute clicks, information retrieval, and other operations, enabling the automated tool to adapt to different filter bar styles (horizontal / vertical) without requiring manual testing. Intent prediction guides subsequent intelligent waiting strategies, significantly improving the efficiency and reliability of functional verification while reducing manual verification costs.

[0119] As an optional implementation, the training process of the visual language model includes:

[0120] Obtain e-commerce training images to construct a training dataset;

[0121] For each of the e-commerce training images, the coordinate range and semantic identifiers of three types of elements are labeled, including the main body of the operation position and its nested sub-elements, product card instances, and filter bar controls and their internal operable buttons.

[0122] A multimodal converter architecture is adopted to jointly optimize the loss function of the visual feature extraction layer and the semantic understanding layer;

[0123] Additionally, random noise and simulated display parameter fluctuations are added during training to enhance the image.

[0124] For training the visual language model, the first step is data construction, collecting historical screenshots of e-commerce pages. Depending on the actual application scenario, this can include one or more of three types of elements: labeled operational areas (including sub-element boundaries), product cards (instance segmentation masks), and filter controls (including behavioral labels). Other types of elements, such as decorative elements, can also be included. Further, a visual encoder extracts local features, and a semantic decoder generates element description text. A cross-modal fusion layer aligns the visual and semantic vector spaces. An anti-interference mechanism is also designed during training, injecting noise to enhance the image. This can be achieved by adding random noise, simulating resolution fluctuations, or through image processing methods such as random cropping. This employs a hard sample optimization strategy to improve boundary recognition capabilities.

[0125] In this implementation, the model is trained using an e-commerce-specific dataset, enabling the visual language model to accurately understand the visual semantic features of elements such as promotional operation slots, product cards, and filter bars. A multimodal architecture is used to jointly optimize visual and semantic representation capabilities, improving the model's accuracy in recognizing element boundaries and types. Noise injection and parameter fluctuation simulation enhance the model's anti-interference capabilities, ensuring stable recognition performance in dynamically rendered pages.

[0126] In the application scenario of this application, the training process of the visual language model includes:

[0127] Obtain e-commerce training images to construct a training dataset;

[0128] For each of the e-commerce training images, the coordinate range and semantic identifier of the associated elements of the filter bar control and the internal operable buttons are marked, and the button coordinates and corresponding interaction behavior types of each operable button are marked.

[0129] A multimodal converter architecture is adopted to jointly optimize the loss function of the visual feature extraction layer, semantic understanding layer, and interaction behavior classification output layer;

[0130] Additionally, random noise and simulated display parameter fluctuations are added during training to enhance the image.

[0131] This implementation constructs a training set based on images from e-commerce scenarios, annotating control coordinates, button positions, and interaction types. It jointly optimizes the loss functions of the visual feature extraction layer, semantic understanding layer, and classification output layer. By adding random noise and simulating display parameter fluctuations, image diversity is enhanced, enabling the model to adapt to different resolutions and noise interference. This training method improves the model's generalization ability to various filter styles, ensuring robustness of recognition and intent prediction in complex page environments.

[0132] In fact, visual language models can also perform iterative optimization of the training dataset. For example, before linearly scaling the position and size information according to the current device display parameters to obtain the physical operation position information, the method further includes:

[0133] The confidence level of the output of the visual language model is verified.

[0134] When the confidence score is less than the preset threshold, the context region expansion screenshot is triggered, the target image is re-acquired, and the recognition operation is re-executed.

[0135] Additionally, the page context features of elements whose confidence scores are less than a preset threshold are recorded to optimize the training dataset.

[0136] Furthermore, a confidence check is performed before coordinate transformation. First, a confidence assessment is conducted, analyzing the confidence score output by the visual language model. When an element's score is below a set threshold, it is considered a low-confidence recognition. A local resampling process is then initiated for low-confidence elements. For example, an extended region image can be cropped based on the predicted coordinates and input into the lightweight model for secondary recognition. A data optimization mechanism can also be designed to record the page context features of low-confidence elements (including the distribution of surrounding elements and color contrast) to generate a hard sample dataset for model iteration.

[0137] This implementation adds a confidence verification mechanism. When the model output confidence score is lower than a threshold, it triggers a context region expansion screenshot for re-identification, avoiding erroneous operations caused by low confidence. The page context features of low-confidence elements are recorded and fed back into the training set, forming a dynamic optimization loop. This design solves the model's recognition bottleneck in edge cases (such as occluded / blurred buttons), continuously improving the model's generalization ability and recognition accuracy.

[0138] Based on practical application scenarios, the method provided in this application can accurately identify filtering and sorting controls on an e-commerce page, predict their interactive intent, and output precise physical coordinates for automated clicking. Implemented through a corresponding system architecture, the system can include an image acquisition module, a Vision-LLM recognition module, an intent prediction module, a coordinate adaptation module, and a command output module.

[0139] Phase 1: Vision-LLM Identification and Intent Prediction

[0140] The Vision-LLM recognition and intent prediction module receives screenshots of e-commerce pages. It identifies the main body of the filter bar and all operable buttons within it (such as "Price Sort" and "Filter Entry").

[0141] The model infers the expected interaction type (e.g., Side_Panel_Popup or InPlace_Update) for each operable button based on both visual semantics and page context.

[0142] The module outputs the center coordinates and width / height information of the original screenshot pixels.

[0143] Phase Two: Screen Coordinate System Restoration and Adaptation

[0144] The coordinate adaptation module obtains the actual resolution of the current device, linearly scales and restores the coordinates of the screenshot output by Vision-LLM, and obtains the physical coordinates and width and height information that can be executed on the current device.

[0145] Phase 3: Output of automated operation instructions:

[0146] The instruction output module outputs the restored physical coordinates and the predicted interaction type together, guiding subsequent automation tools to click accurately and adopt customized intelligent waiting strategies.

[0147] Specifically, the first stage: visual language model recognition and interaction intent prediction:

[0148] When the system initiates the recognition process, the image acquisition module first obtains a complete screenshot of the e-commerce page as input. This screenshot is then transmitted to the pre-trained visual language model processing module, whose core task is to recognize the filter bar controls on the page and all operable buttons within them (such as the "Price Sort" button, "Filter Condition Entry" button, etc.).

[0149] The model achieves accurate recognition and intent reasoning through a dual-path fusion mechanism:

[0150] Visual feature extraction: Convolutional neural networks are used to analyze the button's shape features (such as geometric outlines like rectangles and circles), color distribution (dominant color tone, gradient effects), and spatial relationships. For example, it can identify whether a button has a drop-down arrow icon or whether it uses a highlighted border to indicate its current selection.

[0151] Text semantic understanding: Utilize word embedding models to parse the semantic meaning of the button's on-text (e.g., "Prioritize Sales") and surrounding descriptive text (e.g., "Sorted by Popularity"). Simultaneously, combine page context features, such as the button's order in the filter bar and the functional relationship between adjacent buttons (e.g., the symmetry between "Price from High to Low" and "Price from Low to High").

[0152] Based on the dual-path output described above, the model performs deep matching between visual feature vectors and text semantic vectors through a cross-modal association computation layer:

[0153] Calculate the cosine similarity weight between the two to measure the strength of the association between the visual cue and the text description (e.g., the red exclamation mark icon is highly associated with the text "urgent").

[0154] Based on the weight matrix, the button interaction behavior types are classified, and three standardized identifiers are output:

[0155] Pop-up type identifier: indicates that clicking will trigger a floating panel (such as a panel that expands with filter conditions) or a hidden pop-up (such as a cascading menu).

[0156] Refresh type indicator: Indicates that clicking will only partially update the current area (such as real-time sorting and rearranging of a product list).

[0157] Jump type indicator: Indicates that clicking will switch to a new page (such as entering a separate filter configuration page).

[0158] Finally, the model outputs the position information (center point x and y coordinates) and size information (width and height) of each button in the original screenshot coordinate system. This stage breaks through the limitation of traditional tools that can only locate elements but cannot predict behavior, providing intent guidance for subsequent operations.

[0159] Phase Two: Screen Coordinate System Restoration and Physical Coordinate Adaptation

[0160] The coordinates output by the recognition module are based on the screenshot pixel coordinate system and need to be adapted to the actual display parameters of different terminal devices before operation can be performed. The coordinate adaptation module executes the following process in sequence:

[0161] Get device display parameters: Read the physical width and height values ​​of the current device screen in real time (in pixels).

[0162] Calculate the scaling factor: Compare the device screen parameters with the pre-stored baseline resolution parameters (e.g., the baseline screen size used when training the model):

[0163] Horizontal scaling factor = Device screen width ÷ Base screen width

[0164] Vertical scaling factor = Device screen height ÷ Reference screen height

[0165] Coordinate linear transformation:

[0166] Button horizontal coordinate = Original horizontal coordinate × Horizontal scaling factor

[0167] Button vertical coordinate = Original vertical coordinate × Vertical scaling factor

[0168] The button width and height scale proportionally.

[0169] Special scenario handling: When the aspect ratio of the device screen differs significantly from the baseline ratio (e.g., ultrawide mobile devices), an edge pixel filling strategy is adopted:

[0170] Fill the sides or top and bottom of the screen with non-functional buffer pixels to ensure that the button positions are not distorted after scaling.

[0171] Dynamically adjust the boundaries of the operable area to avoid click position offset caused by stretching.

[0172] This stage enables devices with different resolutions to accurately map operation coordinates, resolving cross-terminal compatibility issues.

[0173] Phase Three: Automated Operation Command Generation and Intelligent Execution

[0174] The instruction output module integrates the results from the first two stages, generates executable instructions, and customizes the verification strategy:

[0175] Instruction generation binds the adapted physical coordinates (accurate to the pixel) with the predicted interaction behavior type and encapsulates them into automated operation instructions.

[0176] The instructions include three types of operations:

[0177] Click operation: Simulates a user click at specified coordinates.

[0178] Information acquisition operation: Capture the state of page elements after a click (such as the first text of the product list after a refresh).

[0179] Analysis procedure: Compare the current state with the expected result.

[0180] Furthermore, the waiting verification logic can be dynamically adjusted based on different interaction types:

[0181] Pop-up type: After the command is triggered, the automation tool will continuously monitor the loading completion signal of the floating panel (such as the semi-transparent overlay layer being fully rendered), rather than waiting for a fixed period of time.

[0182] Refresh type: After the command is triggered, only listen for content update events in a local area (such as the product list container) and ignore other irrelevant changes on the page.

[0183] Redirect type: After the command is triggered, a page redirect timer is started and the new page loading completion is monitored. An error is reported if the timeout occurs or loading fails.

[0184] Verify closed loop:

[0185] After the operation is completed, the system automatically captures images of the page state changes and compares them with the expected process and results generated in the behavior prediction stage.

[0186] If a rendering anomaly (such as image tearing) or a logical anomaly (such as disordered sorting results) is detected, a preset fault handling process (such as retrying or log reporting) will be triggered.

[0187] This stage upgrades the traditional "blind polling" to "precise event monitoring" by linking intent prediction and verification strategies, significantly improving automation efficiency.

[0188] Therefore, this application achieves a closed loop of behavior prediction and verification, enabling the prediction of interaction intent for filter / sort controls. This allows automated tools to know what changes to expect after a click, significantly improving the reliability of functional verification. The method of this application possesses high versatility and requires zero code maintenance. Vision-LLM has universal recognition capabilities for different filter bar styles (horizontal and vertical), eliminating the need for specific code writing. It is universally applicable across devices and resolutions, introducing coordinate restoration and adaptation mechanisms to ensure the accuracy of clicking filter / sort buttons on different devices. It efficiently replaces manual interaction verification, achieving automated and reliable verification of the filter / sort function interaction process through precise control positioning and interaction intent prediction.

[0189] This application also provides a device for identifying filter bar elements and predicting interaction intent, the device comprising:

[0190] The acquisition module is used to acquire the target image corresponding to the target page;

[0191] The processing module is used to input the target image into a pre-trained visual language model, identify the filter bar control and its internal operable buttons, and output position and size information based on the target image screenshot coordinate system;

[0192] The processing module is also used to predict the interaction behavior type of each operable button based on the visual semantics and page context features of the target image;

[0193] The processing module is further configured to generate automated operation instructions based on the location information and the size information, and execute click operations, information acquisition operations and analysis operations associated with the filter bar control according to the automated operation instructions and the interaction behavior type.

[0194] This implementation uses a pre-trained visual language model to identify filter bar controls and internal buttons, and combines visual semantics with page context features to predict interaction behavior types, solving the problem that traditional tools cannot predict interaction intent. Based on location information, it generates automated instructions to execute clicks, information retrieval, and other operations, enabling the automated tool to adapt to different filter bar styles (horizontal / vertical) without requiring manual testing. Intent prediction guides subsequent intelligent waiting strategies, significantly improving the efficiency and reliability of functional verification while reducing manual verification costs.

[0195] As an optional implementation, the specific method by which the processing module predicts the interaction behavior type of each operable button based on the visual semantics and page context features of the target image includes:

[0196] Extract the visual feature vectors and semantic vectors of the surrounding text for each operable button;

[0197] The association weights between visual feature vectors and semantic vectors are calculated using a multimodal fusion method.

[0198] Based on the correlation weight matrix, the interactive behavior type identifier is output.

[0199] The multimodal fusion includes:

[0200] The visual branch uses a convolutional neural network to extract the button shape and color features of each operable button;

[0201] The text branch uses a word embedding model to extract the semantics of the subject and surrounding descriptive text of each operable button;

[0202] The cosine similarity between the visual features and the text features is calculated using a cross-modal attention layer as the association weight.

[0203] This implementation employs a multimodal fusion mechanism, using a convolutional neural network to extract visual features such as button shape and color, a word embedding model to extract semantic features from the text, and a cross-modal attention layer to calculate cosine similarity as association weights. This design deeply associates visual cues with textual descriptions, overcoming the limitations of insufficient information from a single modality. Based on a weight matrix, it classifies and outputs behavior identifiers, improving the accuracy of interaction type prediction and providing more refined intent guidance for automated operations.

[0204] As an optional implementation, the interaction behavior type identifier includes:

[0205] Pop-up type identifier, used to indicate the click behavior that triggers the floating panel or hides the pop-up;

[0206] The refresh type identifier is used to indicate the click behavior that triggers a partial update of the current area;

[0207] Jump type identifier, used to indicate the click behavior that triggers page switching.

[0208] This implementation identifies three types of interaction identifiers: pop-up, refresh, and jump. The pop-up identifier corresponds to a floating panel trigger action, the refresh identifier indicates a local area update, and the jump identifier is associated with a page switching action. By accurately matching subsequent verification strategies using these categorized identifiers (e.g., pop-ups require waiting for the floating layer to load, and jumps require monitoring page transitions), automation tools can customize waiting logic for different behavior types, avoiding invalid polling and improving verification efficiency and response accuracy.

[0209] As an optional implementation, the processing module is further configured to:

[0210] If a first page element with an interface rendering abnormality is detected in the filter bar control or the operable button, the area ratio of each first page element with rendering abnormality is calculated. When the abnormal area ratio exceeds a preset ratio, the operable area of ​​each first page element is optimized, and the coordinate information of a preset number of first page elements is retained according to the priority order of element semantic attributes.

[0211] If a second page element with abnormal business logic is detected in the filter bar control or the operable button, the code segment associated with the second page element is obtained from the document object model, the abnormal situation of the second page element is identified according to the code segment, and the preset fault handling process is called to process the second page element.

[0212] If a third page element for a behavior prediction task to be performed is detected in the filter bar control or the operable button, the expected behavior flow and expected execution result are generated according to the preset behavior prediction model. In the process of determining the execution effect of the target task based on the page state change image, the execution effect of the target task is determined based on the expected behavior flow and the expected execution result.

[0213] This implementation method targets elements with abnormal interface rendering, calculates the percentage of abnormal area, optimizes operable regions, and retains key coordinate information based on the priority of element semantic attributes to ensure the accessibility of core functions. For elements with abnormal business logic, it uses document object model code segments and tree-structured coordinate sets to locate the root cause of the abnormality, triggering a preset fault handling process to avoid interruption of automated processes. For elements with predictable tasks, it generates expected processes and results through a behavior prediction model and compares the actual effects with status monitoring. This multi-dimensional abnormality handling mechanism significantly improves the system's fault tolerance and task continuity in complex page environments.

[0214] As an optional implementation, the specific method by which the processing module trains the visual language model includes:

[0215] Obtain e-commerce training images to construct a training dataset;

[0216] For each of the e-commerce training images, the coordinate range and semantic identifier of the associated elements of the filter bar control and the internal operable buttons are marked, and the button coordinates and corresponding interaction behavior types of each operable button are marked.

[0217] A multimodal converter architecture is adopted to jointly optimize the loss function of the visual feature extraction layer, semantic understanding layer, and interaction behavior classification output layer;

[0218] Additionally, random noise and simulated display parameter fluctuations are added during training to enhance the image.

[0219] This implementation constructs a training set based on images from e-commerce scenarios, annotating control coordinates, button positions, and interaction types. It jointly optimizes the loss functions of the visual feature extraction layer, semantic understanding layer, and classification output layer. By adding random noise and simulating display parameter fluctuations, image diversity is enhanced, enabling the model to adapt to different resolutions and noise interference. This training method improves the model's generalization ability to various filter styles, ensuring robustness of recognition and intent prediction in complex page environments.

[0220] As an optional implementation, the processing module is further configured to perform the following before linearly scaling the position information and size information according to the current device display parameters to obtain the physical operation position information:

[0221] The confidence level of the output of the visual language model is verified.

[0222] When the confidence score is less than the preset threshold, the context region expansion screenshot is triggered, the target image is re-acquired, and the recognition operation is re-executed.

[0223] Additionally, the page context features of elements whose confidence scores are less than a preset threshold are recorded to optimize the training dataset.

[0224] This implementation adds a confidence verification mechanism. When the model output confidence score is lower than a threshold, it triggers a context region expansion screenshot for re-identification, avoiding erroneous operations caused by low confidence. The page context features of low-confidence elements are recorded and fed back into the training set, forming a dynamic optimization loop. This design solves the model's recognition bottleneck in edge cases (such as occluded / blurred buttons), continuously improving the model's generalization ability and recognition accuracy.

[0225] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented by processing element calls to software, while others are implemented in hardware. For example, a processing module can be a separate processing element, or it can be integrated into a chip within the device. Alternatively, it can be stored as program code in the device's memory, and its functions can be called and executed by a processing element. The implementation of other modules is similar. Moreover, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. During implementation, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0226] Indicatively, such as Figure 2 As shown, Figure 2 This is a schematic diagram of the internal structure of a computer device 300 provided in an embodiment of this application. The computer device 300 can be provided as a server. (Refer to...) Figure 2 The computer device 300 includes a processing component 302, which further includes one or more processors, and memory resources represented by memory 301 for storing instructions, such as application programs, that can be executed by the processing component 302. The application programs stored in memory 301 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 302 is configured to execute instructions to perform the methods of any of the embodiments described above.

[0227] The computer device 300 may also include a power supply component 303 configured to perform power management of the computer device 300, a wired or wireless network interface 304 configured to connect the computer device 300 to a network, and an input / output (I / O) interface 305. The computer device 300 may operate on an operating system stored in memory 301, such as Windows Server™, Mac OS X™, Unix™, Linux™, Free BSD™, or similar.

[0228] Those skilled in the art will understand that Figure 2 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0229] This application provides a storage medium storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the method provided in any embodiment.

[0230] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0231] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0232] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method of slot element recognition and interaction intent prediction, the method comprising: Includes the following steps: Get the target image corresponding to the target page; The target image is input into a pre-trained visual language model to identify the filter bar control and its internal operable buttons, and output the position and size information based on the target image screenshot coordinate system. Based on the visual semantics and page context features of the target image, predict the interaction behavior type of each operable button; Based on the location information and the size information, an automated operation instruction is generated, and click operation, information acquisition operation and analysis operation associated with the filter bar control are executed according to the automated operation instruction and the interaction behavior type.

2. The method of claim 1, wherein, The prediction of the interaction behavior type of each operable button based on the visual semantics and page context features of the target image includes: Extract the visual feature vectors and semantic vectors of the surrounding text for each operable button; The association weights between visual feature vectors and semantic vectors are calculated using a multimodal fusion method. Based on the correlation weight matrix, the interactive behavior type identifier is output. The multimodal fusion includes: The visual branch uses a convolutional neural network to extract the button shape and color features of each operable button; The text branch uses a word embedding model to extract the semantics of the subject and surrounding descriptive text of each operable button; The cosine similarity between the visual features and the text features is calculated using a cross-modal attention layer as the association weight.

3. The method of claim 2, wherein, The interaction behavior type identifier includes: Pop-up type identifier, used to indicate the click behavior that triggers the floating panel or hides the pop-up; The refresh type identifier is used to indicate the click behavior that triggers a partial update of the current area; Jump type identifier, used to indicate the click behavior that triggers page switching.

4. The method of claim 1, wherein, The method further includes: If a first page element with an interface rendering error is detected in the filter bar control and its internal operable buttons, the area ratio of each first page element with rendering error is calculated. When the abnormal area ratio exceeds a preset ratio, the operable area of ​​each first page element is optimized, and the coordinate information of a preset number of first page elements is retained according to the priority order of element semantic attributes. If a second page element with abnormal business logic is detected in the filter bar control and its internal operable buttons, the code segment associated with the second page element is obtained from the document object model, the abnormal situation of the second page element is identified according to the code segment, and the preset fault handling process is called to process the second page element. If a third page element for a behavior prediction task is detected in the filter bar control and its internal operable buttons, an expected behavior flow and an expected execution result are generated according to a preset behavior prediction model. In the process of determining the execution effect of the target task based on the page state change image, the execution effect of the target task is determined based on the expected behavior flow and the expected execution result.

5. The method according to any one of claims 1 to 4, characterized in that, The training process of the visual language model includes: Obtain e-commerce training images to construct a training dataset; For each of the e-commerce training images, the coordinate range and semantic identifier of the associated elements of the filter bar control and the internal operable buttons are marked, and the button coordinates and corresponding interaction behavior types of each operable button are marked. A multimodal converter architecture is adopted to jointly optimize the loss function of the visual feature extraction layer, semantic understanding layer, and interaction behavior classification output layer; Additionally, random noise and simulated display parameter fluctuations are added during training to enhance the image.

6. The method of claim 5, wherein, Before performing linear scaling on the position and size information based on the current device display parameters to obtain the physical operation position information, the method further includes: The confidence level of the output of the visual language model is verified. When the confidence score is less than the preset threshold, the context region expansion screenshot is triggered, the target image is re-acquired, and the recognition operation is re-executed. Additionally, the page context features of elements whose confidence scores are less than a preset threshold are recorded to optimize the training dataset.

7. A device for sifting through elements of a bar and predicting an intent of interaction, characterized by, The device includes: The acquisition module is used to acquire the target image corresponding to the target page; The processing module is used to input the target image into a pre-trained visual language model, identify the filter bar control and its internal operable buttons, and output position and size information based on the target image screenshot coordinate system; The processing module is also used to predict the interaction behavior type of each operable button based on the visual semantics and page context features of the target image; The processing module is further configured to generate automated operation instructions based on the location information and the size information, and execute click operations, information acquisition operations and analysis operations associated with the filter bar control according to the automated operation instructions and the interaction behavior type.

8. The apparatus of claim 7, wherein, The specific methods by which the processing module predicts the interaction behavior type of each operable button based on the visual semantics and page context features of the target image include: Extract the visual feature vectors and semantic vectors of the surrounding text for each operable button; The association weights between visual feature vectors and semantic vectors are calculated using a multimodal fusion method. Based on the correlation weight matrix, the interactive behavior type identifier is output. The multimodal fusion includes: The visual branch uses a convolutional neural network to extract the button shape and color features of each operable button; The text branch uses a word embedding model to extract the semantics of the subject and surrounding descriptive text of each operable button; The cosine similarity between the visual features and the text features is calculated using a cross-modal attention layer as the association weight.

9. A computer device, comprising: The method includes one or more processors and a memory storing computer-readable instructions that, when executed by the one or more processors, perform the steps of the method as described in any one of claims 1-6.

10. A storage medium, characterized by The storage medium stores computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the method as described in any one of claims 1-6.

Images