A self-repairing UI automation detection method based on multi-modal fusion

Abstract

The present application relates to the technical field of software test automation, in particular to a self-repairing UI automatic detection method based on multi-modal fusion, mainly including visual feature extraction to obtain a feature vector, and constructing a virtual DOM tree with semantic labels to perform similarity calculation to obtain a structural similarity; a feature fusion decision maker is used to dynamically adjust the weights of the feature vector and the structural similarity, output a fusion weight ratio under a current confidence threshold, and form a decision combination by fusing the feature vector and the structural similarity according to the weight ratio to generate a new locator. Through the above method, the test case vulnerability problem caused by the dynamic user interface is solved, the detection speed of real-time perception of interface changes is improved, and the success rate of positioning in use is also significantly improved.

Description

Technical Field

[0001] This invention relates to the field of software testing automation technology, and more specifically, to a self-healing UI automated detection method based on multimodal fusion. Background Technology

[0002] In scenarios where dynamic user interfaces (UIs) change frequently, test case self-maintenance refers to a systematic solution that uses technical means and design strategies to enable test cases to automatically adapt to UI changes, reduce manual maintenance costs, and maintain test stability. Its core objective is to decouple test scripts from UI elements, addressing issues such as element locators becoming invalid, layout adjustments, or changes in interaction logic through intelligent location, update, and verification mechanisms.

[0003] However, current UI automation testing technologies have the following key problems:

[0004] Locator vulnerability: It relies on XPath / CSS locators and cannot adapt to dynamic DOM IDs generated by front-end frameworks (such as React / Vue randomized IDs).

[0005] Visual layer blindness: Traditional solutions have no ability to perceive purely visual changes (such as CSS style adjustments and element overlap);

[0006] Repair delays: Manual intervention is required for maintenance, and each UI change results in an average of 2.3 hours of test interruption. Summary of the Invention

[0007] The purpose of this invention is to provide a self-healing UI automated detection method based on multimodal fusion to solve the above-mentioned problems in the prior art.

[0008] This invention is achieved through the following technical solution:

[0009] A self-healing UI automated detection method based on multimodal fusion includes:

[0010] The test executor performs anomaly detection on the current UI. If no anomaly is detected, the current instruction is executed normally. If an anomaly is detected, the anomaly point is located and it is determined whether the location was successful.

[0011] If the location is successful, the abnormal point is repaired normally. If the location is unsuccessful, visual features are extracted to obtain feature vectors, and a virtual DOM tree with semantic labels is constructed to calculate the similarity and obtain the structural similarity.

[0012] The feature fusion decision-maker dynamically adjusts the weights of feature vectors and structural similarity, and sets a confidence threshold. It then determines whether the output of the feature fusion decision-maker is less than the confidence threshold. If it is less than the confidence threshold, it outputs the fusion weight ratio under the current confidence threshold. If it is not less than the confidence threshold, it selects the fusion weight ratio based on historical data.

[0013] The feature vector and structural similarity are fused by weight ratio to form a decision combination. A new locator is generated based on the strategy combination. The new locator is then output to the test executor for anomaly detection and localization.

[0014] Preferably, the step of locating the anomaly and determining whether the location was successful includes:

[0015] Get the attributes of all UI elements and check whether the attributes of the elements are consistent with the preset attributes. Mark the target elements that are inconsistent with the preset attributes as anomalies.

[0016] The outlier is regenerated according to the preset attributes, and the attributes of the outlier are retrieved again and checked against the preset attributes. If they are inconsistent, the result of unsuccessful location is output; if they are consistent, the result of successful location is output.

[0017] Preferably, the step of extracting visual features to obtain a feature vector includes:

[0018] The system acquires a region image from the current UI screenshot, detects and identifies the region image using a pre-trained neural network model, extracts composite features from the region image, and outputs a feature vector based on these composite features.

[0019] Preferably, the detection and recognition of the region image through the pre-trained neural network model includes:

[0020] The neural network model includes a YOLOv5 element detection model, which includes a Backbone module, a loss function module, and a data augmentation module.

[0021] The Backbone module is used to extract features from the region image and output a feature image;

[0022] The loss function module is used to measure the difference between the model prediction and the true label;

[0023] The data augmentation module is used to scale, crop, or flip the feature image.

[0024] Preferably, the similarity calculation of constructing a virtual DOM tree with semantic tags includes:

[0025] Obtain DOM tree structure data, encapsulate the DOM tree structure data into reusable components, compile the reusable components to generate virtual DOM nodes containing semantic tags, and select semantic tags according to the data state;

[0026] A virtual DOM tree is generated using template syntax or the h function. The similarity of the virtual DOM nodes is calculated, including attribute similarity and tree structure similarity. The attribute similarity and tree structure similarity are then standardized and scored for structural similarity before being output.

[0027] Preferably, the attribute similarity includes:

[0028] Attributes are compared using Jaccard coefficients, including:

[0029]

[0030] In the formula, Jaccard coefficient, For the first The number of elements in the set of a certain attribute. For the first The number of elements in the set of a certain attribute.

[0031] Preferably, the tree structure similarity includes evaluating subtree matching degree using the Wu-Palmer algorithm, including:

[0032] Obtain the two subtree structures to be compared, each subtree structure including a root node, and set a target node. Find the paths from the root node to the target node of each subtree, and obtain the last common node of the two paths.

[0033] Calculate the number of edges from the root node to the common node and the level depth of the common node in the global tree;

[0034] Calculate the similarity of tree structures based on the number of edges and the level depth using Wu-Palme.

[0035] Preferably, the step of dynamically adjusting the weights of feature vectors and structural similarity using a feature fusion decision maker includes:

[0036]

[0037] In the formula, For the new weight parameters, The current weight parameters, For learning rate, To improve the success rate of historical visuals, This represents the historical DOM success rate.

[0038] Preferably, it also includes establishing a knowledge graph model, including an element module, a positioning strategy module, an application version module, and a DOM node module, and establishing relationships between them;

[0039] The element module is used to store the essential features of UI elements, including semantic fingerprints, visual feature vectors, and classification labels.

[0040] The positioning strategy module is used to record metadata of successful positioning strategies. The metadata includes a hybrid expression that combines XPath and visual coordinates, records of the visual and DOM weight configuration of the strategy, and calculation of strategy confidence based on usage records.

[0041] The application version module is used to establish version context, track version down to code commit hash, associate UI framework version and identify framework features;

[0042] The DOM node module is used to store structural features, which include XPath hashes and structural fingerprints.

[0043] Preferably, establishing the association includes:

[0044] Associate UI elements with multiple positioning strategies and set the strategy association priority;

[0045] Associate the location strategy with several different application versions, and invoke the corresponding application version based on the currently used location strategy;

[0046] Establish dynamic associations between elements and DOM nodes, triggering updates to the mapping relationship when the DOM structure changes.

[0047] The technical solution of the present invention has at least the following advantages and beneficial effects:

[0048] The method provided by this invention mainly includes extracting visual features to obtain feature vectors, constructing a virtual DOM tree with semantic labels for similarity calculation to obtain structural similarity; dynamically adjusting the weights of feature vectors and structural similarity through a feature fusion decision-maker, outputting the fusion weight ratio at the current confidence threshold, and fusing feature vectors and structural similarity according to the weight ratio to form a decision combination, generating a new locator. This method solves the vulnerability problem of test cases caused by dynamic user interfaces, improves the real-time detection speed of interface changes, and significantly improves the success rate of positioning when using the automatically generated new locator. Attached Figure Description

[0049] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a schematic diagram of the overall process of the present invention;

[0051] Figure 2 This invention is based on Figure 1 Detailed flowchart;

[0052] Figure 3 This is a schematic diagram of the knowledge graph structure of the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0054] The module division in this application is a logical division. In actual application, there may be other division methods. For example, multiple modules may be combined into or integrated into another system, or some features may be ignored or not executed.

[0055] The independently described modules or sub-modules may or may not be physically separated; they may be implemented in software or hardware, and some modules or sub-modules may be implemented in software, with the processor calling the software to implement the function of these modules or sub-modules, while other modules or sub-modules may be implemented in hardware, such as through hardware circuits. Furthermore, some or all of the modules can be selected to achieve the purpose of this application's solution according to actual needs.

[0056] Please refer to Figures 1-3 This invention provides a self-healing UI automated detection method based on multimodal fusion, comprising:

[0057] S101: Perform anomaly detection on the current UI through the test executor. If no anomaly is detected, execute the current instruction normally. If an anomaly is detected, locate the anomaly point and determine whether the location was successful.

[0058] In this invention, UI refers to the user interface, specifically the totality of all visual and auditory elements, such as screens, menus, buttons, icons, text, audio, and video, involved in a user's interaction with a computer or other electronic device. The UI needs to be detected during display to reflect whether the current UI interface is functioning correctly and to perform repairs. This invention proposes a further solution regarding the location of anomalies.

[0059] S102: If the location is successful, the abnormal point is repaired normally. If the location is unsuccessful, visual features are extracted to obtain feature vectors, and a virtual DOM tree with semantic labels is constructed to calculate the similarity and obtain the structural similarity.

[0060] A virtual DOM tree with semantic tags refers to the use of identifiers with explicit semantics (such as HTML5 semantic tags) in the tag names (tag attributes) of virtual DOM nodes. <header> 、 <nav> 、 <section>(etc.), instead of just using general container tags (such as...) 、 This tree structure not only describes the hierarchical relationship of interface elements, but also directly conveys their function or content type through tag names, forming a "self-explanatory" code structure.

[0061] To enhance readability and maintainability, semantic tags use intuitive naming (e.g., ...). <navigation> 、 <footer>) Replace the general This enables developers and tools to quickly understand component functionality, reducing code maintenance costs. Combined with the abstraction layer characteristics of the virtual DOM, semantic tags can achieve logic reuse independently of specific platforms; for example, the same semantic tag system can be used on both web and mobile devices.

[0062] Enhanced accessibility (A11Y) support; semantic tags are naturally compatible with assistive technologies such as screen readers. For example, <nav>The labels will be automatically recognized as navigation areas, and <button>The semantic attributes of a label (such as aria-label) can clearly define the interaction intent.

[0063] Optimize search engine optimization (SEO) and rendering performance; search engines can more easily parse the content weight of semantic tags (such as...). <main>(Representing core content), improving page content indexing efficiency. In the virtual DOM's Diff algorithm, semantic tags can serve as key path identifiers, reducing the complexity of cross-level comparisons and thus accelerating difference detection.

[0064] Precise differential updates and cross-platform adaptation: Semantic tags help the Diff algorithm more accurately locate the subtree that needs updating. For example, updating only... <form>Instead of the entire form container, the changing fields are defined, reducing unnecessary DOM operations. In cross-platform scenarios (such as React Native), semantic tags can be mapped to native components (such as iOS's UINavigationBar) for efficient rendering. Furthermore, in this invention, DOM stands for Document Object Model.

[0065] S103: The feature fusion decision-maker dynamically adjusts the weights of feature vectors and structural similarity, sets a confidence threshold, and determines whether the output of the feature fusion decision-maker is less than the confidence threshold. If it is less than the confidence threshold, the fusion weight ratio under the current confidence threshold is output. If it is not less than the confidence threshold, the fusion weight ratio is selected based on historical data.

[0066] In this embodiment, the purpose of setting a confidence threshold is to balance the accuracy and reliability of the results. The confidence threshold reduces the risk of false positives by excluding predictions with low confidence. For example, in object detection, the model may generate multiple bounding box predictions, but only boxes with confidence scores higher than the threshold will be retained. This filtering mechanism effectively reduces erroneous outputs caused by model uncertainty. In this embodiment, it is set to 0.85. Secondly, regarding the selection of fusion weight ratios based on historical data, this essentially involves staff selecting weight ratios based on historical data. Since the model cannot output values ​​below the specified confidence threshold, manual intervention is required.

[0067] S104: The feature vector and structural similarity are fused by weight ratio to form a decision combination. A new locator is generated according to the strategy combination. The new locator is output to the test executor for anomaly detection and localization.

[0068] The decision combination is calculated by taking the feature vector and structural similarity as a percentage, and it reflects the results judged by visual feature extraction and the results judged by the virtual DOM tree. By allocating them in a certain proportion, the final result judgment value is output.

[0069]

[0070] In the formula, The final calculation is used to determine whether the location was successful. The proportion used to calculate the eigenvectors. This represents the proportion used to calculate structural similarity. For feature vectors, For structural similarity.

[0071] The method provided by this invention mainly includes extracting visual features to obtain feature vectors, constructing a virtual DOM tree with semantic labels for similarity calculation to obtain structural similarity; dynamically adjusting the weights of feature vectors and structural similarity through a feature fusion decision-maker, outputting the fusion weight ratio at the current confidence threshold, and fusing feature vectors and structural similarity according to the weight ratio to form a decision combination, generating a new locator. This method solves the vulnerability problem of test cases caused by dynamic user interfaces, improves the real-time detection speed of interface changes, and significantly improves the success rate of positioning when using the automatically generated new locator.

[0072] In one exemplary embodiment of the present invention, the step of locating the anomaly and determining whether the location was successful includes:

[0073] Get the attributes of all UI elements and check whether the attributes of the elements are consistent with the preset attributes. Mark the target elements that are inconsistent with the preset attributes as anomalies.

[0074] The outlier is regenerated according to the preset attributes, and the attributes of the outlier are retrieved again and checked against the preset attributes. If they are inconsistent, the result of unsuccessful location is output; if they are consistent, the result of successful location is output.

[0075] In one exemplary embodiment of the present invention, visual feature extraction to obtain a feature vector includes:

[0076] The system acquires a region image from the current UI screenshot, detects and identifies the region image using a pre-trained neural network model, extracts composite features from the region image, and outputs a feature vector based on these composite features.

[0077] Specifically, the composite features include shape features: extracting 64-dimensional local feature descriptors using the SURF algorithm; and color features: calculating a 16×16×16 three-dimensional histogram in HSV space.

[0078] The output feature vector is a 4160-dimensional visual feature vector (SURF mean + histogram expansion).

[0079] Specifically, the detection and recognition of the region image using the pre-trained neural network model includes:

[0080] The neural network model includes a YOLOv5 element detection model, which includes a Backbone module, a loss function module, and a data augmentation module.

[0081] The Backbone module is used to extract features from the region image and output a feature image. Specifically, it is a convolutional neural network. The backbone network used by YOLOv5 is based on CSPDarknet53 or other lightweight network structures, such as EfficientNeta.

[0082] The loss function module measures the difference between the model's predictions and the true labels, and uses backpropagation to adjust the model parameters to minimize this difference. YOLOv5 uses loss functions commonly found in object detection, such as mean squared error and cross-entropy loss.

[0083] The data augmentation module transforms and distorts the input image during training to expand the training dataset and improve the model's generalization ability. Augmentation operations include random scaling, cropping, rotation, and flipping.

[0084] More specifically, a small object detection layer can be added to detect even smaller features. A new anchor size can be added to the anchors section of the YOLOv5 model configuration file. This solution uses [4,5,8,10,22,18] to accommodate the detection requirements of small objects.

[0085] The location where a small object detection layer needs to be added is determined, and a residual network mechanism for generating 160×160 feature maps is added to the Backbone module. Since the original network's Neck does not have a 160×160 feature map, the 80×80 feature map needs to be upsampled to obtain a 160×160 feature map, thereby improving the model's ability to detect small objects and performing better in dense scenes.

[0086] A new convolutional layer is introduced to achieve a 160×160 detection feature map. This convolutional layer receives upsampled results from an 80×80 feature map as input and is adapted to the small target detection requirements in dense scenes through specific parameter settings. Appropriately adjusting the kernel size and stride can better capture the features of small targets and improve detection accuracy.

[0087] The newly added 160×160 detection feature map is connected to other layers of the backbone network and the backbone module to ensure smooth information transfer and effective feature extraction. This ensures that the model can fully utilize the information provided by the 160×160 feature map, thereby improving performance in dense scenes.

[0088] In one exemplary embodiment of the present invention, the similarity calculation of constructing a virtual DOM tree with semantic tags includes:

[0089] Obtain DOM tree structure data, encapsulate the DOM tree structure data into reusable components, compile and generate virtual DOM nodes containing semantic tags through the reusable components, and select semantic tags according to the data status; generate virtual DOM trees through template syntax or h functions, calculate the similarity of virtual DOM nodes, the similarity includes attribute similarity and tree structure similarity, and output the standardized structural similarity score of attribute similarity and tree structure similarity.

[0090] Specifically, building a virtual DOM tree with semantic tags requires using semantic tag names when creating virtual nodes. For example, using... <main>replace Use this to wrap the main content. At this point, you need to refer to the virtual DOM generation methods in various frameworks, such as React's createElement function or Vue's rendering function.

[0091] Next, we need to consider how to organize the structure of these semantic tags. For example, following the standard HTML5 structure, the page can be divided into sections such as header, main, and footer. In the virtual DOM tree, each node needs to correctly represent these tags and maintain the hierarchical structure.

[0092] Additionally, ARIA roles or other accessibility attributes need to be added to enhance semantics. For example, add `role="navigation"` to the navigation bar. This can be set in the properties object of the virtual DOM node.

[0093] Furthermore, component-based development may involve encapsulating semantic tags into components, such as creating a... <customheader>Components, used internally <header>Tags and other semantic elements. This way, when building the virtual DOM tree, these components are converted into corresponding virtual nodes, preserving the semantic structure.

[0094] It's also important to note how to dynamically generate semantic tags in a data-driven context. This might require selecting different tags based on the data, such as conditionally rendering different semantic elements. In this case, it's crucial to ensure that the generated virtual DOM node tags are correct.

[0095] Finally, testing and verification are also important steps. Use tools to check whether the virtual DOM tree is correctly mapped to semantic tags, ensure that the generated HTML structure conforms to standards, and has good accessibility.

[0096] In one exemplary embodiment of the present invention, the attribute similarity includes:

[0097] Attributes are compared using Jaccard coefficients, including:

[0098]

[0099] In the formula, Jaccard coefficient, For the first The number of elements in the set of a certain attribute. For the first The number of elements in a set with certain attributes, including class, role, etc.

[0100] In one exemplary embodiment of the present invention, the tree structure similarity includes evaluating subtree matching degree using the Wu-Palmer algorithm, comprising:

[0101] Obtain the two subtree structures to be compared, each subtree structure including a root node, and set a target node. Find the paths from the root node to the target node of each subtree, and obtain the last common node of the two paths.

[0102] The subtree structure includes the root node and all its child nodes (recursively defined). Node attributes: Each node needs to record its level depth (the number of edges from the root node to that node).

[0103] Calculate the number of edges from the root node to the common node and the level depth of the common node in the global tree;

[0104] Calculate the similarity of tree structures based on the number of edges and the level depth using Wu-Palme.

[0105] To improve sensitivity to the internal structure of subtrees, a weighted adjustment can be made by combining node coverage and hierarchical alignment.

[0106] Regarding the standardized structural similarity scoring of attribute similarity and tree structure similarity, including:

[0107]

[0108] In the formula, This is the structural similarity index between two subtree structures. The total number of node pairs. For attribute similarity weights, For attribute similarity function, For tree structure similarity weights, This is a tree structure similarity function based on Wu-Palme.

[0109] In one exemplary embodiment of the present invention, the dynamic weight adjustment of feature vectors and structural similarity via a feature fusion decision maker includes:

[0110]

[0111] In the formula, For the new weight parameters, The current weight parameters, For learning rate, To improve the success rate of historical visuals, This represents the historical DOM success rate.

[0112] In object detection, if the model successfully identifies and locates objects in 90% of the test images, then... In image generation tasks, visual verification refers to the proportion of generated images that are judged by humans or automated tools to meet expectations. It is the proportion of test cases that pass visual verification out of the total test cases, and is usually used to detect whether the interface rendering is consistent with expectations.

[0113] This is a metric used to measure the success rate of automated tasks based on DOM (Document Object Model) validation. It is commonly used in front-end development, automated testing, web scraping, or performance monitoring. In this invention, the calculation rule is the number of operations that successfully completed DOM validation divided by the total number of operations multiplied by 100%.

[0114] An exemplary embodiment of the present invention further includes establishing a knowledge graph model, including an element module, a positioning strategy module, an application version module, and a DOM node module, and establishing association relationships;

[0115] The element module (ELEMENT) is used to store the essential features of UI elements, including semantic fingerprints, visual feature vectors, and classification labels;

[0116] Specifically, semantic fingerprints are unique identifiers generated based on element text, ARIA tags, etc., visual feature vectors are 128-dimensional vectors (SURF+HSV histograms), and classification labels guide multimodal weight allocation (buttons emphasize CV, input boxes emphasize DOM).

[0117] The location strategy module (LOCATION STRATEGY) is used to record metadata of successful location strategies. The metadata includes a hybrid expression that combines XPath and visual coordinates, records of the visual and DOM weight configuration of the strategy, and calculation of strategy confidence based on usage records.

[0118] The application version module (APP_VERSION) is used to establish a version context, track versioning down to the code commit hash, associate UI framework version, and identify framework characteristics.

[0119] The DOM node module (DOM_NODE) ​​is used to store structural features, including XPath hashes and structural fingerprints.

[0120] In addition, establishing relationships includes:

[0121] Associate UI elements with multiple positioning strategies and set the strategy association priority;

[0122] One element can correspond to multiple historical positioning strategies;

[0123] Strategy selection priority: success rate > most recently used > version compatibility;

[0124] Example: The login button has 3 successful targeting strategies.

[0125] Associate the location strategy with several different application versions, and invoke the corresponding application version based on the currently used location strategy;

[0126] Policy validity scope: defined through semantic versioning; automatically matches historical policies during version rollback; supports policy reuse across minor versions.

[0127] Establish dynamic associations between elements and DOM nodes. When the DOM structure changes, the mapping relationship is updated, and a bidirectional index is maintained: element → DOM node (primary index) and DOM node → element (inverted index).

[0128] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0129] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. This computer software product, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0130] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.< / header> < / customheader> < / main> < / form> < / main> < / button> < / nav> < / footer> < / navigation> < / section> < / nav> < / header>

Claims

1. A self-repairing UI automation detection method based on multi-modal fusion, characterized in that, The method comprises the following steps: abnormality detection is performed on the current UI by a test executor, if no abnormality is detected, the current instruction is executed normally, if abnormality is detected, the abnormal point is located, and it is judged whether the location is successful; if the location is successful, the abnormal point is repaired normally, if the location is not successful, a feature vector is obtained by visual feature extraction, a virtual DOM tree with a semantic label is constructed, and a structural similarity is calculated to obtain a structural similarity; the feature vector and the structural similarity are dynamically adjusted by a feature fusion decision maker, a confidence threshold is set, it is judged whether the output result of the feature fusion decision maker is less than the confidence threshold, if it is less than the confidence threshold, the fusion weight ratio under the current confidence threshold is output, if it is not less than the confidence threshold, the fusion weight ratio is selected according to historical data; a decision combination is formed by fusing the feature vector and the structural similarity through the weight ratio, a new locator is generated according to the strategy combination, and the new locator is output to the test executor for abnormality detection and location; the feature vector obtained by visual feature extraction comprises: a region image of a current UI interface screenshot is obtained, a composite feature of the region image is extracted by detecting and recognizing the region image through a pre-trained neural network model, and a feature vector is output based on the composite feature; the construction of the virtual DOM tree with the semantic label for similarity calculation comprises: DOM tree structure data is obtained, the DOM tree structure data is encapsulated as a reusable component, a virtual DOM node containing a semantic label is generated by compiling the reusable component, and a semantic label is selected according to the data state; a virtual DOM tree is generated through a template syntax or an h function, the similarity of the virtual DOM node is calculated, the similarity comprises an attribute similarity and a tree structure similarity, and the attribute similarity and the tree structure similarity are standardized to obtain a structural similarity score, and the structural similarity score is output; the dynamic adjustment of the feature vector and the structural similarity by the feature fusion decision maker comprises: wherein, is a new weight parameter, is a current weight parameter, is a learning rate, is a historical visual success rate, is a historical DOM success rate; the standardization of the attribute similarity and the tree structure similarity to obtain the structural similarity score comprises: wherein is the structure similarity index of two sub-tree structures, is the total number of node pairs, is the attribute similarity weight, is the attribute similarity function, is the tree structure similarity weight, is the Wu-Palme based tree structure similarity function.

2. The self-repairing UI automation detection method based on multi-modal fusion according to claim 1, characterized in that, the location of the abnormal point and the judgment of whether the location is successful comprise: all attributes of elements of the UI are obtained, and it is checked whether the attributes of the elements are consistent with preset attributes, target elements that are inconsistent with the preset attributes are marked as abnormal points; the abnormal points are generated again according to the preset attributes, the attributes of the abnormal points are obtained again and checked with the preset attributes, if the attributes are inconsistent with the preset attributes, a result that the location is unsuccessful is output, if the attributes are consistent with the preset attributes, a result that the location is successful is output.

3. The self-repairing UI automation detection method based on multi-modal fusion according to claim 2, characterized in that, the detection and recognition of the region image by the pre-trained neural network model comprise: the neural network model comprises a YOLOv5 element detection model, the YOLOv5 element detection model comprises a Backbone module, a loss function module and a data enhancement module; the Backbone module is used for extracting features from the region image and outputting a feature image; the loss function module is used for measuring the difference between the model prediction and the real label; the data enhancement module is used for scaling, cropping or flipping the feature image.

4. The self-repairing UI automation detection method based on multi-modal fusion according to claim 3, characterized in that, the attribute similarity comprises: Attributes are compared using Jaccard coefficients, including: In the formula, is the Jaccard coefficient, is the number of element sets of the th attribute, is the number of element sets of the th attribute.

5. The self-repairing UI automation detection method based on multi-modal fusion according to claim 4, characterized in that, The tree structure similarity includes evaluating subtree matching using the Wu-Palmer algorithm, including: Obtain the two subtree structures to be compared, each subtree structure including a root node, and set a target node. Find the paths from the root node to the target node of each subtree, and obtain the last common node of the two paths. Calculate the number of edges from the root node to the common node and the level depth of the common node in the global tree; Calculate the similarity of tree structures based on the number of edges and the level depth using Wu-Palme.

6. The self-repairing UI automation detection method based on multi-modal fusion according to claim 5, characterized in that, It also includes building a knowledge graph model, including an element module, a location strategy module, an application version module, and a DOM node module, and establishing relationships between them; The element module is used to store the essential features of UI elements, including semantic fingerprints, visual feature vectors, and classification labels. The positioning strategy module is used to record metadata of successful positioning strategies. The metadata includes a hybrid expression that combines XPath and visual coordinates, records of the visual and DOM weight configuration of the strategy, and calculation of strategy confidence based on usage records. The application version module is used to establish version context, track version down to code commit hash, associate UI framework version and identify framework features; The DOM node module is used to store structural features, which include XPath hashes and structural fingerprints.

7. The self-repairing UI automation detection method based on multi-modal fusion according to claim 6, characterized in that, The establishment of the association includes: Associate UI elements with multiple positioning strategies and set the strategy association priority; Associate the location strategy with several different application versions, and invoke the corresponding application version based on the currently used location strategy; Establish dynamic associations between elements and DOM nodes, triggering updates to the mapping relationship when the DOM structure changes.

Images