A screen interface element detection method based on knowledge guidance and multimodal fusion
By constructing a knowledge mapping base and a multimodal fusion detection method, the problems of missed detection, false detection and low efficiency in screen interface element detection are solved, achieving efficient and accurate screen interface element detection and improving detection accuracy and robustness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- XIDIAN UNIV
- Filing Date
- 2026-01-23
- Publication Date
- 2026-05-26
AI Technical Summary
Existing technologies suffer from several problems when detecting screen interface elements, including missed detections of targets without text, false detections of targets with text, low detection efficiency, and insufficient ability to distinguish similar targets. They are particularly difficult to achieve efficient and accurate detection in complex application environments.
By constructing a knowledge mapping library, prior domain knowledge is transformed into structured detection constraints. Combining OCR text recognition and multimodal fusion detection, a two-stage strategy of candidate region directional detection and full-image supplementation detection is adopted to generate accurate candidate regions and perform fusion and deduplication to output the final detection results.
It significantly reduces the false negative rate for targets without text and the false positive rate for targets with text, improves detection efficiency, enhances the ability to distinguish complex interface layouts and similar targets, and meets real-time requirements.
Smart Images
Figure CN122090469A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision technology, specifically relating to a screen interface element detection method based on knowledge guidance and multimodal fusion. Background Technology
[0002] In today's rapidly developing information technology environment, the accurate identification and positioning of screen interfaces, as the primary medium for human-computer interaction, has become a core requirement in many key technology fields. In software automation testing, precise identification of interface elements is needed to achieve procedural operational interactions; in user behavior research, the identification of interface elements is required to reconstruct user operation sequences and usage preferences; and in the analysis and processing of operation demonstration videos, accurate extraction of key interface elements is necessary to achieve automated tutorial generation.
[0003] However, current mainstream interface element detection technologies still face many challenges in practical application. First, for visual elements without textual information (such as graphic icons and decorative lines), existing methods often struggle to achieve stable and reliable detection due to a lack of effective recognition cues, resulting in insufficient recognition coverage. Second, the abundant textual information in the interface often interferes with the detection process. On the one hand, it may misclassify non-interactive text as operable elements; on the other hand, when the spatial relationship between visual elements and their associated text is complex, positioning errors or misidentifications are prone to occur. Furthermore, to ensure detection coverage, existing technologies typically require a global scan of the entire interface. This approach incurs significant computational overhead, making it difficult to meet the demands of applications with high real-time requirements. Finally, most existing detection methods rely on learning general features from training data, failing to effectively utilize domain-specific knowledge contained in software specifications, design documents, and other materials. Consequently, the accuracy and stability of detection results still need improvement when faced with interfaces with similar visual features and complex layout structures.
[0004] Therefore, the industry urgently needs a screen interface element recognition solution that can achieve efficient and accurate detection in complex application environments to address the various challenges currently facing the technology. Summary of the Invention
[0005] To address the aforementioned problems in existing technologies, this invention provides a screen interface element detection method based on knowledge guidance and multimodal fusion. The technical problem to be solved by this invention is achieved through the following technical solution: This invention provides a screen interface element detection method based on knowledge guidance and multimodal fusion, comprising: Acquire keyframes of images in the target scene; the keyframes refer to image frames selected from image data based on changes in image content. Using an OCR text recognition algorithm, the text information of the keyframes of the image is extracted and a text list is output. The text list contains one or more text entries, and each text entry contains a text content, the region coordinates corresponding to the text content, and the text confidence score. Using a pre-built knowledge mapping library, text entries in the text list that meet preset conditions are filtered to obtain text entries with a matching degree greater than or equal to a matching threshold, thus forming a valid text list. The knowledge mapping library is constructed using standard documents related to the target scenario, including user manuals, design specifications, or development documents. The knowledge mapping library contains one or more knowledge entries, each corresponding to a target category of a screen interface element, as well as the standard text set, text position rules, and target size range for that target category. Based on the region coordinates of each valid text in the list of valid texts, and combined with the text position rules and target size range of the corresponding target category in the knowledge mapping library, a corresponding candidate region is calculated and generated to obtain a candidate region list. Using the trained target detection model, a first round of detection is performed on each candidate region in the candidate region list, and a second round of detection is performed on the image keyframe or the regions in the image keyframe other than the candidate regions, to obtain the candidate region detection results and the full image detection results, respectively. The candidate region detection results and the full image detection results are fused and deduplicated to output the final screen interface element detection results.
[0006] Compared with the prior art, the beneficial effects of the present invention are as follows: To address the shortcomings of existing screen interface element detection methods, such as missed detections of targets without text, false detections of targets with text, low detection efficiency, and insufficient ability to distinguish similar targets, this invention provides a screen interface element detection method based on knowledge guidance and multimodal fusion. By constructing a knowledge mapping library, prior domain knowledge is transformed into structured detection constraints, establishing a new detection paradigm that deeply integrates knowledge guidance and multimodal information. First, text is extracted using OCR and matched with the knowledge library to generate valid text. Then, precise candidate regions are generated by combining text position rules and target size ranges. A two-stage strategy combining candidate region-oriented detection and full-image supplementation detection is then employed. Finally, the detection results are output through fusion and deduplication. This innovative architecture not only significantly reduces the missed detection rate of targets without text and the false detection rate of targets with text, but also greatly improves processing speed by optimizing the detection area. Furthermore, knowledge guidance enhances the model's ability to distinguish complex interface layouts and similar targets, providing a comprehensive and effective solution for screen interface element detection. Attached Figure Description
[0007] Figure 1This is a flowchart illustrating the screen interface element detection method based on knowledge guidance and multimodal fusion provided in an embodiment of the present invention. Figure 2 This is a flowchart illustrating the construction process of the knowledge mapping library provided in this embodiment of the invention. Detailed Implementation
[0008] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.
[0009] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.
[0010] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings, disclosure, and appended claims in carrying out the claimed invention. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.
[0011] The present invention will now be described in detail with reference to the accompanying drawings, a screen interface element detection method based on knowledge guidance and multimodal fusion.
[0012] Figure 1 This is a flowchart illustrating the screen interface element detection method based on knowledge guidance and multimodal fusion provided in an embodiment of the present invention. Figure 1 As shown, steps 110-160 are included. Specifically: S110: Acquire image keyframes in the target scene; image keyframes refer to image frames selected from image data based on changes in the scene content.
[0013] For example, the target scenario can be understood as the specific business area or task type to which the method of the present invention is applied. For instance, in a software automation testing scenario, it is necessary to continuously capture or record the software's interface and detect elements such as the "Save" button, "Font Drop-down," and "Bold" icon in real time to simulate user clicks and input. In a user behavior analysis scenario, after obtaining user authorization, the process of the user operating the screen is recorded. By analyzing the video stream, frequently clicked interface elements (such as the "Moments" icon and the "Send" button) are detected, thereby drawing a user behavior path diagram for optimizing product design. As another example, in an operation demonstration video analysis scenario, key interface elements involved in each operation step (such as the "Layers" panel and the "Filters" menu) are located and associated with operation instructions (such as "Click the 'Filters' menu") to achieve automated tutorial generation.
[0014] Here, image keyframes refer to representative static image frames selected from continuous image data (such as video streams or consecutive screenshots) based on significant changes in the image content. The purpose is to avoid repeatedly calculating consecutive frames with similar content, thereby greatly improving processing efficiency. For example, in a software automation testing scenario, a 30-frame video is recorded. The content change between frame 1 and frame 2 is minimal (only cursor position movement), and is not considered a key change. From frame 1 to frame 29, the main interface elements (menu bar, toolbar, blank document area) do not undergo structural changes. Therefore, the system might select a frame (such as frame 1) from this interval as a representative frame. Frame 30 shows a significant change in content compared to any previous frame—a new "Save As" dialog box appears. The system will then only process frame 1 (representing the "New Document" state) and frame 30 (representing the "After Clicking Save" state) as image keyframes.
[0015] For example, a demonstration video of the target software (resolution 1920×1080, frame rate 30fps) is selected, and video frames are read through the cv2.VideoCapture interface of OpenCV. The inter-frame difference method is used to filter key frames (inter-frame pixel difference ≥5% is determined to be a key frame) to reduce redundant calculations.
[0016] S120: Using an OCR text recognition algorithm, extract text information from keyframes of an image and output a text list. The text list contains one or more text entries, and each entry contains a text content, the corresponding region coordinates, and the text confidence score.
[0017] For example, a high-precision OCR algorithm (such as PP-OCRv4) is used to perform a full-image scan and text recognition on the image, converting the pixel information in the image into structured text data. Each text entry contains information in three dimensions: 1. Text content: The recognized character sequence; for example: "Now playing", "volume" and "settings".
[0018] 2. Region coordinates: The bounding box coordinates of the text instance's location in the image, usually defined in pixels; for example, the region coordinates corresponding to the string "playing now" are (30, 8, 100, 22), the region coordinates corresponding to the string "volume" are (300, 5, 330, 20), and the region coordinates corresponding to the string "settings" are (350, 5, 375, 20).
[0019] 3. Text Confidence: The probability assessment value of the OCR algorithm's accuracy in recognizing this text instance. For example, the text confidence score for the string "Now playing" is 0.92, the text confidence score for the string "volume" is 0.85, and the text confidence score for the string "settings" is 0.90.
[0020] S130: Using a pre-built knowledge mapping library, text entries in the text list that meet preset conditions are filtered to obtain text entries with a matching degree greater than or equal to the matching threshold, forming a valid text list; wherein, the knowledge mapping library is built using the standard documents involved in the target scenario, including user manuals, design specifications or development documents; the knowledge mapping library contains one or more knowledge entries, each knowledge entry corresponds to a target category of a screen interface element, as well as the standard text set, text position rules and target size range of that target category.
[0021] This step aims to associate the text information recognized by OCR with prior domain knowledge, filtering out text content highly relevant to the object detection task. Its processing flow comprises two core stages: confidence filtering and knowledge matching.
[0022] Here, the filtering process includes: filtering out text entries with a text confidence score lower than a preset confidence threshold, resulting in multiple filtered text entries; performing fuzzy matching on these multiple filtered text entries with a standard text set in the knowledge mapping base, generating multiple one-to-one matching scores; where fuzzy matching refers to using a string similarity calculation method based on edit distance to calculate the similarity between the text content of each filtered text entry and each text string in the standard text set. For example, the preset confidence threshold is set to 0.8.
[0023] Then, the filtered text entries are subjected to fuzzy matching with the standard text set in the knowledge mapping base. The matching operation is performed on a single text entry basis, comparing its text content with all standard text strings under a certain target category in the knowledge base one by one. Fuzzy matching uses a string similarity calculation method based on edit distance. It quantifies the similarity by calculating the minimum number of single-character edit operations required to convert between two strings and outputs a normalized matching score.
[0024] Finally, the system will filter out all text entries with a matching score greater than or equal to a preset matching threshold (e.g., 0.9) and classify them as valid text. All these valid texts together constitute a list of valid texts. Each valid text in this list is explicitly associated with a specific target category in the knowledge mapping base, laying a solid foundation for the subsequent generation of targeted candidate detection regions.
[0025] here, Figure 2 This is a flowchart illustrating the construction process of the knowledge mapping library provided in this embodiment of the invention. For example... Figure 2 As shown, the knowledge mapping library is constructed in the following way: S1: Extract and define multiple target categories from the specification document, with each target category corresponding to a screen interface element.
[0026] Here, the target category is composed of two main dimensions: First, the functional dimension. For example, operable elements of interface components that support user interaction, including but not limited to: buttons (which can be further subdivided into subtypes such as "Confirm Button", "Cancel Button", and "Function Button"), input boxes, drop-down menus, checkboxes, and sliders; and identifying elements of interface components that mainly serve to display information or provide visual separation, including but not limited to: text labels, icons, separators, and progress bars. Second, the granularity dimension. For example, the basic category (button), and sub-categories (further divided according to specific application scenarios and functional semantics, such as subdividing "button" into "Confirm Button", "Cancel Button", "Minimize Button", etc.).
[0027] Taking a video player scenario as an example, by analyzing user manuals, UI design guidelines, and development documents, the following target categories can be extracted: (1) Playback controls: play / pause button, progress bar, volume slider; (2) Navigation buttons: Home button, Back button, Menu button; (3) Settings: Subtitle selection drop-down menu, video quality switching button, settings icon; (4) Identifiers: program duration tags, full-screen icon, separator line.
[0028] Table 1 is an example of target category extraction provided in the embodiments of the present invention.
[0029] Table 1
[0030] This systematic extraction and definition process ensures that the knowledge mapping library can fully cover all screen interface elements to be detected in the target scene, providing a complete category system foundation for subsequent detection processes.
[0031] S2: For each target category, perform the following operations to generate its structured knowledge entries: Extract one or more related texts from the specification document to form the standard text set for the target category; define the text position rules of the related texts within the corresponding screen interface elements to be detected. The text position rules are used to limit the relative positional relationship between the related texts and the screen interface elements to be detected, including the alignment and the offset from the element edge; calculate the target size range of the target category at the specified screen resolution; the target size range is used to limit the pixel range of the width and height of the screen interface elements to be detected.
[0032] For example, extract descriptive text related to the target category from user manuals, operation guides, and other documents. For example, the text set corresponding to the "Save Button" is: ["Save", "Save", "Store"]; for elements without text (such as icons), the text set is empty []. Subsequently, based on UI design specifications, clearly define the layout rules of the text within the elements. Specifically, alignment methods include center alignment, left alignment, right alignment, top alignment, bottom alignment, etc.; offset includes the distance between the text and the edge of the element (unit: pixels). For example, the associated text "Font Size" of the "Font Settings Input Box" needs to be left-aligned and located on the left side of the input box, with an offset of 8 pixels. Finally, based on design specifications and actual interface measurements, statistically analyze the typical size of this category at different resolutions: collect the actual size of the element in multiple interface instances; calculate the reasonable range of width and height; and store the data hierarchically according to screen resolution. For example, at 1080P resolution, the size range of the "OK button" is [60-120, 25-40] (width × height), and at 4K resolution, the size range of the "OK button" is [120-240, 50-80].
[0033] S3: Store all knowledge entries for all target categories in a structured format to build a knowledge mapping library.
[0034] For example, it can be stored in JSON format, XML format, database table format, and YAML format.
[0035] It should be noted that the knowledge mapping library supports dynamic updates. Field content can be expanded by importing new scenario manuals, manually adding annotations, etc., to ensure compatibility with different software interfaces.
[0036] S140: Based on the region coordinates of each valid text in the list of valid texts, and combined with the text position rules and target size range of the corresponding target category in the knowledge mapping base, calculate and generate the corresponding candidate region to obtain the candidate region list.
[0037] Specifically, S140 includes performing the following operations for each valid text: Based on the region coordinates of the valid text, determine the geometric center coordinates, width, and height of its text bounding box; according to the text position rules of the target category corresponding to the valid text in the knowledge mapping library, determine the direction and offset of pixel expansion, and based on the corresponding target size range, determine the expected target width and expected target height; based on the difference between the expected target width and the width of the text bounding box, and the difference between the expected target height and the height of the text bounding box, determine the required expansion pixel size to generate a rectangular candidate region that completely covers the screen interface element to be detected corresponding to the valid text; summarize all the rectangular candidate regions to form a candidate region list.
[0038] In one possible implementation, for each valid text, based on its region coordinates... The calculated geometric center coordinates of the text bounding box are represented as follows: Width is represented as The height is .
[0039] Retrieve the text position rules (including alignment and offset) for the target category corresponding to the valid text from the knowledge mapping base. ), and the target size range (including the expected target width). Highly in line with expected goals Therefore, the formula for calculating the required expanded pixels is: ; ; Then, the formula for calculating the rectangular candidate region corresponding to the valid text is: .
[0040] Here, after obtaining the candidate region list, the process also includes: when the overlap between any two candidate regions is greater than or equal to a preset overlap threshold, or the adjacent distance between them is less than or equal to a preset distance threshold, performing a region merging or filtering operation to ensure that each candidate region in the candidate region list corresponds to an independent screen interface element to be detected.
[0041] For example, the intersection-to-union ratio (IoU) is used as the evaluation metric to assess the degree of overlap; the adjacent spacing is determined by calculating the minimum Euclidean distance between the boundaries of two candidate regions. Here, the preset overlap threshold ranges from 0.5 to 0.7, and the preset spacing threshold ranges from 5 to 15 pixels.
[0042] Here, corresponding optimization strategies are adopted based on the different distribution characteristics of candidate regions. Specifically, for candidate regions that overlap or are closely adjacent, if they correspond to the same target category or have semantic relevance, their minimum bounding rectangle is calculated as a new candidate region to be merged, i.e., the region merging operation. For candidate regions that overlap but need to retain independence, the following selection criteria are adopted: 1. Retain the candidate region with the larger area; 2. Retain the candidate region with a higher matching degree with the target size range in the knowledge mapping base; 3. Retain the original candidate region generated based on text confidence. One of these criteria is selected for retention, and the other is removed.
[0043] S150: Using the trained object detection model, perform a first round of detection on each candidate region in the candidate region list, and perform a second round of detection on the image keyframe or the region in the image keyframe other than the candidate region, to obtain the candidate region detection results and the full image detection results respectively.
[0044] Specifically, S150 includes: during the first round of detection for each candidate region in the candidate region list, restricting the predicted category of the trained target detection model to the target category corresponding to the candidate region, thus obtaining a target detection model with locked categories; using the locked target detection model to detect the candidate region, so as to output a detection box that meets the first preset condition as the candidate region detection result; wherein, the first preset condition includes: the confidence of the detection box is higher than the local confidence threshold, and the size of the bounding box does not exceed the preset size range; based on the number of target categories without text association in the knowledge mapping base, selecting to perform full-image detection on the image keyframe, or only to detect the regions other than the candidate regions in the image keyframe, so as to output a detection box that meets the second preset condition as the full-image detection result; wherein, the second preset condition includes: the confidence of the detection box is higher than the global confidence threshold; the global confidence threshold is less than the local confidence threshold.
[0045] For example, the global confidence threshold is set to 0.5, and the local confidence threshold ranges from 0.7 to 0.8. It should be noted that the values of the global and local confidence thresholds can be changed according to actual needs. Furthermore, if the detection box size exceeds the target size range by ±20%, it is considered an abnormal detection box and is removed.
[0046] Here, the predicted categories of the trained object detection model are restricted to the target category corresponding to the candidate region, resulting in a category-locked object detection model. This can be understood as significantly reducing the search space of the model within each candidate region by utilizing prior knowledge. Specifically, the system constructs a temporary classification space containing only a single target category for each candidate region based on the association relationships established by the knowledge mapping base. During model inference, category locking is achieved through an output layer filtering mechanism. This manifests as selective masking of the model's predicted vectors: only the confidence channels related to the target category corresponding to the current candidate region are retained, while the predicted scores of all other categories are forced to zero. This operation essentially reduces the hypothesis space of the detection problem from exponential complexity to linear complexity. The model only needs to complete a binary decision of "yes / no" for the specific category within each candidate region, rather than a traditional multi-class classification task, fundamentally avoiding false detections caused by inter-class confusion.
[0047] Here, a target category without text association refers to a target category in the knowledge mapping library whose standard text set field is empty or contains a specific null value identifier; when the number of target categories without text association is greater than or equal to a preset value, full-image detection is performed on the image keyframe; when the number of target categories without text association is less than a preset value, detection is performed on the region in the image keyframe excluding the candidate region.
[0048] For example, since approximately 15% of the targets in the software interface are without text, a full-image detection method is used, with a global confidence threshold of 0.5. The focus is on capturing targets without text (such as separator lines) and supplementing any missed targets with text. The detection boxes for the "separator line" (coordinates (50, 100, 1870, 102)) with a confidence score of 0.65 and the "undo button" with a confidence score of 0.58 are both retained.
[0049] The training process of the object detection model will now be described.
[0050] In one possible implementation, the object detection model is the YOLO model. This YOLO model is trained as follows: Image data containing various screen interface elements in different application scenarios are collected and labeled to construct a training sample set. For screen interface elements with text association, the associated standard text content and standard text region coordinates are labeled. For screen interface elements without text association, special identifiers of unrelated text are labeled. Constructing the original object detection loss function of the YOLO model And text-target position alignment loss function Total loss function: ; in, For hyperparameters; Using the training sample set and optimizing the total loss function, the initial YOLO model is iteratively trained to obtain a well-trained object detection model.
[0051] For example, text-space alignment loss The calculation method is as follows: For a training sample, if it contains a text with association... visual target Remember visual targets The coordinates of the center point of the predicted bounding box are Record the text in this sample. The coordinates of the center point of the true bounding box are .and then, Defined as and The Euclidean distance between them is: .
[0052] For example, screenshots of different functional interfaces of the target software were collected, covering all core modules, including text targets (buttons, input boxes) and non-text targets (icons, separators), in a ratio of approximately 8:2. LabelStudio was used to annotate each image with: basic information: target bounding box (x1, y1, x2, y2), target category (e.g., "Confirm button," "File dialog box"); associated text information: standard text corresponding to text targets (e.g., "Confirm button" labeled "Confirm"), text region coordinates; and "No associated text" for non-text targets. Data partitioning: the data was divided proportionally into training, validation, and test sets. Here, a batch size of 32-64, an initial learning rate of 0.001 (cosine annealing decay), and 100-150 training iterations were used, with GPU acceleration for training. Before each training iteration, one or more data augmentation processes were performed on the training sample set, including random cropping, brightness adjustment, and minor text position shifting. For example, during training, strategies such as random cropping (cropping ratio 0.7-1.0), brightness adjustment (brightness change ±15%), small text position shifts (offset ±3 pixels), and horizontal flipping (probability 0.5) are employed to improve model robustness. After setting appropriate training parameter configurations, the optimal model is selected and parameters are saved based on precision and recall.
[0053] S160: The candidate region detection results and the full image detection results are fused and deduplicated to output the final screen interface element detection results.
[0054] Specifically, S160 includes: merging the candidate region detection results and the full image detection results into an initial detection result set; performing deduplication processing on the detection boxes in the initial detection result set to obtain multiple deduplicated detection boxes as the final screen interface element detection results; wherein, the deduplication processing includes: traversing the initial detection result set, and when two detection boxes have the same predicted category and their bounding box intersection-union ratio is greater than or equal to a preset overlap threshold, they are determined to be duplicate detection boxes; and filtering and retaining the detection box with the highest confidence among the two.
[0055] To address the shortcomings of existing screen interface element detection methods, such as missed detections of targets without text, false detections of targets with text, low detection efficiency, and insufficient ability to distinguish similar targets, this invention provides a screen interface element detection method based on knowledge guidance and multimodal fusion. By constructing a knowledge mapping library, prior domain knowledge is transformed into structured detection constraints, establishing a new detection paradigm that deeply integrates knowledge guidance and multimodal information. First, text is extracted using OCR and matched with the knowledge library to generate valid text. Then, precise candidate regions are generated by combining text position rules and target size ranges. A two-stage strategy combining candidate region-oriented detection and full-image supplementation detection is then employed. Finally, the detection results are output through fusion and deduplication. This innovative architecture not only significantly reduces the missed detection rate of targets without text and the false detection rate of targets with text, but also greatly improves processing speed by optimizing the detection area. Furthermore, knowledge guidance enhances the model's ability to distinguish complex interface layouts and similar targets, providing a comprehensive and effective solution for screen interface element detection.
[0056] Simulation software was used to verify the technical effects of the present invention. To verify the effectiveness of the algorithm of the present invention, two methods were selected for comparison: traditional YOLOv8x (relying only on visual features, without OCR and knowledge guidance) and YOLOv8x+OCR fusion (fusing only visual and text features, without manual knowledge guidance). Table 2 shows the experimental results of the three methods on the software test dataset.
[0057] Table 2
[0058] As shown in Table 2, the precision of the method of this invention reaches 94.9%, and the recall reaches 93.5%. Compared with Method 1, the precision is improved by 9.7 percentage points, and the recall is improved by 11.4 percentage points. This is mainly due to the prior knowledge provided by the knowledge mapping library, which effectively reduces false positives and false negatives through "category locking" and "text position constraints". Furthermore, the detection frame rate of the proposed method is 42.6 FPS, higher than the two comparative methods, meeting the application requirements of real-time video processing (≥30 FPS). This indicates that the introduction of knowledge guidance and multimodal fusion improves accuracy without sacrificing processing speed; in fact, the candidate region mechanism reduces redundant computation, thereby optimizing efficiency. Moreover, in complex scenarios with interference from similar targets, the false positive rate of the method of this invention is only 3.2%, far lower than the 10.5% of Method 1 and the 7.8% of Method 2. This fully demonstrates that using knowledge from manuals, standards, and other fields for guidance can significantly enhance the model's ability to distinguish similar targets and improve detection robustness.
[0059] In summary, this invention demonstrates significant advantages in detection accuracy, efficiency, and robustness through knowledge-guided and multimodal fusion, verifying the effectiveness and advancement of the proposed technical solution.
[0060] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.
Claims
1. A method for detecting screen interface elements based on knowledge guidance and multimodal fusion, characterized in that, include: Acquire keyframes of images in the target scene; the keyframes refer to image frames selected from image data based on changes in image content. Using an OCR text recognition algorithm, the text information of the keyframes of the image is extracted and a text list is output. The text list contains one or more text entries, and each text entry contains a text content, the region coordinates corresponding to the text content, and the text confidence score. Using a pre-built knowledge mapping library, text entries in the text list that meet preset conditions are filtered to obtain text entries with a matching degree greater than or equal to a matching threshold, thus forming a valid text list. The knowledge mapping library is constructed using standard documents related to the target scenario, including user manuals, design specifications, or development documents. The knowledge mapping library contains one or more knowledge entries, each corresponding to a target category of a screen interface element, as well as the standard text set, text position rules, and target size range for that target category. Based on the region coordinates of each valid text in the list of valid texts, and combined with the text position rules and target size range of the corresponding target category in the knowledge mapping library, a corresponding candidate region is calculated and generated to obtain a candidate region list. Using the trained target detection model, a first round of detection is performed on each candidate region in the candidate region list, and a second round of detection is performed on the image keyframe or the regions in the image keyframe other than the candidate regions, to obtain the candidate region detection results and the full image detection results, respectively. The candidate region detection results and the full image detection results are fused and deduplicated to output the final screen interface element detection results.
2. The screen interface element detection method based on knowledge guidance and multimodal fusion according to claim 1, characterized in that, The filtering includes: Filter out text entries with a confidence level lower than a preset confidence threshold to obtain multiple filtered text entries; The multiple filtered text entries are fuzzily matched with the standard text set in the knowledge mapping base to generate multiple matching degrees that correspond one-to-one; wherein, the fuzzy matching refers to using a string similarity calculation method based on edit distance to calculate the similarity between the text content in each filtered text entry and each text string in the standard text set.
3. The screen interface element detection method based on knowledge guidance and multimodal fusion according to claim 1, characterized in that, The process involves calculating and generating a corresponding candidate region based on the region coordinates of each valid text in the valid text list, combined with the text position rules and target size range of the corresponding target category in the knowledge mapping base, resulting in a candidate region list, including: Perform the following operations for each valid text: Based on the region coordinates of the valid text, determine the geometric center coordinates, width, and height of its text bounding box; Based on the text position rules of the target category corresponding to the valid text in the knowledge mapping library, determine the direction and offset of pixel expansion, and determine the expected target width and expected target height based on the corresponding target size range; Based on the difference between the expected target width and the width of the text bounding box, and the difference between the expected target height and the height of the text bounding box, the required expansion pixel size is determined to generate a rectangular candidate region that completely covers the screen interface element to be detected corresponding to the valid text. All rectangular candidate regions are aggregated to form the candidate region list.
4. The screen interface element detection method based on knowledge guidance and multimodal fusion according to claim 3, characterized in that, After obtaining the candidate region list, the following is also included: When the overlap between any two candidate regions is greater than or equal to a preset overlap threshold, or the adjacent distance between them is less than or equal to a preset distance threshold, a region merging or filtering operation is performed to ensure that each candidate region in the candidate region list corresponds to an independent screen interface element to be detected.
5. The screen interface element detection method based on knowledge guidance and multimodal fusion according to claim 1, characterized in that, The trained object detection model is used to perform a first round of detection on each candidate region in the candidate region list, and a second round of detection is performed on the image keyframe or the regions in the image keyframe excluding the candidate regions, to obtain candidate region detection results and full-image detection results, including: When performing the first round of detection for each candidate region in the candidate region list, the predicted category of the trained target detection model is restricted to the target category corresponding to that candidate region, thus obtaining a target detection model with category locking. The locked target detection model is used to detect the candidate region, and a detection box that meets the first preset condition is output as the candidate region detection result; wherein, the first preset condition includes: the confidence of the detection box is higher than the local confidence threshold, and the size of the bounding box does not exceed the preset size range; Based on the number of target categories without text association in the knowledge mapping base, the system selects to perform full-image detection on the image keyframes, or to detect only the regions in the image keyframes other than the candidate regions, so as to output detection boxes that meet the second preset conditions as the full-image detection result; wherein, the second preset conditions include: the confidence of the detection box is higher than the global confidence threshold; the global confidence threshold is less than the local confidence threshold.
6. The screen interface element detection method based on knowledge guidance and multimodal fusion according to claim 5, characterized in that, The target category without text association refers to the target category in the knowledge mapping library whose standard text set field is empty or contains a specific null value identifier; When the number of target categories without text association is greater than or equal to a preset value, full-image detection is performed on the keyframes of the image. When the number of target categories without text association is less than the preset value, the regions other than the candidate regions in the image keyframe are selected for detection.
7. The screen interface element detection method based on knowledge guidance and multimodal fusion according to claim 1, characterized in that, The process of fusing and deduplicating the candidate region detection results and the full-image detection results to output the final screen interface element detection results includes: The candidate region detection results and the full map detection results are merged into an initial detection result set; The detection boxes in the initial detection result set are deduplicated to obtain multiple deduplicated detection boxes, which are used as the final screen interface element detection results. The deduplication process includes: traversing the initial detection result set; when two detection boxes have the same predicted category and their bounding box intersection-union ratio is greater than or equal to a preset overlap threshold, they are determined to be duplicate detection boxes; and the detection box with the highest confidence among the two is selected and retained.
8. The screen interface element detection method based on knowledge guidance and multimodal fusion according to claim 1, characterized in that, The knowledge mapping library is constructed in the following way: Extract and define multiple target categories from the aforementioned specification document, with each target category corresponding to a screen interface element; For each target category, perform the following operations to generate its structured knowledge entries: Extract one or more related texts from the specification document to form a standard text set for the target category; Define the text position rules of the associated text within the corresponding screen interface element to be detected. The text position rules are used to limit the relative positional relationship between the associated text and the screen interface element to be detected, including the alignment method and the offset from the edge of the element. Calculate the target size range of the target category at a specified screen resolution; The target size range is used to define the pixel range of the width and height of the screen interface element to be detected; All knowledge entries for all target categories are stored in a structured format to construct the knowledge mapping library.
9. The screen interface element detection method based on knowledge guidance and multimodal fusion according to claim 1, characterized in that, The object detection model is a YOLO model, which is trained in the following way: Image data containing various screen interface elements in different application scenarios are collected and labeled to construct a training sample set. Among them, for screen interface elements with text association, the associated standard text content and standard text area coordinates are labeled, and for screen interface elements without text association, special identifiers of unrelated text are labeled. Construct a loss function that integrates the original object detection function of the YOLO model. And text-target position alignment loss function Total loss function: ； in, For hyperparameters; Using the training sample set and with the total loss function as the optimization objective, the initial YOLO model is iteratively trained to obtain the trained object detection model.
10. The screen interface element detection method based on knowledge guidance and multimodal fusion according to claim 9, characterized in that, Before each training session, one or more data augmentation processes are performed on the training sample set, including random cropping, brightness adjustment, and slight text position shifting.