Mobile application user interface component detection method based on improved YOLOv8
By introducing a hybrid structure module and multiple attention mechanisms into the YOLOv8 network, combined with an adaptive threshold focus loss function, the problems of imprecise multi-scale feature fusion and unreasonable attention allocation in UI component detection in the existing technology are solved, achieving UI component detection with higher accuracy and robustness.
Patent Information
- Application Number
- CN202510715507.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-30
- Publication Date
- 2025-09-12
AI Technical Summary
Existing mobile application user interface component detection methods based on YOLOv8 have problems such as imprecise multi-scale feature fusion, unreasonable attention allocation, and high false detection rate caused by sample imbalance when processing UI components.
By introducing a hybrid structure module into the bottleneck layer of the feature pyramid layer C2f of the YOLOv8 network, combined with multi-scale parallel convolution and multiple attention mechanisms, the multi-scale feature extraction and fusion capabilities are enhanced. Layered detection is performed through five detection heads. At the same time, an adaptive threshold focal loss function is used to optimize sample weight distribution.
The accuracy and robustness of UI component detection have been significantly improved, especially the detection performance in small targets and complex scenarios, the false detection rate has been reduced, and the real-time performance of the model has been maintained.
Smart Images
Figure CN120635546A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision and deep learning technologies, and in particular to a mobile application user interface component detection method based on improved YOLOv8. Background Art
[0002] With the rapid development of the mobile internet, mobile app user interfaces (UIs) are becoming increasingly diverse. Accurately detecting components like buttons, text boxes, and icons within these interfaces is crucial for automated testing, interface optimization, and accessibility. YOLOv8, as an efficient real-time object detection algorithm, provides a technical foundation for UI component detection. However, this scenario presents significant challenges: UI components vary widely in size, shape, and visual features, and are easily affected by dense backgrounds. Traditional detection models are prone to missed or false positives.
[0003] Existing detection methods based on YOLOv8 still have limitations when dealing with UI components. For example, the traditional bottleneck layer in the feature pyramid network has a coarse granularity for the fusion of multi-scale features, making it difficult to effectively capture the detailed information of small-sized components; the conventional attention mechanism lacks targeted modeling of the unique spatial distribution of UI components (such as linearly arranged navigation bar components) and visual patterns (such as the contrast characteristics of text and background), resulting in insufficient attention allocation to key component areas. In addition, some lightweight improvement methods sacrifice feature expression capabilities in pursuit of speed and cannot meet the high-precision requirements of UI component detection. Therefore, improving the YOLOv8 network structure based on the characteristics of mobile application UI components and improving the model's adaptability to complex scenarios have become technical issues that need to be urgently addressed. Summary of the Invention
[0004] The purpose of the present invention is to provide a mobile application user interface component detection method based on improved YOLOv8, which can comprehensively improve the accuracy of component detection in mobile application user interfaces and solve problems such as imprecise multi-scale feature fusion, unreasonable attention allocation, and high false detection rate caused by sample imbalance in mobile application UI component detection.
[0005] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is as follows: the present invention provides a mobile application user interface component detection method based on improved YOLOv8, comprising the following steps:
[0006] Step S1: Preprocess the RICO mobile application dataset, including extracting component annotation information from the original data, data cleaning and format conversion, and dividing it into training set, validation set and test set according to the preset ratio;
[0007] Step S2: Improve the YOLOv8 network and introduce a hybrid structure module into the bottleneck layer of the feature pyramid layer C2f of its backbone network. The hybrid structure module enhances the multi-scale feature extraction and fusion capabilities through parallel multi-scale convolution and multiple attention mechanisms, and generates an optimized feature pyramid layer.
[0008] Step S3: Based on the optimized feature pyramid layer, multiple detection heads are constructed. Each detection head is connected to the C2f module in the feature pyramid layer. Hierarchical detection of user interface components of different sizes is achieved through multi-level feature fusion.
[0009] Step S4: introducing an adaptive threshold focus loss function into the loss function, and optimizing the sample weight distribution by dynamically calculating the threshold and using a dual-branch modulation strategy;
[0010] Step S5: Use the preprocessed RICO dataset to train the improved YOLOv8 network to generate a user interface component detection model.
[0011] Preferably, step S1 specifically includes:
[0012] S11: Extract component annotation box information from the mobile application user interface hierarchical structure information provided by the RICO dataset and convert the dataset into a VOC format dataset;
[0013] S12: Parse the XML annotation file for data cleaning. Based on the component type, locate the actual component area through color difference, edge detection, and texture feature analysis, and verify the annotation box offset.
[0014] S13: Convert the cleaned dataset into the Yolo dataset format and divide it into training set, validation set and test set in a ratio of 7:2:1.
[0015] Preferably, step S2 specifically includes:
[0016] S21. Construct a multi-scale parallel convolution module. The feature tensor ([B, C, H, W]) of the feature pyramid layer in the YOLOv8 network is first batch normalized, and then initially transformed through 1×1 convolution and 5×5 convolution. The tensor is then input into three parallel deep dilated convolution layers. Different sizes of convolution kernels and dilation rates are combined with reflection padding for feature extraction. The three convolution results are spliced in the channel dimension, and after the features are fused by the multi-layer perceptron, the first residual connection is performed with the input features.
[0017] The feature tensor [B, C, H, W] represents the dimension of the input feature tensor, B: batch size (such as 1, 8, 16, etc., depending on the training configuration), C: number of input channels (such as 64, 128, 256, 512, 1024, corresponding to the feature map channels of different layers of YOLOv8), H, W: feature map size (such as 640×640, 320×320, 160×160, etc., decreasing as the network layer deepens).
[0018] S22. Construct a parallel attention mechanism module. After batch normalization, the output of the multi-scale parallel convolution module is diverted to four parallel attention mechanisms: simple pixel attention, channel attention, pixel attention, and coordinate attention to generate weighted features. The outputs of the four attention mechanisms are spliced in the channel dimension, and after the features are fused by the multi-layer perceptron, a second residual connection is performed with the output of the first stage.
[0019] S23, add residual to construct a hybrid structure module, perform 1×1 convolution transformation on the original input features, perform a third residual connection on the transformed original input and the second stage output to obtain the final output features of the hybrid structure module;
[0020] S24. Introduce a hybrid structure module to form a new feature pyramid layer. In the bottleneck layer Bottleneck of the feature pyramid layer C2f of the YOLOv8 network, a hybrid structure module including a multi-scale parallel convolution module and a parallel attention mechanism module is introduced to form an optimized feature pyramid layer.
[0021] Preferably, in step S21, the three parallel depth-dilated convolutional layers are DWConv7, DWConv13 and DWConv19, which respectively use 7×7, 5×5, 3×3 convolution kernels and different dilation rates of 1, 3, and 3.
[0022] Preferably, step S3 specifically includes:
[0023] S31: Upsample the feature map of the last layer in the backbone network, concatenate the upsampled features with the layer of specified size and index in the backbone network along the channel dimension, process them through the C2f module of multiple specified channels to form feature output, and repeat this upsampling and concatenation operation multiple times;
[0024] S32: Construct the first detection head branch, concatenate the output of the last upsampling with the layer of specified size and index in the backbone network along the channel dimension, process it through the C2f module of multiple layers of specified channels to form feature output and directly construct the first detection head;
[0025] S33: Based on the feature output of the first detection head branch, the second, third, fourth and fifth detection head branches are constructed in sequence by downsampling and concatenating them with the layers of specified size and index in the backbone network along the channel dimension. After being processed by the C2f module of multiple layers of specified channels, the second, third, fourth and fifth detection head branches are constructed in sequence.
[0026] Preferably, the feature map of the last layer of the backbone network is upsampled four times step by step, and after each upsampling, it is spliced with the feature map of the corresponding layer of the backbone network. After each splicing, the feature is enhanced by a 3-layer C2f module, and the number of channels is halved step by step from 2048 → 1024 → 512 → 256 → 128. The upsampling layers correspond to the feature layers with backbone network indices of 9, 7, 5, and 3.
[0027] Preferably, the first detection head is constructed based on the output of the fourth upsampling, and is spliced with the deep features of the backbone network through step-by-step downsampling, with the index numbers being 21, 18, 15, and 12 respectively; a four-level downsampling detection branch is formed, and each downsampling branch is processed by a 3-layer C2f module, with the number of channels doubling step by step from 128 to 256 to 512 to 1028 to 2048.
[0028] Preferably, step S4 specifically includes:
[0029] S41. Use nn.BCEWithLogitsLoss as the basic loss function to calculate the basic binary cross entropy loss between the predicted value and the true label;
[0030] S42, convert the output of the detection head into a probability value through the sigmoid function and calculate the prediction confidence p_t of each sample;
[0031] S43. Dynamically calculate the threshold based on the sample confidence of the current batch and historical batches, specifically:
[0032] Calculate the average confidence level of the current batch of samples, maintain a list of historical confidence level averages, and use the exponential moving average method to combine the current average and the historical average to update the threshold;
[0033] S44. Apply different modulation factors to the high-confidence samples and the low-confidence samples according to the threshold, specifically:
[0034] For samples with confidence higher than 0.5, the modulation factor is (1.000001-p_t)^γ,
[0035] For samples with confidence less than or equal to 0.5, the modulation factor is (1.5-p_t)^(-log(p_t)), where γ is dynamically determined by the threshold;
[0036] S45. Multiply the modulation factor by the basic loss to obtain the final loss value.
[0037] Compared with the prior art, the present invention has the following beneficial effects:
[0038] 1. A hybrid structure module is introduced in the C2f bottleneck layer of YOLOv8, integrating multi-scale parallel convolution (7×7, 5×5, and 3×3 convolution kernels with dilation rates) and reflection padding, combining deep dilated convolution with residual connections. By using different convolution kernels and dilation rates, it effectively captures the multi-scale features of UI components, avoiding the loss of detailed information in traditional bottleneck layers. The reflection padding strategy reduces feature map boundary distortion, improves the detection accuracy of edge components, enhances network stability, alleviates the gradient vanishing problem, and improves the model's adaptability to complex layouts.
[0039] 2. Four attention mechanisms are introduced in parallel in the hybrid structure module and the outputs are fused through a multi-layer perceptron. Channel attention focuses on global semantics, pixel attention captures local details, and coordinate attention enhances position features, optimizing the spatial distribution and visual patterns of UI components. A parallel mechanism is used to assign higher attention weights to key component areas to reduce false detections caused by background interference.
[0040] 3. Five detection heads are connected to the C2f layer of the neck network, and hierarchical detection is achieved through upsampling, downsampling and feature splicing. Different detection heads correspond to different feature levels, covering the full range of targets from tiny icons to large-sized components. Through multi-level feature fusion, the ability to distinguish densely arranged components is enhanced. The hierarchical detection mechanism reduces the problem of missed detection of small targets due to the coarse granularity of the feature pyramid.
[0041] 4. Balance the loss weights of easy samples and difficult samples through dynamic modulation factors to reduce false detection of background noise and missed detection of small targets.
[0042] 5. We correct annotation offsets through color difference, edge detection, texture analysis, and other methods, and filter abnormal annotations in special areas such as the navigation bar. The cleaned dataset contains 16 types of UI components (such as buttons, text boxes, progress bars, etc.), reducing annotation errors and making model training more stable. We balance accuracy and speed through adaptive loss functions and hybrid structure modules, improving mAP while maintaining the real-time performance of YOLOv8.
[0043] In summary: This solution significantly improves the accuracy (especially for small targets and complex scenarios) and robustness of UI component detection through four core improvements: multi-scale feature enhancement, attention mechanism optimization, multi-level detection head design, and dynamic loss function. While also taking into account real-time requirements, it is suitable for high-precision scenarios such as automated testing and barrier-free access. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.
[0045] Figure 1 A flowchart of a method for detecting user interface components of mobile applications based on improved YOLOv8 provided by an embodiment of the present invention;
[0046] Figure 2 1 is a structural diagram of a Mix Structure Block provided in an embodiment of the present invention;
[0047] Figure 3 This is a diagram of a bottleneck layer structure that introduces a hybrid structure module provided by an embodiment of the present invention;
[0048] Figure 4 This is a C2f structure diagram of a feature pyramid layer that introduces a hybrid structure module according to an embodiment of the present invention;
[0049] Figure 5 is a flow chart of an adaptive threshold focus loss function (ATFL) provided in an embodiment of the present invention;
[0050] Figure 6 This is a diagram of the improved YOLOv8 network structure provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0051] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. Of course, the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.
[0052] like Figure 1 As shown, an embodiment of the present invention provides a mobile application user interface component detection method based on improved YOLOv8, comprising the following steps:
[0053] Step S1: Preprocess the RICO mobile application dataset, including extracting component annotation information from the raw data, data cleaning and format conversion, and dividing the dataset into training, validation, and test sets according to preset proportions. Specifically, the following steps are performed:
[0054] S11: Extract the component annotation box information from the mobile application user interface hierarchical structure information (.json file) provided by the RICO dataset and convert the dataset into a VOC format dataset;
[0055] S12: Parse the XML annotation file for data cleaning. Based on the component type, locate the actual component area through color difference, edge detection, and texture feature analysis. Consider special cases such as whether the annotation box is located in the navigation bar area to determine whether the annotation is offset.
[0056] S13: Convert the cleaned dataset into the Yolo dataset format and divide it into training set, validation set and test set in a ratio of 7:2:1.
[0057] The preprocessed RICO dataset includes 16 categories: IMAGE image, PICTOGRAM graphic symbol, BUTTON button, TEXT text, TEXT_INPUT text input, MAP map, CHECK_BOX checkbox, SWITCH switch selector, PAGER_INDICATOR paging carousel, SLIDER sliding unit, RADIO_BUTTON radio button, SPINNER drop-down unit, PROGRESS_BAR progress bar, ADVERTISEMENT advertisement, DATE_PICKER date selector, NUMBER_STEPPER digital stepper, etc., with a total of 3296 training sets and 470 test set data, which are used for YOLO target detection series training.
[0058] Step S2: Improve the YOLOv8 network by introducing a hybrid structure module into the bottleneck layer of the feature pyramid layer C2f of its backbone network. The hybrid structure module enhances the multi-scale feature extraction and fusion capabilities through parallel multi-scale convolution and multiple attention mechanisms, and generates an optimized feature pyramid layer. Specifically, it includes:
[0059] S21, such as Figure 2 As shown in the figure, a multi-scale parallel convolutional structure is constructed. The feature tensor ([B, C, H, W]) of the feature pyramid layer in the YOLOv8 network is first normalized through a batch normalization layer. Then, a preliminary feature transformation is performed through 1×1 convolution and 5×5 convolution. The features are then input into three parallel deep dilated convolutional layers, namely DWConv7, DWConv13, and DWConv19. Different convolution kernel sizes (7×7, 5×5, 3×3) and dilation rates (dilation=1, 3, 3) are used for feature extraction. Reflection padding (padding_mode='reflect') is used to maintain feature map boundary information and avoid boundary distortion. The three convolution results are spliced in the channel dimension, feature fusion is performed through a multi-layer perceptron, and the first residual connection is performed with the input features.
[0060] The feature tensor [B, C, H, W] represents the dimension of the input feature tensor, B: batch size (such as 1, 8, 16, etc., depending on the training configuration), C: number of input channels (such as 64, 128, 256, 512, 1024, corresponding to the feature map channels of different layers of YOLOv8), H, W: feature map size (such as 640×640, 320×320, 160×160, etc., decreasing as the network layer deepens).
[0061] S22. Construct a parallel attention mechanism structure. The output of the multi-scale parallel convolution module is first subjected to a batch normalization layer. Then the features are diverted to four parallel attention modules: simple pixel attention, channel attention, pixel attention, and coordinate attention mechanism. Weighted features are generated. The outputs of the four attention mechanisms are spliced in the channel dimension, and the features are fused through a multi-layer perceptron. A second residual connection is performed with the output of the first stage.
[0062] S23. Add the residual to construct the hybrid structure module, perform a 1×1 convolution transformation on the original input features, perform a third residual connection between the transformed original input and the second stage output, and generate the final output features of the hybrid structure module.
[0063] S24, such as Figure 3 As shown in Figure 2, a mixed structure module Mix Structure Block is introduced into the bottleneck layer Bottleneck of the feature pyramid layer C2f of the YOLOv8 network. The mixed structure module Mix Structure Block includes a multi-scale parallel convolution module and a parallel attention mechanism module, which ultimately constitute the C2f_MixStructure feature pyramid layer. Figure 4 shown.
[0064] Step S3: Build five detection heads based on the optimized feature pyramid layer. Each detection head is connected to the C2f module in the feature pyramid layer. Multi-level feature fusion is used to achieve hierarchical detection of user interface components of different sizes.
[0065] like Figure 6 As shown in the figure, five detection heads are used for target detection. The specific contents are as follows:
[0066] S31, Neck upsampling 1: Upsample the last layer of 20×20×2048 feature maps in the backbone network, concatenate the upsampled features with the layer with index 9 and size 40×40×1024 in the backbone network along the channel dimension, and process them through a 3-layer 1024-channel C2f module to form the feature output;
[0067] S32, Neck Upsampling 2: Continue upsampling the output of Neck Upsampling 1, concatenate the upsampled features with the layer with index 7 and size 80×80×512 in the backbone network along the channel dimension, and process them through a 3-layer 512-channel C2f module to form the feature output;
[0068] S33, Neck Upsampling 3: Upsample the output of Neck Upsampling 2 again, concatenate the upsampled features with the layer with index 5 and size 160×160×256 in the backbone network along the channel dimension, and process them through a 3-layer 256-channel C2f module to form the feature output;
[0069] S34, neck upsampling 4 and building the first detection head branch: Upsample the output of neck upsampling 3, concatenate the upsampled features with the layer with index 3 and size 320×320×128 in the backbone network along the channel dimension, process them through a 3-layer 128-channel C2f module, form feature output and directly build the first detection head;
[0070] S35. Construct the second detection head branch: downsample the features after C2f in the neck upsampling 4, concatenate them with the layer with index 21 along the channel dimension, and process them through 3 layers of 256-channel C2f modules to form the feature output of the detection head;
[0071] S36. Construct the third detection head branch: downsample the output features of the second detection head branch, concatenate them with the layer with index 18 along the channel dimension, and process them through a 3-layer 512-channel C2f module to form the feature output of this detection head;
[0072] S37. Construct the fourth detection head branch: Continue to downsample the output features of the third detection head branch, concatenate them with the layer with index 15 along the channel dimension, and process them through a 3-layer 1024-channel C2f module to form the feature output of this detection head;
[0073] S38. Construct the fifth detection head branch: downsample the output features of the fourth detection head branch again, concatenate them with the layer with index 12 along the channel dimension, and process them through a 3-layer 2048-channel C2f module to form the feature output of the detection head.
[0074] Step S4: Introduce the adaptive threshold focus loss function into the loss function, and optimize the sample weight distribution by dynamically calculating the threshold and the dual-branch modulation strategy; see Figure 5 , the specific steps are as follows:
[0075] S41. Use nn.BCEWithLogitsLoss as the basic loss function to calculate the basic binary cross entropy loss between the predicted value and the true label;
[0076] S42, convert the output of the detection head into a probability value through the sigmoid function and calculate the prediction confidence p_t of each sample;
[0077] S43. Dynamically calculate the threshold based on the sample confidence of the current batch and historical batches, specifically:
[0078] Calculate the average confidence level of the current batch of samples, maintain a list of historical confidence level averages, and use the exponential moving average method to combine the current average and the historical average to update the threshold;
[0079] S44. Apply different modulation factors to the high-confidence samples and the low-confidence samples according to the threshold, specifically:
[0080] For samples with confidence higher than 0.5, the modulation factor is (1.000001-p_t)^γ,
[0081] For samples with confidence less than or equal to 0.5, the modulation factor is (1.5-p_t)^(-log(p_t)), where γ is dynamically determined by the threshold;
[0082] S45. Multiply the modulation factor by the basic loss to obtain the final loss value.
[0083] Step S5: Use the preprocessed RICO dataset to train the improved YOLOv8 network to generate a user interface component detection model. The specific steps are as follows:
[0084] S51. Create a data set mydata.yaml file, which includes the training set and validation set addresses in the data set as well as the target detection categories and their numbers;
[0085] S52, according to Figure 6 Improved YOLOv8 network structure, modify the model configuration file ultralytics / cfg / models / v8 / yolov8.yaml;
[0086] S53. Create the train.py file and modify the corresponding hyperparameters to train the target detection model.
[0087] This method improves upon the YOLOv8 network structure by introducing a hybrid structure module into the bottleneck layer of the YOLOv8 network feature pyramid layer C2f, enhancing the model's ability to process multi-scale features of mobile application UI components. Five detection heads are connected to the C2f layer of the neck network to achieve layered detection, comprehensively covering all types of UI components from small icons to large buttons, and reducing missed detections. Adaptive threshold focus loss is introduced, dynamically calculating thresholds, down-weighting high-confidence samples and up-weighting low-confidence samples, effectively addressing sample imbalance, reducing false detection and missed detection rates, and improving the model's learning ability and adaptability to difficult samples in complex scenarios.
[0088] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A mobile application user interface component detection method based on improved YOLOv8, characterized in that: The following steps are involved: Step S1: Preprocess the RICO mobile application dataset, including extracting component annotation information from the original data, data cleaning and format conversion, and dividing it into training set, validation set and test set according to the preset ratio; Step S2: Improve the YOLOv8 network and introduce a hybrid structure module into the bottleneck layer of the feature pyramid layer C2f of its backbone network. The hybrid structure module enhances the multi-scale feature extraction and fusion capabilities through parallel multi-scale convolution and multiple attention mechanisms, and generates an optimized feature pyramid layer. Step S3: Based on the optimized feature pyramid layer, multiple detection heads are constructed. Each detection head is connected to the C2f module in the feature pyramid layer. Hierarchical detection of user interface components of different sizes is achieved through multi-level feature fusion. Step S4: introducing an adaptive threshold focus loss function into the loss function, and optimizing the sample weight distribution by dynamically calculating the threshold and using a dual-branch modulation strategy; Step S5: Use the preprocessed RICO dataset to train the improved YOLOv8 network to generate a user interface component detection model.
2. A mobile application user interface component detection method based on improved YOLOv8 according to claim 1, characterized in that: Step S1 specifically includes: S11: Extract component annotation box information from the mobile application user interface hierarchical structure information provided by the RICO dataset and convert the dataset into a VOC format dataset; S12: Parse the XML annotation file for data cleaning. Based on the component type, locate the actual component area through color difference, edge detection, and texture feature analysis, and verify the annotation box offset. S13: Convert the cleaned dataset into the Yolo dataset format and divide it into training set, validation set and test set in a ratio of 7:2:
1.
3. The mobile application user interface component detection method based on improved YOLOv8 according to claim 1, characterized in that: Step S2 specifically includes: S21. Construct a multi-scale parallel convolution module. First, batch normalize the feature tensor of the feature pyramid layer in the YOLOv8 network, perform preliminary feature conversion through 1×1 convolution and 5×5 convolution, and then input it into three parallel deep dilated convolution layers. Use convolution kernels of different sizes and dilation rates combined with reflection padding for feature extraction. The three convolution results are spliced in the channel dimension, and after the features are fused by the multi-layer perceptron, the first residual connection is performed with the input features. S22. Construct a parallel attention mechanism module. After batch normalization, the output of the multi-scale parallel convolution module is diverted to four parallel attention mechanisms: simple pixel attention, channel attention, pixel attention, and coordinate attention to generate weighted features. The outputs of the four attention mechanisms are spliced in the channel dimension, and after the features are fused by the multi-layer perceptron, a second residual connection is performed with the output of the first stage. S23, add residual to construct a hybrid structure module, perform 1×1 convolution transformation on the original input features, perform a third residual connection on the transformed original input and the second stage output to obtain the final output features of the hybrid structure module; S24. Introduce a hybrid structure module to form a new feature pyramid layer. In the bottleneck layer Bottleneck of the feature pyramid layer C2f of the YOLOv8 network, a hybrid structure module including a multi-scale parallel convolution module and a parallel attention mechanism module is introduced to form an optimized feature pyramid layer.
4. A mobile application user interface component detection method based on improved YOLOv8 according to claim 3, characterized in that: In step S21, the three parallel depth-dilated convolutional layers are DWConv7, DWConv13, and DWConv19, which use 7×7, 5×5, and 3×3 convolution kernels and different dilation rates of 1, 3, and 3, respectively.
5. The mobile application user interface component detection method based on improved YOLOv8 according to claim 1, characterized in that: Step S3 specifically includes: S31: Upsample the feature map of the last layer in the backbone network, concatenate the upsampled features with the layer of specified size and index in the backbone network along the channel dimension, process them through the C2f module of multiple specified channels to form feature output, and repeat this upsampling and concatenation operation multiple times; S32: Construct the first detection head branch, concatenate the output of the last upsampling with the layer of specified size and index in the backbone network along the channel dimension, process it through the C2f module of multiple layers of specified channels to form feature output and directly construct the first detection head; S33: Based on the feature output of the first detection head branch, the second, third, fourth and fifth detection head branches are constructed in sequence by downsampling and concatenating them with the layers of specified size and index in the backbone network along the channel dimension. After being processed by the C2f module of multiple layers of specified channels, the second, third, fourth and fifth detection head branches are constructed in sequence.
6. The method for detecting mobile application user interface components based on improved YOLOv8 according to claim 5, characterized in that: The feature map of the last layer of the backbone network is upsampled four times step by step. After each upsampling, it is spliced with the feature map of the corresponding layer of the backbone network. After each splicing, the feature is enhanced by a 3-layer C2f module. The number of channels is halved step by step from 2048 → 1024 → 512 → 256 → 128. The upsampling layers correspond to the feature layers with indexes 9, 7, 5, and 3 of the backbone network.
7. The method for detecting mobile application user interface components based on improved YOLOv8 according to claim 6, characterized in that: The first detection head is constructed based on the output of the fourth upsampling. It is concatenated with the deep features of the backbone network through step-by-step downsampling, with the index numbers being 21, 18, 15, and 12, respectively. Four-level downsampling detection branches are formed, and each downsampling branch is processed by a 3-layer C2f module. The number of channels increases step by step from 128 to 256 to 512 to 1028 to 2048.
8. The method for detecting mobile application user interface components based on improved YOLOv8 according to claim 1, characterized in that: Step S4 specifically includes: S41. Use nn.BCEWithLogitsLoss as the basic loss function to calculate the basic binary cross entropy loss between the predicted value and the true label; S42, convert the output of the detection head into a probability value through the sigmoid function and calculate the prediction confidence p_t of each sample; S43. Dynamically calculate the threshold based on the sample confidence of the current batch and historical batches, specifically: Calculate the average confidence level of the current batch of samples, maintain a list of historical confidence level averages, and use the exponential moving average method to combine the current average and the historical average to update the threshold; S44. Apply different modulation factors to the high-confidence samples and the low-confidence samples according to the threshold, specifically: For samples with confidence higher than 0.5, the modulation factor is (1.000001-p_t)^γ, For samples with confidence less than or equal to 0.5, the modulation factor is (1.5-p_t)^(-log(p_t)), where γ is dynamically determined by the threshold; S45. Multiply the modulation factor by the basic loss to obtain the final loss value.